track_grad_norm doesn't work when accumulate_grad_batches is greater than 1
🐛 Bug

When I set accumulate_grad_batches=2, track_grad_norm doesn't work.
It seems to me the condition for logging gradient norm can hardly be satisfied when accumulate_grad_batches > 1 and row_log_interval(default=10) > 1.
Say accumulate_grad_batches=2 and row_log_interval=2, (batch_idx+1) and batch_idx need to be even numbers at the same time which can never be met.

  
    
      pytorch-lightning/pytorch_lightning/trainer/training_loop.py
    
    
        Lines 608 to 616
      in
      868dde2
    
  
  
    

        
          
           # gradient update with accumulated gradients 
        

        
          
           if (self.batch_idx + 1) % self.accumulate_grad_batches == 0: 
        

        
          
            
        

        
          
               # track gradient norms when requested 
        

        
          
               if batch_idx % self.row_log_interval == 0: 
        

        
          
                   if self.track_grad_norm > 0: 
        

        
          
                       model = self.get_model() 
        

        
          
                       grad_norm_dic = model.grad_norm( 
        

        
          
                           self.track_grad_norm) 
        
    
  


To Reproduce
Steps to reproduce the behavior:

Set accumulate_grad_batches=2
Run train.fit
The gradient norms are not logged


Expected behavior
The gradient norm should be logged.
Environment

CUDA:

GPU:
available:         False
version:           10.2


Packages:

numpy:             1.18.4
pyTorch_debug:     False
pyTorch_version:   1.5.0
pytorch-lightning: 0.7.6
tensorboard:       2.2.0
tqdm:              4.45.0


System:

OS:                Linux
architecture:

64bit



processor:
python:            3.7.6
version:           #1 SMP Debian 4.19.118-2 (2020-04-29)