Gradient accumulation + DeepSpeed LR scheduler
How does gradient accumulation interact with DeepSpeed learning rate scheduling (e.g. the per-step warm-up scheduler)? Is the learning rate updated after every iteration, or only after the model weights are ultimately updated?
it considers the accumulation before doing lr_scheduler_step:

  
    
      pytorch-lightning/pytorch_lightning/loops/epoch/training_epoch_loop.py
    
    
        Lines 387 to 390
      in
      86b177e
    
  
  
    

        
          
           def update_lr_schedulers(self, interval: str, update_plateau_schedulers: bool) -> None: 
        

        
          
               """updates the lr schedulers based on the given interval.""" 
        

        
          
               if interval == "step" and self._should_accumulate(): 
        

        
          
                   return