question: Gradient accumulation + DeepSpeed LR scheduler
context: How does gradient accumulation interact with DeepSpeed learning rate scheduling (e.g. the per-step warm-up scheduler)? Is the learning rate updated after every iteration, or only after the model weights are ultimately updated?
answer: it considers the accumulation before doing lr_scheduler_step:
  
    
      
        pytorch-lightning/pytorch_lightning/loops/epoch/training_epoch_loop.py
      
      
          Lines 387 to 390
        in
        86b177e
      
    
    
      
  
          
            
             def update_lr_schedulers(self, interval: str, update_plateau_schedulers: bool) -> None: 
          
  
          
            
                 """updates the lr schedulers based on the given interval.""" 
          
  
          
            
                 if interval == "step" and self._should_accumulate(): 
          
  
          
            
                     return