For multiple optimizers, model.toggle_optimizer() is setting the requires_grad=True for all the params.
pytorch-lightning/pytorch_lightning/core/lightning.py
    
    
        Lines 1152 to 1170
      in
      dabfeca
    
  
  
    

        
          
               def toggle_optimizer(self, optimizer: Optimizer, optimizer_idx: int): 
        

        
          
                   """ 
        

        
          
                   Makes sure only the gradients of the current optimizer's parameters are calculated 
        

        
          
                   in the training step to prevent dangling gradients in multiple-optimizer setup. 
        

        
          
            
        

        
          
                   .. note:: Only called when using multiple optimizers 
        

        
          
            
        

        
          
                   Override for your own behavior 
        

        
          
            
        

        
          
                   Args: 
        

        
          
                       optimizer: 
        

        
          
                       optimizer_idx: 
        

        
          
                   """ 
        

        
          
                   for param in self.parameters(): 
        

        
          
                       param.requires_grad = False 
        

        
          
            
        

        
          
                   for group in optimizer.param_groups: 
        

        
          
                       for param in group['params']: 
        

        
          
                           param.requires_grad = True 
        
    
  


Setup :
pytorch-lightning 1.1.2
pytorch 1.7.1
When using multiple optimizers, in the toggle_optimizers(..) function, the requires_grad property is set to true for all the params belonging to the param_groups of the optimizer. This is incorrect as in case the user has explicitly disabled the requires_grad property for some parameters permanently, the function would enable requires_grad for those parameters also.
Proposed fix:
The requires_grad value for all the parameters should be stored beforehand in an object variable self.params_dict of lightning object. In the toggle_optimizer() function, instead of setting params.requires_grad=True, set params.requires_grad = self.params_dict[params].This would set the correct value for the params of the concerned optimizer.