cuda runtime error (101) : invalid device ordinal
CUDA_VISIBLE_DEVICES and ddp are not compatible.

  
    
      pytorch-lightning/pytorch_lightning/trainer/distrib_data_parallel.py
    
    
         Line 504
      in
      25ee51b
    
  
  
    

        
          
           if self.on_gpu: 
        
    
  


the pytorch respects the CUDA_VISIBLE_DEVICES.
gpu_idx should be the same as local_rank.
If I set CUDA_VISIBLE_DEVICES to be 1. It will break.