Resolve several Horovod test failures
🐛 Bug
We are currently skipping several Horovod tests. We should eventually address each one.

 Coverage issue: 
  
    
      pytorch-lightning/tests/models/test_horovod.py
    
    
         Line 52
      in
      b9475c1
    
  
  
    

        
          
           # TODO: Find out why coverage breaks CI. 
        
    
  


 Unclear feature: 
  
    
      pytorch-lightning/tests/models/test_horovod.py
    
    
        Lines 154 to 156
      in
      b9475c1
    
  
  
    

        
          
           # https://discuss.pytorch.org/t/torch-cuda-amp-vs-nvidia-apex/74994 
        

        
          
           # Check with (tgaddair) on Horovod issues if this feature is needed 
        

        
          
           @pytest.mark.skip(reason="TODO: Horovod currently doesn't work with Apex") 
        
    
  


 Flaky test on ubuntu 1. Produces a segfault: 
  
    
      pytorch-lightning/tests/models/test_horovod.py
    
    
         Line 243
      in
      b9475c1
    
  
  
    

        
          
           @pytest.mark.skip('TODO: flaky test - Fatal Python error: Aborted') 
        
    
  


 Flaky test on ubuntu 2. Produces a segfault: 
  
    
      pytorch-lightning/tests/models/test_horovod.py
    
    
         Line 380
      in
      9402c05
    
  
  
    

        
          
           @pytest.mark.skip('TODO: flaky test - Fatal Python error: Aborted') 
        
    
  


 Permissions issue 1: 
  
    
      pytorch-lightning/tests/models/test_horovod.py
    
    
         Line 276
      in
      b9475c1
    
  
  
    

        
          
           @pytest.mark.skip(reason="TODO: CI agent.jobstatus=Succeeded: Permission denied") 
        
    
  


 Permissions issue 2: 
  
    
      pytorch-lightning/tests/models/test_horovod.py
    
    
         Line 326
      in
      b9475c1
    
  
  
    

        
          
           @pytest.mark.skip(reason="TODO: CI agent.jobstatus=Succeeded: Permission denied") 
        
    
  



cc: @Borda @tgaddair