ModelCheckpoint tries to remove already removed checkpoint in DDP mode
🐛 Bug
When training in DDP mode with ModelCheckpoint callback, the train process fails, when ModelCheckpoint callback tries to remove previous checkpoint. I assume that it was already deleted by another process.
To Reproduce
Steps to reproduce the behavior:
Run training with "ddp" backend and ModelCheckpoint callback with save_top_k={some_number}
  File "/home/myuser/miniconda3/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 19, in _wrap                                                                                                 
    fn(i, *args)                                                                                
  File "/home/myuser/miniconda3/lib/python3.7/site-packages/pytorch_lightning/trainer/distrib_data_parallel.py", line 342, in ddp_train
    self.run_pretrain_routine(model)                                                                                                        
  File "/home/myuser/miniconda3/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 830, in run_pretrain_routine
    self.train()                                                                                                                                                                                                   
  File "/home/myuser/miniconda3/lib/python3.7/site-packages/pytorch_lightning/trainer/training_loop.py", line 343, in train                                                      
    self.run_training_epoch()                            
  File "/home/myuser/miniconda3/lib/python3.7/site-packages/pytorch_lightning/trainer/training_loop.py", line 452, in run_training_epoch                                                                       
    self.call_checkpoint_callback()                                                                                                                                                   
  File "/home/myuser/miniconda3/lib/python3.7/site-packages/pytorch_lightning/trainer/training_loop.py", line 737, in call_checkpoint_callback
    self.checkpoint_callback.on_validation_end(self, self.get_model())                                                                                                                                             
  File "/home/myuser/miniconda3/lib/python3.7/site-packages/pytorch_lightning/callbacks/model_checkpoint.py", line 204, in on_validation_end                                      
    self._do_check_save(filepath, current, epoch)    
  File "/home/myuser/miniconda3/lib/python3.7/site-packages/pytorch_lightning/callbacks/model_checkpoint.py", line 221, in _do_check_save                                                                      
    self._del_model(delpath)                                                                                                                                                           
  File "/home/myuser/miniconda3/lib/python3.7/site-packages/pytorch_lightning/callbacks/model_checkpoint.py", line 121, in _del_model
    os.remove(filepath)                                                                                                                                                                                            
FileNotFoundError: [Errno 2] No such file or directory: {PREVIOUS_CHECKPOINT_NAME}

Expected behavior
I expect that ModelCheckpoint callbacks from different DDP processes will not concurrent with each other in saving/deleteng files.
I fixed by rewriting _del_model method of ModelCheckpoint callback:
class DDPModelCheckpoint(ModelCheckpoint):
    def _del_model(self, filepath):
        try:
            os.remove(filepath)
        except Exception:
            pass

Environment

PyTorch Version: 1.4
OS: Ubuntu 18.04
How you installed PyTorch - conda
Python version: 3.7
CUDA/cuDNN version: 10.2
GPU models and configuration: 2x2080Ti
pytorch-lightning version: 0.7.1