Repository: Lightning-AI/pytorch-lightning

Issue:
 Checkpoint fails in single node multi-GPU mode using  DDP
## 🐛 Bug

Checkpoint fails in single node multi-GPU mode using  DDP.

### To Reproduce

```bash
python pl_examples/basic_examples/gpu_template.py --distributed_backend ddp --gpus 2
```
```bash
Epoch 2: : 700it [00:28, 42.69it/s, l/home/xz/anaconda3/envs/x/lib/python3.7/multiprocessing/semaphore_tracker.py:144: UserWarning: semaphore_tracker: There appear to be 1 leaked semaphores to clean up at shutdown                                                                                                                                                                                                                 
  len(cache))
Traceback (most recent call last):
  File "gpu_template.py", line 79, in <module>
    main(hyperparams)
  File "gpu_template.py", line 40, in main
    trainer.fit(model)
  File "/home/xz/anaconda3/envs/x/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 590, in fit
    mp.spawn(self.ddp_train, nprocs=self.num_gpus, args=(model,))
  File "/home/xz/anaconda3/envs/x/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 171, in spawn
    while not spawn_context.join():
  File "/home/xz/anaconda3/envs/x/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 118, in join
    raise Exception(msg)
Exception: 

-- Process 1 terminated with the following error:
Traceback (most recent call last):
  File "/home/xz/anaconda3/envs/x/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 19, in _wrap
    fn(i, *args)
  File "/home/xz/anaconda3/envs/x/lib/python3.7/site-packages/pytorch_lightning/trainer/distrib_data_parallel.py", line 342, in ddp_train
    self.run_pretrain_routine(model)
  File "/home/xz/anaconda3/envs/x/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 830, in run_pretrain_routine
    self.train()
  File "/home/xz/anaconda3/envs/x/lib/python3.7/site-packages/pytorch_lightning/trainer/training_loop.py", line 343, in train
    self.run_training_epoch()
  File "/home/xz/anaconda3/envs/x/lib/python3.7/site-packages/pytorch_lightning/trainer/training_loop.py", line 452, in run_training_epoch
    self.call_checkpoint_callback()
  File "/home/xz/anaconda3/envs/x/lib/python3.7/site-packages/pytorch_lightning/trainer/training_loop.py", line 737, in call_checkpoint_callback
    self.checkpoint_callback.on_validation_end(self, self.get_model())
  File "/home/xz/anaconda3/envs/x/lib/python3.7/site-packages/pytorch_lightning/callbacks/model_checkpoint.py", line 204, in on_validation_end
    self._do_check_save(filepath, current, epoch)
  File "/home/xz/anaconda3/envs/x/lib/python3.7/site-packages/pytorch_lightning/callbacks/model_checkpoint.py", line 221, in _do_check_save
    self._del_model(delpath)
  File "/home/xz/anaconda3/envs/x/lib/python3.7/site-packages/pytorch_lightning/callbacks/model_checkpoint.py", line 121, in _del_model
    os.remove(filepath)
FileNotFoundError: [Errno 2] No such file or directory: '/home/xz/pytorch-lightning/pl_examples/basic_examples/lightning_logs/version_1/checkpoints/epoch=0.ckpt'
```


Discussion:
yeah we shall run all examples in CI too

Resolving patch:
diff --git a/pytorch_lightning/callbacks/model_checkpoint.py b/pytorch_lightning/callbacks/model_checkpoint.py
--- a/pytorch_lightning/callbacks/model_checkpoint.py
+++ b/pytorch_lightning/callbacks/model_checkpoint.py
@@ -175,6 +175,10 @@ def format_checkpoint_name(self, epoch, metrics, ver=None):
         return filepath
 
     def on_validation_end(self, trainer, pl_module):
+        # only run on main process
+        if trainer.proc_rank != 0:
+            return
+
         metrics = trainer.callback_metrics
         epoch = trainer.current_epoch
         self.epochs_since_last_check += 1


Test patch: