Repository: Lightning-AI/pytorch-lightning

Issue:
Bug in DDP, but not DP modes.
Pytorch 1.5

```
In [3]: pytorch_lightning.__version__                                                                                                                                                                                                                                                                
Out[3]: '0.7.5rc1'

```


In DP everything works.

In DDP fails with:

```
 File "/home/vladimir/anaconda3/envs/solaris/lib/python3.7/multiprocessing/popen_fork.py", line 20, in __init__
    self._launch(process_obj)
  File "/home/vladimir/anaconda3/envs/solaris/lib/python3.7/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/home/vladimir/anaconda3/envs/solaris/lib/python3.7/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
_pickle.PicklingError: Can't pickle <class 'torch._C._VariableFunctions'>: it's not the same object as torch._C._VariableFunctions
```



Discussion:
can you post a colab? does 0.7.4 work ok? 
https://pytorch-lightning.readthedocs.io/en/latest/multi_gpu.html#make-model-picklable
I am getting the same issue.
* The model is picklable
* It worked fine on 0.7.2
* It breaks after the update to 0.7.4

My stacktrace:
```
Traceback (most recent call last):
  File "./train_stage3.py", line 254, in <module>
    trainer.fit(model)
  File "/net/people/plgquinor/venv/lib/python3.6/site-packages/pytorch_lightning/trainer/trainer.py", line 744, in fit
    mp.spawn(self.ddp_train, nprocs=self.num_processes, args=(model,))
  File "/net/people/plgquinor/venv/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 200, in spawn
    return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')
  File "/net/people/plgquinor/venv/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 149, in start_processes
    process.start()
  File "/net/software/local/python/3.6.5/lib/python3.6/multiprocessing/process.py", line 105, in start
    self._popen = self._Popen(self)
  File "/net/software/local/python/3.6.5/lib/python3.6/multiprocessing/context.py", line 284, in _Popen
    return Popen(process_obj)
  File "/net/software/local/python/3.6.5/lib/python3.6/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/net/software/local/python/3.6.5/lib/python3.6/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/net/software/local/python/3.6.5/lib/python3.6/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/net/software/local/python/3.6.5/lib/python3.6/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
_pickle.PicklingError: Can't pickle <class 'torch._C._VariableFunctions'>: it's not the same object as torch._C._VariableFunctions
```
The model is picklable - the error appears when I try to pickle the `Trainer` instance.

[Update] I had found the culprit. ModelCheckpoint is non-picklable due to:
https://github.com/PyTorchLightning/pytorch-lightning/blob/master/pytorch_lightning/callbacks/model_checkpoint.py#L118
torch.lt, torch.gt etc. being non-picklable. It worked previously when those were np.* ops.

Resolving patch:
diff --git a/pytorch_lightning/callbacks/model_checkpoint.py b/pytorch_lightning/callbacks/model_checkpoint.py
--- a/pytorch_lightning/callbacks/model_checkpoint.py
+++ b/pytorch_lightning/callbacks/model_checkpoint.py
@@ -116,10 +116,10 @@ def __init__(self, filepath: Optional[str] = None, monitor: str = 'val_loss', ve
 
         torch_inf = torch.tensor(np.Inf)
         mode_dict = {
-            'min': (torch.lt, torch_inf, 'min'),
-            'max': (torch.gt, -torch_inf, 'max'),
-            'auto': (torch.gt, -torch_inf, 'max') if 'acc' in self.monitor or self.monitor.startswith('fmeasure')
-            else (torch.lt, torch_inf, 'min'),
+            'min': (torch_inf, 'min'),
+            'max': (-torch_inf, 'max'),
+            'auto': (-torch_inf, 'max') if 'acc' in self.monitor or self.monitor.startswith('fmeasure')
+            else (torch_inf, 'min'),
         }
 
         if mode not in mode_dict:
@@ -127,7 +127,7 @@ def __init__(self, filepath: Optional[str] = None, monitor: str = 'val_loss', ve
                            f'fallback to auto mode.', RuntimeWarning)
             mode = 'auto'
 
-        self.monitor_op, self.kth_value, self.mode = mode_dict[mode]
+        self.kth_value, self.mode = mode_dict[mode]
 
     def _del_model(self, filepath):
         if os.path.isfile(filepath):
@@ -151,7 +151,12 @@ def check_monitor_top_k(self, current):
         if not isinstance(current, torch.Tensor):
             current = torch.tensor(current)
 
-        return self.monitor_op(current, self.best_k_models[self.kth_best_model])
+        monitor_op = {
+            "min": torch.lt,
+            "max": torch.gt,
+        }[self.mode]
+
+        return monitor_op(current, self.best_k_models[self.kth_best_model])
 
     def format_checkpoint_name(self, epoch, metrics, ver=None):
         """Generate a filename according to the defined template.


Test patch: