GPU memory blowup when using pl=1.1.0 and slurm
🐛 Bug
The very same code that runs ok in multigpu regime in "terminal" mode, i.e. python train.py ..., meets CUDA OOM error when launched in slurm. In the meantime the same code worked OK both in slurm and in terminal with the previous version of PL (1.0.8).
Talking about the model, it is gan, and the behaviour happens either using or not the manual optimization.
Environment
* CUDA:
	- GPU:
		- Tesla V100-SXM2-16GB
		- Tesla V100-SXM2-16GB
		- Tesla V100-SXM2-16GB
		- Tesla V100-SXM2-16GB
	- available:         True
	- version:           10.2
* Packages:
	- numpy:             1.19.4
	- pyTorch_debug:     False
	- pyTorch_version:   1.6.0
	- pytorch-lightning: 1.1.0
	- tqdm:              4.54.1
* System:
	- OS:                Linux
	- architecture:
		- 64bit
		- 
	- processor:         x86_64
	- python:            3.7.9
	- version:           #1 SMP Tue Sep 10 10:50:19 EDT 2019

Additional context
While it seems the the processes are set correctly according to the output (see below), gpu memory load happens just on the one card which looks suspiciously (see nvidia-smi output https://yadi.sk/i/j8Etoi14tFuBQg).
[2020-12-12 01:08:29,305][lightning][INFO] - GPU available: True, used: True
[2020-12-12 01:08:29,306][lightning][INFO] - TPU available: None, using: 0 TPU cores
[2020-12-12 01:08:29,306][lightning][INFO] - Multi-processing is handled by Slurm.
[2020-12-12 01:08:29,306][lightning][INFO] - LOCAL_RANK: 3 - CUDA_VISIBLE_DEVICES: [0,1,2,3]
GPU available: True, used: True
TPU available: None, using: 0 TPU cores
Multi-processing is handled by Slurm.
LOCAL_RANK: 3 - CUDA_VISIBLE_DEVICES: [0,1,2,3]
initializing ddp: GLOBAL_RANK: 3, MEMBER: 4/4
[2020-12-12 01:08:29,317][lightning][INFO] - initializing ddp: GLOBAL_RANK: 3, MEMBER: 4/4
GPU available: True, used: True
[2020-12-12 01:08:31,736][lightning][INFO] - GPU available: True, used: True
TPU available: None, using: 0 TPU cores
[2020-12-12 01:08:31,736][lightning][INFO] - TPU available: None, using: 0 TPU cores
Multi-processing is handled by Slurm.
[2020-12-12 01:08:31,737][lightning][INFO] - Multi-processing is handled by Slurm.
LOCAL_RANK: 2 - CUDA_VISIBLE_DEVICES: [0,1,2,3]
[2020-12-12 01:08:31,737][lightning][INFO] - LOCAL_RANK: 2 - CUDA_VISIBLE_DEVICES: [0,1,2,3]
initializing ddp: GLOBAL_RANK: 2, MEMBER: 3/4
[2020-12-12 01:08:31,756][lightning][INFO] - initializing ddp: GLOBAL_RANK: 2, MEMBER: 3/4
GPU available: True, used: True
[2020-12-12 01:08:32,367][lightning][INFO] - GPU available: True, used: True
TPU available: None, using: 0 TPU cores
[2020-12-12 01:08:32,368][lightning][INFO] - TPU available: None, using: 0 TPU cores
Multi-processing is handled by Slurm.
[2020-12-12 01:08:32,368][lightning][INFO] - Multi-processing is handled by Slurm.
LOCAL_RANK: 1 - CUDA_VISIBLE_DEVICES: [0,1,2,3]
[2020-12-12 01:08:32,368][lightning][INFO] - LOCAL_RANK: 1 - CUDA_VISIBLE_DEVICES: [0,1,2,3]
initializing ddp: GLOBAL_RANK: 1, MEMBER: 2/4
[2020-12-12 01:08:32,392][lightning][INFO] - initializing ddp: GLOBAL_RANK: 1, MEMBER: 2/4
GPU available: True, used: True
[2020-12-12 01:08:32,516][lightning][INFO] - GPU available: True, used: True
TPU available: None, using: 0 TPU cores
[2020-12-12 01:08:32,517][lightning][INFO] - TPU available: None, using: 0 TPU cores
Multi-processing is handled by Slurm.
[2020-12-12 01:08:32,517][lightning][INFO] - Multi-processing is handled by Slurm.
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1,2,3]
[2020-12-12 01:08:32,517][lightning][INFO] - LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1,2,3]
[2020-12-12 01:08:32,526][lightning][INFO] - initializing ddp: GLOBAL_RANK: 0, MEMBER: 1/4
initializing ddp: GLOBAL_RANK: 0, MEMBER: 1/4
Set SLURM handle signals.
[2020-12-12 01:08:39,146][lightning][INFO] - Set SLURM handle signals.
Set SLURM handle signals.
[2020-12-12 01:08:39,150][lightning][INFO] - Set SLURM handle signals.
Set SLURM handle signals.
[2020-12-12 01:08:39,150][lightning][INFO] - Set SLURM handle signals.
Set SLURM handle signals.
[2020-12-12 01:08:39,150][lightning][INFO] - Set SLURM handle signals.