EOFError exception raised when using max_steps in distributed training
🐛 Bug
When using max_steps to stop a distributed training (ddp) in the middle of an epoch, the following exception is raised:
Epoch 1364:  64%|█████████████              | 7/11 [00:11<00:06,  1.62s/it, loss=2.057, v_num=13, lr=2.47e-11]
Exception in thread Thread-4:
Traceback (most recent call last):
  File "***/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "***/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "***/lib/python3.8/site-packages/torch/utils/data/_utils/pin_memory.py", line 25, in _pin_memory_loop
    r = in_queue.get(timeout=MP_STATUS_CHECK_INTERVAL)
  File "***/lib/python3.8/multiprocessing/queues.py", line 116, in get
    return _ForkingPickler.loads(res)
  File "***/lib/python3.8/site-packages/torch/multiprocessing/reductions.py", line 282, in rebuild_storage_fd
    fd = df.detach()
  File "***/lib/python3.8/multiprocessing/resource_sharer.py", line 57, in detach
    with _resource_sharer.get_connection(self._id) as conn:
  File "***/lib/python3.8/multiprocessing/resource_sharer.py", line 87, in get_connection
    c = Client(address, authkey=process.current_process().authkey)
  File "***/lib/python3.8/multiprocessing/connection.py", line 508, in Client
    answer_challenge(c, authkey)
  File "***/lib/python3.8/multiprocessing/connection.py", line 752, in answer_challenge
    message = connection.recv_bytes(256)         # reject large message
  File "***/lib/python3.8/multiprocessing/connection.py", line 216, in recv_bytes
    buf = self._recv_bytes(maxlength)
  File "***/lib/python3.8/multiprocessing/connection.py", line 414, in _recv_bytes
    buf = self._recv(4)
  File "***/lib/python3.8/multiprocessing/connection.py", line 383, in _recv
    raise EOFError
EOFError

To Reproduce
Steps to reproduce the behavior:

Train some model in distributed mode (ddp) such that num_steps is defined in the trainer, and it stops training in the middle of an epoch.

Expected behavior
The training should end without any error.
Environment

CUDA:

GPU:

Tesla V100-SXM2-16GB
Tesla V100-SXM2-16GB
Tesla V100-SXM2-16GB
Tesla V100-SXM2-16GB
Tesla V100-SXM2-16GB
Tesla V100-SXM2-16GB
Tesla V100-SXM2-16GB
Tesla V100-SXM2-16GB


available:         True
version:           10.1


Packages:

numpy:             1.18.5
pyTorch_debug:     False
pyTorch_version:   1.6.0
pytorch-lightning: 0.9.0rc12
tensorboard:       2.3.0
tqdm:              4.47.0


System:

OS:                Linux
architecture:

64bit
ELF


processor:         x86_64
python:            3.8.5
version:           #25~18.04.1-Ubuntu SMP Fri Jun 5 15:18:30 UTC 2020