Having a low "val_check_interval" value with many workers in Dataloader leads to extrem overhead
Describe the bug
When running trainer.fit(...) having many workers in dataloaders in combination with many validations during training leads to extrem overhead. This happens because the iterator of the DataLoader class initialises each time an enumerator is called on it a new _MultiProcessingDataLoaderIter (https://github.com/pytorch/pytorch/blob/master/torch/utils/data/dataloader.py#L278). Also when having bigger datasets as MNIST the exiting of the enumerate loop leads to (https://github.com/pytorch/pytorch/blob/e96ea288a8706b9e6dfad89aa47d73198960631c/torch/utils/data/dataloader.py#L926) which takes a lot of time as well.
To Reproduce
Run gpu example
cli params: --distributed_backend=dp --gpus=1
initialise trainer with val_percent_check=0.02, val_check_interval=0.02
output of cProfile (top 20 entries timewise | total of 74s):
         22904807 function calls (22784137 primitive calls) in 74.071 seconds                                                                    │| 23%   35C    P8     9W / 250W |      2MiB / 11178MiB |
                                                                                                                                                 │    0%      Default |
   Ordered by: internal time                                                                                                                     │+-------------------------------+----------------------+--
   List reduced from 5610 to 20 due to restriction <20>                                                                                          │--------------------+
                                                                                                                                                 │|   2  GeForce GTX 108...  Off  | 00000000:09:00.0 Off |
   ncalls  tottime  percall  cumtime  percall filename:lineno(function)                                                                          │                N/A |
    13263   12.603    0.001   12.603    0.001 {method 'item' of 'torch._C._TensorBase' objects}                                                  │| 23%   35C    P8     8W / 250W |      2MiB / 11178MiB |
      340    7.897    0.023    7.897    0.023 {method '_write_file' of 'torch._C.CudaFloatStorageBase' objects}                                  │    0%      Default |
   220320    3.941    0.000    9.971    0.000 functional.py:191(normalize)                                                                       │+-------------------------------+----------------------+--
   220320    3.496    0.000   15.345    0.000 functional.py:42(to_tensor)                                                                        │--------------------+
   220320    3.455    0.000   37.525    0.000 mnist.py:80(__getitem__)                                                                           │|   3  GeForce GTX 108...  Off  | 00000000:0A:00.0 Off |
   220320    3.120    0.000    3.120    0.000 {method 'div' of 'torch._C._TensorBase' objects}                                                   ├──────────────────────────────────────────────────────────
      492    1.978    0.004    1.987    0.004 {method 'cuda' of 'torch._C._TensorBase' objects}                                                  │t/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (t
   440640    1.917    0.000    1.917    0.000 {built-in method as_tensor}                                                                        │ype, 1) or '1type' as a synonym of type is deprecated; in
   220320    1.842    0.000    1.842    0.000 {method 'float' of 'torch._C._TensorBase' objects}                                                 │a future version of numpy, it will be understood as (type,
   220320    1.701    0.000    6.623    0.000 Image.py:2612(fromarray)                                                                           │ (1,)) / '(1,)type'.
     2814    1.616    0.001    1.616    0.001 {method 'run_backward' of 'torch._C._EngineBase' objects}                                          │  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
   220320    1.480    0.000    1.480    0.000 {method 'sub_' of 'torch._C._TensorBase' objects}                                                  │/home/sebastian/.cache/pypoetry/virtualenvs/pl-iterator-_1
   223607    1.459    0.000    1.459    0.000 {method 'view' of 'torch._C._TensorBase' objects}                                                  │H-4oww-py3.7/lib/python3.7/site-packages/tensorboard/compa
   220320    1.263    0.000    1.263    0.000 {method 'clone' of 'torch._C._TensorBase' objects}                                                 │t/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (t
   440640    1.051    0.000    1.051    0.000 {method 'transpose' of 'torch._C._TensorBase' objects}                                             │ype, 1) or '1type' as a synonym of type is deprecated; in
       16    1.013    0.063    1.013    0.063 {built-in method posix.remove}                                                                     │a future version of numpy, it will be understood as (type,
   220320    0.940    0.000    3.322    0.000 Image.py:734(tobytes)                                                                              │ (1,)) / '(1,)type'.
   220320    0.915    0.000    0.915    0.000 {method 'div_' of 'torch._C._TensorBase' objects}                                                  │  np_resource = np.dtype([("resource", np.ubyte, 1)])
   220320    0.772    0.000    1.101    0.000 Image.py:461(_getencoder)                                                                          │TensorFlow installation not found - running with reduced f
   220320    0.736    0.000    4.826    0.000 Image.py:2552(frombuffer)

Now changing:
template
set num_workers in DataLoader to 8
output of cProfile (top 20 entries timewise | total of 103s):
         8656126 function calls (8542297 primitive calls) in 103.223 seconds

   Ordered by: internal time
   List reduced from 5738 to 20 due to restriction <20>

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
     1280   28.357    0.022   28.389    0.022 {built-in method posix.fork}
    13263   28.110    0.002   28.110    0.002 {method 'item' of 'torch._C._TensorBase' objects}
    15745    9.876    0.001    9.894    0.001 {built-in method posix.waitpid}
      340    7.581    0.022    7.581    0.022 {method '_write_file' of 'torch._C.CudaFloatStorageBase' objects}
    14684    4.262    0.000    4.262    0.000 {method 'acquire' of '_thread.lock' objects}
    56449    2.089    0.000    2.089    0.000 {built-in method posix.read}
      492    1.940    0.004    1.947    0.004 {method 'cuda' of 'torch._C._TensorBase' objects}
     2814    1.646    0.001    1.646    0.001 {method 'run_backward' of 'torch._C._EngineBase' objects}
       16    1.023    0.064    1.023    0.064 {built-in method posix.remove}
    11300    0.869    0.000    0.869    0.000 {method '__enter__' of '_thread.lock' objects}
     2814    0.701    0.000    2.556    0.001 adam.py:49(step)
    33768    0.644    0.000    0.644    0.000 {method 'mul_' of 'torch._C._TensorBase' objects}
    33768    0.502    0.000    0.502    0.000 {method 'add_' of 'torch._C._TensorBase' objects}
     6574    0.488    0.000    0.488    0.000 {built-in method torch._C._scatter}
     4032    0.389    0.000    0.389    0.000 {method 'poll' of 'select.poll' objects}
     8064    0.330    0.000    0.330    0.000 {method 'recvmsg' of '_socket.socket' objects}
     6576    0.326    0.000    0.326    0.000 {built-in method addmm}
     1863    0.301    0.000    0.301    0.000 {method 'astype' of 'numpy.ndarray' objects}
      931    0.297    0.000    0.297    0.000 {pandas._libs.writers.write_csv_rows}
    16884    0.267    0.000    0.267    0.000 {method 'sqrt' of 'torch._C._TensorBase' objects}

Expected behavior
I think it would make sense that during one epoch of training the same iterator of the dataloader was used to avoid overhead that is not necessary.
I was able to implement that like this:
https://gist.github.com/fellnerse/326a88473bb51798cfbcfde18fd333a8#file-lightning_module_template-py-L204
I'm basically doing what was proposed on the pytorch github.
Using this implementation with 8 workers cProfile output (only 58s):
         7392427 function calls (7271288 primitive calls) in 57.846 seconds

   Ordered by: internal time
   List reduced from 5707 to 20 due to restriction <20>

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
    13106   29.183    0.002   29.183    0.002 {method 'item' of 'torch._C._TensorBase' objects}
      400    9.162    0.023    9.162    0.023 {method '_write_file' of 'torch._C.CudaFloatStorageBase' objects}
      492    1.935    0.004    1.942    0.004 {method 'cuda' of 'torch._C._TensorBase' objects}
     2814    1.595    0.001    1.595    0.001 {method 'run_backward' of 'torch._C._EngineBase' objects}
       19    1.162    0.061    1.162    0.061 {built-in method posix.remove}
    48217    0.719    0.000    0.719    0.000 {built-in method posix.read}
     2814    0.699    0.000    2.586    0.001 adam.py:49(step)
    33768    0.663    0.000    0.663    0.000 {method 'mul_' of 'torch._C._TensorBase' objects}
       24    0.529    0.022    0.530    0.022 {built-in method posix.fork}
    33768    0.516    0.000    0.516    0.000 {method 'add_' of 'torch._C._TensorBase' objects}
     6574    0.416    0.000    0.416    0.000 {built-in method torch._C._scatter}
     6576    0.311    0.000    0.311    0.000 {built-in method addmm}
      931    0.276    0.000    0.276    0.000 {pandas._libs.writers.write_csv_rows}
    16884    0.271    0.000    0.271    0.000 {method 'sqrt' of 'torch._C._TensorBase' objects}
     1863    0.260    0.000    0.260    0.000 {method 'astype' of 'numpy.ndarray' objects}
     6888    0.228    0.000    0.228    0.000 {method 'recvmsg' of '_socket.socket' objects}
        4    0.218    0.054    0.218    0.054 {method 'uniform_' of 'torch._C._TensorBase' objects}
    16884    0.212    0.000    0.212    0.000 {method 'addcdiv_' of 'torch._C._TensorBase' objects}
    16884    0.199    0.000    0.199    0.000 {method 'addcmul_' of 'torch._C._TensorBase' objects}
        3    0.197    0.066   54.204   18.068 train_loop_mixin.py:68(run_training_epoch)

Screenshots

version_0: num_workers=1
version_1: num_workers=8
version_2: num_workers=8 && new implementation
The curve is obviously not the same anymore, but comparable. And the time is way better.
Desktop (please complete the following information):
Linux 5.0.0-32-generic #34~18.04.2-Ubuntu SMP Thu Oct 10 10:36:02 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux