Memory leaking when using large numpy array in Dataset
🐛 Bug
Thank you for the great library! When migrating a larger project, I am running into memory issues, though, so maybe someone can help me out.
So, I have a pretty complicated DataSet which loads plenty of data and buffers the data into the CPU RAM as a numpy array.
I train using ddp and with num_workers = 6 in the dataloader. The training crashes my machine, because of  CPU memory overflow. It works with num_workers = 0, but the higher the num_workers, the higher the memory consumption.
I figured out that this is much worse when using a large numpy array in the Dataset rather than a PyTorch tensor.
Unfortunately, I need numpy arrays, so I am asking you if there is anything I can do?
To Reproduce
I created a repository to reproduce this. It allows you to train a model on toy data using either a PyTorch tensor or a numpy array in the Dataset.
When running it with the PyTorch tensor the same amount of data uses 5GB of RAM while with Numpy it uses more than 30GB of RAM.
The higher the number of num_workers, the higher the RAM usage - it seems to leak when using numpy?

Clone https://github.com/mpaepper/reproduce_pytorch_lightning_memory_issues
Try the PyTorch tensor with: python minimal.py --num_workers 10
Try the numpy array with: python minimal.py --numpy --num_workers 10
Compare the huge difference in memory consumption

Code sample
https://github.com/mpaepper/reproduce_pytorch_lightning_memory_issues
Expected behavior
I would expect that numpy and PyTorch tensors should behave in the same way when using num_workers > 0, i.e. memory consumption is similar.
Environment

CUDA:
- GPU:
- GeForce RTX 2080 Ti
- GeForce RTX 2080 Ti
- GeForce RTX 2080 Ti
- GeForce RTX 2080 Ti
- available:         True
- version:           10.1
Packages:
- numpy:             1.16.4
- pyTorch_debug:     False
- pyTorch_version:   1.4.0
- pytorch-lightning: 0.7.5
- tensorboard:       1.14.0
- tqdm:              4.46.0
System:
- OS:                Linux
- architecture:
- 64bit
-
- processor:         x86_64
- python:            3.7.3
- version:           #97-Ubuntu SMP Wed Apr 1 03:25:46 UTC 2020