A somewhat cryptic error message (for newcomers) - "Cannot re-initialize CUDA in forked subprocess" - report and suggestion for a possible solution
Labels: module: dataloader, triaged

### 🐛 Describe the bug

I've seen multiple individuals and groups using num_workers=0 in DataLoader, and significantly under-utilizing GPUs due to the following error message which appeared cryptic (to newcomers) when they used num_workers bigger than 0

"Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method"

The error message was gone when they switched to num_workers=0
Using "spawn" start method did not solve the problem.

I analyzed the problem and found that inside their custom Dataset implementation that as a final step there they transfer the tensor into the GPU.
(Which could make sense from the point of view of non-experts, as they remove the need to do this as another step before feeding the tensor to a model.)

I was wondering - perhaps there should be a warning or an exception when a non-cpu tensor is being created inside a worker process spawn by DataLoader? 
Or maybe an exception when a non-cpu tensor is being created inside a Dataset in general?
I can try to investigate and propose a PR if you think it's a good direction.

A short snippet to reproduce the problem:
```python
import torch
from torch.utils.data import DataLoader, Dataset
import torch.nn as nn
import torch.optim as optim
from tqdm import tqdm

class SomeDataset(Dataset):   
    def __len__(self):
        return 100
    def __getitem__(self, idx):
        assert idx>=0 and idx<len(self)
        t = torch.rand(3, 400,400, device=torch.device('cuda'))
        return t

class SomeNet(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(3, 6, 5)
    def forward(self, x):
        x = self.conv1(x)        
        return x

if __name__=='__main__':
    ds = SomeDataset()
    dl = DataLoader(ds, batch_size=8, num_workers=4)
    model = SomeNet().cuda()
    optimizer = optim.SGD(model.parameters(), lr=0.001, momentum=0.9)
    
    for s in tqdm(iter(dl), total=len(dl)):
        optimizer.zero_grad()        
        pred = model(s)
        pred.mean().backward()
        optimizer.step()

```

the output:
```bash
Traceback (most recent call last):
  File "/some/path/reproduce_dataloader_cuda_tensors_problem.py", line 36, in <module>
    for s in tqdm(iter(dl), total=len(dl)):
  File "/some/path/to/download/miniconda3/envs/bio/lib/python3.7/site-packages/tqdm/std.py", line 1195, in __iter__
    for obj in iterable:
  File "/some/path/to/download/miniconda3/envs/bio/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 530, in __next__
    data = self._next_data()
  File "/some/path/to/download/miniconda3/envs/bio/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1224, in _next_data
    return self._process_data(data)
  File "/some/path/to/download/miniconda3/envs/bio/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1250, in _process_data
    data.reraise()
  File "/some/path/to/download/miniconda3/envs/bio/lib/python3.7/site-packages/torch/_utils.py", line 457, in reraise
    raise exception
RuntimeError: Caught RuntimeError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/some/path/to/download/miniconda3/envs/bio/lib/python3.7/site-packages/torch/utils/data/_utils/worker.py", line 287, in _worker_loop
    data = fetcher.fetch(index)
  File "/some/path/to/download/miniconda3/envs/bio/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 49, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/some/path/to/download/miniconda3/envs/bio/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 49, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/some/path/reproduce_dataloader_cuda_tensors_problem.py", line 16, in __getitem__
    t = torch.rand(3, 400,400, device=torch.device('cuda'))
  File "/some/path/to/download/miniconda3/envs/bio/lib/python3.7/site-packages/torch/cuda/__init__.py", line 207, in _lazy_init
    "Cannot re-initialize CUDA in forked subprocess. To use CUDA with "
RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method

### Versions

Collecting environment information...
PyTorch version: 1.11.0+cu102
Is debug build: False
CUDA used to build PyTorch: 10.2
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.6 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: Could not collect
CMake version: version 3.10.2
Libc version: glibc-2.17

Python version: 3.7.13 (default, Mar 29 2022, 02:18:16)  [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-4.15.0-167-generic-x86_64-with-debian-buster-sid
Is CUDA available: True
CUDA runtime version: Could not collect
GPU models and configuration: 
GPU 0: Tesla V100-PCIE-16GB
GPU 1: Tesla V100-PCIE-16GB

Nvidia driver version: 460.106.00
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.7.1.4
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.3.2
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.3.2
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.3.2
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.3.2
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.3.2
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.3.2
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.3.2
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] mypy==0.782
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.21.5
[pip3] pytorch-lightning==1.6.0
[pip3] torch==1.11.0
[pip3] torchmetrics==0.7.3
[pip3] torchvision==0.12.0
[conda] mypy                      0.782                    pypi_0    pypi
[conda] mypy-extensions           0.4.3                    pypi_0    pypi
[conda] numpy                     1.21.5                   pypi_0    pypi
[conda] pytorch-lightning         1.6.0                    pypi_0    pypi
[conda] torch                     1.11.0                   pypi_0    pypi
[conda] torchmetrics              0.7.3                    pypi_0    pypi
[conda] torchvision               0.12.0                   pypi_0    pypi
```

cc @SsnL @VitalyFedyunin @ejguan @NivekT