RuntimeError: Caught RuntimeError in DataLoader worker process 0

neilsambhu: <p>I’m trying to use DataLoader, which performs some operations (neural rendering) on GPU and outputs the result. I’m not able to get the DataLoader to load because it throws “RuntimeError: Caught RuntimeError in DataLoader worker process 0.” I have changed the num_workers to 0; this was not successful and stopped my code from having any output (i.e. no print statements; no error messages). Here is the code I’m referencing: <a href="https://github.com/nlsde-safety-team/DualAttentionAttack/blob/main/src/data_loader.py#L46" class="inline-onebox" rel="noopener nofollow ugc">DualAttentionAttack/data_loader.py at main · nlsde-safety-team/DualAttentionAttack · GitHub</a></p>

srishti-git1110: <p>Hi,<br>
Please try to post a minimum executable snippet that reproduces your error enclosing it within ```.</p>
<p>Such errors are sometimes also caused while sampling. Make sure the indices generated by your <code>Sampler</code> are aligned with your dataset’s indices.</p>

neilsambhu: <p>Here is a minimal version of the dataloader setup:</p>
<pre><code class="lang-auto">from torch.utils.data import Dataset, DataLoader
from torchvision.io import read_image
class MyDataset(Dataset):
  def __init__(self):
    self.image_paths = glob.glob('./dataset/**.png')
    # a torch.nn.module-like model
    self.renderer = NeuralRenderer(img_size=(225,225)).cuda()
  def __getitem__(self, index):
    img = read_image(self.image_paths[index]).cuda()
    return self.renderer(img)
dataset = MyDataset()
loader = DataLoader(dataset=dataset, batch_size=1, shuffle=False, num_workers=0)
</code></pre>
<p>Upon iterating over the loader is when I see the runtime error.</p>

srishti-git1110: <p>Please ensure <code>image_paths</code> isn’t empty and that you are able to retrieve files by indexing it like -<br>
<code>image_paths[0]</code>.</p>
<p>Please post here if you still face the error.</p>

neilsambhu: <p>I’ve ensured the path isn’t empty.</p>

neilsambhu: <p>I’m able to access the images at any given index.</p>

srishti-git1110: <p>Please implement the <code>__len__()</code> function in your <code>MyDataset</code> class.<br>
(This function just returns the length of your dataset and is expected by many implementations of the <code>Sampler</code> class and the <code>DataLoader</code>.)<br>
If it still doesn’t work, please show the full error message that you are getting.</p>
<p>Maybe this could help:</p>
<pre><code class="lang-python">class MyDataset(Dataset):
  def __init__(self):
    # body
  def __getitem__(self, index):
    # body
  def __len__(self):
     return len(image_paths)
</code></pre>

neilsambhu: <p>Thanks for the reply. I only provided a very stripped-down version of a rather large implementation, and the full version does have <strong>len</strong>(). It seems like it doesn’t like that some pre-processing is happening on GPU before the dataloader can collate the dataset into batches?</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "train.py", line 373, in &lt;module&gt;
    run_cam(test_dir, EPOCH)
  File "train.py", line 279, in run_cam
    for i,j in enumerate(loader):
  File "/home/nsambhu/.conda/envs/dualattentionattack/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
    data = self._next_data()
  File "/home/nsambhu/.conda/envs/dualattentionattack/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 856, in _next_data
    return self._process_data(data)
  File "/home/nsambhu/.conda/envs/dualattentionattack/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 881, in _process_data
    data.reraise()
  File "/home/nsambhu/.conda/envs/dualattentionattack/lib/python3.7/site-packages/torch/_utils.py", line 394, in reraise
    raise self.exc_type(msg)
RuntimeError: Caught RuntimeError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/home/nsambhu/.conda/envs/dualattentionattack/lib/python3.7/site-packages/torch/utils/data/_utils/worker.py", line 178, in _worker_loop
    data = fetcher.fetch(index)
  File "/home/nsambhu/.conda/envs/dualattentionattack/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/nsambhu/.conda/envs/dualattentionattack/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in &lt;listcomp&gt;
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/nsambhu/github/DualAttentionAttack/src/data_loader.py", line 87, in __getitem__
    imgs_pred = self.mask_renderer.forward(self.vertices_var, self.faces_var, self.textures)
  File "/home/nsambhu/github/DualAttentionAttack/src/nmr_test.py", line 249, in forward
    return self.RenderFunc(vertices, faces, textures)
  File "/home/nsambhu/github/DualAttentionAttack/src/nmr_test.py", line 167, in forward
    vs = vertices.cuda().numpy() # 11/27/2022 1:28:46 PM: Neil added
  File "/home/nsambhu/.conda/envs/dualattentionattack/lib/python3.7/site-packages/torch/cuda/__init__.py", line 195, in _lazy_init
    "Cannot re-initialize CUDA in forked subprocess. " + msg)
RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method
</code></pre>

srishti-git1110: <p>In that case, have you tried removing the <code>.cuda()</code> calls from the init and getitem methods? Does it work then?</p>

neilsambhu: <p>Unfortunately, it still fails:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "train.py", line 373, in &lt;module&gt;
    run_cam(test_dir, EPOCH)
  File "train.py", line 279, in run_cam
    for i,j in enumerate(loader):
  File "/home/nsambhu/.conda/envs/dualattentionattack/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
    data = self._next_data()
  File "/home/nsambhu/.conda/envs/dualattentionattack/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 856, in _next_data
    return self._process_data(data)
  File "/home/nsambhu/.conda/envs/dualattentionattack/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 881, in _process_data
    data.reraise()
  File "/home/nsambhu/.conda/envs/dualattentionattack/lib/python3.7/site-packages/torch/_utils.py", line 394, in reraise
    raise self.exc_type(msg)
RuntimeError: Caught RuntimeError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/home/nsambhu/.conda/envs/dualattentionattack/lib/python3.7/site-packages/torch/utils/data/_utils/worker.py", line 178, in _worker_loop
    data = fetcher.fetch(index)
  File "/home/nsambhu/.conda/envs/dualattentionattack/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/nsambhu/.conda/envs/dualattentionattack/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in &lt;listcomp&gt;
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/nsambhu/github/DualAttentionAttack/src/data_loader.py", line 97, in __getitem__
    imgs_pred = self.mask_renderer.forward(self.vertices_var, self.faces_var, self.textures)
  File "/home/nsambhu/github/DualAttentionAttack/src/nmr_test.py", line 249, in forward
    return self.RenderFunc(vertices, faces, textures)
  File "/home/nsambhu/github/DualAttentionAttack/src/nmr_test.py", line 167, in forward
    vs = vertices.cuda().numpy() # 11/27/2022 1:28:46 PM: Neil added
  File "/home/nsambhu/.conda/envs/dualattentionattack/lib/python3.7/site-packages/torch/cuda/__init__.py", line 195, in _lazy_init
    "Cannot re-initialize CUDA in forked subprocess. " + msg)
RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method
</code></pre>

ptrblck: <p>Remove all CUDA calls from your <code>Dataset.__getitem__</code> method as <a class="mention" href="/u/srishti-git1110">@srishti-git1110</a> already mentioned. The current code fails in trying to re-initialize the CUDA context in a new process since you are trying to move a tensor to the GPU in:</p>
<pre><code class="lang-python">vs = vertices.cuda().numpy()
</code></pre>
<p>and directly afterwards to the CPU, which also sounds not necessary.<br>
If you want to use a numpy array, just get rid of the <code>.cuda()</code> call to avoid moving the tensor back and forth.</p>

Yami: <p>Removing the .cuda() works for me! Thanks!</p>