Custom VisionDataset fails to iterate when num_worker is bigger than 1 (on Windows)

ondrejs: <p>I am following the following official tutorial <a href="https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Transfer Learning for Computer Vision Tutorial — PyTorch Tutorials 2.1.0+cu121 documentation</a>.</p>
<p>I replaced datasets.ImageFolder  with custom Dataset as so:</p>
<p>class CustomDataset(datasets.ImageFolder):<br>
def <strong>init</strong>(self, path, transforms):<br>
self.path = path<br>
super().<strong>init</strong>(path, transforms)</p>
<pre><code>def __getitem__(self, i):
    image, label = super().__getitem__(i)
    fname, _ = self.samples[i]
    return (image, label, fname)

def __len__(self):
    return super().__len__()
</code></pre>
<p>Then I replaced datasets.ImageFolder with the new class as shown here:</p>
<p>image_datasets = {x: CustomDataset(os.path.join(data_dir, x),<br>
data_transforms<span class="chcklst-box checked fa fa-check-square-o fa-fw"></span>)<br>
for x in [‘train’, ‘val’]}<br>
dataloaders = {x: torch.utils.data.DataLoader(image_datasets<span class="chcklst-box checked fa fa-check-square-o fa-fw"></span>, batch_size=4,<br>
shuffle=True, num_workers=4</p>
<p>If I try to iterate</p>
<p>it = iter(dataloaders[‘val’])<br>
next(it)</p>
<p>I get the following error:</p>
<hr>
<p>Empty                                     Traceback (most recent call last)<br>
File ~\AppData\Local\r-miniconda\envs\r-reticulate\Lib\site-packages\torch\utils\data\dataloader.py:1132, in _MultiProcessingDataLoaderIter._try_get_data(self, timeout)<br>
1131 try:<br>
 → 1132     data = self._data_queue.get(timeout=timeout)<br>
1133     return (True, data)</p>
<p>File ~\AppData\Local\r-miniconda\envs\r-reticulate\Lib\multiprocessing\queues.py:114, in Queue.get(self, block, timeout)<br>
113     if not self._poll(timeout):<br>
 → 114         raise Empty<br>
115 elif not self._poll():</p>
<p>Empty:</p>
<p>The above exception was the direct cause of the following exception:</p>
<p>RuntimeError                              Traceback (most recent call last)<br>
Cell In[91], line 2<br>
1 it = iter(dataloaders[‘val’])<br>
----&gt; 2 next(it)</p>
<p>File ~\AppData\Local\r-miniconda\envs\r-reticulate\Lib\site-packages\torch\utils\data\dataloader.py:630, in _BaseDataLoaderIter.<strong>next</strong>(self)<br>
627 if self._sampler_iter is None:<br>
628     # TODO(<a href="https://github.com/pytorch/pytorch/issues/76750" class="inline-onebox" rel="noopener nofollow ugc">Bug in dataloader iterator found by mypy · Issue #76750 · pytorch/pytorch · GitHub</a>)<br>
629     self._reset()  # type: ignore[call-arg]<br>
 → 630 data = self._next_data()<br>
631 self._num_yielded += 1<br>
632 if self._dataset_kind == _DatasetKind.Iterable and <br>
633         self._IterableDataset_len_called is not None and <br>
634         self._num_yielded &gt; self._IterableDataset_len_called:</p>
<p>File ~\AppData\Local\r-miniconda\envs\r-reticulate\Lib\site-packages\torch\utils\data\dataloader.py:1328, in _MultiProcessingDataLoaderIter._next_data(self)<br>
1325     return self._process_data(data)<br>
1327 assert not self._shutdown and self._tasks_outstanding &gt; 0<br>
 → 1328 idx, data = self._get_data()<br>
1329 self._tasks_outstanding -= 1<br>
1330 if self._dataset_kind == _DatasetKind.Iterable:<br>
1331     # Check for _IterableDatasetStopIteration</p>
<p>File ~\AppData\Local\r-miniconda\envs\r-reticulate\Lib\site-packages\torch\utils\data\dataloader.py:1294, in _MultiProcessingDataLoaderIter._get_data(self)<br>
1290     # In this case, <code>self._data_queue</code> is a <code>queue.Queue</code>,. But we don’t<br>
1291     # need to call <code>.task_done()</code> because we don’t use <code>.join()</code>.<br>
1292 else:<br>
1293     while True:<br>
 → 1294         success, data = self._try_get_data()<br>
1295         if success:<br>
1296             return data</p>
<p>File ~\AppData\Local\r-miniconda\envs\r-reticulate\Lib\site-packages\torch\utils\data\dataloader.py:1145, in _MultiProcessingDataLoaderIter._try_get_data(self, timeout)<br>
1143 if len(failed_workers) &gt; 0:<br>
1144     pids_str = ', ‘.join(str(w.pid) for w in failed_workers)<br>
 → 1145     raise RuntimeError(f’DataLoader worker (pid(s) {pids_str}) exited unexpectedly’) from e<br>
1146 if isinstance(e, queue.Empty):<br>
1147     return (False, None)</p>
<p>RuntimeError: DataLoader worker (pid(s) 26204, 30776, 26256, 32236) exited unexpectedly</p>
<p>The code works fine if either</p>
<ul>
<li>num_worker parameter is set to 0</li>
<li>CustomDataset is replaced with datasets.ImageFolder and num_worker remains set to 4.</li>
</ul>
<p>Any idea what is the problem?</p>