DataLoaders errors with num_workers >0 on IterableDataset

builder173: <p>We are using simulated training data and developed an IterableDataset for this task. When the dataset is wrapped with a standard DataLoader we can use it for training but suspect that the serial data simulation is severely rate limiting. (complex calculations) This seems to be the perfect scenario for using workers in the DataLoader but we see errors when we attempt to do so.</p>
<p>The issue can be reproduced with a fairly simple example here:</p>
<pre><code class="lang-auto"># https://pytorch.org/docs/stable/data.html#torch.utils.data.IterableDataset
import torch
from torch.utils.data import IterableDataset, DataLoader
import numpy as np

class RandomDataset(IterableDataset):
    def __init__(self):
        super(RandomDataset).__init__()
        self.rng = np.random.default_rng()
 
    def __next__(self):
        x = torch.tensor(self.rng.random(1))
        y = 2 * x.clone().detach()
        return (x, y)

    def __iter__(self):
        return self

test_ds = RandomDataset()
test_dl = DataLoader(test_ds, batch_size=20, num_workers=1)

for batch, data in enumerate(test_dl):
    print(torch.mean(data[0]))
    if batch &gt; 10:
        break
</code></pre>
<p>Traceback (most recent call last):<br>
File “”, line 1, in <br>
File “~/python3.10/multiprocessing/spawn.py”, line 116, in spawn_main<br>
exitcode = _main(fd, parent_sentinel)<br>
File “~/python3.10/multiprocessing/spawn.py”, line 126, in _main<br>
self = reduction.pickle.load(from_parent)<br>
AttributeError: Can’t get attribute ‘RandomDataset’ on &lt;module ‘<strong>main</strong>’ (built-in)&gt;<br>
Output exceeds the size limit. Open the full output data in a text editor</p>
<p>RuntimeError                              Traceback (most recent call last)<br>
File ~/python3.10/site-packages/torch/utils/data/dataloader.py:1163, in _MultiProcessingDataLoaderIter._try_get_data(self, timeout)<br>
1162 try:<br>
 → 1163     data = self._data_queue.get(timeout=timeout)<br>
1164     return (True, data)</p>
<p>File ~/python3.10/multiprocessing/queues.py:113, in Queue.get(self, block, timeout)<br>
112 timeout = deadline - time.monotonic()<br>
 → 113 if not self._poll(timeout):<br>
114     raise Empty</p>
<p>File ~/python3.10/multiprocessing/connection.py:262, in _ConnectionBase.poll(self, timeout)<br>
261 self._check_readable()<br>
 → 262 return self._poll(timeout)</p>
<p>File ~/python3.10/multiprocessing/connection.py:429, in Connection._poll(self, timeout)<br>
428 def _poll(self, timeout):<br>
 → 429     r = wait([self], timeout)<br>
430     return bool(r)</p>
<p>File ~/python3.10/multiprocessing/connection.py:936, in wait(object_list, timeout)<br>
935 while True:<br>
 → 936     ready = selector.select(timeout)<br>
937     if ready:<br>
…<br>
 → 1176     raise RuntimeError(‘DataLoader worker (pid(s) {}) exited unexpectedly’.format(pids_str)) from e<br>
1177 if isinstance(e, queue.Empty):<br>
1178     return (False, None)</p>
<p>RuntimeError: DataLoader worker (pid(s) 19584) exited unexpectedly</p>
<pre><code class="lang-auto">#same code above, change workers to zero
test_ds = RandomDataset()
test_dl = DataLoader(test_ds, batch_size=20, num_workers=0)
for batch, data in enumerate(test_dl):
    print(torch.mean(data[0]))
    if batch &gt; 10:
        break
</code></pre>
<p>tensor(0.4931, dtype=torch.float64)<br>
tensor(0.4804, dtype=torch.float64)<br>
tensor(0.5428, dtype=torch.float64)<br>
tensor(0.5644, dtype=torch.float64)<br>
tensor(0.5368, dtype=torch.float64)<br>
tensor(0.4914, dtype=torch.float64)<br>
tensor(0.5419, dtype=torch.float64)<br>
tensor(0.5024, dtype=torch.float64)<br>
tensor(0.6744, dtype=torch.float64)<br>
tensor(0.5859, dtype=torch.float64)<br>
tensor(0.4658, dtype=torch.float64)<br>
tensor(0.5022, dtype=torch.float64)</p>

ptrblck: <p>I cannot reproduce the issue using the <code>1.13.0+cu117</code> binaries.</p>

builder173: <p>Hey, thanks for taking a look. I should add I’m running this in a notebook environment on VSCode right now. Below are the versions for the ipykernel environment. Would the notebook cause an issue with multi-processing?</p>
<blockquote>
<p>conda list | grep torch<br>
pytorch                   1.12.1                 py3.10_0    pytorch<br>
torch-tb-profiler         0.4.0                    pypi_0    pypi<br>
torchvision               0.2.2                      py_3    pytorch</p>
</blockquote>

ptrblck: <aside class="quote no-group" data-username="builder173" data-post="3" data-topic="166158">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/builder173/48/51744_2.png" class="avatar"> builder173:</div>
<blockquote>
<p>Would the notebook cause an issue with multi-processing?</p>
</blockquote>
</aside>
<p>I don’t know (as I’m generally not using notebooks) but this might be an interesting idea to check.<br>
Could you rerun the script in a terminal and see if the same output is seen?</p>

builder173: <p>Yeah, no error as a script with non-zero workers.</p>

builder173: <p>I guess it’s a thing - I’ll avoid workers in notebooks.</p><aside class="onebox stackexchange" data-onebox-src="https://stackoverflow.com/questions/48846085/python-multiprocessing-within-jupyter-notebook">
  <header class="source">

      <a href="https://stackoverflow.com/questions/48846085/python-multiprocessing-within-jupyter-notebook" target="_blank" rel="noopener nofollow ugc">stackoverflow.com</a>
  </header>

  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/9374864/curiouscientist" target="_blank" rel="noopener nofollow ugc">
    <img alt="curiouscientist" src="https://www.gravatar.com/avatar/d5914049ddedfdedb3b5451bdf515121?s=256&amp;d=identicon&amp;r=PG&amp;f=1" class="thumbnail onebox-avatar" width="256" height="256">
  </a>

<h4>
  <a href="https://stackoverflow.com/questions/48846085/python-multiprocessing-within-jupyter-notebook" target="_blank" rel="noopener nofollow ugc">Python Multiprocessing within Jupyter Notebook</a>
</h4>

<div class="tags">
  <strong>python, multiprocessing, jupyter</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://stackoverflow.com/users/9374864/curiouscientist" target="_blank" rel="noopener nofollow ugc">
    curiouscientist
  </a>
  on <a href="https://stackoverflow.com/questions/48846085/python-multiprocessing-within-jupyter-notebook" target="_blank" rel="noopener nofollow ugc">09:26PM - 17 Feb 18 UTC</a>
</div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>