Error while Multiprocessing in Dataloader

gaurav0651: <p>Not sure if this is reported already but I am getting the following Assertion error in Dataloader</p>
<p>Exception ignored in: &lt;bound method _DataLoaderIter.<strong>del</strong> of &lt;torch.utils.data.dataloader._DataLoaderIter object at 0x7fae94071d30&gt;&gt;<br>
Traceback (most recent call last):<br>
File “/home/amit/.local/lib/python3.6/site-packages/torch/utils/data/dataloader.py”, line 677, in <strong>del</strong><br>
self._shutdown_workers()<br>
File “/home/amit/.local/lib/python3.6/site-packages/torch/utils/data/dataloader.py”, line 659, in _shutdown_workers<br>
w.join()<br>
File “/usr/lib/python3.6/multiprocessing/process.py”, line 122, in join<br>
assert self._parent_pid == os.getpid(), ‘can only join a child process’<br>
AssertionError: can only join a child process</p>

matohak: <p>have you found a solution?</p>

uduse: <p>I am using <code>num_workers</code> with <code>IterableDataset</code> and it also has this problem.</p>

karzia: <p>yes…<br>
same here.</p>
<p>testset = torch.utils.data.DataLoader(trainset, batch_size=4, shuffle=True,num_workers=0)</p>
<p>But i just worried that it is possible to use only my local testing.<br>
So I just want to know what is root cause and a solution.</p>
<p>^^</p>

sharkdeng: <aside class="quote no-group" data-username="gaurav0651" data-post="1" data-topic="46845">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/gaurav0651/48/12579_2.png" class="avatar"> gaurav0651:</div>
<blockquote>
<p>can only join a child process</p>
</blockquote>
</aside>
<p>did you solve it. Or it is the problem of num_workers.</p>

Himani_Gulati: <p>Well I am getting the same error, it says can only join a child process. I do not know what that means??</p>

Gabriel_Barello: <p>I was having this issue. Turns out its because there was an error in the dataset object (for me it was in the <code>__getitem__</code> function). I guess the DataLoader in multiprocessing mode doesn’t know how to cleanly provide you with the internal error message. If you have the same problem, try running with <code>num_workers = 0</code> (single-threaded) and it should tell you what the error is. Once you’ve fixed the error, it should work with <code>num_workers &gt; 0</code>.</p>

Qinsheng_Zhang: <p>same issue happens. Not know where is the error.</p>

jpleet: <p>I just got this error. My data is good. And training with <em>num_workers=0</em> is too slow. For whatever reason, I was able to fix by replacing<br>
<em>from tqdm.auto import tqdm</em><br>
with just<br>
<em>from tqdm import tqdm</em><br>
Something seems to bug out with parallel dataloaders wrapped around the fancy notebook tqdm with my versions of nodejs and ipywidgets. Hope this helps others.</p>

JamesLai: <p>This solve my problem.<br>
Thanks a lot!</p>

Hmrishav_Bandyopadhy: <p>This is literally gold!</p>

brianw0924: <p>This solves my problem, too.</p>
<p>but why???</p>

adrofa: <p>This helped! Thank you!</p>

DarQ: <p>I’m not even using tqdm and my code works fine with num_workers=0. What could be the problem?</p>

akos.kiss: <p>Same here, no tqdm, code worked with num_workers=0, 1, 2, but saw a lot of these errors when num_worers&gt;=3.<br>
I ran the code inside docker and increasing the shared memory size (–shm-size 256M → 1G) solved the problem for me, now works fine with num_workers=12.</p>

Fang_Nan: <p>That works for me. Thanks!</p>

drscotthawley: <p>Not using tqdm but changing <code>num_workers</code> from 1 to 0 caused this error to go away on my Colab run! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

Noyii: <p>that really helps by replacing<br>
<em>from tqdm.auto import tqdm</em><br>
with just<br>
<em>from tqdm import tqdm</em></p>
<p>Thx a lot!</p>

steve_ari: <p>I just changed the <code>num_workers</code> to <code>num_workers=0</code>, run the training once and then change it back again to <code>num_workers=4</code>. The error just disappeared afterwards</p>

agg-shambhavi: <aside class="quote no-group" data-username="jpleet" data-post="9" data-topic="46845">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jpleet/48/30125_2.png" class="avatar"> jpleet:</div>
<blockquote>
<p>from tqdm import tqdm</p>
</blockquote>
</aside>
<p>The warnings were annoying me a lot. Thanks :))</p>