Crashing during using DataLoaders with several workers from different threads

Konstantin_Burlachen: <p>In two projects I have the same problem.</p>
<p>Evnironemt: Linux, Ubunitu 16.04<br>
Python: 3.9.1<br>
Torch: 1.7.1</p>
<p>The problem lies in the case of using DataLoaders instantiated in the thread (main) for completely different datasets.  After that data-loaders are used in different threads.</p>
<p>DataLoaderes are configured with default “num_workers = 4”<br>
An attempt to have the ability from different threads to use different data loaders is crashing.</p>
<p>One workaround is to use “num_workers = 0” and it solves the problem…</p>
<p>It very likely has a connection with bug in Python: <a href="https://bugs.python.org/issue41567" class="inline-onebox" rel="noopener nofollow ugc">Issue 41567: multiprocessing.Pool from concurrent threads failure on 3.9.0rc1 - Python tracker</a></p>
<p>Stack traces:<br>
File “/home/burlack/miniconda3/envs/pl/lib/python3.9/threading.py”, line 954, in _bootstrap_inner<br>
self.run()<br>
File “/home/burlack/pl_new/pl_pytorch/pl_pytorch/utils/workers_threads.py”, line 50, in run<br>
retValues = f(self, *args)<br>
File “/home/burlack/pl_new/pl_pytorch/pl_pytorch/utils/model_funcs.py”, line 145, in local_training<br>
metrics_meter = train_model(<br>
File “/home/burlack/pl_new/pl_pytorch/pl_pytorch/utils/model_funcs.py”, line 223, in train_model<br>
for i, (data, label) in enumerate(train_loader):<br>
File “/home/burlack/miniconda3/envs/pl/lib/python3.9/site-packages/torch/utils/data/dataloader.py”, line 352, in <strong>iter</strong><br>
return self._get_iterator()<br>
File “/home/burlack/miniconda3/envs/pl/lib/python3.9/site-packages/torch/utils/data/dataloader.py”, line 294, in _get_iterator<br>
return _MultiProcessingDataLoaderIter(self)<br>
File “/home/burlack/miniconda3/envs/pl/lib/python3.9/site-packages/torch/utils/data/dataloader.py”, line 776, in <strong>init</strong><br>
self._worker_result_queue = multiprocessing_context.Queue()  # type: ignore<br>
File “/home/burlack/miniconda3/envs/pl/lib/python3.9/multiprocessing/context.py”, line 103, in Queue<br>
return Queue(maxsize, ctx=self.get_context())<br>
File “/home/burlack/miniconda3/envs/pl/lib/python3.9/multiprocessing/queues.py”, line 40, in <strong>init</strong><br>
from .synchronize import SEM_VALUE_MAX as maxsize<br>
ImportError: cannot import name ‘SEM_VALUE_MAX’ from partially initialized module ‘multiprocessing.synchronize’ (most likely due to a circular import) (/home/burlack/miniconda3/envs/pl/lib/python3.9/multiprocessing/synchronize.py)<br>
Exception ignored in: &lt;function _MultiProcessingDataLoaderIter.<strong>del</strong> at 0x14c8389d61f0&gt;<br>
Traceback (most recent call last):<br>
File “/home/burlack/miniconda3/envs/pl/lib/python3.9/site-packages/torch/utils/data/dataloader.py”, line 1203, in <strong>del</strong><br>
self._shutdown_workers()<br>
File “/home/burlack/miniconda3/envs/pl/lib/python3.9/site-packages/torch/utils/data/dataloader.py”, line 1150, in _shutdown_workers<br>
if not self._shutdown:</p>

Konstantin_Burlachen: <p>With python version 3.9.2, and torch version 3.8.1 the problem is still persist.</p>

ptrblck: <p>The import error points indeed towards a circular import. Would you mind creating an <a href="https://github.com/pytorch/pytorch/issues">issue</a> with the problem description and (if possible) a code snippet to reproduce the error?</p>
<aside class="quote no-group" data-username="Konstantin_Burlachen" data-post="2" data-topic="116391" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/konstantin_burlachen/48/36090_2.png" class="avatar"> Konstantin_Burlachen:</div>
<blockquote>
<p>With python version 3.9.2, and torch version 3.8.1 the problem is still persist.</p>
</blockquote>
</aside>
<p>Do you mean “Python 3.9.2 and PyTorch 1.8.1”?</p>

Konstantin_Burlachen: <p>Hello. Yes. It was a misprint.</p>