How to debug ConnectionRefusedError in Data Loader?

Vedant_Roy: <p>I’m trying to debug a “ConnectionRefusedError: [Errno 111]” error:</p>
<pre><code class="lang-auto">(RayTrainWorker pid=3779494) This exception is thrown by __iter__ of IterableWrapperIterDataPipe(deepco
py=True, iterable=&lt;ray.train.torch.train_loop_utils._WrappedDataLoader object at 0x7f07bf4bf1c0&gt;)
(RayTrainWorker pid=3779494) Stack trace: Traceback (most recent call last):
(RayTrainWorker pid=3779494)   File "/home/ray/anaconda3/lib/python3.9/site-packages/torch/utils/data/d
ataloader.py", line 1126, in _try_get_data
(RayTrainWorker pid=3779494)     data = self._data_queue.get(timeout=timeout)
(RayTrainWorker pid=3779494)   File "/home/ray/anaconda3/lib/python3.9/multiprocessing/queues.py", line
 122, in get
(RayTrainWorker pid=3779494)     return _ForkingPickler.loads(res)
(RayTrainWorker pid=3779494)   File "/home/ray/anaconda3/lib/python3.9/site-packages/torch/multiprocess
ing/reductions.py", line 305, in rebuild_storage_fd
(RayTrainWorker pid=3779494)     fd = df.detach()
(RayTrainWorker pid=3779494)   File "/home/ray/anaconda3/lib/python3.9/multiprocessing/resource_sharer.
py", line 57, in detach
(RayTrainWorker pid=3779494)     with _resource_sharer.get_connection(self._id) as conn:
(RayTrainWorker pid=3779494)   File "/home/ray/anaconda3/lib/python3.9/multiprocessing/resource_sharer.
py", line 86, in get_connection
(RayTrainWorker pid=3779494)     c = Client(address, authkey=process.current_process().authkey)
(RayTrainWorker pid=3779494)   File "/home/ray/anaconda3/lib/python3.9/multiprocessing/connection.py", 
line 507, in Client
(RayTrainWorker pid=3779494)     c = SocketClient(address)
(RayTrainWorker pid=3779494)   File "/home/ray/anaconda3/lib/python3.9/multiprocessing/connection.py", 
line 635, in SocketClient
(RayTrainWorker pid=3779494)     s.connect(address)
(RayTrainWorker pid=3779494) ConnectionRefusedError: [Errno 111] Connection refused
</code></pre>
<p>Normally, this error seems to happen when something else has gone wrong in the data processing methods, but looking through my logs – I don’t see anything other exception.</p>
<p>The only thing that is fishy is that, almost always, a few batches (~ 10) before, I’ll get the following message:</p>
<pre><code class="lang-auto">(RayTrainWorker pid=3779497) /home/ray/anaconda3/lib/python3.9/site-packages/torch/utils/data/datapipes
/utils/common.py:295: FutureWarning: `FSSpecFileOpener()`'s functional API `.open_file_by_fsspec()` is 
deprecated since 0.4.0 and will be removed in 0.6.0.                                                   
(RayTrainWorker pid=3779497) See https://github.com/pytorch/data/issues/163 for details.               
(RayTrainWorker pid=3779497) Please use `.open_files_by_fsspec()` instead.                             
(RayTrainWorker pid=3779497)   warnings.warn(msg, FutureWarning)                                       
(RayTrainWorker pid=3779497) /home/ray/anaconda3/lib/python3.9/site-packages/torch/utils/data/datapipes
/utils/common.py:295: FutureWarning: `IoPathFileOpener()`'s functional API `.open_file_by_iopath()` is 
deprecated since 0.4.0 and will be removed in 0.6.0.                                                   
(RayTrainWorker pid=3779497) See https://github.com/pytorch/data/issues/163 for details.               
(RayTrainWorker pid=3779497) Please use `.open_files_by_iopath()` instead.                             
(RayTrainWorker pid=3779497)   warnings.warn(msg, FutureWarning)   
</code></pre>
<p>This is kind of surprising because</p>
<ol>
<li>
<code>open_file_by_iopath</code> is not used anywhere in my code.</li>
<li>This is happening in the <em>middle</em> of the training loop, which is suspicious, since no new datapipes should be constructed in the middle of an epoch.</li>
</ol>
<p>Any thoughts on what could be going on?</p>

Vedant_Roy: <p>Tagging <a class="mention" href="/u/ejguan">@ejguan</a> and <a class="mention" href="/u/nivek">@nivek</a> , since this is a torchdata question.</p>

ejguan: <p><code>open_file_by_iopath</code> should not affect your program as it’s just a warning.</p>
<p>Could you please try to use <code>IterableWrapper(ray_dl, deepcopy=False)</code>?</p>

ejguan: <p>BTW, which version of <code>torchdata</code> are you using?</p>

Vedant_Roy: <pre><code class="lang-auto">pip list | grep torchdata
</code></pre>
<p>returns</p>
<pre><code class="lang-auto"> 0.6.0.dev20221027
</code></pre>

Vedant_Roy: <p>Thanks! This seemed to fix the issue. (not sure why)</p>