DataLoader worker (pid xxx) is killed by signal: Hangup

seyuboglu: <p>I’m using a <code>DataLoader</code> with 28 workers. After training for 3.5k iterations (~ 6 hours), I got the following error:</p>
<pre><code class="lang-auto">    idx, data = self._get_data()
  File "...site-packages/torch/utils/data/dataloader.py", line 848, in _get_data
    success, data = self._try_get_data()
  File "...site-packages/torch/utils/data/dataloader.py", line 811, in _try_get_data
    data = self._data_queue.get(timeout=timeout)
  File "...python3.6/queue.py", line 173, in get
    self.not_empty.wait(remaining)
  File ".../python3.6/threading.py", line 299, in wait
    gotit = waiter.acquire(True, timeout)
  File "...site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
    _error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 487) is killed by signal: Hangup. 
</code></pre>
<p>Has anyone had <code>DataLoader</code> worker killed by <code>SIGHUP</code>?</p>
<p>There are a number of issues focused on DataLoader workers being killed by signal <a href="https://github.com/pytorch/pytorch/issues/5040" rel="nofollow noopener">BusError</a> and <a href="https://github.com/pytorch/pytorch/issues/8976" rel="nofollow noopener">Killed</a>. However, I can’t find any mention of DataLoader workers being killed by SIGHUP. My understanding of SIGHUP is that it is a signal sent to processes when their terminal is closed, so it strikes me as an odd signal for a worker process to be killed by.</p>
<p>Has anyone run into this before / have any insight into what kind of issue the SIGHUP signal would suggest?</p>
<p>Thanks!</p>

ptrblck: <p>SIGHUP is not necessarily raised, if a terminal is closed, and yesterday I’ve ran into it while debugging a faulty CUDA kernel.</p>
<p>Were you able to train for a whole epoch or are the mentioned 3.5k iterations still in the first epoch?<br>
In the latter case, I would recommend to use <code>num_workers=0</code> and rerun the code, which might yield a better error message in case the <code>DataLoader</code> isn’t able to load a sample.</p>
<p>You could also just try to iterate the <code>DataLoader</code> (with <code>num_workers=0</code>) standalone without training the model and make sure that all samples can be properly loaded.</p>

seyuboglu: <p>Thanks for the reply – that’s good to know! I am able to train for a whole epoch even with <code>num_workers&gt;0</code>; I just finished a job which completed without issue for 100 epochs. Still though, I’ve had a few more dataloaders killed by SIGHUP since I made this post. My <code>__get_item__()</code> includes some stochastic preprocessing, which could explain why some epochs complete and others do not. I’m also querying S3 through <code>boto3</code> which maybe is causing issues inside a parallelized dataloader? From <a href="https://pytorch.org/docs/stable/notes/multiprocessing.html#multiprocessing-best-practices" rel="nofollow noopener">Torch’s MP best practices</a> it sounds like some Python libraries use multiple threads and could lead to deadlock or other issues when used inside a <code>DataLoader</code>. Do you think a SIGHUP could raised for that kind of issue?</p>

ptrblck: <p>I don’t know, what kind of error will raise the SIGHUP.<br>
What kind of stochastic processing are you using inside the <code>Dataset</code>?<br>
Based on your description it seems you are also using multi-threading inside the <code>__getitem__</code>?</p>