RuntimeError: DataLoader worker (pid 26317) is killed by signal: Aborted

wpwei: <p>Hi, I encountered into the following problem when I was trying to read a batch of relatively large data sample with multi-threaded DataLoader (with num_workers=4 for example).<br>
I have tried increasing the shared memory of ubuntu but did not work.<br>
It will run without the num_workers argument, but it is too slow to learn from a large data set with single thread.</p>
<p>Is there anyone knowing what’s going wrong? Thank you in advance.</p>
<p>Exception Details:</p>
<pre><code class="lang-auto">~/.pyenv/versions/3.6.4/lib/python3.6/site-packages/torch/utils/data/dataloader.py in __next__(self)
    273         while True:
    274             assert (not self.shutdown and self.batches_outstanding &gt; 0)
--&gt; 275             idx, batch = self._get_batch()
    276             self.batches_outstanding -= 1
    277             if idx != self.rcvd_idx:

~/.pyenv/versions/3.6.4/lib/python3.6/site-packages/torch/utils/data/dataloader.py in _get_batch(self)
    252                 raise RuntimeError('DataLoader timed out after {} seconds'.format(self.timeout))
    253         else:
--&gt; 254             return self.data_queue.get()
    255 
    256     def __next__(self):

~/.pyenv/versions/3.6.4/lib/python3.6/queue.py in get(self, block, timeout)
    162             elif timeout is None:
    163                 while not self._qsize():
--&gt; 164                     self.not_empty.wait()
    165             elif timeout &lt; 0:
    166                 raise ValueError("'timeout' must be a non-negative number")

~/.pyenv/versions/3.6.4/lib/python3.6/threading.py in wait(self, timeout)
    293         try:    # restore state no matter what (e.g., KeyboardInterrupt)
    294             if timeout is None:
--&gt; 295                 waiter.acquire()
    296                 gotit = True
    297             else:

~/.pyenv/versions/3.6.4/lib/python3.6/site-packages/torch/utils/data/dataloader.py in handler(signum, frame)
    173         # This following call uses `waitid` with WNOHANG from C side. Therefore,
    174         # Python can still get and update the process status successfully.
--&gt; 175         _error_if_any_worker_fails()
    176         if previous_handler is not None:
    177             previous_handler(signum, frame)

RuntimeError: DataLoader worker (pid 26317) is killed by signal: Aborted.
</code></pre>

BhaveshSethi: <p>Hi, did you find any solution for this problem?</p>

yuanzhi_liang: <p>I met this bug too. Did you solve this problem?</p>

guyue_hu: <p>In my case, I find that it caused by out of <code>memory of machine</code>(note that , not <code>GPU memory</code>). To test that when run your code, you can use command <code> top</code> to see the whether the memory and swap memory are used up. If so , it is the memory problem. You can  use a <code>small batch</code>, use a <code>small num_workers</code>, expand your swap memory,etc. Any way could decrease your memory load would be helpful.<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/a/a0715b54f1639c4d9029d1da30597a0cac1d5c67.png" data-download-href="https://discuss.pytorch.org/uploads/default/a0715b54f1639c4d9029d1da30597a0cac1d5c67" title="113%2489P%25UWQVSN%5DPQ3H%7D_%60T"><img src="https://discuss.pytorch.org/uploads/default/original/2X/a/a0715b54f1639c4d9029d1da30597a0cac1d5c67.png" alt="113%2489P%25UWQVSN%5DPQ3H%7D_%60T" data-base62-sha1="mTlgbmuLQK2wMqar1hbrIEVwNiT" width="690" height="94" data-dominant-color="333131"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">113%2489P%25UWQVSN%5DPQ3H%7D_%60T</span><span class="informations">701×96 7.44 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

sekedoua: <p>Hi <a class="mention" href="/u/guyue_hu">@guyue_hu</a> do you know the code to use a small num_workers and to expand wap memory ?</p>