RuntimeError: received 0 items of ancdata

username: Kyle
cooked: <pre><code>Traceback (most recent call last):
  File "process.py", line 73, in &lt;module&gt;
    eval('../amazon/densenet169_new',128,256,2)
  File "/home/zhenghuabin/jianglibin/amazon/eval.py", line 49, in eval
    for step, (data, target) in enumerate(validate_loader):
  File "/home/zhenghuabin/anaconda3/envs/py35/lib/python3.5/site-packages/torch/utils/data/dataloader.py", line 206, in __next__
    idx, batch = self.data_queue.get()
  File "/home/zhenghuabin/anaconda3/envs/py35/lib/python3.5/multiprocessing/queues.py", line 345, in get
    return ForkingPickler.loads(res)
  File "/home/zhenghuabin/anaconda3/envs/py35/lib/python3.5/site-packages/torch/multiprocessing/reductions.py", line 70, in rebuild_storage_fd
    fd = df.detach()
  File "/home/zhenghuabin/anaconda3/envs/py35/lib/python3.5/multiprocessing/resource_sharer.py", line 58, in detach
    return reduction.recv_handle(conn)
  File "/home/zhenghuabin/anaconda3/envs/py35/lib/python3.5/multiprocessing/reduction.py", line 181, in recv_handle
    return recvfds(s, 1)[0]
  File "/home/zhenghuabin/anaconda3/envs/py35/lib/python3.5/multiprocessing/reduction.py", line 160, in recvfds
    len(ancdata))
RuntimeError: received 0 items of ancdata
</code></pre>
<p>Here is my code:</p>
<pre><code>dset_validate = AmazonDateset_validate(validate_index)
validate_loader = DataLoader(dset_validate, batch_size=batch_size, num_workers=4)
for step, (data, target) in enumerate(validate_loader):
</code></pre>
<p>It always get error in the same position(finish some particular epoch). How to solve it?</p>

username: Kyle
cooked: <p>This problem happen in some particular machine, I mean it happens on one of my machines while the same code works well on other servers, and I can get rid by “num_workers=0”, but it makes my  training too slow.</p>

username: sampathweb
cooked: <p><a class="mention" href="/u/kyle">@Kyle</a> - Check this github issue (<a href="https://github.com/pytorch/pytorch/issues/973" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/973</a>) and see towards the end for the potential fix - increasing <code>ulimit</code></p>

username: Swamy
cooked: <p><code>torch.multiprocessing.set_sharing_strategy('file_system')</code><br>
Using this solved for me.</p>

username: btorb
cooked: <p>Most of the solutions here deal with the symptoms, but not the cause.</p>
<p><a href="https://github.com/pytorch/pytorch/issues/973#issuecomment-459398189" rel="nofollow noopener">This link</a> explains the cause (a memory leak when storing to much data from a dataloader) and how to solve it.</p>
<p>I copy the code from the aforementioned link:</p>
<pre><code class="lang-auto">pred_list = []
target_list = []
# long version
for inputs, targets in DataLoader(dataset, num_workers=6, batch_size=64):
    pred_list.append(model.predict_on_batch(inputs))  # make model prediction
    targets_copy = deepcopy(targets)
    target_list.append(targets_copy)
    del inputs
    del targets
</code></pre>

username: Harry_Liu
cooked: <p>set “num_workers=0”  fixes my issue. Thank you.</p>

username: Ludwig_Friborg
cooked: <p>With larger test sets not using multiprocessing may be a lot slower.<br>
Btorb’s solution the better solution, the essential part here is to <code>deepcopy</code> the targets generated.</p>
<p><code>from copy import deepcopy</code></p>

username: Zador
cooked: <p>This worked for me. Any idea why this is needed?</p>

username: Kris_Man
cooked: <p>Yes!!! It works for me</p>