`persistent_workers=True` for the DataLoader causes errors

lbhm: <p>Hello all,</p>
<p>I upgraded from PyTorch 1.6 to 1.7 today because I wanted to run some experiments with custom datasets and the new <code>persistent_workers</code> argument from the <code>DataLoader</code> class.</p>
<p>When I’m running my experiments, I keep getting errors from some internal files related to pinned memory. To make the issue reproducible, I also ran an experiment with the ResNet on ImageNet example from the PyTorch repo [1]. The only thing that I changed is that I fixed the number of classes to 200 because I am using a small subset of ImageNet for faster experimentation.</p>
<p>This is the error that I’m receiving:</p>
<pre><code class="lang-auto">Exception in thread Thread-1:
Traceback (most recent call last):
  File "/home/lbhm/miniconda3/envs/pt17/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/home/lbhm/miniconda3/envs/pt17/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/home/lbhm/miniconda3/envs/pt17/lib/python3.8/site-packages/torch/utils/data/_utils/pin_memory.py", line 28, in _pin_memory_loop
    idx, data = r
ValueError: not enough values to unpack (expected 2, got 0)

Traceback (most recent call last):
  File "/home/lbhm/Projects/dl2-benchmark/dl2/pytorch/experiments/pytorch_test_raw.py", line 429, in &lt;module&gt;
    main()
  File "/home/lbhm/Projects/dl2-benchmark/dl2/pytorch/experiments/pytorch_test_raw.py", line 112, in main
    main_worker(args.gpu, ngpus_per_node, args)
  File "/home/lbhm/Projects/dl2-benchmark/dl2/pytorch/experiments/pytorch_test_raw.py", line 245, in main_worker
    train(train_loader, model, criterion, optimizer, epoch, args)
  File "/home/lbhm/Projects/dl2-benchmark/dl2/pytorch/experiments/pytorch_test_raw.py", line 280, in train
    for i, (images, target) in enumerate(train_loader):
  File "/home/lbhm/miniconda3/envs/pt17/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 349, in __iter__
    self._iterator._reset(self)
  File "/home/lbhm/miniconda3/envs/pt17/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 852, in _reset
    data = self._get_data()
  File "/home/lbhm/miniconda3/envs/pt17/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1029, in _get_data
    raise RuntimeError('Pin memory thread exited unexpectedly')
RuntimeError: Pin memory thread exited unexpectedly

Process finished with exit code 1
</code></pre>
<p>I just stumbled upon the <code>persistent_worker</code> feature today when I thought about some ideas related to caching information in a custom dataset and couldn’t find any helpful leads so far. Did anyone experience this error before or has an idea where this could come from?</p>
<p>[1] <a href="https://github.com/pytorch/examples/blob/master/imagenet/main.py" class="inline-onebox" rel="noopener nofollow ugc">examples/main.py at master · pytorch/examples · GitHub</a></p>
<p><strong>Update:</strong> To verify that this error is specifically related to pinned memory, I also ran the script above with <code>pin_mememory=False</code> when creating the data loader and the error did not occur. So there seems to be an erroneous interaction between <code>pin_memory=True</code> and <code>persistent_workers=True</code>.</p>

JamesDickens: <p>Has this been fixed? I am running into the same problem.</p>

enterthevoidf22: <p>running into a similar problem!</p>

JamesDickens: <p>Apparently it has been fixed in the nightly build a few days ago:</p>
<aside class="onebox githubcommit">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/commit/54ce171f16c8859f829dde09f87c364c8a6b4130" target="_blank" rel="noopener nofollow ugc">github.com/pytorch/pytorch</a>
  </header>
  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Commit">
    <svg width="60" height="60" class="github-icon" viewBox="0 0 14 16" aria-hidden="true"><path d="M10.86 7c-.45-1.72-2-3-3.86-3-1.86 0-3.41 1.28-3.86 3H0v2h3.14c.45 1.72 2 3 3.86 3 1.86 0 3.41-1.28 3.86-3H14V7h-3.14zM7 10.2c-1.22 0-2.2-.98-2.2-2.2 0-1.22.98-2.2 2.2-2.2 1.22 0 2.2.98 2.2 2.2 0 1.22-.98 2.2-2.2 2.2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/commit/54ce171f16c8859f829dde09f87c364c8a6b4130" target="_blank" rel="noopener nofollow ugc">Fix persistent_workers + pin_memory (#48543)</a>
    </h4>

    <div class="github-info">
      <div class="date">
        committed <span class="discourse-local-date" data-format="ll" data-date="2021-01-08" data-time="14:56:10" data-timezone="UTC">02:56PM - 08 Jan 21 UTC</span>
      </div>

      <div class="user">
        <a href="https://github.com/SsnL" target="_blank" rel="noopener nofollow ugc">
          <img alt="SsnL" src="https://avatars2.githubusercontent.com/u/5674597?v=4" class="onebox-avatar-inline" width="20" height="20">
          SsnL
        </a>
        
      </div>

      <div class="lines" title="changed 3 files with 17 additions and 12 deletions">
        <a href="https://github.com/pytorch/pytorch/commit/54ce171f16c8859f829dde09f87c364c8a6b4130" target="_blank" rel="noopener nofollow ugc">
          <span class="added">+17</span>
          <span class="removed">-12</span>
        </a>
      </div>
    </div>

  </div>
</div>


  <div class="github-row">
    <pre class="github-content" style="white-space: normal;">Summary:
Fixes https://github.com/pytorch/pytorch/issues/48370 https://github.com/pytorch/pytorch/issues/47445
cc emcastillo who authored the original functionality.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/48543
Reviewed By: bdhirsh
Differential Revision: D25277474
Pulled By: ejguan
fbshipit-source-id: 1967002124fb0fff57caca8982bc7df359a059a2</pre>
  </div>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


meshghi: <p>still running into a similar problem!</p>

ptrblck: <p>Which nightly version are you using and could you post the complete error message?</p>