Too many open files when using dataLoader

ahhhh: <p>Hi,<br>
When I use the data loader,  I have met the following error: Too many open files.<br>
In my implementation of the Dataset, I use torch.load(‘xxx’) to load the data files (which are all tensors stored on disks), and when call <strong>getitem</strong>(self, index), it will take the corresponding items from the tensor, and return it.</p>
<p>I construct the dalaloader in the following manner:</p>
<blockquote>
<p>dataset = Dataset(xxxxx)<br>
dataLoader = torch.utils.data.DataLoader(dataset=dataset, batch_size=batchSize, shuffle=shuffle,<br>
num_workers = numWorkers, collate_fn=dataset.collate_fn)</p>
</blockquote>
<p>The details of the error are in below:</p>
<blockquote>
<p>0:     referTypes, nNodes, featureIds, sIds) in enumerate(dataloader):<br>
0:   File “/u/home/home/anaconda2/lib/python2.7/site-packages/torch/utils/data/dataloader.py”, line 195, in <strong>next</strong><br>
0:     idx, batch = self.data_queue.get()<br>
0:   File “/u/home/home/anaconda2/lib/python2.7/multiprocessing/queues.py”, line 378, in get<br>
0:     return recv()<br>
0:   File “/u/home/home/anaconda2/lib/python2.7/site-packages/torch/multiprocessing/queue.py”, line 22, in recv<br>
0:     return pickle.loads(buf)<br>
0:   File “/u/home/home/anaconda2/lib/python2.7/pickle.py”, line 1388, in loads<br>
0:     return Unpickler(file).load()<br>
0:   File “/u/home/home/anaconda2/lib/python2.7/pickle.py”, line 864, in load<br>
0:     dispatch<a>key</a><br>
0:   File “/u/home/home/anaconda2/lib/python2.7/pickle.py”, line 1139, in load_reduce<br>
0:     value = func(*args)<br>
0:   File “/u/home/home/anaconda2/lib/python2.7/site-packages/torch/multiprocessing/reductions.py”, line 68, in rebuild_storage_fd<br>
0:     fd = multiprocessing.reduction.rebuild_handle(df)<br>
0:   File “/u/home/home/anaconda2/lib/python2.7/multiprocessing/reduction.py”, line 155, in rebuild_handle<br>
0:     conn = Client(address, authkey=current_process().authkey)<br>
0:   File “/u/home/home/anaconda2/lib/python2.7/multiprocessing/connection.py”, line 169, in Client<br>
0:     c = SocketClient(address)<br>
0:   File “/u/home/home/anaconda2/lib/python2.7/multiprocessing/connection.py”, line 320, in SocketClient<br>
0:     fd = duplicate(s.fileno())<br>
0: OSError: [Errno 24] Too many open files</p>
</blockquote>
<p>Thanks!</p>

SimonW: <p>Well, if could be that you are hitting your system’s limit. A lot of things counts as files on unix-based systems. Try rebooting. If that doesn’t work, you can try raise the limit as described at <a href="https://stackoverflow.com/questions/39537731/errno-24-too-many-open-files-but-i-am-not-opening-files">https://stackoverflow.com/questions/39537731/errno-24-too-many-open-files-but-i-am-not-opening-files</a></p>

ahhhh: <p>Thanks for the reply!<br>
In fact, I have already tried to set ulimit -100000…<br>
while the error still occurs sometimes.</p>

rudra_saha: <p>I am facing the same error, I have rebooted my ubuntu server as well as tried setting ulimit but I get <code>ulimit: value exceeds hard limit</code> when I try to set something very above 4096. Although, I get the output as <code>unlimited</code> when I run <code>ulimit</code>. Is there any other solution for this?</p>

rudra_saha: <p>I used the instructions mentioned <a href="https://docs.oracle.com/cd/E19623-01/820-6168/file-descriptor-requirements.html" rel="nofollow noopener">here</a> to increase my hard limit previously set as a workaround.</p>

Brando_Miranda: <p>I wonder if this would be helpful:</p>
<pre><code class="lang-auto">import torch.multiprocessing
torch.multiprocessing.set_sharing_strategy('file_system')
</code></pre>
<p><a href="https://pytorch.org/docs/stable/multiprocessing.html#file-descriptor-file-descriptor" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/multiprocessing.html#file-descriptor-file-descriptor</a></p><aside class="onebox githubissue">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/issues/11201" target="_blank" rel="noopener nofollow ugc">github.com/pytorch/pytorch</a>
  </header>
  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewBox="0 0 14 16" aria-hidden="true"><path d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/issues/11201" target="_blank" rel="noopener nofollow ugc">Too many open files error</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2018-09-03" data-time="13:56:30" data-timezone="UTC">01:56PM - 03 Sep 18 UTC</span>
      </div>

        <div class="date">
          closed <span class="discourse-local-date" data-format="ll" data-date="2018-09-14" data-time="18:39:40" data-timezone="UTC">06:39PM - 14 Sep 18 UTC</span>
        </div>

      <div class="user">
        <a href="https://github.com/whucdf" target="_blank" rel="noopener nofollow ugc">
          <img alt="whucdf" src="https://avatars.githubusercontent.com/u/42612199?v=4" class="onebox-avatar-inline" width="20" height="20">
          whucdf
        </a>
      </div>
    </div>
  </div>
</div>

<div class="github-row">
  <p class="github-content">Issue description
While using the dataloader from pytorch 0.4.1:
With num_workers &gt; 0 the workers store the tensors in shared memory, but do...</p>
</div>

<div class="labels">
    <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">medium priority (this tag is deprecated)</span>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


initialNeil: <p>Tried this and fixed it.</p>