RuntimeError: unable to open shared memory object

hongtaesuk: <p>Hey guys, I having trouble with loading indexes of document.<br>
I am testing my code, so I set</p>
<pre><code class="lang-auto">batch_size = 4
number_of_sentences_in_document = 84
number_of_words_in_sentence = 80
</code></pre>
<p>that sums up one mini_batch with<br>
80 * 84 * 4 indexes of documents.</p>
<p>The problem is that when I transform those indexes dataset into a DataLoader as below<br>
and try to loop over trainloader, it results out so many error messages.</p>
<pre><code class="lang-auto">DataManager = DS.NewsDataset(data_examples_gen, Vocab)
trainloader = torch.utils.data.DataLoader(DataManager, batch_size=Args.args.batch_size, shuffle=True, num_workers=32)
</code></pre>
<p>The error messages are below.</p>
<blockquote>
<p>Traceback (most recent call last):<br>
File “/home/nlpgpu3/anaconda3/envs/linohong3/lib/python3.6/multiprocessing/process.py”, line 258, in _bootstrap<br>
self.run()<br>
File “/home/nlpgpu3/anaconda3/envs/linohong3/lib/python3.6/multiprocessing/process.py”, line 93, in run<br>
self._target(*self._args, **self._kwargs)<br>
File “/home/nlpgpu3/anaconda3/envs/linohong3/lib/python3.6/site-packages/torch/utils/data/dataloader.py”, line 61, in _worker_loop<br>
data_queue.put((idx, samples))<br>
File “/home/nlpgpu3/anaconda3/envs/linohong3/lib/python3.6/multiprocessing/queues.py”, line 341, in put<br>
File “/home/nlpgpu3/anaconda3/envs/linohong3/lib/python3.6/multiprocessing/reduction.py”, line 51, in dumps<br>
File “/home/nlpgpu3/anaconda3/envs/linohong3/lib/python3.6/site-packages/torch/multiprocessing/reductions.py”, line 121, in reduce_storage<br>
RuntimeError: unable to open shared memory object &lt;/torch_54163_3383444026&gt; in read-write mode at /opt/conda/conda-bld/pytorch_1525909934016/work/aten/src/TH/THAllocator.c:342</p>
<p>During handling of the above exception, another exception occurred:</p>
<p>Traceback (most recent call last):<br>
File “/home/nlpgpu3/anaconda3/envs/linohong3/lib/python3.6/multiprocessing/util.py”, line 262, in _run_finalizers<br>
File “/home/nlpgpu3/anaconda3/envs/linohong3/lib/python3.6/multiprocessing/util.py”, line 186, in <strong>call</strong><br>
File “/home/nlpgpu3/anaconda3/envs/linohong3/lib/python3.6/shutil.py”, line 476, in rmtree<br>
File “/home/nlpgpu3/anaconda3/envs/linohong3/lib/python3.6/shutil.py”, line 474, in rmtree<br>
OSError: [Errno 24] Too many open files: ‘/tmp/pymp-sgew4xdn’<br>
Process Process-1:<br>
Traceback (most recent call last):<br>
File “/home/nlpgpu3/anaconda3/envs/linohong3/lib/python3.6/multiprocessing/process.py”, line 258, in _bootstrap<br>
self.run()<br>
File “/home/nlpgpu3/anaconda3/envs/linohong3/lib/python3.6/multiprocessing/process.py”, line 93, in run<br>
self._target(*self._args, **self._kwargs)<br>
File “/home/nlpgpu3/anaconda3/envs/linohong3/lib/python3.6/site-packages/torch/utils/data/dataloader.py”, line 61, in _worker_loop<br>
data_queue.put((idx, samples))<br>
File “/home/nlpgpu3/anaconda3/envs/linohong3/lib/python3.6/multiprocessing/queues.py”, line 341, in put<br>
File “/home/nlpgpu3/anaconda3/envs/linohong3/lib/python3.6/multiprocessing/reduction.py”, line 51, in dumps<br>
File “/home/nlpgpu3/anaconda3/envs/linohong3/lib/python3.6/site-packages/torch/multiprocessing/reductions.py”, line 121, in reduce_storage<br>
RuntimeError: unable to open shared memory object &lt;/torch_54163_3383444026&gt; in read-write mode at /opt/conda/conda-bld/pytorch_1525909934016/work/aten/src/TH/THAllocator.c:342<br>
Traceback (most recent call last):<br>
File “/home/nlpgpu3/LinoHong/FakeNewsByTitle/main.py”, line 25, in <br>
for mini_batch in trainloader :<br>
File “/home/nlpgpu3/anaconda3/envs/linohong3/lib/python3.6/site-packages/torch/utils/data/dataloader.py”, line 280, in <strong>next</strong><br>
idx, batch = self._get_batch()<br>
File “/home/nlpgpu3/anaconda3/envs/linohong3/lib/python3.6/site-packages/torch/utils/data/dataloader.py”, line 259, in _get_batch<br>
return self.data_queue.get()<br>
File “/home/nlpgpu3/anaconda3/envs/linohong3/lib/python3.6/multiprocessing/queues.py”, line 335, in get<br>
res = self._reader.recv_bytes()<br>
File “/home/nlpgpu3/anaconda3/envs/linohong3/lib/python3.6/multiprocessing/connection.py”, line 216, in recv_bytes<br>
buf = self._recv_bytes(maxlength)<br>
File “/home/nlpgpu3/anaconda3/envs/linohong3/lib/python3.6/multiprocessing/connection.py”, line 407, in _recv_bytes<br>
buf = self._recv(4)<br>
File “/home/nlpgpu3/anaconda3/envs/linohong3/lib/python3.6/multiprocessing/connection.py”, line 379, in _recv<br>
chunk = read(handle, remaining)<br>
File “/home/nlpgpu3/anaconda3/envs/linohong3/lib/python3.6/site-packages/torch/utils/data/dataloader.py”, line 178, in handler<br>
_error_if_any_worker_fails()<br>
RuntimeError: DataLoader worker (pid 54163) exited unexpectedly with exit code 1.</p>
<p>Process finished with exit code 1</p>
</blockquote>
<p>It has error kinds</p>
<blockquote>
<ol>
<li>RuntimeError: unable to open shared memory object &lt;/torch_54163_3383444026&gt; in read-write mode at /opt/conda/conda-bld/pytorch_1525909934016/work/aten/src/TH/THAllocator.c:342</li>
<li>OSError: [Errno 24] Too many open files: ‘/tmp/pymp-sgew4xdn’</li>
<li>RuntimeError: DataLoader worker (pid 54163) exited unexpectedly with exit code 1.</li>
</ol>
</blockquote>
<p>I thought this is somekind of memory problem so I tried the same thing<br>
only having two sentences for a document and it worked.<br>
However, I am expecting this to get pretty much larger with<br>
batch_size up to 32 or 64,<br>
the number of sentences per document up to 84<br>
the number of words per sentence up to 84.<br>
How can I manage this problem?<br>
Any Idea???</p>

ptrblck: <p>Could you check the suggestions in <a href="https://github.com/pytorch/pytorch/issues/1355">this issue</a>?<br>
It looks like your shared memory is too low.</p>
<p>Does your code run with less <code>num_workers</code> or <code>num_workers=0</code>?</p>

hongtaesuk: <p>WOW!! thanks!!<br>
it worked when I set num_workers equals to 0.<br>
doesn’t work greater or equal to 1.</p>
<p>But, doesn’t that mean that if I set it equals to 0 then<br>
I’m not using any GPUs??</p>

ptrblck: <p>No, <code>num_workers=0</code> means your data is loaded in the main process. You can still use the GPU although your code might be slower. Could you try to increase the shared memory and try setting <code>num_workers&gt;0</code> again?</p>

Maxwell_Albert: <p>I met the same problem, and once I set num_workers up to more than one, I randomly cause this problem. Could you give me any help?</p>

ptrblck: <p>The error usually means that your system doesn’t provide enough shared memory for multiple workers (used via <code>num_workers&gt;0</code>). Check the shared memory limitation of your system and try to increase it.</p>

Maxwell_Albert: <p>Thank you！<br>
But I don’t regard this as the reason. Actually it is about pytorch version. When I use 1.8.0 and nightly, it works well. Hoever when I use 1.10.2(which is latest stable version), it always have this problem.<br>
And I test this for changing sharing memory parttern to ‘file system’. And I also enlarge shared memory.<br>
Some one has created an issue on github. And they fixed it at nightly pyotch</p>

ptrblck: <p>Ah, interesting. Could you post the GitHub issue here for visibility?</p>

Maxwell_Albert: <aside class="onebox githubpullrequest" data-onebox-src="https://github.com/pytorch/pytorch/pull/72232">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/pull/72232" target="_blank" rel="noopener nofollow ugc">github.com/pytorch/pytorch</a>
  </header>

  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Pull Request">
    <svg width="60" height="60" class="github-icon" viewbox="0 0 12 16" aria-hidden="true"><path d="M11 11.28V5c-.03-.78-.34-1.47-.94-2.06C9.46 2.35 8.78 2.03 8 2H7V0L4 3l3 3V4h1c.27.02.48.11.69.31.21.2.3.42.31.69v6.28A1.993 1.993 0 0 0 10 15a1.993 1.993 0 0 0 1-3.72zm-1 2.92c-.66 0-1.2-.55-1.2-1.2 0-.65.55-1.2 1.2-1.2.65 0 1.2.55 1.2 1.2 0 .65-.55 1.2-1.2 1.2zM4 3c0-1.11-.89-2-2-2a1.993 1.993 0 0 0-1 3.72v6.56A1.993 1.993 0 0 0 2 15a1.993 1.993 0 0 0 1-3.72V4.72c.59-.34 1-.98 1-1.72zm-.8 10c0 .66-.55 1.2-1.2 1.2-.65 0-1.2-.55-1.2-1.2 0-.65.55-1.2 1.2-1.2.65 0 1.2.55 1.2 1.2zM2 4.2C1.34 4.2.8 3.65.8 3c0-.65.55-1.2 1.2-1.2.65 0 1.2.55 1.2 1.2 0 .65-.55 1.2-1.2 1.2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/pull/72232" target="_blank" rel="noopener nofollow ugc">use random number in shared file name</a>
    </h4>

    <div class="branches">
      <code>pytorch:master</code> ← <code>pytorch:ngimel/map_alloc_rand</code>
    </div>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2022-02-03" data-time="02:09:02" data-timezone="UTC">02:09AM - 03 Feb 22 UTC</span>
      </div>

      <div class="user">
        <a href="https://github.com/ngimel" target="_blank" rel="noopener nofollow ugc">
          <img alt="ngimel" src="https://avatars.githubusercontent.com/u/15841449?v=4" class="onebox-avatar-inline" width="20" height="20">
          ngimel
        </a>
      </div>

      <div class="lines" title="2 commits changed 1 files with 5 additions and 1 deletions">
        <a href="https://github.com/pytorch/pytorch/pull/72232/files" target="_blank" rel="noopener nofollow ugc">
          <span class="added">+5</span>
          <span class="removed">-1</span>
        </a>
      </div>
    </div>
  </div>
</div>

  <div class="github-row">
    <p class="github-body-container">A tentative fix for #67864</p>
  </div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<p>
here is the fix in the nightly pytorch. And I have tested it.<br>
In this page there is a link for decription for this problem</p>