RuntimeError: unable to open shared memory object (depending on the model)

ClementPla: <p>Hi,</p>
<p>I’m facing an issue I’ve seen covered in others topics but I haven’t found any working solutions.<br>
I’m trying to use my library to train different models on a distributed way on a server containing 4 GPUs.<br>
For most models, it works as expected (DeepLabV3+, ResNet101, UNeT, Transformers…). There is one model though that does not work which is the HRN for Semantic Segmentation,<br>
<a href="https://github.com/HRNet/HRNet-Semantic-Segmentation/blob/pytorch-v1.1/lib/models/seg_hrnet.py" rel="noopener nofollow ugc">following the original implementation</a><br>
I did minor modification to this model to remove anything that wasn’t Pytorch/pure python objects. I also removed any class attributes (not sure if this helps). My <a href="https://pastebin.com/K8wgGJcK" rel="noopener nofollow ugc">re-implementation is here.</a></p>
<p>On a single GPU, the model works fine, but on multi-gpu DDP, I get the following message:</p>
<pre><code class="lang-auto">File "/usagers/clpla/.conda/envs/torch18/lib/python3.8/site-packages/torch/multiprocessing/reductions.py", line 321, in reduce_storage
RuntimeError: unable to open shared memory object &lt;/torch_403534_2756454997&gt; in read-write mode
</code></pre>
<p>This error raises when calling the torch.multiprocessing.spawn function <a href="https://github.com/ClementPla/NNTools/blob/347629c326202af839b002b2bf0b40b7675b7e31/nntools/experiment.py#L311" rel="noopener nofollow ugc">in my code</a>, so before the dataloaders are even created. I’ve still tried varying the num_workers (even set to 0), with no effects.<br>
Following other recommendations, I’ve tried increasing the shared memory size:</p>
<pre><code class="lang-auto"># cat /proc/sys/kernel/shmmni
8192
</code></pre>
<p>Which did not helped neither. Finally, I tried changing the sharing strategy to file_system, as described in the <a href="https://pytorch.org/docs/stable/multiprocessing.html" rel="noopener nofollow ugc">pytorch multiprocessing documentation</a><br>
In this case, the previous error does not appeared, but it leads to memory leaks (in regular RAM) after a few epochs and I suspect it comes from the dataloader. It also does not explain why the file_descriptor strategy would work on certain models and not others.</p>
<p>I’m running out of idea, I’m open to any suggestions. Let me know if more details are needed.  I could try to provide a minimal (non)-working example.</p>
<p>Specifications:</p>
<ul>
<li>OS : CentOS 7</li>
<li>Python 3.8.8</li>
<li>Pytorch: 1.8.0</li>
<li>4 GPUs RTX 2080</li>
<li>CUDA: 11.0</li>
<li>Driver: 450.66</li>
<li>RAM: 125G</li>
</ul>
<p>Thanks!</p>

wayi: <p>To summarize, you have tried 3 approaches (as also suggested in this <a href="https://github.com/facebookresearch/maskrcnn-benchmark/issues/103" rel="noopener nofollow ugc">thread</a>):</p>
<ol>
<li>Set <code>num_workers=0</code> (i.e., <code>self.config['Manager']['num_workers']=0</code>) when calling <code>DataLoader</code> constructor;</li>
<li>Increase shared memory size;</li>
<li>Change the <a href="https://pytorch.org/docs/master/multiprocessing.html?highlight=sharing%20strategy#sharing-strategies" rel="noopener nofollow ugc">sharing strategy</a>:</li>
</ol>
<pre><code class="lang-auto">import torch.multiprocessing
torch.multiprocessing.set_sharing_strategy('file_system')
</code></pre>
<p>Can you try a few more suggestions in <a href="https://github.com/pytorch/pytorch/issues/1355" class="inline-onebox" rel="noopener nofollow ugc">possible deadlock in dataloader · Issue #1355 · pytorch/pytorch · GitHub</a>?</p>
<p>It seems that someone also tried <code>ulimit -n</code> or <code>setNumThreads(0)</code>. If all the suggestions are exhausted, feel free to open a bug on Github.</p>

ClementPla: <p>Hi!<br>
Thanks for your reply, you perfectly summarized my approaches.<br>
And thanks to your suggestion, my model finally works!<br>
As you suggested, I had to call:</p>
<pre><code class="lang-auto">ulimit -n 64000
</code></pre>
<p>The 64000 is kinda arbitrary, it corresponds to the open files limit. I’m not really sure how this is is related to distributed training and what a good value should be. The default value of 1024 was enough for some models but not for other.</p>
<p>Anyway, some relevant infos if someones ends up with the same issue:</p>
<p>You need admin rights to call ulimit. Note that calling ulimit will only affect the current shell. Values will be resetted at reboot.<br>
Thanks again.</p>

tianhan4: <p>Great, it also helps me. Thank you!</p>