Training crashes due to - Insufficient shared memory (shm) - nn.DataParallel

dg18: <p>Hi all,</p>
<p>I am training an image recognition model with dataset size (4M training images 200x200 size)</p>
<p>Here are the configurations of the training setup:</p>
<ol>
<li>pytorch v0.4.1</li>
<li>multi-GPU - 4</li>
<li>num_workers of my dataloader = 16</li>
<li>tried pin_memory=true / pin_memory=false</li>
<li>system configuration: 4 Tesla GPUs (6GB each) RAM: 128GB</li>
</ol>
<p>My training crashes after a few epochs with the error messages like:</p>
<p><strong>ERROR: Unexpected bus error encountered in worker. This might be caused by insufficient shared memory (shm)</strong></p>
<p>Here are my system’s shared memory limits:<br>
$ipcs -lm</p>
<pre><code class="lang-auto">------ Shared Memory Limits --------
max number of segments = 8192
max seg size (kbytes) = 18014398509465599
max total shared memory (kbytes) = 18446744073642442748
min seg size (bytes) = 1
</code></pre>
<p>Any ideas why I am getting shared memory crashes?</p>
<p>Any help is highly appreciated.</p>
<p>Thanks</p>

ptrblck: <p>Are you using a docker container?<br>
If so, you should increase the shared memory for the container as it might be too low.<br>
Have a look at the notes <a href="https://github.com/pytorch/pytorch#docker-image">here</a>.</p>

dg18: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>: No I am not using a docker container. I am using a conda installation.</p>

dg18: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>, pytorch users,</p>
<p>I noticed that this behaviour is related to using nn.DataParallel() (multi-GPUs) and num_workers &gt; 1 in torch.utils.data.DataLoader()</p>
<p>After a few epochs, the training crashes with errors like:<br>
" ERROR: Unexpected bus error encountered in worker. This might be caused by insufficient shared memory (shm)"</p>
<p>Are there any conflicts / problems  when using multiple workers in DataLoader() and nn.DataParallel() ?</p>
<p>Thanks</p>

dg18: <p><a class="mention" href="/u/apaszke">@apaszke</a>, <a class="mention" href="/u/smth">@smth</a></p>
<p>Hi guys,<br>
Please help with this.</p>
<p>Thanks.</p>

smth: <p>we fixed some errors wrt shared memory and DataLoader on master branch. Maybe try out PyTorch “Preview” build from our website, and see if that fixes it.</p>

An_Tran: <p>Hi <a class="mention" href="/u/smth">@smth</a><br>
I tried the PyTorch “Preview”, but the issue still remains the same. Could you please do a investigate for the convenience of users.</p>

Noirmist: <p>Is this issue is totally fixed? I have this issue on Pytorch 1.0.0.</p>

PistonY: <p>I have this issue when I set <code>DataLoader</code> num_workers to 32 or more.<br>
I use Pytorch 1.1.0.</p>

ptrblck: <p>More workers might use more shared memory, so you would need to increase the current limit.</p>

Andybert: <p>I have this issue on Pytorch 1.1.0 too. Any example for increasing the size of shared memory? thanks!</p>

Andybert: <p>I have this issue on Pytorch 1.1.0 too. Is there any way for users to set the size of shared memory? Thanks!</p>

ptrblck: <p>If you are using ubuntu, you could check the max shared memory size via:</p>
<pre><code class="lang-auto">sysctl kernel.shmmax
</code></pre>
<p>and set a new value in <code>/etc/sysctl.conf</code> as:</p>
<pre><code class="lang-auto">kernel.shmmax=6400000
</code></pre>

Andybert: <p>Thanks! When I execute the command:<br>
sysctl kernel.shmmax<br>
The result is:<br>
18446744073692774399<br>
Does that mean the value of shmmax in my system is big enough?</p>

ptrblck: <p>It might be bin enough.<br>
Which errors are you seeing that you assume your shared memory is not large enough?</p>

Andybert: <p>The error message is the same as <a href="https://discuss.pytorch.org/t/error-unexpected-bus-error-encountered-in-worker-this-might-be-caused-by-insufficient-shared-memory-shm/38719" class="inline-onebox">ERROR: Unexpected bus error encountered in worker. This might be caused by insufficient shared memory (shm)</a><br>
When I set <strong>num_works=0</strong>, the code can run normall.</p>

ggjy: <p>I get the same situation with next(iter(data_loader)) (My /dev/shm is 256G). Set num_workers=0 indeed can fix this, but num_workers=0 will take more time to load datas, there is an issue of this situation, <a href="https://github.com/pytorch/pytorch/issues/13246" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/13246</a>,  but can we have a better solution ?</p>

gabeur: <p>For me the issue was that I was already converting numpy arrays to torch tensors in the dataloader <code>__getitem__</code></p>
<p>Numpy arrays should only be converted to torch tensors in the trainer loop, just before being sent to the model. Otherwise the tensors will make the shared memory grow out of bounds.</p>
<p>You can monitor the shared memory by running the command <code>watch -n .3 df -h</code><br>
The shared memory corresponds to the line <code>/dev/shm</code><br>
The used amount should not increase after each epoch.</p>

pkubik: <p>I was always under impression that arrays <strong>should</strong> be converted to tenors in <code>__getitem__</code>. It’s shown in the tutorial: <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/data_loading_tutorial.html</a></p>
<p>That would make some sense since some kinds of data cannot be gathered into array until the <code>collate_fn</code>, e.g. text data. Why would they make the memory grow out of bound? I thought that CPU tensors are just wrappers for <code>ndarray</code>.</p>

akashs: <aside class="quote no-group" data-username="ptrblck" data-post="15" data-topic="26396" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>It might be bin enough.<br>
Which errors are you seeing that you assume your shared memory is not large enough?</p>
</blockquote>
</aside>
<p><a class="mention" href="/u/ptrblck">@ptrblck</a>  am still facing this error of shared memory is not large enough. I face this issue when I use large models. For example, If I use four resnet50 as a sub-models in a single large model, then I face this issue. However, if I change the four resnet50 to four resnet18 in that same single large model, then I didn’t face this shared memory issue. Is there any way I can increase the shared memory in PyTorch? or do I need to modify the UNIX system? Thanks in advance.</p>