RuntimeError: unable to open shared memory object (while using model.share_memory())

zarzen: <p>Hi,</p>
<p>I would like to put multiple models into CPU memory with shared memory support. So that I can easily transfer models among multiple processes.<br>
I am using code logic as following:</p>
<pre><code class="lang-auto">...
a_list_of_models = load_models()

for model in a_list_of_models:
    model.share_memory()
...
</code></pre>
<p>The code works fine with 4-5 models, but if I launched more models, I got <code>RuntimeError: unable to open shared memory object</code>.</p>
<p>I saw many other people have the same runtime error while using dataloader. While I am not using dataloader here, so I am not able to set the parameter <code>num_worker</code> to be 0.</p>
<p>I have also checked the shared memory limitation on my machine is unlimited. And the physical memory is more than enough to hold 100 models.</p>
<p>The most relevant topic is this one: <a href="https://discuss.pytorch.org/t/how-to-configure-shared-memory-size/20088" class="inline-onebox">How to configure shared memory size?</a><br>
but, unfortunately, no answer there.</p>
<p>any suggestions?</p>

rvarm1: <p>Hi,</p>
<p>Could you possibly share a more comprehensive repro of the issue such as the models that you are loading, if possible?</p>
<p>How did you check the shared memory limit on your machine? Does the output of  <code>ipcs  -lm</code> indicate that the shared memory is unlimited (this should tell you the max no. of SHM segments and max SHM size)? Also, just to confirm, have you checked if any other processes are taking up too much shared memory on your machine?</p>
<p>To increase the shared memory limit, you can try setting the <code>kernel.shmmax</code> parameter (i.e. <code>sysctl -w kernel.shmmax=...</code>), and follow the steps here: <a href="https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/5/html/tuning_and_optimizing_red_hat_enterprise_linux_for_oracle_9i_and_10g_databases/sect-oracle_9i_and_10g_tuning_guide-setting_shared_memory-setting_shmmni_parameter" rel="nofollow noopener">https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/5/html/tuning_and_optimizing_red_hat_enterprise_linux_for_oracle_9i_and_10g_databases/sect-oracle_9i_and_10g_tuning_guide-setting_shared_memory-setting_shmmni_parameter</a> to increase the # of shared memory segments available.</p>

zarzen: <p>Hi Rohan,</p>
<p>here is the code snippet for reproducing the bug:</p>
<pre><code class="lang-auto">import torch
from torchvision import models
import time

def main():
    """"""
    model_list = [
        ['resnet152', models.resnet152],
        ['inception_v3', models.inception_v3],
        ['vgg16', models.vgg16],
        ['vgg19', models.vgg19],
        ['vgg19_bn', models.vgg19_bn],
        ['densenet201', models.densenet201],
        ['densenet169', models.densenet169],
        ['resnet152-2', models.resnet152],
        ['resnet152-3', models.resnet152],
        ['resnet152-4', models.resnet152],
        ['resnet152-5', models.resnet152],
        ['resnet152-6', models.resnet152],
        ['resnet152-7', models.resnet152],
        ['resnet152-8', models.resnet152],
        ['resnet152-9', models.resnet152]
    ]
    models_dict = {}

    for m in model_list:
        model = m[1](pretrained=True)
        model.share_memory()
        models_dict[m[0]] = model
        print('loaded ', m[0])

    # while True:
    #     time.sleep(1)

if __name__ == "__main__":
    main()
</code></pre>
<p>Here is the outputs of <code>ipcs -lm</code>:</p>
<pre><code class="lang-auto">
------ Shared Memory Limits --------
max number of segments = 4096
max seg size (kbytes) = 18014398509465599
max total shared memory (kbytes) = 18014398509481980
min seg size (bytes) = 1
</code></pre>
<p>I have tried to increase <code>max number of segments</code> to <code>8192</code> (using <code>sysctl -w kernel.shmmni=8192</code>), it does not help.</p>

zarzen: <p>Another interesting fact:<br>
before the program got crash, I didn’t see any file created in <code>/dev/shm</code> folder. While the disk usage of <code>/dev/shm</code> is increasing (based on <code>df -h</code> command)</p>