Memory allocation errors when attempting to initialize a large number of small feed-forward networks in RAM with shared memory despite having enough memory

ryoukai: <p>Hello,</p>
<p>I am attempting to initialize and allocate space for ~10,000 small, single hidden layer mlps with shared memory. Here is how the models are created:</p>
<pre><code class="lang-auto">def model_factory(device, hidden_size=128, init_type='xavier_uniform', share_memory=True):
    model = AntNN(hidden_size=hidden_size, init_type=init_type).to(device)
    model.apply(model.init_weights)
    if share_memory:
        model.share_memory()
    return model

  mlps = []
  device = torch.device('cpu')
  for _ in range(num_policies):
      mlp = model_factory(device, 128, share_memory=True)
      mlps.append(mlp)
      print(f'RAM Memory % used: {psutil.virtual_memory()[2]}')
</code></pre>
<p>I’m keeping track of the total RAM usage and this is the last statement that printed before the error:</p>
<pre><code class="lang-auto">RAM Memory % used: 56.8
</code></pre>
<p>So I clearly have more than enough memory available. Here is the “cannot allocate memory” error</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "/home/user/map-elites/testing.py", line 164, in &lt;module&gt;
    mlp = model_factory(device, 128, share_memory=True)
  File "/home/user/map-elites/models/ant_model.py", line 11, in model_factory
    model.to(device).share_memory()
  File "/home/user/miniconda3/envs/map-elites/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1805, in share_memory
    return self._apply(lambda t: t.share_memory_())
  File "/home/user/miniconda3/envs/map-elites/lib/python3.8/site-packages/torch/nn/modules/module.py", line 578, in _apply
    module._apply(fn)
  File "/home/user/miniconda3/envs/map-elites/lib/python3.8/site-packages/torch/nn/modules/module.py", line 578, in _apply
    module._apply(fn)
  File "/home/user/miniconda3/envs/map-elites/lib/python3.8/site-packages/torch/nn/modules/module.py", line 601, in _apply
    param_applied = fn(param)
  File "/home/user/miniconda3/envs/map-elites/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1805, in &lt;lambda&gt;
    return self._apply(lambda t: t.share_memory_())
  File "/home/user/miniconda3/envs/map-elites/lib/python3.8/site-packages/torch/_tensor.py", line 482, in share_memory_
    self.storage().share_memory_()
  File "/home/user/miniconda3/envs/map-elites/lib/python3.8/site-packages/torch/storage.py", line 480, in share_memory_
    self._storage.share_memory_()
  File "/home/user/miniconda3/envs/map-elites/lib/python3.8/site-packages/torch/storage.py", line 160, in share_memory_
    self._share_filename_()
RuntimeError: unable to mmap 65600 bytes from file &lt;/torch_6936_3098808190_62696&gt;: Cannot allocate memory (12)

Process finished with exit code 1

</code></pre>
<p>With <code>share_memory=False</code> this works just fine, but for my application, it is critical that these tensors exist in shared memory b/c these tensors are modified by different processes. Is this some bug or a fundamental limitation to how shared memory in pytorch works? Is there any way to get around this problem?</p>
<p><strong>EDIT</strong>: I noticed that PyTorch’s default sharing system <code>file_descriptor</code> opens a lot of file descriptors, and that I might be reaching my system’s soft (or hard) open files limit. So I tried increasing the number from 1024 → 1,000,000 and was able to make it to</p>
<pre><code class="lang-auto">Num mlps: 6966 / 10,000
RAM Memory % used: 62.0
</code></pre>
<p>Before running into the mmap error. I tried playing around with different values for the max number of open file descriptors allowed by the system and couldn’t get past that number, so I don’t think it’s bottlenecked by the number of allowed file descriptors anymore. I also tried<br>
<code>torch.multiprocessing.set_sharing_strategy('file_system')</code> since it seems to keep track of less file descriptors per tensor, but this didn’t help either.</p>