Memory allocation errors when attempting to initialize a large number of small feed-forward networks in RAM with shared memory despite having enough memory 
Labels: module: memory usage, triaged

### 🐛 Describe the bug

Hello,

I am attempting to initialize and allocate space for ~10,000 small, two hidden layer mlps with shared memory in RAM. Here is how the models are created:

```
import torch
import psutil
import torch.nn as nn

class AntNN(nn.Module):
    def __init__(self, input_dim=60, hidden_size=256, action_dim=8, init_type='xavier_uniform'):
        super().__init__()

        self.layers = nn.Sequential(
            nn.Linear(input_dim, hidden_size),
            nn.Tanh(),
            nn.Linear(hidden_size, hidden_size),
            nn.Tanh(),
            nn.Linear(hidden_size, hidden_size),
            nn.Tanh(),
            nn.Linear(hidden_size, action_dim)
        )

    def forward(self, obs):
        return self.layers(obs)

def model_factory(device, hidden_size=128, init_type='xavier_uniform', share_memory=True):
    model = AntNN(hidden_size=hidden_size, init_type=init_type).to(device)
    if share_memory:
        model.share_memory()
    return model

if __name__ == '__main__':
  num_policies = 10000
  mlps = []
  device = torch.device('cpu')
  for _ in range(num_policies):
      mlp = model_factory(device, 128, share_memory=True)
      mlps.append(mlp)
      print(f'RAM Memory % used: {psutil.virtual_memory()[2]}')
```

I’m keeping track of the total RAM usage and this is the last statement that printed before the error:

> RAM Memory % used: 56.8

So I clearly have more than enough memory available. Here is the “cannot allocate memory” error:

> Traceback (most recent call last):
  File "/home/user/map-elites/testing.py", line 164, in <module>
    mlp = model_factory(device, 128, share_memory=True)
  File "/home/user/map-elites/models/ant_model.py", line 11, in model_factory
    model.to(device).share_memory()
  File "/home/user/miniconda3/envs/map-elites/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1805, in share_memory
    return self._apply(lambda t: t.share_memory_())
  File "/home/user/miniconda3/envs/map-elites/lib/python3.8/site-packages/torch/nn/modules/module.py", line 578, in _apply
    module._apply(fn)
  File "/home/user/miniconda3/envs/map-elites/lib/python3.8/site-packages/torch/nn/modules/module.py", line 578, in _apply
    module._apply(fn)
  File "/home/user/miniconda3/envs/map-elites/lib/python3.8/site-packages/torch/nn/modules/module.py", line 601, in _apply
    param_applied = fn(param)
  File "/home/user/miniconda3/envs/map-elites/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1805, in <lambda>
    return self._apply(lambda t: t.share_memory_())
  File "/home/user/miniconda3/envs/map-elites/lib/python3.8/site-packages/torch/_tensor.py", line 482, in share_memory_
    self.storage().share_memory_()
  File "/home/user/miniconda3/envs/map-elites/lib/python3.8/site-packages/torch/storage.py", line 480, in share_memory_
    self._storage.share_memory_()
  File "/home/user/miniconda3/envs/map-elites/lib/python3.8/site-packages/torch/storage.py", line 160, in share_memory_
    self._share_filename_()
RuntimeError: unable to mmap 65600 bytes from file </torch_6936_3098808190_62696>: Cannot allocate memory (12)
Process finished with exit code 1

With share_memory=False this works just fine, but for my application, it is critical that these tensors exist in shared memory b/c these tensors are modified by different processes. Is this some bug or a fundamental limitation to how shared memory in pytorch works? Is there any way to get around this problem?

Also, I noticed that PyTorch’s default sharing system file_descriptor opens a lot of file descriptors, and that I might be reaching my system’s soft (or hard) open files limit. So I tried increasing the number from 1024 → 1,000,000 and was able to make it to

> Num mlps: 6966 / 10,000
RAM Memory % used: 62.0

Before running into the mmap error. I tried playing around with different values for the max number of open file descriptors allowed by the system and couldn’t get past that number, so I don’t think it’s bottlenecked by the number of allowed file descriptors anymore. I also tried 

```torch.multiprocessing.set_sharing_strategy('file_system')```
since it seems to keep track of less file descriptors per tensor, but this didn’t help either.

### Versions

Collecting environment information...
PyTorch version: 1.11.0+cu102
Is debug build: False
CUDA used to build PyTorch: 10.2
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.4 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Clang version: Could not collect
CMake version: version 3.16.3
Libc version: glibc-2.31

Python version: 3.8.5 | packaged by conda-forge | (default, Jul 31 2020, 02:39:48)  [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-5.13.0-44-generic-x86_64-with-glibc2.10
Is CUDA available: True
CUDA runtime version: 10.1.243
GPU models and configuration: GPU 0: NVIDIA GeForce GTX 1080
Nvidia driver version: 510.47.03
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.22.3
[pip3] torch==1.11.0
[pip3] torchvision==0.12.0
[conda] blas                      1.0                         mkl    conda-forge
[conda] cudatoolkit               10.2.89              hfd86e86_1  
[conda] libblas                   3.8.0                    16_mkl    conda-forge
[conda] libcblas                  3.8.0                    16_mkl    conda-forge
[conda] liblapack                 3.8.0                    16_mkl    conda-forge
[conda] liblapacke                3.8.0                    16_mkl    conda-forge
[conda] mkl                       2020.1                      217  
[conda] mkl-service               2.3.0            py38he904b0f_0  
[conda] mkl_fft                   1.1.0            py38h23d657b_0  
[conda] mkl_random                1.1.1            py38h0573a6f_0  
[conda] numpy                     1.22.3                   pypi_0    pypi
[conda] torch                     1.11.0                   pypi_0    pypi
[conda] torchvision               0.12.0                   pypi_0    pypi