Pytorch Cannot allocate memory

Meisam_Hasani: <p>I am trying to load the huge coco dataset (120000-image) and do some training. I am using my docker container for the task.<br>
For faster training I try to load the whole data using pytorch dataloader into a python array (on the system memory not the gpu memory), and feed the model with that python array, so I won’t use the dataloader during the training.<br>
The problem is after loading a bunch of data (something around 10-15 GB) I encounter this strange error:</p>
<pre><code class="lang-auto">File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 990, in _try_get_data
  data = self._data_queue.get(timeout=timeout)
File "/opt/conda/lib/python3.8/multiprocessing/queues.py", line 116, in get
  return _ForkingPickler.loads(res)
File "/opt/conda/lib/python3.8/site-packages/torch/multiprocessing/reductions.py", line 294, in rebuild_storage_fd
  storage = cls._new_shared_fd(fd, size)
RuntimeError: falseINTERNAL ASSERT FAILED at "/opt/pytorch/pytorch/aten/src/ATen/MapAllocator.cpp":303, please report a bug to PyTorch. unable to mmap 8 bytes from file &lt;filename not specified&gt;: Cannot allocate memory (12)
</code></pre>
<p>I know it’s a huge data but the numbers match. I am using a linux server which has 200GB memory, so it’s not a memory lacking problem. I also set the docker shared memory on 200 GB to overcome the docker limit on memory usage, but still the problem exist.<br>
For smaller dataset (which needs less than 20GB) my code works fine.<br>
What should I do for this volume of data? What did I miss?</p>

Meisam_Hasani: <p>I put</p>
<pre><code class="lang-auto">num_worker = 0
</code></pre>
<p>and solved the problem!</p>

Chenkai_Mao: <p>Where and how did you add this argument?</p>

Meisam_Hasani: <p>In the dataloader:</p>
<pre><code class="lang-auto">dataloaders = torch.utils.data.DataLoader( image_datasets['train']	
    	                                            ,batch_size=batch_size
    	                                            ,num_workers=num_workers
    	                                            ,collate_fn=collate_fn
    	                                    )
</code></pre>
<p>Please check</p>
<blockquote>
<p>torch.utils.data.DataLoader</p>
</blockquote>
<p>in the documentation for more details.</p>

visionscaper: <p>The question is, why doesn’t it work with multiple DataLoader workers?<br>
You did not solve the problem, you circumvented it.</p>
<p>I also experience this with FashionMNIST using the latest Pytorch 1.11 (torch vision 0.12.0).</p>
<p>I’m getting the following error:</p>
<pre><code class="lang-auto">unable to mmap 3136 bytes from file &lt;/torch_29142_1440317042_64566&gt;: Cannot allocate memory (12)
</code></pre>
<p><a class="mention" href="/u/ptrblck">@ptrblck</a> Do you know what kind of error this is, and why it could occur specifically when multiple data loaders are used?</p>

ptrblck: <p>Unsure what’s causing the issue but based on e.g. <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=650801#c6">this older bug description</a> it seems as if Python fails to allocate the resident size for a new process.<br>
How much system RAM do you have and how much does the main Python process need?</p>

Lu_Dai: <p>I also met the same problem and solved it by modifying num_workers to 0 as well. Even I changed num_workers to 1, this “cannot allocate memory” problem would recur. I still wonder why this problem would happen?</p>

Charlie_Li: <p>Having the same issue here, setting <code>num_workers</code> to any value beyond 0 will result in cannot allocate memory, but using <code>num_workers </code>0 will result in an extremely slow training process.</p>

YijianZhou: <p>I believe I met the same problem, and as other people here reported, only when num_worker=0 is bug-free</p>

Andreas_Kopp1: <p>I have the same issue. Are there any news on this bug? Setting num_workers to 0 works as workaround but is not a good solution as it takes a lot of time in my case.</p>

Andreas_Kopp1: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>. Do you know if there any update on the issue? Setting workers to zero is not really a solution for large datasets. I am experiencing the issue even on a system with 880 GB of RAM.</p>

ptrblck: <p>No, sorry I don’t have any updates as I was never able to reproduce the issue. So far nobody was able to post a minimal and executable code snippet which would raise this error in a current release, so I don’t have a way to debug it.</p>

Andreas_Kopp1: <p>I can reproduce the issue consistently in an environment that I is too big to share it in a code snippet. I will work on a simplified repro that I can share. In the meantime, do you have additional hints for troubleshooting?<br>
Here are some info about the setup:</p>
<ul>
<li>Image multilabel classification with &gt;700k images with DistributedDataParallel</li>
<li>Training runs in Docker container</li>
<li>96 cores, 880 GB RAM, 4 A100 GPU</li>
<li>Shared mem set to &gt;200GB</li>
<li>“Cannot allocate memory” occurs in eval pass of first epoch</li>
<li>no_workers = 20</li>
<li>Error does not occur when setting no_workers to zero (as others have experienced) but makes training very slow<br>
Thanks for any guidance!</li>
</ul>

ptrblck: <p>I don’t know if you are seeing the same stacktrace, but the first posted one shows the failure in <code>storage = cls._new_shared_fd(fd, size)</code>,so I would start by checking:</p>
<ul>
<li>how many file descriptors are already open in the process (e.g. via <code>lsof</code> or by reading <code>/prod/pid/fd</code>)</li>
<li>all allocated and max. handles (e.g. via <code>sysctl fs.file-nr</code>)</li>
<li>if your docker setup somehow limits the file descriptor usage.</li>
</ul>

Andreas_Kopp1: <p>Thank you very much, <a class="mention" href="/u/ptrblck">@ptrblck</a><br>
Will try your suggestions. Here is the stacktrace (env name redacted):</p>
<p>Traceback (most recent call last):<br>
File “/mnt/azureml/cr/j//exe/wd/train.py”, line 256, in <br>
for i, batch_data in enumerate(valid_loader):<br>
File “/azureml-envs//lib/python3.10/site-packages/torch/utils/data/dataloader.py”, line 530, in <strong>next</strong><br>
data = self._next_data()<br>
File “/azureml-envs//lib/python3.10/site-packages/torch/utils/data/dataloader.py”, line 1207, in _next_data<br>
idx, data = self._get_data()<br>
File “/azureml-envs//lib/python3.10/site-packages/torch/utils/data/dataloader.py”, line 1173, in _get_data<br>
success, data = self._try_get_data()<br>
File “/azureml-envs//lib/python3.10/site-packages/torch/utils/data/dataloader.py”, line 1011, in _try_get_data<br>
data = self._data_queue.get(timeout=timeout)<br>
File “/azureml-envs//lib/python3.10/multiprocessing/queues.py”, line 122, in get<br>
return _ForkingPickler.loads(res)<br>
File “/azureml-envs//lib/python3.10/site-packages/torch/multiprocessing/reductions.py”, line 300, in rebuild_storage_fd<br>
storage = cls._new_shared_fd(fd, size)</p>
<p>RuntimeError: unable to mmap 256 bytes from file : Cannot allocate memory (12)</p>

zyc: <p>There is a MWE in <a href="https://github.com/pytorch/pytorch/issues/92134" class="inline-onebox" rel="noopener nofollow ugc">[Bug][Dataloader] unable to mmap 2048 bytes from file &lt;filename not specified&gt;: Cannot allocate memory (12) · Issue #92134 · pytorch/pytorch · GitHub</a><br>
This bug happens to me too, we have 1.8 TiB of memory, more than enough for 100 copies of the entire dataset in memory.</p>

tiramisuNcustard: <p>Do you mean num_woker in DataLoader? We need to be careful that total samples to be processed (num_woker * memory_size_of_one_batch) will fit in the RAM.</p>

zyc: <p>No, I meant, the dataset we used is relatively small, around 10 GiB in total, and we have 1.8 TiB memory installed, more that 100 times of the size of dataset.<br>
(num_woker * memory_size_of_one_batch) takes less than 0.001% of usable memory.</p>

klex: <p>Hey all,</p>
<p>I encountered the same error as described in the question (<code>unable to mmap 8 bytes from file &lt;filename not specified&gt;: Cannot allocate memory (12)</code>), and I was able to write a MWE to find the solution in my case. I thought I leave it here in case other people find this thread. In the following code, the error will occur after a few minutes. As far as I can see, the issue is that item.numpy() does not make a copy, and therefore it can be easily solved by explicitly doing so. In this example it is pretty obvious, but in my case it was a bit more subtle, because <code>item</code> was handed around a bit in the code (but never copied).</p>
<pre><code class="lang-auto">from tqdm import tqdm
import torch


class Dset(torch.utils.data.Dataset):
    def __getitem__(self, idx):
        return torch.randint(255, size=(2, 20), dtype=int)

    def __len__(self):
        return 100_000


dloader = torch.utils.data.DataLoader(Dset(), batch_size=1, num_workers=8)
stats = []
for items in tqdm(dloader):
    for item in items:
        stats.append(item.numpy()) # will give you the error after ~20-90k batches
        #stats.append(item.numpy().copy()) # works without error
</code></pre>

HELLORPG: <p>I have the same issue. I try to use <code>pin_memory=True</code> in my dataloader, and now I do not receive this error message.</p>