Got error 'Cannot allocate memory'

username: XiaXuehai
cooked: <p>Code like this:</p>
<pre><code class="lang-auto">trainset = COCODataset('data.txt', (416,416), True, is_debug=True)
	trainloader = DataLoader(trainset, batch_size=8,shuffle=False, num_workers=4)

	net = Mynet(file)
	if gpu:
		net.cuda()
	optimizer = optim.SGD(net.parameters(), lr=0.01)
	for epoch in range(Epochs):
		for i, sample in enumerate(trainloader):
			images, labels = sample['image'], sample['label']
</code></pre>
<p>I got an error here <code>for i, sample in enumerate(trainloader):</code> :<br>
<code>OSError: [Errno 12] Cannot allocate memory</code><br>
What’s that mean?My GPU memory-usage is <code>1283MiB / 8118MiB</code></p>

username: ptrblck
cooked: <p>The error message seems to point to your RAM, not the GPU memory.<br>
Could you check it with <code>free -h</code> and see how much memory is left?</p>
<p>This particular error message can be triggered, if new processes would like to spawn / fork and there is not enough memory left.</p>

username: Farid_Alijani
cooked: <p>how can I trigger such particular error message?<br>
Is there any code implementation in pytorch for that?<br>
I obtained this memmory allocation error in loading images from KITTI dataset.</p>

username: ptrblck
cooked: <p>If you are trying to load all images into your RAM, you might trigger this error.<br>
Did you observe the RAM usage?</p>

username: Farid_Alijani
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="4" data-topic="20736">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>id you observe the RAM usage?</p>
</blockquote>
</aside>
<p><code>$ free -h</code> shows the available memory decreasing to almost ZERO and this is the error eventually I get:</p>
<pre><code class="lang-auto">RuntimeError: [enforce fail at CPUAllocator.cpp:64] . DefaultCPUAllocator: can't allocate memory: you tried to allocate 12976128000 bytes. Error code 12 (Cannot allocate memory
</code></pre>
<p>what’s a better approach to load images or read images without allocating RAM ?</p>

username: ptrblck
cooked: <p>Usually you would lazily load the samples using a <code>Dataset</code> and wrap it in a <code>DataLoader</code> to create batches, shuffle the data, use multiple workers to load each batch etc.</p>
<p>Have a look at the <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html" rel="nofollow noopener">Data loading tutorial</a> for more information on how to write a custom <code>Dataset</code>.</p>

username: abtion
cooked: <p>try to set overcommit_memory to 0 in /etc/sysctl.conf</p>

username: karl7
cooked: <p>i also meet the memory leak problem. but i dont know how to correct it…</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "main_multi_model_test.py", line 169, in &lt;module&gt;
    main()
  File "main_multi_model_test.py", line 143, in main
    val_loss, val_acc, val_bacc = validate(model, val_loader, criterion, taskNum, weights=attrWeights)
  File "../validate.py", line 30, in validate
    for i, (inputs, labels) in enumerate(val_loader):
  File "/home/user1/miniconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 193, in __iter__
    return _DataLoaderIter(self)
  File "/home/user1/miniconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 469, in __init__
    w.start()
  File "/home/user1/miniconda3/lib/python3.7/multiprocessing/process.py", line 112, in start
    self._popen = self._Popen(self)
  File "/home/user1/miniconda3/lib/python3.7/multiprocessing/context.py", line 223, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/home/user1/miniconda3/lib/python3.7/multiprocessing/context.py", line 277, in _Popen
    return Popen(process_obj)
  File "/home/user1/miniconda3/lib/python3.7/multiprocessing/popen_fork.py", line 20, in __init__
    self._launch(process_obj)
  File "/home/user1/miniconda3/lib/python3.7/multiprocessing/popen_fork.py", line 70, in _launch
    self.pid = os.fork()
OSError: [Errno 12] Cannot allocate memory
</code></pre>