Training works with 1080Ti but don't with 3090. OOM error

username: patricks
cooked: <p>Hi there,<br>
I’m training a model for a style transfer app.<br>
I’ve used this training code for 2 years now and it’s perfectly working with a GTX1080Ti 12 Go but unfortunatly it does not work on a RTX3090.<br>
Here is the error</p>
<blockquote>
<p>File “train-monet_giverny.py”, line 198, in <br>
train()<br>
File “train-monet_giverny.py”, line 112, in train<br>
content_features = VGG(content_batch.add(imagenet_neg_mean))<br>
File “C:\Users\smartTour\tableaux\lib\site-packages\torch\nn\modules\module.py”, line 1501, in _call_impl<br>
return forward_call(*args, **kwargs)<br>
File “C:\Users\smartTour\Documents\Train-Relook\vgg.py”, line 46, in forward<br>
x = layer(x)<br>
File “C:\Users\smartTour\tableaux\lib\site-packages\torch\nn\modules\module.py”, line 1501, in _call_impl<br>
return forward_call(*args, **kwargs)<br>
File “C:\Users\smartTour\tableaux\lib\site-packages\torch\nn\modules\conv.py”, line 463, in forward<br>
return self._conv_forward(input, self.weight, self.bias)<br>
File “C:\Users\smartTour\tableaux\lib\site-packages\torch\nn\modules\conv.py”, line 459, in _conv_forward<br>
return F.conv2d(input, weight, bias, self.stride,<br>
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 198.00 MiB (GPU 0; 24.00 GiB total capacity; 3.82 GiB already allocated; 17.67 GiB free; 24.00 GiB allowed; 4.00 GiB reserved in total by PyTorch) If reserved memory is &gt;&gt; allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF</p>
</blockquote>
<p>The batch size is 1 and the image size is 800x600. I manage to train on the GTX1080Ti with bigger image size.<br>
The curious thing is that it uses  8 Go to train on the GTX1080Ti but don’t want to allocate more than 4 Gib on the RTX3090.<br>
I’ve tried many version of python or cuda but it changes nothing. I’m using the last version for the driver.<br>
Is there a way to “force” an allocation of 10 Gib ?<br>
Any ideas ?</p>

username: AbdulsalamBande
cooked: <p>“CUDA out of memory,” suggests that your GPU is running out of memory during training. This could be due to the RTX 3090’s architecture differences or the way memory is being managed. You can try the following You can try this</p>
<ol>
<li>Set the environment variable <code>PYTORCH_CUDA_ALLOC_CONF</code> to control the memory allocation behavior in PyTorch. To allocate more memory, you can set the <code>max_split_size_mb</code> configuration option:</li>
</ol>
<pre><code class="lang-auto">import os
os.environ['PYTORCH_CUDA_ALLOC_CONF'] = 'max_split_size_mb=4096
</code></pre>
<p>This line should be placed at the beginning of your script, before importing PyTorch. It sets the maximum split size to 4096 MB (4 GB). You can increase this value as needed.</p>

username: patricks
cooked: <p>Thanks AbdusalamBande,<br>
I’ve tried 4096,1024,2048, 8192, … and I always get the same error</p>
<blockquote>
<p>Traceback (most recent call last):<br>
File “train-monet_giverny.py”, line 200, in <br>
train()<br>
File “train-monet_giverny.py”, line 117, in train<br>
generated_features = VGG(generated_batch.add(imagenet_neg_mean))<br>
File “C:\Users\smartTour\tableaux\lib\site-packages\torch\nn\modules\module.py”, line 1501, in _call_impl<br>
return forward_call(*args, **kwargs)<br>
File “C:\Users\smartTour\Documents\Train-Relook\vgg.py”, line 46, in forward<br>
x = layer(x)<br>
File “C:\Users\smartTour\tableaux\lib\site-packages\torch\nn\modules\module.py”, line 1501, in _call_impl<br>
return forward_call(*args, **kwargs)<br>
File “C:\Users\smartTour\tableaux\lib\site-packages\torch\nn\modules\conv.py”, line 463, in forward<br>
return self._conv_forward(input, self.weight, self.bias)<br>
File “C:\Users\smartTour\tableaux\lib\site-packages\torch\nn\modules\conv.py”, line 459, in _conv_forward<br>
return F.conv2d(input, weight, bias, self.stride,<br>
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 200.00 MiB (GPU 0; 24.00 GiB total capacity; 3.99 GiB already allocated; 17.27 GiB free; 4.40 GiB reserved in total by PyTorch) If reserved memory is &gt;&gt; allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF</p>
</blockquote>
<p>It don’t really get how to use all the vram I’ve.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="patricks" data-post="1" data-topic="176577">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/ac91a4/48.png" class="avatar"> patricks:</div>
<blockquote>
<p>Tried to allocate 198.00 MiB (GPU 0; 24.00 GiB total capacity; 3.82 GiB already allocated; 17.67 GiB free; 24.00 GiB allowed; 4.00 GiB reserved in total by PyTorch)</p>
</blockquote>
</aside>
<p>It seems you are limiting the device memory via <code>torch.cuda.set_per_process_memory_fraction</code> since the <code>allowed</code> stats is shown. It points to 24GB, but could you remove it nevertheless?</p>
<p>Also, does:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch

print(torch.cuda.memory_allocated()/1024**2)
# 0.0
x = torch.randn(1024**3, device="cuda")
print(torch.cuda.memory_allocated()/1024**2)
# 4096.0
x = torch.randn(2 * 1024**3, device="cuda")
print(torch.cuda.memory_allocated()/1024**2)
# 8192.0
</code></pre>
<p>work?</p>

username: patricks
cooked: <p>Thanks ptrblck,</p>
<p>I’m not using <code>torch.cuda.set_per_process_memory_fraction</code></p>
<p>Your code does not work. Always the same error : memory limited at 4Gib… Here is the error :</p>
<blockquote>
<p>0.0<br>
4096.0<br>
Traceback (most recent call last):<br>
File “test_memory.py”, line 10, in <br>
x = torch.randn(2 * 1024**3, device=“cuda”)<br>
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 8.00 GiB (GPU 0; 24.00 GiB total capacity; 4.00 GiB already allocated; 18.77 GiB free; 4.00 GiB reserved in total by PyTorch) If reserved memory is &gt;&gt; allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF</p>
</blockquote>

username: ptrblck
cooked: <p>Your code is using the <code>set_per_process_memory_fraction</code> since the <code>allowed</code> keyword wouldn’t be shown otherwise:</p>
<pre><code class="lang-python">x = torch.randn(1024**4, device="cuda")
# OutOfMemoryError: CUDA out of memory. Tried to allocate 4096.00 GiB (GPU 0; 23.69 GiB total capacity; 0 bytes already allocated; 22.01 GiB free; 0 bytes reserved in total by PyTorch) If reserved memory is &gt;&gt; allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

torch.cuda.set_per_process_memory_fraction(1.0)
x = torch.randn(1024**4, device="cuda")
# OutOfMemoryError: CUDA out of memory. Tried to allocate 4096.00 GiB (GPU 0; 23.69 GiB total capacity; 0 bytes already allocated; 22.00 GiB free; 23.69 GiB allowed; 0 bytes reserved in total by PyTorch) If reserved memory is &gt;&gt; allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
</code></pre>

username: patricks
cooked: <p>Thanks Ptrblck,</p>
<p>Of course, you’re right <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"><br>
I have tested to put a ‘set_per_process_memory_fraction’  two days ago but since it was not working I had it commented.<br>
I trash the contentof the pycache folder and the ‘allowed’ keyword disappear but not the error</p>
<p>I also try to call</p>
<blockquote>
<p>torch.cuda.set_per_process_memory_fraction(1.0, None)</p>
</blockquote>
<p>right after the import and now the training is starting <img src="https://discuss.pytorch.org/images/emoji/apple/+1.png?v=12" title=":+1:" class="emoji" alt=":+1:" loading="lazy" width="20" height="20"></p>
<p>two days ago I had called it in my training function after some torch settings. It seem s to be the reason it was not working…</p>
<p>I’ll let you know in 24h at the end of the training</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="patricks" data-post="7" data-topic="176577">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/ac91a4/48.png" class="avatar"> patricks:</div>
<blockquote>
<p>I have tested to put a ‘set_per_process_memory_fraction’ two days ago but since it was not working I had it commented.</p>
</blockquote>
</aside>
<p>Thanks for confirming!</p>
<aside class="quote no-group quote-modified" data-username="patricks" data-post="7" data-topic="176577">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/ac91a4/48.png" class="avatar"> patricks:</div>
<blockquote>
<p>I trash the contentof the pycache folder and the ‘allowed’ keyword disappear but not the error</p>
<p>I also try to call</p>
<blockquote>
<p>torch.cuda.set_per_process_memory_fraction(1.0, None)</p>
</blockquote>
<p>right after the import and now the training is starting <img src="https://discuss.pytorch.org/images/emoji/apple/+1.png?v=12" title=":+1:" class="emoji" alt=":+1:" loading="lazy" width="20" height="20"></p>
</blockquote>
</aside>
<p>This is really weird. Does it mean you have set it to a specific limit in the past, saw the issue, removed it, and were still unable to allocate more than 4GB?<br>
After removing the <code>pycache</code> (could you describe what exactly you have deleted) you are now able to allocate more than 4GB?</p>

username: patricks
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="8" data-topic="176577">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>This is really weird. Does it mean you have set it to a specific limit in the past, saw the issue, removed it, and were still unable to allocate more than 4GB?</p>
</blockquote>
</aside>
<p>Actually Ihave tried to play with this limit but I have placed it after some torch settings and it was not working. I think this settings was to late in my code. its was after</p>
<pre><code class="lang-auto"># Seeds
    torch.manual_seed(SEED)
    torch.cuda.manual_seed(SEED)
    np.random.seed(SEED)
    random.seed(SEED)

    # Device
    device = ("cuda" if torch.cuda.is_available() else "cpu")
</code></pre>
<p>Nevertheless my training code has the OOM error without it. It was unable to allocate more than 4Gib. I have tried the <code>torch.cuda.set_per_process_memory_fraction</code>, but not in the right place and I gave up with that and have commented it. But when you write of that this morning I try to put that just after the import as the first setting and it works.<br>
I never met this error with the GTX1080Ti. No need to set the <code>torch.cuda.set_per_process_memory_fraction</code> with this card.</p>
<aside class="quote no-group" data-username="ptrblck" data-post="8" data-topic="176577">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>After removing the <code>pycache</code> (could you describe what exactly you have deleted) you are now able to allocate more than 4GB?</p>
</blockquote>
</aside>
<p>In my code I import three python  files with classes and functions</p>
<pre><code class="lang-auto">import vgg
import experimental
import utils
</code></pre>
<p>I have deleted the three files related experimental.cpython-38.pyc, vgg.cpython-38.pyc and utils.cpython-38.pyc</p>
<p>Actually my code is crashing again for an other reason…I will investigate</p>

username: ptrblck
cooked: <p>OK, let me know if you are still seeing issues disallowing you to allocate more than 4GB of memory, as this setting should not be “sticky” between runs and I would consider it a bug.<br>
I was also experimenting with it in my setup, but wasn’t able to reproduce the issue.</p>

username: patricks
cooked: <p>As I said in a previous post I had an other error related to a memory cpuallocator error. When I search about this error I found that increasing the Pagefile size can solve this error.<br>
And I discover that the pagefile size was set to 0 on my computer. Don’t remenber why I have done that but setting it automatically solved  my memory cpuAllocator error.<br>
And  Today I also try to run my code without the</p>
<blockquote>
<p>torch.cuda.set_per_process_memory_fraction(1.0, None)</p>
</blockquote>
<p>And it works now. No 4GiB limitations. no OOM error<br>
It seems that I have to have a pagefile size more than 0 to avoid this error on the Vram. I don’t know if it’s already a warning for the users of pytorch but it could be…</p>