+ 50% VRAM used on torch 1.3 compared to 1.2

hadaev8: <p>1.3<br>
<a href="https://colab.research.google.com/drive/104LtQ1zIioIOMQEPgVve77m5Rd4Gm0wU" class="onebox" target="_blank" rel="nofollow noopener">https://colab.research.google.com/drive/104LtQ1zIioIOMQEPgVve77m5Rd4Gm0wU</a><br>
1.2<br>
<a href="https://colab.research.google.com/drive/1y4LF1a90PYfKvFgQw6fCMYKtIFK07mVr" class="onebox" target="_blank" rel="nofollow noopener">https://colab.research.google.com/drive/1y4LF1a90PYfKvFgQw6fCMYKtIFK07mVr</a><br>
Everything same except torch version.<br>
At the end nvidia-smi show 11gb and 7 gb memory used.</p>

albanD: <p>Why are there all these tensorflow warnings? Do you use it as well?<br>
There are many other files imported. Are you sure that you don’t have a difference in model?</p>

hadaev8: <p>Tensorflow warnings from hparams contrib object.<br>
Codebase exact same, only difference is pytorch version.</p>

albanD: <p>Could you try reducing the code to a small 30/40 lines of code? It is very hard to give an answer here as we don’t even see all the files <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"></p>

hadaev8: <p>Okay.<br>
This is copy past of tutorial.<br>
<a href="https://colab.research.google.com/drive/10YhV2QqG-pZLXbsTkJRE-hVcXtmxzBXS" class="onebox" target="_blank" rel="nofollow noopener">https://colab.research.google.com/drive/10YhV2QqG-pZLXbsTkJRE-hVcXtmxzBXS</a><br>
<a href="https://colab.research.google.com/drive/1IDTOlw3U3ZzCvRAirpISc7UBG8M88oZU" class="onebox" target="_blank" rel="nofollow noopener">https://colab.research.google.com/drive/1IDTOlw3U3ZzCvRAirpISc7UBG8M88oZU</a><br>
700mb vs 330mb</p>

albanD: <p>Rerunning both notebooks above from scratch give ~700MB usage. If you rerun your notebook above from scratch you still see this?</p>

hadaev8: <p>Looks like it is related to type of gpu, on k80 both versions take ~300mb, on p4 535 vs 595, on p100 715mb vs 777mb. So this example is invalid.<br>
Still, top example was made on same gpu.<br>
Maybe it is about rnn layers or zero padding, i will try to check later.</p>

albanD: <p>Hi,</p>
<p>So if you use different GPUs, I could think of few reasons:</p>
<ul>
<li>The cuda driver behaves differently</li>
<li>Different architecture lead to different size of code loaded on the device by the driver</li>
<li>The algorithms chosen to do the compute by cudnn are different and use different amount of memory.</li>
</ul>
<p>You can check how much memory is used when only doing <code>torch.rand(1).cuda()</code>. That will tell you how much memory is used by only the driver + code.<br>
You can use <code>torch.cuda.memory_cached</code> <a href="https://pytorch.org/docs/stable/cuda.html#torch.cuda.memory_cached">doc</a> to know how much memory pytorch hold onto for Tensor allocation.</p>

hadaev8: <p>Just checked in 1.4, problem still here.<br>
After same code run, old version have 8gb allocated, and new 11gb on same gpu type.</p>

albanD: <p>When you mean allocated. It is the one given by <code>torch.cuda.memory_allocated()</code> ?</p>

hadaev8: <p>Yes, you cant check notebooks from top post.<br>
I added this command to it.</p>

albanD: <p>It is reserved, not allocated. The reserved memory can increase due to many reasons (more code in the binary, caching allocator behavior, different GPU, different cudnn, etc). The allocated one corresponds to only Tensors (and this one should be constant).</p>

hadaev8: <p>Okay, allocated is constant, but what should i do with it?<br>
I cant go to 1.3+ version, because i will hit out of memory error.<br>
It is not about GPU type, got same on v100.</p>

ptrblck: <p>Are you running into an out of memory error or are you concerned about the reserved memory?</p>

hadaev8: <p>Im not concern, still if I launch same code on 1.3, i will get OOM quickly, with smaller batch size it is fine.</p>

ptrblck: <p>I might have misunderstood the previous post, but I thought the allocated memory is the same?</p>

hadaev8: <p>From notebook allocated is same.<br>
I should not get OOM if allocated is same?</p>

albanD: <p>The fact that the allocated memory is the same means that we use exactely the same amount of memory for Tensors. So that is good.<br>
What is not tracked in this is:</p>
<ul>
<li>cuda driver internal memory</li>
<li>cudnn buffers</li>
</ul>
<p>To check the first, you can see how much cuda memory is used when you only do:</p>
<pre><code class="lang-python">import torch
a = torch.rand(1).cuda()
</code></pre>
<p>To check the second, it is a bit harder. You can try to set <code>torch.backends.deterministic = True</code>. This forces to use the default deterministic algorithm that is usually not too memory hungry.</p>

hadaev8: <p>With only one tensor both versions have 2gb memory reserved.<br>
With deterministic still 7gb vs 11gb.</p>

albanD: <p>So I guess for this model, the peak memory usage increased between 1.2 and later versions.<br>
I am not familiar with these models. Do you have a short description of its structure and the modules used? Or could you write a small nn.Module that looks similar (and reproduces the memory increase)?</p>