GPU usage increases along epochs whereas no new CUDA tensors are in the scope

username: aRI0U
cooked: <p>Hi,<br>
I am training a model and I saw that the GPU usage increases epoch after epoch, even by calling <code>torch.cuda.empty_cache()</code>.</p>
<p>Here is a snippet of code:</p>
<pre><code class="lang-auto">for sample_id, (packed_blocks, packed_sequences) in tqdm(enumerate(data_loader)):
    monitored_quantities = step(packed_blocks, packed_sequences, train=train)

    means = {
        key: value + means[key]
        for key, value in monitored_quantities.items()
    }

    torch.cuda.empty_cache()
</code></pre>
<p>Before the first loop, GPU memory usage is 489MB. I assume it corresponds to the model stored in GPU. But at the beginning of each new loop, the GPU memory is higher and higher. And yet:</p>
<ul>
<li>forward and backward pass are computed inside of <code>step</code> function</li>
<li>
<code>packed_blocks</code> and <code>packed_sequences</code> are on CPU when the dataloader drops them, they are loaded on GPU inside of <code>step</code> function</li>
<li>
<code>monitored_quantities</code> and <code>means</code> are <code>dict</code>s of <code>float</code>s</li>
</ul>
<p>Since no new CUDA tensor is created in the scope of the <code>for</code> loop, I assume that after calling <code>torch.cuda.empty_cache()</code> the GPU memory usage would be the same than at the beginning of the iteration. But in practice it keeps growing.</p>
<p>Does anyone have an idea about this question?</p>
<p>Best,<br>
Alain</p>

username: ptrblck
cooked: <p>Do the <code>dicts</code> contain CUDATensors or CPUTensors?<br>
In the former case, you should detach them, if you don’t want to use them for the gradient calculation anymore, as they might store the whole computation graph otherwise.<br>
Also, check your code for any storage of CUDATensors, which would be used for printing/logging purposes and are not detached.</p>

username: aRI0U
cooked: <p>Hi,<br>
Thanks a lot for your reply.</p>
<p>Concerning the dicts, <code>means</code> is initialized as a <code>defaultdict(float)</code> and <code>monitored_quantities</code> is defined as <code>{'loss': loss.item()}</code>. So both dicts are dicts of Python floats, and none of them contain any tensor, whether on CPU or GPU.</p>
<p>Moreover, all operations I realize inside of the <code>for</code> loop are written in the snippet and in particular nothing is printed in the loop. I did not include the code of the <code>step</code> function because I assume that local tensors created in this function are out-of-scope and thus should be removed from GPU memory when I call <code>torch.cuda.empty_cache()</code>. Am I right on this point?</p>
<p>EDIT: I tried replacing <code>loss.item()</code> by <code>loss.detach().cpu().item()</code> in the <code>monitored_quantities</code> computation but it changed nothing.</p>

username: aRI0U
cooked: <p>I realized new experiments and it seems that my issue is somehow related to APEX!</p>
<p>When I do not use <code>apex.amp</code>, the GPU memory increases at the end of the first loop but then remains constant (489 MB before the first iteration and then 807MB at the beginning of all other iterations)</p>
<p>On the other hand, when I use <code>apex.amp</code>, the GPU memory seems to increase during the first iterations to some point (between 900MB and 1GB) and then varies in this interval.</p>
<p>Then my two questions are:</p>
<ul>
<li>why does the GPU memory usage vary from one iteration to the other?</li>
<li>why the GPU memory usage is higher using apex than using standard PyTorch?</li>
</ul>
<p>I don’t know to what extent it is still relevant to ask such question in this forum since it seems to be not directly related to PyTorch. Please tell me if you prefer me to close that topic and open an issue directly on apex repo.</p>

username: ptrblck
cooked: <p>I’m a bit confused regarding the initial issue.<br>
Are you seeing an icreased memory usage in each epoch or just a varying one?</p>
<p>The latter case might be expected, especially if you are manually clearing the cache (which will not save any memory, but slow down your code).</p>
<p>Is the allocated memory higher using <code>apex</code> or just the reserved?<br>
You can check it via <code>torch.cuda.memory_allocated()</code> and <code>torch.cuda.memory_reserved()</code>.<br>
The reserved memory might be higher, if some intermediate tensors are additionally created and it also depends on the <code>opt_level</code> you are using.</p>
<p>That being said, we recommend to check out the <a href="https://pytorch.org/docs/master/amp.html">native mixed-precision</a> implementation, which can be used by installing the nightly binaries or by building from source.</p>

username: aRI0U
cooked: <p>During the ~10 first iterations, the GPU memory increases, and after this it is varying but it seems to remain in the same interval. When I wrote this topic first, I measured the GPU usage only during the first epochs, hence the confusion.<br>
Here is a plot of the evolution of memory allocated (blue) and reserved (orange) as calculated by respectively <code>torch.cuda.memory_allocated()</code> and <code>torch.cuda.memory_reserved()</code> when I don’t call <code>torch.cuda.empty_cache()</code> at the end of each iter.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/4/7/479c11e2f5a358e9c97a2de4816ec90d966749a4.png" alt="image" data-base62-sha1="aduhgLqCUNxgokDj2j7RkQhMVlq" width="640" height="480"></p>
<p>Here is a similar plot when I call <code>torch.cuda.empty_cache()</code> at the end of each iter:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/3/f/3f2ab0990054edfa0740fcc0ef0ca8b1718fd70b.png" alt="image" data-base62-sha1="90NxG0HyLv62ymi1YDOTs1GuuTN" width="640" height="480"></p>
<p>Finally, here is the GPU memory usage when I don’t use apex and I call <code>empty_cache()</code> at the end of each epoch:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/c/c/cc13c64f2e83b0e59c08cde33587233c76a83bd6.png" alt="image" data-base62-sha1="t7lLNy56zXbYM3f2EMmvte9vCzI" width="640" height="480"></p>
<p>Both allocated and reserved memories are higher using apex. I used <code>opt_level O1</code>.</p>
<p>I am aware that calling <code>torch.cuda.empty_cache()</code> does not reduce the memory usage but I assumed that it would enable me to get more accurate measurements (please tell me if it’s not the case). Before this answer, I only measured the GPU memory usage by looking at <code>nvidia-smi</code>.<br>
Does those elements help you seeing what could be my problem?<br>
Thanks a lot,<br>
Alain</p>

username: ptrblck
cooked: <p>Thanks for the detailed analysis!<br>
From the plots it seems that the allocated memory stays approx. at the same level.<br>
Could you try the <a href="https://pytorch.org/docs/stable/amp.html">native amp implementation</a> in PyTorch <code>1.6.0</code> and check the memory usage?</p>
<p>Also, which GPU are you using?</p>

username: aRI0U
cooked: <p>With the native AMP implementation, GPU memory usage seems to be stable.<br>
Here is the plot without calling <code>empty_cache()</code>:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/6/5/651b9df45be9f649f1e796a42d56c5e4f7ec6e99.png" alt="image" data-base62-sha1="eqropkqpSuXFnkRCrc4t96sFqE9" width="640" height="480"><br>
Same when calling <code>empty_cache()</code>:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/2/7/2776a691d4aaa00d6891a7b3f199a95decee1266.png" alt="image" data-base62-sha1="5D6PpwkF9USySpi2A6neJFxk7nE" width="640" height="480"></p>
<p>My GPU is a Tesla K80.</p>