Best Practices for Maximum GPU utilization

Adi_R: <p>Hi,<br>
I have read the related posts about best practices for usage on GPU, and this question is different.</p>
<p>I am tuning my batch size such that the largest batch uses around 90% of GPU memory. Thus, some batches will use close to 90% while other (most use around  50%)</p>
<p>I am noticing that if 2 large batches (both using close to 90%)  are processed (i.e. one forward prop and batch prop) consecutively it crashes with out of memory error. I suspect because the GPU is unable to clear the memory of the previous batch. However, if large batches and interspersed with smaller batches it seems to work fine.</p>
<p>Is there a rule of thumb to GPU utilization for such purposes, i.e. an optimal memory usage that works with the garbage collection?</p>

SimonW: <p>You are probably holding reference to some Variable from the previous iteration. This can cause the graph not being freed properly in some cases. See <a href="http://pytorch.org/docs/master/notes/faq.html#my-model-reports-cuda-runtime-error-2-out-of-memory">http://pytorch.org/docs/master/notes/faq.html#my-model-reports-cuda-runtime-error-2-out-of-memory</a></p>

Adi_R: <p>Thanks for the pointer. I checked and as far as I can tell I’m not keeping a variable alive for more than it should be.  Explicitly deleting variables does seem to help, but shouldn’t garbage collection immediately free dereferenced memory anyway?</p>
<p>A follow-up is whether <code>nvidia-smi</code> accurately reports memory usage? I managed to run my training session on a Tesla K20 (4G memory). When I run the same training (same data and batching) on a 10G gpu <code>nvidia-smi</code> says I’m using 9G! Clearly, this can’t be because I just successfully ran it on a 4G GPU.</p>

SimonW: <aside class="quote no-group" data-username="Adi_R" data-post="3" data-topic="13863">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/adi_r/48/3072_2.png" class="avatar"> Adi_R:</div>
<blockquote>
<p>I checked and as far as I can tell I’m not keeping a variable alive for more than it should be.  Explicitly deleting variables does seem to help</p>
</blockquote>
</aside>
<p>This is a sign of holding on to variable for too long. Could you post your code?</p>
<aside class="quote no-group" data-username="Adi_R" data-post="3" data-topic="13863">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/adi_r/48/3072_2.png" class="avatar"> Adi_R:</div>
<blockquote>
<p>A follow-up is whether nvidia-smi accurately reports memory usage?</p>
</blockquote>
</aside>
<p>No. PyTorch uses a cached GPU memory allocator. Read more here: <a href="http://pytorch.org/docs/master/notes/cuda.html#memory-management" class="inline-onebox">CUDA semantics — PyTorch master documentation</a>. Notice that some monitoring methods are not available in a public release yet (they exist on github master).</p>

Adi_R: <p>Thanks again! I’m making changes on a fork of OpenNMT-py, so posting code might be difficult. I will try to make an example code to reproduce this. Also, I realize as I’m writing this that I’m working with an older version of OpenNMT-py so I will try updating as well.</p>
<p>Thanks for the pointer once again regarding GPU mem allocation.</p>

elistevens: <p>Sounds to me like you might be accidentally using the following anti-pattern:</p>
<pre><code class="lang-auto">for input, answer in batches:
    optimizer.zero_grad()
    output = model(input)
    loss_var = loss_func(output, answer)
    loss_var.backward()
    optimizer.step()
    del loss_var # missing this!
</code></pre>
<p>Without the <code>del</code> the <code>loss_var</code> is still holding a reference to gradients all the way through the second iteration of the loop, including during the calculation of <code>loss_func</code>. Python can’t garbage collect the old value until after the new has been computed and assigned to <code>loss_var</code>!</p>
<p>If it’s not clear what I mean, try this:</p>
<pre><code class="lang-auto">loss_var = None
for input, answer in batches:
    optimizer.zero_grad()
    output = model(input)
    print loss_var
    loss_var = loss_func(output, answer)
    loss_var.backward()
    optimizer.step()
</code></pre>
<p>Good luck!</p>

paul_mwaniki: <p>can this be caused by storing a snapshot of weights eg best_weights=copy.deepcopy(model.state_dict()) ?</p>