Memory orvershoot issue

111392: <p>hello</p>
<p>Generally when I try to train model, GPU memory overshoot at the beginning and it will converge after that.</p>
<p>from my test, it overshoot at loss.backward()</p>
<p>Is there way to avoid this overshoot? some model get “CUDA out of memory” because of this</p>
<p>thank you in advance!</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/f/9f95b986d07031722a746fa967cfdc22fd67e3ee.png" data-download-href="https://discuss.pytorch.org/uploads/default/9f95b986d07031722a746fa967cfdc22fd67e3ee" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/f/9f95b986d07031722a746fa967cfdc22fd67e3ee_2_690x383.png" alt="image" data-base62-sha1="mLKHuDVUn3daLrKLlhkMuC08C3s" width="690" height="383" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/f/9f95b986d07031722a746fa967cfdc22fd67e3ee_2_690x383.png, https://discuss.pytorch.org/uploads/default/original/3X/9/f/9f95b986d07031722a746fa967cfdc22fd67e3ee.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/9/f/9f95b986d07031722a746fa967cfdc22fd67e3ee.png 2x" data-dominant-color="FCFCFC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">868×482 15.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>Are you seeing this behavior using a specific model or always?<br>
In any case, could you post a minimal code snippet to reproduce this behavior as well as your current setup (PyTorch, CUDA, cudnn versions, which GPU you are using etc.)?</p>

111392: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="111694">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>avior using a specific model or always?<br>
In any case, could you post a minimal code snippet to reproduce this behavior as well as your current setup (PyTorch, CUDA, cudnn versions, which GPU y</p>
</blockquote>
</aside>
<p>thank you for reply!<br>
I thought this is generally happens</p>
<p>this is code I use for train</p>
<pre><code class="lang-auto">    disp_L = model(imgL, imgR)
    loss = F.smooth_l1_loss(dispL, dispL_GT, size_average=True)
    optimizer.zero_grad()
    loss.backward() # here is peak of overshoot in iteration 1
    optimizer.step()
</code></pre>
<p>current environment<br>
model : any model(ex. PSMNet, Segment, yoloV4, …)<br>
GPU : RTX 2070<br>
CUDA : CUDA 11<br>
pytorch : 1.7</p>
<p>I check GPU memory usage by</p>
<pre><code class="lang-auto">watch -n 0.01 nvidia-smi
</code></pre>

ptrblck: <p>Could you post an executable code snippet, which raises the OOM issue on this device, please?<br>
The memory peak might increase e.g. if cudnn uses its benchmarking to profile different kernels.<br>
However, this should not raise an OOM error, but skip the algorithm and clear the cache afterwards.</p>