Why did pytorch run through the code before, when I run it again this time, I reminded: CUDA out of memory. Tried to allocate 50.00 MiB (GPU 0; 6.00 GiB total capacity; 353.72 MiB already allocated; 3.86 GiB free; 396.00 MiB reserved in total by PyTorch)

username: hg6a6
cooked: <p>Why did pytorch run through the code before, when I run it again this time, I reminded: CUDA out of memory. Tried to allocate 50.00 MiB (GPU 0; 6.00 GiB total capacity; 353.72 MiB already allocated; 3.86 GiB free; 396.00 MiB reserved in total by PyTorch)</p>

username: ptrblck
cooked: <p>Could you check, if the previous script crashed and if zombie processes might be using the device memory? If so, kill them, and rerun the script.</p>

username: learner12345
cooked: <p>Hi hope you are well.<br>
I’m also getting a similar kind of error.<br>
First I finetuned Yolov5 on custom data.<br>
After obtaining best I tried to refine model on those best weights but with different image size.<br>
but now i can’t run model even with small batch size.<br>
<strong>And I can’t run any other mode like detection transformer on machine</strong><br>
I tried to emtpy cache as suggested but all in vain<br>
I tried to run code on different machine with same GPU RTX 3080 and and Vram 10GB<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/e/deeae45bb034bae66c1834e163b3950635fde0ba.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/deeae45bb034bae66c1834e163b3950635fde0ba" title="WhatsApp Image 2022-04-21 at 2.02.34 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/e/deeae45bb034bae66c1834e163b3950635fde0ba_2_690x388.jpeg" alt="WhatsApp Image 2022-04-21 at 2.02.34 PM" data-base62-sha1="vO1fDLrB6uZQAgok6lTgJuoffku" width="690" height="388" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/e/deeae45bb034bae66c1834e163b3950635fde0ba_2_690x388.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/d/e/deeae45bb034bae66c1834e163b3950635fde0ba_2_1035x582.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/d/e/deeae45bb034bae66c1834e163b3950635fde0ba.jpeg 2x" data-dominant-color="8A8C8C"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">WhatsApp Image 2022-04-21 at 2.02.34 PM</span><span class="informations">1280×720 141 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Any idea how can i fix this error.</p>

username: ptrblck
cooked: <p>I guess other (dead) processes might use the device memory, so check them via <code>nvidia-smi</code> and/or <code>ps aux</code> and kill them if needed.</p>

username: learner12345
cooked: <p>i can’t see any heavy projects ruuning<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/f/8/f8437efc9e1070d813b37ed37ab668b8772cc1ca.png" alt="Screenshot from 2022-04-26 13-17-23" data-base62-sha1="zqf1nCGAMDNMMY54YaWQ7hqQpZ0" width="657" height="371"></p>
<p><strong>Why 7.18 GiB reserved in total by PyTorch??</strong></p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="learner12345" data-post="5" data-topic="118287">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/l/5daacb/48.png" class="avatar"> learner12345:</div>
<blockquote>
<p>Why 7.18 GiB reserved in total by PyTorch??</p>
</blockquote>
</aside>
<p>The memory usage depends on the actual model architecture as well as the input shapes besides some other requirements for e.g. workspaces for specific kernels etc.<br>
E.g. you would have to store all model parameters, buffers, input tensors, forward activations, gradients, optimizer states (if used). What are your expectations and how did you estimate the memory requirement?</p>