akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Cuda out of memory. Maybe memory leak

yang_sun: <p>I faced a problem when i try to train my dataset on FCOS3D/MMdetection3D.</p>
<h2><a name="the-error-info-1" class="anchor" href="#the-error-info-1"></a>The error info:</h2>
<p>RuntimeError: CUDA out of memory. Tried to allocate 2.83 GiB (GPU 6; 31.75 GiB total capacity; 28.25 GiB already allocated; 2.09 GiB free; 28.60 GiB reserved in total by PyTorch)<br>
I have a question, why the used memory of CUDA:3 is twice as much as others? My guess is that there was a problem during one iteration that caused the memory to not be freed.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/a/1a3ce48bb83aa49cb0f7ffeeadd1c8c107bc979f.png" data-download-href="https://discuss.pytorch.org/uploads/default/1a3ce48bb83aa49cb0f7ffeeadd1c8c107bc979f" title="97E814D4-BFDF-4CCA-89E7-3C559888B25C"><img src="https://discuss.pytorch.org/uploads/default/original/3X/1/a/1a3ce48bb83aa49cb0f7ffeeadd1c8c107bc979f.png" alt="97E814D4-BFDF-4CCA-89E7-3C559888B25C" data-base62-sha1="3K6SkJxQkLU5XMbZVfHFVdfE2jJ" width="546" height="500" data-dominant-color="2A2A2A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">97E814D4-BFDF-4CCA-89E7-3C559888B25C</span><span class="informations">645×590 17.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>Note that not only is the memory usage of device 3 increased, but also device 6 seems to be empty, so you might want to double check, if your code accidentally reuses GPU3 two times and skips GPU6.</p>

yang_sun: <p>RuntimeError: CUDA out of memory. Tried to allocate 2.83 GiB (GPU 6; 31.75 GiB total capacity; 28.25 GiB already allocated; 2.09 GiB free; 28.60 GiB reserved in total by PyTorch)<br>
Some errors happend, so GPU 6 is empty.</p>