GPU full and code not running

Andre_Amaral_IST: <p>Hey!</p>
<p>I am trying to free the GPU memory, since i am not running any code and it is full.<br>
I also closed the Visual Studio Code, and re-open it and nothing happens.<br>
I also used torch.cuda.empty_cache() but the memory is not released.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/8/7/877ec50db0a0bab2374114b107736043506c914a.png" alt="image" data-base62-sha1="jkE3O7B183ta5BNl2Pld2meRLcK" width="676" height="474"></p>
<p>Anyone can help me?<br>
Regards!<br>
André</p>

Andrei_Cristea: <p>Could you attach the output of <code>nvidia-smi</code>? Someone here will likely be able to help you, but will need that as a starting point.</p>

Andrei_Cristea: <p>I’m really not versed in reading nvidia-smi output but it looks to me like you have a bunch of python processes running that are using up GPU memory (bottom section of your image). Could you kill those and check if that helps release your memory? e.g. run <code>$ kill 111715</code> in your shell, where 111715 is the PID of the first process that shows up in <code>nvidia-smi</code>.</p>

Andre_Amaral_IST: <aside class="quote no-group" data-username="Andrei_Cristea" data-post="4" data-topic="150797">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/andrei_cristea/48/47843_2.png" class="avatar"> Andrei_Cristea:</div>
<blockquote>
<p>$ kill 111715</p>
</blockquote>
</aside>
<p>Thank you! It realy helped, now my cuda memory is empty as i wanted! Thank you very much for your help.</p>
<p>Regards<br>
André Amaral</p>

Andre_Amaral_IST: <p>Hey,<br>
Sorry to re-open this topic but now i have no process running and my cuda memory is full.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/4/4/44296d73ac1acfda5fd95fe40cc1e98085995391.png" alt="image" data-base62-sha1="9IZe17JOic6pYF07sTBUvCLH4Zz" width="593" height="321"></p>
<p>And i did not change the code I was running and now it complains about the CPU too.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/4/14f5146df8f2acd768bc49d6667102ce443c665f.png" data-download-href="https://discuss.pytorch.org/uploads/default/14f5146df8f2acd768bc49d6667102ce443c665f" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/4/14f5146df8f2acd768bc49d6667102ce443c665f_2_690x55.png" alt="image" data-base62-sha1="2ZoD5vxZYekME41VA8LiHszfgFV" width="690" height="55" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/4/14f5146df8f2acd768bc49d6667102ce443c665f_2_690x55.png, https://discuss.pytorch.org/uploads/default/optimized/3X/1/4/14f5146df8f2acd768bc49d6667102ce443c665f_2_1035x82.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/1/4/14f5146df8f2acd768bc49d6667102ce443c665f.png 2x" data-dominant-color="F5F8F9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1357×109 47.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

Andrei_Cristea: <p>Hi Andre -</p>
<p>From this section of <code>nvidia-smi</code>, the GPUs seem to be far from full:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/0/406cbd8db7205b18551e3d9dfebfc709e6511b1d.png" alt="image" data-base62-sha1="9bVx4wrOv13kXmU0pQ2eY8VqLM1" width="126" height="76"></p>
<p>From your error message…<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/2/b/2bc6c1aea3cb89655a7057f1b3bc54b329c2d2da.png" alt="image" data-base62-sha1="6fgmDTSk17axnZYctFZzatv6kH0" width="107" height="27"><br>
…it seems like you’re trying to load a dataset of about 11.7 GB into memory, and each of your GPUs can only 6 GB available (and maybe your CPU also doesn’t have enough memory for this). Perhaps you could somehow split up the dataset and send half of it to one GPU and another half to another GPU (I’ve never done that before so I’m not sure how that works).</p>
<p>However, just wanted to say that it is atypical to load the entire dataset into memory. What’s more typical is to use dynamic / lazy loading, using <code>Dataset</code>s and <code>DataLoader</code>s <a href="https://pytorch.org/tutorials/beginner/basics/data_tutorial.html" rel="noopener nofollow ugc">like here</a>.</p>

Andre_Amaral_IST: <p>Thanks, I will look to it, but it is strange since like for 10 days I was able to upload that dataset on the CPU and the code did not break in that section.</p>
<p>Thanks again for your help<br>
Regards<br>
André</p>

Andre_Amaral_IST: <p>The code that provides the error is the following:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/a/1a0958d723812e9259c8ce5d002663c4c2dee0c2.png" data-download-href="https://discuss.pytorch.org/uploads/default/1a0958d723812e9259c8ce5d002663c4c2dee0c2" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/a/1a0958d723812e9259c8ce5d002663c4c2dee0c2_2_690x142.png" alt="image" data-base62-sha1="3IkrhXJFxajJJhBrFdMJNobKhLs" width="690" height="142" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/a/1a0958d723812e9259c8ce5d002663c4c2dee0c2_2_690x142.png, https://discuss.pytorch.org/uploads/default/original/3X/1/a/1a0958d723812e9259c8ce5d002663c4c2dee0c2.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/1/a/1a0958d723812e9259c8ce5d002663c4c2dee0c2.png 2x" data-dominant-color="ECEEEE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">835×173 38.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Where I am used the DataLoader but for what I debug the code crashes in the np.array(X).</p>

Andrei_Cristea: <p>Your CPU will have more or less memory available depending on what else is running on your system, independent of the GPU and what nvidia-smi might tell you. So it’s possible that it would work at some moments but not others, since the amount of memory consumed by other processes is not going to be consistent over time.</p>

Andre_Amaral_IST: <p>That makes sense!<br>
Thanks very much for your time!<br>
Regards<br>
André</p>

ptrblck: <p>Also double post from <a href="https://discuss.pytorch.org/t/cpu-full-without-any-reason/150882/2">here</a> with the same explanation as given by <a class="mention" href="/u/andrei_cristea">@Andrei_Cristea</a>.</p>