CUDA error: out of memory when load models

cbox: <p>In the last two days, I have often encountered CUDA error when loading the pytorch model: out of memory.<br>
I didn’t change any code, but the error just come from nowhere.<br>
It can work just one hour ago.<br>
I use nvidia-smi to check the GPU also seems fine.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/9/9f30dde89f379fc940905884ee21022df0ea2ad0.png" data-download-href="https://discuss.pytorch.org/uploads/default/9f30dde89f379fc940905884ee21022df0ea2ad0" title="00"><img src="https://discuss.pytorch.org/uploads/default/original/2X/9/9f30dde89f379fc940905884ee21022df0ea2ad0.png" alt="00" data-base62-sha1="mIgC69o0ufb91LU8wtY5YrHHJni" width="689" height="67" data-dominant-color="18343E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">00</span><span class="informations">1268×124 9.26 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "test.py", line 39, in &lt;module&gt;
    top_model.load_state_dict(torch.load('pkls/True_fuck_feature_top_model_basic_loss_loss_10.pkl'))
  File "/home1/cbx/anaconda3/envs/torch_1.0/lib/python3.7/site-packages/torch/serialization.py", line 367, in load
    return _load(f, map_location, pickle_module)
  File "/home1/cbx/anaconda3/envs/torch_1.0/lib/python3.7/site-packages/torch/serialization.py", line 538, in _load
    result = unpickler.load()
  File "/home1/cbx/anaconda3/envs/torch_1.0/lib/python3.7/site-packages/torch/serialization.py", line 504, in persistent_load
    data_type(size), location)
  File "/home1/cbx/anaconda3/envs/torch_1.0/lib/python3.7/site-packages/torch/serialization.py", line 113, in default_restore_location
    result = fn(storage, location)
  File "/home1/cbx/anaconda3/envs/torch_1.0/lib/python3.7/site-packages/torch/serialization.py", line 95, in _cuda_deserialize
    return obj.cuda(device)
  File "/home1/cbx/anaconda3/envs/torch_1.0/lib/python3.7/site-packages/torch/_utils.py", line 76, in _cuda
    return new_type(self.size()).copy_(self, non_blocking)
  File "/home1/cbx/anaconda3/envs/torch_1.0/lib/python3.7/site-packages/torch/cuda/__init__.py", line 496, in _lazy_new
    return super(_CudaBase, cls).__new__(cls, *args, **kwargs)
RuntimeError: CUDA error: out of memory
</code></pre>

ptrblck: <p>Did you update any libraries between the last successful run?<br>
Did you specify any devices using <code>CUDA_VISIBLE_DEVICES</code>?<br>
Is the GPU completely empty before you run the script?</p>

nasTnate: <p>Hey <a class="mention" href="/u/ptrblck">@ptrblck</a> and <a class="mention" href="/u/cbox">@cbox</a>, I am currently running into this issue and am wondering how it was resolved.</p>
<blockquote>
<p>Did you update any libraries between the last successful run?</p>
</blockquote>
<p>I am using the same conda environment, so no change in libraries.</p>
<blockquote>
<p>Did you specify any devices using  <code>CUDA_VISIBLE_DEVICES</code> ?</p>
</blockquote>
<p>I am just specifying the device via: <code>device = torch.device('cuda:4')</code><br>
I am still pretty green here so I am not really sure what the difference is, however, this is the first time I have run into a situation of running out of memory on one of these particular models. (in fact, a different GPU in the cluster is currently training the exact same model).</p>
<blockquote>
<p>Is the GPU completely empty before you run the script?</p>
</blockquote>
<p>The GPU I am targeting is idle (10MiB/16280MiB)</p>
<p>I would appreciate any advice. Thanks!</p>
<p>EDIT: I solved the problem via this previous solution <a href="https://discuss.pytorch.org/t/out-of-memory-error-when-resume-training-even-though-my-gpu-is-empty/30757/2" class="inline-onebox">Out of memory error when resume training even though my GPU is empty - #2 by ptrblck</a></p>
<p>For anyone with a similar problem, the code I used for a fix is as follows:</p>
<pre><code class="lang-auto">model_path = 'path/to/model.pt'
model = UNet(n_channels = 1, n_classes = 1)
state_dict = torch.load(model_path,map_location='cpu')
model.load_state_dict(state_dict)
model.to(device)
</code></pre>
<p>That said, I am still curious why the work-around is necessary…</p>

Minseok: <p>How much is the model size? (like the one from one hour ago)<br>
Anything happened?<br>
Any code snippet we can check? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

whoab: <p>Check this thread: <a href="https://discuss.pytorch.org/t/out-of-memory-error-when-resume-training-even-though-my-gpu-is-empty/30757" class="inline-onebox">Out of memory error when resume training even though my GPU is empty</a></p>
<p>You need to load onto CPU first. Someone should file a bug report about this…</p>

Bruce_Wu: <p>I also faced this problem today, and solved it by loading on ‘cpu’ first.<br>
Later, I think the reason might be that the model was trained and saved from my gpu 0, and I tried to load it using my gpu 1. At the same time, my gpu 0 was doing something else and had no memory left. I guess that’s why loading the model on “cpu” first and sending to “gpu 1” fix the problem.</p>

codeislife99: <p>I had this same issue. This exists even to this date sadly. I had to do</p>
<pre><code class="lang-auto">state_dict = torch.load("model_path.pth",map_location=lambda storage, loc: storage)
</code></pre>
<p>which is similar to the above answers</p>