Out of memory error when resume training even though my GPU is empty

username: jdhao
cooked: <p>I am training a classification model and I have saved some checkpoints. When I try to resume training, however, I got out of memory errors:</p>
<blockquote>
<p>Traceback (most recent call last):<br>
File “train.py”, line 283, in <br>
main()<br>
File “train.py”, line 86, in main<br>
optimizer.load_state_dict(checkpoint[‘optimizer’])<br>
File “/home/haojiedong/tools/anaconda3/lib/python3.6/site-packages/torch/optim/optimizer.py”, line 96, in load_state_dict<br>
state_dict = deepcopy(state_dict)<br>
File “/home/haojiedong/tools/anaconda3/lib/python3.6/copy.py”, line 150, in deepcopy<br>
y = copier(x, memo)<br>
File “/home/haojiedong/tools/anaconda3/lib/python3.6/copy.py”, line 240, in _deepcopy_dict<br>
y[deepcopy(key, memo)] = deepcopy(value, memo)<br>
File “/home/haojiedong/tools/anaconda3/lib/python3.6/copy.py”, line 150, in deepcopy<br>
y = copier(x, memo)<br>
File “/home/haojiedong/tools/anaconda3/lib/python3.6/copy.py”, line 240, in _deepcopy_dict<br>
y[deepcopy(key, memo)] = deepcopy(value, memo)<br>
File “/home/haojiedong/tools/anaconda3/lib/python3.6/copy.py”, line 150, in deepcopy<br>
y = copier(x, memo)<br>
File “/home/haojiedong/tools/anaconda3/lib/python3.6/copy.py”, line 240, in <em>deepcopy_dict<br>
y[deepcopy(key, memo)] = deepcopy(value, memo)<br>
File “/home/haojiedong/tools/anaconda3/lib/python3.6/copy.py”, line 161, in deepcopy<br>
y = copier(memo)<br>
File “/home/haojiedong/tools/anaconda3/lib/python3.6/site-packages/torch/tensor.py”, line 23, in <strong>deepcopy</strong><br>
new_storage = self.storage().<strong>deepcopy</strong>(memo)<br>
File “/home/haojiedong/tools/anaconda3/lib/python3.6/site-packages/torch/storage.py”, line 28, in <strong>deepcopy</strong><br>
new_storage = self.clone()<br>
File “/home/haojiedong/tools/anaconda3/lib/python3.6/site-packages/torch/storage.py”, line 42, in clone<br>
return type(self)(self.size()).copy</em>(self)<br>
RuntimeError: CUDA error: out of memory</p>
</blockquote>
<p>The error is in <code>optimizer.load_state_dict()</code>.  The code to load the checkpoint is like this:</p>
<pre><code class="lang-auto">if args.resume:                                                      
    if os.path.isfile(args.resume):                                  
        print("=&gt; loading checkpoint '{}'".format(args.resume))      
        checkpoint = torch.load(args.resume)                         
        args.start_epoch = checkpoint['epoch']                       
        best_acc = checkpoint['best_acc']                            
        model.load_state_dict(checkpoint['state_dict'])              
        optimizer.load_state_dict(checkpoint['optimizer'])           
        print("=&gt; loaded checkpoint '{}' (epoch {})".format(         
            args.resume, checkpoint['epoch']))                       
    else:                                                            
        print("=&gt; no checkpoint found at '{}'".format(args.resume))  
</code></pre>
<p>I checked the target GPU, it is actually empty.  I am currently using pytorch version 0.4.1.</p>

username: ptrblck
cooked: <p>Could you try to load the checkpoints onto the CPU first using the <code>map_location</code> argument in <a href="https://pytorch.org/docs/stable/torch.html#torch.load">torch.load</a>?<br>
After it was successful, try to push your model onto the GPU again.</p>

username: Ray_Luo
cooked: <p>Thanks. It solved the problem!<br>
<code>checkpoint = torch.load(path, map_location='cpu')</code></p>

username: cosmosaa
cooked: <p>This did not work for me. I am encountering the same issue. I am able to train on 1 fold, then when I reset state_dicts for model, optimizer and scheduler, I get OOM error at <code>optimizer.step()</code></p>
<p>pytorch version <code>'1.10.0+cu102'</code></p>
<p>If I understood correctly, you suggested</p>
<pre><code class="lang-auto">checkpoint = torch.load(..., map_location='cpu')
model.load_state_dict(checkpoint['state_dict'])
model = model.cuda()
</code></pre>
<p>This gets out of memory at <code>optimizer.step()</code> after training successfully on 1 fold.</p>

username: ptrblck
cooked: <p>This would mean that additional tensors are most likely pushed or created on the device, so compare the memory usage before resetting the states and after each step of loading the model to isolate where the memory overhead is coming from.</p>

username: cosmosaa
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> I found the issue using <code>nvidia-smi</code>; it was related to AWS sagemaker /jupyter notebook leaving orphaned processes on the GPU which was why there wasn’t enough memory available. Unsure why there were orphaned processes on the GPU.</p>