CUDA error when loading my model

username: blade
cooked: <p>I’m trying to load a model and run it on GPU nodes. The code snippet is:</p>
<pre><code class="lang-auto">import torch

from torch import nn


class MyModel(nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()

        # -- embedding params
        self.cn1 = nn.Conv2d(1, 256, kernel_size=3, stride=2)
        self.cn2 = nn.Conv2d(256, 256, kernel_size=3, stride=1)
        self.cn3 = nn.Conv2d(256, 256, kernel_size=3, stride=2)
        self.cn4 = nn.Conv2d(256, 256, kernel_size=3, stride=1)
        self.cn5 = nn.Conv2d(256, 256, kernel_size=3, stride=2)
        self.cn6 = nn.Conv2d(256, 256, kernel_size=3, stride=2)

        # -- prediction params
        self.fc1 = nn.Linear(2304, 1700)
        self.fc2 = nn.Linear(1700, 1200)
        self.fc3 = nn.Linear(1200, 964)


class Train:
    def __init__(self):

        self.path_pretrained = './model_stat.pth'
        self.model = self.load_model()

    def load_model(self):

        # -- init model
        model = MyModel()
        old_model = torch.load(self.path_pretrained)
        for old_key in old_model:
            dict(model.named_parameters())[old_key].data = old_model[old_key]

        return model.to('cuda')


my_train = Train()
</code></pre>
<p>I run this with no issue on a CPU (with <code>.to('cpu')</code>). However, when I run it on GPU, I get the following error:</p>
<blockquote>
<p>Traceback (most recent call last):<br>
File “test.py”, line 41, in <br>
my_train = Train()<br>
File “test.py”, line 28, in <strong>init</strong><br>
self.model = self.load_model()<br>
File “test.py”, line 38, in load_model<br>
return model.to(‘cuda’)<br>
File “/…/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 907, in to<br>
return self._apply(convert)<br>
File “/…/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 578, in _apply<br>
module._apply(fn)<br>
File “/…/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 601, in _apply<br>
param_applied = fn(param)<br>
File “/…/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 905, in convert<br>
return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)<br>
RuntimeError: CUDA error: out of memory<br>
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.<br>
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.</p>
</blockquote>
<p>I was wondering how to resolve this issue. The model can be found <a href="https://drive.google.com/file/d/1dYlO--1R_p7q3f6kSMpIduG25h2K-3iF/view?usp=sharing" rel="noopener nofollow ugc">here</a>.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="blade" data-post="1" data-topic="139100">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/blade/48/23861_2.png" class="avatar"> blade:</div>
<blockquote>
<p>RuntimeError: CUDA error: out of memory</p>
</blockquote>
</aside>
<p>This error would mea that your GPU is running out of memory and you would have to lower the memory usage by e.g. lowering the batch size of the input.<br>
If that’s not the case and you are immediately seeing the issue using any CUDATensor, then I guess it’s a double post from <a href="https://discuss.pytorch.org/t/getting-runtimeerror-cuda-error-out-of-memory-when-memory-is-free/138780/4">here</a>, which you’ve marked as solved.</p>

username: blade
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Thanks for the response. As you had correctly mentioned, the issue with that case was environment-related and was caused by conflicting libraries.</p>
<p>Last night I tried running the code in this question again, and it ran smoothly. Since when I log into the remote machine, I get assigned to a randomly selected GPU, perhaps the issue was from the particular GPU node that I was assigned to in that session.</p>

username: blade
cooked: <p>For the future readers that end up here, I figured that the problem in my case was that when I suspended a previous job using <code>ctrl + z</code>, the memory was not freed. Hence, I ran out of memory after a few runs. I used</p>
<p><code>ps -elf | grep python</code></p>
<p>to find the processes and killed them using</p>
<p><code>kill -9 [pid]</code></p>
<p>to free up the space. Here is a relative <a href="https://discuss.pytorch.org/t/pytorch-doesnt-free-gpus-memory-of-it-gets-aborted-due-to-out-of-memory-error/13775/14">discussion</a>.</p>