Model.cuda() fails

Andrey_Bondarenko: <p>import torch</p>
<p>class TinyModel(torch.nn.Module):</p>
<pre><code>def __init__(self):
    super(TinyModel, self).__init__()

    self.linear1 = torch.nn.Linear(100, 200)
    self.activation = torch.nn.ReLU()
    self.linear2 = torch.nn.Linear(200, 10)
    self.softmax = torch.nn.Softmax()

def forward(self, x):
    x = self.linear1(x)
    x = self.activation(x)
    x = self.linear2(x)
    x = self.softmax(x)
    return x
</code></pre>
<p>tinymodel = TinyModel()<br>
tinymodel.cuda()</p>
<p>^^^^^^^^^^^^^^^^^^^^^^</p>
<p>Produces:</p>
<hr>
<p>RuntimeError                              Traceback (most recent call last)<br>
/tmp/ipykernel_34/1502743690.py in <br>
17<br>
18 tinymodel = TinyModel()<br>
—&gt; 19 tinymodel.cuda()</p>
<p>/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py in cuda(self, device)<br>
456             Module: self<br>
457         “”"<br>
 → 458         return self._apply(lambda t: t.cuda(device))<br>
459<br>
460     def cpu(self: T) → T:</p>
<p>/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py in _apply(self, fn)<br>
352     def _apply(self, fn):<br>
353         for module in self.children():<br>
 → 354             module._apply(fn)<br>
355<br>
356         def compute_should_use_set_data(tensor, tensor_applied):</p>
<p>/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py in _apply(self, fn)<br>
374                 # <code>with torch.no_grad():</code><br>
375                 with torch.no_grad():<br>
 → 376                     param_applied = fn(param)<br>
377                 should_use_set_data = compute_should_use_set_data(param, param_applied)<br>
378                 if should_use_set_data:</p>
<p>/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py in (t)<br>
456             Module: self<br>
457         “”"<br>
 → 458         return self._apply(lambda t: t.cuda(device))<br>
459<br>
460     def cpu(self: T) → T:</p>
<p>RuntimeError: CUDA error: device-side assert triggered</p>
<p>pip list | grep torch:<br>
torch                  1.6.0<br>
torchvision            0.7.0</p>
<p>nvcc -V:<br>
nvcc: NVIDIA (R) Cuda compiler driver<br>
Copyright (c) 2005-2019 NVIDIA Corporation<br>
Built on Sun_Jul_28_19:07:16_PDT_2019<br>
Cuda compilation tools, release 10.1, V10.1.243</p>
<p>nvidia-smi:<br>
Fri Mar 25 16:39:08 2022<br>
±----------------------------------------------------------------------------+<br>
| NVIDIA-SMI 465.19.01    Driver Version: 465.19.01    CUDA Version: 11.3     |<br>
|-------------------------------±---------------------±---------------------+<br>
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |<br>
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |<br>
|                               |                      |               MIG M. |<br>
|===============================+======================+======================|<br>
|   0  NVIDIA Tesla K80    Off  | 00000001:00:00.0 Off |                    0 |<br>
| N/A   37C    P0    69W / 149W |    707MiB / 11441MiB |      0%      Default |<br>
|                               |                      |                  N/A |<br>
±------------------------------±---------------------±---------------------+</p>
<p>±----------------------------------------------------------------------------+<br>
| Processes:                                                                  |<br>
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |<br>
|        ID   ID                                                   Usage      |<br>
|=============================================================================|<br>
±----------------------------------------------------------------------------+</p>
<p>(Running from within docker container)</p>
<hr>
<p>Does anyone have any ideas on why model.cuda() rises abovelisted exception?</p>

ptrblck: <p>This sounds like a setup issue. Were you able to use the GPU on this system before and if so, did you change anything (e.g. updated the drivers without a restart etc.)?</p>

Andrey_Bondarenko: <p>I’ve did setup some long time ago, without changing anything.</p>
<p>Just before running code above I’ve launched pytorch on GPU (as confirmed by “watch nvidia-smi” - GPU utilization, memory, power usage).<br>
Code I’ve launched in jupyter notebook next to that failing one (see above) is “<a href="https://github.com/adambielski/siamese-triplet/blob/master/Experiments_MNIST.ipynb" class="inline-onebox" rel="noopener nofollow ugc">siamese-triplet/Experiments_MNIST.ipynb at master · adambielski/siamese-triplet · GitHub</a>”<br>
steps I’ve launched are:</p>
<ul>
<li>Prepare dataset</li>
<li>Common setup</li>
<li>Baseline: Classification with softmax</li>
</ul>
<p>All worked flawlessly, no restarts of the docker or any changes on host.</p>

ptrblck: <p>If I understand your use case correctly the GPU was indeed working, but you are now hitting the <code>device-side assert</code> when trying to run <em>any</em> CUDA code in your notebook?<br>
If so, then you were most likely already running into an <code>assert</code> before and the CUDA context is corrupted.<br>
I.e. once CUDA runs into an assert the context is corrupt and all following CUDA operations will fail until you restart the process.<br>
Try restarting the notebook and see if the simple <code>to()</code> operation is working again.<br>
If so, then debug which assert was triggered before that.</p>

Andrey_Bondarenko: <aside class="quote no-group" data-username="Andrey_Bondarenko" data-post="1" data-topic="147441">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/andrey_bondarenko/48/47960_2.png" class="avatar"> Andrey_Bondarenko:</div>
<blockquote>
<p><code>super(TinyModel</code></p>
</blockquote>
</aside>
<p>Wow! Thank you very much!<br>
Restart helped, as to what exactly caused that error, seems during model building I have tried to create models with mismatching layer sizes (that’s a hypothesis, I’m not 100% sure).</p>
<p>Best wishes!</p>