CUDA RuntimeError: CUDA error: an illegal memory access was encountered

mbacher: <p>Hi, I am using colab, pytorch version 1.6.0+cu101. When I try to allocate a model or a parameter to GPU I get the error below. I have tried to reproduce several tips/corrections that were listed in the forum but no one has worked so far…any ideas?<br>
Thanks!</p>
<pre><code class="lang-auto">device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
print('Using device:', device)
g = nn.Parameter(torch.rand((100,128)), requires_grad=False)
if device.type=='cuda':
  g = g.to(device)
  print(next(g.parameters()).is_cuda) # returns a boolean
</code></pre>
<p>Here is the error message:</p>
<pre><code class="lang-auto">Using device: cuda
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-35-f9356a863a45&gt; in &lt;module&gt;()
      4 g = nn.Parameter(torch.rand((100,128)), requires_grad=False)
      5 if device.type=='cuda':
----&gt; 6   g = g.to(device)
      7   print(next(g.parameters()).is_cuda) # returns a boolean

RuntimeError: CUDA error: an illegal memory access was encountered
</code></pre>

ptrblck: <p>Have you tried to run the code with <code>CUDA_LAUNCH_BLOCKING=1 python script.py args</code>?<br>
If so, could you post the stack trace here, please?</p>

mbacher: <p>for some reason, after restarting the colab, I get a new error:</p>
<p>RuntimeError: cuDNN error: CUDNN_STATUS_NOT_INITIALIZED</p>

ptrblck: <p>Do you get this error message with the blocking command?<br>
If so, could you post the complete stack trace?</p>

mbacher: <p>This is what I am trying to run now. Without .cuda() it runs OK.</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.distributions import Uniform, Normal
import torch.utils.data as tdata
import torch.optim as optim
import numpy as np


class DepthToSpace(nn.Module):
    def __init__(self, block_size):
        super().__init__()
        self.block_size = block_size
        self.block_size_sq = block_size * block_size
 
    def forward(self, input):
        output = input.permute(0, 2, 3, 1)
        (batch_size, d_height, d_width, d_depth) = output.size()
        s_depth = int(d_depth / self.block_size_sq)
        s_width = int(d_width * self.block_size)
        s_height = int(d_height * self.block_size)
        t_1 = output.reshape(batch_size, d_height, d_width, self.block_size_sq, s_depth)
        spl = t_1.split(self.block_size, 3)
        stack = [t_t.reshape(batch_size, d_height, s_width, s_depth) for t_t in spl]
        output = torch.stack(stack, 0).transpose(0, 1).permute(0, 2, 1, 3, 4).reshape(batch_size, s_height, s_width,
                                                                                      s_depth)
        output = output.permute(0, 3, 1, 2)
        return output

# Spatial Upsampling with Nearest Neighbors
class Upsample_Conv2d(nn.Module):
  def __init__(self, in_dim, out_dim, kernel_size=(3, 3), stride=1, padding=1):
    super(Upsample_Conv2d, self).__init__()
    self.depth_to_space = DepthToSpace(block_size=2)
    self.conv2d = nn.Conv2d(in_dim, out_dim, kernel_size, stride=stride, 
                            padding=padding)

  def forward(self, x):
    x_ = torch.cat([x, x, x, x], dim=1)
    x_ = self.depth_to_space(x_)
    x_ = self.conv2d(x_)
    return x_


class ReshapeGan(nn.Module):
  def __init__(self, out_shape):
    super(ReshapeGan, self).__init__()
    self.out_shape = out_shape
  
  def forward(self, x):
    b = x.shape[0]
    return x.view(b, *self.out_shape)

class Generator(nn.Module):
  def __init__(self, n_filters=128):
    super(Generator, self).__init__()
    self.n_filters = n_filters
    self.latent = torch.distributions.Normal(torch.tensor(0.), torch.tensor(1.))
    self.network = nn.Sequential(
        nn.Linear(n_filters, 4*4*256),
        ReshapeGan((256,4,4)),
        ResnetBlockUp(in_dim=256, n_filters=n_filters),
        ResnetBlockUp(in_dim=n_filters, n_filters=n_filters),
        ResnetBlockUp(in_dim=n_filters, n_filters=n_filters),
        nn.BatchNorm2d(n_filters),
        nn.ReLU(),
        nn.Conv2d(n_filters, 3, kernel_size=(3, 3), padding=1),
        nn.Tanh()
    )

  def forward(self, z):
    return self.network(z)

  def sample(self, n):
    z = self.latent.sample([n, self.n_filters])
    return self.forward(z)

CUDA_LAUNCH_BLOCKING=1
g = Generator(128)
g = g.cuda() # HERE IT CRASHES
z = g.sample(10)
print(z.shape)

</code></pre>

ptrblck: <p>The code is not executable, since <code>ResnetBlockUp</code> is undefined.</p>

mbacher: <p>Ups, sorry:</p>
<pre><code class="lang-auto">class ResnetBlockUp(nn.Module):
  def __init__(self, in_dim, kernel_size=(3, 3), n_filters=256):
    super(ResnetBlockUp, self).__init__()
    self.layers = nn.ModuleList(
        [nn.BatchNorm2d(in_dim), 
         nn.ReLU(),
         nn.Conv2d(in_dim, n_filters, kernel_size, padding=1),
         nn.BatchNorm2d(n_filters), 
         nn.ReLU(),
         Upsample_Conv2d(n_filters, n_filters, kernel_size=kernel_size, padding=1)])
    self.shortcut = Upsample_Conv2d(in_dim, n_filters, 
                                    kernel_size=(1, 1), padding=0)

  def forward(self, x):
    x_ = x
    for layer in self.layers:
      x_ = layer(x_)
    x = self.shortcut(x)
    return x + x_
</code></pre>
<p>The strange thing is that it crashes with error code:</p>
<p>RuntimeError: cuDNN error: CUDNN_STATUS_NOT_INITIALIZED</p>
<p>right after resetting the colab at the g.sample(10) line. Then at .cuda() line I get the illegal access error after retrying to run without resetting the colab…</p>

ptrblck: <p>Thanks for the rest of the code.<br>
While running it I get a proper error message:</p>
<pre><code class="lang-auto">RuntimeError: Tensor for argument #2 'mat1' is on CPU, but expected it to be on GPU (while checking arguments for addmm)
</code></pre>
<p>which points to a device mismatch in <code>Generator</code>.<br>
After checking the code it seems that <code>z</code> is created on the CPU, while the parameters of the model are on the GPU.<br>
Use</p>
<pre><code class="lang-python">z = self.latent.sample([n, self.n_filters]).cuda()
</code></pre>
<p>or</p>
<pre><code class="lang-python">z = self.latent.sample([n, self.n_filters]).to(next(self.parameters()).device)
</code></pre>
<p>to fix this issue.</p>

mbacher: <p>Amazing help! Thank you very much, I have just missed the gpu data load.<br>
BTW, what should I have done in order to get the error message?<br>
Again, thanks!</p>

ptrblck: <p>It shouldn’t be the case, but maybe the notebook is somehow raising the “wrong” error message? <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"><br>
In any way, updating to the latest stable release (1.7) or the nightly is also a good idea and might help.</p>

pelin_tas: <p>can you help me??<br>
I’m trying to write a code about disease diagnosis through retinal images. I try different codes but I get errors separately from all of them</p>
<p>first code</p>
<p>import torch<br>
import torch.nn as nn<br>
import torch.nn.functional as F<br>
import torchvision<br>
import torchvision.transforms as transforms<br>
import matplotlib.pyplot as plt<br>
import torch.optim as optim<br>
import numpy as np<br>
train_transforms = transforms.Compose([transforms.Grayscale(),<br>
transforms.Resize(255),<br>
transforms.RandomRotation(38),<br>
transforms.RandomResizedCrop(224),<br>
transforms.ToTensor(),<br>
transforms.Normalize([0.5],[0.5])])<br>
test_transforms=transforms.Compose([transforms.Grayscale(),<br>
transforms.Resize(255),<br>
transforms.CenterCrop(244),<br>
transforms.ToTensor()])<br>
train_dataset=torchvision.datasets.ImageFolder(root=‘C:/OCT2017/train’, transform=train_transforms)<br>
test_dataset=torchvision.datasets.ImageFolder(root=‘C:/OCT2017/test’, transform=test_transforms)<br>
train_loader = torch.utils.data.DataLoader(train_dataset, batch_size = 100, shuffle = False)<br>
test_loader = torch.utils.data.DataLoader(test_dataset, batch_size = 100, shuffle = False)<br>
class NN(nn.Module):<br>
def <strong>init</strong>(self, input_size, num_classes):<br>
super(NN.self).<strong>init</strong>()<br>
self.fc1=nn.Linear(input_size,50)<br>
def forward(self,x):<br>
x=F.relu(self.fc1(x))<br>
return x</p>
<p>class OCTModel(nn.Module):<br>
def <strong>init</strong>(self,in_channels=1, num_classes =4):<br>
super(OCTModel, self).<strong>init</strong>()<br>
# Convolution 1<br>
self.conv1= nn.Conv2d(in_channels=1, out_channels=32, kernel_size=(3,3), stride=(1))<br>
self.relu1 = nn.ReLU()<br>
self.pool = nn.MaxPool2d(kernel_size=2)<br>
self.dropout1=nn.Dropout(p=0.2)<br>
self.relu2=nn.ReLU()<br>
self.fc1 = nn.Linear(256,num_classes)</p>
<pre><code>def forward(self, x):
    # Convolution 1
    x = self.conv1(x)
    x = self.relu1(x)
    x = self.pool(x)
    x = self.dropout1(x)
    x = self.relu2(x)
    x = x.reshape(x.shape[0], -1)
    x = self.fc1(x)
    
    return x
</code></pre>
<p>device = torch.device(‘cuda’ if torch.cuda.is_available() else ‘cpu’)<br>
in_channels=1<br>
num_classes=4<br>
learning_rate=0.001<br>
batch_size=10<br>
num_epochs=1<br>
model = OCTModel().to(device)<br>
Criterion=nn.CrossEntropyLoss()<br>
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)<br>
for epoch in range (num_epochs):<br>
for batch_idx, (data,targets) in enumerate(train_loader,0):<br>
data=data.to(device=device)<br>
targets=targets.to(device=device)<br>
scores=model(data)<br>
loss = Criterion(scores,targets)<br>
optimizer.zero_grad()<br>
loss.backward()<br>
optimizer.step()<br>
def check_accuracy (loader, model):<br>
if loader.datasets.train:<br>
print (“Checking accuracy on training data”)<br>
else:<br>
print(“Checking accuary on test data”)<br>
num_correct=0<br>
num_samples=0<br>
model.eval()<br>
with torch.no_grad():<br>
for x, y in loader:<br>
x=x.to(device=device)<br>
y=y.to(device=device)<br>
scores=model(x)<br>
_, predictions = scores.max(1)<br>
num_correct += (predictions == y).sum()<br>
num_samples += predictions.size(0)<br>
print(f’Got {num_correct} / {num_samples} with accuracy {float(num_correct)/float(num_samples)*100:.2f}’)<br>
model.train()<br>
check_accuracy(train_loader,model)<br>
check_accuracy(test_loader,model)</p>
<p>RuntimeError: CUDA out of memory. Tried to allocate 152.00 MiB (GPU 0; 2.00 GiB total capacity; 1.20 GiB already allocated; 121.74 MiB free; 9.86 MiB cached)</p>
<p>second code</p>
<p>class OCTModel(nn.Module):<br>
def <strong>init</strong>(self,in_channels=1, num_classes =4):<br>
super(OCTModel, self).<strong>init</strong>()<br>
# Convolution 1<br>
self.conv1= nn.Conv2d(in_channels=1, out_channels=32, kernel_size=(3,3), stride=(1))<br>
self.relu1 = nn.ReLU()<br>
self.pool = nn.MaxPool2d(kernel_size=2)<br>
self.dropout1=nn.Dropout(p=0.2)<br>
self.relu2=nn.ReLU()<br>
self.fc1 = nn.Linear(256,num_classes)</p>
<pre><code>def forward(self, x):
    # Convolution 1
    x = self.conv1(x)
    x = self.relu1(x)
    x = self.pool(x)
    x = self.dropout1(x)
    x = self.relu2(x)
    x = self.fc1(x)
    
    return x
</code></pre>
<p>device = torch.device(‘cuda’ if torch.cuda.is_available() else ‘cpu’)<br>
input_size=784<br>
num_classes = 4<br>
learning_rate=0.001<br>
batch_size=32<br>
num_epochs=1<br>
train_transforms = transforms.Compose([transforms.Grayscale(),<br>
transforms.Resize(255),<br>
transforms.RandomRotation(38),<br>
transforms.RandomResizedCrop(224),<br>
transforms.ToTensor(),<br>
transforms.Normalize([0.5],[0.5])])<br>
test_transforms=transforms.Compose([transforms.Grayscale(),<br>
transforms.Resize(255),<br>
transforms.CenterCrop(244),<br>
transforms.ToTensor()])<br>
train_dataset=torchvision.datasets.ImageFolder(root=‘C:/OCT2017/train’, transform=train_transforms)<br>
test_dataset=torchvision.datasets.ImageFolder(root=‘C:/OCT2017/test’, transform=test_transforms)<br>
train_loader = torch.utils.data.DataLoader(train_dataset, batch_size = 32, shuffle = False)<br>
test_loader = torch.utils.data.DataLoader(test_dataset, batch_size = 32, shuffle = False)<br>
model = OCTModel().to(device)<br>
Criterion=nn.CrossEntropyLoss()<br>
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)<br>
for epoch in range (num_epochs):<br>
for batch_idx, (data,targets) in enumerate(train_loader,0):<br>
data=data.to(device=device)<br>
targets=targets.to(device=device)<br>
data = data.reshape(data.shape[0],-1)<br>
scores=model(data)<br>
loss = Criterion(scores,targets)<br>
optimizer.zero_grad()<br>
loss.backward()<br>
optimizer.step()<br>
def check_accuracy (loader, model):<br>
if loader.datasets.train:<br>
print (“Checking accuracy on training data”)<br>
else:<br>
print(“Checking accuary on test data”)<br>
num_correct=0<br>
num_samples=0<br>
model.eval()<br>
with torch.no_grad():<br>
for x, y in loader:<br>
x=x.to(device=device)<br>
y=y.to(device=device)<br>
x=x.reshape(x.shape[0],-1)<br>
scores=model(x.unsqueeze(dim=1))<br>
_, predictions = scores.max(1)<br>
num_correct += (predictions == y).sum()<br>
num_samples += predictions.size(0)<br>
print(f’Got {num_correct} / {num_samples} with accuracy {float(num_correct)/float(num_samples)*100:.2f}’)<br>
model.train()<br>
check_accuracy(train_loader,model)<br>
check_accuracy(test_loader,model)</p>
<p>RuntimeError: Expected 4-dimensional input for 4-dimensional weight 32 1, but got 2-dimensional input of size [32, 50176] instead</p>

ptrblck: <p>The error message is:</p>
<pre><code class="lang-python">RuntimeError: Expected 4-dimensional input for 4-dimensional weight 32 1, but got 2-dimensional input of size [32, 50176] instead
</code></pre>
<p>which doesn’t point towards an illegal memory access, but a shape mismatch.<br>
Most likely you are trying to pass a 2-dimensional input to e.g. an <code>nn.Conv2d</code> layer, which expects an input in the shape <code>[batch_size, channels, height, width]</code>.<br>
Since this error is unrelated, feel free to create a new topic, if you get stuck.</p>
<p>PS: you can post code snippets by wrapping them into three backticks ```, which would make debugging easier. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

pelin_tas: <aside class="quote no-group quote-modified" data-username="ptrblck" data-post="12" data-topic="101318">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>e <code>nn.Conv2d</code> bir girdi bekleyen bir katmana iletmeye çalışıyorsunuz <code>[batch_size, channels, height, width]</code> .<br>
Bu hata ilgisiz olduğundan, takılırsanız yeni bir konu oluşturmaktan çekinmeyin.</p>
<p>Not: Kod parçacıklarını üç geri işaretine sararak gönderebilirsiniz, bu da hata ayıklamayı kolaylaştırır. : göz kırpma:</p>
</blockquote>
</aside>
<p>The database has four categories of gray level retinal images. I guess I’m having a mistake here. I’m very sorry, but I don’t quite understand what to do. <img src="https://discuss.pytorch.org/images/emoji/apple/pensive.png?v=12" title=":pensive:" class="emoji" alt=":pensive:" loading="lazy" width="20" height="20"></p>
<p>[image]</p>

ptrblck: <p>Your input images should have the mentioned shape (<code>[batch_size, channels, height, width]</code>), while they are 2-dimensional tensors.<br>
Have a look at <a href="https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html">this tutorial</a> for an example on how to build a model and how the data tensors should be created.</p>