(Updated) NVIDIA RTX A6000 INCOMPATIBLE WITH PYTORCH

tjk: <p>Hello！<br>
Several days ago I posted a issue about pytorch with NVIDIA RTX A6000 GPU, here is the origianl link:<br>
<a href="https://discuss.pytorch.org/t/nvidia-rtx-a6000-gpu-incompatible-with-pytorch/135450">Nvidia rtx a6000 gpu incompatible with pytorch - windows - PyTorch Forums</a></p>
<p>Many thanks to ptrblck, and I have tried to uninstall CUDA on our workstation and built CUDA 11.3 toolkit and pytorch from source using the following command:<br>
conda install pytorch torchvision torchaudio cudatoolkit=11.3 -c pytorch<br>
But before the finishing of first epoch, the same error occured:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/b/1b02e7e8e95a78a81db4f95b14f7b0d486ab3bbb.png" data-download-href="https://discuss.pytorch.org/uploads/default/1b02e7e8e95a78a81db4f95b14f7b0d486ab3bbb" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/1/b/1b02e7e8e95a78a81db4f95b14f7b0d486ab3bbb.png" alt="image" data-base62-sha1="3QX7hTEFs5uPtsCPWQwZFLlDDK3" width="690" height="295" data-dominant-color="F4FAF4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">850×364 15.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Could you please offer any other suggestions about this issue? Thanks a lot.<br>
PS: My classmates and I think the problem described by ptrblck was rather fancy, because we uninstalled CUDA 10.2 and installed CUDA 11.1 using binary file from MVIDIA official site. We then uninstalled CUDA 11.1 on windows 10 and built cuda and pytorch from source, but it didn’t solve the problem. We tried to run the same program on CUDA 11.1 and pytorch 1.8.1, it can run three epoches, and the error :RuntimeError: CUDA error: an illegal memory access was encountered would pop out.</p>

ptrblck: <aside class="quote no-group" data-username="tjk" data-post="1" data-topic="136046">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/t/dec6dc/48.png" class="avatar"> tjk:</div>
<blockquote>
<p>We tried to run the same program on CUDA 11.1 and pytorch 1.8.1, it can run three epoches, and the error :RuntimeError: CUDA error: an illegal memory access was encountered would pop out.</p>
</blockquote>
</aside>
<p>Since you are seeing the cuDNN issue apparently in both setups, could you post an executable code snippet which would reproduce the illegal memory access on the A6000, please?</p>

tjk: <p>Of course we can, the underlined part of the following picture denote the position where error occured:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/5/a53cb6573275a22cdfae544a02133d0365d56b7c.png" data-download-href="https://discuss.pytorch.org/uploads/default/a53cb6573275a22cdfae544a02133d0365d56b7c" title="loop"><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/5/a53cb6573275a22cdfae544a02133d0365d56b7c.png" alt="loop" data-base62-sha1="nzKRcnlFt4fuKE07Cm6hyZ7cp6Q" width="558" height="500" data-dominant-color="F5F5F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">loop</span><span class="informations">570×510 14.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

tjk: <p>and the network structure is defined as follows:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/8/88dba88787878dad6dec1f52cac31fe75bf49e7a.png" data-download-href="https://discuss.pytorch.org/uploads/default/88dba88787878dad6dec1f52cac31fe75bf49e7a" title="network-structure"><img src="https://discuss.pytorch.org/uploads/default/original/3X/8/8/88dba88787878dad6dec1f52cac31fe75bf49e7a.png" alt="network-structure" data-base62-sha1="jwHycX8fil5ysWM0ZEdNQ5Tu4kO" width="517" height="500" data-dominant-color="F8F7F7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">network-structure</span><span class="informations">669×647 23.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

tjk: <p><img src="https://discuss.pytorch.org/uploads/default/original/3X/2/7/27fe8eb090ca9c1a4475f469befea6987bb79e17.png" alt="image" data-base62-sha1="5HO0uPYyWfZneDlUVGBXiv5HTxB" width="521" height="301"></p>

ptrblck: <p>Could you post it as a code snippet by wrapping it into three backticks ```?<br>
Also, please post the shapes for all input tensors which are needed to execute the code and reproduce the issue.</p>

tjk: <p>OK, the network and parameter updating process is defined as follows:</p>
<pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        #self.conv1 = nn.Conv2d(3, 6, 5)
        #self.fc1 = nn.Linear(72*12, 768)
        #self.pool = nn.MaxPool2d(2, 2)
        self.fc1 = nn.Linear(2*3*64*64, 80*80)
        self.fc2 = nn.Linear(80*80, 125*125)
        #self.fc3 = nn.Linear(2*3*16*16, 60*60)
        self.us = nn.Upsample(scale_factor=2, mode='bilinear', align_corners=True)
        self.conv1 = nn.Conv2d(1, 64, 5,padding=2) # would decrease size 4 in each dimension
        self.conv2 = nn.Conv2d(64, 32, 5,padding=2)
        self.conv3 = nn.Conv2d(32, 16, 5,padding=2)
        self.conv4 = nn.Conv2d(16, 4, 5,padding=2)
        self.conv5 = nn.Conv2d(4, 1, 5,padding=2)


    # With relu and wihout relu has similar result    
    def forward(self, x):
        x = x.view(-1, 2*3*64*64)
        #x = self.conv1(x)
        #x = x.view(-1, 72*12)
        ## Fully connected layers
        x = self.fc1(x)
        #x = F.relu(self.fc1(x))
        x = nn.LeakyReLU(0.1)(x)
        x = self.fc2(x)
        # x = F.relu(self.fc2(x))
        x = nn.LeakyReLU(0.1)(x)	
        x = torch.reshape(x, (np.shape(x)[0], -1, 125, 125)) # 
        ## Upsampling
        x = self.us(x)
        ## Five convolution layers
        x = nn.LeakyReLU(0.1)( self.conv1(x) )
        x = nn.LeakyReLU(0.1)( self.conv2(x) )
        x = nn.LeakyReLU(0.1)( self.conv3(x) )
        x = nn.LeakyReLU(0.1)( self.conv4(x) )
        x = self.conv5(x)
        x = x.view(np.shape(x)[0], -1)
        # x = self.conv1(x)
        return x
#Generate model
net = Net().double()
#Batch size too small, no parallelization, only 1 GPU
#device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
if torch.cuda.is_available():
    device = torch.device("cuda:0")
    print("Use GPU")
else:
    device = torch.device("cpu")
    print("Use CPU")



#Use all 3 GPUs
# if torch.cuda.device_count() &gt; 1:
#     print("Let's use", torch.cuda.device_count(), "GPUs!")
#     net = nn.DataParallel(net)
#Put data on all GPUs

net.to(device)
#Learning Process
# Train the network
for epoch in range(nepoch):
    running_loss = 0.0
    current_loss.append(10)
    print('Epoch: %d\n' %(epoch))
    for i, data in enumerate(trainloader, 0):
        #print(i)
        inputs, labels = data[0].to(device), data[1].to(device)
        # Zero gradient
        optimizer.zero_grad()
        # Forward, backward(gradient), optimize (add gradient to weight)
        outputs = net(inputs)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()
        
        # Print loss information
        running_loss += loss.item()
        current_loss[epoch] += loss.item()
        print(loss.item())
        if i % 4 == 3: # Print every 4 mini-batches
            #print('[%d, %3d] loss: %.3f' % (epoch+1, i+1, running_loss/9))
            scheduler.step(running_loss)
            loss_history.append(running_loss)
            running_loss = 0.0
    current_loss[epoch] = current_loss[epoch] - 10
    if epoch &gt;=1:
        if current_loss[epoch] &lt; min(current_loss[0:epoch]):
            PATH = '%s/Corrosion-conv-fc-%d.pth' % (fd, niter+1)
            torch.save(net.state_dict(), PATH)
    else:
        PATH = '%s/Corrosion-conv-fc-%d.pth' % (fd, niter+1)
        torch.save(net.state_dict(), PATH)
</code></pre>
<p>Shape of input tensors:<br>
inputs: torch.Size([64,2,3,64,64])<br>
labels: torch.Size([64,62500])<br>
when it runs to outputs = net(inputs), the error occured</p>

ptrblck: <p>Thanks for the code!<br>
I’ve let the model train for a few hours using PyTorch <code>1.9.1+cu113</code> as well as <code>1.8.2+cu111</code> on an A6000 using this code:</p>
<pre><code class="lang-python">import torch
import torch.nn as nn
from torch.utils.data import TensorDataset, DataLoader
import numpy as np


class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        #self.conv1 = nn.Conv2d(3, 6, 5)
        #self.fc1 = nn.Linear(72*12, 768)
        #self.pool = nn.MaxPool2d(2, 2)
        self.fc1 = nn.Linear(2*3*64*64, 80*80)
        self.fc2 = nn.Linear(80*80, 125*125)
        #self.fc3 = nn.Linear(2*3*16*16, 60*60)
        self.us = nn.Upsample(scale_factor=2, mode='bilinear', align_corners=True)
        self.conv1 = nn.Conv2d(1, 64, 5,padding=2) # would decrease size 4 in each dimension
        self.conv2 = nn.Conv2d(64, 32, 5,padding=2)
        self.conv3 = nn.Conv2d(32, 16, 5,padding=2)
        self.conv4 = nn.Conv2d(16, 4, 5,padding=2)
        self.conv5 = nn.Conv2d(4, 1, 5,padding=2)


    # With relu and wihout relu has similar result    
    def forward(self, x):
        x = x.view(-1, 2*3*64*64)
        #x = self.conv1(x)
        #x = x.view(-1, 72*12)
        ## Fully connected layers
        x = self.fc1(x)
        #x = F.relu(self.fc1(x))
        x = nn.LeakyReLU(0.1)(x)
        x = self.fc2(x)
        # x = F.relu(self.fc2(x))
        x = nn.LeakyReLU(0.1)(x)	
        x = torch.reshape(x, (np.shape(x)[0], -1, 125, 125)) # 
        ## Upsampling
        x = self.us(x)
        ## Five convolution layers
        x = nn.LeakyReLU(0.1)( self.conv1(x) )
        x = nn.LeakyReLU(0.1)( self.conv2(x) )
        x = nn.LeakyReLU(0.1)( self.conv3(x) )
        x = nn.LeakyReLU(0.1)( self.conv4(x) )
        x = self.conv5(x)
        x = x.view(np.shape(x)[0], -1)
        # x = self.conv1(x)
        return x
#Generate model
net = Net().double()
#Batch size too small, no parallelization, only 1 GPU
#device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
if torch.cuda.is_available():
    device = torch.device("cuda:0")
    print("Use GPU")
else:
    device = torch.device("cpu")
    print("Use CPU")


x = torch.randn(64*100, 2, 3, 64, 64).double()
y = torch.randn(64*100, 62500).double() 

dataset = TensorDataset(x, y)
trainloader = DataLoader(dataset, batch_size=64)
criterion = nn.MSELoss()

#Use all 3 GPUs
# if torch.cuda.device_count() &gt; 1:
#     print("Let's use", torch.cuda.device_count(), "GPUs!")
#     net = nn.DataParallel(net)
#Put data on all GPUs

net.to(device)
#Learning Process
# Train the network
nepoch = 4
optimizer = torch.optim.Adam(net.parameters(), lr=1e-3)
for epoch in range(nepoch):
    print('Epoch: %d\n' %(epoch))
    for i, data in enumerate(trainloader, 0):
        #print(i)
        inputs, labels = data[0].to(device), data[1].to(device)
        # Zero gradient
        optimizer.zero_grad()
        # Forward, backward(gradient), optimize (add gradient to weight)
        outputs = net(inputs)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()
        
        # Print loss information
        print(loss.item())
</code></pre>
<p>and could reproduce any issue.</p>

tjk: <p>Thanks, but I am a little confused, you said: ''and could reproduce any issue", did you mean you ran the code completely and no problem was found?</p>

ptrblck: <p>Ah sorry… missing word. Yes, I executed the posted code and was not able to run in any issue. The code worked for ~5 hours without any problems on the A6000.</p>

tjk: <p>Could you please leave an e-mail so that I can send the complete code to you?</p>

ptrblck: <p>Could you post the code in a GitHub Gist, please?<br>
Also, are you able to run into the error using my posted code snippet?</p>

tjk: <p>Yes, I run into error with your code snippet. Do you know how to build cuda from source in windows10? I see the guide on pytorch and it seems that build from source in ubuntu is easy, but build in windows 10 is complicated.</p>

tjk: <p>Since the computer environment of our workstation is windows10 and we use spyder to run the program, is it possible that the error occured because pytorch is trying to use the memory allocated for windows OS?</p>

ptrblck: <p>I don’t know enough about Windows and its specifics to be able to speculate about the root cause of the issue. Could you try to create the cuDNN API logs as described <a href="https://docs.nvidia.com/deeplearning/cudnn/developer-guide/index.html#api-logging">here</a> and send them to me?</p>

tjk: <p>OK，We are producing the cudnn log according to instructions. By the way，can this A6000 GPU support ubuntu 16.04LTS? It seems that install the cuda11.x driver is possible but the manual of a6000 says the minimum OS requirement is ubuntu 18.</p>

ptrblck: <p>I don’t know if an older Ubuntu could cause issues. Since 16.04LTS’ EOL was April 30th 2021 I would generally recommend to update to a supported OS. For my test I’ve been using 20.04LTS.</p>

tjk: <p>Hello!<br>
The Cudnn log file is produced and sent to the moderators as private messages. The log produced with  the code snippet above ran for one epoch is also uploaded on github.</p>

ptrblck: <p>Thanks! Let me check it. Btw. I assume your workload is not “private” and you’ve thus sent it.</p>

tjk: <p>Hello，have you found out the cause of the problem? Thanks a lot，by the way.</p>