Getting a blue screen error "system thread exception not handled"

Karan_Nautiyal: <p>I am trying to run the following code on WSL2 (Ubuntu 20.04). Cuda version 12</p>
<pre><code class="lang-auto">class CustomDataSet(Dataset):
    
    def __init__(self, data_dir, transform = transforms.ToTensor()):
        self.images=[]
        self.data_dir = data_dir
        labels = os.listdir(self.data_dir)
        labels.sort()
        self.transform = transform
        
        for i, label in enumerate(labels):
            label_dir = os.path.join(data_dir,label)
            for image_name in os.listdir(label_dir):
                image_path = os.path.join(label_dir,image_name)
                self.images.append((image_path,i))
        
    def __len__(self):
        return len(self.images)
    
    def __getitem__(self, idx):
        img_path,label = self.images[idx]
        image = Image.open(img_path).convert('RGB')
        
        if self.transform is not None:
            image= self.transform(image)
        
        return image,label

class CustomTestDataSet(Dataset):
 
    def __init__(self, data_dir, transform = transforms.ToTensor()):
        self.images=[]
        self.data_dir = data_dir
        labels = os.listdir(self.data_dir)
        labels.sort()
        self.transform = transform
        
        for i, label in enumerate(labels):
            label_dir = os.path.join(data_dir,label)
            for image_name in os.listdir(label_dir):
                image_path = os.path.join(label_dir,image_name)
                self.images.append((image_path,i))
        
    def __len__(self):
        return len(self.images)
    
    def __getitem__(self, idx):
        img_path,label = self.images[idx]
        image = Image.open(img_path).convert('RGB')
        
        if self.transform is not None:
            image= self.transform(image)
        
        return image,img_path`
</code></pre>
<pre><code class="lang-auto">train_set = CustomDataSet(data_dir= os.path.join(datapath,"train_set") )


test_set = CustomTestDataSet(data_dir= os.path.join(datapath,"test_set") )

train_set,valid_set = torch.utils.data.random_split(train_set, [0.8, 0.2], generator=torch.Generator().manual_seed(0))

cuda_device_id=0



train_loader = DataLoader(train_set,
                          batch_size=64,
                          shuffle=True,
                          num_workers=2,
                          pin_memory = True,
                          pin_memory_device = "cuda:%i" % cuda_device_id)

test_loader = DataLoader(test_set,
                         batch_size=64,
                         shuffle=False,
                         num_workers=2,
                         pin_memory = True,
                         pin_memory_device = "cuda:%i" % cuda_device_id)

valid_loader = DataLoader(valid_set,
                         batch_size=64,
                         shuffle=True,
                         num_workers=2,
                         pin_memory = True,
                         pin_memory_device = "cuda:%i" % cuda_device_id)
</code></pre>
<p>All this works fine but when I try to run the following code</p>
<pre><code class="lang-auto">class CNN(nn.Module):
    def __init__(self):
        super(CNN, self).__init__()
        self.conv_layers = nn.Sequential(
            nn.Conv2d( 3 , 6, kernel_size = 5, stride = 1,padding = 1),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=5, stride=2),
            nn.Conv2d(6, 12, kernel_size=5, stride=1, padding = 1),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2, stride=2),
            nn.Conv2d(12, 36, kernel_size=5, stride=1, padding = 1),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=5, stride=2)
        )

        self.fc_layers = nn.Sequential(    
            nn.Linear(36*4*4,120),
            nn.Linear(120,84),
            nn.ReLU(),
            nn.Linear(84,30)

        )


    def forward(self, x):
      x = self.conv_layers(x)
      x = x.view(x.size(0), -1)
      x = self.fc_layers(x)
      return x


cnn = CNN()
cnn.to('cuda')
# Define the loss function
loss_function_cnn = nn.CrossEntropyLoss()

# Define the optimizer
optimizer_cnn = optim.SGD(cnn.parameters(), lr=0.01, momentum=0.9)
</code></pre>
<p>I get a blue screen of death nvlddmkm.sys failed in a few seconds. System thread exception not handled. I am using images from TinyImageNet30 to train the model.</p>
<p>I have narrowed it down to the line</p>
<pre><code class="lang-auto">cnn.to('cuda')
</code></pre>

Karan_Nautiyal: <p>Found the solution, the “latest” Nvidia drivers I had had a bug, and downloading a hotfix from the Nvidia forums fixed the issue</p>

StoeckOverflow: <p>Hi, I’m also encountering the same blue screen error when attempting to run a PyTorch program on my Windows 10 system with an Nvidia Quadro P500 GPU (Driver version: 531.4, CUDA Version 12.1). I suspect the issue may also be related to the CUDA toolkit version I’m using.</p>
<p>Here are the specifics of my setup:</p>
<ul>
<li>Python Version: 3.11 (installed via Microsoft Store)</li>
<li>PyTorch Version: 2.1.1+cu121 (installed via Pip)</li>
<li>Associated Libraries: torchaudio 2.1.1+cu121, torchvision 0.16.1+cu121</li>
</ul>
<p>Could you may share me the hotfix from the nvidia forum ?</p>

burhop: <p><a class="mention" href="/u/stoeckoverflow">@StoeckOverflow</a>  <a class="mention" href="/u/karan_nautiyal">@Karan_Nautiyal</a>  Did you all come up with any solutions here?  Training starts but will cause my PC to die after an hour or so:</p>
<p>Windows 10  32Gig<br>
RTX 4800 32Gig<br>
Python 3.11.8<br>
torch 2.2.1+cu121<br>
torchaudio 2.2.1+cu121<br>
torchvision 0.17.1</p>
<p>NVidia Drivers 551.61 (relase date: 02/22/2024)</p>
<p>Minidump info below:<br>
PROCESS_NAME:  System</p>
<p>STACK_TEXT:<br>
ffffe003<code>2671e028 fffff804</code>63e03ad0     : 00000000<code>00000119 00000000</code>00000005 ffffbc84<code>caeaa000 ffffbc84</code>caf30620 : nt!KeBugCheckEx<br>
ffffe003<code>2671e030 fffff804</code>7cae029d     : ffffbc84<code>cafa6000 ffffbc84</code>caeaa000 00000000<code>0000ffff fffff804</code>712a434c : watchdog!WdLogEvent5_WdCriticalError+0xe0<br>
ffffe003<code>2671e070 fffff804</code>7caced0d     : ffffbc84<code>cae78000 ffffbc84</code>caeaa000 ffffe003<code>2671e2c0 fffff804</code>7cacef8e : dxgmms2!VidSchiProcessIsrFaultedPacket+0x26d<br>
ffffe003<code>2671e0f0 fffff804</code>7cabe231     : ffffbc84<code>0001e794 fffff804</code>71d4d690 ffffbc84<code>cae78000 00000000</code>ffffffff : dxgmms2!VidSchDdiNotifyInterruptWorker+0x10a9d<br>
ffffe003<code>2671e150 fffff804</code>64a4d914     : ffffbc84<code>c423a030 ffffe003</code>2671e2c0 ffffbc84<code>c4350000 00000000</code>00000000 : dxgmms2!VidSchDdiNotifyInterrupt+0xd1<br>
ffffe003<code>2671e1a0 fffff804</code>71d6b96f     : ffffbc84<code>c423a030 ffffbc84</code>c4350000 ffffe003<code>0000000e ffffe003</code>00000000 : dxgkrnl!DxgNotifyInterruptCB+0x94<br>
ffffe003<code>2671e1d0 ffffbc84</code>c423a030     : ffffbc84<code>c4350000 ffffe003</code>0000000e ffffe003<code>00000000 fffff804</code>71d6b8f1 : nvlddmkm+0xb9b96f<br>
ffffe003<code>2671e1d8 ffffbc84</code>c4350000     : ffffe003<code>0000000e ffffe003</code>00000000 fffff804<code>71d6b8f1 ffffbc84</code>c4350000 : 0xffffbc84<code>c423a030 ffffe003</code>2671e1e0 ffffe003<code>0000000e     : ffffe003</code>00000000 fffff804<code>71d6b8f1 ffffbc84</code>c4350000 00000000<code>00000000 : 0xffffbc84</code>c4350000<br>
ffffe003<code>2671e1e8 ffffe003</code>00000000     : fffff804<code>71d6b8f1 ffffbc84</code>c4350000 00000000<code>00000000 00000000</code>00000000 : 0xffffe003<code>0000000e ffffe003</code>2671e1f0 fffff804<code>71d6b8f1     : ffffbc84</code>c4350000 00000000<code>00000000 00000000</code>00000000 00000000<code>00000000 : 0xffffe003</code>00000000<br>
ffffe003<code>2671e1f8 ffffbc84</code>c4350000     : 00000000<code>00000000 00000000</code>00000000 00000000<code>00000000 00000000</code>00000000 : nvlddmkm+0xb9b8f1<br>
ffffe003<code>2671e200 00000000</code>00000000     : 00000000<code>00000000 00000000</code>00000000 00000000<code>00000000 ffffbc84</code>c423a030 : 0xffffbc84`c4350000</p>
<p>SYMBOL_NAME:  nvlddmkm+b9b96f</p>
<p>MODULE_NAME: nvlddmkm</p>
<p>IMAGE_NAME:  nvlddmkm.sys</p>
<p>STACK_COMMAND:  .cxr; .ecxr ; kb</p>
<p>BUCKET_ID_FUNC_OFFSET:  b9b96f</p>
<p>FAILURE_BUCKET_ID:  0x119_5_DRIVER_FAULTED_SYSTEM_COMMAND_nvlddmkm!unknown_function</p>
<p>OS_VERSION:  10.0.19041.1</p>
<p>BUILDLAB_STR:  vb_release</p>
<p>OSPLATFORM_TYPE:  x64</p>
<p>OSNAME:  Windows 10</p>
<p>FAILURE_ID_HASH:  {55a61c3c-91b1-e527-dcff-f2f0d7348227}</p>