RuntimeError : device-side assert triggered

Yehan_Kodithuwakku: <p>Hi, I am on a Linux server and the GPU is GeForce RTX3090. I use transfer learning to train my model. I use the model resnet18 for my purpose. Since I have 2 classes to classify, I use 2 output units for my final layer. But when I start the training process it goes to a runtime error. Please help me resolve this.</p>
<p>Error:<br>
RuntimeError: CUDA error: device-side assert triggered<br>
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.<br>
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.<br>
Compile with <code>TORCH_USE_CUDA_DSA</code> to enable device-side assertions.</p>
<p>Following is the code.</p>
<p>import torch<br>
import torch.nn as nn<br>
import torch.optim as optim<br>
from torch.optim import lr_scheduler<br>
import numpy as np<br>
import torchvision<br>
from torchvision import datasets, models, transforms<br>
import matplotlib.pyplot as plt<br>
import time<br>
import os<br>
import copy</p>
<p>mean = np.array([0.5, 0.5, 0.5])<br>
std = np.array([0.25, 0.25, 0.25])</p>
<p>data_transforms = {</p>
<pre><code>'train': transforms.Compose([
    transforms.RandomResizedCrop(224),
    transforms.RandomHorizontalFlip(),
    transforms.ToTensor(),
    transforms.Normalize(mean, std)

]),

'val': transforms.Compose([
    transforms.Resize(256),
    transforms.CenterCrop(224),
    transforms.ToTensor(),
    transforms.Normalize(mean, std)
]),
</code></pre>
<p>}</p>
<p>data_dir = ‘//home//CAMA_cat//trainP1//’</p>
<p>image_datasets = {x: datasets.ImageFolder(os.path.join(data_dir, x),<br>
data_transforms[x])<br>
for x in [‘train’, ‘val’]}</p>
<p>dataloaders = {x: torch.utils.data.DataLoader(image_datasets[x], batch_size=2,<br>
shuffle=True, num_workers=0)<br>
for x in [‘train’, ‘val’]}</p>
<p>dataset_sizes = {x: len(image_datasets[x]) for x in [‘train’, ‘val’]}<br>
class_names = image_datasets[‘train’].classes</p>
<p>device = torch.device(“cuda:0” if torch.cuda.is_available() else “cpu”)<br>
print(class_names)</p>
<p>def imshow(inp, title):<br>
inp = inp.numpy().transpose((1, 2, 0))<br>
inp = std * inp + mean<br>
inp = np.clip(inp, 0, 1)<br>
plt.imshow(inp)<br>
plt.title(title)<br>
plt.show()</p>
<p>inputs, classes = next(iter(dataloaders[‘train’]))<br>
out = torchvision.utils.make_grid(inputs)<br>
imshow(out, title=[class_names[x] for x in classes])</p>
<p>def train_model(model, criterion, optimizer, scheduler, num_epochs=25):<br>
since = time.time()<br>
best_model_wts = copy.deepcopy(model.state_dict())<br>
best_acc = 0.0<br>
for epoch in range(num_epochs):<br>
print(‘Epoch {}/{}’.format(epoch, num_epochs - 1))<br>
print(‘-’ * 10)</p>
<pre><code>    # Each epoch has a training and validation phase

    for phase in ['train', 'val']:
        if phase == 'train':
            model.train()  # Set model to training mode
        else:
            model.eval()   # Set model to evaluate mode
        running_loss = 0.0
        running_corrects = 0



        # Iterate over data.
        for inputs, labels in dataloaders[phase]:
            inputs = inputs.to(device)
            labels = labels.to(device)



            # forward

            # track history if only in train

            with torch.set_grad_enabled(phase == 'train'):
                outputs = model(inputs)
                _, preds = torch.max(outputs, 1)
                loss = criterion(outputs, labels)



                # backward + optimize only if in training phase

                if phase == 'train':
                    optimizer.zero_grad()
                    loss.backward()
                    optimizer.step()



            # statistics

            running_loss += loss.item() * inputs.size(0)
            running_corrects += torch.sum(preds == labels.data)



        if phase == 'train':
            scheduler.step()



        epoch_loss = running_loss / dataset_sizes[phase]
        epoch_acc = running_corrects.double() / dataset_sizes[phase]



        print('{} Loss: {:.4f} Acc: {:.4f}'.format(
            phase, epoch_loss, epoch_acc))



        # deep copy the model

        if phase == 'val' and epoch_acc &gt; best_acc:
            best_acc = epoch_acc
            best_model_wts = copy.deepcopy(model.state_dict())



    print()



time_elapsed = time.time() - since

print('Training complete in {:.0f}m {:.0f}s'.format(
    time_elapsed // 60, time_elapsed % 60))
print('Best val Acc: {:4f}'.format(best_acc))



# load best model weights

model.load_state_dict(best_model_wts)
return model
</code></pre>
<p>model = models.resnet18(pretrained=True)</p>
<p>num_ftrs = model.fc.in_features</p>
<p><span class="hashtag-raw">#ERROR</span> CELL</p>
<p>model.fc = nn.Linear(num_ftrs, 2)<br>
model = model.to(device)<br>
criterion = nn.BCELoss()<br>
optimizer = optim.SGD(model.parameters(), lr=0.001)<br>
step_lr_scheduler = lr_scheduler.StepLR(optimizer, step_size=7, gamma=0.1)<br>
model = train_model(model, criterion, optimizer, step_lr_scheduler, num_epochs=25)</p>

ptrblck: <p>Rerun your code via <code>CUDA_LAUNCH_BLOCKING=1 python script.py args</code> as suggested in the error message and check which operation fails.<br>
Device-side asserts are triggered e.g. by invalid indexing operations etc.</p>

Yehan_Kodithuwakku: <p>Thank you so much for the quick response. I tried, but it says “No module named torch” when I run in the terminal. But PyTorch is installed and the chosen python environment is also correct. I used “pip show torch”</p>

ptrblck: <p>This would indicate that a different Python environment is used in your terminal, so make sure to use the same one as in your original setup.</p>

Yehan_Kodithuwakku: <p>Thanks. Managed to overcome the problem. I rerun the code using CUDA_LAUNCH_BLOCKING=1 python script.py args then found this error:<br>
ValueError: Using a target size (torch.size([2])) that is different to the input size (torch.Size([2,2])) is deprecated. Please ensure they have the same size.</p>