CUDA error 59: Device-side assert triggered | by Perez Ogayo | Towards Data Science

If you happen to run into this error, you know how frustrating it can be. The most frustrating part, for me, was the lack of a clear, step-by-step solution to this problem. This could be caused by the fact that PyTorch is still relatively new.
A little backstory ...
I was working on the Stanford car dataset as part of a hackathon born out of my participation in Udacity Pytorch Challenge when I encountered this problem. It took me quite a while to fix it, and this was worsened by the fact that I was using Kaggle Kernels which presented its own challenges with regards to GPU.
What is the Error?
This error occurs due to the following two reasons:
Inconsistency between the number of labels/classes and the number of output unitsThe input of the loss function may be incorrect.
Inconsistency between the number of labels/classes and the number of output units
The input of the loss function may be incorrect.
Let’s unpack these reasons and their solutions below:
Reason 1: Inconsistency between number of labels and number of output units
When defining the final fully connected layer of my model, instead of putting 196(total number of classes for the Stanford car dataset) as the number of output units, I put 195.
The error is usually identified in the line where you do the backpropagation. Your loss function will be comparing the output from your model and the label of that observation in your dataset. Just in case you are confused between labels and output, see how I define them below:
Label: This is the tags associated with an observation. When working on a classification problem, the label is the class. For example, in the classic dog vs cat problem, the labels are cat and dog.
Output: This is the predicted ‘label’ from your model. You give your model a set of features from an observation and it gives out a prediction called output in the PyTorch ecosystem. Think of it as a predicted label.
In my case, some of the labels will have a value 195 which is beyond the range of output for my model whose greatest possible value is 194(Start counting from zero). This causes the error to be triggered.
How can you fix it?
Make sure the number of output units match the number of your classes
That would be changing my classifier to be as follows:
classifier = nn.Sequential(nn.Linear(2048, 700), nn.ReLU(), nn.Dropout(p = 0.2), nn.Linear(700 , 300), nn.ReLU(), nn.Dropout(p = 0.2), nn.Linear(300 ,196), #changed this from 195 to 196 nn.LogSoftmax(dim = 1))model.fc = classifier
This is how you get the number of classes in your dataset programmatically:
#.classes returns a list of the classes in your dataset, usually #numbered from 0 to number_of_classes-1len(dataset.classes)#Calling len() function on the list will return the number of classes in your dataset
Reason 2: Wrong input for the loss function
Loss functions have different ranges for the possible inputs that they can accept. If you choose an incompatible activation function for your output layer this error will be triggered. For example, BCELoss requires its input to be between 0 and 1. If the input(output from your model) is beyond the acceptable range for that particular loss function, the error gets triggered.
Recap on activation loss functions
Activation functions are the mathematical equations that determine the output of your neural network. The purpose of the activation function is to introduce non-linearity into the output of a model thus making a model capable to learn and perform more complex tasks. They, in turn, determine how accurate your network gets to be.
Loss functions are the equations that compute the error that is used to learn by means of backpropagation.
For a detailed explanation of loss functions and how they work, please check out this article and the Pytorch documentation. Find information about activation function here.
How can you fix it?
Make sure your output layer returns values in the range of the loss function(criterion) that you chose. This implies using the appropriate activation function(Sigmoid, Softmax, LogSoftmax) in your final output layer.
Example of problematic code
model = nn.Linear()input = torch.randn(128, 2)output = model(input)criterion=nn.BCELoss()torch.empty(128).random_(2)loss=criterion(output, target)
The code above will trigger runtime error 59 if we are using GPU. You can fix it by passing your output through sigmoid function or using BCEWithLogitsLoss().
Fix 1: Passing the results through Sigmoid function
model = nn.Linear()  #The sigmoid funtion can also be applied here as model=nn.Sigmoid(nn.Linear())input = torch.randn(128, 2)output = model(input)criterion=nn.BCELoss()target=torch.empty(128).random_(2)loss=criterion(nn.Sigmoid(output), target)
Fix 2: Using BCEWithLogitsLoss() instead
model = nn.Linear()input = torch.randn(128, 2)output = model(input)criterion=nn.BCEWithLogitsLoss() #changed fromBCELosstarget = torch.empty(128).random_(2)loss=criterion(output, target)
Working on Kaggle? Here’s why even after following the above steps, you are still struggling
Once that error is triggered, you cannot continue using your GPU. Even after changing the problematic line and reloading the entire kernel, you will still get the error presented in different forms. The form depends on which line is the first one to attempt to use the GPU. Look at the image below for an idea.
The reason why this is happening even though you have fixed the bug in your code is that once the runtime error 59 is triggered, you are blocked from using the GPU entirely during the same GPU session that the error was triggered in.
How can you fix it?
Stop your current kernel session and start a new one.
To do this, follow the steps below:
From your Kernel, click the ‘K’ on the top left. It automatically takes you to your kernelsYou will see a list of your kernels, having edit option and additional stop for the ones currently running.Click stop kernelOpen your Kernel afresh, every variable will be reset and you will have a brand new GPU session
From your Kernel, click the ‘K’ on the top left. It automatically takes you to your kernels
You will see a list of your kernels, having edit option and additional stop for the ones currently running.
Click stop kernel
Open your Kernel afresh, every variable will be reset and you will have a brand new GPU session
Extra tip
The error messages you get when running into this error may not be very descriptive. To make sure you get the complete and useful stack trace, have this at the very beginning of your code and run it before anything else:
CUDA_LAUNCH_BLOCKING="1"
References
[1]Thread, GPU goes away after an error occurs #1010(2018), Pytorch Github Issues pages
[2] Lernapprat, Debugging CUDA device-side assert in PyTorch (2018), https://lernapparat.de/
I hope this article helped you. Feel free to leave your comments on any aspect of this tutorial in the response section below.