Getting Nan after first iteration with custom loss

aam541: <p>I have a complex model that calculates the low-rank matrix and try to minimize it while training CNN. The training wrapper is the following:</p>
<p>def train_generalization(args, modelc, model1, model2, device, train_loader_combined, optimizer, epoches, criterion,batch_size):<br>
for epoch in range(epoches): # loop over the dataset multiple times<br>
running_loss = 0.0<br>
for i, data in enumerate(train_loader_combined, 0):<br>
input_combined,labels_combined, flag = data<br>
labels_combined,index =torch.sort(labels_combined)<br>
input_combined=input_combined[index]<br>
flag = flag[index]<br>
input_combined, labels_combined, flag = input_combined.to(device), labels_combined.to(device), flag.to(device)<br>
optimizer.zero_grad()<br>
Hc, outputs1 = modelc(input_combined) # domain invariant representation<br>
Lg = criterion(outputs1, labels_combined)<br>
Hs1, outputs2 = model1(input_combined[flag==1])<br>
Hs2, outputs3 = model2(input_combined[flag==2])<br>
max_iteration = 1000<br>
if (epoch &gt;= 2):<br>
max_iteration = 1000<br>
Hs = torch.cat((Hs1, Hs2), 0)<br>
labels_Hs = torch.cat((labels_combined[flag==1], labels_combined[flag==2]), 0)<br>
labels_Hs,index = torch.sort(labels_Hs)<br>
Hs = Hs[index]<br>
Q = get_Q(labels_combined, labels_combined, batch_size)<br>
Z,ZZ,E = calculate_Z(torch.transpose(Hc,0,1),torch.transpose(Hs,0,1), Q, device, batch_size)<br>
Lr = get_nuc_norm(Z)+ get_fib_norm(Z-Q)<br>
Lr = Lr.to(device)<br>
loss = Lr<br>
loss.backward()<br>
optimizer.step()<br>
print(Lr)</p>
<p>My CNN is as the following:</p>
<pre><code class="lang-auto">def __init__(self):
    super(model_gen, self).__init__()
    self.conv1 = nn.Conv2d(1, 10, 5)
    self.conv2 = nn.Conv2d(10, 20, 5)
    self.conv2_drop = nn.Dropout2d()
    self.fc1 = nn.Linear(20 * 5 * 5, 120)
    self.fc2 = nn.Linear(120, 84)
    self.fc3 = nn.Linear(84, 10)

def forward(self, x):
    x = F.max_pool2d(F.relu(self.conv1(x)), (2, 2))
    x = F.max_pool2d(F.relu(self.conv2(x)), 2)
    x= self.conv2_drop(x)
    x = x.view(-1, self.num_flat_features(x))
    x = self.fc1(x) # matrix of 20*5*,120
    x1 = self.fc2(x) # vector of 84
    x2 = self.fc3(x1) # vector of 10 which are the number of classes
    x4 = F.relu(x2)
</code></pre>
<p>For some reason, after the first iteration, the fc1 and fc2 wights gives me NAN while fc3 gives me normal weights</p>
<p>Can anybody direct me on how to figure out the problem?</p>

ptrblck: <p>Currently your model uses the linear layers (<code>fc1</code>, <code>fc2</code>, <code>fc3</code>) without a non-linearity between them, so basically it’s just one single linear transformation. Is this on purpose or did you forget to add the <code>relu</code> or another activation function?<br>
Might be unrelated to this issue, but might be worth a try as a first approach.</p>

aam541: <p>Thank you for your replay</p>
<p>You are right, I did that on purpose because I am trying to mimic a paper that explained the network in this way. However, I tried to add non-linearty between them ,but unfortunately didn’t fix the NaN error.</p>
<p>debugging the code, I notice the NaN appears in the weights of the model after I call the optimizer()</p>

ptrblck: <p>Could you check the gradients in the layers which have the NANs after the update?<br>
You can print them with <code>print(model.fc1.weight.grad)</code>.</p>

aam541: <aside class="quote no-group" data-username="ptrblck" data-post="4" data-topic="25929">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>print(model.fc1.weight.grad)</p>
</blockquote>
</aside>
<p>Sure, I printed the gradient after the backward() and it shows this:<br>
tensor([[nan, nan, nan,  …, nan, nan, nan],<br>
[nan, nan, nan,  …, nan, nan, nan],<br>
[nan, nan, nan,  …, nan, nan, nan],<br>
…,<br>
[nan, nan, nan,  …, nan, nan, nan],<br>
[nan, nan, nan,  …, nan, nan, nan],<br>
[nan, nan, nan,  …, nan, nan, nan]], device=‘cuda:0’)</p>

ptrblck: <p>OK, thanks. Then we would need to see the loss function to track down these nasty NANs.</p>

aam541: <p>Sure,<br>
You can see the loss function in here: <a href="https://www.dropbox.com/s/2kh8lnkrt41rgbv/lslrrd_pytorch.py?dl=0" rel="nofollow noopener">https://www.dropbox.com/s/2kh8lnkrt41rgbv/lslrrd_pytorch.py?dl=0</a><br>
which seems pretty complicated (sorry about that).</p>
<p>I also have my own norm functions as follow:<br>
def get_fib_norm(A):<br>
    B = torch.sqrt(torch.trace(torch.mm(torch.transpose(A,0,1),A)))<br>
    return B</p>
<p>def get_nuc_norm(A):<br>
    B = torch.trace(torch.sqrt(torch.mm(A,A)))<br>
    return B</p>
<p>def get_infinity_norm(A):<br>
    B = torch.max(torch.sum(torch.abs(A),dim=1))<br>
    return B</p>
<p>def get_spec_norm(A):<br>
    l1, B, l2 = torch.svd(A,some=False)<br>
    C = torch.max(B)<br>
    return C</p>

ptrblck: <p>As your script is quite complicated, you could try to build PyTorch from source and try out the <a href="https://pytorch.org/docs/master/autograd.html#anomaly-detection">anomaly detection</a>, which will try to get the method causing the NANs.<br>
You’ll find the build instructions <a href="https://github.com/pytorch/pytorch#from-source">here</a>.<br>
Let me know, if you encounter any problems.</p>
<p>Alternatively, you could create an executable code snippet and I could try to run it on my machine.</p>

aam541: <p>Thank you</p>
<p>I will try the anomaly detection and let you know what I find. If I couldn’t find out what is causing the problem maybe I will give you the executable code</p>
<p>Thank you for your time and help again</p>

Pengbo_Ma: <p>I guess it is because you have zeros in your sqrt which causes a nan in back prop</p>

jbohnslav: <p>Thanks for pointing out anomaly detection! This was very helpful in finding where nans were coming from in my custom loss function. Protip: adding a tiny epsilon where you’re dividing or taking square roots will probably do the trick.</p>

t.ouyang: <p>Here is a way of debuging the nan problem.<br>
First, print your model gradients because there are likely to be nan in the first place.<br>
And then check the loss, and then check the input of your loss…Just follow the clue and you will find the bug resulting in nan problem.</p>
<p>There are some useful infomation about why nan problem could happen:<br>
1.the learning rate<br>
2.sqrt(0)<br>
3.ReLU-&gt;LeakyReLU</p>

aribryan: <p>I was using <a href="https://pytorch.org/docs/stable/torch.html#torch.reciprocal" rel="nofollow noopener">torch.reciprocal_</a> (for element wise reciprocal) function at some point and I had to add a small epsilon to get rid of the ‘nan’ loss value. Thank you for the overall discussion.</p>

azeeshan: <p>Is there a way to have the anomaly detection on by default? I want to avoid inserting <code>with autograd.detect_anomaly():</code> in different parts of the code.</p>

ptrblck: <p>You can add <code>torch.autograd.set_detect_anomaly(True)</code> at the beginning of the script to enable it globally.</p>

Saurabh_Kataria: <p>this solved one of my problem. thanks!</p>

changwn: <p><a class="mention" href="/u/t.ouyang">@t.ouyang</a> Thanks. Your advice solve my NaN problem.<br>
My problem caused due to sqrt(0).<br>
<span class="hashtag">#------------------------------------</span><br>
There are some useful infomation about why nan problem could happen:<br>
1.the learning rate<br>
2.sqrt(0)<br>
3.ReLU-&gt;LeakyReLU</p>

Sentient07: <p>Hello.</p>
<p>I face exactly the same issue. A similarity with OP’s post is that we both seem to compute norm. As <a class="mention" href="/u/pengbo_ma">@Pengbo_Ma</a> mentioned, this seems to vanish when I add a small epsilon to <code>linalg.matrix_norm()</code>. Ideally, this should be supported as a parameter of the function and/or pytorch must handle it internally. <a class="mention" href="/u/ptrblck">@ptrblck</a> can you please comment on what is the best practise ?</p>

ptrblck: <p>I think adding a small <code>eps</code> value to the input matrix is the right approach as it’s explicitly guarding against invalid gradients as seen e.g. here:</p>
<pre><code class="lang-python">x = torch.zeros(4, 4, requires_grad=True)
out = torch.linalg.matrix_norm(x)
out.backward()
print(x.grad)
# &gt; tensor([[nan, nan, nan, nan],
#          [nan, nan, nan, nan],
#          [nan, nan, nan, nan],
#          [nan, nan, nan, nan]])
</code></pre>
<p>You could add e.g. <code>1e-6</code> to the input to avoid the NaN values in the gradient.</p>

duyducvo4444: <p>Dear <a class="mention" href="/u/ptrblck">@ptrblck</a> ,<br>
Is there anyway to get the tensor’s value after <code>set_detect_anomaly</code> raises the error?<br>
Specifically, it returns to me this trace</p>
<blockquote>
<p>File “/home/s1910442/Project/Master3/models/ComplexVQ2_WITHOUT_PRETRAINING.py”, line 111, in si_snr<br>
snr = 10 * torch.log10(target_norm / (noise_norm + eps) + eps)<br>
(function _print_stack)<br>
Traceback (most recent call last):<br>
File “train_no_pretrain.py”, line 128, in <br>
train(**train_config)<br>
File “train_no_pretrain.py”, line 57, in train<br>
loss.backward()<br>
File “/opt/conda/lib/python3.8/site-packages/torch/_tensor.py”, line 256, in backward<br>
torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)<br>
File “/opt/conda/lib/python3.8/site-packages/torch/autograd/<strong>init</strong>.py”, line 147, in backward<br>
Variable._execution_engine.run_backward(<br>
RuntimeError: Function ‘Log10Backward’ returned nan values in its 0th output.</p>
</blockquote>
<p>What I want to ask is how can I get the values of <code>target_norm</code> and <code>noise_norm</code>? eps here is just 1e-9</p>