Loss nan when resuming from a pretrained model

cindybrain: <p>When I trained resnet18 on ImageNet, I stop it at epoch 30. Then in a later period, i train it again resuming from the pretrained model(epoch 30). However, during the training process, the training loss always turns to nan half way. And I try three times, it always has the same problem(loss nan at different iteration of epoch 30)<br>
The training code is the official training example at <a href="https://github.com/pytorch/examples/blob/master/imagenet/main.py" class="inline-onebox" rel="noopener nofollow ugc">examples/imagenet/main.py at main · pytorch/examples · GitHub</a><br>
I don’t know how the solve the problem. If someone can help me, I will be quite grateful!<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/1/8139631c126584d21679370cdead5c4c6af5f4ea.png" data-download-href="https://discuss.pytorch.org/uploads/default/8139631c126584d21679370cdead5c4c6af5f4ea" title="loss_nan_pic"><img src="https://discuss.pytorch.org/uploads/default/original/3X/8/1/8139631c126584d21679370cdead5c4c6af5f4ea.png" alt="loss_nan_pic" data-base62-sha1="iraxNiFmaZ7V60ByXROzF7hf1yO" width="690" height="114" data-dominant-color="1D1D1D"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">loss_nan_pic</span><span class="informations">1431×237 21.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>Could you check the input for invalid values via:</p>
<pre><code class="lang-python">print(torch.isfinite(input), torch.isfinite(target))
</code></pre>
<p>Did you change anything else in the code?</p>

cindybrain: <p>The values all show true by using</p>
<pre><code class="lang-auto">print(torch.isfinite(input), torch.isfinite(target))
</code></pre>
<p>I change the code a little only for recording the accuracy and not showing the Data time. I think it doesnot effect the training process.</p>
<p>And now, when I train it from scratch by using the official code(not change anything this time), I always get nan at iteration 2</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/a/caeb9e453f079e91743415c6d594c4993d75736f.png" data-download-href="https://discuss.pytorch.org/uploads/default/caeb9e453f079e91743415c6d594c4993d75736f" title="error"><img src="https://discuss.pytorch.org/uploads/default/original/3X/c/a/caeb9e453f079e91743415c6d594c4993d75736f.png" alt="error" data-base62-sha1="sX7g2N2k7YlRvzd6xxS9DhtTJev" width="690" height="96" data-dominant-color="121212"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">error</span><span class="informations">1633×228 16.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>after using this script</p>
<pre><code class="lang-auto">torch.autograd.set_detect_anomaly(True)
with torch.autograd.detect_anomaly():
	loss.backward()
</code></pre>
<p>I got the error message as follows:</p>
<pre><code class="lang-auto">(pytorch) shu@hec02:~/GANs/Image Classification/ImageNet$ python main_official.py  /nfs/home/Imagenet_ILSVRC2012
=&gt; creating model 'resnet18'


        ...,


        [[[True, True, True,  ..., True, True, True],
          [True, True, True,  ..., True, True, True],
          [True, True, True,  ..., True, True, True],
          ...,
          [True, True, True,  ..., True, True, True],
          [True, True, True,  ..., True, True, True],
          [True, True, True,  ..., True, True, True]],

         [[True, True, True,  ..., True, True, True],
          [True, True, True,  ..., True, True, True],
          [True, True, True,  ..., True, True, True],
          ...,
          [True, True, True,  ..., True, True, True],
          [True, True, True,  ..., True, True, True],
          [True, True, True,  ..., True, True, True]],

         [[True, True, True,  ..., True, True, True],
          [True, True, True,  ..., True, True, True],
          [True, True, True,  ..., True, True, True],
          ...,
          [True, True, True,  ..., True, True, T
          [True, True, True,  ..., True, True, True],
          [True, True, True,  ..., True, True, True]]]]) tensor([True, True, True, True, True, True, True, True, True, True, True, True,
        True, True, True, True, True, True, True, True, True, True, True, True,
        True, True, True, True, True, True, True, True, True, True, True, True,
        True, True, True, True, True, True, True, True, True, True, True, True,
        True, True, True, True, True, True, True, True, True, True, True, True,
        True, True, True, True, True, True, True, True, True, True, True, True,
        True, True, True, True, True, True, True, True, True, True, True, True,
        True, True, True, True, True, True, True, True, True, True, True, True,
        True, True, True, True, True, True, True, True, True, True, True, True,
        True, True, True, True, True, True, True, True, True, True, True, True,
        True, True, True, True, True, True, True, True, True, True, True, True,
        True, True, True, True, True, True, True, True, True, True, True, True,
        True, True, True, True, True, True, True, True, True, True, True, True,
        True, True, True, True, True, True, True, True, True, True, True, True,
        True, True, True, True, True, True, True, True, True, True, True, True,
        True, True, True, True, True, True, True, True, True, True, True, True,
        True, True, True, True, True, True, True, True, True, True, True, True,
        True, True, True, True, True, True, True, True, True, True, True, True,
        True, True, True, True, True, True, True, True, True, True, True, True,
        True, True, True, True, True, True, True, True, True, True, True, True,
        True, True, True, True, True, True, True, True, True, True, True, True,
        True, True, True, True])
/nfs/home/shu/.local/lib/python3.6/site-packages/torch/autograd/anomaly_mode.py:70: UserWarning: Anomaly Detection has been enabled. This mode will increase the runtime and should only be enabled for debugging.
  warnings.warn('Anomaly Detection has been enabled. '
Epoch: [0][   0/5005]   Time  5.869 ( 5.869)    Data  4.310 ( 4.310)    Loss 7.0919e+00 (7.0919e+00)    Acc@1   0.78 (  0.78)       Acc@5   0.78 (  0.78)


        ...,


        [[[True, True, True,  ..., True, True, True],
          [True, True, True,  ..., True, True, True],
          [True, True, True,  ..., True, True, True],
          ...,
          [True, True, True,  ..., True, True, True],
          [True, True, True,  ..., True, True, True],
          [True, True, True,  ..., True, True, True]],

         [[True, True, True,  ..., True, True, True],
          [True, True, True,  ..., True, True, True],
          [True, True, True,  ..., True, True, True],
          ...,
          [True, True, True,  ..., True, True, True],
          [True, True, True,  ..., True, True, True],
          [True, True, True,  ..., True, True, True]],

         [[True, True, True,  ..., True, True, True],
          [True, True, True,  ..., True, True, True],
          [True, True, True,  ..., True, True, True],
          ...,
          [True, True, True,  ..., True, True, T
          [True, True, True,  ..., True, True, True],
          [True, True, True,  ..., True, True, True]]]]) tensor([True, True, True, True, True, True, True, True, True, True, True, True,
        True, True, True, True, True, True, True, True, True, True, True, True,
        True, True, True, True, True, True, True, True, True, True, True, True,
        True, True, True, True, True, True, True, True, True, True, True, True,
        True, True, True, True, True, True, True, True, True, True, True, True,
        True, True, True, True, True, True, True, True, True, True, True, True,
        True, True, True, True, True, True, True, True, True, True, True, True,
        True, True, True, True, True, True, True, True, True, True, True, True,
        True, True, True, True, True, True, True, True, True, True, True, True,
        True, True, True, True, True, True, True, True, True, True, True, True,
        True, True, True, True, True, True, True, True, True, True, True, True,
        True, True, True, True, True, True, True, True, True, True, True, True,
        True, True, True, True, True, True, True, True, True, True, True, True,
        True, True, True, True, True, True, True, True, True, True, True, True,
        True, True, True, True, True, True, True, True, True, True, True, True,
        True, True, True, True, True, True, True, True, True, True, True, True,
        True, True, True, True, True, True, True, True, True, True, True, True,
        True, True, True, True, True, True, True, True, True, True, True, True,
        True, True, True, True, True, True, True, True, True, True, True, True,
        True, True, True, True, True, True, True, True, True, True, True, True,
        True, True, True, True, True, True, True, True, True, True, True, True,
        True, True, True, True])
Warning: Error detected in LogSoftmaxBackward. Traceback of forward call that caused the error:
  File "main_official.py", line 439, in &lt;module&gt;
    main()
  File "main_official.py", line 118, in main
    main_worker(args.gpu, ngpus_per_node, args)
  File "main_official.py", line 251, in main_worker
    train(train_loader, model, criterion, optimizer, epoch, args)
  File "main_official.py", line 299, in train
    loss = criterion(output, target)
  File "/nfs/home/shu/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/nfs/home/shu/.local/lib/python3.6/site-packages/torch/nn/modules/loss.py", line 932, in forward
    ignore_index=self.ignore_index, reduction=self.reduction)
  File "/nfs/home/shu/.local/lib/python3.6/site-packages/torch/nn/functional.py", line 2317, in cross_entropy
    return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
  File "/nfs/home/shu/.local/lib/python3.6/site-packages/torch/nn/functional.py", line 1535, in log_softmax
    ret = input.log_softmax(dim)
 (print_stack at /pytorch/torch/csrc/autograd/python_anomaly_mode.cpp:60)
Traceback (most recent call last):
  File "main_official.py", line 439, in &lt;module&gt;
    main()
  File "main_official.py", line 118, in main
    main_worker(args.gpu, ngpus_per_node, args)
  File "main_official.py", line 251, in main_worker
    train(train_loader, model, criterion, optimizer, epoch, args)
  File "main_official.py", line 310, in train
    loss.backward()
  File "/nfs/home/shu/.local/lib/python3.6/site-packages/torch/tensor.py", line 198, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/nfs/home/shu/.local/lib/python3.6/site-packages/torch/autograd/__init__.py", line 100, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: Function 'LogSoftmaxBackward' returned nan values in its 0th output.
</code></pre>
<p>I do not know what is wrong with it.</p>

ptrblck: <p>Thanks for the update. Could you add an <code>.all()</code> operation to the check and also check the <code>output</code> tensor as:</p>
<pre><code class="lang-python">print(torch.isfinite(input).all(), torch.isfinite(target).all(), torch.isfinite(output).all())
</code></pre>
<p>It seems that the loss is getting a NaN value, so I guess that the model might output NaNs.</p>

ZdsAlpha: <p>Did you make any changes to the model. It might have something to do with final activation layer.</p>

cindybrain: <p>I did not change the model, the model is created in the code below</p>
<pre><code class="lang-auto">print("=&gt; creating model '{}'".format(args.arch))
        model = models.__dict__[args.arch]()
</code></pre>
<p>and because this problem happens sometimes, and sometimes not. Now, it seems to work fine, therefore, I will update the error message if I meet nan again. Thanks very much! <a class="mention" href="/u/ptrblck">@ptrblck</a> <a class="mention" href="/u/zdsalpha">@ZdsAlpha</a>  <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

cindybrain: <pre><code class="lang-auto">Epoch: [1][2120/5005]   Loss 3.5405e+00 (3.6588e+00)    Acc@1  25.00 ( 25.53)   Acc@5  51.56 ( 49.25)
tensor(True) tensor(True, device='cuda:0') tensor(True, device='cuda:0')
tensor(True) tensor(True, device='cuda:0') tensor(True, device='cuda:0')
tensor(True) tensor(True, device='cuda:0') tensor(True, device='cuda:0')
tensor(True) tensor(True, device='cuda:0') tensor(True, device='cuda:0')
tensor(True) tensor(True, device='cuda:0') tensor(True, device='cuda:0')
tensor(True) tensor(True, device='cuda:0') tensor(True, device='cuda:0')
tensor(True) tensor(True, device='cuda:0') tensor(True, device='cuda:0')
tensor(True) tensor(True, device='cuda:0') tensor(True, device='cuda:0')
tensor(True) tensor(True, device='cuda:0') tensor(True, device='cuda:0')
tensor(True) tensor(True, device='cuda:0') tensor(True, device='cuda:0')
Epoch: [1][2130/5005]   Loss 3.7294e+00 (3.6584e+00)    Acc@1  23.05 ( 25.54)   Acc@5  50.39 ( 49.26)
tensor(True) tensor(True, device='cuda:0') tensor(True, device='cuda:0')
tensor(True) tensor(True, device='cuda:0') tensor(True, device='cuda:0')
tensor(True) tensor(True, device='cuda:0') tensor(True, device='cuda:0')
tensor(True) tensor(True, device='cuda:0') tensor(True, device='cuda:0')
tensor(True) tensor(True, device='cuda:0') tensor(True, device='cuda:0')
tensor(True) tensor(True, device='cuda:0') tensor(True, device='cuda:0')
tensor(True) tensor(True, device='cuda:0') tensor(False, device='cuda:0')
tensor(True) tensor(True, device='cuda:0') tensor(False, device='cuda:0')
tensor(True) tensor(True, device='cuda:0') tensor(False, device='cuda:0')
tensor(True) tensor(True, device='cuda:0') tensor(False, device='cuda:0')
Epoch: [1][2140/5005]   Loss nan (nan)  Acc@1   0.00 ( 25.49)   Acc@5   1.17 ( 49.18)
tensor(True) tensor(True, device='cuda:0') tensor(False, device='cuda:0')
tensor(True) tensor(True, device='cuda:0') tensor(False, device='cuda:0')
tensor(True) tensor(True, device='cuda:0') tensor(False, device='cuda:0')
tensor(True) tensor(True, device='cuda:0') tensor(False, device='cuda:0')
tensor(True) tensor(True, device='cuda:0') tensor(False, device='cuda:0')
tensor(True) tensor(True, device='cuda:0') tensor(False, device='cuda:0')
tensor(True) tensor(True, device='cuda:0') tensor(False, device='cuda:0')
tensor(True) tensor(True, device='cuda:0') tensor(False, device='cuda:0')
tensor(True) tensor(True, device='cuda:0') tensor(False, device='cuda:0')
tensor(True) tensor(True, device='cuda:0') tensor(False, device='cuda:0')
Epoch: [1][2150/5005]   Loss nan (nan)  Acc@1   0.00 ( 25.38)   Acc@5   0.78 ( 48.96)
tensor(True) tensor(True, device='cuda:0') tensor(False, device='cuda:0')
tensor(True) tensor(True, device='cuda:0') tensor(False, device='cuda:0')
tensor(True) tensor(True, device='cuda:0') tensor(False, device='cuda:0')
tensor(True) tensor(True, device='cuda:0') tensor(False, device='cuda:0')
tensor(True) tensor(True, device='cuda:0') tensor(False, device='cuda:0')
tensor(True) tensor(True, device='cuda:0') tensor(False, device='cuda:0')
tensor(True) tensor(True, device='cuda:0') tensor(False, device='cuda:0')
tensor(True) tensor(True, device='cuda:0') tensor(False, device='cuda:0')
tensor(True) tensor(True, device='cuda:0') tensor(False, device='cuda:0')
tensor(True) tensor(True, device='cuda:0') tensor(False, device='cuda:0')
Epoch: [1][2160/5005]   Loss nan (nan)  Acc@1   0.00 ( 25.26)   Acc@5   0.78 ( 48.73)
tensor(True) tensor(True, device='cuda:0') tensor(False, device='cuda:0')
tensor(True) tensor(True, device='cuda:0') tensor(False, device='cuda:0')
tensor(True) tensor(True, device='cuda:0') tensor(False, device='cuda:0')
tensor(True) tensor(True, device='cuda:0') tensor(False, device='cuda:0')
tensor(True) tensor(True, device='cuda:0') tensor(False, device='cuda:0')
tensor(True) tensor(True, device='cuda:0') tensor(False, device='cuda:0')
tensor(True) tensor(True, device='cuda:0') tensor(False, device='cuda:0')
tensor(True) tensor(True, device='cuda:0') tensor(False, device='cuda:0')
tensor(True) tensor(True, device='cuda:0') tensor(False, device='cuda:0')
tensor(True) tensor(True, device='cuda:0') tensor(False, device='cuda:0')
</code></pre>
<p>I got this message when add this code.</p>

ptrblck: <p>For some reason the output is getting NaN values.<br>
Could you <code>break</code> the training loop, after you’ve encountered the first NaN and check all parameters of the model?<br>
E.g. you could print their <code>abs().max()</code> via:</p>
<pre><code class="lang-python">for name, param in model.named_parameters():
    print(name, param.abs().max())
</code></pre>
<p>If this looks alright, you could repeat the last forward iteration (since the input contains valid values) and check all intermediate activations to narrow down, which layer creates the NaN outputs using forward hooks as described <a href="https://discuss.pytorch.org/t/how-can-l-load-my-best-model-as-a-feature-extractor-evaluator/17254/6">here</a>.</p>

chandrak1907: <p>Hi <a class="mention" href="/u/cindybrain">@cindybrain</a> , Could you please advise if you found a solution to this? I’m also encountering a similar situation, where model runs fine on some occasions and gives nan on other occasions.</p>