Embedding layer appear nan

username: JBoRu
cooked: <p>Excuse me,<br>
When I use the Embedding layer and randomly initialize it and update it during training, however, after one or two epochs, the weights in the Embedding layer change to nan, causing all subsequent model outputs to be nan, triggering “CUDA error: device-side assert triggered”, I want to know why the weights in the Embedding layer change to nan during training?</p>

username: ptrblck
cooked: <p>Could you check the gradients in the embedding layer and see, if some large values are seen?<br>
Also, are you sure this layer’s parameters are updated to NaN values? Could you run the code with <code>torch.autograd.set_detect_anomaly(True)</code> and post the stack trace here?</p>

username: JBoRu
cooked: <p>Thanks,<br>
i sure the enbedding layer’s weight are updated to NAN, because i debug the code and find this truth. But i use the clip=50/30/20/5/1, the problem still occur. And i plan to print the max value of the model’s weight between <code> loss.backward()</code> and <code>model.step()</code>. And i will also try your advice.</p>
<p>.</p>

username: JBoRu
cooked: <p>Oh! The stack trace as follow:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/f/afb233f4824b21caff05817c05113ff579891d5f.png" data-download-href="https://discuss.pytorch.org/uploads/default/afb233f4824b21caff05817c05113ff579891d5f" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/f/afb233f4824b21caff05817c05113ff579891d5f.png" alt="image" data-base62-sha1="p4hm26LBKt4fNEI8szeSG1qrKTJ" width="690" height="434" data-dominant-color="343232"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1221×769 44 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Here is my own mask_loss code:</p>
<pre><code class="lang-auto">def maskNLLLoss(inp, target, mask):
    nTotal = mask.sum()
    crossEntropy = -torch.log(torch.gather(inp, 1, target.view(-1, 1)).squeeze(1))
    loss = crossEntropy.masked_select(mask).mean()
    loss = loss.to(device)
    return loss, nTotal.item()
</code></pre>
<p>The <code>inp</code> is the model output after F.softmax() compute and the target is the gold padding sequence and mask is the length mask. Are there some problem?<br>
Here is the instance of the two variant:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/a/0ae9bbea28f16722bb9585fa7c86a1c441873daf.png" data-download-href="https://discuss.pytorch.org/uploads/default/0ae9bbea28f16722bb9585fa7c86a1c441873daf" title="1"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/a/0ae9bbea28f16722bb9585fa7c86a1c441873daf_2_690x384.png" alt="1" data-base62-sha1="1yxxWTv6OB2KmYtIED1Nx0gfGrR" width="690" height="384" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/a/0ae9bbea28f16722bb9585fa7c86a1c441873daf_2_690x384.png, https://discuss.pytorch.org/uploads/default/optimized/3X/0/a/0ae9bbea28f16722bb9585fa7c86a1c441873daf_2_1035x576.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/0/a/0ae9bbea28f16722bb9585fa7c86a1c441873daf_2_1380x768.png 2x" data-dominant-color="2D2B2A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">1</span><span class="informations">1384×771 105 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div> <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/2/923e5ee9090b2ce5ae4f379fe16f8c3292c6a026.png" data-download-href="https://discuss.pytorch.org/uploads/default/923e5ee9090b2ce5ae4f379fe16f8c3292c6a026" title="2"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/2/923e5ee9090b2ce5ae4f379fe16f8c3292c6a026_2_689x389.png" alt="2" data-base62-sha1="kRJl2Bosf58ikAqqHPZe3uZC6yO" width="689" height="389" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/2/923e5ee9090b2ce5ae4f379fe16f8c3292c6a026_2_689x389.png, https://discuss.pytorch.org/uploads/default/optimized/3X/9/2/923e5ee9090b2ce5ae4f379fe16f8c3292c6a026_2_1033x583.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/9/2/923e5ee9090b2ce5ae4f379fe16f8c3292c6a026.png 2x" data-dominant-color="2E2C2B"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">2</span><span class="informations">1299×733 95.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: JBoRu
cooked: <p>And i print the grad as follows:</p>
<pre><code class="lang-auto">---&gt;encoder:
--&gt;name: embedding.weight --&gt;max_grad: tensor(4.5225e-05, device='cuda:1') --&gt;min_grad: tensor(-3.6540e-05, device='cuda:1')
--&gt;name: gru.weight_ih_l0 --&gt;max_grad: tensor(0.0192, device='cuda:1') --&gt;min_grad: tensor(-0.0180, device='cuda:1')
--&gt;name: gru.weight_hh_l0 --&gt;max_grad: tensor(0.0023, device='cuda:1') --&gt;min_grad: tensor(-0.0025, device='cuda:1')
--&gt;name: gru.bias_ih_l0 --&gt;max_grad: tensor(0.0103, device='cuda:1') --&gt;min_grad: tensor(-0.0094, device='cuda:1')
--&gt;name: gru.bias_hh_l0 --&gt;max_grad: tensor(0.0061, device='cuda:1') --&gt;min_grad: tensor(-0.0049, device='cuda:1')
--&gt;name: gru.weight_ih_l0_reverse --&gt;max_grad: tensor(0.0164, device='cuda:1') --&gt;min_grad: tensor(-0.0191, device='cuda:1')
--&gt;name: gru.weight_hh_l0_reverse --&gt;max_grad: tensor(0.0020, device='cuda:1') --&gt;min_grad: tensor(-0.0021, device='cuda:1')
--&gt;name: gru.bias_ih_l0_reverse --&gt;max_grad: tensor(0.0077, device='cuda:1') --&gt;min_grad: tensor(-0.0071, device='cuda:1')
--&gt;name: gru.bias_hh_l0_reverse --&gt;max_grad: tensor(0.0042, device='cuda:1') --&gt;min_grad: tensor(-0.0040, device='cuda:1')
---&gt;decoder:
--&gt;name: embedding.weight --&gt;max_grad: tensor(4.5225e-05, device='cuda:1') --&gt;min_grad: tensor(-3.6540e-05, device='cuda:1')
--&gt;name: gru.weight_ih_l0 --&gt;max_grad: tensor(0.0208, device='cuda:1') --&gt;min_grad: tensor(-0.0201, device='cuda:1')
--&gt;name: gru.weight_hh_l0 --&gt;max_grad: tensor(0.0040, device='cuda:1') --&gt;min_grad: tensor(-0.0039, device='cuda:1')
--&gt;name: gru.bias_ih_l0 --&gt;max_grad: tensor(0.0061, device='cuda:1') --&gt;min_grad: tensor(-0.0059, device='cuda:1')
--&gt;name: gru.bias_hh_l0 --&gt;max_grad: tensor(0.0039, device='cuda:1') --&gt;min_grad: tensor(-0.0045, device='cuda:1')
--&gt;name: concat.weight --&gt;max_grad: tensor(0.0017, device='cuda:1') --&gt;min_grad: tensor(-0.0017, device='cuda:1')
--&gt;name: concat.bias --&gt;max_grad: tensor(0.0012, device='cuda:1') --&gt;min_grad: tensor(-0.0003, device='cuda:1')
--&gt;name: out.weight --&gt;max_grad: tensor(0.0440, device='cuda:1') --&gt;min_grad: tensor(-0.0055, device='cuda:1')
--&gt;name: out.bias --&gt;max_grad: tensor(0.0170, device='cuda:1') --&gt;min_grad: tensor(-0.0022, device='cuda:1')
</code></pre>
<p>The position of detect code as follows:</p>
<pre><code class="lang-auto">    # Perform backpropatation
    loss.backward()

    # Clip gradients: gradients are modified in place
    _ = nn.utils.clip_grad_norm_(encoder.parameters(), clip)
    _ = nn.utils.clip_grad_norm_(decoder.parameters(), clip)

    print("---&gt;encoder:")
    for name, params in encoder.named_parameters():
        print("--&gt;name:",name, "--&gt;max_grad:", params.grad.max(), "--&gt;min_grad:", params.grad.min())
    print("---&gt;decoder:")
    for name, params in decoder.named_parameters():
        print("--&gt;name:",name, "--&gt;max_grad:", params.grad.max(),"--&gt;min_grad:", params.grad.min())

    # Adjust model weights
    encoder_optimizer.step()
    decoder_optimizer.step()
</code></pre>

username: ptrblck
cooked: <p>Could you check, if you could possibly passing a zero or negative values to <code>torch.log</code>?<br>
This would create a NaN/Inf output and thus yield to the mentioned issue.</p>

username: JBoRu
cooked: <p>However, the inp is the softmax output, it’s value in (0,1). So, it’s value not problem.</p>

username: ptrblck
cooked: <p><code>softmax</code> can still return a zero, so it might be a problem:</p>
<pre><code class="lang-python">x = torch.tensor([[-100000000000., 1., 1., 1.]])
print(torch.log(torch.softmax(x, 1)))
&gt; tensor([[   -inf, -1.0986, -1.0986, -1.0986]])
</code></pre>

username: JBoRu
cooked: <p>Thanks,<br>
so using the F.softmax() and F.cross_entropy() can solve this problem. what’s the difference between them.And this maskCrossEntropy is the chatbot tutorial of pytorch. And how to modify it?</p>

username: ptrblck
cooked: <p><code>F.cross_entropy</code> expects logits, so you shouldn’t apply <code>softmax</code> in the input, as internally <code>F.log_softmax</code> will be used.</p>
<aside class="quote no-group" data-username="JBoRu" data-post="9" data-topic="78323">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jboru/48/30937_2.png" class="avatar"> JBoRu:</div>
<blockquote>
<p>what’s the difference between them</p>
</blockquote>
</aside>
<p><code>torch.log(torch.softmax())</code> should not be used, as it’s numerically less stable than <code>torch.log_softmax</code>.<br>
For the dummy inputs of my previous post you’ll get:</p>
<pre data-code-wrap="python"><code class="lang-python">print(torch.log_softmax(x, 1))
&gt; tensor([[-1.0000e+11, -1.0986e+00, -1.0986e+00, -1.0986e+00]])
</code></pre>
<p>instead if the <code>-inf</code> output.</p>
<aside class="quote no-group" data-username="JBoRu" data-post="9" data-topic="78323">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jboru/48/30937_2.png" class="avatar"> JBoRu:</div>
<blockquote>
<p>And this maskCrossEntropy is the chatbot tutorial of pytorch. And how to modify it?</p>
</blockquote>
</aside>
<p>Could you link to the line of code, which is using <code>torch.log(torch.softmax())</code>, please?</p>

username: JBoRu
cooked: <p>sorry, i can’t clearly show you the code detail, i just have screenshot and the code link, sorry again:<br>
The code source as follows:<br>
[<a href="https://github.com/pytorch/tutorials/blob/master/beginner_source/chatbot_tutorial.py#L892" class="inline-onebox" rel="noopener nofollow ugc">tutorials/beginner_source/chatbot_tutorial.py at main · pytorch/tutorials · GitHub</a>]<br>
And the code detail as follows:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/7/c720cf62d3ba734ae73ad271bc0bf8c63a2320c8.png" data-download-href="https://discuss.pytorch.org/uploads/default/c720cf62d3ba734ae73ad271bc0bf8c63a2320c8" title="2"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/7/c720cf62d3ba734ae73ad271bc0bf8c63a2320c8_2_690x384.png" alt="2" data-base62-sha1="spzjl5AvqyRUVHH9AL7c5oO9qdy" width="690" height="384" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/7/c720cf62d3ba734ae73ad271bc0bf8c63a2320c8_2_690x384.png, https://discuss.pytorch.org/uploads/default/optimized/3X/c/7/c720cf62d3ba734ae73ad271bc0bf8c63a2320c8_2_1035x576.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/c/7/c720cf62d3ba734ae73ad271bc0bf8c63a2320c8_2_1380x768.png 2x" data-dominant-color="141518"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">2</span><span class="informations">1533×855 104 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div> <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/c/fcacd1819f7fc7c1e5376ef7560b148e974db3d0.png" data-download-href="https://discuss.pytorch.org/uploads/default/fcacd1819f7fc7c1e5376ef7560b148e974db3d0" title="1"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/c/fcacd1819f7fc7c1e5376ef7560b148e974db3d0_2_690x227.png" alt="1" data-base62-sha1="A3gAdNdixybHCJKM1P3KieMI4Te" width="690" height="227" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/c/fcacd1819f7fc7c1e5376ef7560b148e974db3d0_2_690x227.png, https://discuss.pytorch.org/uploads/default/optimized/3X/f/c/fcacd1819f7fc7c1e5376ef7560b148e974db3d0_2_1035x340.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/f/c/fcacd1819f7fc7c1e5376ef7560b148e974db3d0_2_1380x454.png 2x" data-dominant-color="151417"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">1</span><span class="informations">1450×479 47 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>Thanks for the link. That’s generally not a good idea. Could you create an <a href="https://github.com/pytorch/tutorials/issues">issue</a> and describe the problem you ran into, please?</p>

username: JBoRu
cooked: <p>ok, i’ll try! Thanks for your help! This problem has trouble me 1 week. So, your help is very important ! Thanks again!</p>

username: ptrblck
cooked: <p>Sorry to hear that and thanks again for reporting this issue.</p>

username: SumNeuron
cooked: <p>I think I have something similar? Could you please take a look?</p>
<pre><code class="lang-auto">class TokenPositionSegmentEmbedding(nn.Module):
    def __init__(self,
        vocabulary_size:int,
        embedding_size:int,
        maximum_length:int,
    ):
        super(TokenPositionSegmentEmbedding, self).__init__()
        # embedding for the tokens
        self.token_embedding    = nn.Embedding(vocabulary_size, embedding_size)
        # embedding for corresponding position
        self.position_embedding = nn.Embedding(maximum_length, embedding_size)
        self.norm = nn.LayerNorm(embedding_size)

    def forward(self, tokens):
        sequence_length = tokens.size(1)
        position = torch.arange(sequence_length, dtype=torch.long).to(tokens.device)
        # (sequence_length,) -&gt; (batch_size, sequence_length)
        position = position.unsqueeze(0).expand_as(tokens)
        embedding = (
            self.token_embedding(tokens) + \
            self.position_embedding(position)
        )
        # eventually embedding is nan
        return self.norm(embedding)
</code></pre>

username: Francis_Shaw
cooked: <p><a class="mention" href="/u/sumneuron">@SumNeuron</a>  me too ,how do solve it?</p>