Getting NaN values in backward pass (triplet loss function)

teotfw: <p>Hi there, I’m trying to reimplement <a href="https://github.com/kuanghuei/SCAN" rel="noopener nofollow ugc">SCAN</a>.</p>
<p>My code is <a href="https://github.com/hthoai/image-text-matching/blob/dev/lib/models/encoder.py" rel="noopener nofollow ugc">here</a>.</p>
<p>The loss didn’t change after each iteration. When I called <code>backward</code> to calculate the gradients, it happened to be <code>NaN</code>.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/0/d0d874032bbb0de168bb1128cd366efbb24d9a14.png" data-download-href="https://discuss.pytorch.org/uploads/default/d0d874032bbb0de168bb1128cd366efbb24d9a14" title="158368423_148142287084217_1092993663152853717_n"><img src="https://discuss.pytorch.org/uploads/default/original/3X/d/0/d0d874032bbb0de168bb1128cd366efbb24d9a14.png" alt="158368423_148142287084217_1092993663152853717_n" data-base62-sha1="tNx3SMwQlEoH9UY8cw6QPQ40DVG" width="690" height="230" data-dominant-color="2E323A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">158368423_148142287084217_1092993663152853717_n</span><span class="informations">744×249 13.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Do you guys have any suggestions?</p>

crcrpar: <p>Hi,</p>
<p>I’d try <a href="https://pytorch.org/docs/stable/autograd.html#torch.autograd.detect_anomaly" class="inline-onebox" rel="noopener nofollow ugc">Automatic differentiation package - torch.autograd — PyTorch 1.8.0 documentation</a> to which backward part emit nan first.</p>

teotfw: <p>It showed <code>Function 'DivBackward0' returned nan values in its 0th output.</code></p>
<p>I’m trying to figure out but found nothing :&lt;</p>

teotfw: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Sir please give me a hand!</p>

crcrpar: <p>Oh, it’s a little bit hard to identify which layer.</p>
<p><code>nan</code> can occur for some reasons but mainly it’s oftentimes 0/inf related maths.<br>
For example, in SCAN code (<a href="https://github.com/kuanghuei/SCAN/blob/master/model.py" class="inline-onebox" rel="noopener nofollow ugc">SCAN/model.py at master · kuanghuei/SCAN · GitHub</a>), nan and inf can happen in forward of <code>l1norm</code> and <code>l2norm</code>. So, I think it’s better to investigate where those bad values are generated, for example, by using forward/backward hooks.</p>
<p>I think one implementation of a backward hook with module name is as follows.<br>
Also, forward hook with similar functionalities will be helpful.</p>
<pre><code class="lang-python"># backward hook with module name
def get_backward_hook(module_name: str):
    
    class BackwardHook:
        name: str
            
        def __init__(self, name):
            self.name = name
            
        def __call__(self, module, grad_input, grad_output):
            for i, g_in in enumerate(grad_input):
                print(module_name, torch.any(torch.isnan(g_in)))
                if torch.any(torch.isnan(g_in)):
                    print(f"{module_name}'s {i}th input gradient is nan")
            for i, g_out in enumerate(grad_output):
                if torch.any(torch.isnan(g_out)):
                    print(f"{module_name}'s {i}th output gradient is nan")
                
    return BackwardHook(module_name)

...

model = SomeCoolModel(...)
for name, module in model.named_modules():
    module.register_full_backward_hook(get_backward_hook(name))
</code></pre>
<p>Hope this</p>

teotfw: <p>Thank you for your generous response. I found that there is something wrong in <a href="https://github.com/hthoai/image-text-matching/blob/320db98b5cf5ffa7718172fcb66060584a366fcf/lib/models/scan.py#L93" rel="noopener nofollow ugc">this line of code</a> but I can’t figure out. Could you please have a look at it?!</p>
<pre><code class="lang-auto">def forward(self, images: Tensor, captions: Tensor, cap_lens: Tensor) -&gt; Tensor:
        """
        Parameters
        ----------
        images:      (batch, regions, hidden_dim)
        captions:    (batch, cap_lenghts, hidden_dim)
        cap_lengths: (batch)
        Returns
        -------
        sim_scores: (batch cap, batch image)
        """
        # Projecting the image/caption to embedding dimensions
        img_embs = self.img_enc(images)
        cap_embs = self.txt_enc(captions, cap_lens)
        # -&gt; img_embs: (batch, regions, hidden)
        # -&gt; cap_embs: (batch, max_seq_len, hidden)

        # Compute similarity between each region and each token
        # cos(u, v) = (u @ v) / (||u|| * ||v||)
        #           = (u / ||u||) @ (v / ||v||)
        img_embs = F.normalize(img_embs, dim=-1)
        cap_embs = F.normalize(cap_embs, dim=-1)

        img_embs.unsqueeze_(0)
        cap_embs.unsqueeze_(1)
        # -&gt; img_embs: (1, batch, regions, hidden)
        # -&gt; cap_embs: (batch, 1, max_seq_len, hidden)

        # After normalizing: cos(u, v) = u @ v
        sim_token_region = torch.matmul(cap_embs, img_embs.transpose(-1, -2))
        # -&gt; sim_token_region: (batch, batch, max_seq_len, regions)

        att_score = self.lambda_softmax * sim_token_region
        # -&gt; att_score: (batch, batch, max_seq_len, regions)

        # Create mask from caption length
        # torch.arange(max_seq_len_SIZE).expand(batch_SIZE, max_seq_len_SIZE)
        padding_mask = torch.arange(cap_embs.size(2)).expand(
            cap_embs.size(0), cap_embs.size(2)
        ) &gt;= cap_lens.unsqueeze(1)
        # padding_mask: (batch, max_seq_len)
        
        padding_mask = padding_mask.unsqueeze(-1).unsqueeze(1).to(cap_embs.device)
        # -&gt; padding_mask: (batch, 1, max_seq_len, 1)

        # mask score of padding tokens
        att_score.data.masked_fill_(padding_mask, -float('inf'))

        # softmax along regions axis
        attention_weights = F.softmax(att_score, dim=-1)
        # -&gt; attention_weights: (batch, batch, max_seq_len, regions)

        # Calculate weighted sum of regions -&gt; attended image vectors
        attention = torch.matmul(attention_weights, img_embs)
        # -&gt; attention: (batch, batch, max_seq_len, hidden_dim)

        # Calculate the importance of each attended image vector
        # w.r.t each token of sentence
        r = F.cosine_similarity(cap_embs, attention, dim=-1)
        # -&gt; r: (batch, batch, max_seq_len)

        # Zeros similarity of padding tokens
        r[r.isnan()] = 0

        # Calculate similarity of each caption and each image by averaging all tokens in a caption.
        sim_scores = r.sum(dim=-1) / cap_lens.view(-1, 1).to(cap_embs.device)
        # -&gt; sim_score: (batch, batch)

        return sim_scores
</code></pre>

crcrpar: <p>Did you find that in forward computation? or backward?</p>
<p>Either way, I’m not sure but I’d check the values of <code>img_embs</code> and backward input values.</p>

xandery-geek: <p>I met the ‘nan’ because of empty input tensor.</p>
<p>A simple example is :</p>
<pre><code class="lang-auto">impot torch.nn as nn

loss = nn.TripletMarginLoss(margin=1.0, p=2)

a = torch.randn(0, 100)
p = torch.randn(0, 100)
n = torch.randn(0, 100)

ret = loss(a, p, n)  # the return value is `tensor(nan)`
</code></pre>