CTC loss with variable input_lengths produces NaN values

acobobby: <p>Hello all,</p>
<p>I am working with Pytorch 1.0.1 and I am using the CTC loss of PyTorch.</p>
<p>The code looks like the following one (I am working on GPU):</p>
<pre><code class="lang-auto">criterion = torch.nn.CTCLoss()
outs, (h,c) = lstm(input) # input is padded with zeros
outs = torch.nn.functional.log_softmax(outs, dim=2)
loss = criterion(outs.permute(1,0,2).contiguous(), y.to(device), lengths_input.to(device), lengths_target.to(device) )
</code></pre>
<p>I am permuting <code>outs</code> (FloatTensor) because I am working batch_first (N, T, C) while CTC requires (T, N, C).<br>
<code>y</code> contains the concatenation of the labels for each batch and is a LongTensor.<br>
<code>lengths_input</code> and <code>lengths_target</code> (LongTensors) have 1 dimension with N elements each representing the length of each input sequence in the batch and each target sequence in the labels.<br>
I am working with variable length sequences so the values of <code>lengths_input</code> may be different (and they often are) from one another.<br>
This setting causes the loss to become <code>nan</code> after few iterations.</p>
<p>However, if I fill <code>lengths_input</code> with all equal sizes (i.e. <code>outs.size(1)</code>) the loss decreases fine.<br>
This is the only change I made from one version to the other.</p>
<p>I was previously using PyTorch 1.0.0, then I read about this fixed <a href="https://github.com/pytorch/pytorch/pull/15798" rel="nofollow noopener">bug</a> and I updated to 1.0.1 but nothing changed.</p>
<p>As I wrote in the comment I am using padded inputs to obtain a single batch-tensor. Is this not compatible with variable sized lengths in the CTC loss? Am I missing something?</p>
<p>Thank you!</p>

acobobby: <p>Ops, I found this in the CTC docs:</p>
<blockquote>
<p>In order to use CuDNN, the following must be satisfied: <code>targets</code> must be in concatenated format, all <code>input_lengths</code> must be T. <code>blank</code>=0, <code>target_lengths</code> ≤256, the integer arguments must be of dtype <code>torch.int32</code>.</p>
<p>The regular implementation uses the (more common in PyTorch) torch.long dtype.</p>
</blockquote>
<p>So I guess I cannot use different values for <code>input_lengths</code> in CUDA, I will try on CPU. Anyway even with fixed lengths I was using LongTensor instead of <code>torch.int32</code> and no errors were raised. I don’t know if the result is still correct in this case.</p>

tom: <p>No that is for CuDNN, you’ll be OK if you use the “native” (PyTorch) version.<br>
Two caveats</p>
<ul>
<li>If you feed “invalid” (targets to long for input length) samples, you probably want the <code>zero_infinity</code> option from PyTorch nightlies. This will very likely be the source of your difficulties.</li>
<li>GPU CTC doesn’t like zero length targets currently (sorry, it’s on my to-do list).</li>
</ul>
<p>Best regards</p>
<p>Thomas</p>

acobobby: <p>Thank you for your reply!</p>
<p>I don’t have any zero length targets so this should not be a problem.</p>
<p>I did not understand exactly what do you mean by “invalid” sample: in my example above it happens if lengths_target[i] &gt;  lengths_input[i]  for some i?</p>
<p>I will check the <code>zero_infinity</code> option, thank you!</p>

tom: <p>The precise condition is target length + repeats &gt; input length. This has likelihood 0 and so the negative log likelihood (that ctc loss is) is infinite. Zeroing that will help there, but actually distort your score of used in validation.</p>
<p>Best regards</p>
<p>Thomas</p>

acobobby: <p>I see, this surely happens in my case,</p>
<p>thank you again.</p>

acobobby: <p>Just to keep you updated about this: I used the nightly version with <code>zero_infinity</code> flag set to true but I am still getting <code>nan</code> from a certain time step on.</p>
<p>What seems strange to me is that if I use a constant <code>lengths_input</code> (torch.full) this never happens.<br>
Of course in this way the length of the input stays the same or it is increased so I reduce the number of invalid samples, but by zeroing the inf loss I should get the same result.</p>
<p>I will continue to try to fix it and eventually let you know.</p>

tom: <p>If you can isolate the inputs that cause CTC or gradients to go wrong (before the inputs become NaN) I’d be most interested. So far no-one did that and it almost seems like for most people seeing NaNs the source is somewhere else.</p>
<p>Best regards</p>
<p>Thomas</p>

acobobby: <p>Ok, the problem is gone and the loss is decreasing fine. Maybe on the very first run after your suggestion I mistakenly executed the old script, I don’t know.<br>
Anyway on the subsequent runs I never faced the problem anymore, thank you!</p>

Baoguang_Shi: <p>I’ll just share my observations here. I am also seeing this NaN gradient issue in my training code. I tried to capture the inputs to CTC when gradients are NaN using backward hooks. The code is like</p>
<pre><code class="lang-auto">class CTCWrapper(nn.Module):
    def __init__(self):
        super().__init__()
        self.ctc = nn.CTCLoss(blank=0, reduction='none', zero_infinity=True)

    def forward(self, logits, logit_lengths, labels, label_lengths):
        def _backward_hook(grad):
            if not np.isfinite(grad.sum().item()):
                # save logits, logit_lengths, labels, label_lengths and grad
                torch.save(...) 
        log_probs = F.log_softmax(logits, 2)
        log_probs.register_hook(_backward_hook)
        return self.ctc(log_probs, labels, logit_lengths, label_lengths)
</code></pre>
<p>The variables are dumped successfully when NaN gradient is detected. The dumped <code>grad</code> variable (I’ll call it <code>dump_grad</code>) has NaN values while others (<code>logits</code>, <code>labels</code>) don’t. However, when I tried those inputs on CTCWrapper alone, the backward gradients (I’ll call it <code>offline_grad</code>) have no NaN values.</p>
<p>I further looked into <code>dump_grad</code> and <code>offline_grad</code>, they are only different where <code>dump_grad</code> is NaN. <code>dump_grad</code> is a [396, 93, 96] tensor (T,N,C layout). It has four NaNs in four different samples, all of them at the last time step and in the blank label, i.e. only <code>dump_grad[-1,:,0],</code> has NaN.</p>
<p>I don’t know where to go further so I have to stop here and use a workaround, like setting NaNs to zeros.</p>
<p>I’m using PyTorch 1.1 stable and CUDA 10.0.</p>
<p>The values that contain NaNs are attached. I can upload the dump if anyone is interested.</p>
<pre><code class="lang-auto">&gt;&gt; dump_grad[-1,:,0]

tensor([-1.1732e-05,  1.8528e-04,  1.8528e-04,  1.8528e-04,  1.8528e-04,
         1.8528e-04,  1.8528e-04,  1.8528e-04,  1.8528e-04,  1.8528e-04,
         1.8528e-04,  1.8528e-04,  1.8528e-04,  1.8528e-04,  1.8528e-04,
         1.8528e-04,  1.8528e-04,  1.8528e-04,  1.8528e-04,  1.8528e-04,
         1.8528e-04,  1.8528e-04,  1.8528e-04,  1.8528e-04,  1.8528e-04,
         1.8528e-04,  1.8528e-04,  1.8528e-04,  1.8528e-04,  1.8528e-04,
         1.8528e-04,  1.8528e-04,  1.8528e-04,  1.8528e-04,  1.8528e-04,
         1.8528e-04,  1.8528e-04,  1.8528e-04,  1.8528e-04,  1.8528e-04,
         1.8528e-04,  1.8528e-04,         nan,  1.8528e-04,  1.8528e-04,
         1.8528e-04,         nan,         nan,         nan,  1.8528e-04,
         1.8528e-04,  1.8528e-04,  1.8528e-04,  1.8528e-04,  1.8528e-04,
         1.8528e-04,  1.8528e-04,  1.8528e-04,  1.8528e-04,  1.8528e-04,
         1.8528e-04,  1.8528e-04,  1.8528e-04,  1.8528e-04,  1.8528e-04,
         1.8528e-04,  1.8528e-04,  1.8528e-04,  1.8528e-04,  1.8528e-04,
         1.8528e-04,  1.8528e-04,  1.8528e-04,  1.8528e-04,  1.8528e-04,
         1.8528e-04,  1.8528e-04,  1.8528e-04,  1.8528e-04,  1.8528e-04,
         1.8528e-04,  1.8528e-04,  1.8528e-04,  1.8528e-04,  1.8528e-04,
         1.8528e-04,  1.8528e-04,  1.8528e-04,  1.8528e-04,  1.8528e-04,
         1.8528e-04,  1.8528e-04,  1.8528e-04], device='cuda:0')
</code></pre>
<pre><code class="lang-auto">&gt;&gt; offline_grad[-1,:,0]

tensor([-1.1732e-05,  1.8528e-04,  1.8528e-04,  1.8528e-04,  1.8528e-04,
         1.8528e-04,  1.8528e-04,  1.8528e-04,  1.8528e-04,  1.8528e-04,
         1.8528e-04,  1.8528e-04,  1.8528e-04,  1.8528e-04,  1.8528e-04,
         1.8528e-04,  1.8528e-04,  1.8528e-04,  1.8528e-04,  1.8528e-04,
         1.8528e-04,  1.8528e-04,  1.8528e-04,  1.8528e-04,  1.8528e-04,
         1.8528e-04,  1.8528e-04,  1.8528e-04,  1.8528e-04,  1.8528e-04,
         1.8528e-04,  1.8528e-04,  1.8528e-04,  1.8528e-04,  1.8528e-04,
         1.8528e-04,  1.8528e-04,  1.8528e-04,  1.8528e-04,  1.8528e-04,
         1.8528e-04,  1.8528e-04,  1.8528e-04,  1.8528e-04,  1.8528e-04,
         1.8528e-04,  1.8528e-04,  1.8528e-04,  1.8528e-04,  1.8528e-04,
         1.8528e-04,  1.8528e-04,  1.8528e-04,  1.8528e-04,  1.8528e-04,
         1.8528e-04,  1.8528e-04,  1.8528e-04,  1.8528e-04,  1.8528e-04,
         1.8528e-04,  1.8528e-04,  1.8528e-04,  1.8528e-04,  1.8528e-04,
         1.8528e-04,  1.8528e-04,  1.8528e-04,  1.8528e-04,  1.8528e-04,
         1.8528e-04,  1.8528e-04,  1.8528e-04,  1.8528e-04,  1.8528e-04,
         1.8528e-04,  1.8528e-04,  1.8528e-04,  1.8528e-04,  1.8528e-04,
         1.8528e-04,  1.8528e-04,  1.8528e-04,  1.8528e-04,  1.8528e-04,
         1.8528e-04,  1.8528e-04,  1.8528e-04,  1.8528e-04,  1.8528e-04,
         1.8528e-04,  1.8528e-04,  1.8528e-04], device='cuda:0')
</code></pre>

tom: <p>There was a <a href="https://github.com/pytorch/pytorch/pull/21244" rel="nofollow noopener">bug when there was uninitialized memory in unused parts of the input gradient</a>. This is still in PyTorch releases including 1.1. Thanks to <a class="mention" href="/u/yqwangustc">@yqwangustc</a>’s fix  the nightlies / master would work better.</p>
<p>Best regards</p>
<p>Thomas</p>