Can CTCLoss go down to zero?

jleobernard: <p>Hello,<br>
I’m struggling  while trying to implement <a href="https://www.vision.rwth-aachen.de/media/papers/MDLSTM_final.pdf" rel="noopener nofollow ugc">this paper</a>. After some epochs the loss stops going down but my network only produces blanks. I’ve seen a lot of posts on the forum concerning this issue and most of the time the problem resulted from a wrong understanding of the way CTCLoss works. So I tried to make a minimal example to see where my code went wrong.</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

nn.CTCLoss(blank=0, zero_infinity=True, reduction=none)

predicted = torch.tensor(
[
  [
    [0., 1., 0., 0., 0.],
    [0., 0., 0., 1., 0.]
  ]
]
).detach().requires_grad_()

expected = torch.tensor([
[1.,3.]
], dtype=torch.long)

predicted_lengths = torch.tensor(predicted.shape[0] * [predicted.shape[1]])
ctc_loss(predicted.permute(1, 0, 2).log_softmax(2), expected, predicted_lengths, torch.tensor([2]))
</code></pre>
<p>and it returns</p>
<pre><code class="lang-auto">tensor([1.8097], grad_fn=&lt;SWhereBackward&gt;)
</code></pre>
<p>I was expecting it to be zero as the prediction matches the target perfectly so if someone could explain why the loss can never be zero or where my mistake is I would really appreciate it <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>Thanks</p>

tom: <p>The inputs of log_softmax are in log-space, so you’d need to take the log (-inf and 0 instead of 0 and 1 respectively).</p>

jleobernard: <p>It does work on my small example. I’ll try it on my target network and let you know.<br>
Thanks for the help !</p>

jleobernard: <p>I actually have a hard time understanding why log should be called before log_softmax because as the description of CTCLoss states :</p>
<blockquote>
<p>Blockquote<br>
Log_probs: Tensor of size (T, N, C)… The logarithmized probabilities of the outputs (e.g. obtained with <a href="https://pytorch.org/docs/stable/nn.functional.html#torch.nn.functional.log_softmax" rel="noopener nofollow ugc"><code>torch.nn.functional.log_softmax()</code></a>).</p>
</blockquote>
<p>So I was expecting the call to log_softmax to do the log on its own (as described in the <a href="https://pytorch.org/docs/stable/nn.functional.html#torch.nn.functional.log_softmax" rel="noopener nofollow ugc">doc</a>).</p>
<p>Either way I tried calling .log() on my actual outputs and not this small example and the training produced the same error (predicting only blanks) so I’m back to square one.</p>

SimonW: <p>It’s not about logsoftmax vs softmax. Log softmax means softmax first, then log. But softmax expect logits, not probabilities.</p>

jleobernard: <p>Thank you for your answer <a class="mention" href="/u/simonw">@SimonW</a>.<br>
It means that my training problem does not come from the way I use CTCLoss then as I think I used it right in the first place according to your description. The inputs of the log_softmax came from my last nn.Linear layer.</p>
<p>In the model I have (only kept the relevant parts):</p>
<pre><code class="lang-auto">...
self.dense = to_best_device(nn.Linear(in_features=50, out_features=len(characters)))
...
def forward(self, x):
   ...
   return self.dense(x)
</code></pre>
<p>And in the training loop I have:</p>
<pre><code class="lang-auto">    for i, batch_data in enumerate(dataloader):
        data, labels = batch_data
        optimizer.zero_grad()
        outputs = model(data)
        outputs = outputs.permute(1, 0, 2) # Because I have (batch_size, sequence_length, nb_features)
        bs = len(data)
        curr_loss = loss(outputs.log_softmax(2), labels, bs * [outputs.shape[0]], [get_sentence_length(label) for label in labels])
        curr_loss.backward()
        optimizer.step()
</code></pre>