CTCLoss returns negative loss after some batches

username: bloemy
cooked: <p>Hey !<br>
I’m using CTCLoss to implement a speech recognition recurrent neural network.</p>
<pre><code class="lang-auto">       def forward(self, x):

        hidden = None
        rnn_output, hidden = self.recurrent(x, hidden)

        rnn_output_flat = rnn_output.view(-1, self.hidden_size)

        lin_output = self.output(rnn_output_flat)

        output_flat = self.softmax(lin_output)

        output = output_flat.view(rnn_output.size(0), rnn_output.size(1), output_flat.size(1))

        return output
</code></pre>
<p>This is my forward function, where self.softmax = nn.LogSoftmax, self.output = nn.Linear(…), and self.recurrent = nn.RNN(…)</p>
<p>Now, as input I have an audio files which I have pre-processed using MFCC feature extraction, and encoded using a simple alphabet encoding. I pad my inputs per batch in my DataLoader.</p>
<p>Training seems to work: the loss starts at about 30 for my first input, and then gradually goes down after every batch. But after 7 or 8 batches, I start getting losses, in the [-1, 0] range. At that point, obviously, training doesn’t actually seem to improve the model at all anymore.</p>
<p>I was wondering if I’m missing something obvious here. I’ve been scratching my head for a while now…</p>
<p>Thanks a lot for your help !</p>

username: tom
cooked: <p>Hi,</p>
<p>by definition, a negative log likelihood cannot be negative, and I’ve not seen CTC loss return negative values for valid inputs. Based on that, can you double-check that your inputs are valid (e.g. no blank labels in the target)?</p>
<p>Best regards</p>
<p>Thomas</p>

username: bloemy
cooked: <p>Hi Thomas,<br>
thanks for your fast answer.<br>
I do actually think that I’m using the blank labels in target when I’m padding. What value should I pad by batches (input &amp; ouput) with to get the best results?</p>
<p>Thanks a lot again.</p>

username: tom
cooked: <p>The CTC loss will not look at inputs beyond the target_length you pass, but the first target_length inputs need to be non-blank.</p>

username: bloemy
cooked: <p>Awesome Tom, this was exactly my issue. I just fixed it. Thanks a lot, really appreciate your quick help !</p>

username: Ceday
cooked: <p>I get negative losses out of every 4-5K samples, they are really shorter than others. But input/target lenghts are OK. However cudnnctcloss gives positive values, so I switched them with deterministic flag setted to true.<br>
I have a json file for those inputs if you want to investigate the issue, but apparently it doesnt let me attach them here. (this is with 1.3.0)</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://gofile.io/?c=tfTryc">
  <header class="source">

      <a href="https://gofile.io/?c=tfTryc" target="_blank" rel="noopener nofollow ugc">gofile.io</a>
  </header>

  <article class="onebox-body">
    <img width="" height="" src="https://gofile.io/dist/img/logo-small-og.png" class="thumbnail">

<h3><a href="https://gofile.io/?c=tfTryc" target="_blank" rel="noopener nofollow ugc">Gofile - Free Unlimited File Sharing and Storage</a></h3>

  <p>Gofile is a free, secure file sharing and storage platform. With unlimited bandwidth and storage, you can easily store and share files of any type without any limits. Our advanced features, such as CDN support and password protection, make Gofile the...</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/d/3dd87d8d4928d748ee85ba8b7e5a3102e7078234.png" data-download-href="https://discuss.pytorch.org/uploads/default/3dd87d8d4928d748ee85ba8b7e5a3102e7078234" title="ctcloss"><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/d/3dd87d8d4928d748ee85ba8b7e5a3102e7078234.png" alt="ctcloss" data-base62-sha1="8P6Xap6TuG456z1a9tTPaFEqwBK" width="690" height="130" data-dominant-color="F2F3F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">ctcloss</span><span class="informations">1219×230 18 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<pre><code class="lang-auto">import torch
import codecs
import json
import numpy as np

def dump_ctc_inputs(file, log_probs, target_lengths, targets, input_lengths):
    inputs = (log_probs.detach().cpu().numpy().tolist(), target_lengths.cpu().numpy().tolist(), targets.cpu().numpy().tolist(), input_lengths.cpu().numpy().tolist())
   
    with codecs.open(file, "w", "utf-8") as streamjson:
                    json.dump(inputs, streamjson, indent=4, ensure_ascii=False)

def read_ctc_inputs(file):
    with codecs.open(file, "r", "utf-8") as stream:
        lines = stream.read()
        inputsloaded = json.loads(lines)
        log_probs, target_lengths, targets, input_lengths = inputsloaded
        log_probs = torch.from_numpy(np.array(log_probs)).float()
        target_lengths = torch.from_numpy(np.array(target_lengths)).int()
        targets = torch.from_numpy(np.array(targets)).int()
        input_lengths = torch.from_numpy(np.array(input_lengths)).int()
        return log_probs, targets, input_lengths, target_lengths

ctc_loss = torch.nn.CTCLoss(reduction='none')
file = "C:\\1391713.json"
log_probs, targets, input_lengths, target_lengths = read_ctc_inputs(file)
log_probs.requires_grad_().cuda()
loss = ctc_loss(log_probs, targets, input_lengths, target_lengths)
loss2 = torch.nn.functional.ctc_loss(log_probs.detach().requires_grad_().cuda(), targets[0][0:6], input_lengths, target_lengths, reduction='none', zero_infinity=True)
</code></pre>

username: barakb
cooked: <p>Hi just to verify I understand what you mean, let’s say my target is:<br>
“hello world”<br>
Target’s length is 10 and not 11?</p>

username: tom
cooked: <p>“Hello world” has 11 chars including the space, so it would be 11. “blank” is a special “output-only” character that means “nothing”, it is not space. The article <a href="https://distill.pub/2017/ctc/" class="inline-onebox" rel="noopener nofollow ugc">Sequence Modeling with CTC</a> has a good overview of how CTC works under the hood.</p>
<p>Best regards</p>
<p>Thomas</p>

username: barakb
cooked: <p>Thanks, if I may I have one more question, in this code:</p><aside class="onebox githubblob" data-onebox-src="https://github.com/ASR-project/Multilingual-PR/blob/main/models/BaseModule.py#L52">
  <header class="source">

      <a href="https://github.com/ASR-project/Multilingual-PR/blob/main/models/BaseModule.py#L52" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/ASR-project/Multilingual-PR/blob/main/models/BaseModule.py#L52" target="_blank" rel="noopener nofollow ugc">ASR-project/Multilingual-PR/blob/main/models/BaseModule.py#L52</a></h4>



    <pre class="onebox"><code class="lang-py">
      <ol class="start lines" start="42" style="counter-reset: li-counter 41 ;">
          <li>    unk_token=network_param.unk_token,</li>
          <li>    pad_token=network_param.pad_token,</li>
          <li>    word_delimiter_token=network_param.word_delimiter_token,</li>
          <li>    do_phonemize=False,</li>
          <li>    return_attention_mask=False,</li>
          <li>)</li>
          <li>
          </li>
<li>network_param.vocab_size = self.phonemes_tokenizer.vocab_size</li>
          <li>
          </li>
<li># Loss function</li>
          <li class="selected">self.loss = nn.CTCLoss(</li>
          <li>    blank=self.phonemes_tokenizer.encoder[network_param.word_delimiter_token]</li>
          <li>)</li>
          <li>
          </li>
<li># Feature_extractor</li>
          <li>feature_extractor = Wav2Vec2FeatureExtractor(</li>
          <li>    feature_size=1,</li>
          <li>    sampling_rate=16000,</li>
          <li>    padding_value=0.0,</li>
          <li>    do_normalize=True,</li>
          <li>    return_attention_mask=False,</li>
      </ol>
    </code></pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<p>
They declare blank as space(i.e space between words), as I understand from your comment, it’s a mistake, the blank should be a separate token?</p>

username: SuperTaurus
cooked: <p>The “blank label” means exactly the “blank” itself, which namely means that it can’t represent any specific characters like space or enter and so on. The blank label is merely a placeholder for CTCLoss to be calculated correctly. Thus, naturally the space can’t be regarded as the blank label. CTCLoss needs the placeholder to separate different predicted characters so the blank label is essential.</p>
<p>Hope it helps</p>