Label smoothing with CTCLoss

username: jinserk
cooked: <p>Does anybody know how to implement label smoothing (LS) with CTCLoss? I found a lot of articles about CrossEntropyLoss with Label smoothing, but nothing about CTCLoss.<br>
I’ve found a paper (<a href="https://arxiv.org/pdf/1711.02212.pdf" rel="noopener nofollow ugc">https://arxiv.org/pdf/1711.02212.pdf</a>) mentioning about the LS with CTCLoss, with an equivalent approach with KLDiv, but have no idea how to implement it with PyTorch. TIA!</p>
<p>PS:<br>
I implemented the CTCLoss with Label smoothing like:</p>
<pre><code class="lang-auto">class SmoothCTCLoss(_Loss):

    def __init__(self, num_classes, blank=0, weight=0.01):
        super().__init__(reduction='mean')
        self.weight = weight
        self.num_classes = num_classes

        self.ctc = nn.CTCLoss(reduction='mean', blank=blank, zero_infinity=True)
        self.kldiv = nn.KLDivLoss(reduction='batchmean')

    def forward(self, log_probs, targets, input_lengths, target_lengths):
        ctc_loss = self.ctc(log_probs, targets, input_lengths, target_lengths)

        kl_inp = log_probs.transpose(0, 1)
        kl_tar = torch.full_like(kl_inp, 1. / self.num_classes)
        kldiv_loss = self.kldiv(kl_inp, kl_tar)

        #print(ctc_loss, kldiv_loss)
        loss = (1. - self.weight) * ctc_loss + self.weight * kldiv_loss
        return loss
</code></pre>
<p>Can anybody confirm if this is correct or not? I also referred to this:<br>
</p><aside class="onebox allowlistedgeneric">
  <header class="source">
      <img src="https://leimao.github.io/favicon-32x32.png" class="site-icon" width="32" height="32">
      <a href="https://leimao.github.io/blog/Label-Smoothing/" target="_blank" rel="noopener nofollow ugc">Lei Mao's Log Book</a>
  </header>
  <article class="onebox-body">
    <img src="https://leimao.github.io/images/" class="thumbnail" width="" height="">

<h3><a href="https://leimao.github.io/blog/Label-Smoothing/" target="_blank" rel="noopener nofollow ugc">Label Smoothing</a></h3>

<p>Hello Underworld.</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


username: Abhilash_Srivastava
cooked: <ol>
<li>Your <code>ctc_loss</code> looks straightforward, so there shouldn’t be any issues.</li>
<li>In <code>kl_tar</code>:</li>
</ol>
<aside class="quote no-group" data-username="jinserk" data-post="1" data-topic="103392">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jinserk/48/1431_2.png" class="avatar"> jinserk:</div>
<blockquote>
<pre><code class="lang-auto">kl_inp = log_probs.transpose(0, 1)
kl_tar = torch.full_like(kl_inp, 1. / self.num_classes)
</code></pre>
</blockquote>
</aside>
<p>why are you using <code>kl_inp</code> (which uses input <code>log_probs</code>) for creating the <code>kl_tar</code>? Should it be dependent on the <code>targets</code> input instead?</p>

username: jinserk
cooked: <p>Thanks for your interest, <a class="mention" href="/u/abhilash_srivastava">@Abhilash_Srivastava</a>. I just borrow the <code>kl_inp</code> shape to generate <code>kl_tar</code> tensor, with a constant value of <code>1. / self.num_classes</code>.</p>

username: Abhilash_Srivastava
cooked: <p>Ohkk, got it!<br>
Are you seeing any issues now?</p>

username: jinserk
cooked: <p>No. I just want to make it sure my implementation is correct. It seems work as I expected, but I’m still testing it with multiple choice of label smoothing weights.</p>

username: Abhilash_Srivastava
cooked: <p>Cool! Looks fine to me.<br>
If label smoothening is bothering you, another way to test it is to change label smoothing to 1. ie: simply use one-hot representation with KL-Divergence loss. In this case, your loss values should match exactly the Cross-Entropy loss values.</p>

username: jinserk
cooked: <p>It’s good to know! Thank you for your comment!</p>

username: biG_mOutH_Fr0g
cooked: <p>just curious, why you want to transpose the logits as following?</p>
<pre><code>    kl_inp = log_probs.transpose(0, 1)
</code></pre>
<p>the shape of kl_inp should be something like (BatchSize,TimeStep, Channels) or something else?</p>