Softmax/log_softmax in CTC loss

discort: <p>The docs to suggest using of logarithmized probabilities for an input of CTCLoss<br>
<a href="https://pytorch.org/docs/stable/generated/torch.nn.CTCLoss.html#torch.nn.CTCLoss" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.nn.CTCLoss.html#torch.nn.CTCLoss</a></p>
<p>However, there is example in pytorch repo (the DeepSpeech model) where <code>softmax</code> func (instead of <code>log_softmax</code>) is used just for evaluation but not training.</p><aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/blob/22a34bcf4e5eaa348f0117c414c3dd760ec64b13/benchmarks/functional_autograd_benchmark/torchaudio_models.py#L135-L140" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/22a34bcf4e5eaa348f0117c414c3dd760ec64b13/benchmarks/functional_autograd_benchmark/torchaudio_models.py#L135-L140" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/22a34bcf4e5eaa348f0117c414c3dd760ec64b13/benchmarks/functional_autograd_benchmark/torchaudio_models.py#L135-L140</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="135" style="counter-reset: li-counter 134 ;">
<li>class InferenceBatchSoftmax(nn.Module):</li><li>    def forward(self, input_):</li><li>        if not self.training:</li><li>            return F.softmax(input_, dim=-1)</li><li>        else:</li><li>            return input_</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>So I want to clarify what should I use for training and evaluation in CTCLoss:</p>
<ul>
<li>
<code>softmax/log_softmax</code> for train/eval?</li>
<li>identity for the training and <code>softmax/log_softmax</code> for eval like in example I shared above?</li>
</ul>

ZdsAlpha: <p>As far as I know, for training you need log_softmax. For inference you can just do argmax. But using argmax might only give you Top-1 accuracy. If you use softmax and get top 5 scores you can get Top-5 accuracy. I think DeepSpeech model does something similar.</p>

discort: <p>Thank you for the reply.</p>
<p>So for the training I need to use <code>log_softmax</code> it’s clear now. For the inference I can use <code>softmax</code> to get top <code>k</code> scores.</p>
<p>What isn’t clear is that why <code>DeepSpeech</code> implementation is not using <code>log_softmax</code> in the repo? I suppose there should be an explicit call of <code>log_softmax</code> in the model definition or the model calling, right? Or did I miss something?</p>
<p>model definition</p><aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/blob/22a34bcf4e5eaa348f0117c414c3dd760ec64b13/benchmarks/functional_autograd_benchmark/torchaudio_models.py#L193-L245" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/22a34bcf4e5eaa348f0117c414c3dd760ec64b13/benchmarks/functional_autograd_benchmark/torchaudio_models.py#L193-L245" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/22a34bcf4e5eaa348f0117c414c3dd760ec64b13/benchmarks/functional_autograd_benchmark/torchaudio_models.py#L193-L245</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="193" style="counter-reset: li-counter 192 ;">
<li>class DeepSpeech(nn.Module):</li><li>    def __init__(self, rnn_type, labels, rnn_hidden_size, nb_layers, audio_conf,</li><li>                 bidirectional, context=20):</li><li>        super(DeepSpeech, self).__init__()</li><li></li><li>        self.hidden_size = rnn_hidden_size</li><li>        self.hidden_layers = nb_layers</li><li>        self.rnn_type = rnn_type</li><li>        self.audio_conf = audio_conf</li><li>        self.labels = labels</li><li>        self.bidirectional = bidirectional</li><li></li><li>        sample_rate = self.audio_conf["sample_rate"]</li><li>        window_size = self.audio_conf["window_size"]</li><li>        num_classes = len(self.labels)</li><li></li><li>        self.conv = MaskConv(nn.Sequential(</li><li>            nn.Conv2d(1, 32, kernel_size=(41, 11), stride=(2, 2), padding=(20, 5)),</li><li>            nn.BatchNorm2d(32),</li><li>            nn.Hardtanh(0, 20, inplace=True),</li>
</ol></code></pre>

  This file has been truncated. <a href="https://github.com/pytorch/pytorch/blob/22a34bcf4e5eaa348f0117c414c3dd760ec64b13/benchmarks/functional_autograd_benchmark/torchaudio_models.py#L193-L245" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>model calling</p><aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/blob/22a34bcf4e5eaa348f0117c414c3dd760ec64b13/benchmarks/functional_autograd_benchmark/audio_text_models.py#L51-L63" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/22a34bcf4e5eaa348f0117c414c3dd760ec64b13/benchmarks/functional_autograd_benchmark/audio_text_models.py#L51-L63" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/22a34bcf4e5eaa348f0117c414c3dd760ec64b13/benchmarks/functional_autograd_benchmark/audio_text_models.py#L51-L63</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="51" style="counter-reset: li-counter 50 ;">
<li>model = models.DeepSpeech(rnn_type=nn.LSTM, labels=labels, rnn_hidden_size=1024, nb_layers=5,</li><li>                          audio_conf=audio_conf, bidirectional=True)</li><li>model = model.to(device)</li><li>criterion = nn.CTCLoss()</li><li>params, names = extract_weights(model)</li><li></li><li>def forward(*new_params: Tensor) -&gt; Tensor:</li><li>    load_weights(model, names, new_params)</li><li>    out, out_sizes = model(inputs, inputs_sizes)</li><li>    out = out.transpose(0, 1)  # For ctc loss</li><li></li><li>    loss = criterion(out, targets, out_sizes, targets_sizes)</li><li>    return loss</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>