What is the recommended softmax function?

username: YuhskeHujisaki
cooked: <p>There are nn.Softmax, torch.nn.functional.softmax and torch.nn.functional.log_softmax, but what is your best recommendation? And what’s the difference between them?</p>

username: Matias_Vasquez
cooked: <h2><a name="nnsoftmax-vs-nnfunctionalsoftmax-1" class="anchor" href="#nnsoftmax-vs-nnfunctionalsoftmax-1"></a>nn.Softmax vs nn.functional.softmax</h2>
<p>If you take a look at the source code for <strong>nn.Softmax</strong>, then you can see that in the forward method it actually calls <strong>nn.functiona.softmax</strong> (functional is imported as F then it is written as F.softmax).</p>
<aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/3ee863cb7c07699b94e36715a82a25744560f2d9/torch/nn/modules/activation.py#L1373-L1374">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/3ee863cb7c07699b94e36715a82a25744560f2d9/torch/nn/modules/activation.py#L1373-L1374" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/3ee863cb7c07699b94e36715a82a25744560f2d9/torch/nn/modules/activation.py#L1373-L1374" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/3ee863cb7c07699b94e36715a82a25744560f2d9/torch/nn/modules/activation.py#L1373-L1374</a></h4>



    <pre class="onebox"><code class="lang-py">
      <ol class="start lines" start="1373" style="counter-reset: li-counter 1372 ;">
          <li>def forward(self, input: Tensor) -&gt; Tensor:</li>
          <li>    return F.softmax(input, self.dim, _stacklevel=5)</li>
      </ol>
    </code></pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>So it is the same and if you want to do some functional stuff, then you can use this way.</p>
<p>However, if you want it to be an actual layer and not just a functional method, then I would recommend defining a layer with the constructor. If you use functional instead of a layer at the forward method of your architecture, then if someone tries to extract the children and put them in a container (e.g. <strong>nn.Sequential</strong>), the functional stuff will not be done. If this is not clear then I can give you an example.</p>
<hr>
<h2><a name="softmax-vs-logsoftmax-2" class="anchor" href="#softmax-vs-logsoftmax-2"></a>Softmax vs LogSoftmax</h2>
<p>As you can see, both activation functions are the same, only with a <strong>log</strong>.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/9/c/9c1ebd7a599ad57323d6f724d286c477651fb27c.png" alt="image" data-base62-sha1="mh6lrsDahIZGj7hHDN3jq87naYk" width="249" height="71"><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/7/0/701eee1f1a1598817c22a1d9683f16ecfc47e39c.png" alt="image" data-base62-sha1="fZRK8VglkbCMC6SRq9pcKL3mhxW" width="329" height="78"></p>
<p>As you might already know, the result of <strong>softmax</strong> are probabilities between 0 and 1. This is something useful for us to understand.</p>
<h3><a name="log-3" class="anchor" href="#log-3"></a>Log</h3>
<p>If you plot the values between 0 and 1 for <strong>log</strong>, you get the following plot. Here, values closer to 0 are heavily penalized.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/0/9/0980a637dcd0ea237197aae5882a8067aeff3851.png" alt="image" data-base62-sha1="1m3VtDTmKuiGvb5EYH9vm5YPy9j" width="370" height="248"></p>
<h3><a name="example-4" class="anchor" href="#example-4"></a>Example</h3>
<p>We can look at a little example.</p>
<pre data-code-wrap="python"><code class="lang-python">soft = torch.nn.Softmax(dim=0)
log_soft = torch.nn.LogSoftmax(dim=0)

x = torch.rand(3)

print(x)
#tensor([0.6848, 0.1505, 0.5071])
</code></pre>
<p>If we print the softmax values, we can understand that the first value is about 40% sure to be the correct class. The third class is very close but still a little bit smaller. The second class is much lower, thus less likely to be the correct one.</p>
<pre data-code-wrap="python"><code class="lang-python">print(soft(x))
#tensor([0.4127, 0.2418, 0.3455])
</code></pre>
<p>If we now print the logsoft values, we can sort of understand from the graph above that values closer to 0 are more likely to be the correct class. But other than, it will be difficult for us to determine how good the estimate is.</p>
<pre data-code-wrap="python"><code class="lang-python">print(log_soft(x))
#tensor([-0.8851, -1.4195, -1.0628])
</code></pre>
<h2><a name="which-one-should-you-use-5" class="anchor" href="#which-one-should-you-use-5"></a>Which one should you use</h2>
<p><a href="https://stats.stackexchange.com/a/176563" rel="noopener nofollow ugc">Here</a> is more information on as to why the <strong>log</strong> version might behave better for improved numerical performance and gradient optimization.</p>
<p><a href="https://datascience.stackexchange.com/a/40719" rel="noopener nofollow ugc">Here</a> is another answer on the topic.</p>
<p>But I would suggest trying different approaches and see which works better for your problem. There is not going to be a single function that will be the optimal solution for every problem.</p>
<p>I personally like using <strong>CrossEntropyLoss</strong>, which makes use of <strong>LogSoftmax</strong>. But I think the best thing is to try and see what works best.</p>
<p>Hope this helps <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=12" title=":smile:" class="emoji" alt=":smile:" loading="lazy" width="20" height="20"></p>