Which activation function for multi-class classification gives true probability?

laro: <p>I’m wondering which activation function for multi class classification problem, give true probability.</p>
<p>According to:</p>
<aside class="onebox stackexchange" data-onebox-src="https://ai.stackexchange.com/questions/37889/are-softmax-outputs-of-classifiers-true-probabilities">
  <header class="source">

      <a href="https://ai.stackexchange.com/questions/37889/are-softmax-outputs-of-classifiers-true-probabilities" target="_blank" rel="noopener nofollow ugc">ai.stackexchange.com</a>
  </header>

  <article class="onebox-body">
      <a href="https://ai.stackexchange.com/users/62466/snehal-patel" target="_blank" rel="noopener nofollow ugc">
    <img alt="Snehal Patel" src="https://lh3.googleusercontent.com/a-/AOh14Gg8FPkPrDJ6ppvEo_oMsRobnMxCg-rC31AmUyp8=k-s256" class="thumbnail onebox-avatar" width="256" height="256">
  </a>

<h4>
  <a href="https://ai.stackexchange.com/questions/37889/are-softmax-outputs-of-classifiers-true-probabilities" target="_blank" rel="noopener nofollow ugc">Are softmax outputs of classifiers true probabilities?</a>
</h4>

<div class="tags">
  <strong>activation-functions, probability, softmax, probability-theory</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://ai.stackexchange.com/users/62466/snehal-patel" target="_blank" rel="noopener nofollow ugc">
    Snehal Patel
  </a>
  on <a href="https://ai.stackexchange.com/questions/37889/are-softmax-outputs-of-classifiers-true-probabilities" target="_blank" rel="noopener nofollow ugc">07:11PM - 14 Nov 22 UTC</a>
</div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>it seems that the output probability (for <strong>softmax</strong>) doesn’t represent the true probability.</p>
<p>Is this the same when we use <strong>sigmoid</strong> function ? Are <strong>sigmoid</strong> probabilities (for each class) more relivable than <strong>softmax</strong> ?</p>

KFrank: <p>Hi Amit!</p>
<aside class="quote no-group quote-modified" data-username="laro" data-post="1" data-topic="213065" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/laro/48/47125_2.png" class="avatar"> laro:</div>
<blockquote>
<p>I’m wondering which activation function for multi class classification problem, give true probability.<br>
…<br>
Are <strong>sigmoid</strong> probabilities (for each class) more relivable than <strong>softmax</strong> ?</p>
</blockquote>
</aside>
<p>The short answer is that <code>softmax()</code> and <code>sigmoid()</code> are used for different things.<br>
It’s not that one is true and the other false or that one is more stable and the other<br>
less stable – they’re just different.</p>
<p>Let me give you my perspective on this.  (I haven’t looked at the links you posted.)</p>
<p>For some context, my number-one rule is that the output of a neural network means<br>
whatever you train it to mean.</p>
<p>Let’s say that you have a problem with three classes and the final layer of your network<br>
is a <code>Linear</code> with <code>out_features = 3</code> (not followed by any additional “activation” layer).<br>
The output of your network will be three numbers will run from <code>-inf</code> to <code>inf</code>.</p>
<p>What does the output of such a network mean?  Whatever you train it to mean.</p>
<p>If you pass that output through <code>softmax()</code>, you will get three probabilities (therefore<br>
each between <code>0.0</code> and <code>1.0</code>) that sum to <code>1.0</code>.  This would be used for a <em>multi-class</em><br>
problem where your input is in exactly one class and the result of <code>softmax()</code> is the<br>
set of probabilities for the input being in each of the three classes.  (Because the input<br>
is in exactly in one of the three classes, these probabilities sum to one.)</p>
<p>On the other hand, If you pass your output through <code>sigmoid()</code>, you will get three<br>
<em>independent</em> probabilities, that is, each probability is between zero and one, but<br>
they don’t sum to anything in particular.  (They might sum to zero or one or anything<br>
in between.)  This would be used for a so-called <em>multi-label</em> problem where the input<br>
can independently be in or not in any of the three classes.  That is, the input might<br>
be in none of the classes, in all three classes, or just in one or two of the classes.</p>
<p>In the multi-class case you would typically use <code>CrossEntropyLoss</code> (which has<br>
<code>logsoftmax()</code> built into it) and the output of your network (which is the input to<br>
<code>CrossEntropyLoss</code>) would typically be interpreted as <em>unnormalized log-probabilities</em><br>
(that are converted into probabilities by <code>softmax()</code>), because you’ve <em>trained your<br>
network</em> to predict unnormalized log-probabilities.</p>
<p>In the multi-label case, you would typically use <code>BCEWithLogitsLoss</code> (which has<br>
<code>log_sigmoid()</code> built into it) and the output of your network would be interpreted as<br>
<em>logits</em> (that are converted into probabilities by <code>sigmoid()</code>), because you’ve <em>trained<br>
your network</em> to predict logits.</p>
<p>Last, if you use integer categorical class labels as your ground-truth <code>target</code> when<br>
training with <code>CrossEntropyLoss</code>, you will be training your network to predict a single<br>
class – that is, you will be training your network to predict 100% probability for the single<br>
correct class and 0% for the others.  If your network trains well, it will typically predict a<br>
value that (after being converted by <code>softmax()</code>) is very close to <code>1.0</code> for one class<br>
and values that (after being converted) are very close to <code>0.0</code> for the the other classes.</p>
<p>But if you use floating-point probabilistic “soft labels” as your ground-truth <code>target</code><br>
for <code>CrossEntropyLoss</code>, you will train your network to predict a probability distribution<br>
for your classes.  For example, you might predicts 25% for classes A and B and 50%<br>
for class C.  This would be an exactly-correct prediction for an input whose ground-truth<br>
target was <code>[0.25, 0.25, 0.50]</code>.</p>
<p>Note, these comments should also answer the question in your other post:</p>
<aside class="quote quote-modified" data-post="1" data-topic="213069">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/laro/48/47125_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/why-softmax-training-is-more-stable-than-sigmoid/213069">Why softmax training is more stable (than sigmoid)</a> 
  </div>
  <blockquote>
    I’m wondering about which activation function will be easier to train with (get better accuracy / smallest loss) - with SoftMax or sigmoid (for multiclass classification problem) 
According to: <a href="https://www.quora.com/What-are-the-benefits-of-using-a-softmax-function-instead-of-a-sigmoid-function-in-training-deep-neural-networks" rel="noopener nofollow ugc">https://www.quora.com/What-are-the-benefits-of-using-a-softmax-function-instead-of-a-sigmoid-function-in-training-deep-neural-networks</a> 
Training model for multiclass with SoftMax - the training is more stable vs training with sigmoid 

Why is it true ?
Is it easier to train model (and get better results) …
  </blockquote>
</aside>

<p>Best.</p>
<p>K. Frank</p>