classification using LogSoftmax vs Softmax and calculating precision-recall curve?

username: ni_tempe
cooked: <p>In case of binary classification we could get final output using LogSoftmax or Softmax. In case of softmax we get results that add up to 1. I understand that LogSoftmax penalizes more for a wrong classification and few other mathematical advantage.</p>
<p>I have binary classification problem with class 1 occurring very rarely (&lt;2% times)</p>
<p>my questions:</p>
<ol>
<li>If I am using probability cutoff of 0.5 (predicting to class 1 if prob is above 0.5) with Softmax then will I get same values for overall accuracy, class 1  - recall, precision and f1 as when using LogSoftmax (and using the lower value of output as prediction class)?</li>
<li>How to calculate precision-recall curve when using LogSoftmax ? <a href="https://medium.com/@douglaspsteen/precision-recall-curves-d32e5b290248" rel="noopener nofollow ugc">This</a> link says that “The precision-recall curve is constructed by calculating and plotting the precision against the recall for a single classifier at a <strong>variety of thresholds.</strong>” How are we going to choose those thresholds if output is not between 0 to 1?</li>
</ol>

username: ptrblck
cooked: <aside class="quote no-group" data-username="ni_tempe" data-post="1" data-topic="145428">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/n/e9c0ed/48.png" class="avatar"> ni_tempe:</div>
<blockquote>
<p>I understand that LogSoftmax penalizes more for a wrong classification and few other mathematical advantage.</p>
</blockquote>
</aside>
<p>No, <code>LogSoftmax</code> doesn’t penalize a “more wrong” classification but applies the <code>log</code> to the <code>softmax</code> output in a numerically stable way.</p>
<ol>
<li>
<p>If you are working on a binary classification use case and are thinking about using a threshold, I assume your output has the shape <code>[batch_size, 1]</code> and you would be using <code>nn.BCE(WithLogits)Loss</code>. In this case, no <code>(Log)Softmax</code> would be used, as you have a single output neuron. To get the prediction using a probability threshold you could use <code>torch.sigmoid(output_logits) &gt; threshold</code>.</p>
</li>
<li>
<p>Again, <code>LogSoftmax</code> is used for e.g. <code>nn.NLLLoss</code> and a multi-class classification.</p>
</li>
</ol>

username: ni_tempe
cooked: <p>I am still not clear.</p>
<p>in both <code>softmax</code> and <code>logsoftmax</code> case, my neural network output has shape  <code>[batch_size,2]</code> and in both cases I am using<code> cross_entropy(probs,labels)</code>. Truth labels have shape <code>[batch_size,1]</code>. Do I need to change anything?</p>
<p>question 3. based upon your comment, is using <code>logsoftmax</code> not useful as I am using <code>cross_entropy</code>?</p>
<p>could you answer my questions 1 and 2?</p>
<ol>
<li>just trying to rephrase - would <code>logsoftmax</code> and <code>softmax</code> give exact same output?<br>
2.if i am using <code>logsoftmax</code> then how to get precision-recall curve</li>
</ol>

username: ni_tempe
cooked: <p>please answer <a href="https://discuss.pytorch.org/t/neural-network-binary-classification-softmax-logsofmax-and-loss-function/145532">this</a>  post first…I created it to clear my understanding</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="ni_tempe" data-post="3" data-topic="145428">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/n/e9c0ed/48.png" class="avatar"> ni_tempe:</div>
<blockquote>
<p>in both <code>softmax</code> and <code>logsoftmax</code> case, my neural network output has shape <code>[batch_size,2]</code> and in both cases I am using<code> cross_entropy(probs,labels)</code>. Truth labels have shape <code>[batch_size,1]</code>. Do I need to change anything?</p>
</blockquote>
</aside>
<p>Both, <code>LogSoftMax</code> and <code>Softmax</code> are wrong if you are using <code>nn.CrossEntropyLoss</code> as raw logits are expected and <code>nn.CrossEntropyLoss</code> will internally apply <code>LogSoftmax</code>.<br>
In the multi-class classification, your target should have the shape <code>[batch_size]</code> and contain the class indices in <code>[0, nb_classes-1]</code>. Based on your description you are not working with <code>nn.BCEWithLogitsLoss</code> but are using <code>nn.CrossEntropyLoss</code> for a “2-class multi-class classification”. <a class="mention" href="/u/kfrank">@KFrank</a> describes the difference in your cross-post.</p>
<aside class="quote no-group" data-username="ni_tempe" data-post="3" data-topic="145428">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/n/e9c0ed/48.png" class="avatar"> ni_tempe:</div>
<blockquote>
<p>question 3. based upon your comment, is using <code>logsoftmax</code> not useful as I am using <code>cross_entropy</code>?</p>
</blockquote>
</aside>
<p>That is correct and you should remove it if you are using <code>nn.CrossEntropyLoss</code>. <code>LogSoftmax</code> is used with <code>nn.NLLLoss</code>.</p>
<aside class="quote no-group" data-username="ni_tempe" data-post="3" data-topic="145428">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/n/e9c0ed/48.png" class="avatar"> ni_tempe:</div>
<blockquote>
<p>just trying to rephrase - would <code>logsoftmax</code> and <code>softmax</code> give exact same output?</p>
</blockquote>
</aside>
<p>No, since <code>LogSoftmax</code> applies the logarithm to the <code>Softmax</code> output.</p>
<aside class="quote no-group" data-username="ni_tempe" data-post="3" data-topic="145428">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/n/e9c0ed/48.png" class="avatar"> ni_tempe:</div>
<blockquote>
<p>if i am using <code>logsoftmax</code> then how to get precision-recall curve</p>
</blockquote>
</aside>
<p>As described before (and in your cross-post) you might want to switch to <code>nn.BCEWithLogitLoss</code> for a binary classification to be able to use thresholds to create the predictions.</p>