Neural network binary classification softmax logsofmax and loss function

username: ni_tempe
cooked: <p>I am building a binary classification where the class I want to predict is present only &lt;2% of times.</p>
<p>The last layer could be <code>logosftmax</code> or <code>softmax</code>.</p>
<p><code>self.softmax = nn.Softmax(dim=1)</code> or <code>self.softmax = nn.LogSoftmax(dim=1)</code></p>
<p>my questions</p>
<ol>
<li>
<p>I should use <code>softmax</code> as it will provide outputs that sum up to 1 and I can check performance for various prob thresholds. is that understanding correct?</p>
</li>
<li>
<p>if I use <code>softmax</code> then can I use <code>cross_entropy</code> loss? <a href="https://machinelearningmastery.com/how-to-choose-loss-functions-when-training-deep-learning-neural-networks/" rel="noopener nofollow ugc">This</a> seems to suggest that it is okay to use</p>
</li>
<li>
<p>if i use <code>logsoftmax</code> then can I use <code>cross_entropy</code> loss? <a href="https://stackoverflow.com/questions/64494819/use-crossentropyloss-with-logsoftmax" rel="noopener nofollow ugc">This</a> seems to suggest that I shouldnt.</p>
</li>
<li>
<p>if I use <code>softmax</code> then is there any better option than <code>cross_entropy</code> loss?</p>
<pre><code>    ` cross_entropy = nn.CrossEntropyLoss(weight=class_wts)`
</code></pre>
</li>
</ol>

username: KFrank
cooked: <p>Hi Ni!</p>
<aside class="quote no-group" data-username="ni_tempe" data-post="1" data-topic="145532" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/n/e9c0ed/48.png" class="avatar"> ni_tempe:</div>
<blockquote>
<p>I am building a binary classification</p>
</blockquote>
</aside>
<p>Build a model that outputs a single value (per sample in a batch),<br>
typically by using a <code>Linear</code> with <code>out_features = 1</code> as the final<br>
layer.</p>
<p>This value will be a raw-score <em>logit.</em>  Use <code>BCEWithLogitsLoss</code> as your<br>
loss criterion (and do not use a final “activation” such as <code>sigmoid()</code> or<br>
<code>softmax()</code> or <code>log_softmax()</code>).</p>
<aside class="quote no-group">
<blockquote>
<p>the class I want to predict is present only &lt;2% of times.</p>
</blockquote>
</aside>
<p>Either sample your underrepresented class more heavily when training,<br>
e.g., about fifty times more heavily, or weight the underrepresented class<br>
in your loss computation by using <code>BCEWithLogitsLoss</code>’s <code>pos_weight</code><br>
constructor argument with something like:</p>
<pre data-code-wrap="python"><code class="lang-python">criterion = torch.nn.BCEWithLogitsLoss (pos_weight = torch.tensor ([50.0]))
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

username: ni_tempe
cooked: <p>could you answer my 4 questions? just yes or no would suffice…<br>
I will also look into your reply and try</p>
<p>Few additional questions:<br>
I understand your suggestion " and do not use a final “activation” such as <code>sigmoid()</code> or<br>
<code>softmax()</code> or <code>log_softmax()</code> )." But what should be my final activation? i looked at linear and it doesnt do anything. it is just a pass through. Could you point the exact function?</p>