Single vs multi-label vs multi-class training best practice

username: folterj
cooked: <p>Hi, we use convolutional networks (like shufflenet) and are training single, multi-label and (single label) multi-class PyTorch models. We also need probability values for each prediction.<br>
We would like to check best practice for these cases, and we haven’t found a clear answer online so far.</p>
<p>Describing each scenario:</p>
<p>Single label / binary</p>
<ul>
<li>training label: single label (0/1)</li>
<li>training criterion: CrossEntropyLoss</li>
<li>prediction class: max(outputs)</li>
<li>prediction probability: softmax(outputs)</li>
</ul>
<p>Single label / multi-class</p>
<ul>
<li>training label: single label (0-X)</li>
<li>training criterion: CrossEntropyLoss</li>
<li>prediction class: max(outputs)</li>
<li>prediction probability: softmax(outputs)</li>
</ul>
<p>Multi-label / binary</p>
<ul>
<li>training label: an array of binary labels [0/1, 0/1, …]</li>
<li>training criterion: BCEWithLogitsLoss</li>
<li>prediction class: for each output &gt; 0</li>
<li>prediction probability: sigmoid(outputs)</li>
</ul>
<p>Is this the right way of doing this?</p>
<p>For single binary label, is it better to use arrays with single elements for training data and subsequently treat similar to multi-label? Should the same be done to train multi-class using input values no greater than 1?</p>
<p>Also for single binary label (2 classes), what’s the right way of obtaining a meaningful probability?</p>
<p>All feedback appreciated!</p>

username: KFrank
cooked: <p>Hi Joost!</p>
<aside class="quote no-group" data-username="folterj" data-post="1" data-topic="143089" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/folterj/48/35942_2.png" class="avatar"> folterj:</div>
<blockquote>
<p>Single label / binary</p>
</blockquote>
</aside>
<p>You may treat a binary problem as a two-class multi-class problem,<br>
and then you would use <code>CrossEntropyLoss</code> just as you would for<br>
a more-than-two-class multi-class problem.</p>
<p>But it’s a little clearer and modestly more efficient to treat it explicitly<br>
as a binary problem. in which case you would …</p>
<aside class="quote no-group">
<blockquote>
<ul>
<li>training label: single label (0/1)</li>
</ul>
</blockquote>
</aside>
<p>Your training label would be a single floating-point value (per sample)<br>
between <code>0.0</code> and <code>1.0</code>.  This is the probability that the sample is in<br>
the “positive” class (the “yes” class or “class-1” or whatever you want<br>
to call it).</p>
<p>If you want a purely binary label, just restrict the label to be either<br>
exactly <code>0.0</code> or exactly <code>1.0</code>.</p>
<aside class="quote no-group">
<blockquote>
<ul>
<li>training criterion: CrossEntropyLoss</li>
</ul>
</blockquote>
</aside>
<p>Use <code>BCEWithLogitsLoss</code> (and have your predictions be the output<br>
of your final <code>Linear</code> (or convolutional) layer without any subsequent<br>
non-linear activations).  These predictions will now be raw-score <em>logits</em><br>
that run from <code>-inf</code> to <code>inf</code>.</p>
<aside class="quote no-group">
<blockquote>
<ul>
<li>prediction class: max(outputs)</li>
</ul>
</blockquote>
</aside>
<p>To get your prediction class as an integer class label equal to <code>0</code> or <code>1</code>,<br>
threshold against <code>0.0</code>:</p>
<pre data-code-wrap="python"><code class="lang-python">prediction_class = (outputs &gt; 0.0).long()
</code></pre>
<aside class="quote no-group">
<blockquote>
<ul>
<li>prediction probability: softmax(outputs)</li>
</ul>
</blockquote>
</aside>
<p>For the binary case the predicted probability of being in “class-1” is:</p>
<pre data-code-wrap="python"><code class="lang-python">prediction_probability = outputs.sigmoid()
</code></pre>
<aside class="quote no-group">
<blockquote>
<p>Single label / multi-class</p>
<ul>
<li>training label: single label (0-X)</li>
<li>training criterion: CrossEntropyLoss</li>
</ul>
</blockquote>
</aside>
<p>Yes.</p>
<aside class="quote no-group">
<blockquote>
<ul>
<li>prediction class: max(outputs)</li>
</ul>
</blockquote>
</aside>
<p>This should be <code>argmax()</code>.</p>
<p>Let’s assume that <code>outputs</code> has shape = <code>[nBatch, nClass]</code>.  Then:</p>
<pre data-code-wrap="python"><code class="lang-python">prediction_class = outputs.argmax (dim = 1)
# or equivalently
prediction_class = outputs.max (dim = 1)[1]
</code></pre>
<aside class="quote no-group">
<blockquote>
<ul>
<li>prediction probability: softmax(outputs)</li>
</ul>
</blockquote>
</aside>
<p>Yes.</p>
<aside class="quote no-group">
<blockquote>
<p>Multi-label / binary</p>
<ul>
<li>training label: an array of binary labels [0/1, 0/1, …]</li>
</ul>
</blockquote>
</aside>
<p>Yes.  (Again, these could be probabilistic labels between <code>0.0</code> and <code>1.0</code>.)</p>
<aside class="quote no-group">
<blockquote>
<ul>
<li>training criterion: BCEWithLogitsLoss</li>
<li>prediction class: for each output &gt; 0</li>
<li>prediction probability: sigmoid(outputs)</li>
</ul>
</blockquote>
</aside>
<p>Yes.</p>
<aside class="quote no-group">
<blockquote>
<p>For single binary label, is it better to use arrays with single elements for training data and subsequently treat similar to multi-label?</p>
</blockquote>
</aside>
<p>Yes.  Treat a single-label binary problem as a multi-label, multi-class<br>
problem where you have just one class (and hence just one label).</p>
<aside class="quote no-group">
<blockquote>
<p>Should the same be done to train multi-class using input values no greater than 1?</p>
</blockquote>
</aside>
<p>If I understand your question correctly, the  predictions (“input values”?)<br>
for a (single-label) multi-class problem should be logits that run from<br>
<code>-inf</code> to <code>inf</code>, rather than probabilities that would be restricted to be<br>
no greater than <code>1.0</code>.  This is because pytorch’s <code>CrossEntropyLoss</code><br>
expects its <code>input</code> (the predictions) to be logits.  (It might better be called<br>
<code>CrossEntropyWithLogitsLoss</code>.)</p>
<aside class="quote no-group">
<blockquote>
<p>Also for single binary label (2 classes), what’s the right way of obtaining a meaningful probability?</p>
</blockquote>
</aside>
<p>For a single-label binary problem, your prediction (your <code>outputs</code>) should<br>
be a single (per sample) logit (and you should use <code>BCEWithLogitsLoss</code><br>
rather than <code>BCELoss</code>) and you obtain the predicted probability of your<br>
sample being in “class-1” by taking <code>outputs.sigmoid()</code>.</p>
<p>Best.</p>
<p>K. Frank</p>

username: folterj
cooked: <p>Hi <a class="mention" href="/u/kfrank">@KFrank</a>, thank you very much for your detailed response!</p>
<p>So for single label non-multiclass I understand now it’s better to use a single model output in combination with BCEWithLogitsLoss. This also removes ambiguity of the classes and probabilities.</p>
<p>Regarding the input values used for multi-class, what I meant was the model input / training labels. Re-reading the documentation, CrossEntropyLoss accepts either an array of probabilities or single multiclass label, so the latter seems more convenient and readily usable from our source labels.</p>
<p>Here is the updated overview (re-ordered for logic). (I wrote this in pseudo language, but indeed for multi-class we use argmax getting the index of the maximum value.)</p>
<p>Single label / multi-class</p>
<ul>
<li>training label: single label (0/1/2/…)</li>
<li>training criterion: CrossEntropyLoss</li>
<li>prediction class: argmax(outputs) ( <code>outputs.argmax(dim=1)</code> )</li>
<li>prediction probability: softmax(outputs) ( <code>outputs.softmax(dim=1)</code> )</li>
</ul>
<p>Single label / binary</p>
<ul>
<li>training label: single (binary) labels [0.0/1.0]</li>
<li>training criterion: BCEWithLogitsLoss</li>
<li>prediction class: {output&lt;=0: 0, output&gt;0: 1} ( <code>(outputs &gt; 0.0).long()</code> )</li>
<li>prediction probability: sigmoid(outputs) ( <code>outputs.sigmoid()</code> )</li>
</ul>
<p>Multi-label / binary</p>
<ul>
<li>training label: an array of (binary) labels [0.0/1.0, 0.0/1.0, …]</li>
<li>training criterion: BCEWithLogitsLoss</li>
<li>prediction class: for each {output&lt;=0: 0, output&gt;0: 1}</li>
<li>prediction probability: sigmoid(outputs)</li>
</ul>
<p>The other thing we came across worth noting is that for the loss function usually the same size tensors are needed. So if the output layer has a size of for example 1000 (like Shufflenet), for training X labels the model output needs to be cropped to match the number of labels before passing it into the loss function.</p>