Handling "Nones" in multilabel classification

username: shartzog
cooked: <p>I’m working on a multilabel classification problem.  In my current version, I have four potential labels: “Hospitalized”, “Intubated”, “Deceased”, and “Pneumonia”.  My model trains well and has provided some interesting insights on cases with at least one label, but none of my post-training analyses account for the cases that have NO labels, i.e. patients that contracted the disease but recovered at home with no complications.</p>
<p>I started trying to modify <a href="https://discuss.pytorch.org/t/so-what-did-we-learn/96061">my post training analysis routines</a> to add in a 5th “None” label, but quickly realized this wasn’t going to work.  What was my predicted likelihood of “None”?  Is it (1 - [sum of other label likelihoods])?  Not really…  This value is often negative.  At that point, I considered adding in a 5th “None” label <em>before</em> training and revising my multihot target matrices accordingly, but I wasn’t sure that was the right way to go either given that the truth value of that 5th label can be easily calculated given the truth values for the other four labels.</p>
<p>So my questions are two fold:</p>
<ol>
<li>Is adding in the 5th “None” label prior to training the right way forward?</li>
<li>How do interdependencies between labels (i.e. label E is always 0 if label A, B, C, or D is 1 and always 1 otherwise) impact multilabel classification problems?</li>
</ol>

username: KFrank
cooked: <p>Hi Sam!</p>
<aside class="quote no-group quote-modified" data-username="shartzog" data-post="1" data-topic="96647" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/shartzog/48/27488_2.png" class="avatar"> shartzog:</div>
<blockquote>
<p>I’m working on a multilabel classification problem.  I have four potential labels: “Hospitalized”, “Intubated”, “Deceased”, and “Pneumonia”.<br>
…</p>
<ol>
<li>Is adding in the 5th “None” label prior to training the right way forward?</li>
</ol>
</blockquote>
</aside>
<p>You do not need (or want) a “None” label.  The “None” case is indicated<br>
by none of your four given labels being active.</p>
<p>You have a multi-label, multi-class classification problem.  It is<br>
multi-class because you have four classes, “Hospitalized,” “Intubated,”<br>
“Deceased,” and “Pneumonia.”  It is multi-label because any number,<br>
including all and none, of the labels can be active for any sample.</p>
<p>You say your model trains well.  I’m assuming that it trains well for<br>
all label combinations, including no active labels (and, hopefully, you<br>
have some no-active-label samples in your training data).  Provided<br>
that this is the case, your model is fine, and, if you have an issue with<br>
the “None” case in your post-training analysis, you need to fix it there,<br>
rather than trying to tweak your model to sweep the issue under the rug.</p>
<p>Best.</p>
<p>K. Frank</p>

username: shartzog
cooked: <p><a class="mention" href="/u/kfrank">@KFrank</a><br>
Thank you for your feedback!  That was my initial reaction as well, but between posting this and reading your response, I went ahead and tried training <em>with</em> a None label added.  The new model seemed to do quite well.  The accuracy of my other labels seemed improved if anything (at least as far as I’m able to interpret accuracy at this point).  Maybe I’m just thinking about it wrong since, technically, that last label is “Recovered at Home” (which I DO actually want to be able to predict) rather than “None”, but that still begs the question about how label interdependencies might influence model results.  (NOTE: I realized after posting that there’s another obvious inter-label relation present in my use case, namely you won’t be intubated if you’re not hospitalized.)</p>
<p>My use case is also, to put it mildly, atypical.  In the end, the likelihoods themselves are the critical output, not a hard label prediction.  I still plan to do some threshold analysis and possibly renormalize my  likelihoods, but the ultimate goal is to have the ability to predict “risk of an outcome” rather than the outcome itself.  Does any of that change anything?  Or would you still recommend removing the final label and trying to figure out a method to estimate the “Recovered at Home” aka “None” outcome after the fact?</p>
<p>Thanks again for taking the time!<br>
–SEH</p>

username: KFrank
cooked: <p>Hi Sam!</p>
<aside class="quote no-group" data-username="shartzog" data-post="3" data-topic="96647" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/shartzog/48/27488_2.png" class="avatar"> shartzog:</div>
<blockquote>
<p>technically, that last label is “Recovered at Home”</p>
</blockquote>
</aside>
<p>As a side note, for the “None” case to actually be “Recovered at Home,”<br>
there must be some unstated assumptions in play, namely, that<br>
pneumonia always leads to hospitalization, and that the person was<br>
indeed sick and either died or recovered.  (Otherwise, cases like<br>
“Never Sick,” “Still Sick at Home (but not with Pneumonia)” would be<br>
consistent with the “None” case, and “Recovered at Home (from<br>
Pneumonia)” would not be consistent with “None.”</p>
<aside class="quote no-group">
<blockquote>
<p>label interdependencies might influence model results.</p>
</blockquote>
</aside>
<p>My general view is that if you have uniformly-structured “label<br>
interdependencies,” you should look for a way to build them into<br>
your model.  A clear example is that if you know that exactly one<br>
of your labels will be active for each sample, then you recognize<br>
that you are actually working with a single-label, multi-class problem,<br>
and you model it as such (rather than as a multi-label problem).<br>
This way you make it easier for your training because you have<br>
“told” your model that there will be exactly one label, rather than<br>
having your model “learn” this fact.</p>
<p>But if your label interdependencies are, so to speak, “a little of this,<br>
a little of that,” then you’re probably better off letting your model<br>
learn those interdependencies through the training process, rather<br>
than developing some complicated scheme to build them into your<br>
model.</p>
<aside class="quote no-group">
<blockquote>
<p>there’s another obvious inter-label relation present in my use case, namely you won’t be intubated if you’re not hospitalized.</p>
</blockquote>
</aside>
<p>Assuming that all the other labels can be present or not, independent<br>
of one another – that is, that this interdependency is not part of a larger<br>
uniform structure – I would consider it appropriate for your model to<br>
learn this.  (And I would assume that your training data contains many<br>
samples with “Not Hospitalized and Not Intubated,” “Hospitalized and<br>
Not Intubated,” and “Hospitalized and Intubated,” but no samples with<br>
“Not Hospitalized and Intubated,” so that your model can, in fact, learn<br>
it.)</p>
<aside class="quote no-group">
<blockquote>
<p>In the end, the likelihoods themselves are the critical output, not a hard label prediction.  …  the ultimate goal is to have the ability to predict “risk of an outcome” rather than the outcome itself.</p>
</blockquote>
</aside>
<p>If your training data is labelled with probabilities, then you can sensibly<br>
train your model to predict probabilities.  But – to my mind, at least – if<br>
your labels are hard labels (e.g., 0 or 1, but not 0.625), then I view the<br>
probabilistic predictions (e.g., logits from a final <code>Linear</code> layer) made<br>
by your model to be more an uncertainty about what the model has<br>
successfully learned, rather than a prediction of a probability.  But<br>
perhaps such a distinction doesn’t matter in practice, or perhaps it’s<br>
purely semantic.</p>
<p>Best.</p>
<p>K. Frank</p>

username: shartzog
cooked: <p><a class="mention" href="/u/kfrank">@KFrank</a><br>
Great points re the ability to interpret “None” as “recovered at home”!  Now that you mention it, I <em>KNOW</em> there are examples in my dataset in which Pneumonia was indicated AND the patient still recovered at home!</p>
<p>As for confounding factors related to cases that have not yet “concluded”, I controlled for those by only considering cases where the onset of symptoms was more than a month in the past.  That has it’s own set of issues (e.g. it won’t reflect improvements in treatment in real time), but at least 99+% of cases will be resolved one way or the other by then.</p>
<p>I think my path forward will be to add in the 5th “HomeRecovery” label <em>without</em> excluding pneumonia cases.  I’d also love to be able to capture the “Never Sick” vs. “got sick but still recovered at home” distinction, but it’s just not present in the data as far as I can figure.</p>
<p>Finally, to your last point, I realize I’m out on a limb here by trying to interpret my outputs as probabilities, but…  That’s exactly what I’d like to be able to do.  Hence my struggles interpreting accuracy, I guess…  <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
<p>Regardless, I really appreciate your insights!!!  Thanks a ton!<br>
–SEH</p>
<p>EDIT: Changed “solution” to be your first response rather than my initial reaction to mark the second.  The first response will likely be more valuable to users with a more typical use case.</p>

username: KFrank
cooked: <p>Hi Sam!</p>
<p>Well, I shouldn’t pretend to have more than a superficial understanding<br>
of your use case, however …</p>
<p>Let me make a few more comments.</p>
<aside class="quote no-group" data-username="shartzog" data-post="5" data-topic="96647" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/shartzog/48/27488_2.png" class="avatar"> shartzog:</div>
<blockquote>
<p>I think my path forward will be to add in the 5th “HomeRecovery” label <em>without</em> excluding pneumonia cases.</p>
</blockquote>
</aside>
<p>I think you’re overly fixated on adding another label.  I think the<br>
multi-label approach is your friend, and is already doing the necessary<br>
work for you.</p>
<aside class="quote no-group">
<blockquote>
<p>As for confounding factors related to cases that have not yet “concluded”, I controlled for those by only considering cases where the onset of symptoms was more than a month in the past.</p>
</blockquote>
</aside>
<p>So, as I understand it, in all of your samples the person has been<br>
sick (“onset of symptoms”), and, as an approximation, has either<br>
died or (approximately) recovered (“more than a month in the past”).<br>
So let’s take those as the rules of the game.</p>
<p>Under these assumptions, is seems to me that your “HomeRecovery”<br>
label is exactly equivalent to “Not Hospitalized” and “Not Deceased.”<br>
Again, I think multi-label is your friend.</p>
<aside class="quote no-group">
<blockquote>
<p>I’d also love to be able to capture the “Never Sick” vs. “got sick but still recovered at home” distinction, but it’s just not present in the data as far as I can figure.</p>
</blockquote>
</aside>
<p>Yes, if the reality of your data is that (to a good approximation) each<br>
person has been sick, and (to a good approximation) you don’t have<br>
any “Never Sick” samples in your data, then you won’t be able to<br>
capture a “Never Sick” vs. “got sick” distinction.</p>
<p>To recap:  From what you’ve said so far, it seems to me that the four<br>
labels you discussed originally are just the set you need to capture<br>
all of the cases in your data.  Adding more labels – be they extraneous,<br>
conflicting, or redundant – will, as a general rule, just muddy the waters<br>
and make your model harder to train.</p>
<p>Best.</p>
<p>K. Frank</p>

username: shartzog
cooked: <p><a class="mention" href="/u/kfrank">@KFrank</a><br>
Thanks again for the thoughtful response!</p>
<aside class="quote no-group" data-username="KFrank" data-post="6" data-topic="96647">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>Under these assumptions, is seems to me that your “HomeRecovery”<br>
label is exactly equivalent to “Not Hospitalized” and “Not Deceased.”</p>
</blockquote>
</aside>
<p>Point taken.  How, then, should I calculate a “HomeRecovery” likelihood after the fact?  I can’t really assume independence between “Hospitalized” and “Deceased”, so simple methods e.g.</p>
<pre><code class="lang-auto">P(HomeRec) = 1 - P(Hosp U Deceased)
</code></pre>
<p>won’t work.  Or am I thinking about it wrong?</p>
<p><em>EDIT</em>: Technically, I realize I AM, in many ways, “thinking about it wrong”, not least by interpreting the outputs of my model as probabilities, but those reservations aside, any idea how the math should work?  <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=12" title=":sweat_smile:" class="emoji" alt=":sweat_smile:" loading="lazy" width="20" height="20"></p>
<p>Thanks again!<br>
–SEH</p>

username: KFrank
cooked: <p>Hi Sam!</p>
<aside class="quote no-group" data-username="shartzog" data-post="7" data-topic="96647" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/shartzog/48/27488_2.png" class="avatar"> shartzog:</div>
<blockquote>
<p>How, then, should I calculate a “HomeRecovery” likelihood after the fact?</p>
</blockquote>
</aside>
<p>I’m not really sure what you are asking, but from the perspective of<br>
your dataset, you can simply count:</p>
<p>Empirically observed probability = number of samples having both<br>
“Not Hospitalized” and “Not Deceased” divided by total number of<br>
samples.</p>
<aside class="quote no-group">
<blockquote>
<p>I can’t really assume independence between “Hospitalized” and “Deceased”, so simple methods e.g.</p>
<pre><code class="lang-auto">P(HomeRec) = 1 - P(Hosp U Deceased)
</code></pre>
<p>won’t work.</p>
</blockquote>
</aside>
<p>On the contrary.  If I interpret the <code>U</code> as meaning “union” or “or,” this<br>
is correct.</p>
<p>Using the trivial identity:</p>
<pre><code class="lang-auto">1 = total_samples / total_samples
</code></pre>
<p>your expression becomes:</p>
<pre><code class="lang-auto">P(HomeRec) = (total_samples - samples_with_Hosp_or_Deceased) / total_samples
           = samples_with_Not_Hosp_and_Not_Deceased / total_samples
</code></pre>
<p>This doesn’t assume that “Hospitalized” and “Deceased” are<br>
independent.  Indeed, the “union” (your <code>U</code>) is what accounts for any<br>
interdependency.</p>
<p>Let’s rewrite your probability in terms of an “intersection” (“and”):</p>
<pre><code class="lang-auto">P(HomeRec) = P(NotHosp and NotDeceased)
</code></pre>
<p>What you can’t do (without"Hospitalized" and “Deceased” being<br>
probabilistically independent) is the following:</p>
<pre><code class="lang-auto">P(NotHosp and NotDeceased) = P(NotHosp) * P(NotDeceased)
</code></pre>
<p>That is, you can’t factor the joint probability into a product of two<br>
individual probabilities unless the two conditions are independent.</p>
<p>Best.</p>
<p>K. Frank</p>

username: shartzog
cooked: <p>Thanks again, <a class="mention" href="/u/kfrank">@KFrank</a>!<br>
I especially appreciate the detail in your explanation of the statistics!  I’m following how all of that would apply for the <em>entire dataset</em>, but I’d like to be able to produce an estimation of the likelihood of “HomeRecovery” <em>for an individual sample</em>.  Given:</p>
<pre><code class="lang-auto">P(NotHosp and NotDeceased) != P(NotHosp) * P(NotDeceased)
</code></pre>
<p>as stated above and:</p>
<pre><code class="lang-auto">P(Hosp or Deceased) != P(Hosp) + P(Deceased)
</code></pre>
<p>since Hosp and Deceased are not mutually exclusive, I’m struggling to see how I can estimate P(HomeRec) for an individual sample.  Wouldn’t I need some kind of sophisticated simulation technique?</p>
<p>I feel like I’m missing something simple, so, as always, I appreciate your advice!!!<br>
–SEH</p>
<p><strong>EDIT</strong>: To be clear, I’m using (incorrectly) probabilistic notation to represent <em>model likelihood</em> for a single sample, not “true probability”.  Sorry for any confusion!</p>
<p><strong>EDIT 2</strong>: Maybe my question would be better stated as “How can I estimate P(Hosp or Deceased) (or equivalently, P(NotHosp and NotDeceased)) for a single sample?”</p>

username: KFrank
cooked: <p>Hi Sam!</p>
<aside class="quote no-group" data-username="shartzog" data-post="9" data-topic="96647" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/shartzog/48/27488_2.png" class="avatar"> shartzog:</div>
<blockquote>
<p>I’d like to be able to produce an estimation of the likelihood of “HomeRecovery” <em>for an individual sample</em>.</p>
</blockquote>
</aside>
<p>The more I think about it, I do believe that your proposal to train on<br>
a “HomeRec” label is the way to go.</p>
<p>A disclaimer:  Because your data is not labeled with probabilities,<br>
you’re not training your model to predict probabilities.  Consider a<br>
case where you overfit your model on your training data. You’ve<br>
trained it to predict 0s and 1s, so you get mostly predictions along<br>
the lines of  <code>0.01</code> and <code>0.99</code> with a very low loss and very high<br>
accuracy on your training set.  But let’s say that you get a useful,<br>
but middling accuracy of 75% on your validation set.  Your model<br>
(because it was overfit) has high confidence in its predictions (close<br>
to <code>0.0</code> or <code>1.0</code>), but these are not related to non-training-set<br>
probabilities.</p>
<p>Back to your question:</p>
<p>You could add the additional “HomeRec” label to your multi-label<br>
classification problem, train that, and interpret your “HomeRec”<br>
prediction values (converted from logits to probabilities, as appropriate,<br>
depending on the output of your model) as the probability of a single<br>
sample being “HomeRec.”  To tie this back to the earlier discussion,<br>
we can think of the model “learning” how the correlated “Hospitalized”<br>
and “Deceased” probabilities should be combined to produce the<br>
“HomeRec” probability.</p>
<p>Note, when training such a network, you should probably stop the<br>
training when your training loss and validation loss first start to diverge<br>
<em>even if further training were to produce a model whose validation<br>
and/or test loss and accuracy were better.</em>  This is because you’re<br>
trying to make your predictions best reflect their own uncertainty,<br>
rather than make them as good as possible, but without a good<br>
measure of their uncertainty.</p>
<p>Here’s a couple more things to consider:</p>
<p>If you want to focus narrowly on predicting the “HomeRec” probability,<br>
then perhaps you should train a pure “HomeRec” vs. everything else<br>
binary classifier.  Then the classifier won’t make compromises to<br>
get the other classes right at the expense of not doing as well on<br>
“HomeRec.”  But you could argue to the contrary that by not training<br>
on the other classes you’re giving your training procedure less<br>
information.  Perhaps learning the other classes helps your model<br>
learn relevant “features” that help it do a better on “HomeRec.”</p>
<p>Another point (that I’ll outline in the context of a pure “HomeRec” binary<br>
classifier):  If your dataset is unbalanced, it might be best to train your<br>
classifier with weighted sampling or class weights in the loss function<br>
so that its predictions won’t be biased in favor of the class that occurs<br>
more frequently in your training set.  But, coming back to the issue of<br>
predicting probabilities, if the fraction of “HomeRec” vs.“Not HomeRec”<br>
in your training set is representative of the “HomeRec” fraction in the<br>
“real-world” data to which you’ll be applying your model, then perhaps<br>
you <em>do</em> want your predictions to be so biased.  Consider the extreme<br>
case where your model gives the same prediction value, regardless<br>
of its input:  If your real-world and training data both consist of 30%<br>
“HomeRec” samples, then you would want that single prediction value<br>
to be <code>0.30</code>, rather than the “unbiased” <code>0.50</code>.</p>
<p>Good luck.</p>
<p>K. Frank</p>

username: shartzog
cooked: <p>As always, many, many thanks for your time and advice, <a class="mention" href="/u/kfrank">@KFrank</a>!</p>
<aside class="quote no-group" data-username="KFrank" data-post="10" data-topic="96647">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>Because your data is not labeled with probabilities,<br>
you’re not training your model to predict probabilities.</p>
</blockquote>
</aside>
<p>Thank you for the explanation re the above!  That makes it much clearer <em>why</em> my ‘likelihoods’ are NOT likelihoods OR probabilities in any real sense and how interpreting them as such could lead to false impressions.</p>
<aside class="quote no-group" data-username="KFrank" data-post="10" data-topic="96647">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>Note, when training such a network, you should probably stop the<br>
training when your training loss and validation loss first start to diverge<br>
<em>even if further training were to produce a model whose validation<br>
and/or test loss and accuracy were better.</em> This is because you’re<br>
trying to make your predictions best reflect their own uncertainty,<br>
rather than make them as good as possible, but without a good<br>
measure of their uncertainty.</p>
</blockquote>
</aside>
<p>Great tip and explanation!!!  I’ll be playing around with learning rates and early stops to see how they influence model behavior.  I’m guessing a reduced learning rate will be preferable to aggressive learning with an early stop, but there’s one way to find out… <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<aside class="quote no-group" data-username="KFrank" data-post="10" data-topic="96647">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>If you want to focus narrowly on predicting the “HomeRec” probability,<br>
then perhaps you should train a pure “HomeRec” vs. everything else<br>
binary classifier.</p>
</blockquote>
</aside>
<p>This will be an interesting test.  I’d like to see how my predictions compare.  Prior to now, I’ve been thinking about my multi-label model <em>as if it <strong>were</strong> a set of parallel binary classifiers</em>, but that’s clearly a bad analogy.</p>
<aside class="quote no-group" data-username="KFrank" data-post="10" data-topic="96647">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>if the fraction of “HomeRec” vs.“Not HomeRec”<br>
in your training set is representative of the “HomeRec” fraction in the<br>
“real-world” data to which you’ll be applying your model, then perhaps<br>
you <em>do</em> want your predictions to be so biased.</p>
</blockquote>
</aside>
<p>I think this might be key in my use case.  My dataset is ambiguous to say the least.  It’s possible (and in fact, common) for the <em>exact same input</em> to appear multiple times with <em>different target labels</em>.  E.g. I could have three 50 year old men with hypertension, one who was hospitalized with pneumonia but recovered, one who recovered at home with no complications, and one who died of covid related cardiac arrest without ever being hospitalized.  My input “image” would be the same in all three cases since it’s “calculated” from tabular data, not a true image.  I <em>think</em> this is the key difference between what I’m attempting to do and the traditional image recognition use case.  There really <em>isn’t</em> a right answer.  That’s certainly the reason that I’ve been so hell-bent on interpreting my model results as probabilities.</p>
<p>But back to your insight re bias…  I think I actually <em>do</em> want that bias reflected in my predictions, but if so, I’d be making some strong assumptions regarding the quality, comprehensiveness, and representativeness of my data, i.e. assumptions that seem best avoided.  Either way, I think I’ll do some trial runs with class weights and some with batch normalization to see how they influence my results.  At the very least, that’ll be a valuable training exercise.</p>
<p>At the end of the day, I realize this all might be a fool’s errand.  I’m not sure I’ll ever be able to make statements like “diabetes increases risk by 10% for women aged 40 to 50” as I originally hoped, but I’ve definitely learned a ton about PyTorch, so it’s been worth it either way… <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"></p>
<p>Thanks again for all of your thoughts and advice!<br>
Cheers,<br>
SEH</p>

username: KFrank
cooked: <p>Hi Sam!</p>
<aside class="quote no-group" data-username="shartzog" data-post="11" data-topic="96647" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/shartzog/48/27488_2.png" class="avatar"> shartzog:</div>
<blockquote>
<p>Prior to now, I’ve been thinking about my multi-label model <em>as if it <strong>were</strong> a set of parallel binary classifiers</em>, but that’s clearly a bad analogy.</p>
</blockquote>
</aside>
<p>Just one quick comment:  On the contrary, I think understanding a<br>
multi-label, multi-class classifier as a set of parallel binary classifiers<br>
is exactly the right way to think about it.  The “parallel” classifiers<br>
have an efficiency advantage in training and inference, because<br>
you pass any given sample through one network instead of many,<br>
and much of the work is shared.  They can also have an accuracy<br>
advantage, because training on all of the classes at the same time<br>
can help your classifier learn more “insightful” features that end up<br>
working better for all classes.  (One could certainly imagine, however,<br>
a ten-class dataset, where the first nine classes cause training on<br>
all ten classes to hurt the performance for the tenth class.)</p>
<p>Best.</p>
<p>K. Frank</p>

username: Deeply
cooked: <aside class="quote no-group" data-username="shartzog" data-post="1" data-topic="96647">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/shartzog/48/27488_2.png" class="avatar"> shartzog:</div>
<blockquote>
<p>None</p>
</blockquote>
</aside>
<p>I would like to add my two cents, although I didn’t read all the discussion. If the labels of ‘None’ are known a prior; then you can predict ‘None’ as a form of Anomaly Detection based on the other classes.</p>
<p>Otherwise, I think it’s a good option to add a/the ‘None’ label if they are available at training time. By not doing so, what you are trying is the following:<br>
<strong>Designed labels</strong><br>
<code> [1, 0, 0, 0]; [0, 1, 0, 0]; [0, 0, 1, 0]; [0, 0, 0, 1]</code></p>
<p>Then, the prediction of None is made via the Ad-hoc label <code>[0, 0, 0, 0]</code>. IMHO, having five classes (after including None); the model should output five logits. This is what the machine literature has found to work best.</p>
<p>Finally, it is a good idea to pay a special attention at the data balance;  as it is expect that samples with ‘None’ (presumably healthy subjects) to dominate such datasets.</p>

username: shartzog
cooked: <p>I’ll have to do some research about Anomaly Detection and how it works.  Sounds like it could be useful for a lot of things!  Thanks for the tip, <a class="mention" href="/u/deeply">@Deeply</a>!</p>