Multilabel classification with no label examples

farhadzadeh: <p>What is the best way to model the multi-label classification task with no label examples:<br>
For example, we have the following classes: [A, B, C] and our instances are assigned to one or more than one of them. However, we have also some examples without any of these labels, determined as non-findings.<br>
Let’s say the label vectors for annotators 1, 2, and 3 look like this:<br>
A: [1, 0, 0]<br>
B: [0,1, 0]<br>
C: [0, 0, 1]<br>
AB: [1, 1, 0]<br>
AC: [1, 0, 1]<br>
…<br>
Non-finding: [0, 0, 0, 0]<br>
How should I formulize the BCEWithLogitsLoss?</p>

KFrank: <p>Hi Farhad!</p>
<aside class="quote no-group quote-modified" data-username="farhadzadeh" data-post="1" data-topic="90572" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/farhadzadeh/48/9580_2.png" class="avatar"> farhadzadeh:</div>
<blockquote>
<p>we have the following classes: [A, B, C]<br>
…<br>
Non-finding: [0, 0, 0, 0]<br>
How should I formulize the BCEWithLogitsLoss?</p>
</blockquote>
</aside>
<p>You would label “non-finding” as <code>[0, 0, 0]</code>, with three yes-no labels,<br>
one for each of your three classes.</p>
<aside class="quote no-group" data-username="farhadzadeh" data-post="1" data-topic="90572">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/farhadzadeh/48/9580_2.png" class="avatar"> farhadzadeh:</div>
<blockquote>
<p>our instances are assigned to one or more than one of them.</p>
</blockquote>
</aside>
<p>To rephrase this a little more precisely:  “our instances are assigned<br>
to <em>zero</em> or one or more than one of them.”</p>
<p>To clarify a little further:  If this were a <em>single-label</em>, three-class problem,<br>
[A, B, C], but a given instance could also be “none of the above,” then<br>
you would add a fourth class, [none, A, B, C], and perform a four-class<br>
classification.</p>
<p>But for a multi-label, three-class  problem, you would just use the<br>
three classes, [A, B, C] because, with multiple labels, <code>[0, 0, 0]</code><br>
suffices to indicate “none of the above.”</p>
<p>Best.</p>
<p>K. Frank</p>

farhadzadeh: <p>Hi Frank,<br>
Thanks for reply. I think the second option should be the correct one for my problem multilabel classification. In this case how I can measure the performance? Particularly for no finding labels?</p>

KFrank: <p>Hi Farhad!</p>
<aside class="quote no-group" data-username="farhadzadeh" data-post="3" data-topic="90572" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/farhadzadeh/48/9580_2.png" class="avatar"> farhadzadeh:</div>
<blockquote>
<p>how I can measure the performance?</p>
</blockquote>
</aside>
<p>There is no single measure that tells you everything you want to know<br>
about multilabel-classification performance.  See this brief overview in<br>
Wikipedia’s “Multi-label classification” article:</p>
<p><a href="https://en.wikipedia.org/wiki/Multi-label_classification#Statistics_and_evaluation_metrics" rel="noopener nofollow ugc">Statistics and evaluation metrics</a></p>
<p>Different use cases will care more or less about different kinds of errors,<br>
and, so, will prefer different performance measures.</p>
<aside class="quote no-group">
<blockquote>
<p>Particularly for no finding labels?</p>
</blockquote>
</aside>
<p>If you care about not missing “no finding” cases, you could look at an<br>
accuracy given by the percentage of true “no finding” cases that were<br>
predicted (completely) correctly.</p>
<p>If you <em>really</em> care about not incorrectly predicting “not no finding” cases<br>
as being “no finding,” you could look at a rather weird sort of “accuracy”<br>
given by the percentage of “not no finding” cases that were predicted to<br>
have some labels <em>even if all of the predicted labels were wrong.</em>  (Of<br>
course if this were your use case you would likely be better off training<br>
a simple “no finding” vs. “not no finding” binary classifier.)</p>
<p>Good luck.</p>
<p>K. Frank</p>