Difference between Multilabel-multiclass classification and multitask classification

gass_jb: <p>Dear computer vision community,</p>
<p>I have to find a suitable model for recognizing different events within video clips. I am a little bit confused in considering my task as a multilabel-multiclass classification or multitask classification problem and how should this be implemented</p>
<p>Explanation of the task: From a video clip I need to know if :</p>
<ol>
<li>
<p>an Event A occured or not</p>
</li>
<li>
<p>was it successful, not successful, or out</p>
</li>
<li>
<p>was it performed using right or left hand</p>
</li>
<li>
<p>Which type of event was it (there are 6 types)</p>
</li>
<li>
<p>was it done while running, walking, jumping or standing-up</p>
</li>
</ol>
<p>So the 5 labels need to be returned and each returned label can have one of the subclasses.</p>
<ol>
<li>Should 5 different output layers for 5 different outputs be implemented? Each one of them has n_classes = subclasses</li>
</ol>
<p>self.fc1 = nn.Linear(block_inplanes[3] * block.expansion, n_classes1)</p>
<p>self.fc2 = nn.Linear(block_inplanes[3] * block.expansion, n_classes2)</p>
<p>self.fc3 = nn.Linear(block_inplanes[3] * block.expansion, n_classes3)</p>
<p>self.fc4 = nn.Linear(block_inplanes[3] * block.expansion, n_classes4)</p>
<p>self.fc5 = nn.Linear(block_inplanes[3] * block.expansion, n_classes5)</p>
<p>or</p>
<ol start="2">
<li>should I have only one output layer ;</li>
</ol>
<p>self.fc = nn.Linear(block_inplanes[3] * block.expansion, n_classes)</p>
<p>If the 2nd one is the right answer, how could this be implemented?</p>
<p>I would be very thankful for your precious help.</p>

gass_jb: <p>Hi,</p>
<p>Maybe to be more precise in my question: in case I have multiclass target =&gt; Classes (A,B,C,D,E) and each class have subclasses for example:</p>
<p>Class “A” can be 0 or 1<br>
Class “B” can be 2, 3 or 4<br>
Class “C” can be 5 or 6<br>
Class “D” can be 7,8,9,10,11 or 12<br>
Class “E” can be 13,14, 15 or 16</p>
<p>All information is extracted from the same scene and object<br>
…</p>
<p>Can this task be solved using one hot encoder or multi hot encoder. Or this is a multitask classification problem where I should use multiple outputs and calculate loss for each output independatly then sum them together…?</p>
<p>Thank you.</p>

KFrank: <p>Hi JB!</p>
<aside class="quote no-group" data-username="gass_jb" data-post="1" data-topic="129199">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/g/f1d935/48.png" class="avatar"> gass_jb:</div>
<blockquote>
<p>I have to find a suitable model for recognizing different events within video clips.</p>
</blockquote>
</aside>
<aside class="quote no-group" data-username="gass_jb" data-post="2" data-topic="129199" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/g/f1d935/48.png" class="avatar"> gass_jb:</div>
<blockquote>
<p>Classes (A,B,C,D,E) and each class have subclasses</p>
</blockquote>
</aside>
<p>Could you clarify your use case?</p>
<p>First, let’s temporarily ignore the “subclasses” issue, and just look at<br>
“classes.”</p>
<p>Second, I will <em>purposely</em> speak in the language of two-dimensional<br>
still images rather than “three-dimensional” videos.  There is a clean<br>
analogy between the two, so, to focus the discussion, please describe<br>
your use case in the still-image setting.</p>
<p>Let a single sample (which would typically be part of a batch consisting<br>
of one or more independent samples) be a single still image.</p>
<p>Do you wish to perform (single-label), multi-class classification, e.g.,<br>
this is an image of a cat or a pigeon or a shark (but not more than<br>
one at a time)?</p>
<p>Do you wish to perform multi-label, multi-class classification, e.g.,<br>
this image contains a cat, does not contain a pigeon, and does<br>
contain a shark?</p>
<p>Do you wish to perform object detection, that is, this image contains<br>
one cat in bounding-box A, three pigeons, one each in bounding-boxes<br>
B, C, and D, and no sharks?</p>
<p>(Other possibilities would include semantic segmentation and instance<br>
segmentation.)</p>
<p>Now, about your “subclasses:”  What is the substantive meaning of<br>
a class vs. a subclass?  Why don’t we just say that you have 17<br>
different classes (that happen to be labelled “A0”, “A1”, “B2”, “B3”,<br>
“B4”, …, “E13”, “E14”, “E15”, and “E16”)?</p>
<p>Best.</p>
<p>K. Frank</p>

gass_jb: <p>Hi Frank,  thank you very much for your detailed answer. In my use case I am trying to classify player action within a handball video. I want to know if the player performed a pass with some characteristics but as you said let’s talk about still images and take one image.</p>
<p>Let’s assume that I want to classify an image and want to know for each single image if:</p>
<ul>
<li>contain a car / or not</li>
<li>which color has the car ( red/black/white/blue)</li>
<li>which type of car(Berline/4x4/SUV/Coupe)</li>
<li>is there a conductor within the car / or not</li>
</ul>
<p>About the subclasses:<br>
I thought about considering them as different  classes ( 13 in the example above). So it will be considered as a multilabel classification ( many classes could be activated at the same time). But I cannot have for example:</p>
<ul>
<li>SUV and Berline as car type at the same time.</li>
<li>Or more than one color at the same time …</li>
</ul>
<p>I hope I could explain well the use case.</p>
<p>Thanks a lot for your time.</p>

KFrank: <p>Hi JB!</p>
<aside class="quote no-group" data-username="gass_jb" data-post="4" data-topic="129199" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/g/f1d935/48.png" class="avatar"> gass_jb:</div>
<blockquote>
<p>want to know for each single image if:</p>
<ul>
<li>contain a car / or not</li>
<li>which color has the car ( red/black/white/blue)</li>
<li>which type of car(Berline/4x4/SUV/Coupe)</li>
<li>is there a conductor within the car / or not</li>
</ul>
</blockquote>
</aside>
<p>Continuing with the still-image example use case you gave, I can<br>
see two approaches:</p>
<p>The first approach is a little simpler, but probably wouldn’t be my<br>
choice.  (It should work, though, and is probably worth a try.)</p>
<p>An image can either be a “not-car” or it can be exactly one of 32<br>
kinds of “car.”  That is, there are 4 choices for color, 4 choices for<br>
type, and 2 choices for conductor, so 4 x 4 x 2 = 32.  (If you knew<br>
for a fact that you would never have a white SUV you would,<br>
barring other constraints, only have 30 kinds of “car,” but let’s go<br>
with the full 32.)</p>
<p>Therefore you have 33 classes.  So treat this as a straightforward<br>
single-label, 33-class classification problem.  (Your last layer would<br>
likely be a fully-connected <code>Linear</code> with <code>out_features = 33</code> and<br>
you would likely use <code>CrossEntropyLoss</code> as your loss function.)</p>
<p>But your classes have substantive structure (they’re a “product” of<br>
color and type and conductor) and this is structure that your model<br>
would need to “learn” in the 33-class approach.  You might well get<br>
better results if your model architecture has (some of) this structure<br>
built in.</p>
<p>The second approach takes this structure of your classes into account<br>
(but has the disadvantage that if your classes are not formed by the<br>
full product of your subclasses – say, for example, you never have a<br>
white SUV – there is no natural way to build this in, so the model will<br>
have to “learn” this constraint).</p>
<p>The network will be, in essence, four separate classifiers that share<br>
almost all of the same upstream processing and features.</p>
<p>In its simplest version, you would have a final <code>Linear</code> layer with<br>
<code>out_features = 10</code>.</p>
<p>Suppressing the batch index, <code>output[0]</code> would be your prediction<br>
for “car” / “not-car” (understood to be a single binary-prediction logit,<br>
most likely fed to <code>BCEWithLogitsLoss</code> as its loss function).</p>
<p><code>output[1:5]</code> would be the 4-class prediction for color and<br>
<code>output[5:9]</code> would be the 4-class prediction for type, both<br>
fed to <code>CrossEntropyLoss</code>, and <code>output[9]</code> would be the binary<br>
prediction for “conductor” / “not-conductor”, fed to <code>BCEWithLogitsLoss</code>.</p>
<p>If the <code>target</code> (the ground-truth label) for “car” / “not-car” is “not-car”<br>
you ignore the predictions for color, type, and conductor, skipping<br>
(or multiplying by zero) the computations for their loss functions, and<br>
just use the “car” / “not-car” <code>BCEWithLogitsLoss</code> as the total loss<br>
function.</p>
<p>If the <code>target</code> is “car,” you compute all four loss functions and use<br>
a weighted sum of them as your total loss function.  (The weights<br>
will be non-trainable hyperparameters, tuned by you, based on how<br>
things train, how your results look, and on the relative importance<br>
(to you) of getting, say, the color right vs. getting the type right.)</p>
<p>You could also break your final <code>Linear</code> layer up into four separate<br>
“heads,” one for each of your four classification subproblems, all fed<br>
by the same upstream “feature vector.”  If your four heads are all just<br>
<code>Linear</code> layers with <code>out_features = 1, 4, 4, 1</code>, respectively,<br>
this is mathematically no different than one single <code>Linear</code> layer<br>
with <code>out_features = 10</code>.  Using four <code>Linear</code>-layer heads does<br>
have the advantage, however, that you can use different optimizer<br>
hyperparameters (e.g., different learning rates) for the four heads.</p>
<p>Note that separating your final “layer” into four heads has the<br>
potentially substantive advantage that your four heads can have<br>
different architectures.  For example, maybe a single <code>Linear</code> (with<br>
<code>out_features = 1</code>) gives good-enough results for “car” / “not-car,”<br>
but a sequence of two <code>Linear</code> layers (with the final <code>Linear</code> having<br>
<code>out_features = 4</code>) turns out to be helpful in getting good results<br>
for your type class.</p>
<p>Best.</p>
<p>K. Frank</p>

gass_jb: <p>The proposed solutions are really promising. I tried the four classification subproblems, it worked but the results were not that satisfying. I have to do some adjustments and try it again.</p>
<p>I will also try the other proposed approaches and I think you got the point. I will give you feedback once I have the results.</p>
<p>I am very thankful for your detailed and very constructive response.</p>