Multi-label classification return more than 1 class in each label

tungdop2: <p>How to train a Multi-label classification model when each label should return more than 1 class?<br>
Example:<br>
Image classification have 2 label: style with 4 classes and layout with 5 classes.<br>
An image in list should return 2 style and 3 layout like [1 0 1 0] [1 1  0 0 1]</p>

KFrank: <p>Hi Tùng!</p>
<aside class="quote no-group" data-username="tungdop2" data-post="1" data-topic="133115" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tungdop2/48/42615_2.png" class="avatar"> tungdop2:</div>
<blockquote>
<p>Image classification have 2 label: style with 4 classes and layout with 5 classes.</p>
</blockquote>
</aside>
<p>The simplest approach would be to recognize that you have a<br>
multi-label, nine-class problem.  The fact that the nine classes can<br>
be – conceptually – grouped together into a four-class style group<br>
and a five-class layout group is not something that you need to build<br>
into your network architecture.</p>
<p>So a straightforward approach would be to have the final layer of<br>
your model be a <code>Linear</code> with <code>out_features = 9</code>.  You would typically<br>
use <code>BCEWithLogitsLoss</code>.  The output of your model would have<br>
shape <code>[nBatch, nClass = 9]</code> and your label (the <code>target</code> you pass<br>
to <code>BCEWithLogitsLoss</code>) would have the same shape.  Thus, for<br>
<code>nBatch = 1</code> your example label would be:</p>
<pre data-code-wrap="python"><code class="lang-python">torch.tensor ([[1.0, 0.0, 1.0, 0.0, 1.0, 1.0, 0.0, 0.0, 1.0]])
</code></pre>
<p>(Note the leading <code>nBatch = 1</code> (singleton) dimension.)</p>
<p>Best.</p>
<p>K. Frank</p>

tungdop2: <p>Thank you. But when an image return a Dog and a Cat at the same time, what model learn from this?</p>

KFrank: <p>Hi Tùng!!</p>
<aside class="quote no-group" data-username="tungdop2" data-post="3" data-topic="133115" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tungdop2/48/42615_2.png" class="avatar"> tungdop2:</div>
<blockquote>
<p>when an image return a Dog and a Cat at the same time, what model learn from this?</p>
</blockquote>
</aside>
<p>In a <em>multi-label,</em> multi-class classification task you expect samples<br>
to have more than one label be active at the same time.</p>
<p>Following your example, let’s say that you have three classes:<br>
Dog, Cat, and Horse.  And let’s say that your samples are images<br>
that can contain any (including none) of those three animals at the<br>
same time.  If you have an image where a dog appears on the left<br>
and a cat appears on the right, your ground-truth multi-label label<br>
would be “Yes-Dog, Yes-Cat, No-Horse” (e.g., <code>[1, 1, 0]</code>), and<br>
you would want to train your classifier to predict both a Dog and<br>
a Cat at the same time (and no Horse) for that image.</p>
<p>Best.</p>
<p>K. Frank</p>

tungdop2: <p>This is my model and loss function for 2 labels: style and layout, each have more than 10 classes. Is it true? How can i improve it?</p>
<pre><code class="lang-auto">﻿﻿class MyModel(nn.Module):
    def __init__(self, n__classes=32):
        super().__init__()
        self.base_model = models.resnet50(pretrained=True).to(device)
        last_channel = self.base_model.fc.in_features
        self.base_model.fc = nn.Sequential()
        
        self.layout = nn.Sequential(
            nn.Dropout(0.2),
            nn.Linear(last_channel, n_classes_layout),
            nn.Sigmoid()
        )
        self.style = nn.Sequential(
            nn.Dropout(0.2),
            nn.Linear(last_channel, n_classes_style),
            nn.Sigmoid()
        )
    def forward(self, x):
        base = self.base_model(x)
        return self.layout(base), self.style(base)

def loss_fn(outputs, targets):
    o1, o2 = outputs
    t1, t2 = targets
    l1 = nn.BCELoss()(o1, t1)
    l2 = nn.BCELoss()(o2, t2)
    return (l1 + l2) / 2
</code></pre>

KFrank: <p>Hi Tùng!</p>
<aside class="quote no-group quote-modified" data-username="tungdop2" data-post="5" data-topic="133115" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tungdop2/48/42615_2.png" class="avatar"> tungdop2:</div>
<blockquote>
<p>This is my model and loss function for 2 labels: style and layout, each have more than 10 classes. How can i improve it?</p>
</blockquote>
</aside>
<p>First comment:  For reasons of numerical stability, you should use<br>
<code>BCEWithLogitsLoss</code> and get rid of the final <code>Sigmoid</code> layers.</p>
<p>Other than that, your approach looks reasonable.</p>
<p>However, you don’t need to split your final layer into separate <code>layout</code><br>
and <code>style</code> pieces.  The following is (essentially*) equivalent:</p>
<pre data-code-wrap="python"><code class="lang-python">        self.layout_and_style_together = nn.Sequential(
            nn.Dropout(0.2),
            nn.Linear(last_channel, n_classes_layout + n_classes_style),
            nn.Sigmoid()
        )
</code></pre>
<p>*)  There is a difference in that your version has two separate<br>
<code>Dropout</code>s.  Therefore the choice of which inputs to randomly zero<br>
out is made separately for your <code>layout</code> and <code>style</code> layers, where<br>
the same random choice is used for both the “layout” and “style”<br>
sections of my <code>layout_and_style_together</code> layer.  (I can’t imagine<br>
that this matters at all.)</p>
<p>Also, if you are willing to forgo the <code>Dropout</code> (and do the right thing by<br>
removing the <code>Sigmoid</code>), you can simply replace <code>resnet50</code>’s final layer<br>
with a single <code>Linear</code>:</p>
<pre data-code-wrap="python"><code class="lang-python">        self.base_model = models.resnet50 (pretrained=True).to (device)
        last_channel = self.base_model.fc.in_features
        self.base_model.fc = nn.Linear (last_channel, n_classes_layout + n_classes_style)
</code></pre>
<p>(I don’t have an opinion about whether your final <code>Dropout</code> is helpful<br>
or necessary.  However, the <code>resnet50</code> architecture doesn’t have one.)</p>
<p>Best.</p>
<p>K. Frank</p>

tungdop2: <p>Thank you so much. I have some deeper questions:</p>
<ul>
<li>When i should split my output and when i should combine them like your way?</li>
<li>I think my model will be better if i know the relation between output labels, is it true? And how to research that relations?</li>
<li>And why i shounldn’t  use Sigmoid layer?</li>
</ul>
<p>Sorry for my bad English.</p>

KFrank: <p>Hi Tùng!</p>
<aside class="quote no-group" data-username="tungdop2" data-post="7" data-topic="133115" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tungdop2/48/42615_2.png" class="avatar"> tungdop2:</div>
<blockquote>
<ul>
<li>When i should split my output and when i should combine them like your way?</li>
</ul>
</blockquote>
</aside>
<p>It really doesn’t matter – it is a purely stylistic choice.  The two versions<br>
are completely* equivalent.  In my version there is a single <code>Linear</code><br>
whose <code>weight</code> has shape:</p>
<p><code>[n_classes_layout + n_classes_style, last_channel]</code>,</p>
<p>while your version has two <code>Linear</code>s with <code>weight</code>s with shapes:</p>
<p><code>[n_classes_layout, last_channel]</code>,</p>
<p>and</p>
<p><code>[n_classes_layout + n_classes_style, last_channel]</code>.</p>
<p>But it’s the exact same matrix multiplying the exact same input tensor,<br>
but in your version the matrix and computation are just broken up into<br>
two pieces.</p>
<p>*)  The two versions do actually differ in that your version has two<br>
sets of random <code>Dropout</code> choices, but I think that this difference is<br>
immaterial.</p>
<aside class="quote no-group">
<blockquote>
<ul>
<li>I think my model will be better if i know the relation between output labels</li>
</ul>
</blockquote>
</aside>
<p>When your labels have structure, it is sometimes be better to build<br>
that structure into the architecture of your model so that your model<br>
doesn’t have to “learn” the structure.  But it is also sometimes better<br>
<em>not</em> to build the structure into your model, and, instead, rely on your<br>
model to “learn” the structure.  It depends on your use case.</p>
<p>By way of example, suppose you have a multi-label, ten-class<br>
problem, and you happen to know that you will never have more<br>
than eight labels active at the same time.  That is, your multi-label<br>
label could have any number from zero to eight of the labels active<br>
at the same time, but never nine or ten.  You would be better off<br>
not trying to build that structure into your model – just train your<br>
model well, and it  won’t (very often) predict more than eight labels<br>
being active at the same time.</p>
<p>On the other hand, suppose you have four classes, A, B, C, and D,<br>
and you have a multi-label problem where you know that exactly<br>
two of the labels will be active at the same time.  You may well be<br>
better off recasting your problem as a single-label, six-class problem<br>
where the six classes are AB, AC, AD, BC, BD, and CD.  I could<br>
well imagine this version would train more easily because it doesn’t<br>
have to “learn” that multi-labels such as B or ACD or no label at<br>
all never occur.</p>
<aside class="quote no-group">
<blockquote>
<ul>
<li>And why i shounldn’t  use Sigmoid layer?</li>
</ul>
</blockquote>
</aside>
<p><code>Sigmoid</code> followed by <code>BCELoss</code> is <em>mathematically</em> equivalent to<br>
<code>BCEWithLogitsLoss</code>, but numerically it is less stable.  Internally<br>
<code>BCEWithLogitsLoss</code> combines the <code>Sigmoid</code> with a <code>log()</code> function<br>
into a <code>log_sigmoid()</code> implementation that uses the <a href="https://en.wikipedia.org/wiki/LogSumExp#log-sum-exp_trick_for_log-domain_calculations" rel="noopener nofollow ugc">log-sum-exp trick</a><br>
for improved numerical stability.</p>
<p>Best.</p>
<p>K. Frank</p>

tungdop2: <p>Thank you. Is there any paper about constrain between labels in multi-labels image classification?</p>