MultiLabel Classification and Cross Entropy Weights

anotherone_one: <p>Good afternoon!<br>
I have a model that has 6 classes on which each class has several possible labels. I wanted to ask if it is possible to give a list of weights for each label of each class.</p>
<p>My model:</p>
<pre><code class="lang-auto">class CNN(nn.Module):
    def __init__(self):
        super(CNN, self).__init__()
        
        self.model = pretrainedmodels.__dict__["resnet50"](pretrained="imagenet")
        
        self.fc1 = nn.Linear(2048, 3)    
        self.fc3 = nn.Linear(2048, 5)    
        self.fc4 = nn.Linear(2048, 4)    
        self.fc5 = nn.Linear(2048, 5)    
        self.fc7 = nn.Linear(2048, 2)   
        self.fc9 = nn.Linear(2048, 3)    
        
        
    def forward(self, x):
        bs, _, _, _ = x.shape
        x = self.model.features(x)
        x = F.adaptive_avg_pool2d(x, 1).reshape(bs, -1)
        label1 = torch.sigmoid(self.fc1(x))
        label3 = torch.sigmoid(self.fc3(x))  
        label4 = torch.sigmoid(self.fc4(x))  
        label5 = torch.sigmoid(self.fc5(x))  
        label7 = torch.sigmoid(self.fc7(x))
        label9 = self.fc9(x)
          
        return {'label1': label1, 'label3': label3, 'label4': label4, 'label5': label5, 'label7': label7, 'label9': label9}
</code></pre>
<p>I wanted to do this:</p>
<pre><code class="lang-auto">weights = [[0.3, 0.3, 0.4], [0.2, 0.2, 0.2, 0.4], [0.3, 0.3, 0.3, 0.1], [0.3, 0.3, 0.3, 0.1], [0.7, 0.3], [0.1, 0.5, 0.4]]

class_weights = torch.FloatTensor(weights).cuda()

criterion = nn.CrossEntropyLoss()
</code></pre>
<p>I get this error: <code>ValueError: expected sequence of length 3 at dim 1 (got 4)</code></p>
<p>Thank you in advance</p>

Andrei_Cristea: <p>I don’t totally understand what you’re planning to use the weights for, but your error is coming from the fact that Tensors need to have a well-defined shape, however the elements in your list have different lengths. As a quick fix you can try padding with 0.0 weights to get a consistent shape, like this:</p>
<p><code>weights = [[0.3, 0.3, 0.4, 0.0], [0.2, 0.2, 0.2, 0.4], [0.3, 0.3, 0.3, 0.1], [0.3, 0.3, 0.3, 0.1], [0.7, 0.3, 0.0, 0.0], [0.1, 0.5, 0.4, 0.0]]</code></p>

anotherone_one: <p>Thank you for your answer!!<br>
I want to give a weight o each label of each class because I have an imbalance dataset.</p>
<p>For example, for Class1, I have label1, label2, label3. I want to give different weights to the different labels. Then I have Class2, with label1, label2, label3 and label4. Now I want to give weights to these labels of this class. This is what I want to do.</p>

KFrank: <p>Hi Another!</p>
<aside class="quote no-group" data-username="anotherone_one" data-post="1" data-topic="148576" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/anotherone_one/48/18256_2.png" class="avatar"> anotherone_one:</div>
<blockquote>
<p>I have a model that has 6 classes on which each class has several possible labels. I wanted to ask if it is possible to give a list of weights for each label of each class.</p>
</blockquote>
</aside>
<p>Yes, you can weight your labels / classes individually.</p>
<p>But first, some context and terminology:</p>
<p>At a technical level, you are performing 6 multi-class classification<br>
problems “in parallel.”  What you call “6 classes,” I would call 6<br>
classification problems.  And what you call “several possible labels,”<br>
I would call the separate sets of classes for those 6 classification<br>
problems.</p>
<aside class="quote no-group">
<blockquote>
<p>My model:</p>
<pre><code class="lang-auto">class CNN(nn.Module):
    def __init__(self):
        ...        
        self.fc1 = nn.Linear(2048, 3)    
        self.fc3 = nn.Linear(2048, 5)    
        self.fc4 = nn.Linear(2048, 4)    
        self.fc5 = nn.Linear(2048, 5)    
        self.fc7 = nn.Linear(2048, 2)   
        self.fc9 = nn.Linear(2048, 3)    
        
    def forward(self, x):
        ...
        label1 = torch.sigmoid(self.fc1(x))
        label3 = torch.sigmoid(self.fc3(x))  
        label4 = torch.sigmoid(self.fc4(x))  
        label5 = torch.sigmoid(self.fc5(x))  
        label7 = torch.sigmoid(self.fc7(x))
        label9 = self.fc9(x)
          
        return {'label1': label1, 'label3': label3, 'label4': label4, 'label5': label5, 'label7': label7, 'label9': label9}
</code></pre>
</blockquote>
</aside>
<p>Your final fully-connected <code>Linear</code> layers make sense in this context (but<br>
they can be combined together into a single <code>Linear (2048, 22)</code>, which<br>
is how I would do it).</p>
<p>Get rid of the <code>sigmoid()</code>s.  (They are not appropriate for a multi-class<br>
problem.  Furthermore, <code>CrossEntropyLoss</code>, which you should use,<br>
has, in effect, <code>softmax()</code> built into it, and that plays the role that you<br>
might be imagining for <code>sigmoid()</code>.)</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">weights = [[0.3, 0.3, 0.4], [0.2, 0.2, 0.2, 0.4], [0.3, 0.3, 0.3, 0.1], [0.3, 0.3, 0.3, 0.1], [0.7, 0.3], [0.1, 0.5, 0.4]]

class_weights = torch.FloatTensor(weights).cuda()

criterion = nn.CrossEntropyLoss()
</code></pre>
</blockquote>
</aside>
<p>You have 6 separate classification problems (with differing numbers of<br>
classes), so you will want 6 separate <code>CrossEntropyLoss</code> loss criteria<br>
(that you then sum together, either equally or in some weighted fashion).</p>
<p>However, pass in the slices of your <code>class_weights</code> tensor into the<br>
constructors of your 6 loss criteria as the class weights for each of<br>
the 6 problems separately, e.g., something like:</p>
<pre><code class="lang-auto">loss_fn1 = nn.CrossEntropyLoss (weight = class_weights[0])
loss_fn3 = nn.CrossEntropyLoss (weight = class_weights[1])
...
loss_fn9 = nn.CrossEntropyLoss (weight = class_weights[5])
</code></pre>
<p>The <code>weight</code> constructor argument of <code>CrossEntropyLoss</code> consists of<br>
“class-weight” values that weight the classes differently in the loss<br>
function, and is often used to compensate for unbalanced data.</p>
<p>Best.</p>
<p>K. Frank</p>

anotherone_one: <p>Thank you so much for the detailed explanation! Very useful!<br>
Cheers <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=12" title=":smiley:" class="emoji" alt=":smiley:" loading="lazy" width="20" height="20"></p>