How to do multi-task training?

username: DavexPro
cooked: <p>I am quite confused about how to do multi-task training.</p>
<p>For example, I got a picture with an animal, I want to get four kinds of output:</p>
<ul>
<li>Length of Nose / invisible, long, middle, short</li>
<li>Length of Tail / invisible, long, middle, short, no tail</li>
<li>Length of Hand / invisible, long, middle, short</li>
<li>Length of Leg / invisible, long, middle, short</li>
</ul>
<p>What should I do, if I use ResNet pre-trained model, and do the 4-tasks classify in <code>one</code> network. The Struct like this:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/d/db8a963add4059fa949f7bbfde71a38c186a324e.jpg" data-download-href="https://discuss.pytorch.org/uploads/default/db8a963add4059fa949f7bbfde71a38c186a324e" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/d/db8a963add4059fa949f7bbfde71a38c186a324e_2_690x231.jpg" alt="image" data-base62-sha1="vk9ubQSpNevoRA7yJMDoiDZHpjU" width="690" height="231" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/d/db8a963add4059fa949f7bbfde71a38c186a324e_2_690x231.jpg, https://discuss.pytorch.org/uploads/default/optimized/2X/d/db8a963add4059fa949f7bbfde71a38c186a324e_2_1035x346.jpg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/d/db8a963add4059fa949f7bbfde71a38c186a324e_2_1380x462.jpg 2x" data-dominant-color="F5F5F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1726×580 97.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I search in google, but I can only get links as follow, but it does not work.</p>
<aside class="quote quote-modified" data-post="1" data-topic="4480">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/bartolsthoorn/48/1182_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/triplet-vs-cross-entropy-loss-for-multi-label-classification/4480">Triplet vs Cross entropy loss for multi-label classification</a> <a class="badge-category__wrapper " href="/c/vision/5"><span data-category-id="5" style="--category-badge-color: #AB9364; --category-badge-text-color: #FFFFFF;" data-drop-close="true" class="badge-category " title="Topics related to either pytorch/vision or vision research related topics"><span class="badge-category__name">vision</span></span></a>
  </div>
  <blockquote>
    Hi, this is a general question about multi-label classification I have been thinking about: 
Multi-label classification for &lt; 200 labels can be done in many ways, but here I consider two options: 


CNN (e.g. Resnet, VGG) + Cross entropy loss, the traditional approach, the final layer contains the same number of nodes as there are labels. Samples are taken randomly and compared to the known labels. 


CNN (e.g. Resnet, VGG) + Triplet loss, where the final layer is a “feature vector”. In the end …
  </blockquote>
</aside>
<aside class="quote quote-modified" data-post="29" data-topic="905">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mratsim/48/548_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/multi-label-classification-in-pytorch/905/29">Multi Label Classification in pytorch</a> 
  </div>
  <blockquote>
    Let’s take ResNet finetuning as an example: 
class ResNet50(nn.Module):
    def __init__(self, num_classes):
        super(ResNet50, self).__init__()
        
        # Loading ResNet arch from PyTorch
        original_model = models.resnet50(pretrained=True)
        
        # Everything except the last linear layer
        self.features = nn.Sequential(*list(original_model.children())[:-1])
        
        # Get number of features of last layer
        num_feats = original_model.fc.in_feature…
  </blockquote>
</aside>


username: tom
cooked: <p>If you put your <code>dropout_1</code> output into the 4 <code>Linear</code> layers (probably with 4 outputs, not 36) you can then calculate the losses individually and add them up to form a total loss.</p>
<p>Using that for gradient descent will do the right thing.</p>
<p>Best regards</p>
<p>Thomas</p>

username: DavexPro
cooked: <p>I figured out how to construct my model, but the problem is how to calculate loss. My case is kind of special, each pic only has one kind of label.</p>
<p>To be more specific, there is a picture, and the pic only have Length of Hand ground truth label, there will be no other three labels.</p>
<p>Pic -&gt; [no, no, ‘long’, no] , no means we don’t have the label, and I use 0 represent no which is same as Invisible.</p>
<p>For now, I add the losses up as my loss, but I think is not accurate, do you have any suggestion?</p>
<p>Thanks!<br>
Dave</p>

username: tom
cooked: <p>Just add the losses of labels that you have.</p>

username: DavexPro
cooked: <p>I use <code>nn.CrossEntropyLoss()</code> as my <code>criterion</code> function, and the label as follow:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/2X/c/c995639192667c8e87e5ebd13b8782f29dbafbc4.png" alt="image" data-base62-sha1="sLi2dX8tvak6BTTQkVRxtxsaoGE" width="596" height="264"></p>
<p>And code about loss is as follow:</p>
<pre><code class="lang-auto"># forward
outputs = model(inputs)
loss = 0

for lx in range(len(outputs)):
    tmp_loss = criterion(outputs[lx], labels[:, lx])
    loss += tmp_loss

# backward + optimize only if in training phase
if phase == 'train':
    loss.backward()
    optimizer.step()
</code></pre>
<p>The main problem here is how to add the losses of labels I have, because 0 stands for <code>No Label</code> and <code>Invisible</code>, it is not easy to differ.</p>
<p>Should I use other <code>criterion</code> function?</p>

username: DavexPro
cooked: <p>I tried this <code>nn.MultiLabelSoftMarginLoss()</code> criterion: <a href="https://gist.github.com/bartolsthoorn/36c813a4becec1b260392f5353c8b7cc" rel="nofollow noopener">multilabel_example.py</a></p>
<p>But the accuracy is not that satisfied compared to CrossEntropyLoss, I don’t know what to do now…</p>

username: tom
cooked: <p>So you don’t have the distinction or you loose it in data preparation?</p>

username: DavexPro
cooked: <p>Ahh, I can use 0 stands for no label, and index starts from 1, but there will be 5 outputs per task, which there should be 4 output.</p>
<p>Although I use 0 for no label, but the loss is for a batch, which is a variable(number), I am stuck on here, how to add loss with label.</p>

username: tom
cooked: <p>I would recommend using a mask (1 label, 0 no label) and labels 0…3 (doesn’t matter what if the mask is 0). Then you can take the loss with reduce=False and multiply with the mask, sum over classes and take the mean over the batch.</p>
<p>Best regards</p>
<p>Thomas</p>

username: DavexPro
cooked: <p>Yeah, I am trying this, do I need to change criterion function?</p>

username: tom
cooked: <p>In terms of your code above (<code>CrossEntropyLoss</code> seems good to me unless you have a reason not to use it):</p>
<pre><code class="lang-auto">criterion = CrossEntropyLoss(reduce=False)
...
loss = 0
for lx in range(len(outputs)):
    loss = loss + (mask[:,lx]*criterion(outputs[lx], labels[:, lx])).mean()
</code></pre>
<p>(So the loss with <code>reduce=False</code> reduces over the classes, giving you a float vector of <code>batch_size</code>, <code>mask</code> should be a float of shape <code>batch_size x tasks</code>).</p>
<p>Looking at the <a href="http://pytorch.org/docs/master/nn.html#torch.nn.CrossEntropyLoss" rel="nofollow noopener"><code>CrossEntropyLoss</code> documentation</a>, I also found an alternative to using the mask: If you put <code>-100</code> in your label where you don’t have one, you can use the <code>ignore_index</code> feature and it’ll do the right thing for you without needing changes to your code above.</p>
<p>Best regards</p>
<p>Thomas</p>

username: DavexPro
cooked: <p>Wow, that will be extremely helpful! I will try it now, and feedback later!</p>
<p>Thanks!</p>

username: DavexPro
cooked: <p>From some points, it works but the loss is high compared to Keras with same structure, what can it be? The output block problem?</p>
<pre><code class="lang-auto">(
  ...
  (conv2d_7b): BasicConv2d(
    (conv): Conv2d(2080, 1536, kernel_size=(1, 1), stride=(1, 1), bias=False)
    (bn): BatchNorm2d(1536, eps=0.001, momentum=0.1, affine=True)
    (relu): ReLU()
  )
  (avgpool_1a): AvgPool2d(kernel_size=8, stride=8, padding=0, ceil_mode=False, count_include_pad=False)
  (last_linear): RetBlock(
    (liner_nose): Linear(in_features=1536, out_features=4, bias=True)
    (liner_tail): Linear(in_features=1536, out_features=5, bias=True)
    (liner_hand): Linear(in_features=1536, out_features=4, bias=True)
    (liner_leg): Linear(in_features=1536, out_features=4 bias=True)
  )
)
</code></pre>
<p>My code for output block:</p>
<pre><code class="lang-auto">class RetBlock(nn.Module):

    def __init__(self):
        super(RetBlock, self).__init__()
        in_features = 1536
        self.liner_nose = nn.Linear(in_features, 4)
        self.liner_tail = nn.Linear(in_features, 5)
        self.liner_hand = nn.Linear(in_features, 4)
        self.liner_leg = nn.Linear(in_features, 4)

    def forward(self, x):
        x_nose = self.liner_nose(x)
        x_tail = self.liner_tail(x)
        x_hand = self.liner_hand(x)
        x_leg = self.liner_leg(x)
        
        return x_nose, x_tail, x_hand, x_hand, x_leg

model_ft.last_linear = RetBlock()
</code></pre>
<p>Any wrong 0.0</p>
<p>Dave</p>

username: Luoxd1996
cooked: <p>hi Dave,Do you have solved this problem?Could you put your code in github?I want to learn from it.<br>
Best wish！</p>

username: _Xiaolong
cooked: <p>Hi, Dave, would you please share your codes? thx</p>

username: gaily_sun
cooked: <p>you can use label -1 index no label,<br>
set criterion = nn.CrossEntropyLoss(ignore_index=-1, reduction=“mean”)<br>
I want to know, for multi-task, the label is a label list, so  how to create your Dataset, Could you please show me your defined Dataset? Thank you very much~</p>

username: chrisby
cooked: <p>Shameless plug: I wrote a little helper library that makes it a little easier to do multi-task learning: <a href="https://github.com/chrisby/torchMTL" rel="noopener nofollow ugc">torchMTL</a>. It should work for your example and makes it easy to combine the losses while keeping control over the training loop. I thought it might be of interest for people who are running into similar issues.</p>