Is there an example for multi class multilabel classification in Pytorch?

Shisho_Sama: <p>Hello everyone.<br>
How can I do multiclass multi label classification in Pytorch? Is there a tutorial or example somewhere that I can use?<br>
I’d be grateful if anyone can help in  this regard<br>
Thank you all in advance</p>

Shisho_Sama: <p>I know everything that’s there, and also there was not a single word on <code>multi class multi label classification</code>!<br>
Have you yourself even looked at it before suggesting it?</p>

Shisho_Sama: <p>Thanks, I didnt mean to be rude, I just genuinely wanted to know if you mistook this for some other link or not.<br>
Anyway I appreciate your help.</p>

Shisho_Sama: <p>Does creating different heads (3 classifiers e.g.) count as a multi task? they all have their respective losses and for backpropagating their losses are summed and then the result is backpropagated!<br>
if this is multi-task, what is a multi-lable scenario?  Is this the other name for multi label classification?<br>
if so , what is multi-class multi-label classification?</p>

ptrblck: <p>Have a look at <a href="https://discuss.pytorch.org/t/multi-label-classification-in-pytorch/905/45">this post</a> for a small example on multi label classification.<br>
You could use multi-hot encoded targets, <code>nn.BCE(WithLogits)Loss</code> and an output layer returning <code>[batch_size, nb_classes]</code> (same as in multi-class classification).</p>

Shisho_Sama: <p>Thanks, why did you use <code>nn.BCEWithLogitsLoss()</code> ? and not cross entropy?<br>
Cant we use a sigmoid and a normal crossentropy to have probablities for all classes?</p>

ptrblck: <p><code>nn.CrossEntropyLoss</code> uses the target to index the logits in your model’s output.<br>
Thus it is suitable for multi-class classification use cases (only one valid class in the target).</p>
<p><code>nn.BCEWithLogitsLoss</code> on the other hand treats each output independently and is suitable for multi-label classification use cases.</p>

Shisho_Sama: <p>Thanks a lot as always. Then what about <code>MultiLabelSoftMarginLost</code>? Shouldn’t we use  that? (I know its simplye sigmoid + BCE (<a href="https://github.com/pytorch/pytorch/blob/7b00adf5d31c4fbf5556151514eadea4c1e2aa4b/torch/nn/functional.py#L1116" rel="nofollow noopener">Link</a>)<br>
I guess back in 2017 there was an issue about its numerical instability is it why you chose <code>BCEWithLogitsLoss</code>?</p>

ptrblck: <p>The link points to a legacy version of the loss.<br>
<a href="https://github.com/pytorch/pytorch/blob/0ce7264ed616f7793d3213cb23a5775dcc655b1a/torch/nn/functional.py#L2250">This is the current implementation in the master branch</a>.<br>
The main difference is, that the loss will be averaged over the feature dimension:</p>
<pre data-code-wrap="python"><code class="lang-python">loss = loss.sum(dim=1) / input.size(1)  # only return N loss values
</code></pre>
<p><a href="https://discuss.pytorch.org/t/what-is-the-difference-between-bcewithlogitsloss-and-multilabelsoftmarginloss/14944/2">Here</a> is an older post, which compared both losses, which won’t work anymore due to the shape mismatch.</p>
<p>Here is the updated version:</p>
<pre data-code-wrap="python"><code class="lang-python">x = torch.randn(10, 3)
y = torch.FloatTensor(10, 3).random_(2)

# double the loss for class 1
class_weight = torch.FloatTensor([1.0, 2.0, 1.0])
# double the loss for last sample
element_weight = torch.FloatTensor([1.0]*9 + [2.0]).view(-1, 1)
element_weight = element_weight.repeat(1, 3)

bce_criterion = nn.BCEWithLogitsLoss(weight=None, reduction='none')
multi_criterion = nn.MultiLabelSoftMarginLoss(weight=None, reduction='none')

bce_criterion_class = nn.BCEWithLogitsLoss(weight=class_weight, reduction='none')
multi_criterion_class = nn.MultiLabelSoftMarginLoss(weight=class_weight, reduction='none')

bce_criterion_element = nn.BCEWithLogitsLoss(weight=element_weight, reduction='none')
multi_criterion_element = nn.MultiLabelSoftMarginLoss(weight=element_weight, reduction='none')

bce_loss = bce_criterion(x, y)
multi_loss = multi_criterion(x, y)

bce_loss_class = bce_criterion_class(x, y)
multi_loss_class = multi_criterion_class(x, y)

bce_loss_element = bce_criterion_element(x, y)
multi_loss_element = multi_criterion_element(x, y)

print(torch.allclose(bce_loss.mean(1), multi_loss))
&gt; True
print(torch.allclose(bce_loss_class.mean(1), multi_loss_class))
&gt; True
print(torch.allclose(bce_loss_element.mean(1), multi_loss_element))
&gt; True
</code></pre>
<aside class="quote no-group" data-username="Shisho_Sama" data-post="10" data-topic="53579">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/shisho_sama/48/6926_2.png" class="avatar"> Shisho_Sama:</div>
<blockquote>
<p>I guess back in 2017 there was an issue about its numerical instability is it why you chose <code>BCEWithLogitsLoss</code> ?</p>
</blockquote>
</aside>
<p>Yes, and I think it could be still an issue, as <code>logsigmoid</code> is mathematically more stable than <code>log + sigmoid</code>, since internally the LogSumExp trick will be applied as seen <a href="https://github.com/pytorch/pytorch/blob/517c7c98610402e2746586c78987c64c28e024aa/aten/src/THNN/generic/LogSigmoid.c#L13">here</a>.</p>

chenjus: <p>When I try this, I get the following error:<br>
<code>RuntimeError: Boolean value of Tensor with more than one value is ambiguous</code></p>
<p>Here are my logits</p>
<pre><code class="lang-auto">tensor([[-2.8443, -3.3110, -2.5216,  ..., -2.7601, -3.0928, -2.9031],
        [-2.8533, -2.9637, -2.5839,  ..., -2.3841, -2.8846, -3.0366],
        [-2.8923, -3.2757, -2.6118,  ..., -2.4875, -2.7701, -3.1466],
        ...,
        [-2.9981, -3.2178, -2.5539,  ..., -2.7732, -3.0216, -2.8305],
        [-2.7969, -3.0189, -2.4602,  ..., -2.2811, -2.9239, -3.1404],
        [-2.8644, -2.9294, -2.5960,  ..., -2.4510, -2.8790, -2.9344]],
       grad_fn=&lt;IndexBackward&gt;)
</code></pre>
<p>and labels</p>
<pre><code class="lang-auto">tensor([[0, 0, 0,  ..., 0, 1, 0],
        [0, 0, 0,  ..., 0, 1, 0],
        [0, 0, 0,  ..., 1, 0, 0],
        ...,
        [0, 0, 0,  ..., 0, 0, 0],
        [0, 0, 0,  ..., 0, 0, 0],
        [0, 0, 0,  ..., 0, 0, 1]])
</code></pre>
<p>tensors of shape batch by classes. What am I doing wrong?</p>

chenjus: <p>Nvm. I was treating <code>nn.BCEWithLogitsLoss</code> as a function from <code>torch.nn.functional</code> and was doing <code>nn.BCEWithLogitsLoss(logits, label)</code>. Fixed by changing to <code>n.BCEWithLogitsLoss()(logits, label)</code> in case anyone runs into that.</p>

zuriatib: <p>Hi!</p>
<p>I am doing multilabel classification, and came across your reply. Can you help to explain why they are different? I also looked up the <a href="https://pytorch.org/docs/stable/generated/torch.nn.BCEWithLogitsLoss.html" rel="noopener nofollow ugc">documentation</a> but it’s suggesting nn.BCEWithLogitsLoss(logits, label). Thank you!!!</p>

Ycblue: <p>torch.nn.functional.binary_cross_entropy_with_logits() is a function which will calculate the loss directly:</p>
<pre><code class="lang-auto">torch.nn.functional.binary_cross_entropy_with_logits(logits, label)
</code></pre>
<p>whereas nn.BCEWithLogitsLoss() initiates a class first and calls torch.nn.functional.binary_cross_entropy_with_logits()  when forward is called:</p>
<pre><code class="lang-auto">criterion = torch.nn.BCEWithLogitsLoss(pos_weight=pos_weight)
criterion(output, target)
</code></pre>
<p>For more detailed explanations on the differences between torch.nn and torch.nn.functional, checkout <a href="https://discuss.pytorch.org/t/what-is-the-difference-between-torch-nn-and-torch-nn-functional/33597">this</a> thread here.</p>