Multilabelmarginloss

username: xolotl18
cooked: <p>Hi everyone,<br>
I seem to have problems making this loss function work.<br>
I feed it the output of the network and the labels from the dataset and after the first epoch it says that the loss is 0, but the accuracy is still low.<br>
Do you know where the problem should be?</p>

username: ptrblck
cooked: <p>Could you post the code you are using to calculate the accuracy as well as an example tensor for your model output and the corresponding target, please?</p>

username: xolotl18
cooked: <p>sure, here they are:</p>
<pre><code class="lang-auto">tensor([[-0.4108,  2.3717, -1.2828,  ...,  0.1713,  1.2362, -1.3406],
        [-0.2165,  1.2525, -0.5553,  ...,  0.5631,  0.7271, -0.7836],
        [-0.3744,  1.2202, -0.3709,  ...,  0.3243,  0.6756, -0.9807],
        ...,
        [-0.5486,  1.7626, -0.7162,  ...,  0.2115,  0.9654, -1.0366],
        [-0.7164,  1.5161, -0.2281,  ...,  0.5199,  0.7857, -1.1157],
        [-0.3701,  1.3838, -0.5642,  ...,  0.3960,  0.9117, -0.7888]],
       device='cuda:0', grad_fn=&lt;AddmmBackward&gt;)

torch.Size([64, 100])
tensor([86, 67, 17, 30, 90, 60, 51, 84, 59, 30, 50, 21, 34, 45, 38, 80, 37, 81,
         5, 39, 41, 20, 21,  5, 23, 49,  1, 23, 70, 42, 26, 86, 28, 14, 65, 38,
        53, 80, 98, 79, 29, 41,  6, 77, 23, 16, 85, 59, 46, 14, 73, 43, 43, 39,
        72, 18, 61, 63,  1, 38, 67, 86, 84, 51], device='cuda:0')
torch.Size([64])
</code></pre>
<p>actually now it gives an error:</p>
<pre><code class="lang-auto">-&gt; 2606     return torch._C._nn.multilabel_margin_loss(input, target, reduction_enum)
   2607 
   2608 

RuntimeError: inconsistent target size: [64] for input of size: [64, 100]
</code></pre>
<p>It did not give that error when I used one hot encoded labels, but reading the implementation of the loss one hot labels do not make sense.<br>
Thank you for your help</p>

username: ptrblck
cooked: <p>Based on the <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.MultiLabelMarginLoss">docs</a>, the output and target should have the same shape.<br>
If I understand it correctly, each row of the target should contain all labels for the current sample, where a <code>-1</code> target marks the end of all valid labels.</p>
<p>E.g. if your first sample contains the valid labels <code>3</code> and <code>0</code>, the example label from the docs would be:</p>
<pre><code class="lang-python">&gt;&gt;&gt; # for target y, only consider labels 3 and 0, not after label -1
&gt;&gt;&gt; y = torch.LongTensor([[3, 0, -1, 1]])
</code></pre>
<p>Are you sure your use case is a multi-label classification (each sample can belong to more that a single class) and not a multi-class classification (each sample belongs to one class only)?</p>

username: xolotl18
cooked: <p>you are right, I got confused. I need a multiclass classification loss, not a multi label. Is there an implementation of multiclass hinge loss that I can use? Is MultiMarginLoss the one that I should use?</p>

username: ptrblck
cooked: <p>I think <code>nn.MultiMarginLoss</code> would be the suitable criterion:</p>
<blockquote>
<p>Creates a criterion that optimizes a multi-class classification hinge loss (margin-based loss) between input x (a 2D mini-batch Tensor) and output y</p>
</blockquote>
<p>Based on the shape information it should also work for your current output and target shapes.</p>
<p>Let me know, if it would work for you.</p>

username: xolotl18
cooked: <p>Thank you very much, as soon as I try it I will let you know</p>

username: xolotl18
cooked: <p>It works. But when I tried implementing a squared version of this hinge loss, after the first time it computes the loss it goes to 0, even if the training accuracy is very low.<br>
here is the class I implemented:</p>
<pre><code class="lang-auto">class SqHinge(nn.Module):
  def __init__(self):
        super(SqHinge, self).__init__()

  def forward(self, y_pred, y_true):
        criterion = nn.MultiMarginLoss()
        l1 = criterion(y_pred, y_true)
        l2 = l1*l1
        return l2
</code></pre>
<p>I tried using torch.pow, but the result was the same</p>

username: ptrblck
cooked: <p>The squared loss should be a 0 only if the <code>l1</code> loss is also a 0, shouldn’t it?<br>
For small values &lt;1 it would reduce the loss, but shouldn’t output a perfect 0 loss.</p>

username: xolotl18
cooked: <p>That is what I thought, but it goes to 0 after the first batch. I actually need to confirm the exact moment it goes to 0, maybe that will give a better insight on the problem. Do you see anything wrong with the implementation above?</p>