Problems implementing my own loss

xolotl18: <p>Hi everyone,<br>
I am having problems implementing my own version of the hinge loss function.<br>
In particular it works but after the first epoch the loss computed is zero.<br>
I am pretty sure that I messed up something related to the dimension of the input data of the function.<br>
Here is the code, I would really appreciate your help, I am kinda losing my mind.<br>
ps: the network is a resnet and the main problem is image classification of cifar100</p>
<pre><code class="lang-python">class MultiClassSquaredHingeLoss(nn.Module):
    def __init__(self):
        super(MultiClassSquaredHingeLoss, self).__init__()

    def forward(self, output, y): #output: batchsize*n_class
        n_class = y.size(1)
        #margin = 1 
        margin = 1
        #isolate the score for the true class
        y_out = torch.sum(torch.mul(output, y)).cuda()
        output_y = torch.mul(torch.ones(n_class).cuda(), y_out).cuda()
        #create an opposite to the one hot encoded tensor
        anti_y = torch.ones(n_class).cuda() - y.cuda()
        
        loss = output.cuda() - output_y.cuda() + margin
        loss = loss.cuda()
        #remove the element of the loss corresponding to the true class
        loss = torch.mul(loss.cuda(), anti_y.cuda()).cuda()
        #max(0,_)
        loss = torch.max(loss.cuda(), torch.zeros(n_class).cuda())
        #squared hinge loss
        loss = torch.pow(loss, 2).cuda()
        #sum up
        loss = torch.sum(loss).cuda()
        loss = loss / n_class        
        
        return loss
</code></pre>

albanD: <p>Hi,</p>
<p>Could it be that the first gradient step pushes everything on the flat side of the hinge and so it just returns 0?</p>
<p>Also few question:</p>
<ul>
<li>Why so many <code>.cuda()</code> calls? You should not need them if everything is already on the GPU. If everything is on the CPU, you might not want to shuffle things around that much for perf reasons.</li>
<li>
<code>torch.mul(torch.ones(n_class).cuda(), y_out)</code> is the same as <code>y_out</code> right?</li>
<li>
<code>anti_y = torch.ones(n_class).cuda() - y.cuda()</code> can be done as <code>anti_y = 1 - y.cuda()</code> to make it simpler to read and more efficient</li>
<li>
<code>torch.zeros(n_class).cuda()</code> can be replaced by <code>torch.zeros(n_class, device="cuda")</code> if you want.</li>
<li>
<code>torch.max(loss.cuda(), torch.zeros(n_class).cuda())</code> -&gt; <code>torch.threshold(loss, 0, 0)</code>
</li>
</ul>

xolotl18: <p>thank you for answering!<br>
The cuda calls are there because it kept giving me errors about something tha was on the cpu, while it expected cuda and I tried them everywhere.</p>
<p><code>torch.mul(torch.ones(n_class).cuda(), y_out)</code> is supposed to be a tensor of size n_class that has all values equal to y_out, which is the value of the score for the true class (probabily there is a more elegant solution to do it, I kinda learned python by trial and error…)</p>
<p>do you have any advice on how to avoid the gradient pushing everything on the flat side?</p>

albanD: <p>Well, that’s your goal. Once the hinge is all 0s, then you perfectly classify all your samples. So your job is done <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>

xolotl18: <p>the issue is that the performance is really poor</p>

xolotl18: <p>would you implement it differently?</p>

albanD: <aside class="quote no-group" data-username="xolotl18" data-post="5" data-topic="85813" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/xolotl18/48/25439_2.png" class="avatar"> xolotl18:</div>
<blockquote>
<p>the issue is that the performance is really poor</p>
</blockquote>
</aside>
<p>You mean classification? It should be perfect on your training set.</p>

xolotl18: <p>the problem is that it is very bad on the training set. after the first epoch the accuracy is 0.09 and then the loss goes to 0 and stays there and then of course the training doesn’t go on because it thinks it has reached perfection</p>

albanD: <p>In that case I guess your implementation is not correct. Did you checked on simple examples that it actually computes the multi class squared hinge loss?<br>
Also you can try to compare it to other implementations online (google search returned: <a href="https://github.com/HaotianMXu/Multiclass_LinearSVM_with_SGD">https://github.com/HaotianMXu/Multiclass_LinearSVM_with_SGD</a> for example).</p>

xolotl18: <p>I based my implementation on that repository, I will try swapping them and seeing if I can make it work</p>