When and how shall we use the "torch.nn.MultiMarginLoss"?"

username: crissallan
cooked: <ol>
<li>
<p>Can I use the <code>torch.nn.</code> <code>MultiMarginLoss</code> instead of using the cross-entropy loss for a classification task?</p>
</li>
<li>
<p>Should I apply softmax function to the logits before I send the logits into the “MultiMarginLoss” to make the logits as normalized probobility?</p>
</li>
</ol>

username: tom
cooked: <aside class="quote no-group" data-username="crissallan" data-post="1" data-topic="138314">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/crissallan/48/15254_2.png" class="avatar"> crissallan:</div>
<blockquote>
<p>Can I use the <code>torch.nn.</code> <code>MultiMarginLoss</code> instead of using the cross-entropy loss for a classification task?</p>
</blockquote>
</aside>
<p>Yes.</p>
<aside class="quote no-group" data-username="crissallan" data-post="1" data-topic="138314">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/crissallan/48/15254_2.png" class="avatar"> crissallan:</div>
<blockquote>
<p>Should I apply softmax function to the logits before I send the logits into the “MultiMarginLoss” to make the logits as normalized probobility?</p>
</blockquote>
</aside>
<p>I think typically not. Speaking of which: I would not call them “logits” for this reason but “score” or so because the former always has a “unnormalized log probabilities” ring to me, which is not the case here.</p>
<p>Best regards</p>
<p>Thomas</p>

username: crissallan
cooked: <p>Thanks, I tried it and works. I also had another question: compared with the CrossEntropy, what is the advantages and disadvantages of using the Multi-Margin loss when doing a classification task?</p>

username: InnovArul
cooked: <p>Could you share your observation on the performance when using <code>MultiMarginLoss</code> vs. <code>CrossEntropy</code>? Does it perform better / create better embeddings?</p>

username: tom
cooked: <p>My basic take on this is:</p>
<ul>
<li>Using softmax + negative log likelihood (which combines to CrossEntropyLoss in PyTorch lingo) has the charming property of having NLL being interpretable thanks to the “likelihood” (and the entire training to be a maximum likelihood fitting of sorts). It has proven to work over and over again, typically with number of classes &lt;1000. It seems to not work as well for problems with many more classes.</li>
<li>Using margin losses (which to me always has a <a href="https://en.wikipedia.org/wiki/Support-vector_machine" rel="noopener nofollow ugc">SVM association</a> has the advantage of very directly dealing with the negative examples and I look at it as being less prone to saturation effects than softmax in this when you have many classes. <em>Edit:</em> Also, the margin doesn’t have this push to extreme scores (“overconfidence” if you use the outputs of softmax as probabilities).</li>
</ul>
<p>But you should take this with a lot of caution (“random person on the internet says”) and do your own experimentation for the problem you want to solve. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>Best regards</p>
<p>Thomas</p>

username: crissallan
cooked: <p>I would say it works in my task.But it had a similar performance compared with the CrossEntropy</p>

username: InnovArul
cooked: <p>oh ok. Thanks for sharing the observation in your task.</p>