Why the min loss is not zero in neither of MultiLabelSoftMarginLoss and BCEWithLogitsLoss

seyeeet: <p>I am trying to understand how to use <a href="https://pytorch.org/docs/stable/generated/torch.nn.MultiLabelSoftMarginLoss.html" rel="noopener nofollow ugc">MultiLabelSoftMarginLoss</a> and <a href="https://pytorch.org/docs/stable/generated/torch.nn.BCEWithLogitsLoss.html" rel="noopener nofollow ugc">BCEWithLogitsLoss</a>.<br>
my question is in two parts, for simplicity I consider the case of multi-lable classification for images.</p>
<ol>
<li>
<p>what should be the format for the targets? should it be <code>0</code> and <code>1</code> where <code>0</code> means that the input does not contain that category and <code>1</code> means that the input includes that category?</p>
</li>
<li>
<p>why the minimum of neither of those two losses is not <code>zero</code> when I give them the input equal to targets? shouldn’t that be the case? is it a bug or I am missing something?</p>
</li>
</ol>
<p>Here is an example, where  we have in total 6 categories, and the input has objects from categories <code>1,3,5</code> out of <code>6</code> categories. So we have our target for this image define as follows (please let me know if the target is defined incorrectly).</p>
<pre><code class="lang-auto">BCEWithLogitsLoss = nn.BCEWithLogitsLoss()
MultiLabelSoftMarginLoss= nn.MultiLabelSoftMarginLoss()

target= torch.tensor([[1,0,1,0,1,0]],dtype=torch.float32) 
output= torch.zeros([1,6])
output[0,0]=1
output[0,2]=1
output[0,4]=1
output.requires_grad=True
loss_1 = BCEWithLogitsLoss(output,target) # loss_1 will be equal to 0.5032
loss_2 = MultiLabelSoftMarginLoss(output,target) # loss_2 will be equal to 0.5032
</code></pre>
<p>shouldn’t the losses be <code>zero</code>? what I am missing here? why they are even equal? what should be the min value if not zero?</p>

seyeeet: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> and <a class="mention" href="/u/alband">@albanD</a> appreciate your thoughts on this <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

KFrank: <p>Hi Seyeeet!</p>
<aside class="quote no-group" data-username="seyeeet" data-post="1" data-topic="123434" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/seyeeet/48/21931_2.png" class="avatar"> seyeeet:</div>
<blockquote>
<ol start="2">
<li>why the minimum of neither of those two losses is not <code>zero</code> when I give them the input equal to targets?</li>
</ol>
</blockquote>
</aside>
<p><code>BCEWithLogitsLoss</code> takes raw-score <em>logits</em> (that run from <code>-inf</code> to<br>
<code>inf</code>) as its <code>input</code>, but takes <em>probabilities</em> (<code>0.0</code> to <code>1.0</code>) as its <code>target</code>.</p>
<p>Instead of using <code>0.0</code> and <code>1.0</code> for your input, use very large negative<br>
and positive values.  Then you will get a loss of (something close to)<br>
<code>0.0</code>.</p>
<p>(In general, you shouldn’t use <code>BCELoss</code>, but if you apply it to your<br>
example values, you will get <code>0.0</code> for the loss.)</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">loss_1 = BCEWithLogitsLoss(output,target) # loss_1 will be equal to 0.5032
loss_2 = MultiLabelSoftMarginLoss(output,target) # loss_2 will be equal to 0.5032
</code></pre>
<p>why they are even equal?</p>
</blockquote>
</aside>
<p><code>BCEWithLogitsLoss</code> and <code>MultiLabelSoftMarginLoss</code> are essentially<br>
the same.  See, for example:</p>
<aside class="quote quote-modified" data-post="1" data-topic="14944">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/d/dfb087/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/what-is-the-difference-between-bcewithlogitsloss-and-multilabelsoftmarginloss/14944">What is the difference between BCEWithLogitsLoss and MultiLabelSoftMarginLoss</a> 
  </div>
  <blockquote>
    I think there is no difference between BCEWithLogitsLoss and MultiLabelSoftMarginLoss. 
BCEWithLogitsLoss = One Sigmoid Layer + BCELoss (solved numerically unstable problem) 
MultiLabelSoftMargin’s fomula is also same with BCEWithLogitsLoss. 
One difference is BCEWithLogitsLoss has a ‘weight’ parameter, MultiLabelSoftMarginLoss no has) 
BCEWithLogitsLoss  : 
[image] 
MultiLabelSoftMarginLoss  : 
[image] 
The two formula is exactly the same except for the weight value.
  </blockquote>
</aside>

<p>Best.</p>
<p>K. Frank</p>