Binary Loss mean

username: Jaideep_Valani
cooked: <p>Assuming  size of predictions are<br>
bs *  classes   in case multilabel multiclass classifier.</p>
<p>How does  nn.BCEWithLogits performs the mean in this  case…<br>
is it simply  divide by numel ?</p>
<p>What about the case when we give weights to bce with logits function … would it<br>
Sum weighted rows / size of batch  or it would does it like case one /numel</p>
<p>If in case ml m class classificaion  numel could diminish loss  in case there are more number of classes, so doing weighted sum/bs should be more optimal…</p>
<p>Please let me know how bce does a mean in case of mult label mult class</p>

username: KFrank
cooked: <p>Hi Jaideep!</p>
<aside class="quote no-group" data-username="Jaideep_Valani" data-post="1" data-topic="100570" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jaideep_valani/48/8482_2.png" class="avatar"> Jaideep_Valani:</div>
<blockquote>
<p>How does  nn.BCEWithLogits performs the mean in this  case…<br>
is it simply  divide by numel ?</p>
</blockquote>
</aside>
<p>Yes.</p>
<aside class="quote no-group">
<blockquote>
<p>Sum weighted rows / size of batch</p>
</blockquote>
</aside>
<p>I’m not sure what you mean by “sum weighted rows.”</p>
<aside class="quote no-group">
<blockquote>
<p>or it would does it like case one /numel</p>
</blockquote>
</aside>
<p>Yes.  First the individual per-sample, per-class loss values are<br>
multiplied by the corresponding weights, then summed, and then<br>
divided by <code>numel()</code>.</p>
<p>The following script illustrates these behaviors:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
torch.__version__
torch.random.manual_seed (2020)

inp = torch.randn ((3, 5))
trg = torch.randn ((3, 5)).sigmoid()
torch.nn.BCEWithLogitsLoss (reduction = 'none') (inp, trg)
torch.nn.BCEWithLogitsLoss (reduction = 'none') (inp, trg).sum()
torch.nn.BCEWithLogitsLoss (reduction = 'none') (inp, trg).sum() / inp.numel()
torch.nn.BCEWithLogitsLoss (reduction = 'sum') (inp, trg)
torch.nn.BCEWithLogitsLoss (reduction = 'mean') (inp, trg)

# weights per sample (in batch) and class
wta = torch.randn ((3, 5)).abs()
torch.nn.BCEWithLogitsLoss (reduction = 'none') (inp, trg) * wta
(torch.nn.BCEWithLogitsLoss (reduction = 'none') (inp, trg) * wta).sum()
(torch.nn.BCEWithLogitsLoss (reduction = 'none') (inp, trg) * wta).sum() / inp.numel()
torch.nn.BCEWithLogitsLoss (weight = wta, reduction = 'none') (inp, trg)
torch.nn.BCEWithLogitsLoss (weight = wta, reduction = 'sum') (inp, trg)
torch.nn.BCEWithLogitsLoss (weight = wta, reduction = 'mean') (inp, trg)

# weights just per class
wtb = torch.randn ((5,)).abs()
torch.nn.BCEWithLogitsLoss (reduction = 'none') (inp, trg) * wtb
(torch.nn.BCEWithLogitsLoss (reduction = 'none') (inp, trg) * wtb).sum()
(torch.nn.BCEWithLogitsLoss (reduction = 'none') (inp, trg) * wtb).sum() / inp.numel()
torch.nn.BCEWithLogitsLoss (weight = wtb, reduction = 'none') (inp, trg)
torch.nn.BCEWithLogitsLoss (weight = wtb, reduction = 'sum') (inp, trg)
torch.nn.BCEWithLogitsLoss (weight = wtb, reduction = 'mean') (inp, trg)
</code></pre>
<p>Here is the output:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.6.0'
&gt;&gt;&gt; torch.random.manual_seed (2020)
&lt;torch._C.Generator object at 0x7f72eeef96f0&gt;
&gt;&gt;&gt;
&gt;&gt;&gt; inp = torch.randn ((3, 5))
&gt;&gt;&gt; trg = torch.randn ((3, 5)).sigmoid()
&gt;&gt;&gt; torch.nn.BCEWithLogitsLoss (reduction = 'none') (inp, trg)
tensor([[1.0234, 0.5303, 1.3922, 0.6390, 0.4845],
        [0.6889, 0.6799, 0.6828, 0.8319, 1.5794],
        [0.6843, 0.7374, 0.7945, 0.7029, 1.4041]])
&gt;&gt;&gt; torch.nn.BCEWithLogitsLoss (reduction = 'none') (inp, trg).sum()
tensor(12.8555)
&gt;&gt;&gt; torch.nn.BCEWithLogitsLoss (reduction = 'none') (inp, trg).sum() / inp.numel()
tensor(0.8570)
&gt;&gt;&gt; torch.nn.BCEWithLogitsLoss (reduction = 'sum') (inp, trg)
tensor(12.8555)
&gt;&gt;&gt; torch.nn.BCEWithLogitsLoss (reduction = 'mean') (inp, trg)
tensor(0.8570)
&gt;&gt;&gt;
&gt;&gt;&gt; # weights per sample (in batch) and class
&gt;&gt;&gt; wta = torch.randn ((3, 5)).abs()
&gt;&gt;&gt; torch.nn.BCEWithLogitsLoss (reduction = 'none') (inp, trg) * wta
tensor([[1.2065, 0.2057, 0.4710, 0.8512, 0.1281],
        [0.3199, 1.0705, 0.5096, 0.4725, 0.8348],
        [1.3118, 0.1872, 0.9706, 0.0520, 2.5681]])
&gt;&gt;&gt; (torch.nn.BCEWithLogitsLoss (reduction = 'none') (inp, trg) * wta).sum()
tensor(11.1597)
&gt;&gt;&gt; (torch.nn.BCEWithLogitsLoss (reduction = 'none') (inp, trg) * wta).sum() / inp.numel()
tensor(0.7440)
&gt;&gt;&gt; torch.nn.BCEWithLogitsLoss (weight = wta, reduction = 'none') (inp, trg)
tensor([[1.2065, 0.2057, 0.4710, 0.8512, 0.1281],
        [0.3199, 1.0705, 0.5096, 0.4725, 0.8348],
        [1.3118, 0.1872, 0.9706, 0.0520, 2.5681]])
&gt;&gt;&gt; torch.nn.BCEWithLogitsLoss (weight = wta, reduction = 'sum') (inp, trg)
tensor(11.1597)
&gt;&gt;&gt; torch.nn.BCEWithLogitsLoss (weight = wta, reduction = 'mean') (inp, trg)
tensor(0.7440)
&gt;&gt;&gt;
&gt;&gt;&gt; # weights just per class
&gt;&gt;&gt; wtb = torch.randn ((5,)).abs()
&gt;&gt;&gt; torch.nn.BCEWithLogitsLoss (reduction = 'none') (inp, trg) * wtb
tensor([[0.4109, 0.1231, 0.3938, 0.9537, 0.3403],
        [0.2766, 0.1579, 0.1932, 1.2416, 1.1093],
        [0.2747, 0.1712, 0.2247, 1.0490, 0.9862]])
&gt;&gt;&gt; (torch.nn.BCEWithLogitsLoss (reduction = 'none') (inp, trg) * wtb).sum()
tensor(7.9062)
&gt;&gt;&gt; (torch.nn.BCEWithLogitsLoss (reduction = 'none') (inp, trg) * wtb).sum() / inp.numel()
tensor(0.5271)
&gt;&gt;&gt; torch.nn.BCEWithLogitsLoss (weight = wtb, reduction = 'none') (inp, trg)
tensor([[0.4109, 0.1231, 0.3938, 0.9537, 0.3403],
        [0.2766, 0.1579, 0.1932, 1.2416, 1.1093],
        [0.2747, 0.1712, 0.2247, 1.0490, 0.9862]])
&gt;&gt;&gt; torch.nn.BCEWithLogitsLoss (weight = wtb, reduction = 'sum') (inp, trg)
tensor(7.9062)
&gt;&gt;&gt; torch.nn.BCEWithLogitsLoss (weight = wtb, reduction = 'mean') (inp, trg)
tensor(0.5271)
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

username: Jaideep_Valani
cooked: <aside class="quote no-group" data-username="KFrank" data-post="2" data-topic="100570">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<pre><code class="lang-auto">&gt;&gt;&gt; torch.nn.BCEWithLogitsLoss (weight = wtb, reduction = 'mean') (inp, trg)
tensor(0.5271)
</code></pre>
</blockquote>
</aside>
<p><a class="mention" href="/u/kfrank">@KFrank</a> this explains all possible scenarios…</p>
<pre><code class="lang-auto">
&gt;&gt;&gt; torch.nn.BCEWithLogitsLoss (weight = wtb, reduction = 'mean') (inp, trg)
tensor(0.5271)
</code></pre>
<p>my ask is in this case why pytorch chose to mean using numel  when we are doing weighted sum  of class predictions ,where weights are unequal.<br>
Regular mean is also a kind of weighted Sum where each item in pool has got  an equal weight which is by Numel. …<br>
But when weights are unequal like above then further dividing it by Numel could deflate the loss number.s . I have experienced this ,model is not able to learn well…</p>
<p>eg. check this competition metric loss</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://www.kaggle.com/c/rsna-str-pulmonary-embolism-detection/overview/evaluation">
  <header class="source">
      <img src="https://www.kaggle.com/static/images/favicon.ico" class="site-icon" width="32" height="32">

      <a href="https://www.kaggle.com/c/rsna-str-pulmonary-embolism-detection/overview/evaluation" target="_blank" rel="noopener nofollow ugc">kaggle.com</a>
  </header>

  <article class="onebox-body">
    

<h3><a href="https://www.kaggle.com/c/rsna-str-pulmonary-embolism-detection/overview/evaluation" target="_blank" rel="noopener nofollow ugc">RSNA STR Pulmonary Embolism Detection</a></h3>

  <p>Classify Pulmonary Embolism cases in chest CT scans</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>i tried doing a simple mean but it gave very poor results</p>