How to pass weights to BCELoss()

pascalm: <p>Hello,</p>
<p>I have code like this:</p>
<pre><code class="lang-auto">loss_fn = reg_BCELoss(dim=2)

loss = 0
def train(...loss_fn):
    ...
    for i_batch, (samples, labels) in enumerate(TrainDL):
        ...
        loss_batch = loss_fn(labels_pred, labels)
        loss += loss_batch.item()
        ...
</code></pre>
<p>Now I’d like to set weights inside my training loop but apparently only the constructor can set weights which means, I’d have to create the loss function object inside the loop. Which means, I’d create thousands of objects.</p>
<p>So, what do I do? Do I try to do something stupid?</p>

sagsriv: <p>the weights in the <code>BCELoss</code> could typically be determined before you start your training loop because it usually depends on the label distribution in the <em>whole</em> training data and not in each minibatch.</p>

mMagmer: <p>quote form <a href="https://pytorch.org/docs/stable/generated/torch.nn.BCELoss.html" rel="noopener nofollow ugc">bceloss</a>.</p>
<blockquote>
<ul>
<li><strong>weight</strong> (<a href="https://pytorch.org/docs/stable/tensors.html#torch.Tensor" rel="noopener nofollow ugc"><em>Tensor</em></a><em>,</em> <em>optional</em>) – a manual rescaling weight given to the loss of each batch element. If given, has to be a Tensor of size nbatch.</li>
</ul>
</blockquote>
<p>do you want to use this feature or you want to have weight for each class?</p>

pascalm: <p>How do I write latex in this forum?</p>
<p>BCELoss() takes two vectors: x and y. Each of size N. We then have:</p>
<p>l(x, y) = L = {l_1, …, l_N} wheras</p>
<p>l_n = -w_n * [y_n * log(x_n) + (1- y_n) *log(1 - x_n)]</p>
<p>which is linear. Once we got L we make a reduction. This reduction can be the mean or a sum.</p>
<p>This is the reason people apply the BCELoss() batch wise. The reduction is what puts the information from the whole set into a statement.</p>
<p>In short: I can’t see your argument and I can’t see why there is no weights function in the forward function. I mean, what I could do is simple:</p>
<ol>
<li>Call BCELoss() with reduction=‘none’ and implement my own reduction.</li>
</ol>
<p>But still, I must be missing something because if my argument would be correct, I could provide w_n to forward().</p>

pascalm: <p>What do you mean by class? In any case: I don’t want to pass the weights to the constructor but to the forward function. See argument in my other post. The reason for that is simple: I don’t want to go through GB and GB of data and pass several hundred of megabytes of information to one argument. Just seems wrong. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

mMagmer: <p>you can change <code>loss_fn.weight</code>.<br>
also you can use<br>
F.binary_cross_entropy(input, target, weight=weight, reduction=reduction).<br>
<code>F</code> is <code>torch.nn.functional</code></p>

pascalm: <p>It feels dirty to me to alter a value set by the constructor. Is that really how python people design such a thing?</p>

sagsriv: <p>You can calculate the loss, and before doing the reduction over the batch, multiply it with the scaling factor, and then do the reduction. Here’s something implemented:</p>
<aside class="onebox githubblob" data-onebox-src="https://github.com/dpressel/mead-baseline/blob/master/layers/eight_mile/pytorch/layers.py#L4693">
  <header class="source">

      <a href="https://github.com/dpressel/mead-baseline/blob/master/layers/eight_mile/pytorch/layers.py#L4693" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/dpressel/mead-baseline/blob/master/layers/eight_mile/pytorch/layers.py#L4693" target="_blank" rel="noopener nofollow ugc">dpressel/mead-baseline/blob/master/layers/eight_mile/pytorch/layers.py#L4693</a></h4>



  <pre class="onebox">    <code class="lang-py">
      <ol class="start lines" start="4683" style="counter-reset: li-counter 4682 ;">
          <li>    """</li>
          <li>    def __init__(self):</li>
          <li>        super().__init__()</li>
          <li>        self.loss = nn.NLLLoss(reduction='none')</li>
          <li>
          </li>
<li>    def forward(self, preds, targets, weights):</li>
          <li>        loss = sum([self.loss(pred, targets[:, i]) for i, pred in enumerate(preds)])</li>
          <li>        weights = weights.type_as(loss)</li>
          <li>        return torch.dot(loss, weights)/len(weights)</li>
          <li>
          </li>
<li class="selected">class WeightedSequenceLoss(nn.Module):</li>
          <li>    """Weight individual training examples</li>
          <li>
          </li>
<li>    """</li>
          <li>    def __init__(self, LossFn: nn.Module = nn.NLLLoss, avg: str = "token"):</li>
          <li>        super().__init__()</li>
          <li>        self.avg = avg</li>
          <li>        self.crit = LossFn(ignore_index=Offsets.PAD, reduction="none")</li>
          <li>        if avg == 'token':</li>
          <li>            self._reduce = self._mean</li>
          <li>        else:</li>
      </ol>
    </code>
  </pre>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>This is for sequence though, you probably have <code>BxC</code> and not <code>BxTxC</code>, but I think the idea is similar.</p>

James_Warner: <p>I know this is old news now - but I’m running into this now and feel your pain. I wonder how anyone uses weights with BCELoss successfully? Needing to know the weights a priori for each element of each batch to pass to the constructor is such a weird constraint to have…</p>

J_Johnson: <p>I’m using it like this between each batch on a project:</p>
<pre><code class="lang-auto">criterion = nn.BCELoss()
...
# after getting labels
if targets.sum(0) == 0 | (1 - targets).sum(0)== 0:
    continue

criterion.pos_weight = (1 - targets).sum(0) / targets.sum(0)

...
</code></pre>
<p>The if statement is to avoid divide by zero and any potential anomolies from the loss function internals.</p>