How to calculate epoch loss when using BCEWithLogitsLoss

username: se7en.dreams
cooked: <p>Hello, I’m a bit confused about how to accumulate the batch losses to obtain the epoch loss.</p>
<p>Two questions:</p>
<ol>
<li>Is <code>#1</code> (see comments below) correct way to calculate loss with masks)</li>
<li>Is <code>#2</code> correct way to report epoch loss)</li>
</ol>
<pre><code class="lang-python">optimizer = torch.optim.Adam(model.parameters, lr=1e-3, weight_decay=1e-5)
criterion = torch.nn.BCEWithLogitsLoss(pos_weight=pos_weight)

for epoch in range(10):
    EPOCH_LOSS = 0.
    
    for inputs, gt_labels, masks in training_dataloader:
        optimizer.zero_grad()
        outputs = model(inputs)
        
        #1: Is this the correct way to calculate batch loss? Do I multiply batch_loss with outputs.shape[0[ before adding it to epoch_loss?
        batch_loss = (masks * criterion(outputs, gt_labels.float())).mean()
        EPOCH_LOSS += batch_loss
        loss.backward()
        optimizer.step()

    #2: then what do I do here? Do I divide the EPOCH_LOSS with len(training_dataloader)?
    print(f'EPOCH LOSS: {EPOCH_LOSS/len(training_dataloader)}:.3f')
</code></pre>
<p><a class="mention" href="/u/ptrblck">@ptrblck</a> <a class="mention" href="/u/pargg">@ParGG</a></p>

username: thecho7
cooked: <ol>
<li>
<p><code>BCEWithLogitsLoss</code> returns a float tensor having a single element unless you call it with <code>reduction='none'</code>.<br>Would you explain a bit more about what <code>masks</code> does in your model?</p>
</li>
<li>
<p>Since <code>batch_loss</code> is a tensor, it is recommended to use <code>EPOCH_LOSS += batch_loss.item()</code> instead of <code>EPOCH_LOSS += batch_loss</code>.</p>
</li>
<li>
<p>What I know is the length of dataloader(generator) is determined to <code>round(len(dataset) / batch_size)</code>. <code>EPOCH_LOSS / len(dataset)</code> would be correct.</p>
</li>
</ol>

username: se7en.dreams
cooked: <p>Thanks for your response.</p>
<ol>
<li>
<p>So my outputs shape is <code>(14, 10, 128)</code>, where <code>14</code> is the <code>batch_size</code>, <code>10</code> is the <code>seq_len</code>, and <code>128</code> is the object vector where if an element in sequence belongs to any of <code>128</code> objects, it is marked as <code>1</code> and <code>0</code> otherwise. The mask tells us the true length of the sequences. So, its shape is <code>(14, 10)</code>. For instance, the first sequence might only have <code>3</code> elements (so it’s true shape would be <code>3 x 128</code>), and the rest (<code>7 x 128</code>) is just padding.</p>
</li>
<li>
<p>So basically, I should divide it by <code>len(dataloader.dataset)</code>?</p>
</li>
</ol>

username: ParGG
cooked: <aside class="quote no-group" data-username="se7en.dreams" data-post="3" data-topic="156485">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/a4c791/48.png" class="avatar"> se7en.dreams:</div>
<blockquote>
<p>So basically, I should divide it by <code>len(dataloader.dataset)</code>?</p>
</blockquote>
</aside>
<p>Not always. As you can see from the implementation, the length of the dataset depends on a few factors.</p>
<pre><code class="lang-auto">    def __len__(self) -&gt; int:
        if self._dataset_kind == _DatasetKind.Iterable:
            # NOTE [ IterableDataset and __len__ ]
            #
            # For `IterableDataset`, `__len__` could be inaccurate when one naively
            # does multi-processing data loading, since the samples will be duplicated.
            # However, no real use case should be actually using that behavior, so
            # it should count as a user error. We should generally trust user
            # code to do the proper thing (e.g., configure each replica differently
            # in `__iter__`), and give us the correct `__len__` if they choose to
            # implement it (this will still throw if the dataset does not implement
            # a `__len__`).
            #
            # To provide a further warning, we track if `__len__` was called on the
            # `DataLoader`, save the returned value in `self._len_called`, and warn
            # if the iterator ends up yielding more than this number of samples.

            # Cannot statically verify that dataset is Sized
            length = self._IterableDataset_len_called = len(self.dataset)  # type: ignore[assignment, arg-type]
            if self.batch_size is not None:  # IterableDataset doesn't allow custom sampler or batch_sampler
                from math import ceil
                if self.drop_last:
                    length = length // self.batch_size
                else:
                    length = ceil(length / self.batch_size)
            return length
        else:
            return len(self._index_sampler)
</code></pre>
<ul>
<li>If you have specified the <code>bath_size</code> and <code>drop_last</code> is true: you have to divide by <code>len(dataloader) * batch_size</code></li>
<li>If you have specified the <code>bath_size</code> and <code>drop_last</code> is false: you have to divide by <code>len(dataset)</code></li>
<li>If you didn’t specify the <code>batch_size</code>: you have to look at the <code>sampler</code> or <code>batch_sampler</code></li>
</ul>

username: thecho7
cooked: <p>In that case, you have two choices</p>
<ol>
<li>
<code>masks</code> is applied to <code>outputs</code> directly before calculating the loss.</li>
<li>Create an BCE loss instance with an argument <code>reduction='none'</code> and do masking after calculating the loss.</li>
</ol>
<p>Here’s a simple strategy how to apply a mask to the result.<br>
Let me assume <code>gt_labels</code> has <code>(3 x 1)</code> then,</p>
<pre><code class="lang-auto">B, L = masks.size()
outputs = outputs.view(-1, outputs.shape[-1]) # (B, L_true, D) -&gt; (B x L_true, D)
masks = torch.argwhere(masks.view(-1)).squeeze() # (B, L) -&gt; (B x L)

masked_outputs = torch.index_select(input=outputs, dim=0, index=masks) # (B x L, D)
masked_outputs = masked_outputs.view(B, L, -1)

batch_loss = criterion(masked_outputs, gt_labels.float())
</code></pre>
<p>If anyone knows more effective way to do masking, need your idea</p>

username: ParGG
cooked: <aside class="quote no-group" data-username="thecho7" data-post="5" data-topic="156485">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/t/eada6e/48.png" class="avatar"> thecho7:</div>
<blockquote>
<p>If anyone knows more effective way to do masking, need your idea</p>
</blockquote>
</aside>
<p>I already replied to him about this topic on a different post: <a href="https://discuss.pytorch.org/t/filter-output-using-mask/156389" class="inline-onebox">Filter Output Using Mask</a>.</p>

username: se7en.dreams
cooked: <p>Thank you for your reply. Shouldn’t we also have <code>masked_gt_labels</code> before calculating loss?</p>

username: ParGG
cooked: <p>In his case not, because he selects just the valid data given the mask. If instead you would compute the loss for your original outputs and labels, then you would need to mask the output.</p>