How is the loss for an epoch reported?

stark: <p>Let’s assume, there are 10 examples in a dataset and the batch size is 4.</p>
<p>The number of exmaples per batch and training loss will be:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/5/7/5761bb91eadac6a609f20371f9efb392a44e61ff.png" alt="image" data-base62-sha1="ct0VJM2NMTZ4q0pOp1Jm8JZ7Fmf" width="199" height="94"><br>
<em><strong>Note:</strong> By default <code>nn.MSELoss()</code> returns mean loss for an entire batch!</em></p>
<p><strong>Which of these is the loss for an epoch?</strong></p>
<p><strong>A.</strong>  Mean of individual losses over the training dataset:</p>
<pre><code class="lang-auto">Σ(mean_batch_loss * batch_size) / Σ(batch_size)
= ((27.527*4)+(10.503*4)+(5.6534*2)) / (4+4+2)
= (110.11 + 42.0116 + 11.3068) / 10
= 163.428 / 10 = 16.3428
</code></pre>
<p><strong>B.</strong> Mean of the mean batch losses:</p>
<pre><code class="lang-auto">Σ mean_batch_loss / total_batches
= (27.527 + 10.503 + 5.6534*2) / total_batches
= 43.6837 / 3 
= 14.5612
</code></pre>
<p><strong>C.</strong> Exponentially weighted moving average (EWMA)<br>
<code>s(i) = a * x(i) + (1-a) * x(i-1)</code><br>
where <code>a</code> is a smoothing factor set to <code>0.1</code> and <code>s(0) = 27.527</code><br>
s(0) = 27.527<br>
s(1) = 25.825<br>
s(2) = 23.808</p>
<p><strong>D.</strong> Differently than above</p>
<p><strong>Please also comment on the way validation loss is computed differently.</strong></p>

tom: <p>Good question!</p>
<p>I would view A as the correct way of doing it and B as a (common) approximation. If the number of batches is large and the last error isn’t particularly off, it might not matter as much.<br>
One thing to keep in mind is that during training, the loss is also changed (hopefully decreasing) by the updates. This makes, in my view, some sort of moving average, like <a href="https://en.wikipedia.org/wiki/Moving_average#Exponential_moving_average" rel="nofollow noopener">exponentially weighted moving average</a> a contender for C. Depending on how fast the decay is, the last batch behaviour may be a bit more touchy there. In validation, on the other hand, the there aren’t updates and so there isn’t that much reason to use a moving average scheme.<br>
I don’t think that the loss calculation itself is inherently different for validation (it would defeat the purpose a bit, too), but note that the way the model operates (dropout, batch norm) may be different.</p>
<p>The other thing to keep in mind is that small batch size may have other implications, e.g. for batch norm, adaptive optimizers like adam etc. If you have doubts, it may be safer to drop the last batch.</p>
<p>Best regards</p>
<p>Thomas</p>

stark: <p>Thanks for the reply <img src="https://discuss.pytorch.org/images/emoji/apple/hugs.png?v=12" title=":hugs:" class="emoji" alt=":hugs:" loading="lazy" width="20" height="20"></p>
<p>I’ve included your suggestion about <code>exponentially weighted moving average</code> in the original post as the <code>option C</code>.</p>
<aside class="quote no-group" data-username="stark" data-post="1" data-topic="94295">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/stark/48/29013_2.png" class="avatar"> stark:</div>
<blockquote>
<p><strong>C.</strong> Exponentially weighted moving average (EWMA)<br>
<code>s(i) = a * x(i) + (1-a) * x(i-1)</code><br>
where <code>a</code> is a smoothing factor set to <code>0.1</code> and <code>s(0) = 27.527</code><br>
s(0) = 27.527<br>
s(1) = 25.825<br>
s(2) = 23.808</p>
</blockquote>
</aside>
<p>Don’t you think the final EWMA value i.e. <code>s(2) = 23.808</code> is an overestimate when compared with the final <code>epoch loss</code> obtained from each other method (16.342 and 14.561)? Another issue could be finding an optimal value for the smoothing parameter <code>a</code>.</p>

tom: <p>The vanilla version is not really good for very short sequences, and there is a commonly used scheme to deal with the beginning (where you keep a MA of the weight/batch size as well as the value and then you divide the two, sometimes dubbed debiasing).<br>
While this has a lot of warts, it does serve the purpose of giving a meaningful estimate if we believe that the value changes over the course of the training.</p>

stark: <p><a class="mention" href="/u/tom">@tom</a> Thanks for the reply!</p>
<blockquote>
<p>where you keep a MA of the weight/batch size as well as the value and then you divide the two, sometimes dubbed debiasing</p>
</blockquote>
<p>Do you mean a <code>running_loss</code> by <em>moving average (MA)</em> like it is implemented in the following example?  (from <a href="https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html#train-the-network" class="inline-onebox" rel="noopener nofollow ugc">Training a Classifier — PyTorch Tutorials 2.1.1+cu121 documentation</a>).</p>
<p>In case your reply is positive then it is <strong>Option A.</strong> in my initial post.</p>
<pre><code class="lang-auto">for epoch in range(total_epochs): 
    running_loss = 0.0
    for i, data in enumerate(trainloader, 0):
        inputs, labels = data
        optimizer.zero_grad()
        outputs = net(inputs)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()

        # print statistics
        running_loss += loss.item()
        if i % 2000 == 1999:    # print every 2000 mini-batches
            print('[%d, %5d] loss: %.3f' % (epoch + 1, i + 1, running_loss / 2000))
            running_loss = 0.0

</code></pre>

tom: <p>No, the thing I had in mind as C is what is done e.g. in the optimizers or discussed (not quite as clearly as I would need to understand it) on the <a href="https://en.wikipedia.org/wiki/Moving_average#Exponential_moving_average" rel="noopener nofollow ugc">moving average wikipedia page</a>, see the paragraph starting with  <em>It can also be calculated recursively without introducing the error …</em>.</p>
<p>This isn’t the average over the entire dataset as it weights the later losses more than the former.</p>
<p>Best regards</p>
<p>Thomas</p>

neoglez: <p>I just want to point out that there is small error in B. substitution. That confused me a little, so I actually checked it with the calculator.<br>
<code>(27.527 + 10.503 + 5.6534*2) / total_batches</code>: While the formula is correct, the multiplication by two in the numerator is wrong. Also, the result (14.5612) is correct.<br>
If the multiplication by two would be correct, the result should have been 16.44.<br>
So, the correct substitution is: <code>(27.527 + 10.503 + 5.6534) / total_batches</code><br>
<code>= (27.527 + 10.503 + 5.6534) / 3</code><br>
<code>= 14.5612</code></p>