How to draw loss per epoch

John1231983: <p>I want to draw loss per epoch from the example <a href="https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html#sphx-glr-beginner-blitz-cifar10-tutorial-py" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html#sphx-glr-beginner-blitz-cifar10-tutorial-py</a></p>
<p>The log file is</p>
<pre><code class="lang-auto">Out:

[1,  2000] loss: 2.173
[1,  4000] loss: 1.839
[1,  6000] loss: 1.659
[1,  8000] loss: 1.600
[1, 10000] loss: 1.533
[1, 12000] loss: 1.468
[2,  2000] loss: 1.395
[2,  4000] loss: 1.378
[2,  6000] loss: 1.368
[2,  8000] loss: 1.340
[2, 10000] loss: 1.316
[2, 12000] loss: 1.307
Finished Training

</code></pre>
<p>where the first column is the epoch number. So if I want to draw the loss per epoch, do I need to average the loss when they have same epoch number? It will be</p>
<pre><code class="lang-auto">Epoch   Loss
1          (2.173+1.839+1.659+1.600+1.533+1.468)/6
2         ...
</code></pre>
<p>Have you have more simple way in pytorch?</p>

klory: <pre><code class="lang-auto">for epoch in range(2):  # loop over the dataset multiple times
    epoch_loss = 0.0
    running_loss = 0.0
    for i, data in enumerate(trainloader, 0):
        # get the inputs
        inputs, labels = data

        # zero the parameter gradients
        optimizer.zero_grad()

        # forward + backward + optimize
        outputs = net(inputs)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()

        epoch_loss += outputs.shape[0] * loss.item()

        # print statistics
        running_loss += loss.item()
        if i % 2000 == 1999:    # print every 2000 mini-batches
            print('[%d, %5d] loss: %.3f' %
                  (epoch + 1, i + 1, running_loss / 2000))
            running_loss = 0.0

    # print epoch loss
    print(epoch+1, epoch_loss / len(trainset))

print('Finished Training')
</code></pre>

John1231983: <p>Thanks. So we have to modify the code. I think this may be good option</p>

Dominik_Winterer: <p><a class="mention" href="/u/klory">@klory</a><br>
Why do you multiply the loss.item() with the first dimension of the outputs tensor.<br>
This seems odd to me.</p>
<pre><code class="lang-auto">epoch_loss += outputs.shape[0] * loss.item()
</code></pre>

klory: <p>The loss is averaged by the batch_size, which is the first dimension</p>

Dominik_Winterer: <p>where is the average here?  Shouldn’t you divide instead of multiplying?</p>

klory: <p>Inside the definition of criterion<br>
<a href="https://pytorch.org/docs/stable/nn.html#torch.nn.CrossEntropyLoss" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/docs/stable/nn.html#torch.nn.CrossEntropyLoss</a></p>

MauroCE: <p>I really couldn’t understand this for a long time. I think what Klory is trying to say is this:</p>
<ul>
<li>If you look at most loss functions (e.g. <a href="https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html#torch.nn.CrossEntropyLoss" rel="noopener nofollow ugc">Cross Entropy Loss</a>) you will see that <code>reduction="mean"</code>. This means that the loss is calculated for each item in the batch, summed and then divided by the size of the batch.</li>
<li>If you want to compute the standard loss (without the average) you will need to multiply the mean loss outputted by <code>criterion()</code> with the batch size, which is <code>outputs.shape[0]</code>.</li>
</ul>