On running loss and average loss

oat: <p>[ 1. Context ]<br>
I’m following Udacity’s tutorial <a href="https://www.udacity.com/course/deep-learning-pytorch--ud188" rel="noopener nofollow ugc">Intro to Deep Learning with PyTorch</a>. In Lesson 5 on Convolutional Neural Networks by <a href="https://twitter.com/cezannecam" rel="noopener nofollow ugc">Cezanne Camacho</a> Step 10 Training the Network, Cezanne used the code as quoted below to calculate the training loss and validation loss.</p>
<p>[ 2. Question ]<br>
Although Cezanne has explained in the video, I’m still not clear why she was using <strong><code>train_loss  += loss.item()*data.size(0)</code></strong> to aggregate the <em>total training loss</em> and why she was using <strong><code>train_loss = train_loss/len(train_loader.sampler)</code></strong> to calculate the <em>average training loss</em> with criterion being <code>nn.CrossEntropyLoss()</code>, whereas in the earlier Fashion-MNIST tutorial in the same series they were coded as <strong><code>running_loss += loss.item()</code></strong> and <strong><code>running_loss/len(trainloader)</code></strong> with criterion being <code>nn.NLLLoss()</code>.</p>
<p>[ 3. My understanding ]</p>
<ol>
<li>Following Andrew Ng’s distinguishing between “cost: difference between prediction and target for each sample” and “loss: difference between prediction and target for the entire sample set”.</li>
<li>
<code>loss.item()</code> is the value of “total cost, or, sum of target*log(prediction)” <strong>averaged</strong> across all training examples of the <strong>current batch</strong>, according to the definition of cross entropy loss.</li>
<li>Therefore, <code>loss.item()*data.size(0)</code> is the “total loss of the current batch (not averaged)”.</li>
<li>And, <code>train_loss</code> accumulates these “total loss per batch” for the entire epoch, i.e. “total loss of the current epoch”.</li>
<li>Finally, <code>train_loss = train_loss/len(train_loader.sampler)</code> calculates the “cost/loss <strong>averaged</strong> across all training examples for the <strong>current epoch</strong>”</li>
</ol>
<p>May I ask if the interpretation above is correct?</p>
<pre><code class="lang-auto"># specify loss function (categorical cross-entropy)
criterion = nn.CrossEntropyLoss()

# specify optimizer (stochastic gradient descent) and learning rate = 0.01
optimizer = torch.optim.SGD(model.parameters(), lr=0.01)

# number of epochs to train the model
n_epochs = 50

# initialize tracker for minimum validation loss
valid_loss_min = np.Inf # set initial "min" to infinity

for epoch in range(n_epochs):
    # monitor training loss
    train_loss = 0.0
    valid_loss = 0.0
    
    ###################
    # train the model #
    ###################
    model.train() # prep model for training
    for data, target in train_loader:
        # clear the gradients of all optimized variables
        optimizer.zero_grad()
        # forward pass: compute predicted outputs by passing inputs to the model
        output = model(data)
        # calculate the loss
        loss = criterion(output, target)
        # backward pass: compute gradient of the loss with respect to model parameters
        loss.backward()
        # perform a single optimization step (parameter update)
        optimizer.step()
        # update running training loss
        train_loss += loss.item()*data.size(0)
        
    ######################    
    # validate the model #
    ######################
    model.eval() # prep model for evaluation
    for data, target in valid_loader:
        # forward pass: compute predicted outputs by passing inputs to the model
        output = model(data)
        # calculate the loss
        loss = criterion(output, target)
        # update running validation loss 
        valid_loss += loss.item()*data.size(0)
        
    # print training/validation statistics 
    # calculate average loss over an epoch
    train_loss = train_loss/len(train_loader.sampler)
    valid_loss = valid_loss/len(valid_loader.sampler)
</code></pre>

ptrblck: <p>Yes, your explanation is correct.<br>
The first approach of multiplying the averaged batch loss by the batch size and dividing by the number of samples gives you the correct average sample loss for this particular epoch.<br>
The second approach of dividing the averaged batch loss by the number of batches would yield the same result, if each batch in the epoch contains <code>batch_size</code> samples. This might not always be the case, if the length of the dataset is not divisible by the <code>batch_size</code> without a remainder. The last batch would thus contain less samples and the loss calculation would introduce a small bias.</p>

pkgprateek: <p>you can also use <code>reduction='sum'</code> when calculating loss so you dont have to multiply the <code>data.size(0)</code><br>
<a href="https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html#crossentropyloss" rel="noopener nofollow ugc">PyTorch Docs: CrossEntropyLoss</a></p>

ptrblck: <p>Assuming the <code>DataLoader</code> yields batches with different batch sizes (e.g. the last one could be smaller), this would yield different gradients (since the scaling is now different) and could cause training issues unless you scale the <code>loss</code> before calling <code>backward()</code> (which is just moving the scaling to another part of the code), so be careful with this approach.</p>