Calculating epoch loss incorrectly?

username: hanoody
cooked: <p>Hi,</p>
<p>My epoch loss curve for my multi-class UNET segmentation model looks really off. I get this weird periodic behavior and the loss doesn’t seem to stabilize even with 500 epochs.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/5/b/5b8f3575dd3bb7e827eb5a2fab78a455b9ae3090.png" alt="image" data-base62-sha1="d3YgWCfI6HgZ4OzHHCaQ3gskOVq" width="640" height="480"></p>
<p>Below is my training function:</p>
<pre><code class="lang-auto">def train_fn(loader, model, optimizer, loss_fn, scaler, loss_values):
    loop = tqdm(loader)
    running_loss = 0.0
    for batch_idx, (data, targets, _) in enumerate(loop):
        data = data.to(device=DEVICE)
        targets = targets.long().unsqueeze(1).to(device=DEVICE)
        # forward
        with torch.cuda.amp.autocast():
            predictions = model(data) #should be (N, C) where C = num of classes
            loss = loss_fn(predictions, targets.squeeze(1))
        # backward
        optimizer.zero_grad()
        scaler.scale(loss).backward()
        scaler.step(optimizer)
        scaler.update()
        running_loss =+ loss.item() * data.size(0)
        # update tqdm loop
        loop.set_postfix(loss=loss.item())
    loss_values.append(running_loss/len(data))
</code></pre>
<p>which gets called for each epoch in the following snippet from main():</p>
<pre><code class="lang-auto">    scaler = torch.cuda.amp.GradScaler()
    loss_values = []
    start = time.time()
    for epoch in range(NUM_EPOCHS):
        train_fn(train_loader, model, optimizer, loss_fn, scaler, loss_values)

        # save model
        checkpoint = {
            "state_dict": model.state_dict(),
            "optimizer":optimizer.state_dict(),
        }
        save_checkpoint(checkpoint)

        # check accuracy
        check_accuracy(val_loader, model, device=DEVICE)
    end = time.time()
</code></pre>
<p>Any idea why this might be the case? Thank you!</p>

username: ptrblck
cooked: <p>In this line of code you are scaling the current batch loss with the batch size and accumulate it:</p>
<pre><code class="lang-python">running_loss =+ loss.item() * data.size(0)
</code></pre>
<p>while you are normalizing it with the batch size instead of the <code>Dataset</code> length here:</p>
<pre><code class="lang-python">loss_values.append(running_loss/len(data))
</code></pre>
<p>My guess is that the last batch size might be smaller and would thus increase the loss in the last step.</p>

username: hanoody
cooked: <p>Thank you for your response! That makes sense to me overall but I am confused about some details.</p>
<p>Also I forgot to mention that my batch size is 1…</p>
<p>Did you mean that I am normalizing it with the <code>Dataset</code> length instead of the batch size? So should I have the following instead?</p>
<p><code>loss_values.append(running_loss/data.size(0))</code></p>

username: ptrblck
cooked: <p>No, <code>data.size(0)</code> would return the same value as <code>len(data)</code>:</p>
<pre><code class="lang-python">data = torch.randn(6, 4)
print(data.size(0))
&gt; 6
print(len(data))
&gt; 6
</code></pre>
<p>Since you are scaling the loss via:</p>
<pre><code class="lang-python">running_loss += current_batch_loss * nb_samples_in_batch
</code></pre>
<p>you would have to divide by the number of samples in the entire dataset (<code>len(loader.dataset)</code>).</p>

username: hanoody
cooked: <p>Thank you very much! That makes sense. I corrected my running loss and now I am training my model with 300 epochs. I will let you know if it was indeed the initial running loss calculation that was causing the graph to be unstable.</p>

username: hanoody
cooked: <p>Unfortunately after correcting that, I still get the unstable plot even with a batch size of 1.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/3/2/32096ec058c87ebce1adde343e3a389c847df10a.png" alt="image" data-base62-sha1="78E58n19uUoEos1HYMrww2g3pRE" width="640" height="480"></p>