Training loss drops at the interface between two epochs, but increases within an epoch

username: HELLORPG
cooked: <p>Hey everyone, I encountered a pretty strange situation during training. It was quite unexpected and left me feeling a bit perplexed.</p>
<p>In my experiments, I’m training my model with a simple <code>nn.CrossEntropy</code> criterion and <code>AdamW</code> optimizer. At the beginning of training, everything seems normal. But at the subsequent epochs, the training loss curve looks strange, as shown below:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/3/33f91d57c2ba100ece3bcf011d17e1309c8c9491.png" data-download-href="https://discuss.pytorch.org/uploads/default/33f91d57c2ba100ece3bcf011d17e1309c8c9491" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/3/33f91d57c2ba100ece3bcf011d17e1309c8c9491_2_690x200.png" alt="image" data-base62-sha1="7pM4Ktf45MgGyXr4sHnCBLPzoMp" width="690" height="200" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/3/33f91d57c2ba100ece3bcf011d17e1309c8c9491_2_690x200.png, https://discuss.pytorch.org/uploads/default/optimized/3X/3/3/33f91d57c2ba100ece3bcf011d17e1309c8c9491_2_1035x300.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/3/3/33f91d57c2ba100ece3bcf011d17e1309c8c9491_2_1380x400.png 2x" data-dominant-color="FEFEFF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">3602×1048 116 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
There are two strange characteristics:</p>
<p><strong>1. The loss value slowly increases within an epoch.</strong><br>
<strong>2. The loss value suddenly drops at the interface between two epochs.</strong></p>
<p>BTW, for the loss value in the loss curve, we smooth it over the most recent 50 steps (within an epoch), so I think the loss value smoothing does not cause it. And my dataloader uses <code>suffle=True</code> during training.</p>
<p>I checked previous similar discussions, like <a href="https://discuss.pytorch.org/t/strange-behavior-with-sgd-momentum-training/7442/1">Strange behavior with SGD momentum training - PyTorch Forums</a>, but it doesn’t work for my situation.</p>
<p><strong>Thanks.</strong></p>
<p>P.S.<br>
Due to the large amount of code in my project, I can’t provide all the code in this forum. Here, I try to give the code of training one epoch in my project. However, the code cannot be run directly but is only used as a pseudo-code reference.</p>
<pre data-code-wrap="python"><code class="lang-python">model.train()
metrics = Metrics()   # save metrics

optimizer.zero_grad()   # init optim

last_timestamp = TPS.timestamp()
for i, batch in enumerate(dataloader):
    id_infos, last_frame_detections = batch["id_infos_tensor"], batch["last_frame_detections_tensor"]
    last_frame_id_predicts, last_frame_masks = model(
        id_infos, last_frame_detections
    )
    loss = criterion(
        predicts=last_frame_id_predicts,
        labels=last_frame_detections["id_label"],
        predict_masks=last_frame_masks
    )

    metrics["id_loss"].update(loss.item())

    loss /= config["ACCUMULATE_STEPS"]
    loss.backward()

    if (i + 1) % config["ACCUMULATE_STEPS"] == 0:
        optimizer.step()
        optimizer.zero_grad()
</code></pre>

username: HELLORPG
cooked: <p>I finally figured it out, it is caused by my <code>Dataset</code> class. I forgot to use <code>copy.deepcopy</code> to copy the <code>dict</code>, which saves the image info during data transforms. Therefore, the <code>RandomCrop</code> makes the data augmentation harder and harder, causing the strange loss curve (increasing within an epoch).</p>
<p><em>copy.deepcopy() is all my need</em>.</p>