Loss function oscillations - periodic curve per epoch

yoav: <p>I am training a network, and the batch-averaged per-epoch loss decreases iteratively during training. But, the curve of the per-batch loss is suspiciously periodic.</p>
<p>To the left is the overall curve of the per-batch loss, and to the right is a zoom-in on one area to show the periodicity:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/2/d281eb342bfe320b502187eeeba5263f283777f1.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/d281eb342bfe320b502187eeeba5263f283777f1" title="upload"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/2/d281eb342bfe320b502187eeeba5263f283777f1_2_690x184.jpeg" alt="upload" data-base62-sha1="u2eClnl0KXTdJrZBC3nndYffSql" width="690" height="184" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/2/d281eb342bfe320b502187eeeba5263f283777f1_2_690x184.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/d/2/d281eb342bfe320b502187eeeba5263f283777f1_2_1035x276.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/d/2/d281eb342bfe320b502187eeeba5263f283777f1_2_1380x368.jpeg 2x" data-dominant-color="F9FBFC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">upload</span><span class="informations">1909×511 55 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I am using a DataLoader, and am not utilizing shuffling. batch-size is 80, and the size of the last batch is 62. I realize shuffling would help, but I think that there is a more serious issue resulting in this exact periodicity. I should note that there are 29 batches in an epoch, and the periodicity (distance between the same repeating peak in repeating sections) is 31 epochs apart.</p>
<p>As a loss for the batch, I am using a sum of 5 seperate MSELoss elements:</p>
<p>mse_loss = torch.nn.MSELoss()<br>
total_loss =  mse_loss(guess_a,coord_a) + mse_loss(guess_b,coord_b) + mse_loss(guess_c,coord_c) + mse_loss(guess_d,coord_d) + mse_loss(guess_e,coord_e)</p>

KFrank: <p>Hi Yoav!</p>
<p>Curiouser and curiouser …</p>
<p>I have some clarifying questions and a couple of suggestions.</p>
<aside class="quote no-group" data-username="yoav" data-post="1" data-topic="182347" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/y/58f4c7/48.png" class="avatar"> yoav:</div>
<blockquote>
<p>I am using a DataLoader, and am not utilizing shuffling.</p>
</blockquote>
</aside>
<p>To confirm:</p>
<p>Do you mean by this that every epoch contains the exact same sequence<br>
of batches (in the same order) every time through?  (And that each specific<br>
batch at a given position within the epoch contains the exact same data<br>
samples?)</p>
<aside class="quote no-group">
<blockquote>
<p>I should note that there are 29 batches in an epoch, and the periodicity (distance between the same repeating peak in repeating sections) is 31 epochs apart.</p>
</blockquote>
</aside>
<p>Did you mean that the distance  between repeating peaks is 31 <em>batches</em> or is<br>
it indeed “31 epochs,” as you said?  “Batches” seems to fit the context better.</p>
<p>If it is batches, could you confirm that the periodicity really is different than the<br>
number of batches in an epoch?  This looks very much as if as you go through<br>
the same sequence of batches you get approximately-matching patterns in<br>
your loss function.  It would seem very odd if as you go through the same<br>
sequence of batches from epoch to epoch, different batches give you “matching”<br>
values of the loss function (for example, different batches give you the peak).</p>
<p>Are you using any sort of learning-rate scheduler?</p>
<p>Could you post your zoomed-in loss graph, annotating the top peak and bottom<br>
valley (as well as, perhaps, a couple other landmarks) with the batch number<br>
(or epoch number, if they really are epochs)?</p>
<p>Try making the same graph with training turned off.  That is, run the exact same<br>
code, but simply skip the optimization step or maybe set the learning rate to<br>
zero.  (You might also try training for a bit before turning off the training just to<br>
let things stabilize a little bit.)</p>
<p>Assuming that you are using pytorch’s <code>DataLoader</code> class, you might try setting<br>
<code>drop_last = True</code>.  I think its a long shot (because the last-batch size of 62 is<br>
really not that different from 80), but maybe your last batch is giving your training<br>
an unhelpful jolt that causes your loss function to spike up.</p>
<p>And, yes, turning on shuffling would make sense, but that would mask the<br>
interesting effect that you’re currently seeing.</p>
<p>Best.</p>
<p>K. Frank</p>

yoav: <p>Hello Frank,</p>
<p>Thanks for this!!</p>
<ol>
<li>
<p>Here is the setup of the data loader:<br>
train_loader = DataLoader(train_set, batch_size = batch_size, shuffle = False)<br>
So - during each epoch, batches are provided at the same order.</p>
</li>
<li>
<p>Sorry for the confusion in phrasing my original question - I meant batches, as you mentioned. The periodicity is exactly the same as the number of batches in an epoch, every time.</p>
</li>
<li>
<p>For optimization, I am currently using:</p>
</li>
</ol>
<p>optimizer = torch.optim.Adam(model.parameters(),lr=learning_rate,betas=(0.9,0.999),eps=1e-08,weight_decay=0,amsgrad=True)</p>
<p>where lr is=0.001</p>
<ol start="4">
<li>
<p>In the following chart - lr = 0.001, batch size = 40; the size of the training set is 2,030, therefore there are 51 training batches in an entire epoch. I am not shuffling their order for now, as mentioned. To the left - the overall batch loss curve, to the right - zoomed in on a region. each repeating area is exactly one epoch wide - batch 50 (step 50) is the last batch of the first epoch (starts from 0); step 51 is the first batch of the second epoch and 101 is the last of this epoch, etc.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/d/ad6e6b5d4084e1afe8d1c47482843309668cb30b.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/ad6e6b5d4084e1afe8d1c47482843309668cb30b" title="run_N"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/d/ad6e6b5d4084e1afe8d1c47482843309668cb30b_2_690x388.jpeg" alt="run_N" data-base62-sha1="oKfaQ62OkP97hFmVj4yIlkUPYyT" width="690" height="388" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/d/ad6e6b5d4084e1afe8d1c47482843309668cb30b_2_690x388.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/a/d/ad6e6b5d4084e1afe8d1c47482843309668cb30b_2_1035x582.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/a/d/ad6e6b5d4084e1afe8d1c47482843309668cb30b.jpeg 2x" data-dominant-color="FAFCFD"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">run_N</span><span class="informations">1280×720 80.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
</li>
<li>
<p>Setting lr = 0 resulted in a very similar batch loss curve, only without any decay of the peak height, and instead of being very similar, the periods are identical.</p>
</li>
<li>
<p>Regarding setting drop_last = True and shuffling - thanks, I will incorporate these later after the current mystery gets resolved.</p>
</li>
</ol>
<p>I should mention my dataset is small (won’t get into the details of describing its nature). I would not be surprised if the data/architecture are not sufficient for significant learning, but I remain puzzled about the periodic pattern - if nothing is learned, I wonder why is the overall per-epoch loss decreasing.</p>
<p>Thanks for any advice!</p>

KFrank: <p>Hi Yoav!</p>
<aside class="quote no-group" data-username="yoav" data-post="3" data-topic="182347" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/y/58f4c7/48.png" class="avatar"> yoav:</div>
<blockquote>
<ol start="5">
<li>Setting lr = 0 resulted in a very similar batch loss curve, only without any decay of the peak height, and instead of being very similar, the periods are identical.</li>
</ol>
</blockquote>
</aside>
<p>This strongly suggests that something is “jolting” your training at the<br>
beginning (or perhaps the end) of each epoch.</p>
<p>Are you instantiating a new copy of your optimizer every epoch?  <code>Adam</code><br>
maintains state and smooth training would be disrupted if that state were<br>
zeroed out every epoch.</p>
<p>You might try using <code>SGD</code> (with no momentum) for your optimizer, as it<br>
carries no state.</p>
<p>Could you be calling <code>optimizer.zero_grad()</code> or <code>loss.backward()</code><br>
once per epoch instead of for every batch?</p>
<p>Are you using some kind of learning-rate scheduler that jumps or resets<br>
every epoch.</p>
<p>In any event, look for something in your code that does something different<br>
between epochs than it does between samples within an epoch.</p>
<p>Good luck.</p>
<p>K. Frank</p>