What is updated *per-epoch* (and not *per-batch*)?

jpcbertoldo: <p>I changed my script to use pytorch lightning and now I do not get the same results/performance.</p>
<p><strong>Before</strong>: I had a hand-made torch script with a classic training nested loops (over epochs and over batches in each epoch).</p>
<p><strong>After</strong>: litterally copied and pasted the things in appropriate places to fit PL’s framework.</p>
<p><strong>The catch</strong>: I was defining an epoch as 10 iterations over my training dataset (just replicate the instances in the dataloader 10x), now I define an epoch as a single iteration over the training epoch (no replication) but I multiplied the number of epochs (50) by 10x (so 500) – therefore I have the same number of batches seen during the training.</p>
<p><strong>One solved problem</strong>: as my learning rate was decaying by a ration <code>0 &lt; r &lt; 1</code> (so <code>LR_{k+1} = LR_{k} * r</code> where <code>k</code> is the epoch index), the final learning rate was much smaller than before, so I replaced it by an equivalent rate decay (<code>r' = r ^ (50/500)</code>) such that the learning rate at the begining and at end of the training are the same. The performance had some improvement but I’m still far from the previous one.</p>
<p><strong>The question</strong>: are there other parameters or hyperparameters that are updated  <em>per-epoch</em> (and not <em>per-batch</em>)?</p>
<hr>
<p>PS1: I’m admitting that the problem is comming from some per-epoch update because I reviewed the code and everything should be the same except for the number of epochs although the number of batches is still the same.</p>
<p>PS2: a few things in my training recipe: optimizer SGD with nesterov and momentum, weight decay regularization, model with batchnorm, 2d convolutions, and maxpooling.</p>

ptrblck: <aside class="quote no-group" data-username="jpcbertoldo" data-post="1" data-topic="158427">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jpcbertoldo/48/52037_2.png" class="avatar"> jpcbertoldo:</div>
<blockquote>
<p><strong>The question</strong>: are there other parameters or hyperparameters that are updated <em>per-epoch</em> (and not <em>per-batch</em>)?</p>
</blockquote>
</aside>
<p>PyTorch won’t update anything in your model behind your back per-iteration or per-epoch, so I would recommend to look for explicit per-epoch updates such as learning rate decay, change in data augmentation etc.<br>
Also, usually you would run a validation loop after one training epoch. Make sure to call <code>model.eval()</code> in the validation loop to avoid updating the running stats of batchnorm layers etc.</p>

jpcbertoldo: <blockquote>
<p>PyTorch won’t update anything in your model behind your back per-iteration or per-epoch, so I would recommend to look for explicit per-epoch updates such as learning rate decay, change in data augmentation etc.</p>
</blockquote>
<p>Got it.</p>
<blockquote>
<p>Make sure to call model.eval() in the validation loop to avoid updating the running stats of batchnorm layers etc.</p>
</blockquote>
<p>I’m using <code>pytorch-lightning</code>, which is supposed to manage that under the hood.</p>
<p>Thanks!</p>