Proper way to call mdl.eval() and mdl.train() so that checkpointing saves all parameters?

username: Brando_Miranda
cooked: <p>0</p>
<p>I usually alternate between training and testing in my code. This results in me calling <code>mdl.train()</code> when coming back to the trianing loop and <code>mdl.test()</code> when coming to the testing loop. However I’ve noticed that potentially this creates issues with how parameters are saved (e.g. batch norm, perhaps others) that depend on this eval, train state. Thus, it begs the question how these flags should be called if at all. If I call evaluation before checkpointing I believe this will delete my running average as mentioned here: <a href="https://stackoverflow.com/questions/69846779/how-does-one-use-the-mean-and-std-from-training-in-batch-norm" rel="noopener nofollow ugc">How does one use the mean and std from training in Batch Norm?</a>.</p>
<p>I think a deep copy of the model during evaluation should fix things - but that seems like it will kill my GPU memory (and perhaps my normal memory too).</p>
<p>Thus, what is the proper way to alternate between evaluation and training in PyTorch so that checkpoints are saved properly (e.g. the running stats of training are NOT deleted).</p>
<p>This is a challenge because I usually always run evaluation to see if the current model is better in validation than the previous one and then decide on THAT to save it or not - which requires me to do an evaluation before checkpointing - always.</p>
<p>Perhaps, I can always run it on train so that the running averages are saved correctly but since the actual validation value doesn’t matter it’s fine if the stats from train and val leak to the validation.</p>
<p>related:</p>
<ul>
<li><a href="https://stackoverflow.com/questions/69856207/what-is-the-proper-way-to-do-evaluation-e-g-testing-validation-of-deep-learn" class="inline-onebox" rel="noopener nofollow ugc">What is the proper way to do evaluation (e.g. testing, validation) of deep learning model in PyTorch so that checkpoints are saved correctly? - Stack Overflow</a></li>
</ul>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Brando_Miranda" data-post="1" data-topic="136100">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/brando_miranda/48/14355_2.png" class="avatar"> Brando_Miranda:</div>
<blockquote>
<p>This results in me calling <code>mdl.train()</code> when coming back to the trianing loop and <code>mdl.test()</code> when coming to the testing loop.</p>
</blockquote>
</aside>
<p>Yes, this is the proper way.</p>
<aside class="quote no-group" data-username="Brando_Miranda" data-post="1" data-topic="136100">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/brando_miranda/48/14355_2.png" class="avatar"> Brando_Miranda:</div>
<blockquote>
<p>However I’ve noticed that potentially this creates issues with how parameters are saved (e.g. batch norm, perhaps others) that depend on this eval, train state.</p>
</blockquote>
</aside>
<p>No, that’s not the case and unfortunately you didn’t follow up in <a href="https://discuss.pytorch.org/t/how-does-one-use-the-mean-and-std-from-training-in-batch-norm/136029/8">your other thread</a>.<br>
Saving the <code>state_dict</code> doesn’t have any known issues regarding previous calls to <code>model.train()</code> or <code>model.eval()</code>, so my guess is still that you might be manually deactivating the running stats updates.<br>
If that’s not the case, please post a minimal executable code snippet which would reproduce the issue.</p>