Why not use model.eval() in training_step() method on lightning

andreys42: <p>I’m wonder why we don’t use model.eval() command in training_step method of the “LightningModule”</p>
<blockquote>
<p>def training_step(self, batch, batch_idx):<br>
x, y = batch<br>
pred  = self(x) ### but our model is in training mode now …</p>
</blockquote>

tom: <p>There is two parts to this.</p>
<ul>
<li>
<code>training_step</code> is about training, so it seems natural that the model is in training mode,</li>
<li>Lightning automatically sets the model to training for training_step and to eval for validation.</li>
</ul>
<p>Best regards</p>
<p>Thomas</p>

andreys42: <p>Thanks for clarification, <a class="mention" href="/u/tom">@tom</a> !<br>
Does Litghtning do the same for “on_train_epoch_end” callback or I have to set model.eval() manually?<br>
Which of pre-defined callbacks provided with model.eval “under the hood”?</p>

andreys42: <aside class="quote no-group" data-username="tom" data-post="2" data-topic="122425">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tom/48/3162_2.png" class="avatar"> tom:</div>
<blockquote>
<ul>
<li>Lightning automatically sets the model to training for training_step and to eval for validation.</li>
</ul>
</blockquote>
</aside>
<p>is it correct that training loss calculating using model in training mode?</p>

tom: <aside class="quote no-group" data-username="andreys42" data-post="3" data-topic="122425">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/andreys42/48/35546_2.png" class="avatar"> andreys42:</div>
<blockquote>
<p>Does Litghtning do the same for “on_train_epoch_end” callback or I have to set model.eval() manually?</p>
</blockquote>
</aside>
<p>I don’t know, you could find out yourself with <code>assert not model.training</code> (for eval mode) or so.</p>
<aside class="quote no-group">
<blockquote>
<p>Which of pre-defined callbacks provided with model.eval “under the hood”?</p>
</blockquote>
</aside>
<p>The rule of thumb would be that training is set up with train and validation/testing with eval, but if you want this more detailed, I would probably check with in the <a href="https://github.com/PyTorchLightning/pytorch-lightning/discussions" rel="noopener nofollow ugc">lightning discussions</a>.</p>
<aside class="quote no-group" data-username="andreys42" data-post="4" data-topic="122425">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/andreys42/48/35546_2.png" class="avatar"> andreys42:</div>
<blockquote>
<p>is it correct that training loss calculating using model in training mode?</p>
</blockquote>
</aside>
<p>I think the training loss is just computed during trianing, yes.</p>

andreys42: <p>I guess PL authors took care of switching between eval/train mode within pre defined callbacks… But problem is that when I try to predict test data in “on_fit_end” callback without using model.eval() it gives me different result than predicting outside training routine (and of course using model.eval in advance), thats why I’m wonder If ‘on_fit_end’ callback provided with model.eval … but I guess I chose inappropriate forum</p>

ado_sar: <p>However, it should be emphasized that reporting the loss in <code>train</code> mode might be incorrect if for example the model uses <code>Dropout</code>. In that case, the training loss will be underestimated.</p>