Will not using eval() when validating affect training performance?

username: unami
cooked: <p>I’ve just started using pytorch and am working on my first project. What got me confused was the eval() function. I followed a starting tutorial that showed how to use pytorch and it did not contain using eval() function for validation. So I trained a model with a satisfactory validation performance, but then came across eval and train modes when looking something up. As far as I understand, not using it would only affect how accurate my validation actually is (because dropouts and normalizations would still be used for validation), but it would not affect the training itself. However, when I added the model.eval() and model.train() lines between validation, I noticed that the training loss for the same amount of epochs is now different?</p>
<pre><code class="lang-auto">for i in range(epochs):
    i += 1
    start = time.time()
    y_pred = model(categorical_train_data, numerical_train_data)
    single_loss = loss_function(y_pred, train_outputs)
    aggregated_losses.append(single_loss)
    print(f'epoch: {i:3} loss: {single_loss.item():10.8f}')

    optimizer.zero_grad()
    single_loss.backward()
    optimizer.step()
    with torch.no_grad():
        y_val = model(categorical_test_data, numerical_test_data)
        loss = loss_function(y_val, test_outputs)
</code></pre>
<p>The code above produces</p>
<pre><code class="lang-auto">epoch:   1 loss: 0.80006623
epoch:   2 loss: 0.78904432
</code></pre>
<p>however, if I change it to</p>
<pre><code class="lang-auto">for i in range(epochs):
    i += 1
    start = time.time()
    y_pred = model(categorical_train_data, numerical_train_data)
    single_loss = loss_function(y_pred, train_outputs)
    aggregated_losses.append(single_loss)
    print(f'epoch: {i:3} loss: {single_loss.item():10.8f}')

    optimizer.zero_grad()
    single_loss.backward()
    optimizer.step()
    model.eval()
    with torch.no_grad():
        y_val = model(categorical_test_data, numerical_test_data)
        loss = loss_function(y_val, test_outputs)
    model.train()
</code></pre>
<p>I get a different output:</p>
<pre><code class="lang-auto">epoch:   1 loss: 0.80006623
epoch:   2 loss: 0.78863680
</code></pre>
<p>I used torch.manual_seed(0) so I know it’s not the initial distribution. I can run the code multiple times and get the same output for both these cases.</p>
<p>Which as far as I understand means that weights of the model were updated. Does that mean that without using eval mode I let the validation set influence the training of the actual model and therefore the validation performance is not actually validation?</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="unami" data-post="1" data-topic="104615">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/u/58956e/48.png" class="avatar"> unami:</div>
<blockquote>
<p>Which as far as I understand means that weights of the model were updated.</p>
</blockquote>
</aside>
<p>No, that won’t be the case, since you are never updating the model (and the gradient calculation is also disabled via <code>torch.no_grad()</code>).<br>
The difference in the losses might come from additional calls into the random number generator during the validation loop while the model is still in training mode.<br>
As you’ve already described the dropout layers will be used, which will sample random numbers and would thus change the next call into the random number generator.<br>
Besides that the running stats in all batchnorm layers will be updated with the validation batch statistics, which would be a data leak.</p>