Resnet101 sensitive to previous evaluations in train mode?

username: FaultyBagnose
cooked: <p>I’m retraining resnet101 for an image classification task, and observe that my models behave differently in eval mode if it has previously been run in training mode. Here is a code example:</p>
<pre><code class="lang-auto">from torchvision import models
import torch
from PIL import Image
from torchvision import transforms

transform = transforms.Compose([  # [1]
    transforms.Resize(256),  # [2]
    transforms.CenterCrop(224),  # [3]
    transforms.ToTensor(),  # [4]
    transforms.Normalize(  # [5]
        mean=[0.485, 0.456, 0.406],  # [6]
        std=[0.229, 0.224, 0.225]  # [7]
    )])

img = Image.open("C:/image.jpg")
img_t = transform(img)
batch_t = torch.unsqueeze(img_t, 0).cuda()
resnet = models.resnet101(pretrained=True)
resnet.cuda()
resnet.eval()
print(max(resnet(batch_t)[0]))
# tensor(13.7692, device='cuda:0', grad_fn=&lt;SelectBackward&gt;)
# Peforming a model evaluation in training mode:
resnet.train()
dummy = resnet(batch_t)
# Switching back to eval mode, and the model output is different:
resnet.eval()
print(max(resnet(batch_t)[0]))
# tensor(13.2980, device='cuda:0', grad_fn=&lt;SelectBackward&gt;)
</code></pre>
<p>Apparently, running the model in training mode (without updating anything) changes the behavior of the model in eval mode.<br>
I assume I’m missing something, but I wouldn’t expect the statements<br>
resnet.train()<br>
dummy = resnet(batch_t)<br>
to make any changes to the model?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/7/878f46e6d09fecf75fbb4773f568eefbdf6c3b6e.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/878f46e6d09fecf75fbb4773f568eefbdf6c3b6e" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/7/878f46e6d09fecf75fbb4773f568eefbdf6c3b6e_2_375x500.jpeg" alt="image" data-base62-sha1="jldqx9K6FNMMrCnwNg3Ln15HjRQ" width="375" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/7/878f46e6d09fecf75fbb4773f568eefbdf6c3b6e_2_375x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/8/7/878f46e6d09fecf75fbb4773f568eefbdf6c3b6e_2_562x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/8/7/878f46e6d09fecf75fbb4773f568eefbdf6c3b6e_2_750x1000.jpeg 2x" data-dominant-color="8E796A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">750×1000 91.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="FaultyBagnose" data-post="1" data-topic="93015">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/f/96bed5/48.png" class="avatar"> FaultyBagnose:</div>
<blockquote>
<p>I assume I’m missing something, but I wouldn’t expect the statements<br>
resnet.train()<br>
dummy = resnet(batch_t)<br>
to make any changes to the model?</p>
</blockquote>
</aside>
<p>The forward pass will update the running statistics of all batch norm layers and thus the next output in <code>model.eval()</code> will be changed.<br>
Note that <code>model.train()</code> and <code>model.eval()</code> are exactly used to switch this behavior. I.e. during training the batch statistics will be used to normalize the data and the running stats will be updated, while during evaluation the running stats will be applied to normalize the input data. (Unrelated to this issue, but also dropout will be disabled during <code>model.eval()</code>.)</p>

username: FaultyBagnose
cooked: <p>I thought I understood the answer at the time, but now I’m less sure.<br>
Over which periods are the batch norm running statistics collected?<br>
Are they reset or discarded each time I switch the model between modes?<br>
This would work fine if you run through all your training images in one sequence (which is common, I guess), but I implement a sampling procedure of randomly drawing images from randomly chosen classes, in order to imitate a balanced training set. During training, I (frequently) switch back and forth between traning and eval mode, to monitor the progress.<br>
I guess this is  a bad idea, if it resets running statistics all the time, and these really should be calibrated from the entire data set?</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="FaultyBagnose" data-post="4" data-topic="93015">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/f/96bed5/48.png" class="avatar"> FaultyBagnose:</div>
<blockquote>
<p>Over which periods are the batch norm running statistics collected?</p>
</blockquote>
</aside>
<p>During <code>model.train()</code> the batch statistics will be used to update the running stats using the <code>momentum</code>.</p>
<aside class="quote no-group" data-username="FaultyBagnose" data-post="4" data-topic="93015">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/f/96bed5/48.png" class="avatar"> FaultyBagnose:</div>
<blockquote>
<p>Are they reset or discarded each time I switch the model between modes?</p>
</blockquote>
</aside>
<p>No. You can check them via <code>bn.running_mean</code> and <code>bn.running_var</code>.</p>

username: FaultyBagnose
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="5" data-topic="93015">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>bn.running_mean</p>
</blockquote>
</aside>
<p>Thanks for always answering quickly and accurateky, ptrblck!<br>
I’ve tried to dig into the issue some more, and in another post I found this code where you implement this stuff: “<a href="https://github.com/ptrblck/pytorch_misc/blob/master/batch_norm_manual.py" rel="noopener nofollow ugc">https://github.com/ptrblck/pytorch_misc/blob/master/batch_norm_manual.py</a>”.<br>
If momentum is in use, you set<br>
exponential_average_factor = self.momentum<br>
Your update of the running mean looks like this:<br>
self.running_mean = exponential_average_factor * mean<br>
+ (1 - exponential_average_factor) * self.running_mean<br>
If momentum is close to 1 (typically 0.9 in my application), doesn’t this calculation place most of the weight on the latest observation, and close to none on the history?<br>
Shouldn’t it be the other way around for this calculation to accumulate a stable Ornstein-Uhlenbeck type average?:<br>
exponential_average_factor = 1-self.momentum<br>
Or should momentum be close to 0?</p>

username: ptrblck
cooked: <p>It depends how you specify the momentum and note that PyTorch uses a probably unexpected default value of <code>0.1</code>.<br>
I’ve implemented the manual batchnorm approach to stick to the PyTorch definition. From the docs:</p>
<blockquote>
<p>This momentum argument is different from one used in optimizer classes and the conventional notion of momentum. Mathematically, the update rule for running statistics here is x_hat_new = (1 - momentum) * x_hat + momentum x_t , where x_hat is the estimated statistic and x_t is the new observed value.</p>
</blockquote>