Training a model under eval mode

username: James_W
cooked: <p>I am well aware that a model can be set to train or eval mode and that layers like dropout and batchnorm behave differently depending on this switch.</p>
<p>Recently, however, I have seen repos where they train a model(resnet152 to be specific) in eval mode - i.e the model is set to eval and then trained. Is it legit to do this(they also get sota results for their domain of interest)? if yes, in what situations would one take this approach?</p>

username: ptrblck
cooked: <p>Do you know, if the model is set to <code>eval()</code> from the beginning of the training?<br>
If that’s the case and if only dropout and batchnorm layers are used, I believe you could also remove these layers, since they wouldn’t change anything.<br>
I.e. dropout is disabled, so basically acting as an <code>nn.Identity</code> layer. while batchnorm layers are initialized in a way to not change the incoming activation without training (up to floating point precision):</p>
<pre><code class="lang-python">bn = nn.BatchNorm2d(3)
bn.weight
&gt; Parameter containing:
tensor([1., 1., 1.], requires_grad=True)
bn.bias
&gt; Parameter containing:
tensor([0., 0., 0.], requires_grad=True)
bn.running_mean
&gt; tensor([0., 0., 0.])
bn.running_var
&gt; tensor([1., 1., 1.])

x = torch.randn(2, 3, 24, 24)
bn.eval()
out = bn(x)
print((out - x).abs().max())
&gt; tensor(1.6689e-05, grad_fn=&lt;MaxBackward1&gt;)
</code></pre>

username: James_W
cooked: <p>Hello <a class="mention" href="/u/ptrblck">@ptrblck</a> thank you for your response! To answer your question: yes, the model is set to <code>eval()</code> in the beginning of training. See here for example: <a href="https://github.com/talreiss/Mean-Shifted-Anomaly-Detection/blob/a02ac3001f96fcfa2f9b6a5fcb58cb3e9ba7f5b0/main.py#L32" class="inline-onebox" rel="noopener nofollow ugc">Mean-Shifted-Anomaly-Detection/main.py at a02ac3001f96fcfa2f9b6a5fcb58cb3e9ba7f5b0 · talreiss/Mean-Shifted-Anomaly-Detection · GitHub</a></p>
<p>What is perplexing is that when I change the eval just for the training (<a href="https://github.com/talreiss/Mean-Shifted-Anomaly-Detection/blob/a02ac3001f96fcfa2f9b6a5fcb58cb3e9ba7f5b0/main.py#L46" class="inline-onebox" rel="noopener nofollow ugc">Mean-Shifted-Anomaly-Detection/main.py at a02ac3001f96fcfa2f9b6a5fcb58cb3e9ba7f5b0 · talreiss/Mean-Shifted-Anomaly-Detection · GitHub</a>) I get worser results. They show that using their approach they get SOTA results on CIFAR-10, so I am trying to understand how this <code>eval()</code> thing can be applied to other architectures.</p>
<p>I do not think I understand your comment <code>If that’s the case and if only dropout and batchnorm layers are used..</code> did you mean <code>..NOT used</code>? in <code>eval()</code> mode my understanding was that batchnorm uses stats (m/v) which is stored and the dropout is disabled, meaning they are NOT used. is my understanding correct?</p>
<p>Thank you again.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="James_W" data-post="3" data-topic="131435">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/james_w/48/14698_2.png" class="avatar"> James_W:</div>
<blockquote>
<p>did you mean <code>..NOT used</code>?</p>
</blockquote>
</aside>
<p>I meant: if the only layers in the model, which change their behavior during training/evaluation, are dropout and batchnorm layers, setting <code>model.eval()</code> at the beginning of the script would disable them (as you’ve also said) and you could thus just remove the layers.<br>
However, custom layers can also use the <code>self.training</code> flag internally and thus switch their behavior, but I don’t know if that’s the case in this repository.</p>
<aside class="quote no-group" data-username="James_W" data-post="3" data-topic="131435">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/james_w/48/14698_2.png" class="avatar"> James_W:</div>
<blockquote>
<p>What is perplexing is that when I change the eval just for the training</p>
</blockquote>
</aside>
<p>I don’t understand this, as I thought <code>model.eval()</code> is already used during training and evaluation.</p>

username: James_W
cooked: <blockquote>
<p>I meant: if the only layers in the model, which change their behavior during training/evaluation, are dropout and batchnorm layers, setting <code>model.eval()</code> at the beginning of the script would disable them (as you’ve also said) and you could thus just remove the layers.<br>
However, custom layers can also use the <code>self.training</code> flag internally and thus switch their behavior, but I don’t know if that’s the case in this repository.</p>
</blockquote>
<p>Ah ok I understand now! Thank you.</p>
<blockquote>
<p>I don’t understand this, as I thought model.eval() is already used during training and evaluation.</p>
</blockquote>
<p>Sorry I was not clear - what I had meant was, i switched the eval() flag to train() flag for the training part of the code i.e here: <a href="https://github.com/talreiss/Mean-Shifted-Anomaly-Detection/blob/a02ac3001f96fcfa2f9b6a5fcb58cb3e9ba7f5b0/main.py#L46" rel="noopener nofollow ugc">https://github.com/talreiss/Mean-Shifted-Anomaly-Detection/blob/a02ac3001f96fcfa2f9b6a5fcb58cb3e9ba7f5b0/main.py#L46</a> then I get worser results. Hope I am making myself clear. So basically I do something like this:</p>
<pre><code class="lang-auto">def run_epoch(model, train_loader, optimizer, center, device):
    model.train() #this by default was model.eval() according to repo
    total_loss, total_num = 0.0, 0
</code></pre>

username: ptrblck
cooked: <p>Ah OK, thanks for the explanation.<br>
In that case, could you run an additional experiment by removing all dropout and batchnorm layers from the model (replace them with <code>nn.Identity</code>), call <code>model.train()</code>, and retrain the model?<br>
This should be the same run as using <code>model.eval()</code> during training and I would thus expect to see approx. the same training behavior (assuming it’s stable).</p>

username: James_W
cooked: <p>thanks <a class="mention" href="/u/ptrblck">@ptrblck</a> I will try your suggestion and report back here.</p>

username: James_W
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> I have a follow up question: so, assuming there is no custom layers in this repo (and I can confirm there isnt - it is a vanilla pretrained model), is it legit to train under eval mode?</p>
<p>The model is just this:</p>
<pre><code class="lang-auto">class Model(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.backbone = models.resnet152(pretrained=True)
        self.backbone.fc = torch.nn.Identity()
        freeze_parameters(self.backbone, train_fc=False)

    def forward(self, x):
        z1 = self.backbone(x)
        z_n = F.normalize(z1, dim=-1)
        return z_n

def freeze_parameters(model, train_fc=False):
    for p in model.conv1.parameters():
        p.requires_grad = False
    for p in model.bn1.parameters():
        p.requires_grad = False
    for p in model.layer1.parameters():
        p.requires_grad = False
    for p in model.layer2.parameters():
        p.requires_grad = False
    if not train_fc:
        for p in model.fc.parameters():
            p.requires_grad = False
</code></pre>

username: ptrblck
cooked: <p>Sure, it could be valid to train in <code>eval()</code> and thus disabling the aforementioned layers. It would still be interesting to see if a removal of dropout and batchnorm layers yield the same behavior or if some training/eval logic is still hidden somewhere else in the script.</p>