Model only returns "untrained" predictions that tend to 0 when in eval mode

Oliver: <p>Hi,</p>
<p>So I trained a Gan Style model to return images and it works well (imho), in the convolution blocks I use dropout (p=0.2).</p>
<p>When I return<br>
<code>pred = generator(input)</code><br>
I get what I want.</p>
<p>However when I put the generator in eval mode first, i.e.<br>
<code>generator.eval() pred = generator(input)</code></p>
<p><code>pred</code> is basically just noise, though the output is very small (&lt; 0.3).</p>
<p>I am a little confused, dropout shouldn’t affect it, since the eval just turns it off.<br>
Other than that I only have upsampling, batchnormalization, maxpooling and activation (relu and sigmoid) components.<br>
(tried removing batchnormalization, same thing)</p>
<p>Could it be the <a href="https://pytorch.org/docs/stable/generated/torch.nn.Upsample.html" rel="noopener nofollow ugc">Upsample</a> function?</p>
<p>I use it in U-net like fashion to slowly bring the size up. Should I use transpose convolutions instead?</p>
<p>I think the longer the model trains, the more the predictions in eval mode go to 0.<br>
I.e. as <code>n -&gt; infinity, pred -&gt; 0</code>, but its absolutely fine in train mode.</p>
<p>Many thanks!</p>

ptrblck: <p>Most likely the batchnorm layers are causing this effect and you could double check it by calling <code>.train()</code> on these layers only after calling <code>model.eval()</code>.<br>
Sometimes changing the <code>momentum</code> of the batchnorm layer or turning the running stats off might help.</p>

Oliver: <p>thanks a lot, I thought that too, but I removed them and am still seeing this effect.<br>
It starts out by being randomly between 0 and 1 but the more steps it is trained the closer the prediction goes to 0 in eval.</p>

ptrblck: <p>Could you post your model definition, please?</p>

Oliver: <p>It’s rather big, the main components are contractions and expansions:<br>
So I have a few contractions, and then a few expansion in the end leading to an output that is fed through a sigmoid.</p>
<pre><code class="lang-auto">class ContractingBlock(nn.Module):

    def __init__(self, input_channels, use_dropout=False, use_bn=True):
        super(ContractingBlock, self).__init__()
        self.conv1 = nn.Conv2d(input_channels, input_channels * 2, kernel_size=3, padding=1)
        self.conv2 = nn.Conv2d(input_channels * 2, input_channels * 2, kernel_size=3, padding=1)
        self.activation = nn.LeakyReLU(0.2)
        self.maxpool = nn.MaxPool2d(kernel_size=2, stride=2)
        if use_bn:
            self.batchnorm = nn.BatchNorm2d(input_channels * 2)
        self.use_bn = use_bn
        if use_dropout:
            self.dropout = nn.Dropout()
        self.use_dropout = use_dropout

    def forward(self, x):
=
        x = self.conv1(x)
        if self.use_bn:
            x = self.batchnorm(x)
        if self.use_dropout:
            x = self.dropout(x)
        x = self.activation(x)
        x = self.conv2(x)
        if self.use_bn:
            x = self.batchnorm(x)
        if self.use_dropout:
            x = self.dropout(x)
        x = self.activation(x)
        x = self.maxpool(x)
        return x
</code></pre>
<pre><code class="lang-auto">class ExpandingBlock(nn.Module):
    def __init__(self, input_channels, use_dropout=False, use_bn=True):
        super(ExpandingBlock, self).__init__()
        self.upsample = nn.Upsample(scale_factor=2, mode='bilinear', align_corners=True)
        self.conv1 = nn.Conv2d(input_channels, input_channels // 2, kernel_size=2)
        self.conv2 = nn.Conv2d(input_channels, input_channels // 2, kernel_size=3, padding=1)
        self.conv3 = nn.Conv2d(input_channels // 2, input_channels // 2, kernel_size=2, padding=1)
        if use_bn:
            self.batchnorm = nn.BatchNorm2d(input_channels // 2)
        self.use_bn = use_bn
        self.activation = nn.ReLU()
        if use_dropout:
            self.dropout = nn.Dropout()
        self.use_dropout = use_dropout

    def forward(self, x, skip_con_x):
        x = self.upsample(x)
        x = self.conv1(x)
        skip_con_x = crop(skip_con_x, x.shape)
        x = torch.cat([x, skip_con_x], axis=1)
        x = self.conv2(x)
        if self.use_bn:
            x = self.batchnorm(x)
        if self.use_dropout:
            x = self.dropout(x)
        x = self.activation(x)
        x = self.conv3(x)
        if self.use_bn:
            x = self.batchnorm(x)
        if self.use_dropout:
            x = self.dropout(x)
        x = self.activation(x)
        return x
</code></pre>
<p>So when I set <code>use_bn=False</code> I still get the same behaviour.</p>

ptrblck: <p>I assume you are retraining the model with <code>use_bn=False</code> and once it converged you are still seeing static outputs after calling <code>model.eval()</code>?</p>

Oliver: <p>Yes, though if I put it back to model.train and just feed it to get <code>pred = model(image)</code> it’s fine.</p>
<p>I also tried it in lightning and I have the same behaviour in <code>validation_step</code>.</p>
<p>Thanks a lot for reviewing this with me!</p>

ptrblck: <p>That’s interesting as it seems the dropout layers are causing this effect then, which I haven’t seen before.<br>
Is calling <code>model.eval()</code> and <code>.train()</code> on all <code>nn.Dropout</code> layers also reproducing the effect?</p>

Oliver: <p>I’ve actually tried with turning off all dropout layers too and I still get the same.<br>
Hence I thought it might be the upsampling.<br>
Or, there is some layering that causes <code>model.eval()</code> to not trickle through to the individual blocks below.</p>
<p>When you say <code>.train()</code> on all <code>nn.Droupout</code> do you mean I should manually call it on each of the individual dropout activations? I haven’t tried that. Nor did I try that on the batchnormalization layers.</p>
<p>All I did was call <code>model.eval()</code> or <code>model.train()</code></p>

ptrblck: <p>I’m not sure why you think this behavior would be caused by an upsampling layer as it won’t change its behavior between <code>train</code> and <code>eval</code> mode or what’s the reason?</p>
<aside class="quote no-group" data-username="Oliver" data-post="9" data-topic="174633">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/oliver/48/30054_2.png" class="avatar"> Oliver:</div>
<blockquote>
<p>When you say <code>.train()</code> on all <code>nn.Droupout</code> do you mean I should manually call it on each of the individual dropout activations?</p>
</blockquote>
</aside>
<p>Yes. Call <code>model.eval()</code> first, then iterate the <code>model.named_modules()</code> calling <code>.train()</code> on each dropout layer as another test.</p>

Oliver: <p>ok, I’ll try that. thanks!</p>
<p>I was thinking that (it might be the upsampling) because I have nothing else to test <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"> as I did completely without batchnorm and dropout each.</p>

ptrblck: <aside class="quote no-group" data-username="Oliver" data-post="11" data-topic="174633">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/oliver/48/30054_2.png" class="avatar"> Oliver:</div>
<blockquote>
<p>as I did completely without batchnorm and dropout each.</p>
</blockquote>
</aside>
<p>Just to clarify: did you also train your model completely without batchnorm <em>and</em> dropout layers and were still seeing a change in predictions after <code>model.eval()</code> was called?<br>
If so, we should try to narrow down which other layer is changing its behavior during the validation run.</p>

Oliver: <p>yes, though I didn’t try to turn both off. I tried once without any dropout and got the same ( near all zeros after <code>.eval()</code>) and once without any batchnorm. Though I haven’t tried without either. Maybe I’ll try that too <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"> .</p>

Oliver: <p>hi, I tried without both and iterating why calling .train().</p>
<p>I still have the behaviour, but I think I may have an idea now,<br>
do you think it is possible that this is because I have different sets of images and I am first dividing them by 255.0 and then normalizing them to mean = (0.5, 0.5, 0.5) and std = (0.5, 0.5, 0.5) in the dataset?</p>
<p>I very much appreciate the help, I apologize for not having been able to try sooner.</p>

ptrblck: <aside class="quote no-group" data-username="Oliver" data-post="14" data-topic="174633">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/oliver/48/30054_2.png" class="avatar"> Oliver:</div>
<blockquote>
<p>hi, I tried without both and iterating why calling .train().</p>
<p>I still have the behaviour, but I think I may have an idea now,</p>
</blockquote>
</aside>
<p>So you removed all batchnorm and dropout layers, and are still seeing different outputs for <code>model.train()</code> and <code>model.eval()</code>?<br>
If so, this would indicate another layer’s execution depends on the batch size, which we haven’t narrowed down yet.</p>
<aside class="quote no-group" data-username="Oliver" data-post="14" data-topic="174633">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/oliver/48/30054_2.png" class="avatar"> Oliver:</div>
<blockquote>
<p>do you think it is possible that this is because I have different sets of images and I am first dividing them by 255.0 and then normalizing them to mean = (0.5, 0.5, 0.5) and std = (0.5, 0.5, 0.5) in the dataset?</p>
</blockquote>
</aside>
<p>Could you describe which image sets are used? I assumed you are using the same input data to compare the output of the model after calling <code>.train()</code> and <code>.eval()</code>.</p>

Oliver: <p>Thank you very much.</p>
<p>I am sorry I don’t understand this part, how does a layer other than batchnorm depend on the batch size?<br>
It is possible that because I don’t know this that I am missing an obvious thing.</p>
<p>For the different sets,<br>
I mix them all together and split them into train and evaluation, though they come from different datasets.<br>
It’s a style transfer project (GAN).<br>
So they are the same input data, but not exactly, since I split them in the beginning.<br>
Do you think it possible that it could overfit? The dataset is quite large, about 30k images.</p>

ptrblck: <aside class="quote no-group" data-username="Oliver" data-post="16" data-topic="174633">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/oliver/48/30054_2.png" class="avatar"> Oliver:</div>
<blockquote>
<p>I am sorry I don’t understand this part, how does a layer other than batchnorm depend on the batch size?<br>
It is possible that because I don’t know this that I am missing an obvious thing.</p>
</blockquote>
</aside>
<p>I don’t know which layers you are using, but since you have removed all batchnorm and dropout layer and are still seeing differences between <code>train</code> and <code>eval</code> mode I guess other (maybe custom) layers are using the internal <code>self.training</code> flag to change their behavior. You would have to narrow down which layer is creating the mismatches by e.g. using forward hooks which would allow you to compare each intermediate activation separately.</p>
<aside class="quote no-group" data-username="Oliver" data-post="16" data-topic="174633">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/oliver/48/30054_2.png" class="avatar"> Oliver:</div>
<blockquote>
<p>So they are the same input data, but not exactly, since I split them in the beginning.<br>
Do you think it possible that it could overfit? The dataset is quite large, about 30k images.</p>
</blockquote>
</aside>
<p>I’m not concerned about overfitting and the general training at the moment, but more interested in the unexpected difference in the output of your model using a) exactly the same input data, b) without any usage of batchnorm and dropout layers, and c) since the difference is apparently caused by switching between <code>train</code> and <code>eval</code> mode.</p>

Oliver: <p>Thank you very much.</p>
<p>Aside from the ones above I only use a sigmoid Activation and that only through <code>torch.functional</code> in the forward pass.</p>
<p>After your last comment I was wodnering if it is possible it could overfit, i.e. learn the training data to a point where it memorizes it and just gives blanks for all other inputs, this explains how the output slowly goes from around 0.3/0.4 to 0.01 the longer it trains.</p>
<p>I will try one more thing, I’ll completely remove the batchnorms and dropouts, instead of only setting the flag to False, see how that goes.</p>
<p>Thanks for your patient help!</p>

Oliver: <p>Hmm, ok, so update:</p>
<p>deleted the lines of bn and train completely and similar behaviour but not quite, so I think it’s likely there.<br>
The new behaviour is that that the output is near zero to start with but in both eval and train mode. Then it goes to 0 completely. I.e. it only outputs images of mean == 0.</p>
<p>This behaviour is so strange in that I am quite certain now that I’ve introduced a bug somewhere.</p>
<p>So, other than being a bit slower and undeterministic (dropout), is there any other downside to just predicting and using the model only in <code>.train()</code> mode? As it works quite well there</p>

ptrblck: <aside class="quote no-group" data-username="Oliver" data-post="18" data-topic="174633">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/oliver/48/30054_2.png" class="avatar"> Oliver:</div>
<blockquote>
<p>I will try one more thing, I’ll completely remove the batchnorms and dropouts, instead of only setting the flag to False, see how that goes.</p>
</blockquote>
</aside>
<p>I assumed you already did this.</p>
<aside class="quote no-group" data-username="Oliver" data-post="19" data-topic="174633">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/oliver/48/30054_2.png" class="avatar"> Oliver:</div>
<blockquote>
<p>So, other than being a bit slower and undeterministic (dropout), is there any other downside to just predicting and using the model only in <code>.train()</code> mode?</p>
</blockquote>
</aside>
<p>I’m not aware of more drawbacks than the ones you already mentioned:</p>
<ul>
<li>predictions for exactly the same input sample will have randomness</li>
<li>you are wasting compute</li>
<li>dropout is often creating worse performing models as the actual model capacity is smaller.</li>
</ul>