Dropout in eval mode of GoogLeNet: model(x) or layer(layer(x))

acoder_acoder: <p>Hello!</p>
<p>I find a really tricky situation for the <code>dropout</code> function in the pretrained GoogLeNet. The output results between directly executing the whole model and executing the model layer by layer are different. It seems that when executing the model layer by layer, the <code>dropout</code> function does not work while it works if executing the whole model.</p>
<p>Details are shown below. Thank you!</p>
<p>Version info (google colab):<br>
torch – 1.7.0+cu101<br>
torchvision – 0.8.1+cu101</p>
<pre><code class="lang-auto">import torch
from torch import nn
from torchvision import models
x = torch.randn(10, 3, 224, 224)

glnet = models.googlenet(pretrained=True)
glnet.eval
</code></pre>
<pre><code class="lang-auto">glnet(x)
</code></pre>
<p>outputs</p>
<pre><code class="lang-auto">tensor([[ 0.4788,  0.8616,  1.1871,  ...,  0.3228, -0.7227, -0.7120],
        [-1.3281,  0.3395, -0.9889,  ..., -0.4708, -0.4659,  1.5156],
        [ 1.0499,  1.3531,  0.3119,  ..., -1.3807,  0.2523, -1.0316],
        ...,
        [-0.5240, -1.7480, -0.0047,  ..., -0.8630, -0.2038, -0.6803],
        [ 1.1060,  1.3443, -0.1791,  ..., -1.1149,  1.2490, -0.0174],
        [-1.9743, -0.2748, -0.5077,  ..., -1.3009,  0.3457,  0.0843]],
       grad_fn=&lt;AddmmBackward&gt;)
</code></pre>
<p>while</p>
<pre><code class="lang-auto">out = x
for name, layer in glnet.named_children():
    if name == 'fc':
        out = nn.Flatten()(out)
    out = layer(out)
out
</code></pre>
<p>outputs</p>
<pre><code class="lang-auto">tensor([[ 1.2480,  0.3792,  2.1190,  ...,  0.1834,  0.3580, -1.0001],
        [-0.7878, -0.0876, -0.8435,  ..., -0.8066,  0.2189,  1.0071],
        [ 1.3254, -0.1794, -0.4596,  ..., -0.1602, -0.1182, -0.5954],
        ...,
        [-0.8594, -0.8541,  0.2045,  ..., -0.7589,  0.4976, -0.4356],
        [-0.6665, -0.1939,  0.1854,  ..., -1.1892,  1.4693,  0.5318],
        [-2.6076, -0.0268, -1.8729,  ..., -1.2042,  0.0576,  0.2504]],
       grad_fn=&lt;AddmmBackward&gt;)
</code></pre>

Eta_C: <p>It seems that NOT ONLY THE DROPOUT.<br>
Run this following code and you can find that <code>features[0]</code> is different from <code>features[1]</code> !</p>
<pre><code class="lang-python">import torch
from torch import nn
from torchvision import models

features = []

def hook(module, inputs, outputs):
    features.append(outputs.clone())

x = torch.randn(10, 3, 224, 224).float()
glnet = models.googlenet(pretrained=True)
glnet.eval()

glnet.conv1.register_forward_hook(hook)

with torch.no_grad():
    model_output = glnet(x.clone())
    print(model_output)

    out = x.clone()
    for name, layer in glnet.named_children():
        if name == 'fc':
            out = nn.Flatten()(out)
        out = layer(out)
    print(out)

for feature in features:
    print("feature", feature[0, 0, 0, :10])
</code></pre>
<p>It’s so strange…I don’t know why…</p>

ptrblck: <p>The <code>forward</code> method for this model is not only calling all submodules sequentially, but also transforming the input, if the pretrained model is used as seen <a href="https://github.com/pytorch/vision/blob/ae0d80b3c52dc98b3a9763bdb974c3ef7b6eb83d/torchvision/models/googlenet.py#L207">here</a>.<br>
Generally, I would not recommend to wrap all submodules in e.g. an <code>nn.Sequential</code> container or to call them sequentially without checking the <code>forward</code> method first and make sure that no functional calls would be dropped.</p>
<p>CC <a class="mention" href="/u/acoder_acoder">@acoder_acoder</a></p>

acoder_acoder: <p>Thanks, <a class="mention" href="/u/ptrblck">@ptrblck</a>!<br>
In the <code>forward()</code> function, though <code>_transform_input()</code> is called, it does nothing for the input. Since a parameter <a href="https://github.com/pytorch/vision/blob/ae0d80b3c52dc98b3a9763bdb974c3ef7b6eb83d/torchvision/models/googlenet.py#L69" rel="noopener nofollow ugc">transform_input</a> is controlling whether to use the transform function.</p>
<p>In fact, not only for the GoogLeNet, but the situation also occurs for some other networks. So I wonder is there anything weird for the <code>dropout</code> function.  Thank you very much!</p>

acoder_acoder: <p><a class="mention" href="/u/eta_c">@Eta_C</a> , Thank you! According to ptrblck’s answer. The problem is the <code>transform_input</code> parameter. I thought the parameter <code>transform_input</code> was False, while it was True…<br>
That is the reason…<br>
<a href="https://github.com/pytorch/vision/blob/ae0d80b3c52dc98b3a9763bdb974c3ef7b6eb83d/torchvision/models/googlenet.py#L40" rel="noopener nofollow ugc">This line</a></p>
<p>Thank you very much!</p>

ptrblck: <aside class="quote no-group" data-username="acoder_acoder" data-post="4" data-topic="108951">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/acoder_acoder/48/23900_2.png" class="avatar"> acoder_acoder:</div>
<blockquote>
<p>In the <code>forward()</code> function, though <code>_transform_input()</code> is called, it does nothing for the input. Since a parameter <a href="https://github.com/pytorch/vision/blob/ae0d80b3c52dc98b3a9763bdb974c3ef7b6eb83d/torchvision/models/googlenet.py#L69">transform_input</a> is controlling whether to use the transform function.</p>
</blockquote>
</aside>
<p>Since this attribute is set to <code>True</code> in your example, the input will be transformed.<br>
Using it also yields the same results:</p>
<pre data-code-wrap="python"><code class="lang-python">x = torch.randn(10, 3, 224, 224)
glnet = models.googlenet(pretrained=True)
glnet.eval()
res = glnet(x)

out = x
if glnet.transform_input:
    out = glnet._transform_input(out)
for name, layer in glnet.named_children():
    if name == 'fc':
        out = nn.Flatten()(out)
    out = layer(out)

print((out - res).abs().max())
&gt; tensor(0., grad_fn=&lt;MaxBackward1&gt;)
</code></pre>