Model predictions changing with no_grad and .eval()

username: drj3122
cooked: <p>My model predictions keep changing even though I have set <code>model.eval()</code> and <code>no_grad()</code>. I am following examples from <em>Natural Language Processing with PyTorch</em>. <code>generate_batches</code> is from the book. <code>demo_model</code> is a class that includes <code>model</code> (torch model) and some other attributes.</p>
<pre><code class="lang-auto">demo_model.data.set_split('val')
batch_generator = generate_batches(demo_model.data, batch_size=10, device=demo_model.device, shuffle = False, #demo_model.data._lookup_dict['val'][1]
                                  drop_last = False)
bt = next(batch_generator)
demo_model.model.eval()
with torch.no_grad():    
    y_pred =  demo_model.model(bt['demo_feat'].type(torch.FloatTensor))
bt['demo_feat'], bt['y_feat'], y_pred.flatten()#.sort()

def generate_batches(dataset, batch_size, shuffle=True, drop_last=True, device="cpu"): 
    """
    A generator function which wraps the PyTorch DataLoader. It will 
      ensure each tensor is on the write device location.
    """
    dataloader = DataLoader(dataset=dataset, batch_size=batch_size, shuffle=shuffle, drop_last=drop_last)

    for data_dict in dataloader:
        out_data_dict = {}
        for name, tensor in data_dict.items():
            out_data_dict[name] = data_dict[name].to(device)
        yield out_data_dict  
</code></pre>
<p>My model is a simple 2 layer model with dropout and relu.</p>
<pre><code class="lang-auto">class Demo(nn.Module):

    def __init__(self):
        super(Demo, self).__init__()
        self.agefc1 = nn.Linear(2, 20)
        self.agefc2 = nn.Linear(20, 1)

    def forward(self, demo_in):        
        demo_in = self.agefc1(demo_in)
        demo_in = F.relu(F.dropout(demo_in, p = .5))
        demo_in = self.agefc2(demo_in)
        demo_in = F.dropout(demo_in, p = .5)

        return demo_in
</code></pre>
<p>Example output (2 separate runs in jupyter lab):</p>
<pre><code class="lang-auto">(tensor([[ 1.1856,  0.0000],
         [ 0.6062,  0.0000],
         [-0.4473,  0.0000],
         [ 1.0276,  1.0000],
         [ 1.5016,  0.0000],
         [ 0.1848,  0.0000],
         [ 0.8695,  0.0000],
         [ 1.9757,  1.0000],
         [-0.9740,  0.0000],
         [-1.3427,  0.0000]], dtype=torch.float64),
 tensor([712.5783,  69.1400, 212.7758,  53.3617, 586.6400,  36.5867,  59.6350,
         482.8508, 450.3008, 163.0558], dtype=torch.float64),
 tensor([43.3376,  0.0000, 16.7106, 48.0151, 50.6728, 43.9258, 28.6625, 10.7091,
          4.8271,  0.0000]))
</code></pre>
<pre><code class="lang-auto">(tensor([[ 1.1856,  0.0000],
         [ 0.6062,  0.0000],
         [-0.4473,  0.0000],
         [ 1.0276,  1.0000],
         [ 1.5016,  0.0000],
         [ 0.1848,  0.0000],
         [ 0.8695,  0.0000],
         [ 1.9757,  1.0000],
         [-0.9740,  0.0000],
         [-1.3427,  0.0000]], dtype=torch.float64),
 tensor([712.5783,  69.1400, 212.7758,  53.3617, 586.6400,  36.5867,  59.6350,
         482.8508, 450.3008, 163.0558], dtype=torch.float64),
 tensor([37.5727,  0.0000, 19.0388, 57.4453,  0.0000,  0.0000, 35.3358, 46.8472,
         32.5002, 22.6629]))
</code></pre>
<p>I’m guessing I’m missing something simple.</p>

username: drj3122
cooked: <p>As expected, it was very simple. <code>F.dropout</code> doesn’t disable dropout for eval mode. From <a href="https://stackoverflow.com/questions/53419474/using-dropout-in-pytorch-nn-dropout-vs-f-dropout/53452827#53452827" rel="noopener nofollow ugc">stackoverflow</a>:</p>
<blockquote>
<p>Both are completely equivalent in terms of applying dropout and even though the differences in usage are not that big, there are some reasons to favour the <code>nn.Dropout</code> over <code>nn.functional.dropout</code>:<br>
Dropout is designed to be only applied during training, so when doing predictions or evaluation of the model you want dropout to be turned off.<br>
The dropout module <code>nn.Dropout</code> conveniently handles this and shuts dropout off as soon as your model enters evaluation mode, while the functional dropout does not care about the evaluation / prediction mode.</p>
</blockquote>
<p>Code update:</p>
<p>class Demo(nn.Module):</p>
<pre><code class="lang-auto">    def __init__(self):
        super(Demo, self).__init__()
        self.agefc1 = nn.Linear(2, 20)
        self.agefc2 = nn.Linear(20, 1)
        self.drop_layer = nn.Dropout(p = .5)

    def forward(self, demo_in):        
        demo_in = self.agefc1(demo_in)
        demo_in = F.relu(self.drop_layer(demo_in))
        demo_in = self.agefc2(demo_in)
        demo_in = self.drop_layer(demo_in)

        return demo_in
</code></pre>

username: ptrblck
cooked: <p>The StackOverflow post is unfortunately missing the proper usage of the functional API and sets the <code>training</code> argument of <code>F.dropout</code> to fixed <code>True</code> or <code>False</code> values.<br>
To properly enable/disable <code>F.dropout</code> during training and validation use <code>out = F.dropout(x, training=self.training)</code>, since the <code>self.trainng</code> attribute will be switched by calling <code>model.train()</code> and <code>model.eval()</code>. The <code>nn.Dropout</code> module does indeed the same as seen <a href="https://github.com/pytorch/pytorch/blob/2e49c5dc37bce5ce8ba463a59aacb5d3e4a638b6/torch/nn/modules/dropout.py#L57-L58">here</a>.</p>