Model works for inference without .eval() and doesn't with it

username: JAllsopp
cooked: <p>Hi,<br>
I’ve trained my model and works great, (it’s trying to count points in images) and when I try to infer using it and the trained weights, I get a sensible value but it varies a couple of percent when run on the same data.<br>
(<br>
I realised I’d not set the model to eval using model.eval() before doing the inference, thinking that not turning off the dropout was the source of the variation. However, when I set model.eval() the variation disappears but the answer is completely wrong now. This stability suggests that it’s not numerical instability causing the initial variation.</p>
<p>I’m also worried that it’s the probabilities for the dropout I’m using</p>
<p>My model is this;</p>
<pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(1,75, kernel_size=3, padding=1)
        self.conv2 = nn.Conv2d(75,50, kernel_size=3, padding=1)
        self.conv3 = nn.Conv2d(50,25, kernel_size=3, padding=1)
        self.bn1 = nn.BatchNorm2d(75)
        self.bn2 = nn.BatchNorm2d(50)
        self.bn3 = nn.BatchNorm2d(25)
        self.dropout1 = nn.Dropout(0.2)
        self.dropout2 = nn.Dropout(0.3)
        self.fc1 = nn.Linear(25*66*50, 512)
        self.fc2 = nn.Linear(512,512)
        self.fc3 = nn.Linear(512,1)
        self.float()

    def forward(self, x):
        out = F.max_pool2d(self.bn1(torch.relu(self.conv1(x))),2)
        out = F.max_pool2d(self.bn2(self.dropout1(torch.relu(self.conv2(out)))),2)
        out = F.max_pool2d(self.bn3(torch.relu(self.conv3(out))),2)
        out = out.view(-1,25*66*50)
        out = torch.relu(self.fc1(out))
        out = self.dropout2(out)
        out = torch.relu(self.fc2(out))
        out = self.fc3(out)
        return out
</code></pre>
<p>Here’s the inference code;</p>
<pre><code class="lang-auto">if __name__=="__main__":
    model=Net()
    model.eval()
    device_in_use = get_device()
    weights_file = 'weights.pt'
    model.load_state_dict(torch.load(weights_file,map_location=device_in_use))
    image_to_infer = get_data(sys.argv[1])
    image_to_infer.to(device=device_in_use)

    output = model(image_to_infer).item()

    print(output*max_ions)
</code></pre>
<p>The training loop looks like this;</p>
<pre><code class="lang-auto">def runModel(model, n_epochs, max_ions, optimizer,loss_fn, train_loader, validation_loader, save_weights_as, save\
_graph_as, writerSuffix="" ):
    device_in_use = get_device()
    max_ions = int(max_ions)
    writer = SummaryWriter()
    model.to(device=device_in_use)
    for epoch in range(1, n_epochs + 1):
        loss_train = 0.0
        for imgs, labels in train_loader:
            imgs_gpu= imgs.to(device=device_in_use)
            outputs = model(imgs_gpu)
            normalised_labels = normalise_label2(labels,max_ions).to(device=device_in_use)
            loss = loss_fn(outputs,normalised_labels)
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()
            current_loss = loss.item()
            loss_train += current_loss
        writer.add_scalar("loss/training", loss_train/len(train_loader))
        nll =[]
        ol=[]

        validation_loss = 0
        for imgs, labels in validation_loader:
            imgs= imgs.to(device=device_in_use)
            outputs = model(imgs)
            normalised_labels = normalise_label2(labels,max_ions).to(device=device_in_use)
            #print(f'{outputs.shape} {normalised_labels.shape}')
            output_list = outputs.squeeze().tolist()
            normalised_labels_list = normalised_labels.squeeze().tolist()
            nll.extend(normalised_labels_list)
            ol.extend(output_list)
            loss = loss_fn(outputs,normalised_labels)
            validation_loss += loss.item()

        print(f"Epoch:{epoch} Loss:{loss_train/len(train_loader)} Validation:{validation_loss/len(validation_load\er)} len train_loader:{len(train_loader)} len validation_loader: {len(validation_loader)}")
        writer.add_scalar("loss/validation", validation_loss/len(validation_loader))
        nll = [i * max_ions for i in nll]
        ol = [i * max_ions for i in ol]
        save_weights(model, save_weights_as, epoch)
    draw_box_plot(nll, ol, save_graph_as)
    writer.close()
</code></pre>
<p>Reading through this, I’m wondering if missing out model.train() in the training cycle is the cause of this? Also, do I have to change this back to model.eval() for the validation step?</p>
<p>Love using PyTorch, this is the first project I’ve done with it! Thanks for any advice you can offer.</p>

username: ptrblck
cooked: <p>Yes, calling <code>model.eval()</code> during validation is the common approach as it would disable dropout layers and use the running stats of batchnorm layers, which would avoid creating batch size - dependent outputs. However, the latter seems to cause issues in your model and the running stats of batchnorm layers might not have properly converged or generally might not represent the stats of the forward activations. This issue is discussed a few times here already and you could e.g. play around with the <code>momentum</code> to see if it could help.</p>

username: JAllsopp
cooked: <p>Hi,<br>
Thanks for getting back to me. I started a run last night (for ref, using 500K training images and 70K validation images) with model.train() in the training section and model.eval() in the validation section, and although my train loss is coming down nicely and about where I’d expect, the validation loss is two orders of magnitude greater than I’d expect.</p>
<p>However, I expected something was a bit fishy with the validation loss as it was about 30% of the training loss, which I put down to dropout in the training section.</p>
<p>It’s really odd, by ignoring model.train and model.eval() I seem to have created a huge Frankenstein ensemble network that gives really good results, just slightly inconsistent between runs!</p>

username: JAllsopp
cooked: <p>Also, here’s the optimiser I’m using;<br>
<code>optim.Adam(model.parameters(), lr= learning_rate)</code></p>
<p>where the learning rate is 1e-4.</p>

username: AlphaBetaGamma96
cooked: <p>Hi <a class="mention" href="/u/jallsopp">@JAllsopp</a>,</p>
<aside class="quote no-group" data-username="JAllsopp" data-post="1" data-topic="185585">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/j/b5ac83/48.png" class="avatar"> JAllsopp:</div>
<blockquote>
<pre><code class="lang-auto">        self.float()
</code></pre>
</blockquote>
</aside>
<p>Could this be the issue? I know when casting a model to a new device, it’s not self-referential, i.e.,</p>
<pre><code class="lang-auto">model = model.to("cuda") #works
model.to("cuda") #doesn't work
</code></pre>
<p>I believe (<a class="mention" href="/u/ptrblck">@ptrblck</a> will know for sure) that the same applies for casting to a new dtype.</p>

username: ptrblck
cooked: <p>The <code>to()</code> operation (and direct transformations) are applied recursively on modules (and all properly registered parameters, buffers, and submodules):</p>
<pre><code class="lang-python">model = torchvision.models.resnet18()

print(model.conv1.weight.dtype)
# torch.float32

model.double()
print(model.conv1.weight.dtype)
# torch.float64

model.to(torch.bfloat16)
print(model.conv1.weight.dtype)
# torch.bfloat16
</code></pre>
<p>so this shouldn’t be an issue.<br>
The <code>model.to("cuda")</code> call should also work, so it would be interesting to hear if you’ve encountered any issues with it.</p>