Can't explain the difference test loss vs train loss

username: moher
cooked: <p>Hi, Thanks in advance for looking into my question.<br>
let me first describe the problem:</p>
<p>I have designed a very simple network to classify mnist dataset.<br>
I have refactor as much as possible and make a clean code.<br>
so everything ran smoothly and I would get good results on train and validation set.<br>
<strong>until I read</strong> that If you are using dropout in your modules, then you have to use <code>net.train()</code> in training phase, and <code>net.eval()</code> when in testing.</p>
<p>so I added the changes in my code. now, I can’t explain test loss vs train loss.<br>
train loss is fine, and is decreasing steadily as expected. but test loss is way much lower than train loss from the first epoch until to the end and does not change that much!<br>
this is so weird, and I can’t find out what I am doing wrong.</p>
<p>for your reference I have put the loss and accuracy plots during epochs here:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/5/0/50275704555b147613fae12d4db91d6e3188a513.jpeg" alt="1" data-base62-sha1="br4tWFoY6Bh5A5tQUFbpS1GFPRV" width="470" height="330"></p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/8/b/8b43b3213b54e3eb938aaa2aca02ed2d41426890.jpeg" alt="2" data-base62-sha1="jRZpGTNBBuTuBhEOZFrYE8e6sCY" width="480" height="327"></p>
<p>following, I will upload my entire code.</p>
<h3><a name="imports-1" class="anchor" href="#imports-1"></a>imports</h3>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.optim as optim
import torchvision
import torchvision.transforms as transforms
import matplotlib.pyplot as plt
</code></pre>
<h3><a name="defining-network-custom-nnmodule-2" class="anchor" href="#defining-network-custom-nnmodule-2"></a>Defining Network (Custom nn.Module)</h3>
<pre><code class="lang-auto">class Flatten(nn.Module):
    def forward(self, x):
        batch_size = x.size()[0]
        return x.view(batch_size, -1)

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(1, 6, 3)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.maxpool = nn.MaxPool2d(2, 2)
        self.relu = nn.ReLU()
        self.flatten = Flatten()
        self.dropout = nn.Dropout(0.25)
        self.fc1 = nn.Linear(256, 64)
        self.fc2 = nn.Linear(64, 10)

    def forward(self, x):
        x = self.maxpool(self.relu(self.conv1(x)))
        x = self.maxpool(self.relu(self.conv2(x)))
        x = self.flatten(x)
        x = self.relu(self.dropout(self.fc1(x)))
        x = self.dropout(self.fc2(x))
        return x
</code></pre>
<h3><a name="problem-class-3" class="anchor" href="#problem-class-3"></a>Problem Class</h3>
<p>this class, contains the logic of training and validation. this is just for cleaner code.</p>
<pre><code class="lang-auto">class Problem:
    def __init__(self, net, criterion, opt, device):
        self.net = net
        self.criterion = criterion
        self.opt = opt
        self.device = device
    
    def count_correct(self, y_pred, y):
        return torch.sum(torch.argmax(y_pred, dim=1)==y)
    
    def predict(self, X):
        self.net.eval()
        return self.net(X)
    
    def train(self, trainloader, epochs, testloader=None, verbose=1):
        losses = []
        accs = []
        test_losses = []
        test_accs = []
        for epoch in range(epochs):
            self.net.train()
            
            correct_predictions = 0
            total_loss = 0

            for i, data in enumerate(trainloader):
                X, y = data[0].to(self.device), data[1].to(self.device)
                y_pred = self.net(X)
                self.opt.zero_grad()
                loss = self.criterion(y_pred, y)
                loss.backward()
                self.opt.step()

                correct_predictions += self.count_correct(y_pred, y).item()
                total_loss += loss.item()

            total_loss /= len(trainloader)
            correct_predictions /= (len(trainloader)*trainloader.batch_size)

            losses.append(total_loss)
            accs.append(correct_predictions)
            
            if testloader is not None:
                test_loss, test_acc = self.validate(testloader)
                test_losses.append(test_loss)
                test_accs.append(test_acc)
            
            if verbose == 1 and testloader is not None:
                print("Epoch: {}; Loss: {}; Acc: {}; Test Loss: {}; Test Acc: {}"
                                          .format(epoch, total_loss, correct_predictions, test_loss, test_acc))
            elif verbose == 1:
                print("Epoch: {}; Loss: {}; Acc: {}".format(epoch, total_loss, correct_predictions))

        return losses, accs, test_losses, test_accs

    def validate(self, testloader):
        self.net.eval()
        total_loss = 0
        correct_predictions = 0
        for i, data in enumerate(testloader):
            X, y = data[0].to(self.device), data[1].to(self.device)
            y_pred = self.net(X)
            loss = self.criterion(y_pred, y)

            total_loss += loss.item()
            correct_predictions += self.count_correct(y_pred, y).item()

        total_loss /= len(testloader)
        correct_predictions /= (len(testloader)*testloader.batch_size)

        return total_loss, correct_predictions
</code></pre>
<h3><a name="creating-instances-and-training-the-model-4" class="anchor" href="#creating-instances-and-training-the-model-4"></a>Creating Instances and Training the Model</h3>
<pre><code class="lang-auto"># configurations
lr = 0.01
batch_size = 250
epochs = 20
device = torch.device("cuda") if torch.cuda.is_available() else torch.device("cpu")

# preparing train and test dataloaders
transform = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.5,), (0.5,))])
train_ds = torchvision.datasets.MNIST('/MyFiles/Datasets', train=True, download=True, transform=transform)
test_ds = torchvision.datasets.MNIST('/MyFiles/Datasets', train=False, download=True, transform=transform)
trainloader = torch.utils.data.DataLoader(train_ds, batch_size, shuffle=True)
testloader = torch.utils.data.DataLoader(test_ds, 2*batch_size, shuffle=False)

# creating instances
net = Net()
net.to(device)
criterion = nn.CrossEntropyLoss()
opt = optim.SGD(net.parameters(), lr, 0.9)
problem = Problem(net, criterion, opt, device)

# let model train
train_losses, train_accs, test_losses, test_accs = problem.train(trainloader, epochs, testloader, verbose=1)

# plotting test/train loss
plt.title('loss')
plt.plot(train_losses), plt.plot(test_losses)
plt.legend(['train', 'test'])
plt.show()

# plotting test/train accuracy
plt.title('accuracy')
plt.plot(train_accs), plt.plot(test_accs)
plt.legend(['train', 'test'])
plt.show()
</code></pre>

username: ZdsAlpha
cooked: <p>I don’t think there is something wrong with this. Also I think it is really good that test loss is much lower which means its generalizing well. Here is a little explanation why?<br>
Dropout layer sets some of features to 0. And gradient does not flow backward through those features. In your case 25% of features are dropped making it harder for model to make predictions. In evaluation dropout layer does nothing, making it easier for model to predict. It very rough explanation but something like this happens. If you increase features (e.g increase number of out_channels from 16 to 32 in conv2) train accuracy will increase if thats what you want.</p>

username: moher
cooked: <p>so you don’t suspect anything is wrong?<br>
I understand your explanation, but aren’t the activation maps divided by 1/p to account for those dropped out neurons?<br>
I might not understand it, but at least I want to make sure that I am not seeing this because I have done something wrong.</p>

username: ZdsAlpha
cooked: <aside class="quote no-group" data-username="moher" data-post="1" data-topic="73869">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/moher/48/21700_2.png" class="avatar"> moher:</div>
<blockquote>
<p>net.eval()</p>
</blockquote>
</aside>
<p>Try putting net.train() in place of net.eval() for testing. See if that brings both losses equal.<br>
If dropout is the only thing that caused this change. You shouldn’t worry. Again try increasing features in each layer, it should bring both curves closer.</p>

username: moher
cooked: <p>I have done that, and this problem goes away.<br>
but why this problem  exists when net.eval() is used when computing loss for validation set?</p>

username: ZdsAlpha
cooked: <aside class="quote no-group" data-username="moher" data-post="3" data-topic="73869">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/moher/48/21700_2.png" class="avatar"> moher:</div>
<blockquote>
<p>I understand your explanation, but aren’t the activation maps divided by 1/p to account for those dropped out neurons?</p>
</blockquote>
</aside>
<p>This is not a problem. This is normal. When there are not much feature to account for dropped one, something like this can happen.</p>