Why different validation and train loss for the same data

username: banikr
cooked: <p>Dear Altruists,<br>
I am running some regression analysis with 3D MRI data. But I am getting too low validation loss with respect to the training loss. For 5 fold validation, each having only one epoch(as a trial) I am getting the following loss curves:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/1/4/14c621fb77b2efde11755c2280accc6ec78b95ec.png" alt="image" data-base62-sha1="2XM2U6zTr1iPBnJvcU6YB9sR4Ys" width="480" height="360"></p>
<p>To debug the issue, I used the same input and target for training and validation setups in my codes.</p>
<p><strong>For building model:</strong></p>
<pre><code class="lang-auto">def build_model():
    torch.cuda.manual_seed(1)
    model = ResNet3D().to(device)
    model.apply(weights_init)
    Optimizer = optim.Adam(model.parameters(), lr=lr)
    Criterion = nn.MSELoss().cuda()
    return model, Optimizer, Criterion
</code></pre>
<p>I will run the <code>build_model</code> function for every fold that is my plan.</p>
<pre><code class="lang-auto">Net, Optimizer, Criterion = build_model()
train_loader = data.DataLoader(train_dataset, batch_size=batchsize, shuffle=True) #for every cross-validation fold train_dataset changes.
</code></pre>
<p><strong>Training section:</strong></p>
<pre><code class="lang-auto">Net.train()
sample = train_loader.dataset[0]
input = sample[0]               #just taking one sample for loss calculation
input = input.unsqueeze(0).float().to(device)   #torch.Size([1, 86, 110, 78])
target = sample[1]
target = target.unsqueeze(0).float().to(device) #torch.Size([1, 256]) 

output = Net(input)   #torch.Size([1, 256])
Optimizer.zero_grad()

loss = Criterion(output,target)
print("Loss in training:",loss.item())
</code></pre>
<p>Loss in training: 14.414422035217285</p>
<p>** <em>I did not update Net weights <code>Optimizer.step()</code> or neither did back propagation <code>loss.backward()</code></em></p>
<p><strong>Validation section:</strong></p>
<pre><code class="lang-auto">Net.eval()
with torch.no_grad():
    outputvalid = Net(input)   #same input in same Net 
    lossvalid = Criterion(outputvalid,target)
print("Loss in validation: ",lossvalid.item())
</code></pre>
<p>Loss in validation: 2.8760955333709717</p>
<p>What could be the reason??</p>

username: ptrblck
cooked: <p>If you are using a high drop probability in <code>Dropout</code> layers, this effect is usually expected.<br>
Alternatively, the initial running stats of the batch norm layers might be better than the actual batch statistics, although this effect would be new to me. Usually you would see a lower training loss than validation.</p>
<p>Could you try to call <code>eval()</code> on all batch norm and dropout layers during training and check the losses again?</p>

username: banikr
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a><br>
I am not using any dropout layers. The architecture I am using can be seen below:</p>
<pre><code class="lang-auto">ResNet3D(
  (strConv1): Conv3d(64, 64, kernel_size=(3, 3, 3), stride=(2, 2, 2), padding=(1, 1, 1))
  (strConv2): Conv3d(128, 128, kernel_size=(3, 3, 3), stride=(2, 2, 2), padding=(1, 1, 1))
  (strConv3): Conv3d(256, 256, kernel_size=(3, 3, 3), stride=(2, 2, 2), padding=(1, 1, 1))
  (conv_block1_32): ConvBlock(
    (conv1): Conv3d(1, 32, kernel_size=(3, 3, 3), stride=(1, 1, 1), padding=(1, 1, 1))
    (bn1): BatchNorm3d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (conv2): Conv3d(32, 32, kernel_size=(3, 3, 3), stride=(1, 1, 1), padding=(1, 1, 1))
    (bn2): BatchNorm3d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  )
  (conv_block32_64): residualUnit(
    (conv1): Conv3d(32, 64, kernel_size=(3, 3, 3), stride=(1, 1, 1), padding=(1, 1, 1))
    (bn1): BatchNorm3d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (conv2): Conv3d(64, 64, kernel_size=(3, 3, 3), stride=(1, 1, 1), padding=(1, 1, 1))
    (bn2): BatchNorm3d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (convX): Conv3d(32, 64, kernel_size=(1, 1, 1), stride=(1, 1, 1))
    (bnX): BatchNorm3d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  )
  (conv_block64_128): residualUnit(
    (conv1): Conv3d(64, 128, kernel_size=(3, 3, 3), stride=(1, 1, 1), padding=(1, 1, 1))
    (bn1): BatchNorm3d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (conv2): Conv3d(128, 128, kernel_size=(3, 3, 3), stride=(1, 1, 1), padding=(1, 1, 1))
    (bn2): BatchNorm3d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (convX): Conv3d(64, 128, kernel_size=(1, 1, 1), stride=(1, 1, 1))
    (bnX): BatchNorm3d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  )
  (conv_block128_256): residualUnit(
    (conv1): Conv3d(128, 256, kernel_size=(3, 3, 3), stride=(1, 1, 1), padding=(1, 1, 1))
    (bn1): BatchNorm3d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (conv2): Conv3d(256, 256, kernel_size=(3, 3, 3), stride=(1, 1, 1), padding=(1, 1, 1))
    (bn2): BatchNorm3d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (convX): Conv3d(128, 256, kernel_size=(1, 1, 1), stride=(1, 1, 1))
    (bnX): BatchNorm3d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  )
  (conv1d): ConvBlock1D(
    (conv1): Conv1d(1, 2, kernel_size=(3,), stride=(1,), padding=(1,))
    (bn1): BatchNorm1d(2, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  )
  (fc1): Linear(in_features=788480, out_features=256, bias=True)
  (fc2): Linear(in_features=256, out_features=256, bias=True)
)
</code></pre>
<p>I am using the following weight initialization:</p>
<pre><code class="lang-auto">def weights_init(m):
    if isinstance(m, nn.Conv3d or nn.Conv1d): #nn.Conv3d
        torch.nn.init.xavier_uniform_(m.weight.data, init.calculate_gain('relu'))
        m.bias.data.fill_(0)
        # torch.nn.init.xavier_uniform_(m.bias.data)
    elif isinstance(m, nn.BatchNorm3d):
        m.weight.data.normal_(mean=1.0, std=0.02)
        m.bias.data.fill_(0)
    elif isinstance(m, nn.Linear):
        m.weight.data.normal_(0.0, 0.02)
        m.bias.data.fill_(0)
</code></pre>
<p>Just to clarify how do I call eval on only batchnorm layers?</p>
<p><code>Net.conv_block32_64.bn1.eval()</code></p>
<p>Something like above?</p>

username: ptrblck
cooked: <p>Yes, that should work.<br>
A more convenient method would be:</p>
<pre><code class="lang-python">def set_bn_eval(m):
    if isinstance(m, nn.modules.batchnorm._BatchNorm):
        m.eval()
        
model.apply(set_bn_eval)
</code></pre>
<p>EDIT: Note that in your weight init code you are not initializing <code>nn.BatchNorm1d</code> using the specified method, but only <code>nn.BatchNorm3d</code>. My condition should work on both layer types, in case you need it.</p>

username: banikr
cooked: <p>I ran the codes as below making sure that the batchnorm layers in eval mode(as the code you shared)</p>
<pre><code class="lang-auto">    train_dataset = Subset(dataset, train_idx)
    valid_dataset = Subset(dataset, valid_idx)
    train_loader = data.DataLoader(train_dataset, batch_size=batchsize, shuffle=True)
    valid_loader = data.DataLoader(valid_dataset, batch_size=batchsize, shuffle=True)
    for epoch in range(max_epoch):
        print("training...")
        epoch_wise_loss = 0
        epoch_accuracy = 0
        running_time_batch = 0
        time_batch_start = time.time()
        Net.train()
        Net.apply(set_bn_eval)
</code></pre>
<p>But I think the train loss increased and validation loss did not change or even reduced.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/9/9/9932df8867000ecca73ad228c4b1f175088a5ab4.png" alt="image" data-base62-sha1="lRg2Zuc2EpfcMVGOpQp3UKRb0Jm" width="320" height="240"></p>
<p>Then I changed the <code>valid_loader</code>  also with <code>train_dataset</code> to keep the train and validation is the same data. And removed the <code>Optimizer.zero_grad()</code> and <code>Optimizer.step()</code> from the train section. This time the train and validation losses are almost the same which kind of makes sense.</p>
<p>Why is validation loss decreasing when I use the Optimizer functions?</p>
<pre><code class="lang-auto">    train_loader = data.DataLoader(train_dataset, batch_size=batchsize, shuffle=True)
    valid_loader = data.DataLoader(train_dataset, batch_size=batchsize, shuffle=True)
    for epoch in range(max_epoch):
        print("training...")
        epoch_wise_loss = 0
        epoch_accuracy = 0
        running_time_batch = 0
        time_batch_start = time.time()
        Net.train()
        Net.apply(set_bn_eval)
        for tBatch_idx, sample in enumerate(train_loader):
            time_batch_load = time.time() - time_batch_start
            time_compute_start = time.time()
            mr = sample[0].float().to(device)  #torch.Size([2, 1, 86, 110, 78])

            shimVal = sample[1].float().to(device)  #torch.Size([2, 512])
            out = Net(mr)  #torch.Size([2, 512])
            # Optimizer.zero_grad()
            Loss = Criterion(shimVal, out)
            print("Batch train loss:", Loss.item())
            batch_accuracy = accuracy(shimVal, out)
            Loss.backward()
            # Optimizer.step()
            epoch_wise_loss += Loss.item()
            mean_loss = epoch_wise_loss/(tBatch_idx+1)  #avg loss till current batch
</code></pre>

username: ptrblck
cooked: <p>Did you randomly split the data import the training and validation sets or are you using data e.g. from other data domains?</p>
<aside class="quote no-group" data-username="banikr" data-post="5" data-topic="60487">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/banikr/48/54353_2.png" class="avatar"> banikr:</div>
<blockquote>
<p>Why is validation loss decreasing when I use the Optimizer functions?</p>
</blockquote>
</aside>
<p>If your validation loss decreasing even though you don’t call <code>optimizer.step()</code>?</p>

username: banikr
cooked: <p>I am experimenting with cross-validation.<br>
For splitting the train and validation set I am using the <code>Subset</code> class of PyTorch.</p>
<pre><code class="lang-auto">class Subset(Dataset):
    """
    Subset of a dataset at specified indices.
    Arguments:
        dataset (Dataset): The whole Dataset
        indices (sequence): Indices in the whole set selected for subset
    """
    def __init__(self, dataset, indices):
        self.dataset = dataset
        self.indices = indices

    def __len__(self):
        if self.indices.shape == ():
            print('this happens: Subset')
            return 1
        else:
            return len(self.indices)

    def __getitem__(self, idx):
        return self.dataset[self.indices[idx]]
</code></pre>
<p><code>num_val_samples = len(dataset)//k</code></p>
<pre><code class="lang-auto">for i in range(k):
    print('Processing fold: ', i + 1)
    """%%%% Initiate new model %%%%""" #in every fold
    # torch.cuda.manual_seed(1)
    # model = ResNet3D().to(device)
    # model.apply(weights_init)
    # Optimizer = optim.Adam(model.parameters(), lr=lr)
    # Criterion = nn.MSELoss()
    # Criterion = Criterion.cuda()
    Net, Optimizer, Criterion = build_model()
    """%%%% Split train and validation sets %%%%"""
    valid_idx = np.arange(len(dataset))[i * num_val_samples:(i + 1) * num_val_samples]
    print('Valid fold indices:', valid_idx + 1)
    train_idx = np.concatenate([np.arange(len(dataset))[:i * num_val_samples], np.arange(len(dataset))[(i + 1) * num_val_samples:]], axis=0)
    print('Train fold indices:', train_idx+1)
    train_dataset = Subset(dataset, train_idx)
    valid_dataset = Subset(dataset, valid_idx)
    train_loader = data.DataLoader(train_dataset, batch_size=batchsize, shuffle=True)
    valid_loader = data.DataLoader(train_dataset, batch_size=batchsize, shuffle=True) #same train_set to check
</code></pre>
<p>Without using the <code>Optimizer.step()</code> I am getting the following loss in 1 fold cv:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/b/8/b82ec850d7ebbc39d2103e9a08112965f3f6beb6.png" alt="image" data-base62-sha1="qhm5iW43KJvQOLYbdPmvhHdHwXk" width="320" height="240"></p>
<p>I did not do anything to the validation section. So it was as before:</p>
<pre><code class="lang-auto">        print('validating...')
        Net.eval()
        epoch_wise_loss = 0
        # for vBatch_idx, sample in enumerate(valid_loader):
        with torch.no_grad():
            for vBatch_idx, sample in enumerate(valid_loader):
            # Net.eval()
            # with torch.no_grad():
                mr = sample[0].float().to(device)
                # print("Valid input mean:", mr.mean())
                shimVal = sample[1].float().to(device)
                # print("Valid target mean:", shimVal.mean())
                out = Net(mr)
                Loss = Criterion(shimVal, out)
                print("Batch val loss:", Loss.item())
                batch_accuracy = accuracy(shimVal, out)
                epoch_wise_loss += Loss.item()
                mean_loss = epoch_wise_loss / (vBatch_idx + 1)
</code></pre>
<p>These are all without setting batchnorm to eval mode. The next is setting batchnorm to eval mode as following:</p>
<pre><code class="lang-auto">Net.train()
Net.apply(set_bn_eval)
</code></pre>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/c/7cb930632922cf08b5bae8bf89c0b574f9b8222f.png" alt="image" data-base62-sha1="hNlYgeN0Kg3wMzmxT7CbNMZpqB9" width="320" height="240"></p>
<p>Train and validation loss are exactly the same and increase as you can see from the graph above.</p>

username: Sia_Rezaei
cooked: <p>Hi! Did you figure out what was wrong? I’m having the same issue and it is driving me crazy.</p>