Validation loss lesser than training loss

TheOraware: <p>Hi ,</p>
<p><code>train_df.shape</code><br>
(7111, 8)</p>
<p>I have following predictor variable</p>
<pre><code class="lang-auto">     deg_C	relative_humidity	absolute_humidity	sensor_1	sensor_2	sensor_3	sensor_4	sensor_5
0	  13.1	             46.0	           0.7578	  1387.2	  1087.8  	  1056.0	  1742.8	  1293.4
1	  13.2	             45.3	           0.7255	  1279.1	   888.2	  1197.5	  1449.9	  1010.9
</code></pre>
<p>and here target variables</p>
<pre><code class="lang-auto">  target_carbon_monoxide	target_benzene	target_nitrogen_oxides
0	                 2.5	          12.0	                 167.7
1	                 2.1	           9.9	                  98.9
</code></pre>
<p>My workaround is</p>
<p><strong>Scaling Both Predictor and targets</strong></p>
<pre><code class="lang-auto">X_train, X_val, y_train, y_val = train_test_split(train_df, train_y, test_size=0.2, random_state=0)

scalerX = StandardScaler().fit(X_train)
scalery = StandardScaler().fit(y_train)

X_train = scalerX.transform(X_train)
y_train = scalery.transform(y_train)

X_val = scalerX.transform(X_val)
y_val = scalery.transform(y_val)
</code></pre>
<pre><code class="lang-auto">class ClassifierDataset(Dataset):
    
    def __init__(self, X_data, y_data):
        self.X_data = X_data
        self.y_data = y_data
        
    def __getitem__(self, index):
        return self.X_data[index], self.y_data[index]
        
    def __len__ (self):
        return len(self.X_data)
</code></pre>
<pre><code class="lang-auto">NUM_FEATURES     = 8
BATCHSIZE        = 500


train_dataset = ClassifierDataset(torch.from_numpy(X_train), torch.from_numpy(y_train))
train_loader  = DataLoader(dataset=train_dataset,batch_size=BATCHSIZE)

valid_dataset = ClassifierDataset(torch.from_numpy(X_val), torch.from_numpy(y_val))
valid_loader  = DataLoader(dataset=valid_dataset,batch_size=BATCHSIZE)
</code></pre>
<p><strong>Model</strong></p>
<pre><code class="lang-auto">class multiNet(nn.Module):
    def __init__(self, num_feature):
        super().__init__()

        self.lin0 = nn.Linear(num_feature,5)
        self.lin1 = nn.Linear(5, 4)
        self.lin2 = nn.Linear(4, 3)
        
        self.bn0 = nn.BatchNorm1d(num_feature)
        self.bn1 = nn.BatchNorm1d(5)
        self.bn2 = nn.BatchNorm1d(4)

    def forward(self, x):
        
        x = self.bn0(x)
        x = self.lin0(x) 
        x = F.relu(x)
        x = self.bn1(x)
        
        x = self.lin1(x) 
        x = F.relu(x)

        x = self.lin2(x) # output layer

        return x
</code></pre>
<p><strong>Optimizer</strong></p>
<pre><code class="lang-auto">def get_optimizer(model, lr):
    optimizer = torch.optim.Adam(model.parameters(),lr=lr,weight_decay=0.01) 
    scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(optimizer,patience=10,factor=0.1,threshold=1e1) 
  
    return optimizer,scheduler
</code></pre>
<p><strong>Weights</strong></p>
<pre><code class="lang-auto">def init_weights(m):
    if type(m) == nn.Linear:
        '''
        This is an example i created as instructed by Andrew Ng course
        torch.ones([10, 6])*torch.sqrt(torch.Tensor([2.])/torch.Tensor([1.4142]))
        '''
        m.weight.data = torch.randn(m.weight.size())*torch.sqrt(torch.Tensor([2.])/m.weight.size()[1])
</code></pre>
<p><strong>Model Training</strong></p>
<pre><code class="lang-auto">seed = 4
torch.manual_seed(seed)

def train_loop(model, epochs, lr):
    total    = 0
    sum_loss = 0
    output   = 0
    
    criterion   = nn.L1Loss()
    optim,scheduler = get_optimizer(model, lr = lr)
    
    train_loss_values  = []
    val_loss_values    = []
    
    for epoch in range(epochs):
        train_running_loss = 0.0
        val_running_loss   = 0.0
        
        for data, target in train_loader:
            model.train()
            output = model(data)
            loss = torch.sqrt(criterion(output, target)) # as i need RMSLE
            optim.zero_grad()
            loss.backward()
            optim.step()
            train_running_loss =+ loss.item()
            
            if not  scheduler.__class__ ==  torch.optim.lr_scheduler.ReduceLROnPlateau:
                scheduler.step()
        
        train_loss_values.append(train_running_loss)
        
        for valid_data, valid_target in valid_loader:
            model.eval()
            valid_output = model(valid_data)
            valid_loss = criterion(valid_output,valid_target)
            val_running_loss =+ valid_loss.item()
        val_loss_values.append(val_running_loss)
        
        if epoch % 100 ==0:
            print(f'epoch : {epoch+1},training loss : {loss} , validation loss : {valid_loss}')            
    plt.plot(train_loss_values, 'r')
    plt.plot(val_loss_values, 'g')      

NUM_FEATURES = X_train.shape[1]
multiNetModel = multiNet(NUM_FEATURES)
multiNetModel.apply(init_weights) 
train_loop(multiNetModel, epochs=500, lr=0.001)
</code></pre>
<p>Is my above baseline approach is correct ? If it is then why my validation loss is way lesser than training loss</p>
<p><code>epoch : 1    ,training loss : 1.109604 , validation loss : 0.995102</code><br>
<code>epoch : 101,training loss : 0.608683 , validation loss : 0.294198</code><br>
<code> epoch : 201,training loss : 0.518566 , validation loss : 0.225262</code><br>
<code> epoch : 301,training loss : 0.514474 , validation loss : 0.221938</code><br>
<code> epoch : 401,training loss : 0.512781 , validation loss : 0.220481</code></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/f/cfe6e4b640da6f732544f3071ca472aefb921222.png" data-download-href="https://discuss.pytorch.org/uploads/default/cfe6e4b640da6f732544f3071ca472aefb921222" title="loss"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/f/cfe6e4b640da6f732544f3071ca472aefb921222_2_690x446.png" alt="loss" data-base62-sha1="tFbwrgS7QeYyPmXVsdXQexCE7TQ" width="690" height="446" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/f/cfe6e4b640da6f732544f3071ca472aefb921222_2_690x446.png, https://discuss.pytorch.org/uploads/default/optimized/3X/c/f/cfe6e4b640da6f732544f3071ca472aefb921222_2_1035x669.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/c/f/cfe6e4b640da6f732544f3071ca472aefb921222.png 2x" data-dominant-color="FAF9F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">loss</span><span class="informations">1130×732 61.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

KFrank: <p>Hi Mr. Oraware!</p>
<aside class="quote no-group quote-modified" data-username="TheOraware" data-post="1" data-topic="125628" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/theoraware/48/30315_2.png" class="avatar"> TheOraware:</div>
<blockquote>
<pre><code class="lang-auto">                for data, target in train_loader:
...
                    loss = torch.sqrt(criterion(output, target)) # as i need RMSLE
...
        for valid_data, valid_target in valid_loader:
...
            valid_loss = criterion(valid_output,valid_target)
...
</code></pre>
<p>If it is then why my validation loss is way lesser than training loss</p>
</blockquote>
</aside>
<p>The bulk of the discrepancy is because you have <code>sqrt()</code> in your<br>
training loss and not in your validation loss.  (Even accounting for<br>
this, your training loss is still perhaps about 20% greater than you<br>
validation loss, but this is easier to hand-wave away with more<br>
“hard” samples in your training set or normalization vagaries, etc.)</p>
<p>Best.</p>
<p>K. Frank</p>

TheOraware: <p><a class="mention" href="/u/kfrank">@KFrank</a> thanks for correction , yes i fixed it and now gap between training and validation has been closed , but still validation loss is lesser than training , could you please elaborate more what do you mean by</p>
<p>" but this is easier to hand-wave away with more<br>
“hard” samples in your training set or normalization vagaries, etc."</p>
<pre><code class="lang-auto">epoch : 1,training loss : 1.1062123730735536 , validation loss : 0.9584911298922756
epoch : 101,training loss : 0.5904753306208222 , validation loss : 0.5017905667378958
epoch : 201,training loss : 0.4886256533068133 , validation loss : 0.43560073343017525
epoch : 301,training loss : 0.4826799846960937 , validation loss : 0.43197416186192145
epoch : 401,training loss : 0.47715110642727154 , validation loss : 0.4264545226974467
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/0/b09eb7a01a77e0224244febbcb41c5a891b6b61a.png" data-download-href="https://discuss.pytorch.org/uploads/default/b09eb7a01a77e0224244febbcb41c5a891b6b61a" title="loss"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/0/b09eb7a01a77e0224244febbcb41c5a891b6b61a_2_690x459.png" alt="loss" data-base62-sha1="pcs5gO47o1SlzcTusqz5c15sW1A" width="690" height="459" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/0/b09eb7a01a77e0224244febbcb41c5a891b6b61a_2_690x459.png, https://discuss.pytorch.org/uploads/default/optimized/3X/b/0/b09eb7a01a77e0224244febbcb41c5a891b6b61a_2_1035x688.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/b/0/b09eb7a01a77e0224244febbcb41c5a891b6b61a.png 2x" data-dominant-color="FAF9F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">loss</span><span class="informations">1130×752 67.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

KFrank: <p>Hi Mr. Oraware!</p>
<aside class="quote no-group" data-username="TheOraware" data-post="3" data-topic="125628" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/theoraware/48/30315_2.png" class="avatar"> TheOraware:</div>
<blockquote>
<p>" but this is easier to hand-wave away with more<br>
“hard” samples in your training set or normalization vagaries, etc."</p>
</blockquote>
</aside>
<p>Some context:</p>
<p>As a general rule, you would expect (after some training) your training<br>
loss to be smaller than your validation loss.  This is because, when<br>
training on your training data, your model “learns” not only the general<br>
features of the problem you are trying to solve, but also some specific<br>
features of your training set.  That is, it “learns” (or perhaps “memorizes”)<br>
that it should make certain specific predictions for certain specific<br>
samples in your training set.  So your model performs better than it<br>
“should” on these training samples it has “memorized” and therefore<br>
produces a smaller training loss.  But, as you note, you are seeing<br>
the opposite.</p>
<p>“Hard” samples:  Consider digit classification.  If sloppy enough a “1” and<br>
a “7” can be hard to tell apart, as can be a “4” and a “9.”  Suppose that,<br>
by happenstance, your training set has a lot of sloppy "1"s and "4"s, but<br>
your evaluation set doesn’t.  When you make predictions on your training<br>
set, you’re asking it to something harder – correctly recognize those<br>
sloppy "1"s – than you’re asking it to do with your validation set.  So you<br>
get a larger (worse) loss on your training set.</p>
<p>Normalization:  You normalize both your training set and validation set<br>
using statistics computed from you training set.  (I think this is the right<br>
thing to do.)  But your loss criterion – <code>L1Loss</code> scales with the size of<br>
<code>target</code> (and predictions).  Let’s say that your training set has a “mean”<br>
of 1.0 so that (in simplified terms) your normalization doesn’t actually<br>
change anything, but, by happenstance, your validation set has a “mean”<br>
of 0.9 – and this is not changed by normalizing it with the training-set<br>
statistics.  Now, all else being equal, your validation-set loss will be<br>
smaller by a factor of 0.9 than your training-set loss, merely because<br>
the values given to your <code>L1Loss</code> criterion are, themselves, smaller by<br>
a factor of 0.9.</p>
<p>One comment here:  From your graph, it appears that your training<br>
loss starts out higher than your validation loss, even on the first batch<br>
of each, which, if I understand your code, is <em>before</em> any optimizer steps<br>
for your first training batch, but <em>after</em> one full epoch of training for your<br>
first validation batch.  So this is not an apples-to-apples comparison.</p>
<p>In order for your training and validation losses to be comparable,<br>
you want your training and validation set to have the same overall<br>
characteristics.  Do you have a single larger dataset that you randomly<br>
split into a training and validation set?  Or do your training and validation<br>
data come from two separate sources?</p>
<p>Try taking a single dataset (for example, combine together your current<br>
training and validation sets if they come from different sources) and<br>
randomly split that single dataset into training and validation a handful<br>
of times and run your training on those multiple random choices for<br>
your training and validation data.  See if any systematic discrepancy<br>
occurs across multiple choices of the dataset split.</p>
<p>Lastly, I don’t have good intuition about how the differing behavior of<br>
your <code>BatchNorm1d</code> layers in <code>train()</code> and e<code>val()</code> modes might affect<br>
your losses.</p>
<p>To explore this you could, after each epoch of training, put your model<br>
in <code>eval()</code> mode (and, of course, not call <code>optim.step()</code>) and calculate<br>
your training and validation loss (on batches of the same size).  Plot those<br>
as a function of <code>epoch</code> and see if the discrepancy persists.</p>
<p>Best.</p>
<p>K. Frank</p>

TheOraware: <p><a class="mention" href="/u/kfrank">@KFrank</a> i am working on your suggestion , i will get back to you please bear me</p>