1) Confusion about training loss, 2) why validation loss is so high?

username: swap
cooked: <p>I am constructing a DNN model using pytorch. I came across two ‘loss’ terms for training (see code below). My questions:</p>
<ol>
<li>Which training loss should I compare against validation loss? Shall I use loss or running loss?</li>
<li>In both cases, training loss differs by several folds as compare to validation loss. What am I doing wrong?</li>
</ol>
<pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.fc1 = nn.Linear(1105, 1024)
        self.fc2 = nn.Linear(1024, 256)
        self.fc3 = nn.Linear(256, 64)
        self.fc4 = nn.Linear(64, 16)
        self.fc5 = nn.Linear(16, 2)
        self.bn_1 = nn.BatchNorm1d(1024, momentum=0.5)
        self.bn_2 = nn.BatchNorm1d(256, momentum=0.5)
        self.bn_3 = nn.BatchNorm1d(64, momentum=0.5)
        self.bn_4 = nn.BatchNorm1d(16, momentum=0.5)
        self.dropout = nn.Dropout(p=0.2)
    
    def forward(self, inputs):
        x = inputs
        x = self.dropout(self.bn_1(F.relu(self.fc1(x))))
        x = self.dropout(self.bn_2(F.relu(self.fc2(x))))
        x = self.dropout(self.bn_3(F.relu(self.fc3(x))))
        x = self.dropout(self.bn_4(F.relu(self.fc4(x))))
        x = self.fc5(x)
        return F.log_softmax(x, dim=1)

my_model = Net().to(device)

# data shape:: Train: (163082, 1105) Validation: (40771, 1105)
BATCH_SIZE = 32
nSamples = [157544, 5538] #composition of classes in my dataset for class 0 and 1, resp
normedWeights = [1 - (x / sum(nSamples)) for x in nSamples]
normedWeights = torch.FloatTensor(normedWeights).to(device, dtype=torch.float64)
loss_function = nn.NLLLoss(weight=normedWeights)
optimizer = optim.Adam(my_model.parameters(), lr=0.0001, eps=0.0001, amsgrad=True, weight_decay=1e-5)
scheduler = optim.lr_scheduler.ReduceLROnPlateau(optimizer, patience=10, verbose=True)

def train(my_model):
    EPOCHS = 100
    for epoch in range(EPOCHS):
        train_batch_loss = 0.0
        val_loss = 0.0
        correct = 0
        total = 0
        val_correct = 0
        val_total = 0
        for i, (x, y) in enumerate(train_loader):
            x, y = x.to(device, dtype=torch.float32), y.to(device, dtype=torch.long)
            optimizer.zero_grad()  
            output = my_model(x)
            loss = loss_function(output.double(), y)
            train_batch_loss += F.nll_loss(output.double(), y, reduction='sum')
            loss.backward()
            #torch.nn.utils.clip_grad_norm_(my_model.parameters(), 0.5)
            optimizer.step()
        _, predicted = torch.max(output.data, 1)
        correct += predicted.eq(y).sum().item()
        total += len(y)
        scheduler.step(i)
        running_loss = train_batch_loss/len(train_loader)
            
        for j, (x_val, y_val) in enumerate(val_loader):
            x_val, y_val = x_val.to(device, dtype=torch.float32), y_val.to(device, dtype=torch.long)
            with torch.no_grad():
                my_model.eval()
                val_outputs = my_model(x_val)
                val_loss1 = loss_function(val_outputs.double(), y_val)
                val_loss += val_loss1.item()
        _, val_predicted = torch.max(val_outputs.data, 1)
        val_correct += val_predicted.eq(y_val).sum().item()
        val_total += len(y_val)
        print(f"Epoch {epoch}. LOSS:: Train_loss {loss:.4f}  Train_running_loss {running_loss:.4f}   Val_loss {val_loss:.4f}")
        print(f"Accuracy:: Train {correct/total:.4f}  Validation {val_correct/val_total:.4f}")
</code></pre>
<p>output</p>
<pre><code class="lang-auto">Epoch 0. LOSS:: Train_loss 0.1771  Train_running_loss 6.0901   Val_loss 385.8817
         Accuracy:: Train 0.9000  Validation 1.0000
Epoch 1. LOSS:: Train_loss 0.0815  Train_running_loss 5.8791   Val_loss 450.4003
         Accuracy:: Train 1.0000  Validation 1.0000
Epoch 2. LOSS:: Train_loss 0.1021  Train_running_loss 5.5039   Val_loss 442.1833
         Accuracy:: Train 0.9000  Validation 1.0000
Epoch 3. LOSS:: Train_loss 0.2438  Train_running_loss 5.4403   Val_loss 415.6329
         Accuracy:: Train 0.9000  Validation 1.0000
Epoch 4. LOSS:: Train_loss 0.0914  Train_running_loss 5.1918   Val_loss 380.8049
         Accuracy:: Train 1.0000  Validation 1.0000
Epoch 5. LOSS:: Train_loss 0.0973  Train_running_loss 5.0323   Val_loss 401.0601
         Accuracy:: Train 0.9000  Validation 1.0000
Epoch 6. LOSS:: Train_loss 0.1308  Train_running_loss 4.8982   Val_loss 409.4360
         Accuracy:: Train 0.9000  Validation 1.0000
Epoch 7. LOSS:: Train_loss 0.0257  Train_running_loss 4.7714   Val_loss 456.5145
         Accuracy:: Train 1.0000  Validation 1.0000
Epoch 8. LOSS:: Train_loss 0.0241  Train_running_loss 4.9047   Val_loss 533.7724
         Accuracy:: Train 1.0000  Validation 1.0000
Epoch 9. LOSS:: Train_loss 0.0700  Train_running_loss 4.7236   Val_loss 423.4640
         Accuracy:: Train 1.0000  Validation 1.0000
Epoch 10. LOSS:: Train_loss 0.0446  Train_running_loss 4.4727   Val_loss 410.2520
         Accuracy:: Train 1.0000  Validation 1.0000
Epoch    12: reducing learning rate of group 0 to 1.0000e-04.
Epoch 11. LOSS:: Train_loss 0.0138  Train_running_loss 4.2515   Val_loss 492.4581
         Accuracy:: Train 1.0000  Validation 1.0000
Epoch 12. LOSS:: Train_loss 0.0056  Train_running_loss 3.7988   Val_loss 496.0938
         Accuracy:: Train 1.0000  Validation 1.0000
Epoch 13. LOSS:: Train_loss 0.0163  Train_running_loss 3.5083   Val_loss 493.1467
         Accuracy:: Train 1.0000  Validation 1.0000
Epoch 14. LOSS:: Train_loss 0.0043  Train_running_loss 3.4447   Val_loss 522.9645
         Accuracy:: Train 1.0000  Validation 1.0000
Epoch 15. LOSS:: Train_loss 0.0061  Train_running_loss 3.3147   Val_loss 533.8937
         Accuracy:: Train 1.0000  Validation 1.0000
Epoch 16. LOSS:: Train_loss 0.0708  Train_running_loss 3.3023   Val_loss 505.2642
         Accuracy:: Train 1.0000  Validation 1.0000
Epoch 17. LOSS:: Train_loss 0.0039  Train_running_loss 3.1817   Val_loss 529.2422
         Accuracy:: Train 1.0000  Validation 1.0000
Epoch 18. LOSS:: Train_loss 0.0055  Train_running_loss 3.1025   Val_loss 548.8464
         Accuracy:: Train 1.0000  Validation 1.0000
Epoch 19. LOSS:: Train_loss 0.0276  Train_running_loss 3.0549   Val_loss 560.4675
         Accuracy:: Train 1.0000  Validation 1.0000
Epoch 20. LOSS:: Train_loss 0.0037  Train_running_loss 3.0281   Val_loss 586.8056
         Accuracy:: Train 1.0000  Validation 1.0000
Epoch 21. LOSS:: Train_loss 0.0060  Train_running_loss 2.9651   Val_loss 603.7631
         Accuracy:: Train 1.0000  Validation 1.0000
Epoch    23: reducing learning rate of group 0 to 1.0000e-05.
Epoch 22. LOSS:: Train_loss 0.0027  Train_running_loss 2.9114   Val_loss 487.0047
         Accuracy:: Train 1.0000  Validation 1.0000
Epoch 23. LOSS:: Train_loss 0.0024  Train_running_loss 2.8859   Val_loss 565.6329
         Accuracy:: Train 1.0000  Validation 1.0000
Epoch 24. LOSS:: Train_loss 0.0097  Train_running_loss 2.8546   Val_loss 591.2556
         Accuracy:: Train 1.0000  Validation 1.0000
Epoch 25. LOSS:: Train_loss 0.0021  Train_running_loss 2.8961   Val_loss 588.1403
         Accuracy:: Train 1.0000  Validation 1.0000
Epoch 26. LOSS:: Train_loss 0.0045  Train_running_loss 2.8124   Val_loss 506.0154
         Accuracy:: Train 1.0000  Validation 1.0000
Epoch 27. LOSS:: Train_loss 0.0070  Train_running_loss 2.7557   Val_loss 505.5475
         Accuracy:: Train 1.0000  Validation 1.0000
Epoch 28. LOSS:: Train_loss 0.0048  Train_running_loss 2.8138   Val_loss 626.8176
         Accuracy:: Train 1.0000  Validation 1.0000
Epoch 29. LOSS:: Train_loss 0.0009  Train_running_loss 2.7847   Val_loss 540.7521
         Accuracy:: Train 1.0000  Validation 1.0000
Epoch 30. LOSS:: Train_loss 0.0098  Train_running_loss 2.7887   Val_loss 539.4330
         Accuracy:: Train 1.0000  Validation 1.0000
Epoch 31. LOSS:: Train_loss 0.0185  Train_running_loss 2.7797   Val_loss 574.4599
         Accuracy:: Train 1.0000  Validation 1.0000
Epoch 32. LOSS:: Train_loss 0.0026  Train_running_loss 2.7819   Val_loss 527.2850
         Accuracy:: Train 1.0000  Validation 1.0000
Epoch    34: reducing learning rate of group 0 to 1.0000e-06.
Epoch 33. LOSS:: Train_loss 0.0035  Train_running_loss 2.7485   Val_loss 533.4987
         Accuracy:: Train 1.0000  Validation 1.0000
Epoch 34. LOSS:: Train_loss 0.0033  Train_running_loss 2.7217   Val_loss 548.1089
         Accuracy:: Train 1.0000  Validation 1.0000
Epoch 35. LOSS:: Train_loss 0.0086  Train_running_loss 2.7513   Val_loss 528.2917
         Accuracy:: Train 1.0000  Validation 1.0000
Epoch 36. LOSS:: Train_loss 0.0017  Train_running_loss 2.7649   Val_loss 544.4396
         Accuracy:: Train 1.0000  Validation 1.0000
Epoch 37. LOSS:: Train_loss 0.0131  Train_running_loss 2.7475   Val_loss 570.9015
         Accuracy:: Train 1.0000  Validation 1.0000
Epoch 38. LOSS:: Train_loss 0.0010  Train_running_loss 2.7525   Val_loss 586.9201
         Accuracy:: Train 1.0000  Validation 1.0000
Epoch 39. LOSS:: Train_loss 0.0047  Train_running_loss 2.7542   Val_loss 568.6170
         Accuracy:: Train 1.0000  Validation 1.0000
Epoch 40. LOSS:: Train_loss 0.0269  Train_running_loss 2.7772   Val_loss 551.5580
         Accuracy:: Train 1.0000  Validation 1.0000
Epoch 41. LOSS:: Train_loss 0.0059  Train_running_loss 2.7409   Val_loss 618.6259
         Accuracy:: Train 1.0000  Validation 1.0000
Epoch 42. LOSS:: Train_loss 0.0051  Train_running_loss 2.7635   Val_loss 597.0938
         Accuracy:: Train 1.0000  Validation 1.0000
Epoch 43. LOSS:: Train_loss 0.0018  Train_running_loss 2.7605   Val_loss 528.7895
         Accuracy:: Train 1.0000  Validation 1.0000
Epoch    45: reducing learning rate of group 0 to 1.0000e-07.
Epoch 44. LOSS:: Train_loss 0.0014  Train_running_loss 2.7500   Val_loss 565.9728
         Accuracy:: Train 1.0000  Validation 1.0000
Epoch 45. LOSS:: Train_loss 0.0008  Train_running_loss 2.7274   Val_loss 650.7986
         Accuracy:: Train 1.0000  Validation 1.0000
Epoch 46. LOSS:: Train_loss 0.0032  Train_running_loss 2.7719   Val_loss 640.0589
         Accuracy:: Train 1.0000  Validation 1.0000
Epoch 47. LOSS:: Train_loss 0.0015  Train_running_loss 2.7303   Val_loss 514.7046
         Accuracy:: Train 1.0000  Validation 1.0000
Epoch 48. LOSS:: Train_loss 0.0015  Train_running_loss 2.7317   Val_loss 545.7140
         Accuracy:: Train 1.0000  Validation 1.0000
Epoch 49. LOSS:: Train_loss 0.0058  Train_running_loss 2.7294   Val_loss 536.0183
         Accuracy:: Train 1.0000  Validation 1.0000
Epoch 50. LOSS:: Train_loss 0.0033  Train_running_loss 2.7457   Val_loss 568.7015
         Accuracy:: Train 1.0000  Validation 1.0000
</code></pre>
<p>NOTE: several measures are taken so far to reduce loss</p>
<ol>
<li>Data is standardized</li>
<li>Batch normalization</li>
<li>simplest architecture (single layer, 4 neurons) up to complex architecture (as shown above)</li>
<li>drop out layer</li>
<li>batchnorm momentum increased to 0.5</li>
<li>gradient clipping (clip 0.5)<br>
My data consist of numerical values so no data augmentation can be done!</li>
</ol>

username: KFrank
cooked: <p>Hello Swapnil!</p>
<aside class="quote no-group quote-modified" data-username="swap" data-post="1" data-topic="87833" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/a698b9/48.png" class="avatar"> swap:</div>
<blockquote>
<ol start="2">
<li>In both cases, training loss differs by several folds as compare to validation loss. What am I doing wrong?</li>
</ol>
<pre><code class="lang-auto">loss_function = nn.NLLLoss(weight=normedWeights)
...
            loss = loss_function(output.double(), y)
            train_batch_loss += F.nll_loss(output.double(), y, reduction='sum')
...
        running_loss = train_batch_loss/len(train_loader)
...
                val_loss1 = loss_function(val_outputs.double(), y_val)
                val_loss += val_loss1.item()
...
        val_total += len(y_val)
        print(f"Epoch {epoch}. LOSS:: Train_loss {loss:.4f}  Train_running_loss {running_loss:.4f}   Val_loss {val_loss:.4f}")
        print(f"Accuracy:: Train {correct/total:.4f}  Validation {val_correct/val_total:.4f}")
</code></pre>
</blockquote>
</aside>
<p>I don’t really understand what you are trying to do nor what your issue is.</p>
<p>But I have a couple of comments:</p>
<p>When you calculate <code>loss</code> and <code>val_loss</code> you use <code>normedWeights</code> in<br>
<code>loss_function</code>.  But when you calculate <code>running_loss</code> you do not<br>
use <code>normedWeights</code>.  Also, <code>running_loss</code> is an average – you<br>
normalize it by dividing by <code>len(train_loader)</code>, but <code>val_loss</code> is just<br>
a sum over losses, and you don’t average it over the length of your<br>
<code>enumerate(val_loader)</code> loop.</p>
<p>So your three loss values are normalized and/or class-weighted<br>
differently, and are not directly comparable.</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">        self.fc5 = nn.Linear(16, 2)
...
        x = self.fc5(x)
        return F.log_softmax(x, dim=1)
</code></pre>
<p>…<br>
nSamples = [157544, 5538] <span class="hashtag-raw">#composition</span> of classes in my dataset for class 0 and 1, resp</p>
</blockquote>
</aside>
<p>As an aside, you are performing a <em>binary</em> (two-class) classification.</p>
<p>You can treat this as a multi-class classification (as you are doing), but<br>
you will be a little bit better off if you treat it as a binary classification.</p>
<p>So I would recommend that you have a single output from your final<br>
layer, get rid of the <code>log_softmax()</code>, and use <code>BCEWithLogitsLoss</code> as<br>
your loss function.</p>
<p>Something like this:</p>
<pre data-code-wrap="python"><code class="lang-python">       self.fc5 = nn.Linear(16, 1)
...
       x = self.fc5(x)
       return x
...
posWeight = torch.FloatTensor ([nSamples[1] / (nSamples[0] + nSamples[1])]).to(device, dtype=torch.float64)
loss_function = nn.BCEWithLogitsLoss (pos_weight = posWeight)
</code></pre>
<p>Good luck.</p>
<p>K. Frank</p>