Interpreting loss value

username: Diego
cooked: <p>Hello, I am using a resnet18 to do some Finetuning classification on a dataset of dogs breeds and I’m struggling to get good results. This is how the losses and accuracies look like.<br>
What does it mean that the validation loss is smaller all the time than the train loss?. Is 80% a reasonable accuracy for this kind of problem?.<br>
Any tips on how to approach this kind of classification would very much be appreciated!  Thanks in advance. <img src="https://discuss.pytorch.org/uploads/default/original/2X/4/4f7a18aa65577c90e73491c560ea561a9865fdac.png" alt="results" data-base62-sha1="bl5jf8nU6FSzwtmziJAejJ4yOeM" width="640" height="480"></p>

username: johnolafenwa
cooked: <p>Your results depends on many factors;  train size, test size,  train batch size, test batch size, learning rate + optimizer and how accurately you compute the metrics. Might you include your full code, that would make it easier to advice you.</p>

username: Diego
cooked: <p>Thanks for your reply here its the code:</p>
<p><strong>The model</strong></p>
<pre><code class="lang-auto">resnet_model = models.resnet18(pretrained=True)

for param in resnet_model.parameters():
    param.requires_grad = False


num_ftrs = resnet_model.fc.in_features
resnet_model.fc = nn.Linear(num_ftrs, len(classes))


resnet_model = resnet_model.to(device)
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(resnet_model.fc.parameters(), lr=1e-3, weight_decay=0)
</code></pre>
<p><strong>Data</strong></p>
<pre><code class="lang-auto">data_transforms = {
    'train': transforms.Compose([
        transforms.RandomResizedCrop(224),
        transforms.RandomHorizontalFlip(),
        transforms.ToTensor()
    ]),

    'val': transforms.Compose([
        transforms.Resize(256),
        transforms.CenterCrop(224),
        transforms.ToTensor()
    ])
}

images = {x: datasets.ImageFolder(os.path.join(data_dir, x),
                                  data_transforms[x])
          for x in ['train', 'val']}


dataloaders = {x: torch.utils.data.DataLoader(images[x], batch_size=128,
                                              shuffle=True, num_workers=4)
               for x in ['train', 'val']}

</code></pre>
<p><strong>Training</strong></p>
<pre><code class="lang-auto">def train():

    resnet_model.train(True)
    torch.set_grad_enabled(True)

    running_loss = 0.0
    running_corrects = 0

    for data in dataloaders['train']:

        inputs, labels = data
        inputs = inputs.to(device)
        labels = labels.to(device)

        optimizer.zero_grad()

        outputs = resnet_model(inputs)
        _, preds = torch.max(outputs, 1)
        loss = criterion(outputs, labels)

        loss.backward()
        optimizer.step()

        running_loss += loss.item()
        running_corrects += torch.sum(preds == labels).item()

    epoch_loss = running_loss / len(dataloaders['train'])
    epochs_acc = running_corrects / dataset_sizes['train']

    return epoch_loss, epochs_acc

</code></pre>
<p><strong>Validation</strong></p>
<pre><code class="lang-auto">def evaluate():

    resnet_model.train(False)

    running_loss = 0
    running_corrects = 0

    for data in dataloaders['val']:

        inputs, labels = data
        inputs = inputs.to(device)
        labels = labels.to(device)

        outputs = resnet_model(inputs)
        _, preds = torch.max(outputs, 1)
        loss = criterion(outputs, labels)

        running_loss += loss.item()
        running_corrects += torch.sum(preds == labels).item()

    epoch_loss = running_loss / len(dataloaders['val'])
    epochs_acc = running_corrects / dataset_sizes['val']

    return epoch_loss, epochs_acc
</code></pre>
<p><strong>Main Loop</strong></p>
<pre><code class="lang-auto">while True:
        epoch_start_time = time.time()

        train_loss, train_acc = train()
        train_losses.append(train_loss)
        train_accuracies.append(train_acc)

        print('-' * 73)
        print('| End of epoch: {:3d} | Time: {:6.2f}s | Train loss: {:.2f} | Train acc: {:.2f}|'
              .format(epoch, (time.time() - epoch_start_time), train_loss, train_acc))

        val_loss, val_acc = evaluate()
        val_losses.append(val_loss)
        val_accuracies.append(val_acc)

        print('-' * 73)
        print('| End of epoch: {:3d} | Time: {:6.2f}s | Valid loss: {:.2f} | Valid acc: {:.2f}|'
              .format(epoch, (time.time() - epoch_start_time), val_loss, val_acc))

        if val_loss &lt; best_val_loss:

            with open('resnet_model.pt', 'wb') as f:
                torch.save(resnet_model, f)

            best_val_loss = val_loss
            bad_epochs = 0

        else:
            bad_epochs += 1

        if bad_epochs == 10:
            break

        epoch += 1
</code></pre>

username: johnolafenwa
cooked: <p>I have taken a look at it, there are a few problems with it.  I suggest the following solutions<br>
First add</p>
<pre><code class="lang-auto">transforms.Normalize((0.5,0.5,0.5),(0.5,0.5,0.5))
</code></pre>
<p>just before ToTensor in both the train and test transforms.</p>
<p>Step2.<br>
You don’t need to shuffle the test set, I suggest you should just load the test and train sets without using the loops over array,  that is even cleaner.</p>
<p>Step3:<br>
Here is where the real problem is.<br>
The way you compute the average loss and accuracy.</p>
<p>len(data_loader) would return number of batches,  hence you are dividing the total metrics by the number of batches instead of number of examples.</p>
<p>Here is what you need to do,  define</p>
<pre><code class="lang-auto">num_images = 0
</code></pre>
<p>At each batch iteration do</p>
<pre><code class="lang-auto">num_images += inputs.size(0)
</code></pre>
<p>Note that inputs.size(0) would give the total images in each batch.<br>
Finally</p>
<pre><code class="lang-auto">epoch_loss = running_loss/num_images
epoch_acc = running_acc/num_images
</code></pre>
<p>That would give you accurate metrics<br>
Do same in the evaluate function</p>
<p>Also,  torch.save accepts filenames too, you really shouldn’t bother using open</p>

username: roaffix
cooked: <aside class="quote no-group" data-username="johnolafenwa" data-post="4" data-topic="17665">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/johnolafenwa/48/4641_2.png" class="avatar"> johnolafenwa:</div>
<blockquote>
<p>Note that inputs.size(0) would give the total images in each batch.<br>
Finally</p>
<pre><code class="lang-auto">epoch_loss = running_loss/num_images
epoch_acc = running_acc/num_images
</code></pre>
</blockquote>
</aside>
<p>what about <code>running_loss/len(dataloader['val'].dataset)</code> ? Its has the same effect, but more efficient in runtime.</p>

username: johnolafenwa
cooked: <p>That’s cool, cuts out the extra computation.<br>
However, the batch by batch implementation is useful if logging the metrics after N batches rather than at the end of the epoch. You can simply divide the running loss by the num_images at any number of iterations to get current average.<br>
However, if averaging is done only at the end of each epoch, your code is just okay.</p>

username: Diego
cooked: <p>I am confused about a couple things:</p>
<p>For each batch you get one loss value, so why should I average over all the samples instead of over the batches?<br>
Wouldn’t that give me a very small loss value? Since I am adding 64 loss values (64 batches) and dividing by all the images (8000 more or less)?</p>
<p>Also how can I decide which values to normalize with is 0.5 a good choice by default?</p>

username: johnolafenwa
cooked: <p>Although the loss is one value, it represents the sum of losses over all examples in the batch, hence the effective total loss is the sum of losses over all N images rather than sum over M batches. The normalization values is mean and std values, it differs for various datasets, however, 0.5 is generally a good default to use for any generic dataset.</p>

username: Diego
cooked: <p>Thanks a lot for your help. I was looking at the <a href="https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html" rel="nofollow noopener">Transfer Learning Tutorial</a> and I found the normalization it’s done with:</p>
<pre><code class="lang-auto">transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
</code></pre>
<p>It seems that resnet18 was trained using this normalization, so I am using that one and the results seem to improve somewhat.</p>
<p>When it comes to the loss part, he calculates it like this:</p>
<pre><code class="lang-auto"> # statistics
running_loss += loss.item() * inputs.size(0)
running_corrects += torch.sum(preds == labels.data)
epoch_loss = running_loss / dataset_sizes[phase]
epoch_acc = running_corrects.double() / dataset_sizes[phase]
</code></pre>
<p>With phase being either ‘train’ or ‘val’ and <code>dataset_sizes[phase]</code> being the all of the examples used in training or validation. So yes he does divide by all of the samples but before accumulating the loss value he multiplies by all the examples in one batch.</p>
<p>Any insight on why this is done this way?</p>
<p>If I divide the loss value by the totality of examples like you suggest, I get really small loss values, like 0.003 and 0.002.<br>
Help.</p>

username: johnolafenwa
cooked: <p>There is a flaw in my earlier explanation<br>
Running loss should be<br>
<code>running_loss += loss.item() * inputs.size(0)</code><br>
Not<br>
<code>running_loss += loss.item()</code></p>
<p>Contrary to what I earlier said, the loss returned is an average over all examples in a batch, given this, you could simply divide the running loss by the number of batches, however since the final batch is often not the same size with others, your results would be inaccurate, hence, the solution is to reverse the average by multiplying the loss in a batch by the total number of images in the batch, this gives unaveraged sum of losses over all examples, when you divide the running loss by the total number of images ,you get accurate loss metrics.</p>
<p>The line<br>
<code>dataset_sizes[phase]</code><br>
Uses the approach suggested by <a class="mention" href="/u/roaffix">@roaffix</a><br>
It returns the number of Images in the dataset.</p>

username: Diego
cooked: <p>Great! Thank a lot for your help. Now that we know that I am calculating the metrics correctly. What does it mean that the validation loss is always smaller than the train loss, this is certainly not normal IMO and I don’t now how to deal with it. Any tips?<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/b/baf142be515c19a759e169863b6aaf11c4981a7b.png" alt="last_layer_only" data-base62-sha1="qFLHUl1q6uhHHbRmxntLLjc9eNd" width="640" height="480"></p>

username: johnolafenwa
cooked: <p>You are welcome. It is possible for your train loss to be higher than your test loss. Often this occurs early in training. This might be due to the total number of test images being substantially smaller than the train set. How many images is in your train and test sets respectively?</p>

username: Diego
cooked: <p>I use 8221 images for training and about 25% of that (2001) for validation</p>

username: ptrblck
cooked: <p>This issue often occurs when using regularization like <code>Dropout</code>, i.e. <code>Dropout</code> makes your models “smaller” (lowers the capacity), thus your training loss might be higher than your validation loss.<br>
However, since you are using a resnet, this won’t be the reason.</p>
<p>Another reason is the intrinsic variance of the dataset as <a class="mention" href="/u/johnolafenwa">@johnolafenwa</a> said. If your dataset is quite small, the inner variance of the training dataset is just higher than in the validation set.</p>
<p>The most likely reason for me is, because you are collecting your training loss during the whole epoch while you are training the model. So at the beginning of each epoch the loss will be higher, at the end of an epoch lower. Since you are just summing the losses and dividing by the total number of images, your estimate might be a bit biased.</p>

username: Diego
cooked: <p>Thanks a lot for your reply. Is there any way to reduce this bias or do I just have to accept it and move on. Is there any better way to calculate and/or plot the losses?</p>

username: ptrblck
cooked: <p>Well, you could calculate the resubstitution error (using the training data) after each epoch with <code>model.eval()</code>, but to be honest, I would just stick to this estimate.</p>

username: Lionking
cooked: <p>In fact the loss is average rather than the sum of losses unless you indicate <code>reduction = 'sum'</code> in the loss function.</p>

username: maureyes
cooked: <aside class="quote no-group" data-username="Diego" data-post="3" data-topic="17665">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/diego/48/6989_2.png" class="avatar"> Diego:</div>
<blockquote>
<p>best_val_loss</p>
</blockquote>
</aside>
<p>Hi Diego, what was the first value that you use for ‘best_val_loss’ ? Could you explain a little bit that part? I am interested to save the best model. Thanks</p>

username: SpandanMadan
cooked: <p>Your validation accuracy is consistently higher than train accuracy. Strongly suggest to check this, it shouldn’t be happening.</p>