I’m confused about the way that I calculate my loss function

Bahaa_Kattan: <p>I’m confused about the way that I calculate my loss</p>
<p>here is the function:</p>
<pre><code class="lang-auto">
def test_epoch(iterator, model, criterion):

    train_loss = 0

    all_y = []

    all_y_hat = []

    model.eval()

    for batch in iterator:

        y = torch.stack([batch.toxic,

                         batch.severe_toxic,

                         batch.obscene,

                         batch.threat,

                         batch.insult,

                         batch.identity_hate],dim=1).float().to(device)

        text, length = batch.comment_text

        length = length.to('cpu')

        with torch.no_grad():

            y_hat = model(text, length)

        loss = criterion(y_hat, y)

        train_loss += loss.item()

        all_y.append(y)

        all_y_hat.append(y_hat)

    y = torch.vstack(all_y)

    y_hat = torch.vstack(all_y_hat)

    roc = roc_auc_score(y.cpu(),y_hat.round().detach().cpu())

    return train_loss / len(y) , roc

</code></pre>
<p>the way I calculated my loss in the above function is here:</p>
<pre><code class="lang-auto">
train_loss = 0

...

loss = criterion(y_hat, y)

...

train_loss += loss.item()

...

return train_loss / len(y) , roc

</code></pre>
<p>and it gives at the first epoch</p>
<p><code>Loss: 0.0148(valid)  | roc: 0.547727 (valid)</code></p>
<p>but when I calculate the loss in this way:</p>
<pre><code class="lang-auto">
all_loss = []

...

loss = criterion(y_hat, y)

...

all_loss.append(loss.item())

...

return np.mean(all_loss), roc

</code></pre>
<p>it gives at the first epoch</p>
<p>` Loss: 0.7691(valid) | roc: 0.548824 (valid)</p>
<p>`</p>
<p>why the loss in the first way is totally different from the loss in the second way . and which one should I use or rely on ?</p>
<p>THANKS !</p>

ptrblck: <p>Could you provide an executable code snippet using random data to reproduce the different results, please?</p>

Alexey_Demyanchuk: <p>Most likely because in first case:</p>
<ol>
<li>You collect <code>loss.item()</code> which by default already averaged to a batch size</li>
<li>and later you average it by number of samples in the full epoch <code>len(y)</code>
</li>
</ol>
<p>and in the second case you:</p>
<ol>
<li>append all <code>loss.items()</code> to the list</li>
<li>average it by the number of items in the list <code>np.mean(all_loss)</code>
</li>
</ol>
<p>Second way, I suppose gives a correct result. To correct the first case, you need to average by the number of batches  <code>train_loss / len(iterator)</code>. This way results should be equal, I guess.</p>
<p>To summirize: <code>len(y)</code> from the first case is not equal <code>len(all_loss)</code> in the second case.</p>

Alexey_Demyanchuk: <p>In case you want some (sorry it is lengthy <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> ) code example:</p>
<pre><code class="lang-auto">import numpy as np
import torch
import torchvision
import torchvision.transforms as transforms
import torch.nn as nn
import torch.nn.functional as F

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
transform = transforms.Compose(
    [transforms.ToTensor(),
     transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])
testset = torchvision.datasets.CIFAR10(root='./data', train=False,
                                       download=True, transform=transform)
testloader = torch.utils.data.DataLoader(testset, batch_size=64,
                                         shuffle=False, num_workers=2)
net = torchvision.models.resnet18()
net.fc = nn.Linear(512, 10)  # for CIFAR10
net.to(device)
criterion = nn.CrossEntropyLoss()
</code></pre>
<p>Case 1:</p>
<pre><code class="lang-auto">def test_epoch_case1(iterator, model, criterion):

    train_loss = 0
    all_y = []

    model.eval()

    for batch in iterator:
        X, y = batch[0], batch[1]
        X, y = X.to(device), y.to(device)

        with torch.no_grad():
            y_hat = model(X)

        loss = criterion(y_hat, y)
        train_loss += loss.item()

        all_y.append(y)

    y = torch.cat(all_y)
    print(f'This function returns collected train_loss: {train_loss} averaged by number of samples in y: {len(y)}')

    return train_loss / len(y)

test_epoch_case1(testloader, net, criterion)
</code></pre>
<p>prints:</p>
<pre><code class="lang-auto">This function returns collected train_loss: 362.682909488678 averaged by number of samples in y: 10000
0.0362682909488678
</code></pre>
<p>Case 2:</p>
<pre><code class="lang-auto">def test_epoch_case2(iterator, model, criterion):

    all_loss = []
    model.eval()

    for batch in iterator:
        X, y = batch[0], batch[1]
        X, y = X.to(device), y.to(device)

        with torch.no_grad():
            y_hat = model(X)

        loss = criterion(y_hat, y)
        all_loss.append(loss.item())

    print(f'This function returns collected train_loss: {np.sum(all_loss)} averaged by number of batches in dataloader: {len(iterator)} = {np.sum(all_loss) / len(iterator)}')

    return np.mean(all_loss)

test_epoch_case2(testloader, net, criterion)
</code></pre>
<p>prints:</p>
<pre><code class="lang-auto">This function returns collected train_loss: 362.682909488678 averaged by number of batches in dataloader: 157 = 2.310082226042535
2.310082226042535
</code></pre>
<p>Case 1 (corrected):</p>
<pre><code class="lang-auto">def test_epoch_case1_correct(iterator, model, criterion):

    train_loss = 0
    all_y = []

    model.eval()

    for batch in iterator:
        X, y = batch[0], batch[1]
        X, y = X.to(device), y.to(device)

        with torch.no_grad():
            y_hat = model(X)

        loss = criterion(y_hat, y)
        train_loss += loss.item()

        all_y.append(y)

    y = torch.cat(all_y)
    print(f'This function returns collected train_loss: {train_loss} averaged by number of batches in dataloader: {len(iterator)}')

    return train_loss / len(iterator)

test_epoch_case1_correct(testloader, net, criterion)
</code></pre>
<p>prints:</p>
<pre><code class="lang-auto">This function returns collected train_loss: 362.682909488678 averaged by number of batches in dataloader: 157
2.310082226042535
</code></pre>
<p>Hope it helps <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>