How to calculate the loss on Resnet

bioinfo-dirty-jobs: <p>Dear all!<br>
I would like to write this code for training a  model of Resnet34 (<a href="https://github.com/bioinfo-dirty-jobs/pytorch_study/blob/master/train.py" rel="nofollow noopener">train</a>) from image data. I don’t understands some parts in particular row 6 it is correct to do calculation of the loss in this way? I use batch size of 4  and the same for row 23.<br>
It is correct  calculate the ACC  after training  in this way?<br>
Thanks in advance for any help</p>

ptrblck: <p>If you are referring to line17, then yes, this is the standard way to calculate the loss:</p>
<pre data-code-wrap="python"><code class="lang-python">loss = criterion(outputs_tr, labels_tr)
</code></pre>
<aside class="quote no-group" data-username="bioinfo-dirty-jobs" data-post="1" data-topic="76879">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/b/f9ae1b/48.png" class="avatar"> bioinfo-dirty-jobs:</div>
<blockquote>
<p>It is correct calculate the ACC after training in this way?</p>
</blockquote>
</aside>
<p>The script does neither import not define the metric functions (<code>mcor</code>, <code>acc</code>, <code>precision</code>, <code>recall</code>), so we cannot answer, if the implementation is correct.</p>

bioinfo-dirty-jobs: <p>Thanks so much for you kind help! I add that information:</p>
<pre><code class="lang-auto">from sklearn.metrics import accuracy_score as acc

from sklearn.metrics import confusion_matrix

from sklearn.metrics import matthews_corrcoef as mcor

from sklearn.metrics import precision_score as precision

from sklearn.metrics import recall_score as recall

from torch.utils.data import DataLoader

from torch.utils.tensorboard import SummaryWriter

from torchvision import transforms
</code></pre>
<p>I’m sorry I don’t explain well… my problem are refereed to this lines where I start to check the test :</p>
<pre><code class="lang-auto">if j % int(len(loader_train) / 2) == 0 and j != 0:
                model.eval()
                with torch.no_grad():
</code></pre>
<p>Here are performed in this way in some script I found after 10  accumulation… what is the right way to do if I have batch of 4 image at time?</p>

ptrblck: <p>The training batch size doesn’t matter, since you are using <code>loader_test</code>.<br>
The loss calculation looks wrong:</p>
<pre><code class="lang-python">loss_test_avg = losses_sum / num_samples_test
mean_loss_train = losses_sum / (
    len(loader_train) * loader_train.batch_size
)
</code></pre>
<p>I’m not sure which <code>reduction</code> you are using in the <code>criterion</code>, but based on the variable names I would assume <code>reduction='sum'</code>?<br>
On the other hand, since you are dividing by <code>num_samples_test</code> (which is in fact the number of batches), you might be using the default <code>reduction='mean'</code>?</p>
<p>The <code>mean_loss_train</code> calculation doesn’t seem to be correct, since you are dividing the test loss by the number of samples.</p>

bioinfo-dirty-jobs: <p>II don’t found   <strong>reduction</strong> variable. So the loss calculation are corrected in this way?</p>
<pre><code class="lang-auto">for epoch in range(EPOCHS):
      
        for j, data in enumerate(loader_train):
            global_i += 1

            if j % 10 == 0:
                print(time.time() - start_time)
                start_time = time.time()

            optimizer.zero_grad()

            images_tr = data["data"].to(device)
            labels_tr = torch.LongTensor(data["label"]).to(device)
            outputs_tr = model(images_tr).to(device)

            # backward
            loss = criterion(outputs_tr, labels_tr)
            loss.backward()

            optimizer.step()

            # check test set
            if j % int(len(loader_train) / 2) == 0 and j != 0:
                model.eval()
                with torch.no_grad():

                    losses_sum = 0
                    num_samples_test = 0

                    for data_test in loader_test:

                        images_ts = data_test["data"].to(device)
                        labels_ts = torch.LongTensor(data["label"]).to(device)

                        outputs_ts = model.forward(images_ts)

                        loss_test_sum = criterion(outputs_ts, labels_ts).item()
                        losses_sum += loss_test_sum
                        num_samples_test += 1

                    loss_test_avg = losses_sum / num_samples_test
                    
                    

                    last_loss_test = loss_test_avg
                val_epoch_tr_loss=loss.item()/len(loader_train)
                losses_tr.append(val_epoch_tr_loss)
                losses_ts.append(loss_test_avg)

                del images_ts, labels_ts

            iteration += 1
            del images_tr, labels_tr
            gc.collect()
            model.train()

</code></pre>
<p>Is it correct? the loss of training are calculate on the number of data in the training and the loss of  test are calculate with the number of element  of training<br>
Thanks for the kind help!</p>

bioinfo-dirty-jobs: <p><a href="https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html" rel="nofollow noopener">transfer learning tutorial</a> In the tutorial I found  also this way to do loss calculation</p>
<p><code>running_loss += loss.item() * inputs.size(0)</code></p>
<p>So I don’t understand which is the way to  calculate correctly.  inputs.size(0) correspond to the batch that I use?</p>

ptrblck: <p>The <code>running_loss</code> calculation multiplies the averaged batch loss (<code>loss</code>) with the current batch size, and divides this sum by the total number of samples.</p>
<p>In your example you are summing the averaged batch losses and divide by the number of batches.<br>
This might create an offset, if your last batch is smaller than the others.</p>
<p>Your code snippet looks alright, if you want to ignore the potential offset in the loss calculation.<br>
However, <code>val_epoch_tr_loss</code> uses the loss of the last batch during training and divides by the number of batches in the training <code>DataLoader</code>, which still seems to be wrong.</p>

bioinfo-dirty-jobs: <p>thanks for your help. So you suggest to do only this calculation:</p>
<pre><code class="lang-auto">losses_tr.append(loss.item())
print(
                    "Train_loss:{:.4f} Test_loss {:.4f}".format(
                        train_loss_t, last_loss_test
                    )
                )
</code></pre>

ptrblck: <p>In the usual use case you would compute the average of both, the training and test loss.<br>
Your suggested approach of using:</p>
<pre><code class="lang-python">    loss = criterion(outputs_ts, labels_ts)
    losses_sum += loss.item()
    num_samples += 1
...
loss_test_avg = losses_sum / num_samples
</code></pre>
<p>would work for both losses. Although you might see a slight bias (as explained before), it could be “good enough”.</p>
<p>Alternatively, you could use the <a href="https://github.com/pytorch/examples/blob/391be73bbd191a08776c9ba05d29ef1e05c0c83b/imagenet/main.py#L359-L380">AverageMeter</a> from the ImageNet example and update the loss as seen <a href="https://github.com/pytorch/examples/blob/391be73bbd191a08776c9ba05d29ef1e05c0c83b/imagenet/main.py#L292">here</a>.</p>