Plotting loss curve

ahmed: <p>I am trying to plot a loss curve by each epoch, but I’m not sure how to do that. I can do it for 1 epoch using the following method:</p>
<pre><code class="lang-auto">def train(model, num_epoch):

    for epoch in range(num_epoch):
        running_loss = 0.0
        loss_values = []
        for i, data in enumerate(trainloader, 0):
            images, labels, bbox = data
            
            images = Variable(images).to(device)
            labels = Variable(labels).to(device)
                        
            optimizer.zero_grad()
            outputs = model(images)#, th_images)
            loss = criterion(outputs, labels.to(device))
            loss.backward()
            optimizer.step()
            
            running_loss += loss.item()
            
            loss_values.append(running_loss)
            
            if i % 10 == 9:    # print every 2000 mini-batches
                
                print('[%d, %5d] loss: %.3f' %
                      (epoch + 1, i + 1, running_loss / 10))
                running_loss = 0.0
       
    print('Finished Training Trainset')
            
    plt.plot(np.array(loss_values), 'r')

model = train(vgg16, 2)
</code></pre>
<p>However, I get the following plot, which doesn’t seem correct:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/2X/6/62703077ee6bd23a2f261aed5b7b4adaa8d8387f.png" alt="image" data-base62-sha1="e2P9I6vlZOcjDoFV5nLnzjoyXy7" width="183" height="126"></p>
<p>Any advice would be grealty appreciated.</p>

ptrblck: <p>Currently you are accumulating the batch loss in <code>running_loss</code>. If you just would like to plot the loss for each epoch, divide the <code>running_loss</code> by the number of batches and append it to <code>loss_values</code> in each epoch.<br>
Note, that this might give you a slightly biased loss if the last batch is smaller than the others, so let me know if you need the exact loss.</p>

ahmed: <p>Thank you for your reply. I will probably need an exact loss when I use the full dataset, but I’m only using a small portion of it, so probably don’t require that right now.</p>
<p>If you could tell me how to get exact values, then that would be great for when I use the full dataset.</p>
<p>I’m just a bit coursed about ‘append it to  <code>loss_values</code>  in each epoch’. Based on the image I posted, the x axis is not the epochs. I believe it is using the optimizer.step values instead.</p>

ptrblck: <p>You could use the <a href="https://github.com/pytorch/examples/blob/master/imagenet/main.py#L327" rel="nofollow noopener">ImageNet example</a> or the following manual approach:</p>
<pre><code class="lang-python">for epoch in range(num_epochs):
    running_loss = 0.0
    for i, data in enumerate(trainloader, 0):
        running_loss =+ loss.item() * images.size(0)

    loss_values.append(running_loss / len(train_dataset))

plt.plot(loss_values)
</code></pre>
<p>This code would plot a single loss value for each epoch. Would that work?</p>

ahmed: <p>Thank you for your reply. I can already plot the loss curve for one epoch, but I was looking to plot the loss for a number of epochs. Using the manual approach you suggested above, this is the loss curve generated:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/2X/e/e177827e072bb293bee00a3e972153aeaada0a9b.png" alt="image" data-base62-sha1="waztIluHXphdBSOA6To2eksCETp" width="187" height="126"></p>
<p>What I would like to do is something like this:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/8/864224b4224e47216fdf443996b517f9e259438d.png" data-download-href="https://discuss.pytorch.org/uploads/default/864224b4224e47216fdf443996b517f9e259438d" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/8/864224b4224e47216fdf443996b517f9e259438d_2_331x250.png" alt="image" data-base62-sha1="j9HGWdxgayXa7xoi4dTDITpfgeN" width="331" height="250" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/8/864224b4224e47216fdf443996b517f9e259438d_2_331x250.png, https://discuss.pytorch.org/uploads/default/optimized/2X/8/864224b4224e47216fdf443996b517f9e259438d_2_496x375.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/8/864224b4224e47216fdf443996b517f9e259438d_2_662x500.png 2x" data-dominant-color="F9FAFA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">812×612 43.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I hope that clears up what I wanting to do.</p>

ptrblck: <p>The code I’ve posted should plot a single loss values for each epoch.<br>
I assume you let your model train for 25 epochs, is that correct?</p>
<p>If so, the plots should show basically the same with the difference that the second plot shows the train and validation loss for each epoch.</p>

ahmed: <p>I am attempting to do only 2 epochs according to the line <code>model = train(vgg16, 2)</code>. I am unsure where the 25 on the x axis is coming from. I believe it may have something to do with the optimist.step? Also, my trainset is 100 images and my batch size in the trainloader is 4, so perhaps it maybe something to with that instead?</p>

Manu: <p>You can also try <strong>Pierogi</strong>:<br>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicon.ico" class="site-icon" width="16" height="16">
      <a href="https://github.com/nalepae/pierogi/" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars1.githubusercontent.com/u/4943830?s=400&amp;amp;v=4" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://github.com/nalepae/pierogi/" target="_blank" rel="nofollow noopener">nalepae/pierogi</a></h3>

<p>Plot your learning curves in real time in your web browser. - nalepae/pierogi</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>

ahmed: <p>Thank you for your suggestion. I managed to get it working, but I will also have a look at this.</p>

AMEL_BENAIDA: <p>Please how did you get it work ? I am stuck at the same problem<br>
I train for 2 epochs but here is what I get<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/0/0da0e25e1858efb6dab00cca22b8a01db3e44056.png" alt="image" data-base62-sha1="1WyTNHT6nPBlP39R341rofRyXHw" width="392" height="257"></p>

ahmed: <p>If I remember correctly, I did something like this:</p>
<pre><code class="lang-auto">    losses = []
    for epoch in range(num_epochs):
        running_loss = 0.0
        for data in dataLoader:
            images, labels = data
            
            outputs = model(images)
            loss = criterion_label(outputs, labels)
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()

            running_loss += loss.item() * images.size(0) 

        epoch_loss = running_loss / len(dataloaders['train'])
        losses.append(epoch_loss)
</code></pre>
<p>Let me know if this helps. It’s based on the solution the suggestion made by ptrblck in this post.</p>

AMEL_BENAIDA: <p>Yes that’s it, thank you a lot for your fast reply</p>

saba: <p>HI Ptrblck,</p>
<p>Sorry, I am trying to show different histograms on one figures by seaborn.</p>
<p>One time I use (sns.displot(), and another time I use sns.histplot().<br>
The problem is that the order of the histograms has change  and it shouldn’t happen. I used these command, the hue is same. every thing is same I just replace sns.displot() with sns.histplot()</p>
<p>ax = sns.displot(Data, x=‘Minimum intensity value’,hue=‘method’,bw_adjust=.1,kind=“kde”,fill=False)<br>
ax=sns.histplot(Data, x=‘Minimum intensity value’,hue=‘method’,bins=np.linspace(0,1,200),stat=“density”,multiple=“stack”,common_norm=“False”)</p>
<p>I try different bins in sns.histplot, but the order of the histogram are not sames in two figures.<br>
I appreciate your comment.</p>

ptrblck: <p>This question is very seaborn-specific and I would recommend to ask in their discussion board or any support channel.</p>

pgou: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> I wonder how to properly plot the training and validation curves wrt iterations (not epochs) as they do in, e.g., ResNet paper on ImageNet.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/5/f5f46654a4f0662c2fa915daf5e3c61e6f610daf.png" data-download-href="https://discuss.pytorch.org/uploads/default/f5f46654a4f0662c2fa915daf5e3c61e6f610daf" title="Screenshot from 2022-02-07 10-54-54"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/5/f5f46654a4f0662c2fa915daf5e3c61e6f610daf_2_690x239.png" alt="Screenshot from 2022-02-07 10-54-54" data-base62-sha1="z5OBsNb4G6Qdtig7JeY4CsKe2dh" width="690" height="239" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/5/f5f46654a4f0662c2fa915daf5e3c61e6f610daf_2_690x239.png, https://discuss.pytorch.org/uploads/default/optimized/3X/f/5/f5f46654a4f0662c2fa915daf5e3c61e6f610daf_2_1035x358.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/f/5/f5f46654a4f0662c2fa915daf5e3c61e6f610daf.png 2x" data-dominant-color="F4F3F3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2022-02-07 10-54-54</span><span class="informations">1075×373 61.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>In case of loss wrt epochs, it is clear, as you described, you just average over the whole dataset for both train/val sets, but for error wrt iterations it is less obvious to me.</p>
<p>To get the validation error/loss, I guess they freeze the model time-to-time (for a given number of training iterations ~5K) then they evaluate error on the whole val set (an epoch of the val set). Is that correct?</p>
<p>But how to get the training error they plot wrt iterations? Do they compute a running error on the last train iterations (how many?) ? Do they reset training running error at some point? Do they fix the weights and compute error on the whole train set as for the val set (I guess not because the training curve is much less smooth than the val one) ?</p>
<p><strong>More general question</strong>: when we do not want to work “per epochs”, but “per iterations” (when the dataset is very large for example), what are good practices to report train/val loss/error?</p>
<p>Thank you <a class="mention" href="/u/ptrblck">@ptrblck</a> for all you very good answers you provide every times on this forum!</p>

ptrblck: <aside class="quote no-group" data-username="pgou" data-post="15" data-topic="42632">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/4af34b/48.png" class="avatar"> pgou:</div>
<blockquote>
<p>I guess they freeze the model time-to-time (for a given number of training iterations ~5K) then they evaluate error on the whole val set (an epoch of the val set). Is that correct?</p>
</blockquote>
</aside>
<p>That might be the case, but I would hope the actual research paper would describe their approach (or maybe you could find a repository with their implementation?).</p>
<aside class="quote no-group" data-username="pgou" data-post="15" data-topic="42632">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/4af34b/48.png" class="avatar"> pgou:</div>
<blockquote>
<p>But how to get the training error they plot wrt iterations? Do they compute a running error on the last train iterations (how many?) ? Do they reset training running error at some point? Do they fix the weights and compute error on the whole train set as for the val set (I guess not because the training curve is much less smooth than the val one) ?</p>
</blockquote>
</aside>
<p>The definite answer would come from the authors, but I would guess the current (mean) batch loss is used to create the training loss curves.</p>

pgou: <p>Actually I was not able to find answers in the paper/code that is why I was asking for good practices/general practices when we work wrt iterations and not epochs <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

elemeo: <p>Sorry for the out of date question but I came across this answer because I have a similar issue and wanted to ask, why do you multiply the loss with the image size ?</p>
<p>If I have 3 epochs and 12942 steps per epoch, can I not do the following in my training loop ?</p>
<pre><code class="lang-auto">total_losses = []
for epoch_i in range(0, epochs):
    losses = []
    for step, batch in enumerate(train_dataloader):

        b_input_ids = batch[0].to(device)
        b_labels = batch[0].to(device)
        b_masks = batch[1].to(device)

        model.zero_grad()        

        outputs = model(b_input_ids, labels=b_labels, attention_mask=b_masks, token_type_ids=None)

        loss = outputs[0]  

        batch_loss = loss.item()
        losses.append(batch_loss)
  total_losses.append(losses)
</code></pre>
<p>And so fill the total_losses array with 3 sub-arrays with each having the epoch losses ?</p>

ptrblck: <p>I assume you are referring to <a href="https://discuss.pytorch.org/t/plotting-loss-curve/42632/4">this post</a> where the loss of the current batch is multiplied with the batch size and later divided by the number of images (i.e. the length of the <code>dataset</code>).<br>
Your calculation accumulates the batch loss and divides by the number of batches afterwards (i.e. the length of the <code>DataLoader</code>). If not all batches contain the same number of samples your calculation would add a small error to the mean epoch loss while my approach would not run into this issue.<br>
This would be the case if the number of samples are not divisible by the batch size without a remainder and if <code>drop_last</code> is set to <code>False</code> (the default) in the <code>DataLoader</code>.</p>

trex: <p>Hi,</p>
<p>I believe this line of your code needs to be fixed:</p>
<p>Currently reads:<br>
<code>        running_loss =+ loss.item() * images.size(0)</code></p>
<p>Should be:<br>
<code>       running_loss += loss.item() * images.size(0)</code></p>
<p>Regards,<br>
Trex</p>