Is my loss function wrong?

username: Oussama_Bouldjedri
cooked: <p>I was wondering if my loss function calculation is wrong :</p>
<pre><code>def train(model, optimizer, train_loader, epoch,writer):
      model.train()
    
      train_loss=0
      train_accuracy=0
      correct=0
   
    
    
      train_predicted=[]
      train_target=[]
   
      for data, target in train_loader:    
     
        if use_cuda:    
            data, target = data.cuda(), target.cuda()
            
            
           
            data=data.float()
      
            
       
            optimizer.zero_grad()
        
       
            output = model(data)
    
        
        
            _, predicted = torch.max(output.data, 1)
        
        
     
       
            target = target.long() 
        
        

            target_top_numpy=target.cpu().detach().numpy()
        
            predicted_top_numpy=output.cpu().detach().numpy()
  
        
            loss = criterion(output,target)  
       
            loss.backward()
            optimizer.step()
        
            train_loss+=loss.data 
   
            predicted_cpu=predicted.cpu()
            predicted_list=predicted.tolist()
        
            target_cpu=target.cpu()
            target_list=target.tolist()
        
        
            train_predicted=train_predicted+predicted_list
        
        
            train_target=train_target+target_list
        
        
            correct +=(target==predicted).sum().item()
        
   
    
    
    #################################################################################
    
   
    
    
    
      train_accuracy=100. * correct / len(train_loader.dataset)
      print('accuracy is ',train_accuracy)
    
      train_loss /= len(train_loader.dataset) 
      return (train_accuracy,train_loss)</code></pre>

username: ptrblck
cooked: <p>Your code doesn’t show any implementation of a luss function, just the usage of <code>criterion</code>, which looks alright assuming the input tensor shapes and <code>dtype</code>s are correct.</p>

username: Oussama_Bouldjedri
cooked: <p>I taught the right way of doing is following one of the 2 methods explained in this post :</p><aside class="quote quote-modified" data-post="1" data-topic="107890">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/oat/48/34409_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/on-running-loss-and-average-loss/107890">On running loss and average loss</a> 
  </div>
  <blockquote>
    [ 1. Context ] 
I’m following Udacity’s tutorial <a href="https://www.udacity.com/course/deep-learning-pytorch--ud188" rel="noopener nofollow ugc">Intro to Deep Learning with PyTorch</a>. In Lesson 5 on Convolutional Neural Networks by <a href="https://twitter.com/cezannecam" rel="noopener nofollow ugc">Cezanne Camacho</a> Step 10 Training the Network, Cezanne used the code as quoted below to calculate the training loss and validation loss. 
[ 2. Question ] 
Although Cezanne has explained in the video, I’m still not clear why she was using train_loss  += loss.item()*data.size(0) to aggregate the total training loss and why she was using train_loss = train_loss/len(tr…
  </blockquote>
</aside>
<p>
either I use the avereged batch loss calculated by the creterion sum them and devide by the number of batches or<br>
use the loss of all samples of every batch sum all and devide by the size of the data set<br>
which I was missing here ? so either i add the multiplication by the batch size inside the loop or I devide by the number of the batches outside the loop instead of the data set size ?</p>

username: ptrblck
cooked: <p>Right now you are dividing the accumulated loss by the number of samples in the <code>Dataset</code> via:</p>
<pre><code class="lang-python">train_loss /= len(train_loader.dataset)
</code></pre>
<p>which assumes <code>train_loss</code> is the sum of the losses of all samples.<br>
Based on:</p>
<pre><code class="lang-python">train_loss+=loss.data 
</code></pre>
<p>this would be the case if your are using the <code>sum</code> reduction. If not (e.g. if you are using the <code>mean</code> reduction) you would need to scale it with the current batch size (i.e. <code>output.size(0)</code>).</p>

username: Oussama_Bouldjedri
cooked: <p>if i am using a default definition of the loss function  (which by default uses mean()),  assume that I can also use this instead :</p>
<p>train_loss /= len(train_loader) which point the way2 in the previous post you repiled to, the sum of average loss over batches devided by the number of batches ?</p>

username: ptrblck
cooked: <p>This would work if all batches contain the same number of samples. In case the last one has fewer samples, you would add a small error to the loss calculation as also described <a href="https://discuss.pytorch.org/t/on-running-loss-and-average-loss/107890/2">here</a>.</p>

username: Oussama_Bouldjedri
cooked: <p>then I would go with the way1 summing the loss of all the samples along the data set and deviding by the size of the dataset,  my validation cross entropy loss is high very high (1&lt;) compared to the training one, and the validation accuracy is around 65 %, the train accuracy is 95%, this is a clear overffiting ? isnt ?<br>
PS i have a small validation data set around 2000 same for test set , but for the train data is around 22000</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Oussama_Bouldjedri" data-post="7" data-topic="145121">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/oussama_bouldjedri/48/64172_2.png" class="avatar"> Oussama_Bouldjedri:</div>
<blockquote>
<p>and the validation accuracy is around 65 %, the train accuracy is 95%, this is a clear overffiting ?</p>
</blockquote>
</aside>
<p>Yes, due to the large gap between the training and validation accuracy your model would overfit to the training dataset.</p>