Kaggle Data Science Bowl - Loss not decreasing and predictions are not good

yashkatariya: <p>I am attaching the url of the jupyter notebook. I can’t seem to figure out why my loss is not decreasing. I have been stuck on this for 3 days, any help is very much appreciated. The command to download the data is in the jupyter notebook.</p>
<p><a href="https://drive.google.com/file/d/1ZrwYiyddRsT6Hi9iemGIXn-Y5PhwmBqR/view?usp=sharing" class="onebox" target="_blank" rel="noopener nofollow ugc">https://drive.google.com/file/d/1ZrwYiyddRsT6Hi9iemGIXn-Y5PhwmBqR/view?usp=sharing</a></p>
<p>Labels shape: (batch_size, 1, 256, 256)<br>
Prediction shape: (batch_size, 1, 256, 256)</p>
<p>Is something wrong with how I calculate loss?</p>
<pre data-code-wrap="python"><code class="lang-python">for epoch in range(15):
    for i, data in enumerate(trainloader):
        inputs, labels = data
        inputs = Variable(inputs.cuda())
        labels = Variable(labels.cuda())
        
        # forward + backward + optimize
        
        # zeroes the gradient buffers of all parameters
        optimizer.zero_grad()
        #forward pass
        outputs = model_pytorch(inputs)
        # calculate the loss
        loss = loss_function(outputs, labels)
        # backpropagation
        loss.backward()
        # Does the update after calculating the gradients
        optimizer.step()
        
        if (i+1) % 5 == 0:
            print('[%d, %5d] loss: %.4f' % (epoch, i+1, loss.data[0]))
</code></pre>
<p><strong>Image of loss values</strong></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/8/8e0fdd99ee9963525783fd11e18dc50a7fda96a5.png" data-download-href="https://discuss.pytorch.org/uploads/default/8e0fdd99ee9963525783fd11e18dc50a7fda96a5" title="loss"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/8/8e0fdd99ee9963525783fd11e18dc50a7fda96a5_2_690x387.png" alt="loss" data-base62-sha1="kgJN9mLswuLf6oGZY8xoY6RsoPX" width="690" height="387" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/8/8e0fdd99ee9963525783fd11e18dc50a7fda96a5_2_690x387.png, https://discuss.pytorch.org/uploads/default/optimized/2X/8/8e0fdd99ee9963525783fd11e18dc50a7fda96a5_2_1035x580.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/8/8e0fdd99ee9963525783fd11e18dc50a7fda96a5.png 2x" data-dominant-color="EBE9EA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">loss</span><span class="informations">1366×768 173 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>How did you define your <code>loss_function</code>?<br>
Your collab notebook wants some permission, am I doing something wrong?</p>

yashkatariya: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Here is the link for the notebook (I have updated it in the main post also)<br>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://ssl.gstatic.com/docs/doclist/images/icon_14_generic_favicon.ico" class="site-icon" width="16" height="16">
      <a href="https://drive.google.com/file/d/1ZrwYiyddRsT6Hi9iemGIXn-Y5PhwmBqR/view?usp=sharing" target="_blank" rel="nofollow noopener">Google Docs</a>
  </header>
  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:114/60;"><img src="https://lh5.googleusercontent.com/CA0lA0RxSbtbBbwgDEeUyspmpbQLlpEGGdxvVlJbxUBxyvA4vEaUAKVS07Y=w1200-h630-p" class="thumbnail"></div>

<h3><a href="https://drive.google.com/file/d/1ZrwYiyddRsT6Hi9iemGIXn-Y5PhwmBqR/view?usp=sharing" target="_blank" rel="nofollow noopener">Nuclei Detection.ipynb</a></h3>

<p>Colaboratory notebook</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>

ptrblck: <p>Thanks for the link!</p>
<p>There seem to be some issues in your training loop.<br>
You are doing the following:</p>
<pre><code>inputs = Variable(inputs).cuda()
labels = Variable(labels).cuda()
    
outputs = model_pytorch(inputs)
probs = F.sigmoid(outputs)
probs_flat = probs.view(-1)
labels_flat = probs.view(-1)
loss = loss_function(probs_flat, labels_flat.float())
</code></pre>
<p>You are using <code>probs</code> for <code>probs_flat</code> and <code>labels_flat</code>. Your loss function should therefore return a zero loss.<br>
However, you also call <code>F.sigmoid()</code> twice on <code>outputs</code>! The first time in the training loop and then again in your loss definition:</p>
<pre><code>class DiceLoss(_Loss):
    def forward(self, input, target):
        return 1 - dice_coeff(F.sigmoid(input), target)
</code></pre>
<p>This seems to be wrong.<br>
Could you fix that and see if your model is learning?</p>

yashkatariya: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> I changed the loss function to BCELoss. Can you please check it now?</p>

ptrblck: <p>Looks good! What does the loss do?<br>
Also, your dice loss might be a good idea. I didn’t mean to criticize it!<br>
You should however remove one of the sigmoid calls. <img src="//discuss.pytorch.org/images/emoji/apple/wink.png?v=5" title=":wink:" class="emoji" alt=":wink:"></p>

yashkatariya: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> I just trained the model again and as you can see the loss is not decreasing. BCELoss is the binary cross-entropy loss.  I have also given you edit access.</p>