Tresholding the prediction image to binary before sending to loss function

username: einrone
cooked: <p>Hi!</p>
<p>There is a question that I have wondered and I have tried to search for it but cant find anything. My question is:</p>
<p><strong>Is it wrong to treshold the prediction image (output from a segmentation network) and send it to the loss function.</strong> By that I mean:</p>
<pre><code class="lang-auto">prediction_image = model(image)
prediction_image[prediction_image &gt;= treshold] = 1.0
prediction_image[prediction_image &lt; treshold] = 0.0

loss = loss_function(prediction_image, ground_truth)
</code></pre>
<p>The ground truth is already a mask consisting of 0 and 1. Meaning if we have modified dice score (tversky), which is</p>
<pre><code class="lang-auto">inputs = inputs.view(-1)
        targets = targets.view(-1)
        
        #True Positives, False Positives &amp; False Negatives
        TP = (inputs * targets).sum()    
        FP = ((1-targets) * inputs).sum()
        FN = (targets * (1-inputs)).sum()
       
        Tversky = (TP + smooth) / (TP + alpha*FP + beta*FN + smooth)  
        
        return 1 - Tversky
</code></pre>
<p>woudnt that be more correct?</p>

username: Andrei_Cristea
cooked: <p>Naively I would worry that creating this kind of kink in your output would zero out your gradients unless your output happens to be right around the threshold. This would make learning slower (and perhaps jumpier?) However, perhaps you’ve already tried and found that it works, in which case you could post a a plot of your validation loss vs baseline.</p>

username: einrone
cooked: <p>Thanks for the reply Andrei, really appriciated it! I have not done it yet. I have done it for metrics though. Before I calculate recall and dice score, I set</p>
<pre><code class="lang-auto">prediction_image[prediction_image &gt;= treshold] = 1.0
prediction_image[prediction_image &lt; treshold] = 0.0
</code></pre>
<p>And then calculate TP, FN and FP. Do you recommend that, or should I just have the raw prediction image when calculating different metrics (by this I do not mean loss function, only evaluation metrics)</p>

username: Andrei_Cristea
cooked: <p>Suppose we call model A the one trained without thresholding, and model B your suggested new model, trained by first thresholding and then applying the loss function. The two models have the same architecture, but different parameters (due to the different training approaches used).</p>
<p>I think when looking at metrics, the important thing for this particular question is to do the same thing for both model A and B so we are comparing apples to apples. Depending on your problem, you may very likely be “forced” to threshold by the type of answer your model is required to provide anyway (like you may have to allocate every pixel to either type X or type Y) so I don’t see the issue with doing it for evaluation.</p>
<p>EDIT: The “kinked” prediction may actually not be differentiable, in which case you should expect to see no training at all.</p>
<p>EDIT 2: per the snippet below, indeed I think this type of thresholding would zero out the gradients and would prevent any training.</p>
<p>normally you expect this:</p>
<pre><code class="lang-auto">vec1 = torch.rand(10, requires_grad=True)
vec2 = torch.rand(10, requires_grad=True)
s = vec1 + vec2
print(s)
s.sum().backward()
print(vec1.grad)
print(vec2.grad)

Output:
tensor([0.6766, 1.4995, 0.7432, 1.7373, 1.1667, 0.8818, 1.2005, 0.6173, 0.8586,
        0.6945], grad_fn=&lt;AddBackward0&gt;)
tensor([1., 1., 1., 1., 1., 1., 1., 1., 1., 1.])
tensor([1., 1., 1., 1., 1., 1., 1., 1., 1., 1.])
</code></pre>
<p>With thresholding:</p>
<pre><code class="lang-auto">vec1 = torch.rand(10, requires_grad=True)
vec2 = torch.rand(10, requires_grad=True)
s = vec1 + vec2
s[s &gt; 1.0] = 1.0  # thresholding
s[s &lt; 1.0] = 0.0  # thresholding
print(s)
s.sum().backward()
print(vec1.grad)
print(vec2.grad)

Output:
tensor([1., 1., 1., 1., 1., 0., 0., 1., 0., 0.], grad_fn=&lt;IndexPutBackward&gt;)
tensor([0., 0., 0., 0., 0., 0., 0., 0., 0., 0.])
tensor([0., 0., 0., 0., 0., 0., 0., 0., 0., 0.])
</code></pre>
<p>Note the gradients (last two tensors in the output) are zero everywhere, which means they can’t train.</p>
<p>However I think even if you came up with some implementation that’s not quite so kinked and still accomplished thresholding to some degree (raising the difference from the threshold to some power, for example) you would still find that it will get gradients closer to zero and slow down training.</p>

username: InnovArul
cooked: <aside class="quote no-group" data-username="einrone" data-post="3" data-topic="149092">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/einrone/48/34783_2.png" class="avatar"> einrone:</div>
<blockquote>
<pre><code class="lang-auto">prediction_image[prediction_image &gt;= treshold] = 1.0
prediction_image[prediction_image &lt; treshold] = 0.0
</code></pre>
</blockquote>
</aside>
<p>If at all possible, would you be able to try how straight-through estimator works in your case?</p>
<aside class="quote" data-post="2" data-topic="149024">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/innovarul/48/5282_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/the-parameters-of-the-model-with-custom-loss-function-doesnt-upgraded-thorough-its-learning-over-epochs/149024/2">The parameters of the model with custom loss function doesn't upgraded thorough its learning over epochs</a> <a class="badge-category__wrapper " href="/c/deployment/14"><span data-category-id="14" data-drop-close="true" class="badge-category " title="A category of posts focused on production usage of PyTorch. Mobile deployment is out of scope for this category (for now…  )"><span class="badge-category__name">deployment</span></span></a>
  </div>
  <blockquote>
    I am suspecting that torch.where may not be backpropagating gradients. 
If you really want to threshold, try to use straight-through estimator trick as follows: 
thresholded_inputs = torch.where(thresholded_inputs &lt; threshold, 0, 1)
inputs = (inputs + thresholded_inputs) - inputs.detach()

... calculate IoU loss ...
  </blockquote>
</aside>


username: einrone
cooked: <p>I do lesion segmentation for my thesis, so forcing the prediction image to either background and foreground in this case is good?</p>
<p>Background : 0<br>
Foreground/lesion : 1</p>

username: Andrei_Cristea
cooked: <p>For evaluation, yes it should be fine.</p>
<p>Please do check out InnovArul’s reply and link above as well where is suggesting you try something called “straight through estimator” before calculating loss. In thinking about it more, direct thresholding (like what we were discussing) is guaranteed to zero out the gradients and to therefore will not work (though you should confirm that yourself) however the estimator he suggests at least has a chance (though I’m still trying to get my head around it, based on the debate in the linked thread).</p>
<p>straight-through estimator, taken from other thread:</p>
<pre><code class="lang-auto">thresholded_inputs = torch.where(thresholded_inputs &lt; threshold, 0, 1)
inputs = (inputs + thresholded_inputs) - inputs.detach()
</code></pre>

username: einrone
cooked: <p>Hi innovarul! Thank you for the reply☺️</p>
<p>The thing is, i dont have problem with my model learning, I just want to help the model and optimizer a bit, and see if the validation dice score increases or not. Do you think this technique will help?</p>

username: InnovArul
cooked: <p>I am not so sure if it will improve. You can give it a try and see how it goes.</p>
<p>When the thresholding operation (at 0.5) is applied, all probabilities between <code>[0.5, 1]</code> are mapped to <code>1</code>. Thus it may easily satisfy the loss function, even though the actual probabilities might be less (&lt; 0.7 for example). There is a chance that it might degrade the performance too. You have to train and check how it works.</p>

username: KFrank
cooked: <p>Hi Andrei (and Einrone and Arul)!</p>
<aside class="quote no-group" data-username="Andrei_Cristea" data-post="7" data-topic="149092" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/andrei_cristea/48/47843_2.png" class="avatar"> Andrei_Cristea:</div>
<blockquote>
<p>Please do check out InnovArul’s reply and link above as well where is suggesting you try something called “straight through estimator”</p>
</blockquote>
</aside>
<p>For completeness, let me link to some further comments on this issue<br>
that I posted in the aforementioned thread:</p>
<aside class="quote quote-modified" data-post="7" data-topic="149024">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/the-parameters-of-the-model-with-custom-loss-function-doesnt-upgraded-thorough-its-learning-over-epochs/149024/7">The parameters of the model with custom loss function doesn't upgraded thorough its learning over epochs</a> <a class="badge-category__wrapper " href="/c/deployment/14"><span data-category-id="14" data-drop-close="true" class="badge-category " title="A category of posts focused on production usage of PyTorch. Mobile deployment is out of scope for this category (for now…  )"><span class="badge-category__name">deployment</span></span></a>
  </div>
  <blockquote>
    Hi Arul! 

I am aware that this “straight-through estimator” trick is recommended 
from time to time as a work-around for zero gradients (or a broken 
computation graph) in the presence of thresholding. 
Let me expand a little on what I see as its problems: 
First, it is easy to implement and use “soft” thresholds for which you get 
useful (and correct) gradients.  So, what is the benefit of using something 
that has demonstrable problems?  (The fact that you might want to look at 
a performance…
  </blockquote>
</aside>

<p>Best.</p>
<p>K. Frank</p>