[SOLVED] Should I use torch.clamp after torch.sigmoid?

username: tzeny
cooked: <p>Hello, I’ve tried to write a custom loss function, a Weighted Binary Cross Entropy. As suggested by <a class="mention" href="/u/miguelvr">@miguelvr</a> here:</p><aside class="quote" data-post="1" data-topic="3114">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/miguelvr/48/366_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/solved-class-weight-for-bceloss/3114">[SOLVED] Class Weight for BCELoss</a> 
  </div>
  <blockquote>
    Hey there, 
I’m trying to increase the weight of an under sampled class in a binary classification problem. 
torch.nn.BCELoss has a weight attribute, however I don’t quite get it as this weight parameter is a constructor parameter and it is not updated depending on the batch of data being computed, therefore it doesn’t achieve what I need. 
What is the correct way of simulating a class weight, similar to the way Keras does? 
Cheers
  </blockquote>
</aside>
<p>
I’ve tried to use the following function (wrapped inside a class):</p>
<pre><code class="lang-auto">def weighted_binary_cross_entropy(output, target, weights=None):
        
    if weights is not None:
        assert len(weights) == 2
        
        loss = weights[1] * (target * torch.log(output)) + \
               weights[0] * ((1 - target) * torch.log(1 - output))
    else:
        loss = target * torch.log(output) + (1 - target) * torch.log(1 - output)

    return torch.neg(torch.mean(loss))
</code></pre>
<p>The problem is that sometimes this outputs nan or -inf.</p>
<p>Tracing it back I reached the conclusion that sometimes my model outputs very small numbers, for example -136. This leads to:</p>
<p><code>torch.sigmoid(torch.tensor([-136.])) -&gt; tensor([0.])</code> which leads to -inf in <code>torch.log()</code>.</p>
<p>I am on pyTorch 1.0.1.</p>
<p>Is it ok to use <code>torch.clamp(torch.sigmoid(torch.tensor([-136.])), min=1e-8, max=1 - 1e-8)</code> ?</p>

username: miguelvr
cooked: <p>that function had that problem, in the end I used the NLLLoss.</p>
<p>But it is indeed okay to do that to fix the problem.</p>

username: tzeny
cooked: <p>Thank you, I tried to use it like this, but it doesn’t work. The network does not train, even when the weights are (0.5, 0.5). Do you have any idea why?</p>

username: KFrank
cooked: <p>Hello Andrei!</p>
<aside class="quote no-group quote-modified" data-username="tzeny" data-post="1" data-topic="46819" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tzeny/48/12689_2.png" class="avatar"> tzeny:</div>
<blockquote>
<p>…<br>
Tracing it back I reached the conclusion that sometimes my model outputs very small numbers, for example -136. This leads to:</p>
<p><code>torch.sigmoid(torch.tensor([-136.])) -&gt; tensor([0.])</code> which leads to -inf in <code>torch.log()</code>.</p>
<p>Is it ok to use <code>torch.clamp(torch.sigmoid(torch.tensor([-136.])), min=1e-8, max=1 - 1e-8)</code> ?</p>
</blockquote>
</aside>
<p>The short (narrow) answer – use <code>LogSigmoid</code>.</p>
<p>Please see my reply to your “Weighted Binary Crossentropy”<br>
post:</p>
<aside class="quote quote-modified" data-post="2" data-topic="47055">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/class-weighed-binary-crossentropy-not-working-even-with-equal-weights/47055/2">[SOLVED] Class Weighed Binary Crossentropy not working, even with equal weights</a> <a class="badge-category__wrapper " href="/c/autograd/7"><span data-category-id="7" data-drop-close="true" class="badge-category " title="A category of posts relating to the autograd engine itself."><span class="badge-category__name">autograd</span></span></a>
  </div>
  <blockquote>
    Hi Andrei! 

First, to answer the question I think you’re asking: 
You should be using (as in the comment in your code) 
BCEWithLogitsLoss. 
BCEWithLogitsLoss supports sample weights, which you 
can use for class weights. 
Let’s say you have class weight w_1 for class 1, and w_0 
for class 0.  Let w_n be the sample weight for sample n. 
Simply set w_n = w_1 if y_n = 1, and w_n = w_0 if 
y_n = 0.  (This assumes that the y_n are either 0 or 1, as 
they should be if they are binary class labels.) 
N…
  </blockquote>
</aside>

<p>Best.</p>
<p>K. Frank</p>

username: tzeny
cooked: <p>Thank you for your replies, I will mark this as solved.</p>
<p>One thing to note for anyone who might be thinking about implemented custom loss functions in the future: be careful about the numeric stability of your implementation.</p>