Modified PyTorch loss function BCEWithLogitsLoss returns NaNs

shlomish3: <p>I’m trying to solve a binary classification problem ( <code>target=0</code>  and  <code>target=1</code> ) with an exception: Some of my labels are classified as  <code>target=0.5</code>  on purpose, and I wish to have  <em>zero loss</em>  for either classifying it as 0 or 1 (i.e both classes are “correct”).</p>
<p>I tried to implement a custom loss from scratch, based on PyTorch’s <a href="https://pytorch.org/docs/master/generated/torch.nn.BCEWithLogitsLoss.html" rel="noopener nofollow ugc">BCEWithLogitsLoss</a>:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/1/b1aeccf63578de3e83c72879f61ce265f45cbbbe.png" data-download-href="https://discuss.pytorch.org/uploads/default/b1aeccf63578de3e83c72879f61ce265f45cbbbe" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/1/b1aeccf63578de3e83c72879f61ce265f45cbbbe_2_690x42.png" alt="image" data-base62-sha1="plR1ik6nwP0M3UXFUhX7FRGvnYi" width="690" height="42" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/1/b1aeccf63578de3e83c72879f61ce265f45cbbbe_2_690x42.png, https://discuss.pytorch.org/uploads/default/original/3X/b/1/b1aeccf63578de3e83c72879f61ce265f45cbbbe.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/b/1/b1aeccf63578de3e83c72879f61ce265f45cbbbe.png 2x" data-dominant-color="E9E9E9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1026×63 28.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<pre><code class="lang-auto">class myLoss(torch.nn.Module):

def __init__(self, pos_weight=1):
    super().__init__()
    self.pos_weight = pos_weight

def forward(self, input, target):
    epsilon = 10 ** -44
    my_bce_loss = -1 * (self.pos_weight * target * F.logsigmoid(input + epsilon)
                        + (1 - target) * log(1 - sigmoid(input) + epsilon))
    add_loss = (target - 0.5) ** 2 * 4
    mean_loss = (my_bce_loss * add_loss).mean()
    return mean_loss
</code></pre>
<p><code>epsilon</code>  was chosen so the log will be bounded to -100, as suggested in <a href="https://pytorch.org/docs/stable/nn.html#bceloss" rel="noopener nofollow ugc">BCE loss</a>.</p>
<p>However I’m still getting NaN errors:</p>
<pre><code class="lang-auto">Function 'LogBackward' returned nan values in its 0th output.
</code></pre>
<p>or</p>
<pre><code class="lang-auto">Function 'SigmoidBackward' returned nan values in its 0th output.
</code></pre>
<p>Any suggestions how can I correct my loss function? maybe by somehow inherit and modify  <code>forward</code>  function?</p>

albanD: <p>Hi,</p>
<p>If the goal is that these samples are always properly classified, why do you use them in the model at all?<br>
Why not filter them out before the forward based on the target? This way you won’t do useless computations in the forward/backward and will be able to use the vanilla loss function.</p>

shlomish3: <p>Hi <a class="mention" href="/u/alband">@albanD</a>, thank you for replying.<br>
I didn’t mention that, but I’m using a Temporal Convolutional Network since I’m looking for temporal causality between samples, then I shouldn’t remove some of my inputs.</p>
<p>In the meanwhile apparently I solved  my problem in a similar manner to what you proposed:<br>
I remove those ‘always true’ samples after doing <code>input=model(x)</code> based on target locations, and then I use the formal BCEWithLogitsLoss. I guess the formal implementation is more robust.</p>