BCELoss are unsafe to autocast

talhaanwarch: <p>I am trying to do autocast but I am getting the following error</p>
<pre><code class="lang-auto">RuntimeError: torch.nn.functional.binary_cross_entropy and torch.nn.BCELoss are unsafe to autocast.
Many models use a sigmoid layer right before the binary cross entropy layer.
In this case, combine the two layers using torch.nn.functional.binary_cross_entropy_with_logits
or torch.nn.BCEWithLogitsLoss.  binary_cross_entropy_with_logits and BCEWithLogits are
safe to autocast.
</code></pre>
<p>This is  saying me to use <code>BCEWithLogitsLoss</code> instead of <code>BCELoss</code>, but my last layer contain <code>tanh</code> instead of <code>sigmoid</code>. How can I use <code>BCEWithLogitsLoss</code> though I don’t want <code>sigmoid</code> layer.<br>
Thanks</p>

ptrblck: <p><code>nn.BCELoss</code> expects probabilities in the range <code>[0, 1]</code> as inputs and will yield an error otherwise:</p>
<pre><code class="lang-python">criterion = nn.BCELoss()
x = torch.randn(10, 1)
criterion(torch.tanh(x), torch.randint(0, 2, (10, 1)).float())
&gt; RuntimeError: all elements of input should be between 0 and 1
</code></pre>
<p>so you would have to use an activation function, which creates these values (such as <code>sigmoid</code>) and could thus switch to <code>nn.BCEWithLogitsLoss</code>.</p>

Mona_Jalal: <p>Hi Piotr, using sigmoid and then BCEWithLogitsLoss doesn’t work for me. Do you know what could be the problem? <a href="https://discuss.pytorch.org/t/predicted-labels-stuck-at-1-for-test-set-where-class-0-is-20-of-data/142751/3" class="inline-onebox">Predicted labels stuck at 1 for test set where class 0 is 20% of data - #3 by Mona_Jalal</a></p>
<p>As you see in that post, I have converted the values to [0, 1] using sigmoid: <code>sig:  tensor([0.4937, 0.3869, 0.5705, 0.4727, 0.5873, 0.5160, 0.4642, 0.3229, 0.4283,         0.5563, 0.4485, 0.4954, 0.5997, 0.4649, 0.4181, 0.3966],</code></p>

ptrblck: <p>That’s expected, since <code>nn.BCEWithLogitsLoss</code> expects logits (remove the <code>sigmoid</code> activation) while <code>nn.BCELoss</code> expects probabilities (use <code>sigmoid</code> here).<br>
Generally using logits and <code>nn.BCEWithLogitsLoss</code> is the recommended approach as it has better numerical stability.</p>

Mona_Jalal: <p>do you mean:<br>
<code>        self.criterion = nn.BCEWithLogitsLoss(reduce=False)</code><br>
with:</p>
<pre><code class="lang-auto">stacked_X = torch.stack(X)
        out = self.transformer(stacked_X)
       
        with torch.autocast('cuda'):
            # https://discuss.pytorch.org/t/unclear-about-weighted-bce-loss/21486/2 
            labels = torch.tensor(labels)
            weight = torch.tensor([0.1, 0.9]) # how to decide on this weights?
            weight_ = weight[labels.data.view(-1).long()].view_as(labels)
 
            loss = self.criterion(out[:,1]-out[:,0], torch.tensor(labels).cuda())
</code></pre>
<p>Because I am still getting error:</p>
<pre><code class="lang-auto">return F.binary_cross_entropy_with_logits(input, target,
  File "/home/jalal/research/venv/dpcc/lib/python3.8/site-packages/torch/nn/functional.py", line 2982, in binary_cross_entropy_with_logits
    return torch.binary_cross_entropy_with_logits(input, target, weight, pos_weight, reduction_enum)
RuntimeError: result type Float can't be cast to the desired output type Long
</code></pre>
<p>could you please show how to use it in my context?</p>

ptrblck: <p>Could you post an executable code snippet using random tensors, which would reproduce this issue, so that I could debug it, please?</p>

Mona_Jalal: <p>Yeah, definitely. Thanks a lot for your time. Here it is:</p>
<pre><code class="lang-auto">import torch
out = torch.tensor([[-0.2422, -0.1971],
        [-0.4763,  0.2703],
        [-0.5091,  0.1275],
        [-0.4697,  0.3374],
        [-0.1803, -0.2947],
        [-0.6293,  0.1251],
        [-0.1562,  0.3778],
        [-0.5306,  0.1107],
        [-0.3032,  0.4657],
        [-0.8656,  0.2032],
        [-0.3078,  0.2820],
        [-0.4398,  0.1044],
        [-0.4838, -0.0568],
        [-0.7053,  0.4384],
        [-0.4647, -0.0030],
        [-0.4993,  0.2855]], device='cuda:0')
labels = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 0]
labels = torch.tensor(labels)


with torch.autocast('cuda'):
        criterion = torch.nn.BCEWithLogitsLoss(reduce=False)
        weight = torch.tensor([0.1, 0.9]) # how to decide on this weights?
        weight_ = weight[labels.data.view(-1).long()].view_as(labels)
        m = torch.nn.Sigmoid()
        print('out[:,1]-out[:,0]: ', out[:,1]-out[:,0])
        print('m(out[:,1]-out[:,0]): ', m(out[:,1]-out[:,0]))
        #loss = self.criterion(torch.cuda.LongTensor(m(out[:,1]-out[:,0])),        labels.cuda())
        #loss = criterion(out[:,1]-out[:,0], labels.cuda()) #I think this is wrong with a high probability
        loss = criterion(m(out[:,1]-out[:,0]), labels.cuda())

        print('loss: ', loss)
        print('weight_: ', weight_)
        loss_class_weighted = loss * weight_.cuda()
        loss_class_weighted = loss_class_weighted.mean()
</code></pre>
<p>Error/output is:</p>
<pre><code class="lang-auto">out[:,1]-out[:,0]:  tensor([ 0.0451,  0.7466,  0.6366,  0.8071, -0.1144,  0.7544,  0.5340,  0.6413,
         0.7689,  1.0688,  0.5898,  0.5442,  0.4270,  1.1437,  0.4617,  0.7848],
       device='cuda:0')
m(out[:,1]-out[:,0]):  tensor([0.5113, 0.6784, 0.6540, 0.6915, 0.4714, 0.6801, 0.6304, 0.6550, 0.6833,
        0.7444, 0.6433, 0.6328, 0.6052, 0.7584, 0.6134, 0.6867],
       device='cuda:0')
/home/jalal/research/venv/dpcc/lib/python3.8/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
  warnings.warn(warning.format(ret))
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Input In [16], in &lt;module&gt;
     28 print('m(out[:,1]-out[:,0]): ', m(out[:,1]-out[:,0]))
     29 #loss = self.criterion(torch.cuda.LongTensor(m(out[:,1]-out[:,0])), labels.cuda())
     30 #loss = criterion(out[:,1]-out[:,0], labels.cuda()) #I think this is wrong with a high probability
---&gt; 31 loss = criterion(m(out[:,1]-out[:,0]), torch.FloatTensor.cuda(labels))
     33 print('loss: ', loss)
     34 print('weight_: ', weight_)

File ~/research/venv/dpcc/lib/python3.8/site-packages/torch/nn/modules/module.py:1102, in Module._call_impl(self, *input, **kwargs)
   1098 # If we don't have any hooks, we want to skip the rest of the logic in
   1099 # this function, and just call forward.
   1100 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1101         or _global_forward_hooks or _global_forward_pre_hooks):
-&gt; 1102     return forward_call(*input, **kwargs)
   1103 # Do not call functions when jit is used
   1104 full_backward_hooks, non_full_backward_hooks = [], []

File ~/research/venv/dpcc/lib/python3.8/site-packages/torch/nn/modules/loss.py:704, in BCEWithLogitsLoss.forward(self, input, target)
    703 def forward(self, input: Tensor, target: Tensor) -&gt; Tensor:
--&gt; 704     return F.binary_cross_entropy_with_logits(input, target,
    705                                               self.weight,
    706                                               pos_weight=self.pos_weight,
    707                                               reduction=self.reduction)

File ~/research/venv/dpcc/lib/python3.8/site-packages/torch/nn/functional.py:2982, in binary_cross_entropy_with_logits(input, target, weight, size_average, reduce, reduction, pos_weight)
   2979 if not (target.size() == input.size()):
   2980     raise ValueError("Target size ({}) must be the same as input size ({})".format(target.size(), input.size()))
-&gt; 2982 return torch.binary_cross_entropy_with_logits(input, target, weight, pos_weight, reduction_enum)

RuntimeError: result type Float can't be cast to the desired output type Long
</code></pre>

ptrblck: <p>Thanks!<br>
<code>nn.BCEWithLogitsLoss</code> expects <code>FloatTensor</code> targets, so use:</p>
<pre><code class="lang-python">labels = torch.tensor(labels, dtype=torch.float32)
</code></pre>
<p>and it should work.</p>

Mona_Jalal: <aside class="quote no-group" data-username="ptrblck" data-post="8" data-topic="110407">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p><code>labels = torch.tensor(labels, dtype=torch.float32)</code></p>
</blockquote>
</aside>
<p>Thanks a lot. This resolved the issue.</p>