Implementation of Binary cross Entropy?

sgaur: <p>Hi All,</p>
<p>I want to write a code for label smoothing using <strong>BCEWithLogitsLoss</strong> .</p>
<p>Q1) Is BCEWithLogitLoss =  BCELoss + sigmoid() ?<br>
Q2) While checking the pytorch github docs I found following code in which sigmoid implementation is not there maybe I am looking at wrong Documents ?</p>
<p>Can someone tell me where they write proper BCEWithLogitLoss Code. ??</p>
<pre><code class="lang-auto">class BCEWithLogitsLoss(_Loss):

    def __init__(self, weight: Optional[Tensor] = None, size_average=None, reduce=None, reduction: str = 'mean',
                 pos_weight: Optional[Tensor] = None) -&gt; None:
        super(BCEWithLogitsLoss, self).__init__(size_average, reduce, reduction)
        self.register_buffer('weight', weight)
        self.register_buffer('pos_weight', pos_weight)

    def forward(self, input: Tensor, target: Tensor) -&gt; Tensor:
        return F.binary_cross_entropy_with_logits(input, target,
                                                  self.weight,
                                                  pos_weight=self.pos_weight)



def binary_cross_entropy_with_logits(input, target, weight=None, size_average=None,
                                     reduce=None, reduction='mean', pos_weight=None):
    # type: (Tensor, Tensor, Optional[Tensor], Optional[bool], Optional[bool], str, Optional[Tensor]) -&gt; Tensor
    r"""Function that measures Binary Cross Entropy between target and output
    logits.

    See :class:`~torch.nn.BCEWithLogitsLoss` for details.

    Args:
        input: Tensor of arbitrary shape
        target: Tensor of the same shape as input
        weight (Tensor, optional): a manual rescaling weight
            if provided it's repeated to match input tensor shape
        size_average (bool, optional): Deprecated (see :attr:`reduction`). By default,
            the losses are averaged over each loss element in the batch. Note that for
            some losses, there multiple elements per sample. If the field :attr:`size_average`
            is set to ``False``, the losses are instead summed for each minibatch. Ignored
            when reduce is ``False``. Default: ``True``
        reduce (bool, optional): Deprecated (see :attr:`reduction`). By default, the
            losses are averaged or summed over observations for each minibatch depending
            on :attr:`size_average`. When :attr:`reduce` is ``False``, returns a loss per
            batch element instead and ignores :attr:`size_average`. Default: ``True``
        reduction (string, optional): Specifies the reduction to apply to the output:
            ``'none'`` | ``'mean'`` | ``'sum'``. ``'none'``: no reduction will be applied,
            ``'mean'``: the sum of the output will be divided by the number of
            elements in the output, ``'sum'``: the output will be summed. Note: :attr:`size_average`
            and :attr:`reduce` are in the process of being deprecated, and in the meantime,
            specifying either of those two args will override :attr:`reduction`. Default: ``'mean'``
        pos_weight (Tensor, optional): a weight of positive examples.
                Must be a vector with length equal to the number of classes.

    Examples::

         &gt;&gt;&gt; input = torch.randn(3, requires_grad=True)
         &gt;&gt;&gt; target = torch.empty(3).random_(2)
         &gt;&gt;&gt; loss = F.binary_cross_entropy_with_logits(input, target)
         &gt;&gt;&gt; loss.backward()
    """
    if not torch.jit.is_scripting():
        tens_ops = (input, target)
        if any([type(t) is not Tensor for t in tens_ops]) and has_torch_function(tens_ops):
            return handle_torch_function(
                binary_cross_entropy_with_logits, tens_ops, input, target, weight=weight,
                size_average=size_average, reduce=reduce, reduction=reduction,
                pos_weight=pos_weight)
    if size_average is not None or reduce is not None:
        reduction_enum = _Reduction.legacy_get_enum(size_average, reduce)
    else:
        reduction_enum = _Reduction.get_enum(reduction)

    if not (target.size() == input.size()):
        raise ValueError("Target size ({}) must be the same as input size ({})".format(target.size(), input.size()))

    return torch.binary_cross_entropy_with_logits(input, target, weight, pos_weight, reduction_enum)
</code></pre>
<p>Thanks !!!</p>

pytorchtester0: <aside class="quote no-group" data-username="sgaur" data-post="1" data-topic="98715">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/3da27b/48.png" class="avatar"> sgaur:</div>
<blockquote>
<p><code>BCEWithLogitsLoss</code></p>
</blockquote>
</aside>
<p>Hello,</p>
<ol>
<li>Yes equivalent but less stable for BCELoss</li>
<li>The code of the BCEWithLogitsLoss Class can be found in <a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/modules/loss.py" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/blob/master/torch/nn/modules/loss.py</a><br>
You will find a call to</li>
</ol>
<pre><code class="lang-auto">    def forward(self, input: Tensor, target: Tensor) -&gt; Tensor:
        return F.binary_cross_entropy_with_logits(input, target,
                                                  self.weight,
                                                  pos_weight=self.pos_weight,
                                                  reduction=self.reduction)
</code></pre>
<p>The F oject is imported from functionnal.py here : <a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/functional.py" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/blob/master/torch/nn/functional.py</a></p>
<p>You will find the function called</p>
<pre><code class="lang-auto">def binary_cross_entropy_with_logits(input, target, weight=None, size_average=None,
                                     reduce=None, reduction='mean', pos_weight=None):
</code></pre>
<p>It calls the handle_torch_function in <a href="https://github.com/pytorch/pytorch/blob/master/torch/overrides.py" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/blob/master/torch/overrides.py</a><br>
You will find an entry of the function binary_cross_entropy_with_logits in the ret dictionnary wich contain every function that can be overriden in pytorch.<br>
This is the Python implementation of <strong>torch_function</strong><br>
More info in <a href="https://github.com/pytorch/pytorch/issues/24015" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/issues/24015</a></p>
<p>Then the code called is in the C++ File<br>
<a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/Loss.cpp" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/Loss.cpp</a></p>
<pre><code class="lang-auto">
Tensor binary_cross_entropy_with_logits(const Tensor&amp; input, const Tensor&amp; target, const Tensor&amp; weight, const Tensor&amp; pos_weight, int64_t reduction) {
    Tensor loss;
    auto max_val = (-input).clamp_min_(0);
    if (pos_weight.defined()) {
        // pos_weight need to be broadcasted, thus mul(target) is not inplace.
        auto log_weight = (pos_weight - 1).mul(target).add_(1);
        loss = (1 - target).mul_(input).add_(log_weight.mul_(((-max_val).exp_().add_((-input - max_val).exp_())).log_().add_(max_val)));
    } else {
        loss = (1 - target).mul_(input).add_(max_val).add_((-max_val).exp_().add_((-input -max_val).exp_()).log_());
    }

    if (weight.defined()) {
        loss.mul_(weight);
    }

    return apply_loss_reduction(loss, reduction);
}
</code></pre>
<p>take advantage of the log-sum-exp trick for numerical stability<br>
<a href="https://en.wikipedia.org/wiki/LogSumExp" rel="noopener nofollow ugc">https://en.wikipedia.org/wiki/LogSumExp</a></p>
<p>You can compare it with the  BCELoss in binary_cross_entropy_out_cpu( function<br>
it is less stable it apply L = -w (y ln(x) + (1-y) ln(1-x))</p>
<pre><code class="lang-auto">
                // Binary cross entropy tensor is defined by the equation:
                // L = -w (y ln(x) + (1-y) ln(1-x))
                return (target_val - scalar_t(1))
                    * std::max(scalar_t(std::log(scalar_t(1) - input_val)), scalar_t(-100))
                    - target_val * std::max(scalar_t(std::log(input_val)), scalar_t(-100));
</code></pre>

KFrank: <p>Hello Surya and Pytorchtester!</p>
<aside class="quote no-group" data-username="sgaur" data-post="1" data-topic="98715" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/3da27b/48.png" class="avatar"> sgaur:</div>
<blockquote>
<p>Q1) Is BCEWithLogitLoss =  BCELoss + sigmoid() ?</p>
</blockquote>
</aside>
<p>To clarify a bit:</p>
<p><em>Mathematically,</em> <code>BCEWithLogitsLoss</code> is <code>sigmoid()</code> followed by<br>
<code>BCELoss</code>.  But numerically they are different, with <code>BCELoss</code><br>
numerically less stable.</p>
<aside class="quote no-group">
<blockquote>
<p>Q2) While checking the pytorch github docs I found following code in which sigmoid implementation is not there</p>
</blockquote>
</aside>
<p>Elaborating on the above, <code>sigmoid()</code> is not there, because it is<br>
not <em>explicitly</em> part of <code>BCEWithLogitsLoss</code>.  It is hiding in the<br>
<code>log (sigmoid())</code> version of the “log-sum-exp trick,” in this line<br>
from the c++ code that Pytorchtester posted:</p>
<pre data-code-wrap="C++"><code class="lang-C++">loss = (1 - target).mul_(input).add_(max_val).add_((-max_val).exp_().add_((-input -max_val).exp_()).log_());
</code></pre>
<p>Best.</p>
<p>K. Frank</p>