Equivalent of TensorFlow's Sigmoid Cross Entropy With Logits in Pytorch

varunagrawal: <p>I am trying to find the equivalent of <code>sigmoid_cross_entropy_with_logits</code> loss in Pytorch but the closest thing I can find is the <code>MultiLabelSoftMarginLoss</code>.</p>
<p>Can someone direct me to the equivalent loss? If it doesn’t exist, that information would be useful as well so I can submit a suitable PR.</p>

Chun_Li: <p>I think it’s class torch.nn.CrossEntropyLoss.</p>

varunagrawal: <p>That doesn’t seem to be the case. As per the docs CrossEntropyLoss only takes a single class index. Please correct me if I am wrong.</p>

jekbradbury: <p>You’re looking for <code>KLDivLoss</code>, which takes two log-probability inputs. If you have logits, you will need to apply <code>F.log_softmax</code> first.</p>

varunagrawal: <p>The objective function formulation is different from the Cross Entropy formulation given in TensorFlow. I don’t think this is the correct loss.</p>

AjayTalati: <p>Maybe the answer to this stackoverflow question is helpful,</p>
<aside class="onebox stackexchange">
  <header class="source">
      <a href="https://stackoverflow.com/questions/34240703/what-is-logits-softmax-and-softmax-cross-entropy-with-logits" target="_blank" rel="nofollow noopener">stackoverflow.com</a>
  </header>
  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/4341948/shubhashis" target="_blank" rel="nofollow noopener">
    <img alt="Shubhashis" src="https://i.stack.imgur.com/jpmcC.png?s=128&amp;g=1" class="thumbnail onebox-avatar" width="128" height="128">
  </a>
<h4>
  <a href="https://stackoverflow.com/questions/34240703/what-is-logits-softmax-and-softmax-cross-entropy-with-logits" target="_blank" rel="nofollow noopener">What is logits, softmax and softmax_cross_entropy_with_logits?</a>
</h4>

<div class="tags">
  <strong>python, machine-learning, tensorflow</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://stackoverflow.com/users/4341948/shubhashis" target="_blank" rel="nofollow noopener">
    Shubhashis
  </a>
  on <a href="https://stackoverflow.com/questions/34240703/what-is-logits-softmax-and-softmax-cross-entropy-with-logits" target="_blank" rel="nofollow noopener">02:03PM - 12 Dec 15 UTC</a>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>In mathematical terms, what exactly do you want to do? That might be easier for people to help you with, rather than trying to port over a TF function?</p>
<p>If you want to do multi-label classification, so do I, but I haven’t figured out yet how to do it in PyTorch? So I’m also interested in your question <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=9" title=":smile:" class="emoji" alt=":smile:"></p>
<p>Best,</p>
<p>Ajay</p>

varunagrawal: <p>From the implementation details, it would seem that the <code>MultiLabelSoftMarginLoss</code> is indeed the equivalent of the <code>sigmoid_cross_entropy_with_logits</code> loss. Closing this!</p>

AjayTalati: <p>Hi <a class="mention" href="/u/varunagrawal">@varunagrawal</a>,</p>
<p>did you get <code>MultiLabelSoftMarginLoss</code> to work on a multi-label classification test problem?</p>
<p>I tried applying it to a multi-label MNIST test, (each image is label by it’s original class, and the class-1), but it didn’t work?</p>

Misha_E: <p>I believe you are talking about BCELoss<br>
<a href="http://pytorch.org/docs/nn.html#torch.nn.BCELoss" class="onebox" target="_blank" rel="nofollow noopener">http://pytorch.org/docs/nn.html#torch.nn.BCELoss</a><br>
or <a href="http://pytorch.org/docs/nn.html#binary-cross-entropy" rel="nofollow noopener">http://pytorch.org/docs/nn.html#binary-cross-entropy</a><br>
but you’d have to apply sigmoid activation yourself before that</p>

mratsim: <p><a class="mention" href="/u/ajaytalati">@AjayTalati</a> I managed to use BCELoss, binary_crossentropy and MultiLabelSoftMarginLoss on a MultiLabel problem</p>
<p>Here is the basic code</p>
<pre><code class="lang-python">def train(epoch):
    model.train()
    for batch_idx, (data, target) in enumerate(train_loader):
        # data, target = data.cuda(async=True), target.cuda(async=True) # On GPU
        data, target = Variable(data), Variable(target)
        optimizer.zero_grad()
        output = model(data)
        loss = F.binary_cross_entropy(output, target)
        loss.backward()
        optimizer.step()
        if batch_idx % 10 == 0:
            print('Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}'.format(
                epoch, batch_idx * len(data), len(train_loader.dataset),
                100. * batch_idx / len(train_loader), loss.data[0]))
</code></pre>
<p>And the source is <a href="https://www.kaggle.com/mratsim/planet-understanding-the-amazon-from-space/starting-kit-for-pytorch-deep-learning" rel="nofollow noopener">here</a>.</p>
<p>For BCELoss you can use <code>criterion = BCELoss()</code> and then <code>loss = criterion(output, target)</code> but as <a class="mention" href="/u/misha_e">@Misha_E</a> said, the NN must return a sigmoid activation.</p>

AjayTalati: <p>Hi Mamy, <a class="mention" href="/u/mratsim">@mratsim</a></p>
<p>thanks a lot for posting your code <img src="//discuss.pytorch.org/images/emoji/apple/smile.png?v=5" title=":smile:" class="emoji" alt=":smile:"></p>
<p>All the best,</p>
<p>Aj</p>

mattrobin: <p>Just for anyone else who finds this from Google (as I did), <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.BCEWithLogitsLoss" rel="nofollow noopener"><code>BCEWithLogitsLoss</code></a> now does the equivalent of <code>sigmoid_cross_entropy_with_logits</code> from TensorFlow. It is a numerically stable sigmoid followed by a cross entropy combination.</p>

moscow25: <p>Worth noting that <code>KLDivLoss</code> still needs to run with <code>reduction='batchmean'</code> – to get the “soft cross_entropy” behavior that people are asking. Surprised this isn’t a more clearly documented…</p>