Cross-entropy with one-hot targets

Dawid_S: <p>I’d like to use the cross-entropy loss function that can take one-hot encoded values as the target.</p>
<pre><code class="lang-auto"># Fake NN output
out = torch.FloatTensor([[0.05, 0.9, 0.05], [0.05, 0.05, 0.9], [0.9, 0.05, 0.05]])
out = torch.autograd.Variable(out)

# Categorical targets
y = torch.LongTensor([1, 2, 0])
y = torch.autograd.Variable(y)

# One-hot encoded targets
y1 = torch.FloatTensor([[0, 1, 0], [0, 0, 1], [1, 0, 0]])
y1 = torch.autograd.Variable(y1)

# Calculating the loss
loss_val = nn.CrossEntropyLoss()(out, y)
loss_val1 = nn.BCEWithLogitsLoss()(out, y1)

print(loss_val)
print(loss_val1)
</code></pre>
<blockquote>
<p>Variable containing:<br>
0.6178<br>
[torch.FloatTensor of size 1]</p>
<p>Variable containing:<br>
0.5927<br>
[torch.FloatTensor of size 1]</p>
</blockquote>
<p>The output of the two functions is not the same.</p>
<p>How to implement the cross-entropy to get to same output passing one-hot encoded targets?</p>

richard: <aside class="quote no-group" data-username="Dawid_S" data-post="1" data-topic="13580">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/dawid_s/48/15829_2.png" class="avatar"> Dawid_S:</div>
<blockquote>
<p>y1 = torch.FloatTensor([[0, 1, 0], [0, 0, 1], [1, 0, 0]])</p>
</blockquote>
</aside>
<p>Try:</p>
<pre><code class="lang-auto">_, targets = y1.max(dim=0)
nn.CrossEntropyLoss()(out, Variable(targets))
</code></pre>

Dawid_S: <p>That’s not what I mean. I need to pass one-hot vector, because later I want to use smoothed values as targets<br>
(example [0.1, 0.1, 0.8]).  max() won’t help here.</p>
<p>The first thing I want to achieve is to get the same results using CrossEntropyLoss() and some loss that takes one-hot encoded values without smoothing</p>

richard: <p><a href="http://pytorch.org/docs/master/nn.html?highlight=crossentropyloss#torch.nn.CrossEntropyLoss">nn.CrossEntropyLoss</a> doesn’t take a one-hot vector, it takes class values. You can create a new function that wraps <code>nn.CrossEntropyLoss</code>, in the following manner:</p>
<pre><code class="lang-auto">def cross_entropy_one_hot(input, target):
    _, labels = target.max(dim=0)
    return nn.CrossEntropyLoss()(input, labels)
</code></pre>
<p>Also I’m not sure I’m understanding what you want. <a href="http://pytorch.org/docs/master/nn.html?highlight=bceloss#bcewithlogitsloss">nn.BCELossWithLogits</a> and nn.CrossEntropyLoss are different in the docs; I’m not sure in what situation you would expect the same loss from them.</p>

Dawid_S: <p>Sorry for not being clear enough.</p>
<p>I want to use soft targets with Cross Entropy. Soft target is one-hot-like vector, but continuous, not zero/one. The example target is in the example code below.</p>
<p>This is what I want:</p>
<pre><code class="lang-auto">def cross_entropy(input, target, size_average=True):
    """ Cross entropy that accepts soft targets
    Args:
         pred: predictions for neural network
         targets: targets, can be soft
         size_average: if false, sum is returned instead of mean

    Examples::

        input = torch.FloatTensor([[1.1, 2.8, 1.3], [1.1, 2.1, 4.8]])
        input = torch.autograd.Variable(out, requires_grad=True)

        target = torch.FloatTensor([[0.05, 0.9, 0.05], [0.05, 0.05, 0.9]])
        target = torch.autograd.Variable(y1)
        loss = cross_entropy(input, target)
        loss.backward()
    """
    logsoftmax = nn.LogSoftmax()
    if size_average:
        return torch.mean(torch.sum(-target * logsoftmax(input), dim=1))
    else:
        return torch.sum(torch.sum(-target * logsoftmax(input), dim=1))

</code></pre>
<p>i’m wondering if I can use any PyTorch function to pass soft targets to it, like in the example above. I was trying with BCELoss.</p>

richard: <p>Ah I see. Thank you for your clarification.</p>
<p><code>BCELoss</code> doesn’t quite do what you want it to do, because it has that extra term on the right (and I presume you only want the term on the left?)<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/c/c98570d1854f9fafa103a4ea9b6b9e6db3a22838.png" data-download-href="https://discuss.pytorch.org/uploads/default/c98570d1854f9fafa103a4ea9b6b9e6db3a22838" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/2X/c/c98570d1854f9fafa103a4ea9b6b9e6db3a22838.png" alt="image" data-base62-sha1="sKJRKuUA6NH2coH6RTKcH9FO6es" width="690" height="87" data-dominant-color="F4F4F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">754×96 6.29 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>There’s no built in PyTorch function to do this right now, but you can use the <code>cross_entropy</code> function you defined and autograd will work with it. Are you finding that this function is too slow as it is?</p>

Dawid_S: <p>You’re right. I’ve heard however a few times (like here<br>
<a href="https://discuss.pytorch.org/t/labels-smoothing-and-categorical-loss-functions-alternatives/11339">Labels smoothing and categorical loss functions - alternatives?</a>), that one can use BCELoss for soft targets somehow and I was trying to figure it out.</p>
<p>The function I defined above is twice faster than CrossEntropyFunction on cpu. It’s not a real reason but somehow I’m not sure I can trust it</p>

richard: <pre><code class="lang-auto"># Calculating the loss
loss_val = nn.CrossEntropyLoss()(out, y)         # (1)
loss_val1 = nn.BCEWithLogitsLoss()(out, y1)   # (2)
</code></pre>
<p>If you’re referring to these, you definitely shouldn’t trust (1) because it doesn’t give you the behavior you want. (2) works, but has an extra term that may or may not affect your training.</p>

evanthebouncy: <p>best to implement your own.</p>
<p>cross entropy loss is something like this I think . . .</p>
<p>[0.1, 0.2, 0.7] (prediction) ------------------ [1.0, 0.0, 0.0] (target)</p>
<p>what you want is - (1.0 * log(0.1) + 0.0 * log(0.2) + 0.0 * log(0.7)) this is the cross entropy loss</p>
<p>so to translate that into code, you have prediction (a vector of length k) and target (a vector of length k, not nessesarily 1 hot)</p>
<p>what you would do would be something like -1 * sum(log(prediction) * target)</p>
<p>so this is what I have in my own code, hopefully it’s helpful</p>
<blockquote>
<p>48 # simple cross entropy cost (might be numerically unstable if pred has 0)<br>
49 def xentropy_cost(x_target, x_pred):<br>
50     assert x_target.size() == x_pred.size(), <br>
51         "size fail ! "+str(x_target.size()) + " " + str(x_pred.size())<br>
52     logged_x_pred = torch.log(x_pred)<br>
53     cost_value = -torch.sum(x_target * logged_x_pred)<br>
54     return cost_value</p>
</blockquote>

Zhang_Chi: <p>hello.  if the target label, i.e. y_n is the one-hot label, should it be the same with crossentropy loss? but why they result in different results in the examples?</p>

jiahuei: <p><code>nn.CrossEntropyLoss</code> computes softmax cross entropy.<br>
<code>nn.BCEWithLogitsLoss</code> computes sigmoid cross entropy.</p>
<p>That’s why they are different.</p>

jinhyun-so: <p>I would like to implement exactly same function as you, Cross Entropy which takes soft target one-hot-like vector as an input.</p>
<p>I tried to train CNN with your cross_entropy function, but it seems that backward propagation of this function does not work properly.</p>
<p>Should I implement further this function for the backward propagation?<br>
If yes, please provide me some guide.</p>