Cross entropy with softmax (4 outputs) with target being multichannel continuous values

YASJAY: <p>Hi,</p>
<p>As understood from the topic, I want to implement cross entropy with softmax on logits with target of format [batchsize,C,H,W] where values are in the range [0,1). Values across axes 1 does not sum to 1.</p>
<p><code>torch.nn.functional.</code> <code>cross_entropy</code> does not allow 4D targets.</p>
<p>How do I do this?</p>

Henry_Chibueze: <p>U cannot do this with crossEntropy</p>

Henry_Chibueze: <p>The crossEntropy is used for classification problems where the length of output of ur network is the number of classes u have<br>
So if u pass an image through the NN it outputs a length of C values where C=classes. So the shape of this output is (N, C) N= batch size, C is number of classes what crossentropy does it to then find the probability score of the outputs with softmax and then compute the negative log likelyhood between the scored output and the Target.</p>
<p>So in other words crossEntropy only take values with shape(N, C).</p>

YASJAY: <p><a class="mention" href="/u/henry_chibueze">@Henry_Chibueze</a>, The  tensorflow version of cross entropy:  tf.nn.softmax_cross_entropy_with_logits allows me to pass 4D target with continuous values by passing axes argument representing class. I am trying to find pytorch version of this.</p>

KFrank: <p>Hi Srinath!</p>
<aside class="quote no-group" data-username="YASJAY" data-post="1" data-topic="95690" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/yasjay/48/12858_2.png" class="avatar"> YASJAY:</div>
<blockquote>
<p>I want to implement cross entropy with softmax on logits with target of format [batchsize,C,H,W] where values are in the range [0,1).</p>
</blockquote>
</aside>
<p>Pytorch’s <code>cross_entropy()</code> takes targets that are integer class labels.<br>
You can’t use so-called <em>soft labels</em> that are probabilities.</p>
<p>However, you can easily write your own version that does take soft<br>
labels.  See this thread:</p>
<aside class="quote" data-post="1" data-topic="69501">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/soulslicer/48/9454_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/soft-cross-entropy-loss-tf-has-it-does-pytorch-have-it/69501">Soft Cross Entropy Loss (TF has it does Pytorch have it)</a> 
  </div>
  <blockquote>
    <a href="https://www.tensorflow.org/api_docs/python/tf/nn/softmax_cross_entropy_with_logits" rel="noopener nofollow ugc">softmax_cross_entropy_with_logits</a> 
TF supports not needing to have hard labels for cross entropy loss: 
logits = [[4.0, 2.0, 1.0], [0.0, 5.0, 1.0]] labels = [[1.0, 0.0, 0.0], [0.0, 0.8, 0.2]] tf.nn.softmax_cross_entropy_with_logits(labels=labels, logits=logits) 
Can we do the same thing in Pytorch?
  </blockquote>
</aside>

<p>Best.</p>
<p>K. Frank</p>

YASJAY: <p>Hi <a class="mention" href="/u/kfrank">@KFrank</a>, my target values do not sum to 1, that is, they are not soft-labels. Values along class axis are normalized gaussian pulses with values between 0 to 1.</p>

Henry_Chibueze: <p>Well I haven’t used tensorflow b4 and I haven’t seen what u r looking for on pytorch</p>
<p>Maybe u should try flattening the data to be of shape (N, C) when parsing to the Cross Entropy loss function</p>

KFrank: <p>Hi Srinath!</p>
<aside class="quote no-group" data-username="YASJAY" data-post="6" data-topic="95690" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/yasjay/48/12858_2.png" class="avatar"> YASJAY:</div>
<blockquote>
<p>my target values do not sum to 1, that is, they are not soft-labels.</p>
</blockquote>
</aside>
<p>That’s okay.  You can still use the “soft-label cross-entropy” I linked to<br>
above – the math still goes through.  It won’t – precisely speaking –<br>
be a true cross-entropy (which compares two true probability<br>
distributions), but it will give you a reasonable loss function.</p>
<p>(And, to be sure, if you pass your targets that don’t sum to one to<br>
<code>tf.nn.softmax_cross_entropy_with_logits()</code>, nothing changes,<br>
and you will still have a the same issue.)</p>
<p>Best.</p>
<p>K. Frank</p>