Sum of cross entropy over all classes

username: moh_sen
cooked: <p>I want to compute sum of cross entropy over all classes for each prediction, where the input is batch (size n), and the output is batch (size n).</p>
<p><img src="https://i.stack.imgur.com/WW4e9.gif" alt width="557" height="56"></p>
<p>The simplest way is for loop (for 1000 classes):</p>
<pre><code class="lang-auto">def sum_of_CE_lost(input):
  CE = torch.nn.CrossEntropyLoss()
  L = 0
  for x in range(1000):
    L = L + CE(logit_unknown,tensor( np.ones( input.shape[0] ) *x , dtype=torch.int64 ).cuda() )
  loss = L.sum()
  return loss
</code></pre>
<p>However, it is very slow. What is a better way? How can we parallelized it for GPU (CUDA)?</p>
<p>input is prediction of imagenet_resnet50 on bathc, i.e., shape of input is [batch_size, number-of_classes]</p>

username: KFrank
cooked: <p>Hi Mohsen!</p>
<aside class="quote no-group quote-modified" data-username="moh_sen" data-post="1" data-topic="70516" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/ea666f/48.png" class="avatar"> moh_sen:</div>
<blockquote>
<p>I want to compute sum of cross entropy over all classes for each prediction</p>
<pre><code class="lang-auto">def sum_of_CE_lost(input):
    L = 0
    for c in range(1000):
        L = L + torch.nn.CrossEntropyLoss(input, c)
    return L
</code></pre>
<p>However, it is very slow. What is a better way? How can we parallelized it for GPU (CUDA)?</p>
</blockquote>
</aside>
<p>Yes, this is straightforward, based on the definition of <code>cross_entropy</code>.</p>
<p>Let <code>input</code> be a tensor of shape <code>(nBatch, nClass)</code>.  Then (mentally<br>
ignoring the minor errors in your code such as using the class version<br>
of <code>CrossEntropyLoss</code> as if it were a function and <code>c</code> not being a tensor)<br>
the following will match the result of the code you posted:</p>
<pre data-code-wrap="python"><code class="lang-python">-torch.nn.functional.log_softmax (input).sum() / input.shape[0]
</code></pre>
<p>(We divide by <code>input.shape[0]</code> because <code>cross_entropy()</code> takes, by<br>
default the mean across the batch dimension.)</p>
<p>Because this expression uses pytorch tensor functions, you will<br>
automatically get the benefit of pytorch’s gpu support (if you move<br>
your tensors to the gpu) (as well as autograd, if you care).</p>
<p>(As an aside, though, it’s not really clear to me that your calculation<br>
makes a lot of sense.  But you can compute it with pytorch tensors<br>
on the gpu without explicit loops.)</p>
<p>Good luck!</p>
<p>K. Frank</p>

username: moh_sen
cooked: <aside class="quote no-group" data-username="KFrank" data-post="2" data-topic="70516">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>Let <code>input</code> be a tensor of shape <code>(nBatch, nClass)</code> .</p>
</blockquote>
</aside>
<p>The input is not shape of ‘(n batch, nClass)’. The input is in shape (‘nBatch, mLogit’). i.e., each row has 1000 elements, corresponding to each classes.</p>
<p>I want to get sum over all classes.</p>

username: KFrank
cooked: <p>Hello Mohsen!</p>
<aside class="quote no-group quote-modified" data-username="moh_sen" data-post="3" data-topic="70516" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/ea666f/48.png" class="avatar"> moh_sen:</div>
<blockquote>
<aside class="quote no-group" data-username="KFrank" data-post="2" data-topic="70516">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>Let <code>input</code> be a tensor of shape <code>(nBatch, nClass)</code> .</p>
</blockquote>
</aside>
<p>The input is not shape of ‘(n batch, nClass)’. The input is in shape (‘nBatch, mLogit’). i.e., each row has 1000 elements, corresponding to each classes.</p>
<p>I want to get sum over all classes.</p>
</blockquote>
</aside>
<p>Whether we call this number, 1000, <code>nClass</code> or <code>mLogit</code> is immaterial.<br>
I maintain that my expression reproduces the result of your (corrected)<br>
pseudo-code.  Have you tried it?</p>
<p>If you come to the contrary conclusion, please post a complete,<br>
runnable sample script that illustrates your contrary result.  (You<br>
can generate random sample data for you <code>input</code> tensor, but please<br>
trim down the number 1000 to something more manageable like<br>
5 or 10 to make everyone’s life easier.)</p>
<p>Also, to get our vocabulary straight, what do you mean by “class”<br>
when you say “sum over all classes?”  How many classes do you<br>
have in your example?  Should I understand the value of the quantity<br>
“C” in the image of the equation you posted to be your number of<br>
classes (and equal to 1000 in your example)?</p>
<p>Best.</p>
<p>K. Frank</p>