Why cross entropy loss has a separate input format for pixel level classification?

username: ritwik.m07
cooked: <p>I was working with cross entropy loss for a sequence labeling problem. Just like pixel-level classification, sequence labeling predicts the class for every token in the sequence.</p>
<p>I read in <a href="https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html#torch.nn.CrossEntropyLoss" rel="noopener nofollow ugc">the documentation of cross entropy loss</a> that there is a separate format for such fine-grained classification.</p>
<p>Let us stick to class probabilities in the ground-truth, NOT class indices.</p>
<p>The standard input/output format is:</p>
<p>N x C</p>
<p>Whereas, input/output format for multidimensional data or fine-grained classification is:</p>
<p>N x C x d_1 x d_2 x … x d_k</p>
<p>Where N is Batch size or number of training examples, C is number of classes, and d_1 … d_k are the dimensions of the input (in case of image k=2, in case of text sequence k=1).</p>
<p>But I am thinking why is it even needed? Because we can modify our problem to the standard format. How? Let us take pixel level classification only. We can simply assume each pixel is our training example, instead of an image. That’s it.</p>
<p>I have tested, this approach works.</p>
<p>Here is a working code:</p>
<p>Let us take classes (C) =3, image size (d1 x d2) = 3x3, and number of images (N) = 3</p>
<p>Let us try with the separate input format.</p>
<pre><code class="lang-auto">import torch
loss = torch.nn.CrossEntropyLoss()
predictions = torch.tensor([ [
                              [[1.,10.,1.], [1.,1.,10.], [1.,1.,1.]], 
                              [[10.,1.,10.], [10.,1.,1.], [1.,10.,1.]], 
                              [[1.,1. ,1.], [1.,10.,1.], [10.,1.,10.]]
                              ] , 
                            [
                             [[1.,1.,1.], [1.,10.,1.], [10.,1.,1.]], 
                             [[10.,10,1], [1,1,1], [1,1,10]],
                             [[1.,1.,10.], [10.,1.,10.], [1.,10.,1.]]
                             ] ],requires_grad=True) # NxCx3x3 = 2x3x3x3
target = torch.tensor([ [
                              [[0.,1.,0.], [0.,0.,1.], [0,0,0]], 
                              [[1.,0.,1.], [1.,0,0], [0.,1.,0.]], 
                              [[0,0,0], [0.,1.,0.], [1.,0.,1.]]
                              ] , 
                            [
                             [[0,0,0.], [0.,1.,0.], [1.,0,0.]], 
                             [[1.,1,0], [0,0,0], [0,0.,1]],
                             [[0,0.,1.], [1.,0.,1.], [0.,1.,0.]]
                             ] ]) # NxCx3x3 = 2x3x3x3
output = loss(predictions, target)
print(output)
</code></pre>
<p>Now, let us test the same thing by permuting the axes i.e. we are going to treat each pixel as an independent training example.</p>
<pre><code class="lang-auto">import torch
loss = torch.nn.CrossEntropyLoss()
predictions = torch.tensor([ [
                              [[1.,10.,1.], [1.,1.,10.], [1.,1.,1.]], 
                              [[10.,1.,10.], [10.,1.,1.], [1.,10.,1.]], 
                              [[1.,1. ,1.], [1.,10.,1.], [10.,1.,10.]]
                              ] , 
                            [
                             [[1.,1.,1.], [1.,10.,1.], [10.,1.,1.]], 
                             [[10.,10,1], [1,1,1], [1,1,10]],
                             [[1.,1.,10.], [10.,1.,10.], [1.,10.,1.]]
                             ] ],requires_grad=True) # NxCx3x3 = 2x3x3x3

predictions = predictions.permute(0,2,3,1).reshape(-1,3) # (-1,C)

target = torch.tensor([ [
                              [[0.,1.,0.], [0.,0.,1.], [0,0,0]], 
                              [[1.,0.,1.], [1.,0,0], [0.,1.,0.]], 
                              [[0,0,0], [0.,1.,0.], [1.,0.,1.]]
                              ] , 
                            [
                             [[0,0,0.], [0.,1.,0.], [1.,0,0.]], 
                             [[1.,1,0], [0,0,0], [0,0.,1]],
                             [[0,0.,1.], [1.,0.,1.], [0.,1.,0.]]
                             ] ]) # NxCx3x3 = 2x3x3x3

target = target.permute(0,2,3,1).reshape(-1,3) # (-1,C)
output = loss(predictions, target)
print(output)
</code></pre>
<p>Both gives the same value. Same thing can be done with class indices. Then why not ask users to permute their tensors first before passing them to the loss functions.</p>
<p>Or Is it just to make the life easy? Under the hood same procedure of permuting is happening. I couldn’t comprehend the source code for cross entropy (<a href="https://pytorch.org/docs/stable/_modules/torch/nn/functional.html#cross_entropy" rel="noopener nofollow ugc">it refers to some torch._C.nn.cross_entropy</a>), that is why I am not sure of this.</p>

username: ptrblck
cooked: <p>The main reason would be for convenience and to avoid code duplication, which could add errors to user code. The same applies for e.g. weighted losses, as you could simply multiply the unreduced loss with a weight tensor and reduce it afterwards. However, would you remember that the reduction would divide by the used weights or would you just call <code>mean()</code> on the weighted loss?<br>
A similar approach is also possible for e.g. grouped convolutions and one can permute the channel blocks to the batch dim, but this is also too error prone and it makes sense to provide these methods to the user.</p>

username: ritwik.m07
cooked: <p>so basically convenience to the user, isnt it?</p>

username: ptrblck
cooked: <p>No, not only convenience, but also avoiding code duplication and thus errors.</p>