Activation and loss function for multi dimensional one hot encoded output

pydisc: <p>I have a multi dimensional output model with the shape of <code>(B,C,T)</code> before the softmax layer. Its target is a row wise one hot encoded matrix with the same shape  of model prediction ie <code>(B,C,T)</code> . The trouble is PyTorch softmax method doesn’t working for row wise one hot encoded values. I wrote this sample code to show that the output value after the softmax layer is not row summed to one.</p>
<pre><code class="lang-auto">import torch

# shape (Batch, Class, Time Step)
target = torch.tensor([[[0,1,0,0],
                        [1,0,0,0],
                        [0,0,0,1]],
                        
                        [[1,0,0,0],
                        [0,0,1,0],
                        [0,1,0,0]],
                         
                        [[0,0,1,0],
                        [0,0,0,1],
                        [1,0,0,0]]],dtype=torch.float)

input = torch.nn.Softmax(1)(target)

assert torch.all(target[0].sum(1).to(torch.float)==input[0].sum(1).to(torch.float)),"Failed: sum is not equal to one"

loss = torch.nn.BCELoss()(input,target)
</code></pre>
<p>What is a suitable activation and loss function for this kind of multi dimensional one hot encoded data?</p>

ptrblck: <p><code>nn.Softmax(dim=1)</code> will create an output, where <code>output.sum(dim=1)</code> returns ones as seen in your code snippet:</p>
<pre><code class="lang-python">input = torch.nn.Softmax(1)(target)
print(input.sum(1))
&gt; tensor([[1.0000, 1.0000, 1.0000, 1.0000],
          [1.0000, 1.0000, 1.0000, 1.0000],
          [1.0000, 1.0000, 1.0000, 1.0000]])
</code></pre>
<p>However, <code>nn.BCELoss</code> is used for a binary classification/segmentation or a multi-label classification/segmentation use case, where zero, one or multiple classes can be active per sample.</p>
<p>Based on your target it seems you are dealing with a multi-class classification, where only one class is active per sample.<br>
In that case, you should use <code>nn.CrossEntropyLoss</code>, which expects raw logits as the model output and a target as a <code>LongTensor</code> containing the class indices in the range <code>[0, nb_classes-1]</code>.<br>
To create the target, use <code>target = torch.argmax(target, dim=2)</code>.</p>

pydisc: <p>This doesn’t working</p>
<pre><code class="lang-auto">t =  torch.argmax(target, dim=2)
torch.nn.CrossEntropyLoss()(input,target)
</code></pre>
<blockquote>
<p>ValueError: Expected target size (3, 4), got torch.Size([3, 3, 4])</p>
</blockquote>

ptrblck: <p>The output of <code>torch.argmax</code> is <code>t</code>, while you are still passing <code>target</code> to the criterion.<br>
Could you pass <code>t</code> to it and rerun the code?</p>

pydisc: <p>sorry that was my mistake the real code and error is this</p>
<pre><code class="lang-auto">input = torch.nn.Softmax(1)(target)
t =  torch.argmax(target, dim=2)
torch.nn.CrossEntropyLoss()(input,t)
</code></pre>
<blockquote>
<p>ValueError: Expected target size (3, 4), got torch.Size([3, 3])</p>
</blockquote>

ptrblck: <p><code>nn.CrossEntropyLoss</code> expects raw logits, so you should not use a softmax on the <code>input</code> tensor.</p>
<p>Since the one-hot encoding is in <code>dim2</code> in the <code>target</code> tensor and deals with 4 classes, your <code>input</code> should thus have the shape <code>[3, 4, 3]</code>.<br>
This code snippet shows its usage:</p>
<pre><code class="lang-python">target = torch.tensor([[[0,1,0,0],
                        [1,0,0,0],
                        [0,0,0,1]],
                        
                        [[1,0,0,0],
                        [0,0,1,0],
                        [0,1,0,0]],
                         
                        [[0,0,1,0],
                        [0,0,0,1],
                        [1,0,0,0]]],dtype=torch.float)

input = torch.randn(3, 4, 3)
criterion = nn.CrossEntropyLoss()
target = torch.argmax(target, dim=2)
loss = criterion(input, target)
</code></pre>