Dimension for logsoftmax

username: NOP
cooked: <pre><code class="lang-auto">class M(nn.Module):
    def __init__(self):
        super().__init__()
        self.l1 = nn.Linear(50*50,100)
        self.l2 = nn.ReLU()
        self.l3 = nn.Linear(100,100)
        self.l4 = nn.Tanh()        
        self.l5 = nn.Linear(100,10) 
        self.l6 = nn.LogSoftmax()
</code></pre>
<p>Having module M, if I don’t set<br>
<code>self.l6 = nn.LogSoftmax(dim=0)</code><br>
or<br>
<code>self.l6 = nn.LogSoftmax(dim=1)</code></p>
<p>I get the warning:</p>
<blockquote>
<p>UserWarning: Implicit dimension choice for log_softmax has been deprecated. Change the call to include dim=X as an argument.</p>
</blockquote>
<p>What does it mean to set <code>dim=0</code> and what <code>dim=1</code>?</p>

username: ptrblck
cooked: <p>The <code>dim</code> argument defines which dimension should be used to calculate the log softmax, i.e. in which dimension the class logits are located.<br>
Have a look at this small example using softmax:</p>
<pre><code class="lang-python">x = torch.randn(5, 3)
x0 = F.softmax(x, dim=0)
print(x0)
&gt; tensor([[0.1313, 0.0170, 0.4122],
        [0.0167, 0.6336, 0.0440],
        [0.1764, 0.0804, 0.3689],
        [0.4540, 0.0501, 0.0967],
        [0.2217, 0.2189, 0.0782]])
print(x0.sum(0))
&gt; tensor([1.0000, 1.0000, 1.0000])

x1 = F.softmax(x, dim=1)
print(x1)
&gt; tensor([[0.2528, 0.0482, 0.6990],
        [0.0169, 0.9438, 0.0393],
        [0.2847, 0.1908, 0.5245],
        [0.7409, 0.1202, 0.1389],
        [0.3620, 0.5255, 0.1124]])
print(x1.sum(1))
&gt; tensor([1.0000, 1.0000, 1.0000, 1.0000, 1.0000])
</code></pre>
<p>As you can see, the sum of all probabilities will be 1. in the specified dimension.</p>
<p>In your use case, you should use <code>dim=1</code> to calculate the log probabilities for each sample in the batch over all classes (which are in <code>dim1</code>).</p>

username: NOP
cooked: <p>OK, just to clarify,</p>
<p>the dimension 0 would be the number of batches (5 in your case),<br>
dimension 1 is number of samples in the batch (3 in your case)<br>
so I set <code>dim=1</code> to calculate logits (log probabilities) of each sample in the batch).</p>

username: ptrblck
cooked: <p>Dimension 0 is the batch dimension and gives the number of samples in the current batch.<br>
You can consider  <code>x</code>  being a single batch of samples.<br>
Dimension 1 is the feature dimension in my use case and gives the number of different features.</p>

username: ananda2020
cooked: <p>Hi,<br>
I have a question. What I understood from this question is that the softmax layer is a classifier of 10 classes. So when you are saying - for this case dim = 1 is to be used- so can I generalize that for classification problem one should use dim =1 or I am understanding wrong.<br>
Could you please give an example where dim =0 can be used.<br>
Thanks in advance.</p>

username: ptrblck
cooked: <p>The majority of PyTorch layers use tensors with the batch dimension in <code>dim0</code>.<br>
The typical multi-class classification output would have a shape of <code>[batch_size, nb_classes]</code>, and you would calculate the probability for each class in each sample:</p>
<pre><code class="lang-python">batch_size = 2
nb_classes = 3
x = torch.randn(batch_size, nb_classes)
prob = F.softmax(x, dim=1)
print(prob)
&gt; tensor([[0.6935, 0.1843, 0.1223],
          [0.8212, 0.0705, 0.1083]])
</code></pre>
<p>Here you can see that for sample0, class0 has a probability of 69.35%, class1 18.43%, and class2 12.23%.</p>
<p>If you are using <code>F.softmax</code> or <code>F.log_softmax</code> with <code>dim=0</code>, you would calculate the (log) probability in the batch dimension.</p>
<pre><code class="lang-python">prob = F.softmax(x, dim=0)
print(prob)
&gt; tensor([[0.2748, 0.5397, 0.3364],
          [0.7252, 0.4603, 0.6636]])
</code></pre>
<p>Now you are looking at: for class0, sample0 has a probability of 27.48%, while sample1 has 72.52%.</p>
<p>RNNs are an exception and are using the temporal dimension in <code>dim0</code>, so it might depend on your use case, if you want to apply the (log)softmax in this dimension.</p>

username: ananda2020
cooked: <p>Thank you ptrblck for such a clear explanation. I am learning so much from this forum.<br>
Regards,<br>
ananda2020</p>