Implicit dimension choice for softmax warning

hongtaesuk: <p>Hey guys,<br>
I was following exactly the same as the tutorial says<br>
which official <a href="http://PyTorch.org" rel="nofollow noopener">PyTorch.org</a> had given on their site.<br>
However, I got stuck on the softmax function which shows no warning according to the tutorial, but my python gives me a warning message it says,</p>
<p>UserWarning: Implicit dimension choice for log_softmax has been deprecated. Change the call to include dim=X as an argument.<br>
return F.log_softmax(self.linear(bow_vec))</p>
<ol>
<li>Can anyone of you guys knows what ‘dim’ variable do?</li>
<li>Why does it produce a warning with my python?</li>
</ol>

chenyuntc: <aside class="quote no-group" data-username="hongtaesuk" data-post="1" data-topic="12314">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hongtaesuk/48/18907_2.png" class="avatar"> hongtaesuk:</div>
<blockquote>
<p>Can anyone of you guys knows what ‘dim’ variable do?</p>
</blockquote>
</aside>
<p><code>A dimension along which softmax will be computed.</code></p>
<p>if</p>
<p>output = softmax(input,dim=2)</p>
<p>then</p>
<p>output.sum(dim=2) is all 1</p>

hongtaesuk: <p>do you mean if the input of a softmax function is a variable that is comprised of, let’s say, 6 elements then we should clarify that it is dimension of 6<br>
with variable dim=6 ???</p>

chenyuntc: <p>No, I mean:</p>
<pre><code class="lang-auto">a = t.Tensor(3,4,5)
b = softmax(Variable(a), dim=2)
b.sum(2) # is all 1
</code></pre>

hongtaesuk: <p>I don’t think I’m aware of what ‘dim’ actually means.<br>
On the example you wrote, python says the expected range<br>
of dim is [-3, 2].<br>
Think I’m confused with the definition of what dim actually is in that context since so many meaning exist as a word dim in machine learning.<br>
Could you explain more specifically what that means?</p>

aherreracep1: <p>I came across exactly the same error.<br>
Just by following the tutorial <a href="http://opennmt.net/OpenNMT-py/quickstart.html" rel="nofollow noopener">http://opennmt.net/OpenNMT-py/quickstart.html</a> , when typing</p>
<pre><code class="lang-auto">python train.py -data data/demo -save_model demo-model
</code></pre>
<p>it gives me the same warning.</p>
<pre><code class="lang-auto">/OpenNMT-py/onmt/modules/GlobalAttention.py:177: UserWarning: Implicit dimension choice for softmax has been deprecated. Change the call to include dim=X as an argument.
</code></pre>
<p>Can anyone please help as to how can this be resolved?</p>

ptrblck: <p>You could add the <code>dim</code> parameter to the <code>softmax</code> function call, to get rid of this warning.<br>
Have a look at the dimensions of the input to <code>softmax</code> and try to figure out, in which dimension the <code>softmax</code> should be calculated.</p>

Diego: <p>In this context dim refers to the dimension in which the softmax function will be applied.</p>
<pre><code class="lang-auto">&gt;&gt;&gt; a = Variable(torch.randn(5,2))
&gt;&gt;&gt; F.softmax(a, dim=1)
Variable containing:
 0.6360  0.3640
 0.3541  0.6459
 0.2412  0.7588
 0.0860  0.9140
 0.6258  0.3742
[torch.FloatTensor of size 5x2]

&gt;&gt;&gt; F.softmax(a, dim=0)
Variable containing:
 0.6269  0.3177
 0.0543  0.0877
 0.1482  0.4128
 0.0103  0.0969
 0.1603  0.0849
[torch.FloatTensor of size 5x2]

</code></pre>
<p>On the first case (using dim=1) the softmax function is applied along the axis 1 . That’s why all rows add up to 1. On the second case (using dim=0) the softmax function is applied along the axis 0. Making all the columns add up to 1.</p>

KeepingItClassy11: <p>Just out of curiosity, how did the softmax function determine the dimension before the deprecation? Was it always 1?</p>

ruotianluo: <p><a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/functional.py#L814" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/master/torch/nn/functional.py#L814</a>.</p>
<p>For matrices, it’s 1. For others, it’s 0.</p>

prasad_sogalad: <p>If your batch size is say=64, and number of classes you want the CNN/NN to classify is say=10. Then the dimension of output tensor will be 64*10.<br>
Set dim=0 to apply softmax along the rows.</p>
<p>m=nn.Softmax(dim=0)  <span class="hashtag">#softmax</span> along row</p>

Brando_Miranda: <p>hmmm might it be a good idea to just throw an error rather a warning and have the code actually stop running? Might save ppl headaches on debugging. I couldn’t even see the warning cuz I was printing a bunch of other stuff…if the code had just failed it would have saved me a bunch of time.</p>

HashRocketSyntax: <p><code>Softmax(dim=1)</code> for linear output.</p>

Mona_Jalal: <p>I kept getting the following error:</p>
<pre><code class="lang-auto">main_classifier.py:86: UserWarning: Implicit dimension choice for log_softmax has been deprecated. Change the call to include dim=X as an argument.
  logpt = F.log_softmax(input)
</code></pre>
<p>Then I used <code>dim=1</code></p>
<pre><code>    #logpt = F.log_softmax(input)
    logpt = F.log_softmax(input, dim=1)
</code></pre>
<p>based on <a href="https://discuss.pytorch.org/t/implicit-dimension-choice-for-softmax-warning/12314/10" class="inline-onebox">Implicit dimension choice for softmax warning - #10 by ruotianluo</a></p>
<p>but I get this error</p>
<pre><code class="lang-auto">train: True test: False
preparing datasets and dataloaders......
creating models......

=&gt;Epoches 1, learning rate = 0.0010000, previous best = 0.0000
training...
feats shape:  torch.Size([64, 419, 512])
labels shape:  torch.Size([64])
Traceback (most recent call last):
  File "main_classifier.py", line 347, in &lt;module&gt;
    loss = criterion(m(output[:,1]-output[:,0]), labels.float())
  File "/home/jalal/research/venv/dpcc/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "main_classifier.py", line 87, in forward
    logpt = F.log_softmax(input, dim=1)
  File "/home/jalal/research/venv/dpcc/lib/python3.8/site-packages/torch/nn/functional.py", line 1769, in log_softmax
    ret = input.log_softmax(dim)
IndexError: Dimension out of range (expected to be in range of [-1, 0], but got 1)
</code></pre>
<p>so how should I find out what dim is?</p>

ptrblck: <p>As given in the new error message, you are trying to specify <code>dim=1</code> while <code>input</code> has only a single dimension.<br>
Use <code>dim=0</code> instead and make sure that’s really the expected shape.<br>
Common use cases use at least two dimensions as <code>[batch_size, feature_dim]</code> and use then the <code>log_softmax</code> in the feature dimension, but I’m also not familiar with your use case so <code>input</code> having a single dimension might be alright.</p>

Mona_Jalal: <p>Thanks for your response. It is part of an older code for focal loss here:</p>
<pre><code class="lang-auto">        logpt = F.log_softmax(input)
</code></pre>
<aside class="quote quote-modified" data-post="1" data-topic="145225">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mona_jalal/48/25715_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/logpt-logpt-gather-1-target-indexerror-dimension-out-of-range-expected-to-be-in-range-of-1-0-but-got-1/145225">Logpt = logpt.gather(1,target) IndexError: Dimension out of range (expected to be in range of [-1, 0], but got 1)</a> <a class="badge-category__wrapper " href="/c/vision/5"><span data-category-id="5" data-drop-close="true" class="badge-category " title="Topics related to either pytorch/vision or vision research related topics"><span class="badge-category__name">vision</span></span></a>
  </div>
  <blockquote>
    I found this implementation of focal loss in GitHub and I am using it for an imbalanced dataset binary classification problem. 
# IMPLEMENTATION CREDIT: https://github.com/clcarwin/focal_loss_pytorch
    class FocalLoss(nn.Module):
    def __init__(self, gamma=0.5, alpha=None, size_average=True):
        super(FocalLoss, self).__init__()
        self.gamma = gamma
        self.alpha = alpha
        if isinstance(alpha,(float,int)): self.alpha = torch.Tensor([alpha,1-alpha])
        if isinstance…
  </blockquote>
</aside>


ptrblck: <p>Thanks for the link.<br>
Based on both errors I don’t think you should fix the issues by changing the <code>dim</code> argument in <code>log_softmax</code> and <code>gather</code>, but should instead debug why the input has a single dimension (while more are expected).</p>

clockzhong: <p>My pytorch version is 2.1.1<br>
I also got the warning:</p>
<blockquote>
<blockquote>
<blockquote>
<p>input<br>
tensor([[ 0.5228, -1.4060, -0.8637],<br>
[-0.4285, -1.1788,  0.6950]])<br>
m = nn.Softmax()<br>
m(input)<br>
/home/clock/Work/venvs/torchgpu/lib/python3.10/site-packages/torch/nn/modules/module.py:1518: UserWarning: Implicit dimension choice for softmax has been deprecated. Change the call to include dim=X as an argument.<br>
return self._call_impl(*args, **kwargs)<br>
tensor([[0.7167, 0.1042, 0.1792],<br>
[0.2199, 0.1038, 0.6763]])<br>
torch.<strong>version</strong><br>
‘2.1.1+cu121’</p>
</blockquote>
</blockquote>
</blockquote>
<p>But according pytorch official document here (for 2.1.0):<br>
<a href="https://pytorch.org/docs/stable/generated/torch.nn.Softmax.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.nn.Softmax.html</a><br>
nn.Softmax() still accept implicit dimension choice as dim=None, so the document is not synched to the  latest code yet, we need fix it in document explanation.</p>