Torch.max and softmax confusion

username: John_J_Watson
cooked: <p>I am quite new to pytorch, and I am going through the [transfer learning tutorials] on resnet(<a href="https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html</a>), and I notice this line (which is also there in the inference mode):</p>
<pre><code>outputs = model(inputs)
_, preds = torch.max(outputs, 1)
</code></pre>
<p>which, when I go through the docs, suggests that it is simply the max on each tensor along with the indices. So, <code>preds</code> just contains the indices, which indirectly represents the classes.</p>
<p>Now, when I look at <a href="https://pytorch.org/hub/pytorch_vision_mobilenet_v2/" rel="nofollow noopener">mobilenet</a> tutorial (I adapted it to output 20 classes and retrained the final layer for feature extraction), I see the line</p>
<pre><code class="lang-auto"># The output has unnormalized scores. To get probabilities, you can run a softmax on it.
print(torch.nn.functional.softmax(output[0], dim=0))
</code></pre>
<p>Why is there this difference in resnet and mobilenet outputs? Strangely enough, when I simply use:</p>
<pre><code>_, preds = torch.max(outputs, 1)
</code></pre>
<p>in my mobilenet transfer learning code, I seem to get the correct answers (i.e the classification is done correctly). So, I am now seriously confused as to why I would need the probablities as described in mobilenet, when I do not seem to need them in resnet.</p>
<p>Also, when I use these probabiliities via softmax and train, like so:</p>
<pre><code>outputs = model(inputs)
outputs = torch.nn.functional.softmax(outputs, dim=1)
_, preds = torch.max(outputs, 1)
</code></pre>
<p>I get the same results as one without this softmax. so, I am not entirely sure what is the correct thing to do.</p>
<p>Any pointers would really help.</p>
<p>Thank you.</p>

username: KFrank
cooked: <p>Hello John!</p>
<p>To answer your most concrete question first:</p>
<aside class="quote no-group" data-username="John_J_Watson" data-post="1" data-topic="80697" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/john_j_watson/48/16421_2.png" class="avatar"> John_J_Watson:</div>
<blockquote>
<p>Also, when I use these probabiliities via softmax and train, like so:</p>
<pre><code>outputs = model(inputs)
outputs = torch.nn.functional.softmax(outputs, dim=1)
_, preds = torch.max(outputs, 1)
</code></pre>
</blockquote>
</aside>
<p>In this case <code>preds</code> will be the same whether you include <code>softmax()</code><br>
or remove it.  This is because <code>softmax()</code> maps its (algebraically)<br>
largest input value to the largest output value.  Therefore the <em>index</em><br>
of the largest value won’t change.</p>
<p>(It’s not clear to me what you mean by “train.”  If you pass <code>outputs</code><br>
to a loss function, call <code>loss.backward()</code>, and then take an optimizer<br>
step, you <em>will</em> get different results if you leave out the <code>softmax()</code>.)</p>
<p>As an aside, (with recent versions of pytorch) you can call<br>
<code>preds = torch.argmax (outputs, 1)</code>, if you prefer.</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto"># The output has unnormalized scores. To get probabilities, you can run a softmax on it.
print(torch.nn.functional.softmax(output[0], dim=0))
</code></pre>
</blockquote>
</aside>
<p>To comment a little more on this:  If your model returns the output of a<br>
<code>Linear</code> layer (without passing it through something like <code>softmax()</code>),<br>
the values returned should be understood as raw-score <code>logits</code> that<br>
run, in principle, from <code>-inf</code> to <code>inf</code>.  Such logits are what is expected<br>
by some loss functions, such as <code>CrossEntropyLoss</code>.</p>
<p><code>softmax()</code> converts a set of logits to <code>probabilities</code> that run from<br>
<code>0.0</code> to <code>1.0</code> and sum to <code>1.0</code>.  If you wish to work with probabilities for<br>
some reason, for example, if your loss function expects probabilities,<br>
then you would pass your logits through <code>softmax()</code>.  But, conceptually,<br>
they’re just different ways of representing the same thing – the logits<br>
and probabilities contain (almost) the same information.</p>
<p>Best.</p>
<p>K. Frank</p>

username: John_J_Watson
cooked: <p>Thanks a tonne for this Frank - it really helps my understanding of things.</p>
<aside class="quote no-group" data-username="KFrank" data-post="2" data-topic="80697">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>(It’s not clear to me what you mean by “train.” If you pass <code>outputs</code><br>
to a loss function, call <code>loss.backward()</code> , and then take an optimizer<br>
step, you <em>will</em> get different results if you leave out the <code>softmax()</code> .)</p>
</blockquote>
</aside>
<p>Could you please elaborate on this? Infact, this is what I am doing, and I am not sure what is the correct value to pass the loss function - raw logits or the values obtained by passing them to softmax(). What should be the expected difference? In the little tests I am doing, I do not actually see a difference and hence my question.</p>

username: KFrank
cooked: <p>Hello John!</p>
<aside class="quote no-group" data-username="John_J_Watson" data-post="3" data-topic="80697" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/john_j_watson/48/16421_2.png" class="avatar"> John_J_Watson:</div>
<blockquote>
<p>I am not sure what is the correct value to pass the loss function - raw logits or the values obtained by passing them to softmax().</p>
</blockquote>
</aside>
<p>It depends on what you are trying to do, and what your loss function is.</p>
<p>It sounds as if you might be working on a twenty-class classification<br>
problem.  In such a case, you would typically use <code>CrossEntropyLoss</code><br>
as your loss function.  Your model would output directly the results of<br>
its last fully-connected <code>Linear</code> layer as logits.  <code>CrossEntropyLoss</code><br>
would take these logits and a batch of integer class labels as its<br>
inputs.  You would not pass anything through <code>softmax()</code> (although<br>
a <code>softmax()</code> is, in effect, built into <code>CrossEntropyLoss</code>).</p>
<aside class="quote no-group">
<blockquote>
<p>In the little tests I am doing, I do not actually see a difference and hence my question.</p>
</blockquote>
</aside>
<p>I’m not sure exactly what you are doing, what your tests are, and how<br>
you are looking for a difference.</p>
<p>But here is a (pytorch version 0.3.0) sample script that shows the<br>
effect of adding <code>softmax()</code>:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
torch.__version__
torch.manual_seed (2020)

some_numbers = torch.autograd.Variable (torch.randn (5), requires_grad = True)
some_numbers

some_outputs = some_numbers
loss = some_outputs.sum()
loss
loss.backward()
some_numbers.grad

some_numbers.grad.zero_()
some_outputs = some_numbers
some_outputs = torch.nn.functional.softmax (some_outputs, 0)
loss = some_outputs.sum()
loss
loss.backward()
some_numbers.grad
</code></pre>
<p>And here is the output:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'0.3.0b0+591e73e'
&gt;&gt;&gt; torch.manual_seed (2020)
&lt;torch._C.Generator object at 0x000001BAF61C6630&gt;
&gt;&gt;&gt;
&gt;&gt;&gt; some_numbers = torch.autograd.Variable (torch.randn (5), requires_grad = True)
&gt;&gt;&gt; some_numbers
Variable containing:
 1.2372
-0.9604
 1.5415
-0.4079
 0.8806
[torch.FloatTensor of size 5]

&gt;&gt;&gt;
&gt;&gt;&gt; some_outputs = some_numbers
&gt;&gt;&gt; loss = some_outputs.sum()
&gt;&gt;&gt; loss
Variable containing:
 2.2911
[torch.FloatTensor of size 1]

&gt;&gt;&gt; loss.backward()
&gt;&gt;&gt; some_numbers.grad
Variable containing:
 1
 1
 1
 1
 1
[torch.FloatTensor of size 5]

&gt;&gt;&gt;
&gt;&gt;&gt; some_numbers.grad.zero_()
Variable containing:
 0
 0
 0
 0
 0
[torch.FloatTensor of size 5]

&gt;&gt;&gt; some_outputs = some_numbers
&gt;&gt;&gt; some_outputs = torch.nn.functional.softmax (some_outputs, 0)
&gt;&gt;&gt; loss = some_outputs.sum()
&gt;&gt;&gt; loss
Variable containing:
 1
[torch.FloatTensor of size 1]

&gt;&gt;&gt; loss.backward()
&gt;&gt;&gt; some_numbers.grad
Variable containing:
 0
 0
 0
 0
 0
[torch.FloatTensor of size 5]
</code></pre>
<p>As you can see adding the line</p>
<pre data-code-wrap="python"><code class="lang-python">some_outputs = torch.nn.functional.softmax (some_outputs, 0)
</code></pre>
<p>causes the loss to change to <code>1</code> (from <code>2.2911</code>), and the gradient to<br>
change to <code>[0, 0, 0, 0, 0]</code> (from <code>[1, 1, 1, 1, 1]</code>).</p>
<p>Typically you would call <code>loss.backward()</code> to calculate the gradient<br>
of your loss function with respect to your model parameters.  Then you<br>
would call <code>optim.step()</code> to adjust your model parameters a little bit in<br>
the direction opposite to your gradient.  So a different gradient means<br>
different updated model parameters, and this means different results.</p>
<p>Best.</p>
<p>K. Frank</p>

username: John_J_Watson
cooked: <p>Hello Frank,</p>
<p>Thanks so much for your detailed explanation. I completely understand what you are saying. Indeed, I should not be using softmax() when using it to compute loss.backward().</p>
<p>Thank you again for your patience explaining a nn to a beginner like me. Cant thank you enough.</p>