What's difference of nn.Softmax(), nn.softmax(), nn.functional.softmax()?

username: I_H_Yoo
cooked: <p>Hi. I guess there are three way to use activation function in a custom module:</p>
<ol>
<li>Use <code>nn.Softmax()</code> in the initializer in the custom model.</li>
<li>Use <code>nn.softmax()</code> in the forward function in the custom model.</li>
<li>Use <code>nn.functional.softmax()</code> in the forward function in the custom model.</li>
</ol>
<p>What are the differences among them and what is the proper way to use an activation function in my custom model?</p>

username: ptrblck
cooked: <p><code>nn.Softmax</code> is an <code>nn.Module</code>, which can be initialized e.g. in the <code>__init__</code> method of your model and used in the <code>forward</code>.</p>
<p><code>torch.softmax()</code> (I assume <code>nn.softmax</code> is a typo, as this function is undefined) and <code>nn.functional.softmax</code> are equal and I would recommend to stick to <code>nn.functional.softmax</code>, since it’s documented. <a class="mention" href="/u/tom">@tom</a> gives a better answer <a href="https://discuss.pytorch.org/t/why-there-isnt-a-method-named-torch-softmax/90554/2">here</a>. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

username: tom
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> nails it with his comment.</p>
<p>We also do have a short discussion in the <a href="https://pytorch.org/deep-learning-with-pytorch" rel="nofollow noopener">Deep Learning with PyTorch book</a> in chapter 8, similar to the brief mention in <a href="https://github.com/t-vi/acdl2020/raw/master/pytorch_introduction_slides.pdf" rel="nofollow noopener">slide 54 for a course I taught earlier this month</a> - copied here for your convenience:</p>
<p>Some thoughts on functional vs. Module</p>
<ol>
<li>If you write for re-use, the functional / Module split of PyTorch has turned out<br>
to be a good idea.</li>
<li>Use functional for stuff without state (unless you have a quick and dirty Sequential).</li>
<li>Never re-use modules (define one torch.nn.ReLU and use it 5 times). It’s a trap!<br>
When doing analysis or quantization (when ReLU becomes stateful due to quantization params), this will break.</li>
</ol>
<p>The latter two seem to cover your case here, but 2. is more a matter of personal preference, while 3. is really about writing better (as in less risky, more clear) code.<br>
Item 1. ist more is more for when you write a library or something you expect to be re-used a lot.</p>
<p>Best regards</p>
<p>Thomas</p>

username: I_H_Yoo
cooked: <p>Thank you for your answer but I’m still not sure what are the differences of them.</p>
<p>So can I say that functional is for re-use? What do ‘stuff without state’ and ‘quantization’ mean? Which should I use between functional and Module?</p>

username: tom
cooked: <aside class="quote no-group" data-username="I_H_Yoo" data-post="4" data-topic="90934">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/i_h_yoo/48/19022_2.png" class="avatar"> I_H_Yoo:</div>
<blockquote>
<p>So can I say that functional is for re-use?</p>
</blockquote>
</aside>
<p>That is more “if you implement something new, provide both because you don’t know which is more convenient to your users”.</p>
<aside class="quote no-group" data-username="I_H_Yoo" data-post="4" data-topic="90934">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/i_h_yoo/48/19022_2.png" class="avatar"> I_H_Yoo:</div>
<blockquote>
<p>What do ‘stuff without state’ and ‘quantization’ mean?</p>
</blockquote>
</aside>
<p>That’s another slide, there.<br>
Or, try the <a href="https://pytorch.org/tutorials/beginner/nn_tutorial.html" rel="noopener nofollow ugc">“what is nn” tutorial</a>.<br>
Quantization <a href="https://pytorch.org/tutorials/advanced/static_quantization_tutorial.html" rel="noopener nofollow ugc">has its own set of tutorials</a>.</p>
<aside class="quote no-group">
<blockquote>
<p>Which should I use between functional and Module?</p>
</blockquote>
</aside>
<p>For softmax, and if you aren’t building a  <code>Sequential</code>, I’d use the functional interface. But there is a modicum of taste in that.</p>
<p>Best regards</p>
<p>Thomas</p>

username: I_H_Yoo
cooked: <p>Thank you for your kind answer.I understand the ‘state’. I guess it refers a weight in a  neural network right? <code>Quantization</code> is something I’ve never seen so I need to see youre reference. However, I’m little bit confuse becuase I think there is no conclusion for some kind of a threshold that using functional or Module. Maybe that’s why I’m not an native english speaker.</p>
<p>I apologise for the inconvenience but if I’m not asking you this question, I will never know. Please excuse me for bothering you. And I want to quote what you mentioned just like you did to mine, but I don’t know how to do it so I just write them down.</p>
<blockquote>
<p>That is more “if you implement something new, provide both because you don’t know which is more convenient to your users”</p>
</blockquote>
<p>I don’t want to make an activation function. I just want to know <strong>proper way</strong> to use them (softmax, ReLU, whatever it is). However, I guess there is no clear standard for it. The functional are not saved in the stae of a neural network so I should avoid it if I want to save them as <code>state_dict</code> right?</p>
<blockquote>
<p>For softmax, and if you aren’t building a <code>Sequential</code> , I’d use the functional interface.</p>
</blockquote>
<p>Why would you do that? What’s the beneficial of using functional interface instead of Module?</p>
<p>Once again, I apologise for the inconvenience.</p>

username: tom
cooked: <aside class="quote no-group quote-modified" data-username="I_H_Yoo" data-post="6" data-topic="90934">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/i_h_yoo/48/19022_2.png" class="avatar"> I_H_Yoo:</div>
<blockquote>
<blockquote>
<p>For softmax, and if you aren’t building a <code>Sequential</code> , I’d use the functional interface.</p>
</blockquote>
<p>Why would you do that? What’s the beneficial of using functional interface instead of Module?</p>
</blockquote>
</aside>
<p>I think about modules as holding state (e.g. weights, as you mention) and using that and the inputs I pass to produce the output. So for softmax, using the functional interface expresses how I think about it (i.e. as a function of the inputs and no weights etc.).</p>
<p>But don’t make it a science, just go with what you prefer.</p>