What is the difference between torch.nn and torch.nn.functional?

micklexqg: <p>They look like a little same…<br>
so, is there any difference between them?</p>

ptrblck: <p>While the former defines <code>nn.Module</code> classes, the latter uses a functional (stateless) approach.<br>
To dig a bit deeper: <code>nn.Modules</code> are defined as Python classes and have attributes, e.g. a <code>nn.Conv2d</code> module will have some internal attributes like <code>self.weight</code>. <code>F.conv2d</code> however just defines the operation and needs all arguments to be passed (including the weights and bias). Internally the modules will usually call their functional counterpart in the <code>forward</code> method somewhere.</p>
<p>That being said, it depends also on your coding style how you would like to work with your modules/parameters etc. While modules might be good enough in most use cases, the functional API might give you additional flexibility which is needed sometimes.<br>
We’ve have a similar discussion recently in <a href="https://discuss.pytorch.org/t/beginner-should-relu-sigmoid-be-called-in-the-init-method/18689/6?u=ptrblck">this thread</a>.</p>

micklexqg: <p>en…, thank you, <img src="https://discuss.pytorch.org/images/emoji/apple/hugs.png?v=6" title=":hugs:" class="emoji" alt=":hugs:"></p>

n0obcoder: <p>how does gradients flow in the case of nn.functional ?  i am a little confused. How do the weights get trained in case of nn.functional ?</p>

ptrblck: <p>Each operation is tracked by Autograd, if parameters are involved with require gradients.<br>
The output of such operations get a <code>.grad_fn</code> attribute, which points to the backward function for the last operation:</p>
<pre><code class="lang-python">x = torch.randn(1, 1)
w = nn.Parameter(torch.randn(1, 1))

output = x * w
print(output)
&gt; tensor([[2.5096]], grad_fn=&lt;MulBackward0&gt;)
</code></pre>
<p>The backward call uses these <code>grad_fn</code>s to calculate the gradient and store it in the <code>.grad</code> attribute of the parameters:</p>
<pre><code class="lang-python">output.backward()
print(w.grad)
&gt; tensor([[1.1757]])
</code></pre>

n0obcoder: <p>thanks for the reply<br>
but what are the  fundamental differences between <code>torch.nn.Conv1d</code> and <code>torch.nn.functional.conv1d</code> ?<br>
i guess nn.Conv1d initializes the kernel weights automatically and nn.functional.conv1d needs an input kernel…</p>
<p>My doubts…</p>
<ul>
<li>does gradient calculation and back-prop work in the same way for both of the above mentioned methods?</li>
<li>where would i want to use nn over nn.functional and vice-versa ? (what is the need for nn.functional.conv1d when you already have nn.Conv1d ? )</li>
</ul>

ptrblck: <p>Have a look at <a href="https://discuss.pytorch.org/t/beginner-should-relu-sigmoid-be-called-in-the-init-method/18689/6">this post</a> for some more information and my point of view.</p>
<p>TLDR: the modules (<code>nn.Module</code>) use internally the functional API.<br>
There is no difference as long as you store the parameters somewhere (manually if you prefer the functional API or in an <code>nn.Module</code> “automatically”).<br>
Having the <code>nn.Module</code> containers as an abstraction layer makes development easy and keeps the flexibility to use the functional API.</p>

krtkgrwl: <p>For my use case, I have added an extra parameter apart from weight and bias. Somehow the gradients for that extra parameter are zero even though requires_grad is true. I can’t seem to figure out why this can happen. Any ideas that anyone can think of? Thanks in advance.</p>