How to choose between torch.nn.Functional and torch.nn module?

Harry_Zhi: <p>Hi, everyone. I am new to Pytorch from Keras. I find Pytorch has sufficient existing modules and functions that I would like to use in my work. It is great.</p>
<p>I have a problem understanding the codes about /examples/minist <a href="http://main.py" rel="nofollow noopener">main.py</a>, where the CNN for MNIST is defined, as shown below:</p>
<pre><code>class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(1, 10, kernel_size=5)
        self.conv2 = nn.Conv2d(10, 20, kernel_size=5)
        self.conv2_drop = nn.Dropout2d()
        self.fc1 = nn.Linear(320, 50)
        self.fc2 = nn.Linear(50, 10)

    def forward(self, x):
        x = F.relu(F.max_pool2d(self.conv1(x), 2))
        x = F.relu(F.max_pool2d(self.conv2_drop(self.conv2(x)), 2))
        x = x.view(-1, 320)
        x = F.relu(self.fc1(x))
        x = F.dropout(x, training=self.training)
        x = self.fc2(x)
        return F.log_softmax(x)
</code></pre>
<p>I see some functions like <strong>Dropout used in both <strong>init</strong> and forward methods</strong>, however in <em>init</em>_ we use Dropout from torch.nn module while the latter one is from torch.nn.functional,<br>
<strong>is there any reason for this configuration?</strong> Why not use the same dropout twice.</p>
<p><strong>Also, can we use nn.relu instead of F.relu in forward part?</strong></p>
<p>Hope someone can help me distinguish the same function module like convolution, dropout , activation in nn. and nn.functional. When shoud I better use the nn module and when is the nn.functional works better?</p>
<p>By the way, the parameters received by nn.Conv2d and F.con2d are also different, and when we use Class nn.Con2d, it uses the function F.conv2d in its forward method.</p>

miguelvr: <p>In PyTorch you define your Models as subclasses of torch.nn.Module.</p>
<p>In the <code>__init__</code> function, you are supposed to <strong>initialize</strong> the layers you want to use. Unlike keras, Pytorch goes more low level and you have to specify the sizes of your network so that everything matches.</p>
<p>In the <code>forward</code> method, you <strong>specify the connections</strong> of your layers. This means that you will use the layers you already initialized, in order to re-use the same layer for each forward pass of data you make.</p>
<p>torch.nn.Functional contains some useful functions like activation functions a convolution operations you can use. However, these are not full layers so if you want to specify a layer of any kind you should use torch.nn.Module.</p>
<p>You would use the torch.nn.Functional conv operations to define a custom layer for example with a convolution operation, but not to define a standard convolution layer.</p>
<p>Let me know if it is clear!</p>
<p>Cheers</p>

Harry_Zhi: <p>Thanks a lot for your detailed response, Miguel.</p>
<p>From your explanation, I get to know that torch.nn module is a real layer which can be added or connected to other layers or network models. However, the functions in torch.nn.functional are just some arithmetical <strong>operations</strong> , not the layers which have trainable parameters such as weights and bias terms.</p>
<p>As a result, <strong>layers</strong> with parameters are usually initialized in <em>init</em> to be shared by the whole module, while some connections or simple operations without parameters can be defined in forward to be used in  forward propagation.</p>
<p>As for dropout,  because it does not have trainable weights, so that we can use either the layer form (from torch.nn) or connection form (from torch.nn.functional), is that correct?<br>
By this I mean, <strong>if I delete the layer self.conv2_drop layer in <em>init</em>, I should change the forawrd method like this:</strong></p>
<pre><code>    class Net(nn.Module):
        def __init__(self):
            super(Net, self).__init__()
            self.conv1 = nn.Conv2d(1, 10, kernel_size=5)
            self.conv2 = nn.Conv2d(10, 20, kernel_size=5)

            #delete this layer 
            #self.conv2_drop = nn.Dropout2d()

            self.fc1 = nn.Linear(320, 50)
            self.fc2 = nn.Linear(50, 10)

        def forward(self, x):
            x = F.relu(F.max_pool2d(self.conv1(x), 2))

           # replace 
           #x = F.relu(F.max_pool2d(self.conv2_drop(self.conv2(x)), 2))
            x = self.conv2(x)
            x = F.dropout(x, training = self.training)
            x = F.relu(F.max_pool2d(x, 2))
           #Or use 
           #x = F.relu(F.max_pool2d(F.dropout(self.conv2(x)), 2))
         
            x = x.view(-1, 320)
            x = F.relu(self.fc1(x))
            x = F.dropout(x, training=self.training)
            x = self.fc2(x)
            return F.log_softmax(x)
</code></pre>
<p>Does the change of dropout layer to connections affect something?</p>

miguelvr: <p>Although nn.Dropout is not a trainable layer, it is still a nn.Module, so you should initialize it in <code>__init__</code> first and then use it. I think it would raise an error otherwise.</p>

Harry_Zhi: <p>HI Miguel,<br>
I have just tried the modified codes and no error was raised.<br>
From model.named_modules() ,I can see the difference of these two networks:<br>
<strong>As for original Net:</strong></p>
<pre><code>(conv1): Conv2d(1, 10, kernel_size=(5, 5), stride=(1, 1))
(conv2): Conv2d(10, 20, kernel_size=(5, 5), stride=(1, 1))
(conv2_drop): Dropout2d (p=0.5)
(fc1): Linear (320 -&gt; 50)
(fc2): Linear (50 -&gt; 10)
</code></pre>
<p><strong>As for the modified version:</strong></p>
<pre><code>('conv1', Conv2d(1, 10, kernel_size=(5, 5), stride=(1, 1)))
('conv2', Conv2d(10, 20, kernel_size=(5, 5), stride=(1, 1)))
('fc1', Linear (320 -&gt; 50))
('fc2', Linear (50 -&gt; 10))
</code></pre>
<p>There is a dropout layer missed. But both network models works and should behave the same operation in forward propagation.</p>
<p>I wonder if these two networks are naturally the same during training and testing?</p>
<p><a class="mention" href="/u/apaszke">@apaszke</a> <a class="mention" href="/u/smth">@smth</a> <a class="mention" href="/u/colesbury">@colesbury</a><br>
Hope you can also give me some suggestions if possible.<br>
Thanks.</p>

miguelvr: <p>I’m just guessing here, but I would say, based on the documentation, that only nn.Dropout changes behaviour when doing model.eval() and model.train(), while F.dropout is the standard dropout operation. You should probably keep using nn.Dropout</p>

fmassa: <p>Just another point here.<br>
A <code>nn.Module</code> is actually a OO wrapper around the functional interface, that contains a number of utility methods, like <code>eval()</code> and <code>parameters()</code>, and it automatically creates the parameters of the modules for you.<br>
you can use the functional interface whenever you want, but that requires you to define the weights by hand. Here is an example <a href="https://github.com/szagoruyko/wide-residual-networks/tree/master/pytorch">https://github.com/szagoruyko/wide-residual-networks/tree/master/pytorch</a></p>

Harry_Zhi: <p>Thanks Miguelvr, I get your points.<br>
nn. Dropout is exactly the layers needed to be added into the neural networks, while functional.dropout will not behave differents in train and eval. mode.</p>

Harry_Zhi: <p>Thansk for the explanation.</p>
<p>It seems using exsiting nn. modules is better choice for building networks while nn.functional are basic building blocks of those layers.</p>
<p>If some custom layers need be defined, then nn.functional may be used.</p>

mratsim: <p><a class="mention" href="/u/harry_zhi">@Harry_Zhi</a> You can have the functional <code>Dropout</code> be aware of training/eval mode with <code>F.dropout(x, training=self.training)</code></p>

Royi: <p><a class="mention" href="/u/mratsim">@mratsim</a>, Your comment is very important one.</p>
<p>Would it b correct to say that if you don’t need your layer parameters to be optimized, define them using the Functional Class?</p>

Rafael_Valle: <p>Yeah, that can be done manually as well.<br>
For things that do not change between training/eval like sigmoid, relu, tanh, I think it makes sense to use functional; for others like dropout, I think it’s better to not use functional and use the module instead such that you get the expected behavior when calling model.eval() or model.train()</p>

ashish_arora: <p><a class="mention" href="/u/rafael_valle">@Rafael_Valle</a> ,<br>
Assuming nn.dropout2d is always a better choice as compared to F.dropout, if  we want to use dropout in a usual way (things change between training/eval).<br>
Is there a situation when F.dropout is preferred over nn.dropout2d.</p>

Rafael_Valle: <p>In Tacotron 2 dropout is used in the decoder input during training and inference. This is one example where one can use F.dropout,  assuming it has the same behavior on model.train() and model.eval().</p>
<p>I’m assuming from pytorch’s dropout API <code>torch.nn.functional.dropout(input, p=0.5, training=False, inplace=False)</code>, that it doesn’t automatically change if one calls net.train() and net.eval() with functional dropout inside the model.</p>
<p>I would be good to have one of the pytorch devs commenting on this, for example <a class="mention" href="/u/apaszke">@apaszke</a>.</p>

apaszke: <p>You can use the functional API and have it work with train/eval mode by doing this:</p>
<pre><code class="lang-auto">def forward(self, x):
    y = ...
    return F.dropout(y, training=self.training)
</code></pre>
<p>However, as the usual advice goes, I think it’s clearer to use modules for stateful function (in this case dropout can be considered stateful, because of this flag), and functional for everything else.</p>

Royi: <p>Performance wise, is there any preference between the 2?</p>

tian_wen: <p>Unhan…You did not give the “dropout rate”…</p>

mratsim: <p>The <code>Module</code> module of PyTorch is built on top of functional so there is a bit more overhead but it is completely dwarfed by the time spent on computing linear, convolutions, RNNs and other layers. (We’re speaking into seconds vs minutes/hours/days here)</p>

Ancalagon: <p>“There is a dropout layer missed” not because you used nn.functional in forward, but because you commented out the nn.Dropout in <strong>int</strong><br>
If you uncomment that in <strong>init</strong>, the Dropout Layer still appears in print(model), even if in forward you used F.dropout</p>
<p>In Pytorch, print(model) gives whatever defined in <strong>init</strong>, not used in forward.</p>

rustagiadi95: <p>I was having a lot of confusion on this as well. Although the nn.modules layers give us the .train() and .eval() flexibilities, but if we initialize an activation or pooling operation as layer having its weight, then I would like to know that how giving parameters to these operations help in achieving the goal for which we have the model for?</p>
<p>This question is bumping in my mind a lot and I would like to end it once and for all so please help.<br>
Thanks,</p>