Beginner: Should ReLU/sigmoid be called in the __init__ method?

biggusbackpropagatus: <p>I am trying to rebuild a Keras architecture in pytorch, which looks like this</p>
<pre><code class="lang-auto">    rnn_layer1 = GRU(25) (emb_seq_title_description)
    # [...]
    main_l = Dropout(0.1)(Dense(512,activation='relu') (main_l))
    main_l = Dropout(0.1)(Dense(64,activation='relu') (main_l))
    
    #output
    output = Dense(1,activation="sigmoid") (main_l)
</code></pre>
<p>So I tried to adjust the basic RNN example in pytorch and add ReLUs to the Linear layers.  However, I am not sure if I can call ReLU directly in the forward method or should call it in the <strong>init</strong> method.</p>
<p>My first try looks like this:</p>
<pre><code class="lang-auto">import torch.nn as nn

class RNN(nn.Module):
    def __init__(self, input_size, hidden_size, output_size):
        super(RNN, self).__init__()
        self.hidden_size = hidden_size
        self.i2h = nn.Linear(input_size + hidden_size, hidden_size)
        self.i2o = nn.Linear(input_size + hidden_size, output_size)


    def forward(self, input, hidden):
        combined = torch.cat((input, hidden), 1)
        hidden = nn.ReLU(self.i2h(combined))
        output = nn.ReLU(self.i2o(combined))
        output = nn.sigmoid(output)
        return output, hidden

    def initHidden(self):
        return torch.zeros(1, self.hidden_size)

n_hidden = 128
n_letters = 26
n_categories = 2
rnn = RNN(n_letters, n_hidden, n_categories)
</code></pre>
<p>However, when I look at the rnn object in python, I will not see the ReLUs, so maybe it’s not right to call nn.ReLU directly in the forward method…</p>
<pre><code class="lang-auto">RNN(
  (i2h): Linear(in_features=154, out_features=128, bias=True)
  (i2o): Linear(in_features=154, out_features=2, bias=True)
)
</code></pre>

ptrblck: <p>Since <code>nn.ReLU</code> is a class, you have to instantiate it first. This can be done in the <code>__init__</code> method or if you would like in the <code>forward</code> as:</p>
<pre><code class="lang-auto">hidden = nn.ReLU()(self.i2h(combined))
</code></pre>
<p>However, I would create an instance in <code>__init__</code> and just call it in the <code>forward</code> method.</p>
<p>Alternatively, you don’t have to create an instance, because it’s stateless, and could directly use the functional API in <code>forward</code>:</p>
<pre><code class="lang-auto">hidden = F.relu(...)
</code></pre>

quazi: <p>let me repeat the question differently: what is the PyTorch-idiomatic way to use the relu() and WHY? I think the answer is to use F.relu() in the forward() function. The WHY part is important here and I’d love to hear a full answer. Hope I’m not complicating things more than necessary.</p>

ptrblck: <p>My personal preference is to use the functional API in the forward for stateless objects, e.g. <code>F.relu</code>.<br>
Since <code>nn.ReLU</code> doesn’t store any parameters it’s not really necessary to define it using the module way.</p>
<p>However, there is one exception when I prefer the module init and that’s when I know I will try out different activation functions in the forward pass.<br>
So instead of changing the <code>F.relu</code> to another non-linearity repeatedly, I use a definition like in this example:</p>
<pre><code class="lang-python">class MyModel(nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
        self.conv1 = nn.Conv2d(3, 6, 3, 1, 1)
        self.act = nn.ReLU()
        
    def forward(self, x):
        x = self.act(self.conv1(x))
        return x
</code></pre>
<p>and just switch <code>self.act</code> for another module.</p>
<p>I think it just comes down to your personal preference, since I think neither of both coding approaches is bad in any way.</p>

quazi: <p>my point isn’t really about coding style per se, but more about correct PyTorch semantics. The motivation for my followup question is more like the o.p. : What goes into <strong>init</strong>() and what goes into forward()? as I found that confusing as well, especially coming from Keras. I think it boils down to wether a layer (module??) holds state or not as you put it. Maybe there are other reasons I have not learned yet??. For example, you can even add new layers in the forward(), e.g. F.conv2d(…), but that layer weights are lost once the forward() exits, so it can’t be used as part of the trained model for example at prediction time. So the semantics -not the syntax- of what goes into <strong>init</strong>() vs what [should] go into forward() is the main point of the question.</p>

ptrblck: <p>OK, I see.<br>
Let me try to boil it down to the standard approach and possible advantages of changing this approach.</p>
<p>In the standard use case, you are writing some kind of model with some layers. The layers hold most likely some parameters which should be trained. <code>nn.Conv</code> would be an example. On the other side some layers don’t have any trainable parameters like <code>nn.MaxPool</code>. However, usually also these are created in <code>__init__</code>. Other “layers” don’t have parameters and can also be seen as simple functions instead of a proper layer (at least in my opinion) like <code>nn.ReLU</code>.<br>
In your <code>forward</code> method you are creating the logic of your forward (and thus also backward) pass.<br>
In a very simple use case you would just call all created layers one by one passing the output of one layer to the other. However, since the computation graph is created dynamically, you can also create some crazy forward passes, e.g. using a random number to select a repetition of some layers, split your input into different parts and call different “sub modules” with each part, use some statistics of your input to select a specific part of your model, etc. You are not bound to any convention during this as long as all shapes etc. match.<br>
For me this is one of the most beautiful parts of PyTorch. Basically you can let your imagination flow without worrying too much about some limited API which can only call layers in a sequential manner.<br>
And this is also the reason to break some of these conventions I’ve mentioned before.</p>
<p>Think about a specific use case where you would like to use a conv layer, but for whatever reason you need to access and maybe manipulate its weight often. The first approach of creating the layer in <code>__init__</code> and applying it in <code>forward</code> would certainly work. However, the weight access might be a bit cumbersome.<br>
So how about we just store the filter weights as <code>nn.Parameters</code> in <code>__init__</code> and just use the functional API (<code>F.conv2d</code>) in the <code>forward</code> method. Would that work at all? Sure! Since you’ve properly registered the filter weights in <code>__init__</code>, they will be trained as long as they are used somewhere in the computation during your forward pass.<br>
As you can see these are somewhat advanced use cases and I wouldn’t say they are breaking some kind of PyTorch semantics. Using the functional API is totally fine for advanced use cases. I would not recommend to use the functional API for every layer from now on. It’s much easier to use <code>nn.Modules</code> in most use cases.</p>
<p>Have a look at the implementations of <a href="https://github.com/pytorch/vision/tree/master/torchvision/models">torchvision.models</a>.<br>
You will see all kind of different coding styles depending on the complexity of the problem. While simpler models might be implemented using some <code>nn.Sequential</code> blocks in <code>__init__</code> and just calling them in <code>forward</code> (e.g. <a href="https://github.com/pytorch/vision/blob/master/torchvision/models/alexnet.py">AlexNet</a>, other models will be implemented in a different manner using more functional calls (e.g. <a href="https://github.com/pytorch/vision/blob/master/torchvision/models/inception.py">Inception</a> since it’s a bit more complicated to split and merge the activations as well as getting the aux loss).</p>
<p>To sum it up, my two cents are: use whatever feels good and easy for you. Although there are some “standard” approaches for some use cases, I have to say that even after working with PyTorch for a while now, I probably change the one or other coding style every few weeks (because I suddenly have the feeling the code logic is easier to follow using this new approach <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=6" title=":wink:" class="emoji" alt=":wink:"> ). If you ask 10 devs to implement <code>Inception</code>, you’ll probably get 10 different but all beautiful and useful implementations.</p>

quazi: <p>Thank you for the time and effort in providing a great answer. Now that I understand your advice more deeply I think I gained a lot more intuition into PyTorch. Much appreciated.</p>

alex.veuthey: <aside class="quote no-group" data-username="ptrblck" data-post="6" data-topic="18689">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>So how about we just store the filter weights as <code>nn.Parameters</code> in <code>__init__</code> and just use the functional API ( <code>F.conv2d</code> ) in the <code>forward</code> method. Would that work at all? Sure! Since you’ve properly registered the filter weights in <code>__init__</code> , they will be trained as long as they are used somewhere in the computation during your forward pass.</p>
</blockquote>
</aside>
<p>What about the following use case: I need a convolution layer (therefore learnable, unlike <code>nn.MaxPool</code> which you already mentionned) where the <code>kernel_size</code> parameter is the shape <code>[height, width]</code> of the input tensor, which is unknown at <code>__init__</code> time. This results in a pooling effect, so that the output tensor has a <code>NxCx1x1</code> shape.</p>
<p>Is it still possible to implement this, using the <code>__init__</code> and <code>forward</code> behaviour?</p>
<p>(Thinking out loud: could this be replaced by a generic convolution in the <code>__init__</code> and a global pooling layer in the <code>forward</code> method?)</p>

ptrblck: <p>If you have static input shapes, that would be possible, but that would also be kind of the “standard approach” initializing the conv layer just a bit later.<br>
I guess you are dealing with variable sized inputs and would like to use the corresponding conv kernel “on the fly”.<br>
In that case it’s a bit trickier, since there are probably some approaches, which would have some shortcomings.</p>
<ol>
<li>
<p>You could pass your new input to your model and the shape would be checked in <code>forward</code>. If the corresponding kernel was already created, just use it for processing the input and training. Else create this kernel and do the same.<br>
Depending on the variety of your input shapes this approach would create a lot of kernels, which are maybe hardly being trained. E.g. if a specific size occurs only once in your training data, this kernel will just be used once in the training. Also what should happen, if you’ll get new shapes in your validation/test dataset?</p>
</li>
<li>
<p>Use a single kernel and manipulate it, i.e. increase/decrease somehow its spatial size. In that case I’m not sure how the kernel will behave if you have large differences between the input sizes. The filter will not only see the features in a completely other resolution, but also if you just append some random values to the filter or cut the filter, the weights will have some “discrepancy”, e.g. while the center was already trained, the borders are completely random weights. I’m not sure, if it would be a good idea to interpolate the kernel weights to another size based on the input size, but that seems to me like a valid approach to try out.</p>
</li>
</ol>
<p>If you just need global pooling, have a look at the <a href="https://pytorch.org/docs/stable/nn.html#adaptivemaxpool2d" rel="nofollow noopener">adaptive pooling layers</a>.</p>

alex.veuthey: <p>Thanks for the very detailed answer <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=6" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>Actually, it’s more of a “convenience” hack than anything, since the inputs have static size. However, the architecture of the network might change, and the shape would be dependent on these changes, since the “pooling-conv” layer is just before the last layer (FC). Also, it might be required to use the network with a database of different dimensions.</p>
<p>Having this shape computed “on the fly” would save me from having to calculate it manually (i.e. by failing a forward pass and checking the dimensions, or <em>actually</em> thinking about convolution padding, strides and their effect on the output shape) when either change occurs.</p>
<p>I’ve settled on the following approach for now, since the inputs have static size: run a partial <code>forward</code> on a tensor (optional argument) in the <code>__init__</code>, get the resulting size, and initialize the “pooling-conv” layer accordingly. This way I can recompute the value as necessary.</p>
<p>Some trial and error is still required, but that will do for the moment!</p>

one_way: <p>Could I use the nn.Con2d Module directly in forward function with out stating in <em>init</em> ?  e.g.</p>
<pre><code class="lang-python">class MyModel(nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
        self.act = nn.ReLU()
        
    def forward(self, x):
        x= nn.Conv2d(3, 6, 3, 1, 1)(x)
        x = self.act(x)
        return x

</code></pre>
<p>Could this still be learnable while training?</p>

ptrblck: <p>You would reinitialize the module including its parameters in each forward pass, so that you would at most update them once then overwrite them in the next forward pass.</p>

one_way: <p>Oh, Thanks!<br>
so the learnable parameters should be held in the _init_ part within MyModel. So during the forward and backward process, these parameters could be updated progressively.  So in this way, MyModel can be either <strong>written in this way</strong>:</p>
<pre><code class="lang-python">class MyModel(nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
        self.conv_like = nn.convlike() # which has its own hidden parameters 
        
    def forward(self, x):
        x = self.conv_like(x)
</code></pre>
<p><strong>or written in this way:</strong></p>
<pre><code class="lang-auto">class MyModel(nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
        self.func_params = params # which will be used in nn.functional.funs 
        
    def forward(self, x):
        x = nn.functional.funs(x,self.func_params)
</code></pre>
<p>other nn.funs or nn.functional.funs which do not have hidden learnable parameters could be placed wherever you like.<br>
is it correct?</p>

ptrblck: <p>Yes, your explanations and assumptions are correct.</p>

mohit117: <aside class="quote no-group" data-username="ptrblck" data-post="6" data-topic="18689">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>So how about we just store the filter weights as <code>nn.Parameters</code> in <code>__init__</code> and just use the functional API ( <code>F.conv2d</code> ) in the <code>forward</code> method. Would that work at all? Sure! Since you’ve properly registered the filter weights in <code>__init__</code> , they will be trained as long as they are used somewhere in the computation during your forward pass.</p>
</blockquote>
</aside>
<p>Thankyou very much for this use case. I in essence understand what is being illustrated in these lines. But because i am still a novice in coding, can a very quick code be given so that i can understand how a weights of conv can be set as parameter in <code>init</code> and then using <code>F.conv</code> they will be updated. Thankyou for this seemingly redundant need but helps me a lot.</p>

ptrblck: <p>This code shows how to initialize a conv in the <code>__init__</code> method and use its parameters in the <code>forward</code>:</p>
<pre><code class="lang-python">class MyModel(nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
        conv = nn.Conv2d(1, 6, 3)
        self.weight = conv.weight
        self.bias = conv.bias
        
    def forward(self, x):
        x = F.conv2d(x, self.weight, self.bias)
        return x

x = torch.randn(1, 1, 4, 4)
model = MyModel()
out = model(x)
</code></pre>
<p>You would have to add potential padding via <code>F.pad</code>, if it’s used in your conv layer.</p>

mohit117: <p>Thankyou very much. So if I modify the code as follows,</p>
<pre><code class="lang-auto">
class MyModel(nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
        conv = nn.Conv2d(1, 6, 3)
        self.weight = conv.weight
        self.bias = conv.bias

    def modify_conv_weights(self, weights, bias):
        # DO SOME OPERATIONS
        return weights, bias
        
    def forward(self, x):
        x = F.conv2d(x, self.weight, self.bias)
       self.weight, self.bias = modify_conv_weights(self.weight, self.bias)
        return x

x = torch.randn(1, 1, 4, 4)
model = MyModel()
out = model(x)
</code></pre>
<p>Then this way I will be able to modify the the weights of the conv layer or in each forward pass of the model the new weights will have no correspondence to previous value?</p>
<p>Yours sincerely</p>

ptrblck: <p>Depending on the operations in <code>modify_conv_weights</code>, you might break the computation graph and just reassign new parameters, so that the <code>weight</code> and <code>bias</code> would not be trained by the optimizer.</p>

mohit117: <p>Oh I see. If the operation is a function which operates on incoming weights say taking log of the weights or say normalising them such as L2 norm is 1, I hope that does not break the computation graph?</p>

ptrblck: <p>These operations should create a non-leaf parameter, and the <code>.grad</code> attribute would disappear.<br>
In the end you’ve replaced the original parameter which was used to compute the output activation, so that the backward pass won’t be able to compute the valid gradients anymore, no?</p>
<p>Could you explain your use case a bit?</p>