Using a module outside a module (very confused)

Gears_Gears: <p>Hi! I’m looking for guidance on defining a network with modules inside functions<br>
Beginner tutorials say says to do (1):</p>
<pre><code class="lang-auto">class network(torch.nn.Module)
    and then self.network = torch.nn.Sequential(
           layer 1
           layer 2)
    def forward(x)
        return self.network(x)
# and to train this model, do
x = network(x)
# and then loss.backward ect.
</code></pre>
<p>But I found another way (2) on Github</p>
<pre><code class="lang-auto">def calculate(x)
  # either:
  return torch.nn.somelayer()(x)
  # or: 
  return torch.nn.functional.thatlayer(x)
# and then 
x = calculate(x)
# and the loss.backward stuff
</code></pre>
<p>In this simple example, are (1) and (2) doing the same thing? Is (2) going to train and learn just like (1)<br>
In particular, I’m worried/wondering that if a layer L doesn’t the ‘keep layers in a super class of torch.nn.module’ and ‘thatsuperclass.forward(x),’ it won’t be part of the network (i.e. the parameters of L will only used as a calculator but won’t get trained/updated</p>

ptrblck: <aside class="quote no-group" data-username="Gears_Gears" data-post="1" data-topic="188042">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/gears_gears/48/60932_2.png" class="avatar"> Gears_Gears:</div>
<blockquote>
<p>In this simple example, are (1) and (2) doing the same thing?</p>
</blockquote>
</aside>
<p>It depends on the used module. If it’s stateless (i.e. the module does not contain trainable parameters or buffers such as <code>nn.ReLU()</code>) both approaches would result in the same output even though creating the module before calling it via <code>nn.ReLU()(x)</code> looks wasteful so use the functional API instead via <code>F.relu(x)</code>.<br>
However, if the modules contain parameters and buffers you would either recreate the module in the second approach via: <code>nn.Linear(10, 10)(x)</code> and would thus never train it, which is wrong, or you would need to explicitly pass registered parameters and buffers to the functional API via <code>F.linear(x, weight, bias)</code>.</p>

Gears_Gears: <p>Thanks very much for your help ptrblck! I was reading up on the documentations<br>
To to make sure I understand, is there any chance you could review my understanding of the following?<br>
My motivation is looking at this <a href="https://github.com/katerakelly/pytorch-maml/blob/master/src/omniglot_net.py" rel="noopener nofollow ugc">PyTorch implementation of MAML algorithm</a>, where the author writes something like this:</p>
<pre><code class="lang-auto"># layers.py helper file
def conv2d(input, weight, bias=None, stride=1, padding=0, dilation=1, groups=1):
    return F.conv2d(input, weight.cuda(), bias.cuda(), stride, padding, dilation, groups)
# omnigolot.py network object file
from layers import conv2d
class Omnigolot(torch.nn.Module):
    # something something
    def __init__():
       self.network = torch.nn.Sequential([ "something" ])
    def forward(self, x, weights):
       x = self.network(x) # if there aren't any weights
       x = conv2d(x, weights) # if there are weights
</code></pre>
<p>So in this case, when we do <code>loss.backward()</code>:</p>
<pre><code class="lang-auto">result = self.network(x) # &lt;- parameters in the layers are being updated (there's learning going on)
result = conv2d(x, weights) # &lt;- this is just a one-time calculation. The layer just holds the weights to do calculate 'result'
</code></pre>
<p>Is this interpretation correct?</p>
<p>Many Thanks!</p>

ptrblck: <p>Yes, your interpretation is not wrong, but let me clarify a few things.</p>
<pre><code class="lang-python">result = self.network(x) # &lt;- parameters in the layers are being updated (there's learning going on)
</code></pre>
<p>Using the <code>self.network</code> module does not automatically guarantee its parameters will be updated.<br>
You would still need to calculate the gradients w.r.t. the used parameters and pass these parameters also to an optimizer which can then update them in its <code>optimizer.step()</code> call.<br>
The <code>nn.Module</code> just holds the parameters and uses them in its <code>forward</code> method in the same way your custom <code>Omnigolot</code> module creates submodules, parameters, etc. and uses these in its <code>forward</code>.</p>
<pre><code class="lang-python">result = conv2d(x, weights) # &lt;- this is just a one-time calculation. The layer just holds the weights to do calculate 'result'
</code></pre>
<p><code>conv2d</code> is not a “layer” but just a function in your example. The <code>weights</code> seem to be passed to it and you would need to make sure they were passed to an <code>optimizer</code> and will be updated.</p>
<p>Let me know if you have more questions. The <a href="https://pytorch.org/tutorials/beginner/nn_tutorial.html">“What is <code>torch.nn</code> really” tutorial</a> might also be interesting.</p>

Gears_Gears: <p>I’m grateful for your detailed reply and explanation! I will read the tutorial to learn more.<br>
Also, I want to say that your replies to many other questions on this forum are really helpful as well; they unstuck-ed me many times</p>