How does torch module list work?

desert_ranger: <p>I am trying to understand how torch module list works. I slightly modified the example from the official documentation here -</p>
<h1>
<a name="taken-from-httpspytorchorgdocsstablegeneratedtorchnnmodulelisthtml-1" class="anchor" href="#taken-from-httpspytorchorgdocsstablegeneratedtorchnnmodulelisthtml-1"></a>Taken from <a href="https://pytorch.org/docs/stable/generated/torch.nn.ModuleList.html" class="inline-onebox" rel="noopener nofollow ugc">ModuleList — PyTorch 1.13 documentation</a>
</h1>
<pre><code class="lang-auto">class MyModule(nn.Module):
    def __init__(self):
        super(MyModule, self).__init__()
        self.linears = nn.ModuleList([nn.Linear(10, 10) for i in range(10)])

    def forward(self, x):
        # ModuleList can act as an iterable, or be indexed using ints
        for i, l in enumerate(self.linears):
            # Why would PyTorch give such a hard example?
            # x = self.linears[i // 2](x) + l(x)
            x = self.linears(x)
        return x
</code></pre>
<p>If I understand correctly, the tensor x is sent through 10 hidden layers as per the definition of self.linears. What is the <code>for</code> loop in <code>forward</code> doing?</p>
<p>Perhaps I am understanding the whole thing wrong?</p>

srishti-git1110: <p>Hi,<br>
Your modification won’t work as you probably expect (that is, passing the tensor x through each layer) as ModuleList does not have a forward method implemented so self.linears(x) won’t work.</p>
<p>You might want to use <code>x = l(x)</code> to simplify the example given in the docs.</p>
<p>As for the example given in docs, I think it’s that way to probably demonstrate both ways of using the layers in a <code>ModuleList</code> - accessing the layers by indexing the <code>ModuleList</code> just how a plain python list can be indexed, and accessing by enumerating the layers.</p>

desert_ranger: <p>Thank you for the correction. I have made the changes here -</p>
<pre><code class="lang-auto"># Taken from https://pytorch.org/docs/stable/generated/torch.nn.ModuleList.html
import torch
import torch.nn as nn

class MyModule(nn.Module):
    def __init__(self):
        super(MyModule, self).__init__()
        self.linears = nn.ModuleList([nn.Linear(10, 10) for i in range(10)])

    def forward(self, x):
        # ModuleList can act as an iterable, or be indexed using ints
        for i, l in enumerate(self.linears):
            print("i = ", i)
            print("l = ", l)
            # Why would PyTorch give such a hard example?
            # x = self.linears[i // 2](x) + l(x)
            x = l(x)
        return x

x = torch.randn(5, 10)
print("x = ", x.shape)
model = MyModule()
y = model(x)
</code></pre>
<p>I have a few follow-up questions to clarify my understanding -</p>
<ol>
<li>Does <code>self.linears = nn.ModuleList([nn.Linear(10, 10) for i in range(10)])</code> help create a 10 layer neural network?</li>
<li>Does the following code iterate over this 10 layer neural network 10 times -</li>
</ol>
<pre><code class="lang-auto"> for i, l in enumerate(self.linears):

            # Why would PyTorch give such a hard example?
            # x = self.linears[i // 2](x) + l(x)
            x = l(x)
</code></pre>
<p>Actually, the entire purpose of asking this is to understand the code given <a href="https://github.com/aladdinpersson/Machine-Learning-Collection/blob/master/ML/Pytorch/more_advanced/transformer_from_scratch/transformer_from_scratch.py" rel="noopener nofollow ugc">here</a>. The author wrote code for the original Transformer architecture from scratch. I am unable to understand the following lines -</p>
<pre><code class="lang-auto">
self.layers = nn.ModuleList(
            [
                TransformerBlock(
                    embed_size,
                    heads,
                    dropout=dropout,
                    forward_expansion=forward_expansion,
                )
                for _ in range(num_layers)
            ]
        )
</code></pre>

srishti-git1110: <p>Hi, sure.</p>
<aside class="quote no-group" data-username="desert_ranger" data-post="3" data-topic="169102">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/desert_ranger/48/31726_2.png" class="avatar"> desert_ranger:</div>
<blockquote>
<p>Does <code>self.linears = nn.ModuleList([nn.Linear(10, 10) for i in range(10)])</code> help create a 10 layer neural network?</p>
</blockquote>
</aside>
<p>Yes, you can use it that way.</p>
<aside class="quote no-group" data-username="desert_ranger" data-post="3" data-topic="169102">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/desert_ranger/48/31726_2.png" class="avatar"> desert_ranger:</div>
<blockquote>
<p>Does the following code iterate over this 10 layer neural network 10 times -</p>
</blockquote>
</aside>
<p>No, only once.<br>
This code passes the tensor x through subsequent layers of the network to produce a final prediction from the last layer.<br>
In fact, this code is using the ModuleList as a 10-layered NN.</p>
<p>In the transformer code, <code>num_layers</code> number of transformer blocks are being stored in a ModuleList.</p>
<p>A module list is very similar to a plain python list and is meant to store nn.Module objects just how a plain python list is used to store int, float etc. objects. The purpose for having ModuleList is to ensure that the parameters of the layers it holds are registered properly.<br>
The layers it contains aren’t connected in any way.</p>