Nested Modules in PyTorch and the parameters update

sinaabdollahi: <p>Suppose that I defined a class as follows:</p>
<pre><code class="lang-auto">class A(nn.Module):
    def __init__(self, some_params):
          #SOMETHING HERE (for example several Linear)
    def forward(self, x):
          #SOMETHING ELSE HERE
</code></pre>
<p>Now, suppose I want to use an array of class A in another model. Does the following way work?</p>
<pre><code class="lang-auto">class B(nn.Module):
    def __init__(self, some_params):
          self.a = [ ]
          for i in range(100):
                 self.a.append(A(some_params))
    def forward(self, x):
          outs = [ ]
          for i in range(100):
                 out = self.a(x[ i ])
                 outs.append(out)
           outs_concat = outs[0]
           for i in range(1,100):
                 outs_concat = torch.cat((outs_concat, outs[ i ]), 1)
</code></pre>
<p>Moreover, would someone please tell me how can I be sure about the parameters? Do the parameters in module A act as paramters of module B? Because I want to update all parameters when I call optimizer and I do not know how to become sure about updating all parameters.<br>
I am looking forward to see your solutions.<br>
Regards,<br>
Sina</p>

ptrblck: <p>This won’t work, as your <code>A</code> module won’t be registered in a plain Python list, so you would have to use a <code>nn.ModuleList</code> instead.<br>
This will make sure to register the passed module as well as all parameters in the parent module.<br>
Calling <code>B.parameters()</code> will thus yield all parameters.<br>
You could double check it via <code>print(dict(modelB.named_parameters()))</code>.</p>

sinaabdollahi: <p>Thanks.<br>
Would you please tell me how can I iterate ModuleList in forward method?<br>
Thanks</p>

ptrblck: <p>You can iterate it like a plain Python <code>list</code>. It’s a drop-in replacement, which makes sure to register all parameters.</p>

sinaabdollahi: <p>Whenever I use the following code in “forward” method of class B, it does not work for multi-instances:</p>
<pre><code class="lang-auto">for i in range(10):
    out = self.a[i](X[i])
</code></pre>
<p>My mean is that suppose the input array be in shape (batch_size, 5, 10) and batch_size = 3<br>
then, when I input the input array into instance of class B, it will rise an error of array index out of bound.</p>
<pre><code class="lang-auto">b = B()
y_hat = b(input_array[index : index + batch_size])
</code></pre>
<p>Would you please help me to know how to work with batches in this case?</p>

ptrblck: <p>I assume you would like to feed the batch to the first layer and then pipe the output through all others?<br>
If so, then this code snippet should work:</p>
<pre><code class="lang-python">out = self.a[0](X)
for i in range(1, 10):
    out = self.a[i](out)
</code></pre>

sinaabdollahi: <p>Actually, each instance (sample) of X has different features and each feature must feed to each self.a[ i ] separately, then I would like to concatenate all of their outputs.</p>

ptrblck: <p>What’s the shape of <code>X</code>?<br>
If it’s <code>[batch_size, nb_features=10]</code>, then this should work:</p>
<pre><code class="lang-python">out = []
for i in range(10):
    out.append(self.a[i](X[:, i])
out = torch.stack(out)
</code></pre>

sinaabdollahi: <p>In this example, we have (batch_size, 5, 10) input.<br>
It means that I have batch_size samples, 5 features, and each feature is a vector of size 10.<br>
Now, I want to feed each feature i to its corresponding self.a[ i ].</p>