nn.ModuleList not working as I think it should

oafolabi: <p>I created a network with linear layers using moduleList, however when I run forward the shape of the output of each layer does not correspond to what I would expect. Concretely, I used the code below to create the network:</p>
<pre><code class="lang-auto">class decoder_with_fc_only(nn.Module):
    
    def __init__(self, latent_size,layer_sizes, non_linearity=nn.ReLU):
        super(decoder_with_fc_only, self).__init__()
        n_layers  = len(layer_sizes)
        if n_layers &lt; 2:
            raise( ValueError('For an FC decoder with single a layer use simpler code.'))
            
        size_list = [latent_size] + layer_sizes[0:-1]
        list_len = len(size_list)
        self.linear_layers = nn.ModuleList([non_linearity(nn.Linear(size_list[i-1], size_list[i]))  \
                                             for i in range(1,list_len) ])
        
        self.final_layer = nn.Linear(layer_sizes[-2], layer_sizes[-1])
 
        
    def forward(self,x):
        for i, l in enumerate(self.linear_layers):
            x =  self.linear_layers[i](x)
        x = self.final_layer(x)
        x = torch.reshape(x, (-1,5000, 3))
        return x
</code></pre>
<p>Upon construction, I pass in arguments latent size = 256, and layer_sizes = [256, 256, 512, 1025 ,15000].</p>
<p>The layers are then<br>
layer1 with size (256 x 256)<br>
layer2 with size (256 x 256)<br>
layer3 with size (256 x 512)<br>
layer4 with size (512 x 1025)<br>
final_layer with size (1025 x 15000)</p>
<p>I would then expect that with batch size = 32 and an input of size 256, the output of the layers are<br>
[32x256, 32x256, 32x512, 32x1025, 32x15000].</p>
<p>However, the sizes I get are<br>
[32x256, 32x256, 32x256, 32x256, -].<br>
The - is there since the final_layer won’t work due to incompatible input size.</p>
<p>However, with the following code, it works:</p>
<pre><code class="lang-auto">class decoder_with_fc_only(nn.Module):
    
    def __init__(self, latent_size,layer_sizes, non_linearity=nn.ReLU):
        super(decoder_with_fc_only, self).__init__()
        n_layers  = len(layer_sizes)
        if n_layers &lt; 2:
            raise( ValueError('For an FC decoder with single a layer use simpler code.'))
            
        size_list = [latent_size] + layer_sizes[0:-1]
        list_len = len(size_list)
        self.linear_layers = nn.ModuleList([nn.Linear(size_list[i-1], size_list[i])  \
                                            for i in range(1,list_len) ])
        self.final_layer = nn.Linear(layer_sizes[-2], layer_sizes[-1])

    def forward(self,x):
        for i, l in enumerate(self.linear_layers):
            x = F.relu(l(x))
        x = self.final_layer(x)
        x = torch.reshape(x, (-1,5000, 3))
        return x
</code></pre>
<p>I’m curious to see if anyone else has encountered this problem or can spot the bug. Thanks!</p>

SimonW: <aside class="quote no-group" data-username="oafolabi" data-post="1" data-topic="66773">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/o/e5b9ba/48.png" class="avatar"> oafolabi:</div>
<blockquote>
<p>non_linearity(nn.Linear(size_list[i-1], size_list[i]))</p>
</blockquote>
</aside>
<p>your <code>non_linearity</code> is a module <code>nn.ReLU</code> whose first constructor arg is <code>inplace</code>, a boolean flag. To chain modules together, use <code>nn.Sequential</code> instead.</p>