Model parameters not shown in model.parameters() generator unless modules initialized in constructor

kirk86: <p>Is there a way to define a model without initializing every module in the  constructor?<br>
It seems that this way <code>model.parameters()</code> is an empty list and would not work for the optimizer.<br>
Is there a way for this to work dynamically calling modules in the <code>forward</code> method?</p>
<pre><code class="lang-auto">class TransformerEncoder(nn.Module):

    def __init__(self):
        super().__init__()
        self.linear = nn.Linear(3, 32, bias=False)

    def forward(self, x):
        # x = nn.Linear(x.shape[1], 32, bias=False)(x)
        x = self.linear(x)
        x = nn.Dropout(0.1)(x)
        x = nn.Linear(32, 64, bias=False)(x)
        x = nn.Dropout(0.01)(x)
        x = nn.Linear(64, 128)(x)
        x = nn.ReLU()(x)
        x = nn.Linear(128, 32)(x)
        x = nn.Dropout(0.3)(x)

        return x
</code></pre>
<pre><code class="lang-auto">model = TransformerEncoder()
list(model.parameters())
[Parameter containing:
 tensor([[-0.4792,  0.5095,  0.0218],
         [-0.4992, -0.0950, -0.1015],
         [-0.4775,  0.4182,  0.3103],
         [-0.5134, -0.3572, -0.4396],
         [ 0.0881, -0.0877, -0.2812],
         [ 0.3770,  0.4974,  0.4990],
         [ 0.0478, -0.3706, -0.1895],
         [-0.3901, -0.3802,  0.3413],
         [ 0.4509, -0.0291, -0.1843],
         [ 0.3662,  0.1571, -0.3094],
         [ 0.3420,  0.2866,  0.2167],
         [ 0.5187,  0.2042, -0.5334],
         [-0.0020,  0.2633,  0.4540],
         [ 0.5211,  0.4651,  0.5746],
         [ 0.3727,  0.5121,  0.1946],
         [ 0.4460, -0.0212, -0.2560],
         [ 0.1041,  0.1689, -0.2890],
         [-0.1101, -0.0870,  0.0905],
         [ 0.1388,  0.3151, -0.3811],
         [ 0.3124, -0.4644,  0.2603],
         [ 0.4458, -0.1589,  0.4804],
         [-0.2198, -0.3407,  0.1858],
         [ 0.3404, -0.1001, -0.1115],
         [-0.5361,  0.5403, -0.1926],
         [-0.1439, -0.2009,  0.2738],
         [-0.3845, -0.5399,  0.3298],
         [-0.1042,  0.0114,  0.2496],
         [ 0.3915, -0.0160, -0.2393],
         [ 0.0127,  0.1600,  0.1321],
         [-0.2192,  0.1396,  0.5114],
         [-0.4857,  0.2361, -0.5637],
         [-0.1308, -0.1101, -0.2792]], requires_grad=True)]
</code></pre>

ptrblck: <p>This is expected behavior since you are not registering the modules.<br>
The newly created layers won’t be trained and will use randomly initialized parameters.<br>
You could assign these layers to <code>self</code> in the forward and add a condition which would call the already initialized modules in future iterations, but you would also need to perform a forward pass before passing the parameters to the optimizer. I don’t fully understand your use case here. If you don’t want to device the features, use <code>nn.Lazy*</code> modules.</p>

kirk86: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="196136">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>This is expected behavior since you are not registering the modules</p>
</blockquote>
</aside>
<p>Shouldn’t the modules in the forward be registered automatically? Is there any specific reason why we have to register them in the constructor?<br>
This way we have to write twice the same thing, first create a module instance in the constructor and then call that object instance in the forward.</p>
<p>The naive question is why not create and call the object at the same time <code>nn.Linear(fan_in, fan_out)(x)</code>?</p>
<p>Or, something like a context manager?</p>
<pre><code class="lang-auto">    def forward(self, x):
        with torch.registered_modules():
            x = nn.Linear(x.shape[1], 32, bias=False)(x)
            x = nn.Linear(32, 64, bias=False)(x)
        x = nn.Dropout(0.01)(x)
        x = nn.Linear(64, 128)(x)
        x = nn.ReLU()(x)
        x = nn.Linear(128, 32)(x)
        x = nn.Dropout(0.3)(x)

        return x
</code></pre>
<p>Then, the <code>model.parameters()</code> will have of 2 linear modules</p>
<aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="196136">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>You could assign these layers to <code>self</code> in the forward and add a condition which would call the already initialized modules in future iterations, but you would also need to perform a forward pass before passing the parameters to the optimizer.</p>
</blockquote>
</aside>
<p>
You’re insinuating that there’s a way to achieve this but didn’t quite get it, could you elaborate with a MWE?
</p>
<p>
Another aspect where what I’m proposing might be interesting is the following scenario.<br>
You want a model that works with any input but in order to construct that you’d need to dynamically create/construct the model based on the input data dimensions. But, what if you don’t know a priori  the shape of your data? For instance, when we register modules in the constructor you have to know a priori the shape of your data to decide the <code>fan_in</code>, <code>fan_out</code> variables, but, in the <code>forward</code> you can directly get that from whatever shape is the input <code>x</code> we pass. So, having modules being registered in the forward might make sense?
</p>
<pre><code class="lang-auto">class SomeModel(nn.Module):

def __init__(self, config):
        super().__init__()
        self._config = config
        self.linear1 = nn.Linear(fan_in, fan_out) # Here we need to know a priori the fan_in dim

    def forward(self, x):
        self.linear1 = nn.Linear(x.shape[-1], fan_out) # Instead, here we directly get it from the data
</code></pre>
<p>I guess <code>LazyLinear</code> is one solution to infering input shapes. Wasn’t aware of it, thanks for bringing it up. It’s a nice addition hope it becomes more mainstream and the <code>warning</code> about being a developmental feature slowly fades away in the future.</p>

ptrblck: <aside class="quote no-group" data-username="kirk86" data-post="3" data-topic="196136">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kirk86/48/151_2.png" class="avatar"> kirk86:</div>
<blockquote>
<p>Shouldn’t the modules in the forward be registered automatically?</p>
</blockquote>
</aside>
<p>No and I wouldn’t know how it should work. E.g. which attribute names should be used?<br>
You can explicitly register modules in the <code>forward</code> as already explained, but you are not doing it either.</p>
<aside class="quote no-group" data-username="kirk86" data-post="3" data-topic="196136">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kirk86/48/151_2.png" class="avatar"> kirk86:</div>
<blockquote>
<p>Is there any specific reason why we have to register them in the constructor?</p>
</blockquote>
</aside>
<p>Simplicity, as you can finish the setup (including the optimizer) before executing the model.</p>
<aside class="quote no-group" data-username="kirk86" data-post="3" data-topic="196136">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kirk86/48/151_2.png" class="avatar"> kirk86:</div>
<blockquote>
<p>This way we have to write twice the same thing, first create a module instance in the constructor and then call that object instance in the forward.</p>
</blockquote>
</aside>
<p>Splitting code into an initialization and execution creates a clean and understandable code. It e.g. allows you to reuse modules easily and is the standard approach in PyTorch.</p>
<aside class="quote no-group" data-username="kirk86" data-post="3" data-topic="196136">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kirk86/48/151_2.png" class="avatar"> kirk86:</div>
<blockquote>
<p>The naive question is why not create and call the object at the same time <code>nn.Linear(fan_in, fan_out)(x)</code>?</p>
</blockquote>
</aside>
<p>Already explained:</p>
<aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="196136">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>The newly created layers won’t be trained and will use randomly initialized parameters.</p>
</blockquote>
</aside>

kirk86: <aside class="quote no-group" data-username="ptrblck" data-post="4" data-topic="196136">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>You can explicitly register modules in the <code>forward</code> as already explained, but you are not doing it either</p>
</blockquote>
</aside>
<p>Can you point me to a reference or example of that. Where should I look?</p>