How to automatically get in-features from nn.conv2d to nn.linear

username: jnx
cooked: <p>Hi,</p>
<p>Is there any way that we don’t have to declare in-features of an nn.linear module explicitly? I am using Conv2d modules and I know that we need to use .view to flatten the values but can it be automatically done by extracting the incoming tensor shapes?</p>

username: ptrblck
cooked: <p>The <code>in_features</code> depend on the shape of your input, so what could be done is to set the input shape as an argument, pass a random tensor through the conv layers, get the shape and initialize the linear layers using this shape.<br>
That’s basically the automatic way of passing an input through your model, print the shape right before the linear layer and set the right <code>in_features</code> afterwards. At least that’s my workflow, if I’m too lazy to calculate the shape. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=6" title=":wink:" class="emoji" alt=":wink:"></p>

username: Kalle
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="27385">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p><code>in_features</code></p>
</blockquote>
</aside>
<p>I am currently dealing with this. Do you have an example of how your workflow is of what you are suggesting (“At least that’s my workflow, if im too lazy to calculate the shape”)?</p>

username: ptrblck
cooked: <p>Here is a simple example of this workflow:</p>
<pre><code class="lang-python"># initial setup
class MyModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv = nn.Conv2d(3, 6, 3, 1, 1)
        self.linear = nn.Linear(1, 10) # initialize with any value
        
    def forward(self, x):
        x = self.conv(x)
        x = x.view(x.size(0), -1)
        # unknown shape here as I don't want to calculate it manually
        print(x.shape)
        x = self.linear(x) # will raise a shape mismatch, but that's OK
        return x

model = MyModel()
x = torch.randn(1, 3, 224, 224)
out = model(x)
# &gt; RuntimeError: mat1 and mat2 shapes cannot be multiplied (1x301056 and 1x10)

# Now change the in_features to 301056 
class MyModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv = nn.Conv2d(3, 6, 3, 1, 1)
        self.linear = nn.Linear(301056, 10)
        
    def forward(self, x):
        x = self.conv(x)
        x = x.view(x.size(0), -1)
        x = self.linear(x)
        return x

model = MyModel()
out = model(x) # works
</code></pre>
<p>In the first iteration I’m expecting a shape mismatch if the shape calculation and will refine the <code>in_features</code> later.<br>
Of course one could calculate the output shape of the single conv layer manually, but this is just used as an example in case the previous layers are more complicated.<br>
In the latest PyTorch releases you could also use the “lazy” layers e.g. <code>nn.LazyLinear</code> which don’t expect the <code>in_features</code> anymore.</p>

username: Eta_C
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a><br>
Does lazy definition work?</p>
<pre><code class="lang-python">class Model(nn.Module):
    def __init__(self, num_classes):
        super().__init__()
        self.conv = nn.Conv2d(3, 6, 3, 1, 1)
        self.linear = None 
        self.num_classes = num_classes

    def forward(self, x):
        x = self.conv(x)
        x = x.view(x.size(0), -1)
        if self.linear is None:
            self.linear = nn.Linear(x.size(1), self.num_classes)
        x = self.linear(x)
        return x 
</code></pre>

username: ptrblck
cooked: <p>Your approach would generally work, but you would have to be careful about creating the optimizer as the parameters are not fully initialized during the instantiation of the model.<br>
E.g. this standard approach would be wrong:</p>
<pre><code class="lang-python">model = Model()
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)
</code></pre>
<p>since <code>model.parameters()</code> does not yet contain the parameters of <code>self.linear</code>.<br>
You would thus have to use:</p>
<pre><code class="lang-python">model = Model()
out = model(dummy_input)
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)
</code></pre>
<p>The same limitation applies to the usage of <code>Lazy*</code> modules as described <a href="https://pytorch.org/docs/stable/generated/torch.nn.modules.lazy.LazyModuleMixin.html#torch.nn.modules.lazy.LazyModuleMixin">here</a>.</p>
<p>For your manual approach you would also have to take care of pushing the module to the <code>device</code> after the first forward pass, which should be done automatically using the <code>Lazy*</code> modules.</p>

username: Kalle
cooked: <p>Thanks for your reply <a class="mention" href="/u/ptrblck">@ptrblck</a>.</p>
<p>However, I am still wondering how this is automatically?</p>
<pre><code class="lang-auto">x = torch.randn(1, 3, 224, 224)

# Now change the in_features to 301056 
class MyModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv = nn.Conv2d(3, 6, 3, 1, 1)
        self.linear = nn.Linear(301056, 10)
        
    def forward(self, x):
        x = self.conv(x)
        x = x.view(x.size(0), -1)
        x = self.linear(x)
        return x

model = MyModel()
out = model(x) # works
</code></pre>
<p>Then if I change the input it fails:</p>
<pre><code class="lang-auto">x = torch.randn(1, 3, 128, 128)
</code></pre>
<p>And to make it work I have to manually change self.linear to:</p>
<pre><code class="lang-auto">self.linear = nn.Linear(98304, 10)
</code></pre>
<p>Is this the expected behaviour?</p>
<p>Is the following the right approach if I want to make it automatically? Or is there a better approach?</p>
<pre><code class="lang-auto">class MyModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv = nn.Conv2d(3, 6, 3, 1, 1)
        
    def forward(self, x):
        x = self.conv(x)
        linear1 = nn.Linear(x.size(1), 10)
        x = linear1(x)
        return x
</code></pre>

username: ptrblck
cooked: <p>It’s not an automatic approach and I’m not sure if auto-correct changed the phrasing back in 2018 or what I meant back then.<br>
In any case, you would have these approaches:</p>
<ul>
<li>calculate the output activation shapes and set the right values</li>
<li>run a dummy forward pass, print the activation shapes, and set the expected values</li>
<li>use the <code>Lazy*</code> layers to avoid setting the <code>in_features</code> manually</li>
</ul>
<p>Your approach would have the same shortcomings described before. I.e. be careful when to pass the parameters to the optimizer and make sure the parameters are moved to the right device.</p>

username: Kalle
cooked: <p>Thanks for your answer <a class="mention" href="/u/ptrblck">@ptrblck</a>! I think I found a solution that can work from <a href="https://github.com/yjlolo/vae-audio/blob/3a43e3122da55daf89fffaf5bbf11fdb805a597f/model/model.py#L205" rel="noopener nofollow ugc">here</a>:</p>
<pre><code class="lang-auto">def _infer_flat_size(self, model):
    model = nn.Sequential(*model)
    model_output = model(torch.ones(1, *self.input_size))
    return int(np.prod(model_output.size()[1:]))  
</code></pre>
<p>So using this function to calculate the size and use it to set the <strong>in_features</strong>.</p>

username: Kalle
cooked: <p>Hi again <a class="mention" href="/u/ptrblck">@ptrblck</a>.</p>
<p>So I went with the option: <em>run a dummy forward pass, print the activation shapes, and set the expected values</em>.</p>
<pre><code class="lang-auto">def _infer_flat_size(self, model):
    model = nn.Sequential(*model)
    model_output = model(torch.ones(1, *self.input_size))
    return int(np.prod(model_output.size()[1:])) 
</code></pre>
<p>However, then I tried an experiment with a setup (fixed seeds) where I know I can reproduce the exact same result. Here I did two different experiments:</p>
<ol>
<li>One where I initialized the <em><strong>in_features</strong></em> with a specific value (e.g. 301056).</li>
<li>One where I used the <em><strong>_infer_flat_size function()</strong></em> to calculate the value to automatically set it.</li>
</ol>
<p>But then I get two different training_loss, training_accuracy etc. How come? Is it because of the following line?</p>
<pre><code class="lang-auto">    model_output = model(torch.ones(1, *self.input_size))
</code></pre>

username: ptrblck
cooked: <p>In case you are using e.g. batchnorm layers, the dummy forward pass should be performed in <code>model.eval()</code> mode. Otherwise the fake inputs would be used to update the running stats of all batchnorm layers.<br>
Could this be the case?</p>

username: Kalle
cooked: <p>I made a simplified version of my code below. Here I added the model.eval() part as you suggested <a class="mention" href="/u/ptrblck">@ptrblck</a> (or atleast I think this is how you suggested). However, I still don’t see the models being reproducible and I am wondering why that is? Is there something different I should add?</p>
<pre><code class="lang-auto">class Network(nn.Module):
    def __init__(self, input_size):
        # calling constructor of parent class
        super(Network, self).__init__()
        self.input_size = input_size
        model = []
        model += [
            nn.BatchNorm2d(1),
            nn.Conv2d(in_channels=1, out_channels=50, kernel_size=(5, 5), stride=1, padding=0),
            nn.ReLU(inplace=True),
            nn.BatchNorm2d(num_features=50),
            nn.Flatten()
        ]
        # Use this method or use lazyLinear (newer pytorch) that dont need in_features as input
        self.flat_size = self._infer_flat_size(model)
        model += [
            nn.Linear(in_features=self.flat_size, out_features=450),
            nn.ReLU(inplace=True),
            nn.Linear(in_features=450, out_features=10)
        ]
        self.model = nn.Sequential(*model)

    def _infer_flat_size(self, model):
        model = nn.Sequential(*model)
        model.eval()
        model_output = model(torch.ones(1, *self.input_size))
        return int(np.prod(model_output.size()[1:]))

    def forward(self, x):
        return self.model(x)
</code></pre>

username: my3bikaht
cooked: <aside class="quote no-group" data-username="Kalle" data-post="13" data-topic="27385">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/aeb1de/48.png" class="avatar"> Kalle:</div>
<blockquote>
<p><code>self.input_size = input_size</code></p>
</blockquote>
</aside>
<ol>
<li>kernel 5 with padding 0 will eat two rows and two columns from each side away in convolution, so after first conv2d your image dimension will be (H-4, W-4)</li>
<li>out_channels = 50</li>
<li>thus your nn.Linear input_features after flatten will be (H-4) * (W-4) * out_channels or  (input_size.shape[2] - 4) * (input_size.shape[3] - 4) * out_channels (==50)</li>
</ol>
<p>also what <a class="mention" href="/u/ptrblck">@ptrblck</a> is proposing is much easier. Fill input_channels with some arbitrary value, run dummy pass (I prefer torchinfo.summary in this case) and let torch to calculate these values for you in form of error)</p>

username: Kalle
cooked: <p>The network is not meant to work <a class="mention" href="/u/my3bikaht">@my3bikaht</a>, but just to show how I use the model.eval() and infer_flat_size function. So I was not expecting the model to actually work. I just randomly inserted some numbers.</p>
<p>What you are suggesting, can you give an example of this?</p>

username: my3bikaht
cooked: <pre><code class="lang-auto">class Network(nn.Module):
    def __init__(self):
        super().__init__()
        self.model = nn.Sequential(
            nn.Conv2d(1, 50, kernel_size=5, stride=1, padding=0),
            nn.ReLU(inplace=True),
            nn.BatchNorm2d(50),
            nn.Flatten(),
            nn.Linear(9999, 450),
            nn.ReLU(inplace=True),
            nn.Linear(450, 10)
        ])
def forward(self, x):
        return self.model(x)
</code></pre>
<pre><code class="lang-auto">import torchinfo
model = Network()
torchinfo.summary(model, (32,1,224,224))

</code></pre>
<p>just run it like this once and torch will give you error what shape it expects for the nn.Linear layer</p>
<p>or, knowing input shape, you can calculate output shape (I prefer padding=‘same’ for these situations)</p>
<pre><code class="lang-auto">class Network(nn.Module):
    def __init__(self, input_shape=(224,224), conv_out_channels=50):
        super().__init__()
        shape_after_conv = input_shape[0]*input_shape[1]*conv_out_channels
        self.model = nn.Sequential(
            nn.Conv2d(1, conv_out_channels, kernel_size=5, stride=1, padding='same'),
            nn.ReLU(inplace=True),
            nn.BatchNorm2d(50),
            nn.Flatten(),
            nn.Linear(shape_after_conv, 450),
            nn.ReLU(inplace=True),
            nn.Linear(450, 10)
        ])
def forward(self, x):
        return self.model(x)
</code></pre>
<p>it’s not like your data will have diferent shapes every day. Even traditional ‘default’ models like resnets have predefined input shape like 224x224 and tinkering with it is possible but there’s very little reason since it is pretrained for this exact shape</p>