Calculation for the input to the Fully Connected Layer

mehra-deepak: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/1/b17d29dac866154ccf5825481c991eefba8f864e.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/b17d29dac866154ccf5825481c991eefba8f864e" title="doubt"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/1/b17d29dac866154ccf5825481c991eefba8f864e_2_690x388.jpeg" alt="doubt" data-base62-sha1="pk8FMa9O49YWp9NpcnKN03p92AC" width="690" height="388" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/1/b17d29dac866154ccf5825481c991eefba8f864e_2_690x388.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/b/1/b17d29dac866154ccf5825481c991eefba8f864e_2_1035x582.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/b/1/b17d29dac866154ccf5825481c991eefba8f864e.jpeg 2x" data-dominant-color="91908E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">doubt</span><span class="informations">1280×720 102 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Do we always need to calculate this 64<em>4</em>4 manually using formula, i think there might be some optimal way of finding the last features to be passed on to the Fully Connected layers otherwise it could become quiet cumbersome to calculate for thousands of layers.<br>
Right now im doing it manually for every layer like first calculating the dimension of images then calculating the output of convolved layer then maxpool layer and so on to find out the last features.<br>
Guide me for optimal ways. Please.</p>

Nikronic: <p>Hi,</p>
<p>Let’s say you have already implemented your model, but you do not know what is the proper input number of neurons for linear layer.</p>
<p>The manual approach would be like this:<br>
You can run your model and add a <code>print(x.shape)</code> in <code>forward</code> function right after <code>self.pool</code>. It will print final shape something like <code>[batch, channel, height, width]</code>. So, you just multiply them as the number of inputs.</p>
<p>About automatic way, you can do this way:</p>
<pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self):
        ...

    def forward(self, x):
        ...
        x = ...  # let's say x is the output of self.pool
        self.linear_input_size = x.view(-1, x.shape[-1]).shape[0]
        ...

</code></pre>
<p>This <a href="https://discuss.pytorch.org/t/linear-layer-input-neurons-number-calculation-after-conv2d/28659/2">thread</a> may be able to help.</p>
<p>bests</p>

mehra-deepak: <p>Thank you very Much <a class="mention" href="/u/nikronic">@Nikronic</a> this helps a lot it took me almost 2 days to find a solution something like this <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

Nikronic: <p>You are welcome! I hope you solve other possible issues much faster.</p>
<p>Good luck</p>

mehra-deepak: <p>Thanks <a class="mention" href="/u/nikronic">@Nikronic</a> also i wanted  like can i use nn.AdaptiveAveragePool1D(argument) as an alternative to it , im not sure if i can althoguh i will use what u suggested.<br>
Thank You.</p>

Nikronic: <p>You are absolutely right, that is another possible solution!<br>
Although, as in your post, the model contains a <code>nn.MaxPool2d</code>, you may want to use same pooling layer.<br>
On top of that, before the <code>Linear</code> layer, the model is still using images so 2d pooling is needed.</p>
<aside class="quote no-group" data-username="mehra-deepak" data-post="5" data-topic="82774">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mehra-deepak/48/30471_2.png" class="avatar"> mehra-deepak:</div>
<blockquote>
<p>nn.AdaptiveAveragePool1D(argument)</p>
</blockquote>
</aside>
<p>So, The alternative is <code>nn.AdaptiveMaxPool2d</code>. See its <a href="https://pytorch.org/docs/stable/nn.html#adaptivemaxpool2d" rel="noopener nofollow ugc">documentation</a>.</p>
<p>The benefit of using <em>Adaptive</em> pooling layer is that you explicitly define your desired output size so not matter what input size is, model always will produce tensors with the identical shape. Also, it is has been used in official PyTorch implementation of ResNet models right before <code>Linear</code> layer. Please see <a href="https://discuss.pytorch.org/t/1-channel-resnet50-not-working-with-different-input-sizes/81803/8">this post</a>.</p>

mehra-deepak: <p>Thanks <a class="mention" href="/u/nikronic">@Nikronic</a> brother u are such an awesome person i must say, u helped me in a great way and u don’t know how much respect i have for you.<br>
Please bear with me on this PyTorch Journey.<br>
Thank You. <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=9" title=":smile:" class="emoji" alt=":smile:"></p>

Nikronic: <p>Thank you so much! I really did not expect that. I am really appreciated.<br>
Actually, I am trying to do same (at least) thing other great members/developers of PyTorch have done to me.</p>
<p>Feel free to ask your questions whenever you have any issues!</p>
<p>Bests</p>

mehra-deepak: <p>hey <a class="mention" href="/u/nikronic">@Nikronic</a> so i was trying to implement it the way u told me but im kind of stuck like how can i pass the output from maxpool layers just the way u told me.<img src="https://discuss.pytorch.org/uploads/default/original/3X/4/1/41a744253941420cef8734a4f7d057adc9e81f94.png" alt="1st" data-base62-sha1="9mNoUj3B33VjkLGCfK0Wxx4LSV6" width="669" height="363"></p>
<p>how can i write it further more , like conventionally i should have written inside</p>
<p>self.fc1 = nn.Linear(64<em>4</em>4 ,500 ) but since i don’t want to calculate the image dimension 4* 4 manually as i said</p>

Nikronic: <p>Using which one of those methods?</p>
<ol>
<li>Printing shape</li>
<li>Extracting shape by passing one arbitrary tensor</li>
<li>Using <code>AdaptiveMaxPool2d</code>
</li>
</ol>

mehra-deepak: <p>like what to add in nn.Linear() in the above code such that it automatically calculates the 64 * 4 * 4 for me  in the first fully connected layer</p>

Nikronic: <p>If your input images are have same size, I go with print method. It will just take few seconds and as the input size is same all the time, you do not need to change the input to linear layer. You can also make sure that input is always in a particular size using <code>transforms.Resize</code> class.</p>
<pre><code class="lang-auto">class Test(nn.Module):
    def __init__(self, pool_size=(4, 4)):
        super(Test, self).__init__()
       
        self.conv1 = nn.Conv2d(3, 16, 3, padding=1)
        self.conv2 = nn.Conv2d(16, 32, 3, padding=1)
        self.conv3 = nn.Conv2d(32, 64, 3, padding=1)
        self.maxpool = nn.MaxPool2d(2, 2)
        self.fc1 = nn.Linear(_____, 500)  # fill it after one run of model
        self.fc2 = nn.Linear(500, 10)
   
    def forward(self, x):
        out = F.relu(self.conv1(x))
        out = F.relu(self.conv2(out))
        out = F.relu(self.conv3(out))
        out = self.maxpool(out)
        print(out.shape)  # use this to fill input to linear layer
        out = out.view(out.size(0), -1)
        out = self.fc1(out)
        out = self.fc2(out)
        return out


model = Test()
x = torch.randn(1, 3, 32, 32)
model(x)
</code></pre>
<p>But if you are using input with different sizes and do not want to resize them into a identical shape, I prefer <code>AdaptiveMaxPool</code>. In this case:</p>
<pre><code class="lang-auto">class Test(nn.Module):
    def __init__(self, pool_size=(4, 4)):
        super(Test, self).__init__()
       
        self.conv1 = nn.Conv2d(3, 16, 3, padding=1)
        self.conv2 = nn.Conv2d(16, 32, 3, padding=1)
        self.conv3 = nn.Conv2d(32, 64, 3, padding=1)
        self.maxpool = nn.AdaptiveMaxPool2d(pool_size)
        self.fc1 = nn.Linear(64*pool_size[0]*pool_size[1], 500)
        self.fc2 = nn.Linear(500, 10)
   
    def forward(self, x):
        out = F.relu(self.conv1(x))
        out = F.relu(self.conv2(out))
        out = F.relu(self.conv3(out))
        out = self.maxpool(out)
        out = out.view(out.size(0), -1)
        out = self.fc1(out)
        out = self.fc2(out)
        return out


model = Test()
x = torch.randn(1, 3, 32, 32)
model(x)

</code></pre>
<p>Also it would be great if you could put the codes within two pair of ```</p>

mehra-deepak: <p>Thnaks <a class="mention" href="/u/nikronic">@Nikronic</a> for the same u are so kind i’m so sorry i could not reply earlier i had an interview earlier for a job which i lost unfortunately but thanks for all the hardships u did for me  i owe you very much.</p>

Nikronic: <p>Thank you,<br>
Don’t worry, you fill find an appropriate job soon. Just do not give up!</p>

mehra-deepak: <p><a class="mention" href="/u/nikronic">@Nikronic</a> btw do u know about Fellowship Program i think u can the right candidate for same or is there anyway where i can DM you like Linkedin etc ?</p>

Arohan_Ajit: <p>Another good way to know the input dimensions of the image is to use <code>torchsummary</code> module for <code>summary</code> library.</p>
<pre><code class="lang-auto">import torch.nn as nn
import torch.nn.functional as F
from torchsummary import summary

# define the CNN architecture
class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        ## Define layers of a CNN
        self.conv1 = nn.Conv2d(3,16,3,padding=1) 
        self.conv2 = nn.Conv2d(16,32,3,padding=1)
        self.conv3 = nn.Conv2d(32,64,3,padding=1)
        self.conv4 = nn.Conv2d(64,128,3,padding=1)
        self.conv5 = nn.Conv2d(128,256,3,padding=1)

        self.batch_norm1 = nn.BatchNorm2d(16)
        self.batch_norm2 = nn.BatchNorm2d(32)
        self.batch_norm3 = nn.BatchNorm2d(64)
        self.batch_norm4 = nn.BatchNorm2d(128)
        self.batch_norm5 = nn.BatchNorm2d(256)
        
        
        self.pool = nn.MaxPool2d(2)
        self.dropout = nn.Dropout(p=0.2)

        self.fc1 = nn.Linear(256*7*7,1000,bias=True)
        self.fc2 = nn.Linear(1000,133)
        
    
    def forward(self, x):
        ## Define forward behavior
        x = F.relu(self.conv1(x))
        x = self.batch_norm1(x)
        x = self.pool(x)

        x = F.relu(self.conv2(x))
        x = self.pool(x)
        x = self.batch_norm2(x)

        x = F.relu(self.conv3(x))
        x = self.pool(x)
        x = self.batch_norm3(x)

        x = F.relu(self.conv4(x))
        x = self.pool(x)
        x = self.batch_norm4(x)

        x = F.relu(self.conv5(x))
        x = self.pool(x)
        x = self.batch_norm5(x)

        x = x.view(-1,256*7*7)
        x = self.dropout(x)
        x = F.relu(self.fc1(x))
        x = self.dropout(x)
        x = self.fc2(x)


        return x
</code></pre>
<p>Then you can use library as follows:</p>
<pre><code class="lang-auto">model_scratch = Net().to(use_cuda)
print(summary(model_scratch,(3,224,224)))
</code></pre>
<p>Which will generate a tensorflow type output highlighting changes after each layer. Something like:</p>
<pre><code class="lang-auto">----------------------------------------------------------------
        Layer (type)               Output Shape         Param #
================================================================
            Conv2d-1         [-1, 16, 224, 224]             448
       BatchNorm2d-2         [-1, 16, 224, 224]              32
         MaxPool2d-3         [-1, 16, 112, 112]               0
            Conv2d-4         [-1, 32, 112, 112]           4,640
         MaxPool2d-5           [-1, 32, 56, 56]               0
       BatchNorm2d-6           [-1, 32, 56, 56]              64
            Conv2d-7           [-1, 64, 56, 56]          18,496
         MaxPool2d-8           [-1, 64, 28, 28]               0
       BatchNorm2d-9           [-1, 64, 28, 28]             128
           Conv2d-10          [-1, 128, 28, 28]          73,856
        MaxPool2d-11          [-1, 128, 14, 14]               0
      BatchNorm2d-12          [-1, 128, 14, 14]             256
           Conv2d-13          [-1, 256, 14, 14]         295,168
        MaxPool2d-14            [-1, 256, 7, 7]               0
      BatchNorm2d-15            [-1, 256, 7, 7]             512
          Dropout-16                [-1, 12544]               0
           Linear-17                 [-1, 1000]      12,545,000
          Dropout-18                 [-1, 1000]               0
           Linear-19                  [-1, 133]         133,133
================================================================
Total params: 13,071,733
Trainable params: 13,071,733
Non-trainable params: 0
----------------------------------------------------------------
Input size (MB): 0.57
Forward/backward pass size (MB): 22.51
Params size (MB): 49.86
Estimated Total Size (MB): 72.95
----------------------------------------------------------------
</code></pre>

mehra-deepak: <aside class="quote no-group" data-username="Arohan_Ajit" data-post="16" data-topic="82774">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/b77776/48.png" class="avatar"> Arohan_Ajit:</div>
<blockquote>
<pre><code class="lang-auto">  [-1, 16, 224, 224]              32
         MaxPool2d-3         [-1, 16, 112, 112]               0
            Conv2d-4         [-1, 32, 112, 112]           4,640
         MaxPool2d-5       
</code></pre>
</blockquote>
</aside>
<p>Hey <a class="mention" href="/u/arohan_ajit">@Arohan_Ajit</a> good to see you it’s me Deepak Mehra do u remember haha world is just too small</p>

Arohan_Ajit: <p>Haha hi <a class="mention" href="/u/mehra-deepak">@mehra-deepak</a>, I actually thought it maybe you, saw your name while going through pytorch weekly summary in mail and I looked into what it was. Cheers mate!</p>

mehra-deepak: <p>so good to see u answer my query that beautifully only <a class="mention" href="/u/nikronic">@nikronic</a> had stated me the soultion otherwise i had asked a lot of professional all said its manual haha u seem like a pro developer now good thing so native vibes</p>

Arohan_Ajit: <p>Lol, not really, I’m still exploring. Someone once told me and I’m always looking to forwards some tips and tricks to friends</p>