Transition from Conv2d to Linear Layer Equations

username: DanBenton
cooked: <p>Hi everyone,<br>
First post here. Having trouble finding the right resources to understand how to calculate the dimensions required to transition from conv block, to linear block. I have seen several equations which I attempted to implement unsuccessfully:</p>
<hr>
<ol>
<li>“The formula for output neuron:<br>
Output = ((I-K+2P)/S + 1), where<br>
I - a size of input neuron,<br>
K - kernel size,<br>
P - padding,<br>
S - stride.”</li>
</ol>
<p>and</p>
<ol start="2">
<li>“𝑊′=(𝑊−𝐹+2𝑃/𝑆)+1”</li>
</ol>
<hr>
<p>The example network that I have been trying to understand is a CNN for CIFAR10 dataset</p>
<p><strong>Below is the third conv layer block, which feeds into a linear layer w/ 4096 as input:</strong></p>
<pre><code>        # Conv Layer block 3
        nn.Conv2d(in_channels=128, out_channels=256, kernel_size=3, padding=1),
        nn.BatchNorm2d(256),
        nn.ReLU(inplace=True),
        nn.Conv2d(in_channels=256, out_channels=256, kernel_size=3, padding=1),
        nn.ReLU(inplace=True),
        nn.MaxPool2d(kernel_size=2, stride=2),
    )


    self.fc_layer = nn.Sequential(
        nn.Dropout(p=0.1),
        nn.Linear(4096, 1024),
        nn.ReLU(inplace=True),
        nn.Linear(1024, 512),
        nn.ReLU(inplace=True),
        nn.Dropout(p=0.1),
        nn.Linear(512, 10)
    )
</code></pre>
<p>I need to figure out the equations/resources/protocol to calculate this transition between Conv and linear. How did we arrive at 4096?</p>
<p>EDIT: I have also used ptrblck’s print-layer (below) for help, but still struggle to understand this transition intuitively.</p>
<p>class Print(nn.Module):<br>
def forward(self, x):<br>
print(x.size())<br>
return x</p>
<p>Any and all help greatly appreciated, Dan</p>
<p>cited:</p>
<ol>
<li><a href="https://discuss.pytorch.org/t/linear-layer-input-neurons-number-calculation-after-conv2d/28659" class="inline-onebox">Linear layer input neurons number calculation after conv2d</a></li>
<li><a href="https://datascience.stackexchange.com/questions/40906/determining-size-of-fc-layer-after-conv-layer-in-pytorch" rel="nofollow noopener">https://datascience.stackexchange.com/questions/40906/determining-size-of-fc-layer-after-conv-layer-in-pytorch</a></li>
</ol>

username: ptrblck
cooked: <p>Your output formula is missing the dilation and also the subtraction from the kernel size.<br>
The <a href="https://pytorch.org/docs/stable/generated/torch.nn.Conv2d.html#torch.nn.Conv2d">Conv2d docs</a> show you the formula which is used.</p>
<p>That being said, your printed conv layer block would keep the spatial dimensions equal in the first layers, since conv layers with a kernel size of 3 and padding of 1 would not reduce the height or width of the activation. The max pooling layer would halve the spatial dimensions.</p>
<p>Based on the <code>out_channels=256</code> I thus assume that the input to the conv block would be <code>[batch_size, 128, 8, 8]</code>, the output thus <code>[batch_size, 256, 4, 4]</code>, which would be flattened to <code>[batch_size, 256*4*4=4096]</code>.</p>
<p>The <a href="http://deeplearning.net/software/theano/tutorial/conv_arithmetic.html">Convolution arithmetic tutorial</a> is a very good website to learn more about how convolution layers perform the window sliding.</p>

username: RaLo4
cooked: <p>Here is some demo-code where I have put in comments and prints to try and explain this as easy as possible:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.nn.functional as F

class CNN(torch.nn.Module):
    def __init__(self):
        super(CNN, self).__init__()
            
        #image-tensor goes in as batch_sizex3x32x32
        #print-1 will show this state
        
        self.conv1 = nn.Conv2d(3, 16, kernel_size=3, stride=1, padding=1, bias=False)
        self.bn1 = nn.BatchNorm2d(16)
        
        #image-tensor is batch_sizex16x32x32 since: (32-3+2*1)/1+1=32
        #print-2 will show this state
        
        self.conv2 = nn.Conv2d(16, 32, kernel_size=3, stride=2, padding=1, bias=False)
        self.bn2 = nn.BatchNorm2d(32)
        
        #image-tensor is batch_sizex32x16x16 since: (32-3+2*1)/2+1=16
        #print-3 will show this state
        
        self.max_pool = nn.MaxPool2d(kernel_size=2, stride=2)
        
        #image-tensor is batch_sizex32x8x8
        #print-4 will show this state

        #now we flatten image tensor to batch_sizex32*8*8 which is batch_sizex2048
        #print-5 will show this state
        self.fc1 = nn.Linear(32 * 8 * 8, 10) #same as: self.fc1 = nn.Linear(2048, 10)
            
            
    def forward(self, x):
        print("print-1:")
        print(x.shape)
        x = F.relu(self.bn1(self.conv1(x)))
        print("print-2:")
        print(x.shape)
        x = F.relu(self.bn2(self.conv2(x)))
        print("print-3:")
        print(x.shape)
        x = self.max_pool(x)
        print("print-4:")
        print(x.shape)
        x = x.view(-1, 32 * 8 * 8)
        print("print-5:")
        print(x.shape)
        x = self.fc1(x)
        return x
    
model = CNN()
x = torch.ones(4, 3, 32, 32)
model(x)
</code></pre>
<p><strong>Out:</strong></p>
<pre><code class="lang-auto">print-1:
torch.Size([4, 3, 32, 32])
print-2:
torch.Size([4, 16, 32, 32])
print-3:
torch.Size([4, 32, 16, 16])
print-4:
torch.Size([4, 32, 8, 8])
print-5:
torch.Size([4, 2048])
</code></pre>

username: RaLo4
cooked: <p>If it is still unclear you should really check out some tutorials on this topic.<br>
Like the one ptrblck suggested:</p>
<aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="93850">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>The <a href="http://deeplearning.net/software/theano/tutorial/conv_arithmetic.html" rel="noopener nofollow ugc">Convolution arithmetic tutorial</a> is a very good website to learn more about how convolution layers perform the window sliding.</p>
</blockquote>
</aside>
<p>Or check out <a href="http://cs231n.stanford.edu/slides/2020/lecture_9.pdf" rel="noopener nofollow ugc">this very popular lecture</a> from Stanford University</p>

username: DanBenton
cooked: <p>Thank you both for the help, rly appreciate it</p>

username: Andrei_Cristea
cooked: <p>Below is a cheeky little utility that helps you avoid doing all the tedious math by piggy-backing on torch’s error messages. The assumption is that <strong>net</strong> is your Module which is composed of a sequence of input shape-agnostic layers (such as Conv &amp; ReLU &amp; BatchNorm &amp; MaxPool layers), followed at some point by a flattening and a linear layer whose required size you’re trying to figure out. <strong>height</strong> and <strong>width</strong> refer to your desired input image shape. Hope this is useful!</p>
<pre><code class="lang-auto">try:
    net.forward(torch.rand((1, 3, height, width)))
    print("Image size is compatible with layer sizes.")
except RuntimeError as e:
    e = str(e)
    if e.endswith("Output size is too small"):
        print("Image size is too small.")
    elif "shapes cannot be multiplied" in e:
        required_shape = e[e.index("x") + 1:].split(" ")[0]
        print(f"Linear layer needs to have size: {required_shape}")
    else:
        print(f"Error not understood: {e}")
</code></pre>