PyTorch neural network parameters and tensor shapes

Machupicchu: <p>I am following this tutorial: <a href="https://pytorch.org/tutorials/beginner/blitz/neural_networks_tutorial.html" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/blitz/neural_networks_tutorial.html</a></p>
<p>it’s quite nicely done, however I do not understand/see where you can know the expected image input size for the small network they have defined. They say that the images must be of size 32x32.</p>
<p>I know there is a formula (given in <a href="https://www.youtube.com/watch?v=bNb2fEVKeEo&amp;list=PL3FW7Lu3i5JvHM8ljYj-zLfQRF3EO8sYv&amp;index=6&amp;t=0s" rel="nofollow noopener">stanford CS231n</a> ) that states that ((N-K)/strideLen) +1 = , where N is the (padded) input size and K is the 2D convolution kernel size (e.g. 3x3).</p>
<p>But that does not seem to help me deduce what I want.</p>
<p>How can I know the expect input size ? I would be happy if someone whats to go in more details for the example of this simple PyTorch conv nn.</p>
<p>(Please note that for sizes significanlty different that 32x32 the net outputs an error message. Also, I don’t understand why it accepts inputs like 30x30 and 31x31, e.g. this is accepted:</p>
<pre><code class="lang-auto">input = torch.randn(1, 1, 31, 31)
out = net(input)
print(out)
</code></pre>
<p>The code is:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.nn.functional as F

class Net(nn.Module):

    def __init__(self):
        super(Net, self).__init__()
        # 1 input image channel, 6 output channels, 3x3 square convolution
        # kernel
        self.conv1 = nn.Conv2d(1, 6, 3)
        self.conv2 = nn.Conv2d(6, 16, 3)
        # an affine operation: y = Wx + b
        self.fc1 = nn.Linear(16 * 6 * 6, 120)  # 6*6 from image dimension
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

    def forward(self, x):
        # Max pooling over a (2, 2) window
        x = F.max_pool2d(F.relu(self.conv1(x)), (2, 2))
        # If the size is a square you can only specify a single number
        x = F.max_pool2d(F.relu(self.conv2(x)), 2)
        x = x.view(-1, self.num_flat_features(x))
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

    def num_flat_features(self, x):
        size = x.size()[1:]  # all dimensions except the batch dimension
        num_features = 1
        for s in size:
            num_features *= s
        return num_features


net = Net()
print(net)
</code></pre>
<p>It is not explained where from the shape (16 * 6 * 6, 120) comes from, for the linear layers, e.g. why 120 ?</p>

ptrblck: <p>Based on the description in CS231n, we know, that a conv layer with a kernel size of 3 and no padding will reduce the spatial size by ones pixel on each side.<br>
Max pooling with a kernel size and stride of 2 will halve the spatial size.</p>
<p>Let’s have a look at the model and split the layers to calculate the shape based on these assumptions.<br>
Have a look at the comments to see the output shape:</p>
<pre data-code-wrap="python"><code class="lang-python">class Net(nn.Module):

    def __init__(self):
        super(Net, self).__init__()
        # 1 input image channel, 6 output channels, 3x3 square convolution
        # kernel
        self.conv1 = nn.Conv2d(1, 6, 3)
        self.conv2 = nn.Conv2d(6, 16, 3)
        # an affine operation: y = Wx + b
        self.fc1 = nn.Linear(16 * 6 * 6, 120)  # 6*6 from image dimension
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

    def forward(self, x):
        # assuming input is [N, 1, 32, 32]
        x = F.relu(self.conv1(x)) # outputs [N, 6, 30, 30]
        x = F.max_pool2d(x, (2, 2)) # outputs [N, 6, 15, 15]
        x = F.relu(self.conv2(x)) # outputs [N, 16, 13, 13]
        x = F.max_pool2d(x, 2) # outputs [N, 16, 6, 6]
        x = x.view(-1, self.num_flat_features(x)) # flattens x to [N, 16*6*6]
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

    def num_flat_features(self, x):
        size = x.size()[1:]  # all dimensions except the batch dimension
        num_features = 1
        for s in size:
            num_features *= s
        return num_features
</code></pre>
<aside class="quote no-group" data-username="Machupicchu" data-post="1" data-topic="54689">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/b19c9b/48.png" class="avatar"> Machupicchu:</div>
<blockquote>
<p>Also, I don’t understand why it accepts inputs like 30x30 and 31x31</p>
</blockquote>
</aside>
<p>For an input of <code>31x31</code> and <code>30x30</code>, we’ll get the following shapes:</p>
<pre><code class="lang-auto"># For x = [N, 1, 31, 31]:
[N, 6, 29, 29]
[N, 6, 14, 14]
[N, 16, 12, 12]
[N, 16, 6, 6 ]

# For x = [N, 1, 30, 30]:
[N, 6, 28, 28]
[N, 6, 14, 14]
[N, 16, 12, 12]
[N, 16, 6, 6 ]
</code></pre>
<p>So we got lucky, that the final result is the same. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"><br>
As you can see, the reason for the equal output shapes is that the pooling layer uses <code>floor</code> for odd input shapes as described in the <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.MaxPool2d">docs</a>.<br>
You can change it by setting <code>ceil_mode=True</code>, if necessary.</p>

Machupicchu: <p>Great answer, thanks!</p>
<p>May I just also ask :</p>
<pre><code class="lang-auto">net = Net()
input = torch.randn(1, 1, 32, 32)
out = net(input)
</code></pre>
<p>Why is the method forward() not explicitely called? I mean how does just calling net(output) calls forward()  ? (which is what happens as far as I understand)</p>
<p>By the way I dont understand what this line means:</p>
<p><code>super(Net, self).__init__()</code></p>
<p>I can imagine super() is calling the constructor of a parent class but …?</p>

ptrblck: <aside class="quote no-group" data-username="Machupicchu" data-post="3" data-topic="54689">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/b19c9b/48.png" class="avatar"> Machupicchu:</div>
<blockquote>
<p>I mean how does just calling net(output) calls forward() ? (which is what happens as far as I understand)</p>
</blockquote>
</aside>
<p>Yes, you are right. If you directly call the <code>nn.Module</code>, its internal <a href="https://github.com/pytorch/pytorch/blob/d2a8435c08f9d87603d4d9a9179a989b632bc2cb/torch/nn/modules/module.py#L535"><code>__call__</code></a> method will be called, which makes sure to call hooks etc. and finally calls the <code>forward</code> method.<br>
You should stick to this approach and not call <code>model.forward</code> manually.</p>
<aside class="quote no-group" data-username="Machupicchu" data-post="3" data-topic="54689">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/b19c9b/48.png" class="avatar"> Machupicchu:</div>
<blockquote>
<p>I can imagine super() is calling the constructor of a parent class but …?</p>
</blockquote>
</aside>
<p>That’s again right. This call initializes the parent class as seen <a href="https://github.com/pytorch/pytorch/blob/d2a8435c08f9d87603d4d9a9179a989b632bc2cb/torch/nn/modules/module.py#L70">here</a> which then initializes all necessary dicts etc. for the parameters, buffers, …</p>