How the first paramter of nn.Linear is selected?

aleemsidra: <p>I am trying to understand how to build a NN in pytorch. I am playing with the MNIST dataset. I am not able to understand that how the first parameter in <code>nn.Linear</code> is set to <code>9216</code>?</p>
<pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self):
      super(Net, self).__init__()
      self.conv1 = nn.Conv2d(1, 32, 3, 1)
      self.conv2 = nn.Conv2d(32, 64, 3, 1)
      self.dropout1 = nn.Dropout2d(0.25)
      self.dropout2 = nn.Dropout2d(0.5)
      self.fc1 = nn.Linear(9216, 128)
      self.fc2 = nn.Linear(128, 10)
my_nn = Net()
print(my_nn)
</code></pre>

nikhil6041: <p>The first parameter also known as in_parameters basically means that the no. of parameters the current linear layers expects from the previous layer. You can refer to the pytorch docs for better explaination.<br>
I would suggest to take the pytorch 60 minute blitz <a href="https://pytorch.org/tutorials/beginner/deep_learning_60min_blitz.html" class="inline-onebox" rel="noopener nofollow ugc">Deep Learning with PyTorch: A 60 Minute Blitz — PyTorch Tutorials 1.8.1+cu102 documentation</a> .<br>
For your current issue you can refer to this <a href="https://pytorch.org/tutorials/beginner/basics/buildmodel_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Build the Neural Network — PyTorch Tutorials 1.8.1+cu102 documentation</a></p>

KFrank: <p>Hi Aleemsidra!</p>
<aside class="quote no-group" data-username="aleemsidra" data-post="1" data-topic="124172" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/aleemsidra/48/38723_2.png" class="avatar"> aleemsidra:</div>
<blockquote>
<p>I am not able to understand that how the first parameter in <code>nn.Linear</code> is set to <code>9216</code>?</p>
</blockquote>
</aside>
<p>The first parameter – <code>in_features</code> –  used to construct a <code>Linear</code><br>
must match the number of features in the tensor passed to it.  That<br>
is, that tensor should have shape <code>[nBatch, in_features]</code> (where<br>
the <code>Linear</code> doesn’t care about the batch size, <code>nBatch</code>, which can<br>
be anything and vary from call to call).</p>
<p>In a CNN you will typically have some convolutions that are applied<br>
to images of arbitrary size (but that have to be at least as large as<br>
the kernel size and the number of channels must match).  These<br>
convolutions preserve the spatial structure of the image.</p>
<p>But when you get to the <code>Linear</code> layers you typically <em>flatten</em> the<br>
processed image – throwing away the spatial structure – so it<br>
becomes a vector of “features.”  This is where <code>9216</code> comes from.</p>
<p>In your case if you start with images of <code>height = 16</code> and <code>width = 16</code>,<br>
the <code>Conv2d</code> layers will eat away a little at the edges of the images, but<br>
increase the number channels (to <code>64</code>).  Each processed sample in the<br>
batch will now have <code>9216</code> elements and will turn into a vector of <code>9216</code><br>
features when flattened.</p>
<p>The MNIST dataset has images of size 28x28, so the numbers<br>
don’t quite work without some further processing.  The dataloader<br>
might hypothetically resize the images (for whatever reason).  The<br>
network might well have a pooling layer (common in CNNs) that<br>
isn’t shown in your code snippet.  If you input images of size 28x28<br>
but also have a 2x2 pooling layer after the convolutions, you will<br>
also end up with <code>9216</code> features.</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self):
      super(Net, self).__init__()
      self.conv1 = nn.Conv2d(1, 32, 3, 1)
      self.conv2 = nn.Conv2d(32, 64, 3, 1)
      self.dropout1 = nn.Dropout2d(0.25)
      self.dropout2 = nn.Dropout2d(0.5)
      self.fc1 = nn.Linear(9216, 128)
      self.fc2 = nn.Linear(128, 10)
</code></pre>
</blockquote>
</aside>
<p>Here’s an illustrative example that uses the numbers from your above<br>
code snippet:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
print (torch.__version__)

nBatch = 32
height = 16
width = 16

conv1 = torch.nn.Conv2d (1, 32, 3, 1)
conv2 = torch.nn.Conv2d (32, 64, 3, 1)

s = torch.randn (nBatch, 1, height, width)
print ('s.shape =', s.shape)
print ('conv1 (s).shape =', conv1 (s).shape)
print ('conv2 (conv1 (s)).shape =', conv2 (conv1 (s)).shape)
print ('torch.flatten (conv2 (conv1 (s)), 1).shape =', torch.flatten (conv2 (conv1 (s)), 1).shape)

height = 28
width = 28

pool = torch.nn.MaxPool2d (2)

t = torch.randn (nBatch, 1, height, width)
print ('t.shape =', t.shape)
print ('conv1 (t).shape =', conv1 (t).shape)
print ('conv2 (conv1 (t)).shape =', conv2 (conv1 (t)).shape)
print ('pool (conv2 (conv1 (t))).shape =', pool (conv2 (conv1 (t))).shape)
print ('torch.flatten (pool (conv2 (conv1 (t))), 1).shape =', torch.flatten (pool (conv2 (conv1 (t))), 1).shape)
</code></pre>
<p>Here is its output:</p>
<pre><code class="lang-plaintext">1.7.1
s.shape = torch.Size([32, 1, 16, 16])
conv1 (s).shape = torch.Size([32, 32, 14, 14])
conv2 (conv1 (s)).shape = torch.Size([32, 64, 12, 12])
torch.flatten (conv2 (conv1 (s)), 1).shape = torch.Size([32, 9216])
t.shape = torch.Size([32, 1, 28, 28])
conv1 (t).shape = torch.Size([32, 32, 26, 26])
conv2 (conv1 (t)).shape = torch.Size([32, 64, 24, 24])
pool (conv2 (conv1 (t))).shape = torch.Size([32, 64, 12, 12])
torch.flatten (pool (conv2 (conv1 (t))), 1).shape = torch.Size([32, 9216])
</code></pre>
<p>You can see how each <code>Conv2d</code> layer trims a row of pixels off of the<br>
image edges and how <code>flatten()</code> produces a batch of vectors with<br>
<code>9216</code> features.</p>
<p>Note that such a network architecture (without further logic in it) will<br>
only work on images of the appropriate fixed size (but the batch size<br>
can be arbitrary).</p>
<p>Best.</p>
<p>K. Frank</p>