Configuration of the input channels for the first linear layer in a convolutional neural network is tricky

Nikos_Spanos: <p>I have a training dataset of melgrams with each melgram having shape [21,128]. The training sample has 3200 melgrams. The dataloader object that I create with a batch size of 16 has batches of shape [16,21,128]. Thus 16 melgrams per batch.</p>
<p>I have created the following neural network with PyTorch (<code>torch</code> module). Briefly, the network has 4 convolutional layers and 4 linear layers in a sequential order. The model structure is as follows:</p>
<pre data-code-wrap="python"><code class="lang-python">class ConvolutionalNeuralNetwork_pooling(nn.Module):
  def __init__(self):
    super(ConvolutionalNeuralNetwork_pooling, self).__init__()
    #initialize features
    self.input_units = 1
    self.output_units = 4
    self.kernel_size = 5
    self.pool_kernel_size=2
    #convolutional layers
    #(in_channels, out_channels, kernel_size)
    self.conv1 = nn.Conv2d(self.input_units, 16, self.kernel_size, padding=2)
    self.conv2 = nn.Conv2d(16, 32, self.kernel_size, padding=2)
    self.conv3 = nn.Conv2d(32, 64, self.kernel_size, padding=2)
    self.conv4 = nn.Conv2d(64, 128, self.kernel_size, padding=2)
    self.fc1 = nn.Linear(128*2*4, 1024) #here is the tricky part
    self.fc2 = nn.Linear(1024, 256)
    self.fc3 = nn.Linear(256, 32)
    self.fc4 = nn.Linear(32, self.output_units)
    #initialiaze max pooling layer
    self.max_pool = nn.MaxPool2d(kernel_size=self.pool_kernel_size)
    #initialize non-linear activation function
    self.activation = nn.ReLU()
    #initialized weights
    self.apply(self._init_weights)

  def _init_weights(self, module):
    if isinstance(module, nn.Linear):
      module.weight.data.normal_(mean=0.0, std=1.0)
      if module.bias is not None:
        module.bias.data.zero_()
    elif isinstance(module, nn.Conv2d):
        nn.init.uniform_(module.weight)
        if module.bias is not None:
          module.bias.data.zero_()

  def forward(self, x):
    x = x.unsqueeze(1)
    x = self.max_pool(self.activation(self.conv1(x)))
    x = self.max_pool(self.activation(self.conv2(x)))
    x = self.max_pool(self.activation(self.conv3(x)))
    x = self.max_pool(self.activation(self.conv4(x)))
    x = x.view(x.size(0), -1)
    x = self.activation(self.fc1(x))
    x = self.activation(self.fc2(x))
    x = self.activation(self.fc3(x))
    x = self.fc4(x)
    return x
</code></pre>
<p>My question is the following:</p>
<p>If I have the first linear layer as:<br>
<code>self.fc1 = nn.Linear(128, 1024)</code> I receive the following error during weight calculation in the first linear layer:</p>
<blockquote>
<p>RuntimeError: mat1 and mat2 shapes cannot be multiplied (16x1024 and 128x1024)</p>
</blockquote>
<p>The shape of <code>x</code> during training is:</p>
<pre data-code-wrap="python"><code class="lang-python">input: torch.Size([16, 1, 21, 128])
after conv1: torch.Size([16, 16, 10, 64])
after conv2: torch.Size([16, 32, 5, 32])
after conv3: torch.Size([16, 64, 2, 16])
after conv4: torch.Size([16, 128, 1, 8])
after flattening-before 1st linear layer: torch.Size([16, 1024])
</code></pre>
<p>However, when I replace</p>
<p>this <code>self.fc1 = nn.Linear(128, 1024)</code> to <code>self.fc1 = nn.Linear(128*2*4, 1024)</code></p>
<p>The <code>x</code> has now shape <code>[1024, 1024]</code> and the calculation of the weights is completed successfully. Is there any general rule that is applied here when using <code>max_pooling()</code> and <code>padding</code> in the convolutional networks?. Because it’s not clear to me why I should apply this multiplication in the input channels of the first linear layer.</p>

KFrank: <p>Hi Nikos!</p>
<aside class="quote no-group quote-modified" data-username="Nikos_Spanos" data-post="1" data-topic="181448" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/nikos_spanos/48/44970_2.png" class="avatar"> Nikos_Spanos:</div>
<blockquote>
<blockquote>
<p>RuntimeError: mat1 and mat2 shapes cannot be multiplied (16x1024 and 128x1024)</p>
</blockquote>
<p>…</p>
<p>However, when I replace</p>
<p>this <code>self.fc1 = nn.Linear(128, 1024)</code> to <code>self.fc1 = nn.Linear(128*2*4, 1024)</code></p>
<p>The <code>x</code> has now shape <code>[1024, 1024]</code> and the calculation of the weights is completed successfully.</p>
</blockquote>
</aside>
<p>The convolutional section of a convolutional neural network has the nice<br>
feature that it can accept input of more-or-less any spatial shape.  But, as<br>
you’ve noticed, when the convolutional section is then joined with some<br>
fully-collected layers, the spatial-shape-independence can break down.</p>
<p>If, in your use case, the input to your network will always have the same<br>
shape (except for the batch size, which can vary), your solution of modifying<br>
<code>fc1</code> to  <code>self.fc1 = nn.Linear(128*2*4, 1024)</code> is perfectly reasonable.</p>
<p>On the other hand, if you want to preserve the ability of your network to<br>
accept inputs of varying shapes, your solution of modifying <code>fc1</code> will break<br>
again when you pass into your network input of some other shape.</p>
<p>Pytorch offers adaptive-pooling layers that support the most common<br>
approach to this issue.</p>
<p>In your case, it appears that you want the output of:</p>
<pre data-code-wrap="python"><code class="lang-python">x = self.max_pool(self.activation(self.conv4(x)))
</code></pre>
<p>to have shape <code>[16, 128, 1, 1]</code> rather than shape <code>[16, 128, 1, 8]</code>.</p>
<p>You can achieve this by replacing your last <code>MaxPool2d</code> layer with<br>
<a href="https://pytorch.org/docs/2.0/generated/torch.nn.AdaptiveMaxPool2d.html" rel="noopener nofollow ugc">AdaptiveMaxPool2d</a>:</p>
<pre data-code-wrap="python"><code class="lang-python">self.adapt_pool = nn.AdaptiveMaxPool2d (output_size = 1)
...
x = self.adapt_pool (self.activation (self.conv4 (x)))
</code></pre>
<p><code>AdaptiveMaxPool2d (1)</code> adjusts its pooling algorithm so that its output<br>
has spatial shape <code>[1, 1]</code>, regardless of the spatial shape of its input, and,<br>
hence, regardless of the spatial shape the input to the network.</p>
<p>Best.</p>
<p>K. Frank</p>

Nikos_Spanos: <aside class="quote no-group" data-username="KFrank" data-post="2" data-topic="181448">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p><code>self.adapt_pool = nn.AdaptiveMaxPool2d (output_size = 1)</code></p>
</blockquote>
</aside>
<p>Thanks fro the reply Frank. I was wondering why <code>nn.Linear(128*2*4, 1024)</code> is a reasonable input channel and for the first linear layer and not something like nn.Linear(128<em>16</em>4, 1024) for example. Why should I multiply by 2*4.</p>
<p>Thanks again for your suggestions and the solution with adaptive pooling. Good to know there is also this option.</p>