Concatenate CNN AvgPool with Embedding layer

chrome: <p>I have an embedding layer with each sentence having a length of 20 and the dimension as 16. I pass this through a 1D convolutional layer -&gt; relu -&gt; avgPool and the output dimension of avgPool is <code>[128, 10, 10]</code> where 128 is the batch size. Now I want to know how to concatenate this avgPool with the embedding with dimension <code>[128, 20, 16]</code> so that I can pass this to the next CNN layer. I have been stuck with the error <code>invalid argument 0: Sizes of tensors must match except in dimension 0. Got 20 and 10 in dimension 1 at /pytorch/aten/src/TH/generic/THTensorMoreMath.cpp:1307</code>.</p>
<pre><code class="lang-auto">self.c_1 = nn.Conv1d(in_channels=self.embedding_dim, out_channels=10, kernel_size=2,
                             padding=1)
self.relu_1 = nn.ReLU()
self.avg_pool_1 = nn.AvgPool1d(2)
</code></pre>

ptrblck: <p>As the error message states you cannot concatenate these tensors, as two dimensions have different sizes.<br>
As far as I understand you would like to create something like a residual connection, where you pass the output of the embedding layer to this conv -&gt; avg_pool block and try to concatenate it with the input.</p>
<p>If that’s the case and your embedding output has a shape of <code>[128, 20, 16]</code>, the output shape of the conv-block will be <code>torch.Size([128, 10, 8])</code>:</p>
<pre><code class="lang-python">
x = torch.randn(128, 20, 16)
c_1 = nn.Conv1d(in_channels=20, out_channels=10, kernel_size=2,
                             padding=1)
avg_pool_1 = nn.AvgPool1d(2)
output = c_1(x)
output = avg_pool_1(output)
</code></pre>
<p>Could you explain your use case a bit, as I’m not sure how you would like to concatenate these tensors now?</p>

chrome: <p>My goal is to concatenate my sentence embedding with shape <code>[128, 20, 16]</code> with the output of the averagepool with shape <code>[128, 10, 10]</code> at position 2 which is <strong>10 + 16</strong>. The output of this concatenation is given as input to the next <code>Conv1d</code> layer.</p>
<pre><code class="lang-auto">torch.cat((embed.view(-1, sent.shape[1], embed.shape[2]), avg_pool_level_1), 2)
</code></pre>
<p>where shape of embed is <code>torch.Size([128, 20, 16])</code> and shape of avg_pool_1 is <code>torch.Size([128, 10, 10])</code></p>

ptrblck: <p>Concatenating these tensors won’t work unfortunately, as the sizes of two dimensions are different.<br>
You could pad one dimension (e.g. dim1) and concatenate it in dim2.</p>

chrome: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a>. I wanted to know if there is a way I can concatenate the input word embedding with the output of CNN-POOL layer considering that the dimensions keep reducing depending on the size of the POOL layer.  Any technique that can be followed for this?</p>

ptrblck: <p>For a 1-dimensional signal, you could try to match the number of channels and concatenate in dim2.<br>
To do this just set <code>out_channels=20</code> in your conv layer.</p>

chrome: <p>Cool thanks! Also in pytorch how can we add the dimensions of a FC layer which comes after avgpool if I don’t know what the output dimension of avgpool will be? Like if the dimensions of avgpool is <code>[128, 10, 57]</code> and FC comes after this then initially I won’t be able to know the value 57 as the first dimension unless I execute the code once. Here 10 is the out_channel and 114 was the sequence length of each sentence which halved due to avgpool.</p>

ptrblck: <p>You could calculate the output shapes looking at the formulas in the <a href="https://pytorch.org/docs/stable/nn.html" rel="nofollow noopener">docs</a>.<br>
If you don’t want to do that, you could just run a single iteration, add some print statements to show the output shapes, and change the number of input features accordingly.</p>
<p>However, if you are dealing with variable sized inputs, I would recommend to use an adaptive pooling layer, which will output a defined shape.</p>

chrome: <p>Yes I had so far used the single iteration and printing the shapes and modifying accordingly.</p>