BERT Based CNN - Convolution and Maxpooling

Ah_Mass: <p>I’m trying to fine-tune a pre-trainted BERT model by inserting a CNN layer.<br>
In this model, the outputs of all transformer encoders are used, not only the output of the latest<br>
transformer encoder.<br>
So that the output vectors of each transformer encoder are concatenated, and a matrix is produced:</p>
<p>The convolutional operation is performed with a window of size (3, hidden size of BERT which is 768 in BERT_base model) and the maximum value is generated for each transformer encoder by<br>
applying max pooling on the convolution output.</p>
<p>By concatenating these values, a vector is generated which is given as input to a fully connected network. By applying softmax on the input, the classification operation is performed.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/7/67af915bcf6d5a743f45678beb04e2f7c5fdd5c7.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/67af915bcf6d5a743f45678beb04e2f7c5fdd5c7" title="7c78d40f-2f5f-4664-bb9a-766283ac9b4d"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/7/67af915bcf6d5a743f45678beb04e2f7c5fdd5c7_2_492x500.jpeg" alt="7c78d40f-2f5f-4664-bb9a-766283ac9b4d" data-base62-sha1="eNfkzeUpyJ44RAg7Kb6RtZPhKOH" width="492" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/7/67af915bcf6d5a743f45678beb04e2f7c5fdd5c7_2_492x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/6/7/67af915bcf6d5a743f45678beb04e2f7c5fdd5c7_2_738x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/6/7/67af915bcf6d5a743f45678beb04e2f7c5fdd5c7_2_984x1000.jpeg 2x" data-dominant-color="F5F5F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">7c78d40f-2f5f-4664-bb9a-766283ac9b4d</span><span class="informations">985×999 74.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>My problem is that I can’t seem to find the right arguments to perform the convolution and the maxpooling on that matrix.</p>
<p>With batch size = 32, there are 13 layers of Transformer encoders, each one get as an input [64, 768] of encoded tokenized text and outputs an encoding of the same dimensions. (64 is the max-length in tokenization)</p>
<p>I want to perform convolution on each transformer’s output matrix ([64,768]) separately, then perform maxpooling on that convolution’s output. So, I should get a max-value per each transformer, and these max values are inserted into the neural network.</p>
<p>My code is:</p>
<pre><code class="lang-auto">class BERT_Arch(nn.Module):

    def __init__(self, bert):
        super(BERT_Arch, self).__init__()
        self.bert = BertModel.from_pretrained('bert-base-uncased')
        self.conv = nn.Conv2d(in_channels=13, out_channels=13, kernel_size= (3, 768), padding=True) 
        self.relu = nn.ReLU()
        self.pool = nn.MaxPool1d(kernel_size=768, stride=1)
        self.dropout = nn.Dropout(0.1)
        self.fc = nn.Linear(9118464, 3)
        self.flat = nn.Flatten()
        self.softmax = nn.LogSoftmax(dim=1)

    def forward(self, sent_id, mask):
        _, _, all_layers = self.bert(sent_id, attention_mask=mask, output_hidden_states=True)
        # all_layers  = [32, 13, 64, 768]
        x = torch.cat(all_layers, 0) # x= [416, 64, 768]
        x = self.conv(x)
        x = self.relu(x)
        x = self.pool(x)
        x = self.flat(x)
        x = self.fc(x)
        return self.softmax(x)
</code></pre>
<p>I keep getting an error saying that the convolution method expected a certain dimensions as input but got a different one.</p>
<pre><code class="lang-auto">&lt;generator object BERT_Arch.forward.&lt;locals&gt;.&lt;genexpr&gt; at 0x7fbeffc2d200&gt;
torch.Size([416, 64, 768])
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-12-3a2c2cd7c02d&gt; in &lt;module&gt;()
    362 
    363         # train model
--&gt; 364         train_loss, _ = train()
    365 
    366         # evaluate model

5 frames
&lt;ipython-input-12-3a2c2cd7c02d&gt; in train()
    148 
    149         # get model predictions for the current batch
--&gt; 150         preds = model(sent_id, mask)
    151 
    152         # compute the loss between actual and predicted values

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--&gt; 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

&lt;ipython-input-12-3a2c2cd7c02d&gt; in forward(self, sent_id, mask)
     42         x = torch.cat(all_layers, 0) # torch.Size([13, 32, 64, 768])
     43         print(x.shape)
---&gt; 44         x = self.conv(x)
     45         x = self.relu(x)
     46         x = self.pool(x)

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--&gt; 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/conv.py in forward(self, input)
    421 
    422     def forward(self, input: Tensor) -&gt; Tensor:
--&gt; 423         return self._conv_forward(input, self.weight)
    424 
    425 class Conv3d(_ConvNd):

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/conv.py in _conv_forward(self, input, weight)
    418                             _pair(0), self.dilation, self.groups)
    419         return F.conv2d(input, weight, self.bias, self.stride,
--&gt; 420                         self.padding, self.dilation, self.groups)
    421 
    422     def forward(self, input: Tensor) -&gt; Tensor:

RuntimeError: Expected 4-dimensional input for 4-dimensional weight [13, 13, 3, 768], but got 3-dimensional input of size [416, 64, 768] instead
</code></pre>
<p>I tried different values for the convolution method arguments, I still got a similar error.<br>
And sometimes an error saying that the maxpooling output size is too small:</p>
<pre><code class="lang-auto">Given input size: (64x62x1). Calculated output size: (64x31x0). Output size is too small
</code></pre>
<p>I would be grateful for any help on how to do this CNN layer correctly.</p>