Run Time Error: RuntimeError: Expected hidden[0] size (2, 1, 512), got [2, 128, 512] - Seq2Seq Model with PreTrained BERT Model
Author: Ninja16180
Pull request: no

Hi,

I am facing this run time error while training a seq2seq model with pretrained bert model.

```
RuntimeError                              Traceback (most recent call last)

<ipython-input-63-472071541d41> in <module>()
      8     start_time = time.time()
      9 
---> 10     train_loss = train(model, train_iterator, optimizer, criterion, CLIP)
     11     valid_loss = evaluate(model, valid_iterator, criterion)
     12 

8 frames

/usr/local/lib/python3.7/dist-packages/torch/nn/modules/rnn.py in check_hidden_size(self, hx, expected_hidden_size, msg)
    221                           msg: str = 'Expected hidden size {}, got {}') -> None:
    222         if hx.size() != expected_hidden_size:
--> 223             raise RuntimeError(msg.format(expected_hidden_size, list(hx.size())))
    224 
    225     def check_forward_args(self, input: Tensor, hidden: Tensor, batch_sizes: Optional[Tensor]):

RuntimeError: Expected hidden[0] size (2, 1, 512), got [2, 128, 512]
```

Related code snippets:

```
from torchtext.legacy.data import BucketIterator,TabularDataset

BATCH_SIZE = 128

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

train_iterator, valid_iterator = data.BucketIterator.splits(
    (train_data, valid_data), 
    batch_size = BATCH_SIZE, 
    device = device)

```
# Encoder

class Encoder(nn.Module):
    def __init__(self, bert, hid_dim, n_layers, dropout):
        super().__init__()
        
        self.hid_dim = hid_dim
        self.n_layers = n_layers

        self.bert = bert
        
        emb_dim = bert.config.to_dict()['hidden_size']
              
        self.rnn = nn.LSTM(emb_dim, hid_dim, n_layers, batch_first = True,dropout = dropout)
        
        self.dropout = nn.Dropout(dropout)
        
    def forward(self, sent1):
        
        #sent1 = [sent1 len, batch size]
        
        with torch.no_grad():
            embedded = self.bert(sent1)[0]
        
        #embedded = [sent1 len, batch size, emb dim]
        
        outputs, (hidden, cell) = self.rnn(embedded)
        
        #outputs = [sent1 len, batch size, hid dim * n directions]
        #hidden = [n layers * n directions, batch size, hid dim]
        #cell = [n layers * n directions, batch size, hid dim]
        
        #outputs are always from the top hidden layer
        
        return hidden, cell

The detailed code with error description is available here for your reference:
https://github.com/Ninja16180/BERT/blob/main/Training_Seq2Seq_Model_using_Pre-Trained_BERT_Model.ipynb

Kindly help me in resolving the issue

Thanks in advance!
```
```

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

Kindly help in resolving the issue.
It will help to build a seq2seq conversation model using pretrained bert model.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

Maybe of interest to @patrickvonplaten and @patil-suraj 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.