Encoder-decoders in Transformers 
Author: anandhperumal
Pull request: no
Labels: name: wontfix

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....):

Language I am using the model on (English, Chinese....):

The problem arise when using:
* [X ] the official example scripts: (give details)
* [ ] my own modified scripts: (give details)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ X] my own task or dataset: (give details)

## To Reproduce

Steps to reproduce the behavior:

1. I'm trying to use the hybrid seq2seq model described in this [article](https://medium.com/huggingface/encoder-decoders-in-transformers-a-hybrid-pre-trained-architecture-for-seq2seq-af4d7bf14bb8). It is stated that the library is available from 2.2.0 version. I tried in both 2.2.0 and 2.2.1
I don't find the respective libraries working as expected.

```
from transformers import PreTrainedEncoderDecoder
model = PreTrainedEncoderDecoder.from_pretrained('bert-base-uncased','gpt2')
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
encoder_input_ids=tokenizer.encode("Hi How are you")
ouput = model(torch.tensor( encoder_input_ids).unsqueeze(0),torch.tensor( encoder_input_ids).unsqueeze(0) )
```
and I get the following error:
```
TypeError: forward() got an unexpected keyword argument 'encoder_hidden_states'
```
I checked the code of modelling [gpt2](https://github.com/huggingface/transformers/blob/1d189304624db17749aee23fa2345f009cc48215/transformers/modeling_gpt2.py#L541) it doesn't take any input as encoder_hidden_states.

2. I also tried the another example from that article
![1_my6KF5Wa2AFa54tPnQp3Yw](https://user-images.githubusercontent.com/12907396/70492861-013de580-1ac3-11ea-9197-86ebae611f7e.png)
 
There is no decode method in Model2Model, but then do you mean decoder?
But then Bert using decoder I get the following error

```
TypeError: forward() got an unexpected keyword argument 'length'
```
As bert doesn't take the length as the input or any of the parameters shown in the example.



* OS: Windows
* Python version: 3.7.4
* PyTorch version:1.3.0
* PyTorch Transformers version (or branch):2.2.0
* Using GPU ? No
* Distributed of parallel setup ? No
* Any other relevant information:

## Additional context

<!-- Add any other context about the problem here. -->


Comments:
Hi @anandhperumal,

Thank you for posting an issue. Just to clarify:

1. Indeed, as I specified in the article, `PreTrainedEncoderDecoder` only works with BERT as an encoder and BERT as a decoder. GPT2 shouldn't take too much work to adapt, but we haven't had the time to do it yet. Try `PreTrainedEncoderDecoder.from_pretrained('bert-base-uncased', 'bert-base-uncased')` should work. Let me know if it doesn't.
2.  We mean `decode`. Again, as written in the article this is not available as of now but will be very soon. You can follow the progress here: #1840 

Hi @rlouf ,
Thanks for getting back. 
The `PreTrainedEncoderDecoder` works like a charm but what was your intuition behind using BERT as a decoder. I mean there is nothing wrong with using it as a decoder but it was never trained as a decoder.  Did you test that on any dataset by combining two BERT trained for the different tasks?

It was this paper that we originally intended to reproduce: https://arxiv.org/abs/1907.12461

@rlouf  Thanks.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


how to fine tune the encoder-decoder model for training on new corpus?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


@rlouf I am using transformers(3.0.2). The module EncoderDecoderModel has some problem same as its predecessor. I am getting the following error for using BERT+GPT2 as well as Bert+XLNet for encoder-decoder:
```
forward() got an unexpected keyword argument 'encoder_hidden_states'
```
Is the problem has been fixed? If yes then please clarify me and tell me how to use it.