Connot correctly fine-tune Bert for generation
Author: wyu97
Pull request: no
Labels: name: Migration

# 📚 Migration

## Information

<!-- Important information -->

Model I am using Bert for generation (a Seq2Seq style):

The problem arises when using:
* The official example scripts from : [transformers/examples/legacy/seq2seq/finetune_trainer.py](https://github.com/huggingface/transformers/blob/master/examples/legacy/seq2seq/finetune_trainer.py)
```
config = AutoConfig.from_pretrained(model_args.config_name)
tokenizer = AutoTokenizer.from_pretrained(model_args.tokenizer_name)
model = AutoModelForSeq2SeqLM.from_pretrained(model_args.model_name_or_path)
```

The above is the official code to fine-tune a text generation model such as BART (i.e., just specify model_name_or_path as facebook/BART-base). **I am trying to use the BERTGEN instead of using BART of T5.** So, I modified the above code into the following one.

* My own modified scripts: 
```
config = BertConfig.from_pretrained("bert-large-uncased")
tokenizer = BertTokenizer.from_pretrained("bert-large-uncased")
encoder = BertGenerationEncoder.from_pretrained("bert-large-uncased", bos_token_id=101, eos_token_id=102)
decoder = BertGenerationDecoder.from_pretrained("bert-large-uncased", add_cross_attention=True, is_decoder=True, bos_token_id=101, eos_token_id=102)
model = EncoderDecoderModel(encoder=encoder, decoder=decoder)
```

The tasks I am working on is:
* An official task: Gigaword, CNN/DailyMail


## Details

Since the examples in the github repo do not contain `Bert for generation`, so I adopt the code from the documents [here](https://huggingface.co/transformers/model_doc/bertgeneration.html). The only modification has been shown as aboves. However, the model performance cannot reach the reported performance in their [paper](https://arxiv.org/pdf/1907.12461.pdf), even has been left by a large margin. I felt the model has not been correctly trained. 

**I am wondering whether there is an example code for using Bert for generation. Thank you very much for your help in advance.**

## Environment info
<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
 
- `transformers` version: 4.7.0
- Python version: 3.6
- PyTorch version (GPU?): 1.8
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: Yes

<!-- IMPORTANT: which version of the former library do you use? -->
* `pytorch-transformers` or `pytorch-pretrained-bert` version (or branch): 4.7.0



Comments:
Hi Stas, Suraj, Sylvain (@stas00 @patil-suraj @sgugger),

Would you please to give some helps on using the `finetune_trainer.py` [file](https://github.com/huggingface/transformers/blob/master/examples/legacy/seq2seq/finetune_trainer.py) for BertGen?

Thank you very much!



It looks like @patrickvonplaten ported this model, https://huggingface.co/transformers/model_doc/bertgeneration.html
so he is probably the best person to ask.


Yes, I followed that document and changed the example script [transformers/examples/legacy/seq2seq/finetune_trainer.py](https://github.com/huggingface/transformers/blob/master/examples/legacy/seq2seq/finetune_trainer.py). However, it did not work.

@patrickvonplaten Hi Patrick, do you have any script for training BertGen model?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

Hey@wyu97,

Could you try to follow this blog post and the accompanying google colab to fine-tune a Beer for generation Seq2Seq model? 

https://huggingface.co/blog/warm-starting-encoder-decoder#warm-starting-encoder-decoder-models-with-%F0%9F%A4%97transformers-practice

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.