mBART finetuned on XSUM
Author: mbelcen
Pull request: no
Labels: name: wontfix

## Environment info

- `transformers` version: 4.1.0.dev0
- Platform: Linux-5.4.0-58-generic-x86_64-with-glibc2.29
- Python version: 3.8.5
- PyTorch version (GPU?): 1.8.0.dev20201216+cu110 (True)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: distributed


### Who can help

 mBART: @patrickvonplaten
 examples/seq2seq: @patil-suraj


## Information

Model I am using: mBART

The problem arises when using:
* [ X ] the official example scripts:
 I used the official seq2seq training example here:
 https://github.com/huggingface/transformers/tree/master/examples/seq2seq
* [ X ] my own modified scripts: (give details below)
my training script is as follows (no changes to finetune_trainer.py):
```shell
python -m torch.distributed.launch --nproc_per_node=2 finetune_trainer.py \
    --data_dir "./xsum" \
    --output_dir "./my_models" \
    --overwrite_output_dir \
    --model_name_or_path "facebook/mbart-large-cc25" \
    --fp16 \
    --freeze_encoder \
    --per_device_train_batch_size=4 \
    --per_device_eval_batch_size=4 \
    --learning_rate=3e-5 \
    --do_train --do_eval --do_predict \
    --evaluation_strategy steps \
    --predict_with_generate \
    --n_val 1000 \
    --max_target_length=60 \
    --val_max_target_length=60 \
    --test_max_target_length=100 \
    "$@"
```
The tasks I am working on is:
* [ X ] XSUM


## To reproduce

Steps to reproduce the behavior:

1. follow https://github.com/huggingface/transformers/blob/master/examples/seq2seq/README.md
2. launch training script with my modifications (batch_size, freeze_encoder, max_target_length ..)
3. inference on two texts (french and english) using the following code:
```python
def sum_mbart_xsum(text):
    print("---------------------------------------------------------------------------------")
    print("                                MBART large xsum                                  ")
    print("---------------------------------------------------------------------------------")

    tokenizer = MBartTokenizer.from_pretrained("/home/mohamed/Desktop/Summarization/mbart-xsum")
    model = MBartForConditionalGeneration.from_pretrained("/home/mohamed/Desktop/Summarization/mbart-xsum")

    article_input_ids = tokenizer.batch_encode_plus([text], return_tensors='pt', max_length=1024, truncation=True)[
        'input_ids']
    summary_ids = model.generate(article_input_ids,
                                 num_beams=6,
                                 length_penalty=1.0,
                                 max_length=142,
                                 no_repeat_ngram_size=3)

    summary_txt = tokenizer.decode(summary_ids.squeeze(), skip_special_tokens=True)
    return summary_txt

```
## Results
1- eval/test results:
```jsonc
{
    "epoch": 3.0,
    "test_gen_len": 28.1,
    "test_loss": 1.7692,
    "test_n_ojbs": -1,
    "test_rouge1": 32.7618,
    "test_rouge2": 12.022,
    "test_rougeL": 25.6512,
    "test_rougeLsum": 25.6499,
    "test_runtime": 2778.8939,
    "test_samples_per_second": -0.0,
    "train_n_ojbs": -1,
    "train_runtime": 94633.1507,
    "train_samples_per_second": -0.0,
    "val_gen_len": 28.0,
    "val_loss": 1.7993,
    "val_n_ojbs": 1000,
    "val_rouge1": 32.9862,
    "val_rouge2": 11.528,
    "val_rougeL": 25.6517,
    "val_rougeLsum": 25.7055,
    "val_runtime": 267.0092,
    "val_samples_per_second": 3.745
}
```
2- Inference:
* Out of context summarizations (gives sth related to training data) --> (sth wrong with my finetuning configuration or inference function?)
* For French texts, results are in English and poor summary --> (why the language changes?)

## Expected behavior

My main objective of finetuning mBART on Xsum is to evaluate the multilingual level of mBART. Basically answering the following question: Should I finetune on a dataset with multiple languages to be able to summarize in multiple languages? Or the multilingual characteristic is preserved with mBART and just finetune on english (xsum) only
Current problems:
1- (inference results and questions above)
2- Why `facebook/bart-large-xsum` understands french (even if bart is trained on english)?


Comments:
> For French texts, results are in English and poor summary --> (why the language changes?)

Since you are fine-tuning on English data, I don't think it will be good at generating french summaries. Probably a good idea to fine-tune with multiple languages.

> Why facebook/bart-large-xsum understands french (even if bart is trained on English)?

since the training data is scraped from the web there is a chance that there could be some french text in it. I think the authors would answer this question better.


@patil-suraj


> > For French texts, results are in English and poor summary --> (why the language changes?)
> 
> Since you are fine-tuning on English data, I don't think it will be food at generating french summaries. Probably a good idea to fine-tune with multiple languages.

So I imagined mBART would be even better at this since even english BART finetuned on Xsum only can summarize pretty well in French. 

 However, I don't understand why my model (mbart FT on Xsum) gives out of context results (in any language) like I mentioned above? I followed the steps at https://github.com/huggingface/transformers/blob/master/examples/seq2seq/README.md and made very few changes.





This issue has been automatically marked as stale and been closed because it has not had recent activity. Thank you for your contributions.

If you think this still needs to be addressed please comment on this thread.