Finetune mbart rouge score difference between training and evaluation part
Author: eymenkagantaspinar
Pull request: no
Labels: name: wontfix

### Environment info
- transformers from source
- google colab

### Information
When I use the `sshleifer/student_cnn_12_6` model for `finetune_trainer.py` and then I run finetuned model in `run_eval.py`,  I can get close and high rouge scores.However when I give `facebook/mbart-large-cc25` as model and tokenizer to `finetune_trainer.py` and also, I used the same `LID` for both eos and bos , the generated text and rouge score produced by the `finetune_trainer.py` evaluation and prediction section are not good, and when I run the fine-tuned model with the `run_eval.py` rouge scores are very close to 0. What could be reason for low rouge scores and particularly the rouge score difference between training and evaluation when using mbart model.

`Finetune_trainer.py` arguments
>!python /content/transformers/examples/seq2seq/finetune_trainer.py 
    --model_name_or_path facebook/mbart-large-cc25 \ 
    --tokenizer_name facebook/mbart-large-cc25 \ 
    --data_dir /content/transformers/cnn_dm_tr \ 
    --output_dir finetuned_model --overwrite_output_dir \
    --learning_rate=3e-5 \
    --warmup_steps 500 --sortish_sampler \
    --fp16 \
    --n_val 500 \
    --freeze_encoder --freeze_embeds \
    --src_lang tr_TR --tgt_lang tr_TR \
    --gradient_accumulation_steps=1 \
    --per_device_train_batch_size=4 --per_device_eval_batch_size=4 \
    --num_train_epochs=2 \
    --save_steps 3000 --eval_steps 3000 \
    --logging_first_step \
    --max_target_length 56 --val_max_target_length 142 --test_max_target_length 142 \
    --do_train --do_eval --do_predict \
    --evaluation_strategy steps \
    --predict_with_generate --sortish_sampler \
    "$@"

`Finetune_trainer.py` results
>test_loss = 7.9716
test_rouge1 = 5.6445
test_rouge2 = 1.6458
test_rougeL = 4.8763
test_rougeLsum = 5.3712

>val_loss = 7.9894
val_rouge1 = 5.0368
val_rouge2 = 1.6249
val_rougeL = 4.1304
val_rougeLsum = 4.6041

`Run_eval.py` arguments
>!python /content/transformers/examples/seq2seq/run_eval.py 
    /content/finetuned_model \
   /content/transformers/cnn_dm_tr/test.source \
   dbart_cnn_12_6_test_gens.txt \
   --reference_path /content/transformers/cnn_dm_tr/test.target \ 
   --score_path dbart_cnn_12_6_test_rouge.json \
   --n_obs 100 \
   --task summarization --bs 2 --fp16

`Run_eval.py` results
>{'rouge1': 0.1091, 'rouge2': 0.0, 'rougeL': 0.1091, 'rougeLsum': 0.1091, 'n_obs': 50, 'runtime': 601, 'seconds_per_sample': 12.02}

### Who can help
@patil-suraj @sshleifer

### Expected behavior
Be able to get high and close rouge scores for `run_eval.py` and `finetune_trainer.py` when I use `mbart-large-cc25` as model and tokenizer.

Comments:
Hi @Eymen3455 

In the `run_eval` command I see that you are setting `n_obs` to 100 while the `finetune_trainer` uses all test examples, could you maybe run eval again with all test examples and see if you get close results?

First of all, thank you very much @patil-suraj for your reply. I did as you said and used all the test dataset for `n_obs`, but the result remained unchanged. In addition, when we examine the texts produced in `finetune_trainer.py` and `run_eval.py`, the texts produced in `finetune_trainer.py` are acceptable. While the texts produced in `run_eval.py` are different, it is worse and the rouge score is less than half of obtained scores in `finetune_trainer.py`.

Then when I saw this issue (https://github.com/huggingface/transformers/issues/9236), I used Xsum dataset as dataset and rouge score increased in `run_eval.py`. Just changing the dataset caused to get logical text in `run_eval.py`, but still I could not understand the difference between the rouge score and generated texts in `finetune_trainer.py` and `run_eval.py`.

I do not understand what has affected the dataset so much. When I use the mbart model, why can't I get the same success in `finetune_train.py` and `run_eval.py`, what could be the reason for this? By the way, when I try the `sshleifer/student_cnn_12_6` model instead of the mbart model, I can achieve exactly the same success in `finetune_trainer.py` and `run_eval.py`. I would appreciate if you could help.


This issue has been automatically marked as stale and been closed because it has not had recent activity. Thank you for your contributions.

If you think this still needs to be addressed please comment on this thread.

I ran into the same problem and positioned it to the max_length parameters in Seq2SeqTrainer.evaluate() . Whether to set max_length parameters will result in different results.

I have encountered the same issue:

val_rouge2 obtained during training is different from scores I got with "run_eval.py" script

do you have any suggestions ? @patil-suraj