[examples] Verify marian and mbart BLEU scores with examples/seq2seq/run_eval.py
Author: sshleifer
Pull request: no
Labels: name: Examples

name: translation

name: marian

Got MBART 26.8 before postprocessing on wmt_en_ro,  37.1 after.
6 minutes. (first # same as fairseq, second should be also).
Marian: 27.7/37.4 90 Seconds

Comments:
Surprised. Maybe the opus data overlaps the wmt test set?

![image](https://user-images.githubusercontent.com/6045025/90946657-faf75b80-e3fc-11ea-856a-dc598d992112.png)


Opus #s are legit!

@sshleifer do you using the default configuration in Readme ? I didn't get 37.1, and I have tried the configuration mentioned as original paper, I have got a 36.6 after postprocessing wmt_en_ro enro.

In addition, I have a confusion about the code. If I don't add extra parameters, there is a warning in the output log. I try to solve it. I can't set a breakpoint at the place where the warning is generated. It's difficult for me to locate the reason. Can you help me look at it?

`Keyword arguments {'add_prefix_space': False} not recognized.
`

(1) Which split, val or test?
(2) What was the score before post-processing?

The warning is from [here](https://github.com/huggingface/transformers/blob/master/src/transformers/tokenization_utils.py#L261), you can safely ignore it.


### Results on wmt_en_ro/test
```bash
python run_eval.py Helsinki-NLP/opus-mt-en-ro wmt_en_ro/test.source enro_test_translations.txt --reference_path wmt_en_ro/test.target -
-task translation --score_path mar_test_bleu.json --fp16 --bs 64
{'bleu': 27.6865, 'n_obs': 1999, 'runtime': 85, 'seconds_per_sample': 0.0425}
```
```bash
ro_post_process enro_finetune/test_generations.txt wmt_en_ro/test.target
# 37.4
```

### Postprocessing Setup
```bash
cd $HOME
git clone git@github.com:moses-smt/mosesdecoder.git
cd mosesdecoder  
git clone git@github.com:rsennrich/wmt16-scripts.git
ro_post_process () {
  sys=$1
  ref=$2
  export MOSES_PATH=$HOME/mosesdecoder
  REPLACE_UNICODE_PUNCT=$MOSES_PATH/scripts/tokenizer/replace-unicode-punctuation.perl
  NORM_PUNC=$MOSES_PATH/scripts/tokenizer/normalize-punctuation.perl
  REM_NON_PRINT_CHAR=$MOSES_PATH/scripts/tokenizer/remove-non-printing-char.perl
  REMOVE_DIACRITICS=$MOSES_PATH/wmt16-scripts/preprocess/remove-diacritics.py
  NORMALIZE_ROMANIAN=$MOSES_PATH/wmt16-scripts/preprocess/normalise-romanian.py
  TOKENIZER=$MOSES_PATH/scripts/tokenizer/tokenizer.perl



  lang=ro
  for file in $sys $ref; do
    cat $file \
    | $REPLACE_UNICODE_PUNCT \
    | $NORM_PUNC -l $lang \
    | $REM_NON_PRINT_CHAR \
    | $NORMALIZE_ROMANIAN \
    | $REMOVE_DIACRITICS \
    | $TOKENIZER -no-escape -l $lang \
    > $(basename $file).tok
  done
  # compute BLEU
  cat $(basename $sys).tok | sacrebleu -tok none -s none -b $(basename $ref).tok
}
```


```bash
ro_post_process enro_test_translations.txt wmt_en_ro/test.target
```


@sshleifer That's warning can be sefaly ignore it, that is great, I have try to figure out the reason for occuring.

I using the test set, I have got 21.9 before post-processing. I'm using this postprocessing setup and run_eval.py, My analysis is that my model performance is not enough, maybe the fine-tuning parameters are inaccurate, or there are too many training steps during fine-tuning.  

I am using this version with following code link.
https://github.com/huggingface/transformers/blob/b9772897ec9f54c1a83263b059bfd37acda936d5/examples/seq2seq/finetune.py#L371

this has been changed in lastest version. 
https://github.com/huggingface/transformers/blob/9e89390ce1e785e72452207139a334cd3bf745ff/examples/seq2seq/finetune.py#L396

I am not a native English speaker, If my words is unclear, you can let me repeat it in time.


I don't understand what code you ran, what you expected to happen, and what happened.

The change you highlight should not affect your results.

my last comment has two question.
the first question is my results can not be 37.7.
the second question is I set save_top_k ==-1,  I have solved it 

the follows is my runing parameters for the first question.
`bash train_mbart_cc25_enro_pap.sh --output_dir $OUTPUT_DIR --gpus 1 --sortish_sampler`

where train_mbart_cc25_enro_pap.sh as follows :

```
BS=16
MAX_LEN=128
python finetune.py \
    --learning_rate=3e-5 \
    --do_train \
    --val_check_interval=0.25 \
    --adam_eps 1e-06 \
    --num_train_epochs 4 --src_lang en_XX --tgt_lang ro_RO \
    --data_dir $ENRO_DIR \
    --max_source_length $MAX_LEN --max_target_length $MAX_LEN --val_max_target_length $MAX_LEN --test_max_target_length $MAX_LEN \
    --train_batch_size=$BS --eval_batch_size=$BS \
    --task translation \
    --warmup_steps 2500 \
    --freeze_embeds \
    --model_name_or_path=$MODEL_PATH \
    --label_smoothing 0.2 \
    --dropout 0.3 \
    "$@"
```

that's my transformers report bleu as follows,  21.9 is using sacrebleu before postprocessing. @sshleifer 
```{'bleu':` 26.0506, 'n_obs': 1999, 'runtime': 681, 'seconds_per_sample': 0.3407}```

and I have try to using the configuration as README with using `--fp16`,  and I get a error log as follows :
```
2020-09-15 15:58:46.000 [INFO] [Driver] RuntimeError: CUDA out of memory. Tried to allocate 490.00 MiB (GPU 0; 31.75 GiB total capacity; 29.17 GiB already allocated; 241.44 MiB free; 30.35 GiB reserved in total by PyTorch)
```

and I have see the README mentioned as follows

> This should take < 6h/epoch on a 16GB v100 and achieve test BLEU above 26 To get results in line with fairseq, you need to do some postprocessing. 


+ 26.06 is definitely expected behavior, as the README indicates. 
+ The best I've ever scored from finetuning is 26.8 after training for 6 epochs. (Took 24h).
+ `--save_top-k=-1`, I can look into.
+ What command did you run with `sacrebleu` to get 21.9? The HF run_eval.py uses the `sacrebleu.corpus_bleu` python function.
+ I would recommend playing with the Helsinki-NLP/ models for faster MT finetuning.

> * 26.06 is definitely expected behavior, as the README indicates.
> * The best I've ever scored from finetuning is 26.8 after training for 6 epochs. (Took 24h).
> * `--save_top-k=-1`, I can look into.
> * What command did you run with `sacrebleu` to get 21.9? The HF run_eval.py uses the `sacrebleu.corpus_bleu` python function.
> * I would recommend playing with the Helsinki-NLP/ models for faster MT finetuning.

@sshleifer  Hi , Thank you for your kindness help. I have got a closer score 37.2 to 37.7, I have a curious question, why is `Helsinki-NLP/ models` faster ?



They are way smaller: 74 million parameters (Marian) vs 610M (mBART)