Repository: facebookresearch/fairseq

Issue:
BLEU score calculation error for character-based languages such as Chinese, Japanese, etc.
## 🐛 Bug

<!-- A clear and concise description of what the bug is. -->
When training model using translation task, if the `target_lang` is char base language, such as zh(Chinese) and ja(Japanese), the score calculated by sacrebleu is problematic.

### To Reproduce
This is my scripts:
Environment variable. Create a file .env and source .env.
```bash
FAIRSEQ_SRC=/path/to/fairseq
RAW_DATA_DIR=raw
DATA_DIR=data
DATA_BIN=databin
CHECKPOINTS=checkpoints
CUDA_VISIBLE_DEVICES=0
MAX_TOKENS=4096  # decide by gpu memory
```
Download wmt18 en-zh processed training data
```bash
curl -o $RAW_DATA_DIR/corpus.gz  https://data.statmt.org/wmt18/translation-task/preprocessed/zh-en/corpus.gz
gunzip $RAW_DATA_DIR/corpus.gz
cut -f 1 $DATA_DIR/corpus > en.train
cut -f 2 $DATA_DIR/corpus > zh.train

sacrebleu -t wmt17 -l en-zh --echo src > $RAW_DATA_DIR/en.valid
sacrebleu -t wmt17 -l en-zh --echo ref > $RAW_DATA_DIR/zh.valid
sacrebleu -t wmt18 -l en-zh --echo src > $RAW_DATA_DIR/en.test
sacrebleu -t wmt18 -l en-zh --echo ref > $RAW_DATA_DIR/zh.test
```

Train sentencepiece model and tokenize data
```bash
python $FAIRSEQ_SRC/scripts/spm_train.py \
    --input=$RAW_DATA_DIR/en.train,$RAW_DATA_DIR/zh.train \
    --model_prefix=$DATA_DIR/sentencepiece.bpe \
    --vocab_size=32000 \
    --character_coverage=1.0 \
    --model_type=bpe \
    --input_sentence_size=2000000 \
    --shuffle_input_sentence=true

python $FAIRSEQ_SRC/scripts/spm_encode.py \
    --model $DATA_DIR/sentencepiece.bpe.model \
    --inputs $RAW_DATA_DIR/en.train $RAW_DATA_DIR/zh.train \
    --outputs $DATA_DIR/train.en $DATA_DIR/train.zh

python $FAIRSEQ_SRC/scripts/spm_encode.py \
    --model $DATA_DIR/sentencepiece.bpe.model \
    --inputs $RAW_DATA_DIR/en.valid $RAW_DATA_DIR/zh.valid \
    --outputs $DATA_DIR/valid.en $DATA_DIR/valid.zh

python $FAIRSEQ_SRC/scripts/spm_encode.py \
    --model $DATA_DIR/sentencepiece.bpe.model \
    --inputs $RAW_DATA_DIR/en.test $RAW_DATA_DIR/zh.test \
    --outputs $DATA_DIR/test.en $DATA_DIR/test.zh
```

Binarize data
```
fairseq-preprocess --source-lang en --target-lang zh \
    --trainpref $DATA_DIR/train \
    --validpref $DATA_DIR/valid \
    --testpref $DATA_DIR/test \
    --joined-dictionary \
    --thresholdtgt 10 --thresholdsrc 10 \
    --destdir $DATA_BIN \
    --workers 40 
```
Train normal transformer model without adapters.
```
fairseq-train \
    $DATA_BIN \
    --arch transformer --share-all-embeddings \
    --source-lang en --target-lang zh \
    --optimizer adam --adam-betas '(0.9, 0.98)' --clip-norm 0.0 \
    --lr 5e-4 --lr-scheduler inverse_sqrt --warmup-updates 4000 \
    --dropout 0.1 --weight-decay 0.0001 \
    --criterion label_smoothed_cross_entropy --label-smoothing 0.1 \
    --max-tokens $MAX_TOKENS \
    --eval-bleu \
    --eval-bleu-args '{"beam": 5, "max_len_a": 1.2, "max_len_b": 10}' \
    --eval-bleu-remove-bpe sentencepiece \
    --eval-bleu-print-samples \
    --best-checkpoint-metric bleu --maximize-best-checkpoint-metric \
    --save-dir $CHECKPOINTS
```
<img width="1445" alt="截屏2022-09-02 00 28 36" src="https://user-images.githubusercontent.com/25839309/187965638-9b6688ab-2501-4683-809f-e10b5fd2924f.png">
From the output sample, the translation result is good, but the calculated bleu value is only 4.63. This is because the default tokenizer of sacrebleu cannot tokenize Chinese correctly.

<!-- If you have a code sample, error messages, stack traces, please provide it here as well -->


#### Code sample
<!-- Ideally attach a minimal code sample to reproduce the decried issue.
Minimal means having the shortest code but still preserving the bug. -->

### Expected behavior
The latest api of sacrebleu supports the specified language and automatically selects the appropriate tokenize method.  Is it possible to use the latest sacrebleu api to solve this issue. I can also make a Pull request for it.

/tree/main/fairseq/tasks/translation.py:494
```py
        if self.cfg.eval_tokenized_bleu:
            return sacrebleu.corpus_bleu(hyps, [refs], tokenize="none")
        else:
            return sacrebleu.corpus_bleu(hyps, [refs])
```
should be changed to
```py
        if self.cfg.eval_tokenized_bleu:
            return sacrebleu.metrics.BLEU().corpus_score(hyps, [refs])
        else:
            return sacrebleu.metrics.BLEU(trg_lang=self.cfg.target_lang).corpus_score(hyps, [refs])
```

<!-- A clear and concise description of what you expected to happen. -->

### Environment

 - fairseq Version: main
 - PyTorch Version: any
 - OS (e.g., Linux): Linux
 - How you installed fairseq (`pip`, source):
 - Build command you used (if compiling from source):
 - Python version:
 - CUDA/cuDNN version:
 - GPU models and configuration:
 - Any other relevant information:

### Additional context

<!-- Add any other context about the problem here. -->



Discussion:

Resolving patch:
diff --git a/fairseq/tasks/translation.py b/fairseq/tasks/translation.py
--- a/fairseq/tasks/translation.py
+++ b/fairseq/tasks/translation.py
@@ -492,6 +492,8 @@ def decode(toks, escape_unk=False):
             logger.info("example hypothesis: " + hyps[0])
             logger.info("example reference: " + refs[0])
         if self.cfg.eval_tokenized_bleu:
-            return sacrebleu.corpus_bleu(hyps, [refs], tokenize="none")
+            return sacrebleu.metrics.BLEU().corpus_score(hyps, [refs])
         else:
-            return sacrebleu.corpus_bleu(hyps, [refs])
+            return sacrebleu.metrics.BLEU(trg_lang=self.cfg.target_lang).corpus_score(
+                hyps, [refs]
+            )
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -205,7 +205,7 @@ def do_setup(package_data):
             'numpy<1.20.0; python_version<"3.7"',
             'numpy; python_version>="3.7"',
             "regex",
-            "sacrebleu>=1.4.12",
+            "sacrebleu>=2.0.0",
             "torch",
             "tqdm",
             "bitarray",


Test patch: