Machine Translation Evaluation with sacreBLEU and BERTScore | by Ng Wai Foong | Towards Data Science

By reading this piece, you will learn to evaluate your machine translation models using the following packages:
sacreBLEU
BERTScore
For your information, BLEU (bilingual evaluation understudy) is one of the most popular metric for evaluating machine-translated text. It can be used to evaluate translations of any language provided that there exists some form of word boundary in the text.
BLEU’s output is usually a score between 0 and 100, indicating the similarity value between the reference text and hypothesis text. The higher the value, the better the translations.
Having said that, one of the major downside for BLEU is the need to tokenize the text properly. For example, you can easily tokenize any English sentences using whitespace as the delimiter. However, tokenization can be challenging for languages such as Burmese, Chinese and Thai.
In addition, there are criticism that an increase in the BLEU score does not guarantee better translations. This is fairly noticeable when the translated text contains similar words or synonyms that convey the same message. For example, the following sentences are in fact identical from human perspective but will not score 100 when evaluating them with BLEU.
# Reference textHi, how are you?# Hypothesis textHello, how are you?
Nevertheless, let’s explore how you can evaluate the BLEU score easily using sacreBLEU which:
... provides hassle-free computation of shareable, comparable, and reproducible BLEU scores.
It is highly recommended to create a new virtual environment before you continue. Activate it and run the following command to install sacrebleu:
pip install sacrebleu
Create a new file called bleu_scorer.py and append the following import statement at the top of the file:
from sacrebleu.metrics import BLEU
Define the hypothesis text and reference text as follows:
refs = [['The dog bit the guy.', 'It was not unexpected.', 'The man bit him first.']]hyps = ['The dog bit the man.', "It wasn't surprising.", 'The man had just bitten him.']
hyps consists of a list of text while refs must be a list of list of text. In actual use cases, you should read it from a file as follows (strip to remove the ending newline):
with open('hyps.txt', 'r', encoding='utf8') as f:    lines = [x.strip() for x in f]
For multi-references, you should define it as follows:
refs = [['The dog bit the guy.', 'It was not unexpected.', 'The man bit him first.'],        ['The dog had bit the man.', 'No one was surprised.', 'The man had bitten the dog.']]hyps = ['The dog bit the man.', "It wasn't surprising.", 'The man had just bitten him.']
The following code highlights the base structure for multi-references:
refs = [[ref1_for_sent1, ref1_for_sent2 ...,], [ref2_for_sent1, ref2_for_sent2, ...], ...]
Then, create a new instance using the metrics.BLEU object-oriented API:
bleu = BLEU()
It accepts the following input parameters:
lowercase — If True, lowercased BLEU is computed.
tokenize — The tokenizer to use. If None, defaults to language-specific tokenizers with ‘13a’ as the fallback default.
smooth_method — The smoothing method to use (‘floor’, ‘add-k’, ‘exp’ or ‘none’).
smooth_value — The smoothing value for `floor` and `add-k` methods. `None` falls back to default value.
max_ngram_order — If given, it overrides the maximum n-gram order (default: 4) when computing precisions.
The max ngram is default to 4 as the it was found to be the highest correlation with monolingual human judgements based on the following research paper.
Call the corpus_score function to calculate the BLUE score of the entire corpus:
result = bleu.corpus_score(hyps, refs)
You can find the complete code at the following gist:
The output is as follows:
29.44 82.4/42.9/27.3/12.5 (BP = 0.889 ratio = 0.895 hyp_len = 17 ref_len = 19)
29.44 refers to the final BLEU score
82.4/42.9/27.3/12.5 represents the precision value for 1–4 ngram order
BP is the brevity penalty
ratio indicates the ratio between hypothesis length and reference length
hyp_len refers to the total number of characters for hypothesis text
ref_len is the total number of characters for reference text
You can find more information on each individual metrics at the following repository.
On the other hand, BERTScore has a few outstanding points that differs from BLEU. It
... leverages the pre-trained contextual embeddings from BERT and matches words in candidate and reference sentences by cosine similarity.
It has been shown to correlate with human judgment on sentence-level and system-level evaluation. Moreover, BERTScore computes precision, recall, and F1 measure, which can be useful for evaluating different language generation tasks.
This provides better evaluation for sentences that use synonyms or similar words. For example:
# reference textThe weather is cold today.# hypothesis textIt is freezing today.
Run the following command to install BERTScore via pip install:
pip install bert-score
Create a new file called bert_scorer.py and add the following code inside it:
from bert_score import BERTScorer
Next, you need to define the reference and hypothesis text. BERTScore accepts both a list of text (single reference) or list of list of text (multiple references):
refs = [['The dog bit the guy.', 'The dog had bit the man.'],                               ['It was not unexpected.', 'No one was surprised.'],                               ['The man bit him first.', 'The man had bitten the dog.']]hyps = ['The dog bit the man.', "It wasn't surprising.", 'The man had just bitten him.']
The structure for multi-references is slightly different compared to sacreBLEU. It is based on the following syntax:
refs = [[ref1_for_sent1, ref2_for_sent1 ...,], [ref1_for_sent2, ref2_for_sent2, ...], ...]
Instantiate a new instance of the BERTScorer object-oriented API as follows:
scorer = BERTScorer(lang="en", rescale_with_baseline=True)
It accepts:
model_type — contexual embedding model specification, default using the suggested model for the target language; has to specify at least one of model_type or lang
verbose — turn on intermediate status update
device — if None, the model lives on cuda:0 if cuda is available
batch_size — processing batch size
nthreads — number of threads
lang — language of the sentences. It needs to be specified when rescale_with_baseline is True
rescale_with_baseline — rescale bertscore with pre-computed baseline
baseline_path — customized baseline file
use_fast_tokenizer — to use HuggingFace’s fast tokenizer
By default, it uses the following model based on the lang specified:
en — roberta-large
en-sci — allenai/scibert_scivocab_uncased
zh — bert-base-chinese
tr — dbmdz/bert-base-turkish-cased
The rest of the language will be based on bert-base-multilingual-cased. If you intend to customize the model, kindly check the following Google sheet to find out more.
Rescale baselines helps to adjust the output score to be more readable. This is simply because some contextual embedding models tend produce scores in a very narrow range. You can find more information on this at the following blog post.
The repository also contains a few pre-computed baseline file for you. Simply head to the following repository and download the desired file based on the mode that you intend to use.
It will download the model and dependencies from HuggingFace during the initial run. Subsequently, it will reuse the same downloaded model. On Linux operating system, the model is located at the following directory:
~/.cache/huggingface/transformers/
Call the score function as follows:
P, R, F1 = scorer.score(hyps, refs)
It will returns the tensors for:
P — precision
R — recall
F1 — F1-score
Here is an example output for F1-score:
tensor([0.9014, 0.8710, 0.5036, 0.7563, 0.8073, 0.8103, 0.7644, 0.8002, 0.6673, 0.7086])
Please be noted that the underlying values range from -1 to 1.
You can easily compute the corpus score by calling the mean function as follows:
print(F1.mean())# 0.6798698902130217
The complete code is located at the following gist:
Please be noted that running BERTScore is a lot slower compared to sacreBLEU. Make sure that you have sufficient GPU memory during evaluation.
For more information on BERTScore, kindly check the example notebook from the official repository.
let’s recap what you have learned today.
This article started off with a brief introduction on BLEU, the advantages and disadvantages of using it to evaluate translated text.
Next, it covered on using sacreBLEU to compute the corpus-level BLEU score. The output also includes the precision value for 1–4 ngrams.
Subsequently, it explored on BERTScore which evaluate the translated text by cosine similarity. The score function returns the tensors for precision, recall and F1-score respectively.
Thanks for reading this piece. Have a great day ahead!
Github — sacreBLEUGithub — BERTScore
Github — sacreBLEU
Github — BERTScore