NLP Metrics Made Simple: The BLEU Score
By: Boaz Shmueli
Tags: Machine Learning, Machine Translation, Bleu, NLP, Metrics

That’s where BLEU, which stands for Bi-Lingual Evaluation Understudy, kicks in. It’s a popular and inexpensive way to automatically measure the performance of your MT model. In a nutshell, BLEU compares the machine’s translation — what’s known as the candidate translation — with existing human-generated translations, known as the reference translations.

The easiest way to explain BLEU is by example. And for that we’ll look at a classic maxim from the Hebrew Bible — וְאָהַבְתָּ לְרֵעֲךָ כָּמוֹךָ — which depicts in three words the principle known as the Golden Rule: treat others as you would like others to treat you 🤗. We use the Bible because it’s been translated many times and to many languages, so it’s easy to find reference translations. Also, most of the translations are in the public domain. The Bible is in fact the most translated book in history (even more than Harry Potter!). In our case, we chose the following three reference translations among the many English translations for our maxim:

R1: but thou shalt love thy neighbor as thyself

thy neighbor as thyself R2: but have love for your neighbor as for yourself

for your neighbor as for yourself R3: but love your neighbors as you love yourself

And here is our candidate translation for which we want to compute the BLEU score:

D: but love other love friend for love yourself

As you can see, our MT system is not very good 😔…

BLEU works by computing the precision — the fraction of tokens from the candidate that appear, or are “covered”, by the references— but with a twist. Like any precision-based metric, the value of the BLEU score is always a number between 0 (worst) and 1 (best). Let’s compute BLEU for our candidate translation. In our example, the candidate consist of 8 words: but love other love friend for love yourself. Had none of the words appeared in any of the references, the precision would have been be 0/8=0. Luckily most of them appear in the references. It’s easy to see that 6 of the 8 words, but love love for love yourself — that is, all words except other and friend — appear in at least one of the references. One is thus tempted to compute the precision as 6/8 = 0.75. But if you look carefully, you will see that the word love appears 3 times in the candidate, yet it appears at most twice in any single reference (it appears twice in R3 and once in R1 and R2 ). BLEU takes this into account — it penalizes words that appear in the candidate more times than it appears in any of the references. Why? Well, the rationale is: since love appeared at most twice in any reference translation, it’s reasonable for it to appear up to 2 times also in the candidate. Thus, up to 2 love words are covered. Any excess usage of love in the candidate doesn’t make sense and so it is not considered as covered. And because the 3rd love is not covered, only 5 of the tokens are covered — but love love for yourself — and we get a BLEU score of 5/8=0.675.

Imagine for a moment that we didn’t have this correction; an 8-word candidate translation like love love love love love love love love would then get a perfect BLEU score of 1, since love appears in a reference translation. With BLEU, we would consider only the first 2 love’s as covered and discard the other 6. The BLEU score, in this case, is just 2/8 = 0.25, indeed indicating very low precision (albeit lots of love!).