NLP Metrics Made Simple: The BLEU Score | by Boaz Shmueli | Towards Data Science

Natural language processing (NLP) has become a super-hot topic. One of the most successful applications of NLP is Machine Translation (MT) — the ability to automatically translate text from one language to another. Perhaps the most well-known MT application is the very useful Google Translate, with its ability to translate from and to 100+ languages.
How can we evaluate the performance of an MT system? One obvious way is to use human evaluation: we can show the original sentence with its machine translation side-by-side, and ask humans to rate the accuracy of the translation. Human evaluation usually requires experts who are fluent in both the source and the target languages. This is expensive and time-consuming. Moreover, every time you tweak your model, you need to re-run the evaluation. Wouldn’t it be nice if we had an automatic way to evaluate the performance of an MT system?
That’s where BLEU, which stands for Bi-Lingual Evaluation Understudy, kicks in. It’s a popular and inexpensive way to automatically measure the performance of your MT model. In a nutshell, BLEU compares the machine’s translation — what’s known as the candidate translation — with existing human-generated translations, known as the reference translations.
The easiest way to explain BLEU is by example. And for that we’ll look at a classic maxim from the Hebrew Bible — וְאָהַבְתָּ לְרֵעֲךָ כָּמוֹךָ — which depicts in three words the principle known as the Golden Rule: treat others as you would like others to treat you 🤗. We use the Bible because it’s been translated many times and to many languages, so it’s easy to find reference translations. Also, most of the translations are in the public domain. The Bible is in fact the most translated book in history (even more than Harry Potter!). In our case, we chose the following three reference translations among the many English translations for our maxim:
R1: but thou shalt love thy neighbor as thyself
R2: but have love for your neighbor as for yourself
R3: but love your neighbors as you love yourself
And here is our candidate translation for which we want to compute the BLEU score:
D: but love other love friend for love yourself
As you can see, our MT system is not very good 😔...
BLEU works by computing the precision — the fraction of tokens from the candidate that appear, or are “covered”, by the references— but with a twist. Like any precision-based metric, the value of the BLEU score is always a number between 0 (worst) and 1 (best). Let’s compute BLEU for our candidate translation. In our example, the candidate consist of 8 words: but love other love friend for love yourself. Had none of the words appeared in any of the references, the precision would have been be 0/8=0. Luckily most of them appear in the references. It’s easy to see that 6 of the 8 words, but love love for love yourself — that is, all words except other and friend — appear in at least one of the references. One is thus tempted to compute the precision as 6/8 = 0.75. But if you look carefully, you will see that the word love appears 3 times in the candidate, yet it appears at most twice in any single reference (it appears twice in R3 and once in R1 and R2 ). BLEU takes this into account — it penalizes words that appear in the candidate more times than it appears in any of the references. Why? Well, the rationale is: since love appeared at most twice in any reference translation, it’s reasonable for it to appear up to 2 times also in the candidate. Thus, up to 2 love words are covered. Any excess usage of love in the candidate doesn’t make sense and so it is not considered as covered. And because the 3rd love is not covered, only 5 of the tokens are covered — but love love for yourself — and we get a BLEU score of 5/8=0.675.
Imagine for a moment that we didn’t have this correction; an 8-word candidate translation like love love love love love love love love would then get a perfect BLEU score of 1, since love appears in a reference translation. With BLEU, we would consider only the first 2 love’s as covered and discard the other 6. The BLEU score, in this case, is just 2/8 = 0.25, indeed indicating very low precision (albeit lots of love!).
Let’s look at the calculation more formally. For each unique word w in the candidate, we count how many times it appears in the candidate. Let’s call this number D(w). In our example:
D(but)=1D(love)=3D(other)=1D(friend)=1D(for)=1D(yourself)=1
For each unique word w, we also define R(w) to be the largest number of times the word appears in any of the references. We calculate this by looking at how many times w appears in each reference, and taking the maximum value. In our example:
R(but)=1 R(love)=2 [appears twice in R3]R(other)=0 R(friend)=0R(for)=2 [appears twice in R2]R(yourself)=1
Our very naïve and basic BLEU score — we will call it BLEU* — can be computed as the ratio of Covered candidate words out of the Total number of candidate words:
BLEU*=Covered/Total
Let’s start with the denominator Total, which is super simple to compute. It’s the number of words in the candidate. A fancy way to write it is:
Total = D(W1)+D(W2)+...
In our case
Total=D(but)+D(love)+D(other)+D(friend)+D(for)+D(yourself)=1+3+1+1+1+1=8
Now for the numerator Covered, which is the total number of covered words. For each unique word w, the number of words in the candidate is D(w), but the coverage is limited by R(w). So if D(w)≤R(w), all D(w) words are covered. Otherwise only R(w) words are covered. The number of covered words for each unique word w can simply be written as MIN(R(w), D(w)) where MIN is the minimum of the two values.
Let’s see how this works out for our example:
MIN(D(but), R(but))=MIN(1, 1)=1MIN(D(love), R(love))=MIN(3, 2)=2MIN(D(other), R(other))=MIN(1, 0)=0MIN(D(friend), R(friend))=MIN(1,0)=0MIN(D(for), R(for))=MIN(1, 2)=1MIN(D(yourself), R(yourself))=MIN(1,1)=1
The total coverage is the sum of the above values:
Covered=1+2+0+0+1+1=5
We can finally calculate our BLEU* score for our candidate:
BLEU*(but love other love friend for love yourself)=Covered/Total=5/8=0.625
The naïve BLEU* score that I described above is not used in practice because it has many issues that render it highly inaccurate and not practical. I’ve simply introduced it to give the idea behind the “true” BLEU score.
Here are some of the problems with the naïve BLEU*.
First, very short translations —i.e., candidates with very few words— can do absurdly well although they are likely to be horrible translations. Imagine the candidate is simply the 1-word love or even the two-word but for. These candidates get a perfect BLEU* score of 1 because the tokens are nicely covered by the references.
In addition, the BLEU* metric is completely oblivious to word order. The BLEU* score for but love other love friend for love yourself is exactly the same as other love love love for friend yourself but. In languages where word order is important (English and many others) this doesn’t really make sense.
Lastly, we only calculated the BLEU* score for a single sentence. To measure the performance of our MT model, it makes sense not to rely on a single instance, but to check the performance on many sentences, and combine the scores for a more comprehensive and accurate evaluation of the model.
To rectify these issues, the true BLEU score incorporates several corrections to my naïve BLEU* calculations. I explain how the true BLEU score is calculated in a follow up story.
Finally, I wrote some simple Python code that computes the BLEU* score. Note that this is for educational purposes only — do not use it for industrial or academic purposes! The code includes two versions for the score — they are functionally equivalent and so they calculate identical scores. The first (BLUE_star) is longer but is hopefully easier to understand. The shorter version (BLUE_star_compact) uses (or perhaps even abuses) Python’s list comprehension and is hence more compact.
from collections import Counter

# compute a very naïve BLEU score -- for educational purposes only

def BLEU_star(refs, candidate):
    
    # tokenize the references and the candidate
    refs = [ref.split() for ref in refs]
    candidate = candidate.split()

    # compute word frequencies for the references and the candidate
    refs_counts = [Counter(ref) for ref in refs]
    candidate_counts = Counter(candidate)

    covered = 0
    total = 0
    
    # compute the coverage for each word
    for word, count in candidate_counts.items():
        covered += min(count, max([ref[word] for ref in refs_counts]))
        total += count
    
    # note: we can also use len(candidate) instead of total :)
    return covered / total

# a more "pythonic" way to compute BLUE_star 

def BLEU_star_compact(refs, candidate):
    refs = [ref.split() for ref in refs]
    candidate = candidate.split()

    return sum([min(count, max([ref[word] for ref in [Counter(ref) for ref in refs]])) for word, count in Counter(candidate).items()])/len(candidate)

refs=['but thou shalt love thy neighbor as thyself',
        'but have love for your neighbour as for yourself',
        'but love your neighbors as you love yourself']

candidate = 'but love other love friend for love yourself'

print(BLEU_star(refs, candidate))
print(BLEU_star_compact(refs, candidate))

I hope you liked this beginner’s BLEU tutorial.
Feel free to leave feedback below. Thank you!