Repository: nltk/nltk

Issue:
BLEU score brevity penalty does not follow definition
Papineni et al. (2002), if I understand correctly, define the BLEU brevity penalty as follows:
- let _c_ be the length of the candidate
- let _r_ be the length of the reference which is closest in length to the candidate

Then, BP = 1 if c > r, and BP = exp(1 - r / c) otherwise.

(There are some details about doing this at the corpus level, but they're not relevant to the point at hand.) 

But this not what `nltk.align.bleu_score._brevity_penalty` (called by `nltk.align.bleu_score`) computes. Rather, it computes _r_ as follows:

```
c = len(candidate)
r = min(abs(len(r) - c) for r in references)
```

So if _c_ is 12 and _r_ under Papineni et al.'s definition is 28, then this function will set `r` to 16, not 28.

However, it is possible I have misunderstood the original paper. And it would be ideal to test this against a canonical implementation. But if I have understood correctly, the fix would be something like:

```
c = len(candidate)
ref_lens = [len(ref) for ref in references]
r = min(ref_lens, key=lambda ref_len: abs(ref_len - c))
```



Discussion:
Hi,

Thank you for the report.

I might be overseeing some small details in the snippets you provided, but your solution does exactly the same as the current in NLTK.

> So if c is 12 and r under Papineni et al.'s definition is 28, then this function will set r to 16, not 28.

If c (12) is less then r (28) then the function returns 1, because c is less then r.

I'll check the paper, but first I need to have a cup of coffee :)

> I might be overseeing some small details in the snippets you provided, but your solution does exactly the same as the current in NLTK.

I respectively disagree. A worked example follows:

```
from math import exp
candidate = "MANY WILL LOSE THEIR RIGHT TO A PENSION IN THEIR OWN NAME".split()
ref0 = "MANY WILL LOSE THEIR RIGHT TO A PENSION IN THEIR OWN NAME BECAUSE OF THEIR HUSBAND 'S INCOME , WHILE MANY OTHERS WILL SEE THEIR PENSIONS REDUCED .".split()
ref1 = "MANY WILL LOSE THEIR RIGHT TO A PENSION IN THEIR OWN NAME BECAUSE OF THEIR HUSBANDS ' INCOME WHILE OTHERS WILL REDUCE THE AMOUNT OF THEIR PENSION .".split()
references = [ref0, ref1]
c = len(candidate)
ref_lens = [len(ref) for ref in references]
r = min(ref_lens, key=lambda ref_len: abs(ref_len - c))
BP = 1. if c > r else exp(1. - r / c)
print "c:\t{}".format(c)
print "r:\t{}".format(r)
print "BP:\t{:.4f}".format(BP)
```

This prints:

```
c:  12
r:  28
BP: 0.3679
```

Compare the following (with the same initializations):

```
from nltk.align.bleu_score import _brevity_penalty
print "BP:\t{:.4f}".format(_brevity_penalty(candidate, references))
```

which prints:

```
BP: 0.7165
```

Even intuitively, this is wrong. The candidate is far, far too short, and so an exponential brevity penalty should penalize it aggressively.

> If c (12) is less then r (28) then the function returns 1, because c is less then r.

I believe you have that backwards. The brevity penalty is always _less than_ 1 if c is less than r, because it is intended to penalize candidates that are too short; the brevity penalty is 1 only if c is _greater than_ or equal to r. The NLTK implementation agrees:

```
    if c > r:
        return 1
    else:
        return math.exp(1 - r / c)
```

I've finally got it. Thanks for the explanation. (I should not try to read code early in the morning)

Running the example on python 3, I get 0.2636, not 0.3679 (but that's because of true division)

```
c:  12
r:  28
BP: 0.2636
```

Another question, just to be sure. For `c = 12` and the reference lengths of `13` and `2` the brevity penalty should be 0.92, not 1.

``` python
>>> c = 12
>>> ref_lens = [2, 13]
>>> r = min(ref_lens, key=lambda ref_len: abs(ref_len - c))
>>> r
13
>>> 1. if c > r else exp(1. - r / c)
0.92...
```


Resolving patch:
diff --git a/nltk/align/bleu_score.py b/nltk/align/bleu_score.py
--- a/nltk/align/bleu_score.py
+++ b/nltk/align/bleu_score.py
@@ -203,9 +203,57 @@ def _brevity_penalty(candidate, references):
     length sentence, brevity penalty is used to modify the overall BLEU
     score according to length.
 
+    An example from the paper. There are three references with length 12, 15
+    and 17. And a terse candidate of the length 12. The brevity penalty is 1.
+
+    >>> references = [['a'] * 12, ['a'] * 15, ['a'] * 17]
+    >>> candidate = ['a'] * 12
+    >>> _brevity_penalty(candidate, references)
+    1.0
+
+    In case a candidate translation is shorter than the references, penalty is
+    applied.
+
+    >>> references = [['a'] * 28, ['a'] * 28]
+    >>> candidate = ['a'] * 12
+    >>> _brevity_penalty(candidate, references)
+    0.2635...
+
+    The length of the closest reference is used to compute the penalty. If the
+    length of a candidate is 12, and the reference lengths are 13 and 2, the
+    penalty is applied because the candidate length (12) is less then the
+    closest reference length (13).
+
+    >>> references = [['a'] * 13, ['a'] * 2]
+    >>> candidate = ['a'] * 12
+    >>> _brevity_penalty(candidate, references)
+    0.92...
+
+    The brevity penalty doesn't depend on reference order. More importantly,
+    when two reference sentences are at the same distance, the shortest
+    reference sentence length is used.
+
+    >>> references = [['a'] * 13, ['a'] * 11]
+    >>> candidate = ['a'] * 12
+    >>> _brevity_penalty(candidate, references) == _brevity_penalty(candidate, reversed(references)) == 1
+    True
+
+    A test example from mteval-v13a.pl (starting from the line 705):
+
+    >>> references = [['a'] * 11, ['a'] * 8]
+    >>> candidate = ['a'] * 7
+    >>> _brevity_penalty(candidate, references)
+    0.86...
+
+    >>> references = [['a'] * 11, ['a'] * 8, ['a'] * 6, ['a'] * 7]
+    >>> candidate = ['a'] * 7
+    >>> _brevity_penalty(candidate, references)
+    1.0
+
     """
     c = len(candidate)
-    r = min(abs(len(r) - c) for r in references)
+    ref_lens = (len(reference) for reference in references)
+    r = min(ref_lens, key=lambda ref_len: (abs(ref_len - c), ref_len))
 
     if c > r:
         return 1


Test patch: