Repository: nltk/nltk

Issue:
Critical bug in BLEU score evaluation
The BLEU score, as computed [here](https://github.com/nltk/nltk/blob/develop/nltk/align/bleu.py), returns extremely broken results

It's currently implemented as:

``` python
p_ns = (BLEU.modified_precision(candidate, references, i)
                for i, _ in enumerate(weights, start=1))
p_ns_nonzero = list(filter(None, p_ns))
if p_ns_nonzero:
  s = math.fsum(w * math.log(p_n) for w, p_n in zip(weights, p_ns_nonzero))
  bp = BLEU.brevity_penalty(candidate, references)
   return bp * math.exp(s)
else: # no alignments
  return 0
```

Now, what the `s =` part is doing is it's trying to compute a weighted **geometric average**, meaning the product of the individual modified precisions, to get the final precision across ngrams. Therefore, you cannot simply take out the `p_ns` that are zero by doing the filtering. If any one of the `p_ns` is zero, that should make the entire score zero.

I suggest something like the following:

``` python
p_ns = [BLEU.modified_precision(candidate, references, i) for i, _ in enumerate(weights, start=1)]
if all([x > 0 for x in p_ns]):
    s = math.fsum(w * math.log(p_n) for w, p_n in zip(weights, p_ns))
    bp = BLEU.brevity_penalty(candidate, references)
    return bp * math.exp(s)
else: # bad
    return 0
```

The paper reference on BLEU score where all of this is discussed is here:
http://www.aclweb.org/anthology/P02-1040.pdf



Discussion:
Thanks @karpathy. @dimazest would you please look into this?

ahh, i see. i'll also try to come up with a test for this situation.

btw, do you think it will be useful to remove the BLEU class and make its static methods just functions?


Resolving patch:
diff --git a/nltk/align/bleu.py b/nltk/align/bleu.py
--- a/nltk/align/bleu.py
+++ b/nltk/align/bleu.py
@@ -6,6 +6,7 @@
 # Contributors: Dmitrijs Milajevs
 # URL: <http://nltk.org/>
 # For license information, see LICENSE.TXT
+"""BLEU score implementation."""
 
 from __future__ import division
 
@@ -16,10 +17,12 @@
 from nltk.util import ngrams
 
 
-class BLEU(object):
-    """
-    This class implements the BLEU method, which is used to evaluate
-    the quality of machine translation. [1]
+def bleu(candidate, references, weights):
+    """Calculate BLEU score.
+
+    :param list(str)) candidate: A candidate sentence.
+    :param list(list(str)) references: Reference sentiences.
+    :param list(float) weights: Weights for unigrams, bigrams, trigrams and so on.
 
     Consider an example:
 
@@ -45,9 +48,35 @@ class BLEU(object):
     ...               'army', 'always', 'to', 'heed', 'the', 'directions',
     ...               'of', 'the', 'party']
 
-    The BLEU method mainly consists of two parts:
+    >>> bleu(candidate1, [reference1, reference2, reference3], weights)
+    0.504...
+
+    >>> bleu(candidate2, [reference1, reference2, reference3], weights)
+    0
+
+    Papineni, Kishore, et al. "BLEU: a method for automatic evaluation of
+    machine translation." Proceedings of the 40th annual meeting on association for
+    computational linguistics. Association for Computational Linguistics, 2002.
+    http://www.aclweb.org/anthology/P02-1040.pdf
+
+    """
+    p_ns = (
+        _modified_precision(candidate, references, i)
+        for i, _ in enumerate(weights, start=1)
+    )
+
+    try:
+        s = math.fsum(w * math.log(p_n) for w, p_n in zip(weights, p_ns))
+    except ValueError:
+        # some p_ns is 0
+        return 0
 
-    Part 1 - modified n-gram precision
+    bp = _brevity_penalty(candidate, references)
+    return bp * math.exp(s)
+
+
+def _modified_precision(candidate, references, n):
+    """Calculate modified ngram precision.
 
     The normal precision method may lead to some wrong translations with
     high-precision, e.g., the translation, in which a word of reference
@@ -55,159 +84,133 @@ class BLEU(object):
     n-gram precision, a reference word will be considered exhausted after
     a matching candidate word is identified.
 
+    Paper examples:
+
+    >>> _modified_precision(
+    ...    'the the the the the the the'.split(),
+    ...    ['the cat is on the mat'.split(), 'there is a cat on the mat'.split()],
+    ...    n=1,
+    ... )
+    0.28...
+
+    >>> _modified_precision(
+    ...    'the the the the the the the'.split(),
+    ...    ['the cat is on the mat'.split(), 'there is a cat on the mat'.split()],
+    ...    n=2,
+    ... )
+    0.0
+
+    >>> _modified_precision(
+    ...    'of the'.split(),
+    ...    [
+    ...        'It is a guide to action that ensures that the military will forever heed Party commands.'.split(),
+    ...        'It is the guiding principle which guarantees the military forces always being under the command of the Party.'.split(),
+    ...        'It is the practical guide for the army always to heed the directions of the party'.split(),
+    ...    ],
+    ...    n=1,
+    ... )
+    1.0
+
+    >>> _modified_precision(
+    ...    'of the'.split(),
+    ...    [
+    ...        'It is a guide to action that ensures that the military will forever heed Party commands.'.split(),
+    ...        'It is the guiding principle which guarantees the military forces always being under the command of the Party.'.split(),
+    ...        'It is the practical guide for the army always to heed the directions of the party'.split(),
+    ...    ],
+    ...    n=2,
+    ... )
+    1.0
+
+    More examples:
+
+    >>> weights = [0.25, 0.25, 0.25, 0.25]
+    >>> candidate1 = ['It', 'is', 'a', 'guide', 'to', 'action', 'which',
+    ...               'ensures', 'that', 'the', 'military', 'always',
+    ...               'obeys', 'the', 'commands', 'of', 'the', 'party']
+
+    >>> candidate2 = ['It', 'is', 'to', 'insure', 'the', 'troops',
+    ...               'forever', 'hearing', 'the', 'activity', 'guidebook',
+    ...               'that', 'party', 'direct']
+
+    >>> reference1 = ['It', 'is', 'a', 'guide', 'to', 'action', 'that',
+    ...               'ensures', 'that', 'the', 'military', 'will', 'forever',
+    ...               'heed', 'Party', 'commands']
+
+    >>> reference2 = ['It', 'is', 'the', 'guiding', 'principle', 'which',
+    ...               'guarantees', 'the', 'military', 'forces', 'always',
+    ...               'being', 'under', 'the', 'command', 'of', 'the',
+    ...               'Party']
+
+    >>> reference3 = ['It', 'is', 'the', 'practical', 'guide', 'for', 'the',
+    ...               'army', 'always', 'to', 'heed', 'the', 'directions',
+    ...               'of', 'the', 'party']
+
     Unigrams:
 
-    >>> BLEU.modified_precision(
+    >>> _modified_precision(
     ...    candidate1,
     ...    [reference1, reference2, reference3],
     ...    n=1,
     ... )
     0.94...
 
-    >>> BLEU.modified_precision(
+    >>> _modified_precision(
     ...    candidate2,
     ...    [reference1, reference2, reference3],
     ...    n=1,
     ... )
     0.57...
 
-    Bigrmas:
+    Bigrams:
 
-    >>> BLEU.modified_precision(
+    >>> _modified_precision(
     ...    candidate1,
     ...    [reference1, reference2, reference3],
     ...    n=2,
     ... )
     0.58...
 
-    >>> BLEU.modified_precision(
+    >>> _modified_precision(
     ...    candidate2,
     ...    [reference1, reference2, reference3],
     ...    n=2,
     ... )
     0.07...
 
+    """
+    counts = Counter(ngrams(candidate, n))
 
-    Part 2 - brevity penalty
-
-    As the modified n-gram precision still has the problem from the short
-    length sentence, brevity penalty is used to modify the overall BLEU
-    score according to length.
-
-    >>> BLEU.compute(candidate1, [reference1, reference2, reference3], weights)
-    0.504...
-
-    >>> BLEU.compute(candidate2, [reference1, reference2, reference3], weights)
-    0.457...
+    if not counts:
+        return 0
 
-    2. Test with two corpus that one is a reference and another is
-    an output from translation system:
+    max_counts = {}
+    for reference in references:
+        reference_counts = Counter(ngrams(reference, n))
+        for ngram in counts:
+            max_counts[ngram] = max(max_counts.get(ngram, 0), reference_counts[ngram])
 
-    >>> weights = [0.25, 0.25, 0.25, 0.25]
-    >>> ref_file = open('newstest2012-ref.en')  # doctest: +SKIP
-    >>> candidate_file = open('newstest2012.fr-en.cmu-avenue')  # doctest: +SKIP
+    clipped_counts = dict((ngram, min(count, max_counts[ngram])) for ngram, count in counts.items())
 
-    >>> total = 0.0
-    >>> count = 0
+    return sum(clipped_counts.values()) / sum(counts.values())
 
-    >>> for candi_raw in candidate_file:  # doctest: +SKIP
-    ...		ref_raw = ref_file.readline()
-    ...		ref_tokens = word_tokenize(ref_raw)
-    ...		candi_tokens = word_tokenize(candi_raw)
-    ...		total = BLEU.compute(candi_tokens, [ref_tokens], weights)
-    ...		count += 1
 
-    >>> total / count  # doctest: +SKIP
-    2.787504437460048e-05
+def _brevity_penalty(candidate, references):
+    """Calculate brevity penalty.
 
-    [1] Papineni, Kishore, et al. "BLEU: a method for automatic evaluation of
-    machine translation." Proceedings of the 40th annual meeting on
-    association for computational linguistics. Association for Computational
-    Linguistics, 2002.
+    As the modified n-gram precision still has the problem from the short
+    length sentence, brevity penalty is used to modify the overall BLEU
+    score according to length.
 
     """
+    c = len(candidate)
+    r = min(abs(len(r) - c) for r in references)
+
+    if c > r:
+        return 1
+    else:
+        return math.exp(1 - r / c)
 
-    @staticmethod
-    def compute(candidate, references, weights):
-        candidate = [c.lower() for c in candidate]
-        references = [[r.lower() for r in reference] for reference in references]
-
-        p_ns = (BLEU.modified_precision(candidate, references, i)
-                for i, _ in enumerate(weights, start=1))
-        p_ns_nonzero = list(filter(None, p_ns))
-
-        if p_ns_nonzero:
-            s = math.fsum(w * math.log(p_n) for w, p_n in zip(weights, p_ns_nonzero))
-            bp = BLEU.brevity_penalty(candidate, references)
-            return bp * math.exp(s)
-        else: # no alignments
-            return 0
-
-    @staticmethod
-    def modified_precision(candidate, references, n):
-        """ Calculate modified ngram precision.
-
-        >>> BLEU.modified_precision(
-        ...    'the the the the the the the'.split(),
-        ...    ['the cat is on the mat'.split(), 'there is a cat on the mat'.split()],
-        ...    n=1,
-        ... )
-        0.28...
-
-        >>> BLEU.modified_precision(
-        ...    'the the the the the the the'.split(),
-        ...    ['the cat is on the mat'.split(), 'there is a cat on the mat'.split()],
-        ...    n=2,
-        ... )
-        0.0
-
-        >>> BLEU.modified_precision(
-        ...    'of the'.split(),
-        ...    [
-        ...        'It is a guide to action that ensures that the military will forever heed Party commands.'.split(),
-        ...        'It is the guiding principle which guarantees the military forces always being under the command of the Party.'.split(),
-        ...        'It is the practical guide for the army always to heed the directions of the party'.split(),
-        ...    ],
-        ...    n=1,
-        ... )
-        1.0
-
-        >>> BLEU.modified_precision(
-        ...    'of the'.split(),
-        ...    [
-        ...        'It is a guide to action that ensures that the military will forever heed Party commands.'.split(),
-        ...        'It is the guiding principle which guarantees the military forces always being under the command of the Party.'.split(),
-        ...        'It is the practical guide for the army always to heed the directions of the party'.split(),
-        ...    ],
-        ...    n=2,
-        ... )
-        1.0
-
-        """
-        counts = Counter(ngrams(candidate, n))
-
-        if not counts:
-            return 0
-
-        max_counts = {}
-        for reference in references:
-            reference_counts = Counter(ngrams(reference, n))
-            for ngram in counts:
-                max_counts[ngram] = max(max_counts.get(ngram, 0), reference_counts[ngram])
-
-        clipped_counts = dict((ngram, min(count, max_counts[ngram])) for ngram, count in counts.items())
-
-        return sum(clipped_counts.values()) / sum(counts.values())
-
-    @staticmethod
-    def brevity_penalty(candidate, references):
-        c = len(candidate)
-        r = min(abs(len(r) - c) for r in references)
-
-        if c > r:
-            return 1
-        else:
-            return math.exp(1 - r / c)
 
 # run doctests
 if __name__ == "__main__":


Test patch:
diff --git a/nltk/test/bleu.doctest b/nltk/test/bleu.doctest
--- a/nltk/test/bleu.doctest
+++ b/nltk/test/bleu.doctest
@@ -1,15 +1,12 @@
-
 ==========
 BLEU tests
 ==========
 
->>> from nltk.align.bleu import BLEU
-
+>>> from nltk.align.bleu import bleu
 
-If the candidate has no alignment to any of the references, the BLEU score is
-0.
+If the candidate has no alignment to any of the references, the BLEU score is 0.
 
->>> BLEU.compute(
+>>> bleu(
 ...     'John loves Mary'.split(),
 ...     ['The candidate has no alignment to any of the references'.split()],
 ...     [1],