Repository: nltk/nltk

Issue:
nltk.translate.gleu_score problems
I see a few possible issues with the gleu_score implementation.

1. It doesn't allow multiple references, despite the [docstring saying it does](https://github.com/nltk/nltk/blob/develop/nltk/translate/gleu_score.py#L72). This is either a code or a doc issue. Can anyone clarify which?

2. I don't see why there is no `corpus_gleu()` function. The Wu et al. 2016 paper states (emphasis added):

   >According to our experiments, GLEU score correlates quite well with the BLEU metric **on a corpus level** but does not have its drawbacks for our per sentence reward objective

   So I would think there's a way to calculate this for the corpus. Maybe just `min(corpus_precision, corpus_recall)`, or alternatively the average of sentence-level GLEU scores?

3. The name GLEU is unfortunate; there are at least 2 other GLEUs that predate the 2016 paper:
   * Mutton et al. (2007) "GLEU: Automatic Evaluation of Sentence-Level Fluency" http://anthology.aclweb.org/P/P07/P07-1044.pdf
   * Napoles et al. (2015) "Ground Truth for Grammatical Error Correction Metrics" http://www.aclweb.org/anthology/P/P15/P15-2097.pdf

   Maybe the Wu et al. 2016 paper will choose a different name if they publish outside of arXiv? But for NLTK, would `google_bleu` or `gnmt_bleu` be less likely to step on toes?

I'd be happy to submit a PR when the above issues get resolved.


Discussion:
@goodmami I'll try to answer some of the questions:

> It doesn't allow multiple references, despite the docstring saying it does. This is either a code or a doc issue. Can anyone clarify which?

It's a docstring problem. It doesn't take multiple references because the GMNT paper didn't account for multiple references. 

> I don't see why there is no corpus_gleu() function. The Wu et al. 2016 paper states (emphasis added):
> 
> According to our experiments, GLEU score correlates quite well with the BLEU metric on a corpus level but does not have its drawbacks for our per sentence reward objective
>
> So I would think there's a way to calculate this for the corpus. Maybe just min(corpus_precision, corpus_recall), or alternatively the average of sentence-level GLEU scores?

Yes, this was unclear because they were using GLEU to do some sort of reinforcement learning (I would call it nbest reranking) at the decoder, so it must have been something at sentence level. 

As for the corpus level, I don't think it's `min(corpus_precision, corpus_recall)`, the reinforcement learning part wouldn't make sense then. the sentences would have been read in mini-batches. so the closest it gets to `corpus_precision`/`corpus_recall` is `minibatch_precision`/`minibatch_recall`. 

But since it's taking the min at every sentence during the reinforcement learning,  I think the macro-average of sentence level might be more viable. 

I have no idea how they calculate correlations though, p-value?


> The name GLEU is unfortunate; there are at least 2 other GLEUs that predate the 2016 paper:
? 
> Mutton et al. (2007) "GLEU: Automatic Evaluation of Sentence-Level Fluency"  http://anthology.aclweb.org/P/P07/P07-1044.pdf
>
> Napoles et al. (2015) "Ground Truth for Grammatical Error Correction Metrics" 
http://www.aclweb.org/anthology/P/P15/P15-2097.pdf
>
> Maybe the Wu et al. 2016 paper will choose a different name if they publish outside of arXiv? But for NLTK, would google_bleu or gnmt_bleu be less likely to step on toes?

Cool! I didn't know there's other GLEU =) 

Then I agree, the name in NLTK should be changed. 
Ok, thanks @alvations. Maybe we should contact an author of the paper to get a more precise description of the metric before modifying the implementation?

And the docstring problem was not as bad as I thought. It was just the plural `references` instead of `reference`. (I misinterpreted `list(str)` as the type to mean a list of strings rather than a list of tokens.) Anyway, I corrected the plural, added a note about the single-reference limitation, and fixed an unrelated copy-paste docstring bug on the return value (it currently says it returns the `CHRF score`). Pull request here: https://github.com/nltk/nltk/pull/1668

So maybe let's wait to (a) rename the module and (b) possibly add multi-reference or corpus metrics until after we've heard back from someone on the project or have seen an official implementation.
Just summarizing what Mike Schuster (2nd author on Wu et al. 2016) said on an nltk-dev thread:

Nothing special for multiple-references; suggests using the max score:
* Q:
   > [...] Are we correct in assuming that GLEU does not provide any similar special formulation for multiple references?
* A:
   > Yes, there is nothing special for multiple references (we used the GLEU score before we were using multiple references). I would use the reference with the highest score in this case.

Corpus GLEU is not just the macro-average:
* Q:
  > [...] is the corpus-level GLEU just the mean of the sentence-level scores?
* A:
  > For the corpus, we just add up the two statistics n_match and n_all = max(n_all_output, n_all_target) for all sentences, then calculate gleu_score = n_match / n_all, so it is *not* just a mean of the sentence gleu scores (in our case, longer sentences count more, which I think makes sense as they are more difficult to translate).

"GLEU" name was used internally and team was not aware of previous instances. Fate uncertain? At least we should be clear about which "GLEU" we are using, which I think we are.
* Q:
  > Is there a finalized name for the metric? "GLEU" has been used [...] (e.g., http://anthology.aclweb.org/P/P07/P07-1044.pdf) [...].
* A:
  > Wow, we didn't know about this -- this is unfortunate! We first called it internally Google-BLEU which became GLEU -- I am not sure what is the best to do in this case. Maybe just reference the paper to make clear which GLEU score it is?

Resolving patch:
diff --git a/nltk/translate/gleu_score.py b/nltk/translate/gleu_score.py
--- a/nltk/translate/gleu_score.py
+++ b/nltk/translate/gleu_score.py
@@ -3,7 +3,7 @@
 #
 # Copyright (C) 2001-2017 NLTK Project
 # Authors:
-# Contributors:
+# Contributors: Mike Schuster, Michael Wayne Goodman, Liling Tan
 # URL: <http://nltk.org/>
 # For license information, see LICENSE.TXT
 
@@ -13,7 +13,8 @@
 
 from nltk.util import ngrams, everygrams
 
-def sentence_gleu(reference, hypothesis, min_len=1, max_len=4):
+
+def sentence_gleu(references, hypothesis, min_len=1, max_len=4):
     """
     Calculates the sentence level GLEU (Google-BLEU) score described in
 
@@ -45,15 +46,15 @@ def sentence_gleu(reference, hypothesis, min_len=1, max_len=4):
          metric on a corpus level but does not have its drawbacks for our per
          sentence reward objective."
 
-    Note: The GLEU score is designed for sentence based evaluation thus there is
-          no corpus based scores implemented in NLTK. Also, unlike
-          multi-reference BLEU, GLEU only supports a single reference.
+    Note: The initial implementation only allowed a single reference, but now
+          a list of references is required (which is consistent with
+          bleu_score.sentence_bleu()).
 
     The infamous "the the the ... " example
 
         >>> ref = 'the cat is on the mat'.split()
         >>> hyp = 'the the the the the the the'.split()
-        >>> sentence_gleu(ref, hyp)  # doctest: +ELLIPSIS
+        >>> sentence_gleu([ref], hyp)  # doctest: +ELLIPSIS
         0.0909...
 
     An example to evaluate normal machine translation outputs
@@ -64,13 +65,13 @@ def sentence_gleu(reference, hypothesis, min_len=1, max_len=4):
         ...            'always obeys the commands of the party').split()
         >>> hyp2 = str('It is to insure the troops forever hearing the activity '
         ...            'guidebook that party direct').split()
-        >>> sentence_gleu(ref1, hyp1) # doctest: +ELLIPSIS
+        >>> sentence_gleu([ref1], hyp1) # doctest: +ELLIPSIS
         0.4393...
-        >>> sentence_gleu(ref1, hyp2) # doctest: +ELLIPSIS
+        >>> sentence_gleu([ref1], hyp2) # doctest: +ELLIPSIS
         0.1206...
 
-    :param reference: a reference sentence
-    :type reference: list(str)
+    :param references: a list of reference sentences
+    :type references: list(list(str))
     :param hypothesis: a hypothesis sentence
     :type hypothesis: list(str)
     :param min_len: The minimum order of n-gram this function should extract.
@@ -80,16 +81,113 @@ def sentence_gleu(reference, hypothesis, min_len=1, max_len=4):
     :return: the sentence level GLEU score.
     :rtype: float
     """
-    # For each order of ngram, calculate the no. of ngram matches and
-    # keep track of no. of ngram in references.
-    ref_ngrams = Counter(everygrams(reference, min_len, max_len))
-    hyp_ngrams = Counter(everygrams(hypothesis, min_len, max_len))
-    overlap_ngrams = ref_ngrams & hyp_ngrams
-    tp = sum(overlap_ngrams.values()) # True positives.
-    tpfp = sum(hyp_ngrams.values()) # True positives + False positives.
-    tffn = sum(ref_ngrams.values()) # True posities + False negatives.
-
-    precision = tp / tpfp
-    recall = tp / tffn
-
-    return min(precision, recall)
+    return corpus_gleu(
+        [references],
+        [hypothesis],
+        min_len=min_len,
+        max_len=max_len
+    )
+
+def corpus_gleu(list_of_references, hypotheses, min_len=1, max_len=4):
+    """
+    Calculate a single corpus-level GLEU score (aka. system-level GLEU) for all
+    the hypotheses and their respective references.
+
+    Instead of averaging the sentence level GLEU scores (i.e. macro-average
+    precision), Wu et al. (2016) sum up the matching tokens and the max of
+    hypothesis and reference tokens for each sentence, then compute using the
+    aggregate values.
+
+    From Mike Schuster (via email):
+        "For the corpus, we just add up the two statistics n_match and
+         n_all = max(n_all_output, n_all_target) for all sentences, then
+         calculate gleu_score = n_match / n_all, so it is not just a mean of
+         the sentence gleu scores (in our case, longer sentences count more,
+         which I think makes sense as they are more difficult to translate)."
+
+    >>> hyp1 = ['It', 'is', 'a', 'guide', 'to', 'action', 'which',
+    ...         'ensures', 'that', 'the', 'military', 'always',
+    ...         'obeys', 'the', 'commands', 'of', 'the', 'party']
+    >>> ref1a = ['It', 'is', 'a', 'guide', 'to', 'action', 'that',
+    ...          'ensures', 'that', 'the', 'military', 'will', 'forever',
+    ...          'heed', 'Party', 'commands']
+    >>> ref1b = ['It', 'is', 'the', 'guiding', 'principle', 'which',
+    ...          'guarantees', 'the', 'military', 'forces', 'always',
+    ...          'being', 'under', 'the', 'command', 'of', 'the', 'Party']
+    >>> ref1c = ['It', 'is', 'the', 'practical', 'guide', 'for', 'the',
+    ...          'army', 'always', 'to', 'heed', 'the', 'directions',
+    ...          'of', 'the', 'party']
+
+    >>> hyp2 = ['he', 'read', 'the', 'book', 'because', 'he', 'was',
+    ...         'interested', 'in', 'world', 'history']
+    >>> ref2a = ['he', 'was', 'interested', 'in', 'world', 'history',
+    ...          'because', 'he', 'read', 'the', 'book']
+
+    >>> list_of_references = [[ref1a, ref1b, ref1c], [ref2a]]
+    >>> hypotheses = [hyp1, hyp2]
+    >>> corpus_gleu(list_of_references, hypotheses) # doctest: +ELLIPSIS
+    0.5673...
+
+    The example below show that corpus_gleu() is different from averaging
+    sentence_gleu() for hypotheses
+
+    >>> score1 = sentence_gleu([ref1a], hyp1)
+    >>> score2 = sentence_gleu([ref2a], hyp2)
+    >>> (score1 + score2) / 2 # doctest: +ELLIPSIS
+    0.6144...
+
+    :param list_of_references: a list of reference sentences, w.r.t. hypotheses
+    :type list_of_references: list(list(list(str)))
+    :param hypotheses: a list of hypothesis sentences
+    :type hypotheses: list(list(str))
+    :param min_len: The minimum order of n-gram this function should extract.
+    :type min_len: int
+    :param max_len: The maximum order of n-gram this function should extract.
+    :type max_len: int
+    :return: The corpus-level GLEU score.
+    :rtype: float
+    """
+    # sanity check
+    assert len(list_of_references) == len(hypotheses), "The number of hypotheses and their reference(s) should be the same"
+
+    # sum matches and max-token-lengths over all sentences
+    corpus_n_match = 0
+    corpus_n_all = 0
+
+    for references, hypothesis in zip(list_of_references, hypotheses):
+        hyp_ngrams = Counter(everygrams(hypothesis, min_len, max_len))
+        tpfp = sum(hyp_ngrams.values())  # True positives + False positives.
+        
+        hyp_counts = []
+        for reference in references:
+            ref_ngrams = Counter(everygrams(reference, min_len, max_len))
+            tpfn = sum(ref_ngrams.values())  # True positives + False negatives.
+
+            overlap_ngrams = ref_ngrams & hyp_ngrams
+            tp = sum(overlap_ngrams.values())  # True positives.
+
+            # While GLEU is defined as the minimum of precision and
+            # recall, we can reduce the number of division operations by one by
+            # instead finding the maximum of the denominators for the precision
+            # and recall formulae, since the numerators are the same:
+            #     precision = tp / tpfp
+            #     recall = tp / tpfn
+            #     gleu_score = min(precision, recall) == tp / max(tpfp, tpfn)
+            n_all = max(tpfp, tpfn)
+
+            if n_all > 0:
+                hyp_counts.append((tp, n_all))
+
+        # use the reference yielding the highest score
+        if hyp_counts:
+            n_match, n_all = max(hyp_counts, key=lambda hc: hc[0]/hc[1])
+            corpus_n_match += n_match
+            corpus_n_all += n_all
+
+    # corner case: empty corpus or empty references---don't divide by zero!
+    if corpus_n_all == 0:
+        gleu_score = 0.0
+    else:
+        gleu_score = corpus_n_match / corpus_n_all
+
+    return gleu_score


Test patch: