Repository: pytorch/ignite

Issue:
Using Bleu for batch input
The example provided in docs for Bleu is for Single input. In that case the output from the engine should be in this format
```python

def evaluate_step():
  ...
  predictions = "Predicted Sentence 1".split()
  references = ["Reference Sentence 1".split(), "Reference Sentence 1.2".split()]
  return (predictions, references)

```

When calculating Bleu score for a Batch what is the format for output from the engine?
It should be something like
```python
#For a batch of size 2
predictions = ["Predicted Sentence 1".split(), "Predicted Sentence 2".split()]
references = [["Reference Sentence 1.1".split(), "Reference Sentence 1.2".split()], ["Reference Sentence 2.1".split(), "Reference Sentence 2.2".split()]]
```
Doing this gives an error. 
```python
TypeError: unhashable type: 'list'
```
The typing for `update` requires predictions to be a `Sequence[Any]` and References to be `Sequence[Sequence[Any]]]`. Does this mean the batch input is not possible?


Discussion:
Cc @sdesrozis @gucifer 
@Ishan-Kumar2 Thanks for pointing this out.

You are right, we didn't expose a batchwise api. Note that the internal function to compute the bleu score could be use in that sense. However, it will compute a bleu score per batch (then average the batch scores) which would be different to the actual sentence bleu score.

https://github.com/pytorch/ignite/blob/cc2c1aa9e31c297f8eb8fe14f3301dd201be8e48/ignite/metrics/nlp/bleu.py#L190

Would you like to help on this ? It would be a great improvement to fix the batch usage.

EDIT : diving further on this, we compute the average of the sentence bleu score as the corpus bleu score. It is the macro average bleu score. We could improve the implementation computing the corpus bleu score, or the micro average bleu score, meaning an internal logic refactoring (basically an accumulation of ngram counter, then the brevity penalty computing after sync). 

For details, see the following comment from nltk (inspiration of our implementation)

https://github.com/nltk/nltk/blob/193cf488a3aa31b0e99219f1a8c4eb7567903069/nltk/translate/bleu_score.py#L113
@sdesrozis thanks for the update, I am interested in contributing to this.
I agree we should implement the micro avg version also. Then the `corpus_bleu` can be renamed to `sentence_bleu` and the new `corpus_bleu` should be used for corpus/batch level calculations.
Firstly, the `_corpus_bleu` should be split in 2 parts : (1) the accumulation of ngrams counter (maybe the lenghts of cand/hyp too) and (2) the computation of bp, smoothing and geometric mean. The sentence bleu score (ie macro avg) is (1) + (2) for each sentence and (3) average. The corpus bleu score (ie micro avg) is (1) for each sentence then (2). 

For batch version, the macro avg means apply the sentence score to each sentence of the batch. We must add a loop over the batch, it is quite straighforward. The micro avg is natively fine with batch version (apply (1) on batch then (2)).

For DDP, the work is fine for the actual macro avg. Concerning the micro avg version, a synchronization is needed to sum the different counters and lenghts. 

HTH
Hey @sdesrozis 
Since the `_corpus_bleu` already has a loop to iterate over a batch, what if instead of splitting it into 2 seperate functions, we call it once for micro-avg case and multiple times(for each hypothesis in batch) for macro avg.
Like this

```python
def update(self, output: Tuple[Sequence[Sequence[Any]], Sequence[Sequence[Sequence[Any]]]], macro_averaging=False) -> None:
        y_pred, y = output
        
        if macro_averaging:
            #Calculate bleu score for each hypothesis indiviually and then average
            for refs, hyp in zip(y, y_pred):
                self._sum_of_bleu += self._corpus_bleu(references=[refs], candidates=[hyp])
                self._num_sentences += 1

        else:
            #Calculate bleu for entire corpus
            self._sum_of_bleu += self._corpus_bleu(references=y, candidates=y_pred)
            self._num_sentences += 1
```
What you mentioned is correct only if the input is the whole corpus. 

The corpus is split in batch, so we have to firstly accumulate the ngram counter, then finally compute the bp and mean when the corpus has been covered. It implies multiple update calls (accumulation) and one compute (bp + mean). In other words, the micro average you suggested is not correct. 

And if you consider the distributed computing, it means each processor has a part of the corpus. Again, we accumulate, (sync) then bp + mean.
Oh okay got it. My understanding of how ignite metrics work was slightly incorrect, cleared now!
Will raise a PR soon.

Resolving patch:
diff --git a/ignite/metrics/nlp/bleu.py b/ignite/metrics/nlp/bleu.py
--- a/ignite/metrics/nlp/bleu.py
+++ b/ignite/metrics/nlp/bleu.py
@@ -1,6 +1,5 @@
 import math
-from collections import Counter
-from typing import Any, Callable, Sequence, Tuple, Union, ValuesView
+from typing import Any, Callable, Sequence, Tuple, Union
 
 import torch
 
@@ -29,35 +28,37 @@ def __init__(self, method: str):
             raise ValueError(f"Smooth is not valid (expected: {valid}, got: {method})")
         self.smooth = method
 
-    def __call__(self, numerators: Counter, denominators: Counter) -> Sequence[float]:
+    def __call__(self, numerators: torch.Tensor, denominators: torch.Tensor) -> Sequence[float]:
         method = getattr(self, self.smooth)
         return method(numerators, denominators)
 
     @staticmethod
-    def smooth1(numerators: Counter, denominators: Counter) -> Sequence[float]:
+    def smooth1(numerators: torch.Tensor, denominators: torch.Tensor) -> Sequence[float]:
         epsilon = 0.1
-        denominators_ = [max(1, d) for d in denominators.values()]
-        return [n / d if n != 0 else epsilon / d for n, d in zip(numerators.values(), denominators_)]
+        denominators_ = [max(1, d.item()) for d in denominators]
+        return [n.item() / d if n != 0 else epsilon / d for n, d in zip(numerators, denominators_)]
 
     @staticmethod
-    def nltk_smooth2(numerators: Counter, denominators: Counter) -> Sequence[float]:
-        denominators_ = [max(1, d) for d in denominators.values()]
-        return _Smoother._smooth2(numerators.values(), denominators_)
+    def nltk_smooth2(numerators: torch.Tensor, denominators: torch.Tensor) -> Sequence[float]:
+        denominators_ = torch.tensor([max(1, d.item()) for d in denominators])
+        return _Smoother._smooth2(numerators, denominators_)
 
     @staticmethod
-    def smooth2(numerators: Counter, denominators: Counter) -> Sequence[float]:
-        return _Smoother._smooth2(numerators.values(), denominators.values())
+    def smooth2(numerators: torch.Tensor, denominators: torch.Tensor) -> Sequence[float]:
+        return _Smoother._smooth2(numerators, denominators)
 
     @staticmethod
-    def _smooth2(
-        numerators: Union[ValuesView[int], Sequence[int]], denominators: Union[ValuesView[int], Sequence[int]]
-    ) -> Sequence[float]:
-        return [(n + 1) / (d + 1) if i != 0 else n / d for i, (n, d) in enumerate(zip(numerators, denominators))]
+    def _smooth2(numerators: torch.Tensor, denominators: torch.Tensor) -> Sequence[float]:
+
+        return [
+            (n.item() + 1) / (d.item() + 1) if i != 0 else n.item() / d.item()
+            for i, (n, d) in enumerate(zip(numerators, denominators))
+        ]
 
     @staticmethod
-    def no_smooth(numerators: Counter, denominators: Counter) -> Sequence[float]:
-        denominators_ = [max(1, d) for d in denominators.values()]
-        return [n / d for n, d in zip(numerators.values(), denominators_)]
+    def no_smooth(numerators: torch.Tensor, denominators: torch.Tensor) -> Sequence[float]:
+        denominators_ = [max(1, d) for d in denominators]
+        return [n.item() / d for n, d in zip(numerators, denominators_)]
 
 
 class Bleu(Metric):
@@ -97,6 +98,9 @@ class Bleu(Metric):
         device: specifies which device updates are accumulated on. Setting the
             metric's device to be the same as your ``update`` arguments ensures the ``update`` method is
             non-blocking. By default, CPU.
+        average: specifies which type of averaging to use (macro or micro)
+            for more details refer https://www.nltk.org/_modules/nltk/translate/bleu_score.html
+            Default: "macro"
 
     Example:
 
@@ -114,6 +118,8 @@ class Bleu(Metric):
         print(m.compute())
 
     .. versionadded:: 0.4.5
+    .. versionchanged:: 0.5.0
+        added ``average`` option to handle micro and macro averaging modes.
     """
 
     def __init__(
@@ -122,19 +128,27 @@ def __init__(
         smooth: str = "no_smooth",
         output_transform: Callable = lambda x: x,
         device: Union[str, torch.device] = torch.device("cpu"),
+        average: str = "macro",
     ):
         if ngram <= 0:
             raise ValueError(f"ngram order must be greater than zero (got: {ngram})")
         self.ngrams_order = ngram
         self.weights = [1 / self.ngrams_order] * self.ngrams_order
         self.smoother = _Smoother(method=smooth)
+
+        if average not in ["macro", "micro"]:
+            raise ValueError(f'Average must be either "macro" or "micro" (got: {average})')
+        self.average = average
+
         super(Bleu, self).__init__(output_transform=output_transform, device=device)
 
-    def _corpus_bleu(
-        self, references: Sequence[Sequence[Sequence[Any]]], candidates: Sequence[Sequence[Any]],
-    ) -> float:
-        p_numerators: Counter = Counter()
-        p_denominators: Counter = Counter()
+    def _n_gram_counter(
+        self,
+        references: Sequence[Sequence[Sequence[Any]]],
+        candidates: Sequence[Sequence[Any]],
+        p_numerators: torch.Tensor,
+        p_denominators: torch.Tensor,
+    ) -> Tuple[int, int]:
 
         if len(references) != len(candidates):
             raise ValueError(
@@ -142,6 +156,9 @@ def _corpus_bleu(
                 f"{len(references)})"
             )
 
+        hyp_lengths = 0
+        ref_lengths = 0
+
         # Iterate through each hypothesis and their corresponding references.
         for refs, hyp in zip(references, candidates):
             # For each order of ngram, calculate the numerator and
@@ -151,54 +168,113 @@ def _corpus_bleu(
                 p_numerators[i] += numerator
                 p_denominators[i] += denominator
 
+            # Calculate the hypothesis lengths
+            hyp_lengths += len(hyp)
+
+            # Calculate the closest reference lengths.
+            ref_lengths += _closest_ref_length(refs, len(hyp))
+
+        return hyp_lengths, ref_lengths
+
+    def _brevity_penalty_smoothing(
+        self, p_numerators: torch.Tensor, p_denominators: torch.Tensor, hyp_length_sum: int, ref_length_sum: int,
+    ) -> float:
+
         # Returns 0 if there's no matching n-grams
         # We only need to check for p_numerators[1] == 0, since if there's
         # no unigrams, there won't be any higher order ngrams.
         if p_numerators[1] == 0:
             return 0
 
-        # If no smoother, returns 0 if there's at least one a not matching n-grams
-        if self.smoother.smooth == "no_smooth" and min(p_numerators.values()) == 0:
+        # If no smoother, returns 0 if there's at least one a not matching n-grams]
+        if self.smoother.smooth == "no_smooth" and min(p_numerators[1:]).item() == 0:
             return 0
 
-        # Calculate the hypothesis lengths
-        hyp_lengths = [len(hyp) for hyp in candidates]
-
-        # Calculate the closest reference lengths.
-        ref_lengths = [_closest_ref_length(refs, hyp_len) for refs, hyp_len in zip(references, hyp_lengths)]
-
-        # Sum of hypothesis and references lengths
-        hyp_len = sum(hyp_lengths)
-        ref_len = sum(ref_lengths)
-
         # Calculate corpus-level brevity penalty.
-        if hyp_len < ref_len:
-            bp = math.exp(1 - ref_len / hyp_len) if hyp_len > 0 else 0.0
+        if hyp_length_sum < ref_length_sum:
+            bp = math.exp(1 - ref_length_sum / hyp_length_sum) if hyp_length_sum > 0 else 0.0
         else:
             bp = 1.0
 
         # Smoothing
-        p_n = self.smoother(p_numerators, p_denominators)
+        p_n = self.smoother(p_numerators[1:], p_denominators[1:])
 
         # Compute the geometric mean
         s = [w_i * math.log(p_i) for w_i, p_i in zip(self.weights, p_n)]
         gm = bp * math.exp(math.fsum(s))
         return gm
 
+    def _sentence_bleu(self, references: Sequence[Sequence[Any]], candidates: Sequence[Any],) -> float:
+        return self._corpus_bleu([references], [candidates])
+
+    def _corpus_bleu(
+        self, references: Sequence[Sequence[Sequence[Any]]], candidates: Sequence[Sequence[Any]],
+    ) -> float:
+
+        p_numerators: torch.Tensor = torch.zeros(self.ngrams_order + 1)
+        p_denominators: torch.Tensor = torch.zeros(self.ngrams_order + 1)
+
+        hyp_length_sum, ref_length_sum = self._n_gram_counter(
+            references=references, candidates=candidates, p_numerators=p_numerators, p_denominators=p_denominators,
+        )
+        bleu_score = self._brevity_penalty_smoothing(
+            p_numerators=p_numerators,
+            p_denominators=p_denominators,
+            hyp_length_sum=hyp_length_sum,
+            ref_length_sum=ref_length_sum,
+        )
+
+        return bleu_score
+
     @reinit__is_reduced
     def reset(self) -> None:
-        self._sum_of_bleu = torch.tensor(0.0, dtype=torch.double, device=self._device)
-        self._num_sentences = 0
+
+        if self.average == "macro":
+            self._sum_of_bleu = torch.tensor(0.0, dtype=torch.double, device=self._device)
+            self._num_sentences = 0
+
+        if self.average == "micro":
+            self.p_numerators = torch.zeros(self.ngrams_order + 1)
+            self.p_denominators = torch.zeros(self.ngrams_order + 1)
+            self.hyp_length_sum = 0
+            self.ref_length_sum = 0
 
     @reinit__is_reduced
     def update(self, output: Tuple[Sequence[Sequence[Any]], Sequence[Sequence[Sequence[Any]]]]) -> None:
         y_pred, y = output
-        for _y_pred, _y in zip(y_pred, y):
-            self._sum_of_bleu += self._corpus_bleu(references=[_y], candidates=[_y_pred])
-            self._num_sentences += 1
+
+        if self.average == "macro":
+            for refs, hyp in zip(y, y_pred):
+                self._sum_of_bleu += self._sentence_bleu(references=refs, candidates=hyp)
+                self._num_sentences += 1
+
+        elif self.average == "micro":
+            hyp_lengths, ref_lengths = self._n_gram_counter(
+                references=y, candidates=y_pred, p_numerators=self.p_numerators, p_denominators=self.p_denominators
+            )
+            self.hyp_length_sum += hyp_lengths
+            self.ref_length_sum += ref_lengths
 
     @sync_all_reduce("_sum_of_bleu", "_num_sentences")
-    def compute(self) -> torch.Tensor:
+    def _compute_macro(self) -> torch.Tensor:
         if self._num_sentences == 0:
             raise NotComputableError("Bleu must have at least one example before it can be computed.")
+
         return self._sum_of_bleu / self._num_sentences
+
+    @sync_all_reduce("p_numerators", "p_denominators", "hyp_length_sum", "ref_length_sum")
+    def _compute_micro(self) -> float:
+
+        bleu_score = self._brevity_penalty_smoothing(
+            p_numerators=self.p_numerators,
+            p_denominators=self.p_denominators,
+            hyp_length_sum=self.hyp_length_sum,
+            ref_length_sum=self.ref_length_sum,
+        )
+        return bleu_score
+
+    def compute(self) -> None:
+        if self.average == "macro":
+            return self._compute_macro()
+        elif self.average == "micro":
+            return self._compute_micro()


Test patch:
diff --git a/tests/ignite/metrics/nlp/test_bleu.py b/tests/ignite/metrics/nlp/test_bleu.py
--- a/tests/ignite/metrics/nlp/test_bleu.py
+++ b/tests/ignite/metrics/nlp/test_bleu.py
@@ -1,9 +1,10 @@
 import os
 import warnings
+from collections import Counter
 
 import pytest
 import torch
-from nltk.translate.bleu_score import SmoothingFunction, corpus_bleu
+from nltk.translate.bleu_score import SmoothingFunction, corpus_bleu, sentence_bleu
 
 import ignite.distributed as idist
 from ignite.exceptions import NotComputableError
@@ -28,11 +29,13 @@ def test_wrong_inputs():
     with pytest.raises(NotComputableError):
         Bleu().compute()
 
+    with pytest.raises(ValueError, match='Average must be either "macro" or "micro"'):
+        Bleu(average="macros")
 
-@pytest.mark.parametrize(
-    "candidate, references",
+
+parametrize_args = (
+    "candidates, references",
     [
-        ([["a"], ["a"]]),
         ([["a", "a", "a", "b", "c"]], [[["a", "b", "c"], ["a", "a", "d"]]]),
         corpus.sample_1,
         corpus.sample_2,
@@ -40,105 +43,98 @@ def test_wrong_inputs():
         corpus.sample_4,
     ],
 )
-def test_corpus_bleu(candidate, references):
-    print(candidate, references)
-    for i in range(1, 8):
+
+
+def _test(candidates, references, average, smooth="no_smooth", smooth_nltk_fn=None, ngram_range=8):
+    for i in range(1, ngram_range):
         weights = tuple([1 / i] * i)
-        with warnings.catch_warnings():
-            warnings.simplefilter("ignore")
-            reference = corpus_bleu(references, candidate, weights=weights)
-        bleu = Bleu(ngram=i)
-        assert pytest.approx(reference) == bleu._corpus_bleu(references, candidate)
-        bleu.update((candidate, references))
+        bleu = Bleu(ngram=i, average=average, smooth=smooth)
+
+        if average == "macro":
+            with warnings.catch_warnings():
+                warnings.simplefilter("ignore")
+                reference = sentence_bleu(
+                    references[0], candidates[0], weights=weights, smoothing_function=smooth_nltk_fn
+                )
+            assert pytest.approx(reference) == bleu._sentence_bleu(references[0], candidates[0])
+
+        elif average == "micro":
+            with warnings.catch_warnings():
+                warnings.simplefilter("ignore")
+                reference = corpus_bleu(references, candidates, weights=weights, smoothing_function=smooth_nltk_fn)
+            assert pytest.approx(reference) == bleu._corpus_bleu(references, candidates)
+
+        bleu.update((candidates, references))
         assert pytest.approx(reference) == bleu.compute()
 
 
-@pytest.mark.parametrize(
-    "candidate, references",
-    [
-        ([["a", "a", "a", "b", "c"]], [[["a", "b", "c"], ["a", "a", "d"]]]),
-        corpus.sample_1,
-        corpus.sample_2,
-        corpus.sample_3,
-        corpus.sample_4,
-    ],
-)
-def test_corpus_bleu_smooth1(candidate, references):
-    for i in range(1, 8):
-        weights = tuple([1 / i] * i)
-        with warnings.catch_warnings():
-            warnings.simplefilter("ignore")
-            reference = corpus_bleu(
-                references, candidate, weights=weights, smoothing_function=SmoothingFunction().method1
-            )
-        bleu = Bleu(ngram=i, smooth="smooth1")
-        assert reference == bleu._corpus_bleu(references, candidate)
-        bleu.update((candidate, references))
-        assert reference == bleu.compute()
+@pytest.mark.parametrize(*parametrize_args)
+def test_macro_bleu(candidates, references):
+    _test(candidates, references, "macro")
 
 
-@pytest.mark.parametrize(
-    "candidate, references",
-    [
-        ([["a", "a", "a", "b", "c"]], [[["a", "b", "c"], ["a", "a", "d"]]]),
-        corpus.sample_1,
-        corpus.sample_2,
-        corpus.sample_3,
-        corpus.sample_4,
-    ],
-)
-def test_corpus_bleu_nltk_smooth2(candidate, references):
-    for i in range(1, 8):
-        weights = tuple([1 / i] * i)
-        with warnings.catch_warnings():
-            warnings.simplefilter("ignore")
-            reference = corpus_bleu(
-                references, candidate, weights=weights, smoothing_function=SmoothingFunction().method2
-            )
-        bleu = Bleu(ngram=i, smooth="nltk_smooth2")
-        assert reference == bleu._corpus_bleu(references, candidate)
-        bleu.update((candidate, references))
-        assert reference == bleu.compute()
+@pytest.mark.parametrize(*parametrize_args)
+def test_micro_bleu(candidates, references):
+    _test(candidates, references, "micro")
 
 
-@pytest.mark.parametrize(
-    "candidate, references",
-    [
-        ([["a", "a", "a", "b", "c"]], [[["a", "b", "c"], ["a", "a", "d"]]]),
-        corpus.sample_1,
-        corpus.sample_2,
-        corpus.sample_3,
-        corpus.sample_4,
-    ],
-)
-def test_corpus_bleu_smooth2(candidate, references):
-    for i in range(1, 3):
-        weights = tuple([1 / i] * i)
-        with warnings.catch_warnings():
-            warnings.simplefilter("ignore")
-            reference = corpus_bleu(
-                references, candidate, weights=weights, smoothing_function=SmoothingFunction().method2
-            )
-        bleu = Bleu(ngram=i, smooth="smooth2")
-        assert reference == bleu._corpus_bleu(references, candidate)
-        bleu.update((candidate, references))
-        assert reference == bleu.compute()
+@pytest.mark.parametrize(*parametrize_args)
+def test_macro_bleu_smooth1(candidates, references):
+    _test(candidates, references, "macro", "smooth1", SmoothingFunction().method1)
+
+
+@pytest.mark.parametrize(*parametrize_args)
+def test_micro_bleu_smooth1(candidates, references):
+    _test(candidates, references, "micro", "smooth1", SmoothingFunction().method1)
+
+
+@pytest.mark.parametrize(*parametrize_args)
+def test_macro_bleu_nltk_smooth2(candidates, references):
+    _test(candidates, references, "macro", "nltk_smooth2", SmoothingFunction().method2)
+
+
+@pytest.mark.parametrize(*parametrize_args)
+def test_micro_bleu_nltk_smooth2(candidates, references):
+    _test(candidates, references, "micro", "nltk_smooth2", SmoothingFunction().method2)
+
+
+@pytest.mark.parametrize(*parametrize_args)
+def test_macro_bleu_smooth2(candidates, references):
+    _test(candidates, references, "macro", "smooth2", SmoothingFunction().method2, 3)
+
 
+@pytest.mark.parametrize(*parametrize_args)
+def test_micro_bleu_smooth2(candidates, references):
+    _test(candidates, references, "micro", "smooth2", SmoothingFunction().method2, 3)
 
-def test_bleu():
+
+def test_accumulation_macro_bleu():
     bleu = Bleu(ngram=4, smooth="smooth2")
     bleu.update(([corpus.cand_1], [corpus.references_1]))
     bleu.update(([corpus.cand_2a], [corpus.references_2]))
     bleu.update(([corpus.cand_2b], [corpus.references_2]))
     bleu.update(([corpus.cand_3], [corpus.references_2]))
-    value = bleu._corpus_bleu([corpus.references_1], [corpus.cand_1])
-    value += bleu._corpus_bleu([corpus.references_2], [corpus.cand_2a])
-    value += bleu._corpus_bleu([corpus.references_2], [corpus.cand_2b])
-    value += bleu._corpus_bleu([corpus.references_2], [corpus.cand_3])
+    value = bleu._sentence_bleu(corpus.references_1, corpus.cand_1)
+    value += bleu._sentence_bleu(corpus.references_2, corpus.cand_2a)
+    value += bleu._sentence_bleu(corpus.references_2, corpus.cand_2b)
+    value += bleu._sentence_bleu(corpus.references_2, corpus.cand_3)
     assert bleu.compute() == value / 4
 
 
-def test_bleu_batch():
+def test_accumulation_micro_bleu():
+    bleu = Bleu(ngram=4, smooth="smooth2", average="micro")
+    bleu.update(([corpus.cand_1], [corpus.references_1]))
+    bleu.update(([corpus.cand_2a], [corpus.references_2]))
+    bleu.update(([corpus.cand_2b], [corpus.references_2]))
+    bleu.update(([corpus.cand_3], [corpus.references_2]))
+    value = bleu._corpus_bleu(
+        [corpus.references_1, corpus.references_2, corpus.references_2, corpus.references_2],
+        [corpus.cand_1, corpus.cand_2a, corpus.cand_2b, corpus.cand_3],
+    )
+    assert bleu.compute() == value
+
+
+def test_bleu_batch_macro():
     bleu = Bleu(ngram=4)
 
     # Batch size 3
@@ -149,25 +145,61 @@ def test_bleu_batch():
     with warnings.catch_warnings():
         warnings.simplefilter("ignore")
         reference_bleu_score = (
-            corpus_bleu([refs[0]], [hypotheses[0]])
-            + corpus_bleu([refs[1]], [hypotheses[1]])
-            + corpus_bleu([refs[2]], [hypotheses[2]])
+            sentence_bleu(refs[0], hypotheses[0])
+            + sentence_bleu(refs[1], hypotheses[1])
+            + sentence_bleu(refs[2], hypotheses[2])
         ) / 3
-    assert bleu.compute() == reference_bleu_score
+    assert pytest.approx(bleu.compute()) == reference_bleu_score
 
     value = 0
     for _hypotheses, _refs in zip(hypotheses, refs):
-        value += bleu._corpus_bleu([_refs], [_hypotheses])
+        value += bleu._sentence_bleu(_refs, _hypotheses)
         bleu.update(([_hypotheses], [_refs]))
 
     ref_1 = value / len(refs)
     ref_2 = bleu.compute()
 
-    assert ref_1 == reference_bleu_score
-    assert ref_2 == reference_bleu_score
+    assert pytest.approx(ref_1) == reference_bleu_score
+    assert pytest.approx(ref_2) == reference_bleu_score
+
 
+def test_bleu_batch_micro():
+    bleu = Bleu(ngram=4, average="micro")
+
+    # Batch size 3
+    hypotheses = [corpus.cand_1, corpus.cand_2a, corpus.cand_2b]
+    refs = [corpus.references_1, corpus.references_2, corpus.references_2]
+    bleu.update((hypotheses, refs))
+
+    with warnings.catch_warnings():
+        warnings.simplefilter("ignore")
+        reference_bleu_score = corpus_bleu(refs, hypotheses)
+    assert pytest.approx(bleu.compute()) == reference_bleu_score
 
-def _test_distrib_integration(device):
+    assert pytest.approx(bleu._corpus_bleu(refs, hypotheses)) == reference_bleu_score
+
+
+@pytest.mark.parametrize(
+    "candidates, references",
+    [
+        (corpus.cand_1, corpus.references_1),
+        (corpus.cand_2a, corpus.references_2),
+        (corpus.cand_2b, corpus.references_2),
+        (corpus.cand_1, corpus.references_1),
+    ],
+)
+def test_n_gram_counter(candidates, references):
+    bleu = Bleu(ngram=4)
+    hyp_length, ref_length = bleu._n_gram_counter([references], [candidates], Counter(), Counter())
+    assert hyp_length == len(candidates)
+
+    ref_lens = (len(reference) for reference in references)
+    closest_ref_len = min(ref_lens, key=lambda ref_len: (abs(ref_len - len(candidates)), ref_len))
+
+    assert ref_length == closest_ref_len
+
+
+def _test_macro_distrib_integration(device):
 
     from ignite.engine import Engine
 
@@ -192,12 +224,12 @@ def _test(metric_device):
         assert "bleu" in engine.state.metrics
 
         ref_bleu = 0
-        for candidate, references in data:
+        for candidates, references in data:
             with warnings.catch_warnings():
                 warnings.simplefilter("ignore")
-                ref_bleu += corpus_bleu(
-                    references,
-                    candidate,
+                ref_bleu += sentence_bleu(
+                    references[0],
+                    candidates[0],
                     weights=[0.25, 0.25, 0.25, 0.25],
                     smoothing_function=SmoothingFunction().method2,
                 )
@@ -210,13 +242,61 @@ def _test(metric_device):
         _test(idist.device())
 
 
+def _test_micro_distrib_integration(device):
+
+    from ignite.engine import Engine
+
+    rank = idist.get_rank()
+
+    size = len(corpus.chunks)
+
+    data = []
+    for c in corpus.chunks:
+        data += idist.get_world_size() * [c]
+
+    def update(_, i):
+        return data[i + size * rank]
+
+    def _test(metric_device):
+        engine = Engine(update)
+        m = Bleu(ngram=4, smooth="smooth2", average="micro")
+        m.attach(engine, "bleu")
+
+        engine.run(data=list(range(size)), max_epochs=1)
+
+        assert "bleu" in engine.state.metrics
+
+        ref_bleu = 0
+        references = []
+        candidates = []
+        for _candidates, _references in data:
+            references.append(_references[0])
+            candidates.append(_candidates[0])
+        with warnings.catch_warnings():
+            warnings.simplefilter("ignore")
+            ref_bleu += corpus_bleu(
+                references,
+                candidates,
+                weights=[0.25, 0.25, 0.25, 0.25],
+                smoothing_function=SmoothingFunction().method2,
+            )
+
+        assert pytest.approx(engine.state.metrics["bleu"]) == ref_bleu
+
+    _test("cpu")
+
+    if device.type != "xla":
+        _test(idist.device())
+
+
 @pytest.mark.distributed
 @pytest.mark.skipif(not idist.has_native_dist_support, reason="Skip if no native dist support")
 @pytest.mark.skipif(torch.cuda.device_count() < 1, reason="Skip if no GPU")
 def test_distrib_nccl_gpu(distributed_context_single_node_nccl):
 
     device = idist.device()
-    _test_distrib_integration(device)
+    _test_macro_distrib_integration(device)
+    _test_micro_distrib_integration(device)
 
 
 @pytest.mark.distributed
@@ -224,7 +304,8 @@ def test_distrib_nccl_gpu(distributed_context_single_node_nccl):
 def test_distrib_gloo_cpu_or_gpu(distributed_context_single_node_gloo):
 
     device = idist.device()
-    _test_distrib_integration(device)
+    _test_macro_distrib_integration(device)
+    _test_micro_distrib_integration(device)
 
 
 @pytest.mark.distributed
@@ -235,7 +316,8 @@ def test_distrib_hvd(gloo_hvd_executor):
     device = torch.device("cpu" if not torch.cuda.is_available() else "cuda")
     nproc = 4 if not torch.cuda.is_available() else torch.cuda.device_count()
 
-    gloo_hvd_executor(_test_distrib_integration, (device,), np=nproc, do_init=True)
+    gloo_hvd_executor(_test_macro_distrib_integration, (device,), np=nproc, do_init=True)
+    gloo_hvd_executor(_test_micro_distrib_integration, (device,), np=nproc, do_init=True)
 
 
 @pytest.mark.multinode_distributed
@@ -244,7 +326,8 @@ def test_distrib_hvd(gloo_hvd_executor):
 def test_multinode_distrib_gloo_cpu_or_gpu(distributed_context_multi_node_gloo):
 
     device = idist.device()
-    _test_distrib_integration(device)
+    _test_macro_distrib_integration(device)
+    _test_micro_distrib_integration(device)
 
 
 @pytest.mark.multinode_distributed
@@ -253,7 +336,8 @@ def test_multinode_distrib_gloo_cpu_or_gpu(distributed_context_multi_node_gloo):
 def test_multinode_distrib_nccl_gpu(distributed_context_multi_node_nccl):
 
     device = idist.device()
-    _test_distrib_integration(device)
+    _test_macro_distrib_integration(device)
+    _test_micro_distrib_integration(device)
 
 
 @pytest.mark.tpu
@@ -261,12 +345,14 @@ def test_multinode_distrib_nccl_gpu(distributed_context_multi_node_nccl):
 @pytest.mark.skipif(not idist.has_xla_support, reason="Skip if no PyTorch XLA package")
 def test_distrib_single_device_xla():
     device = idist.device()
-    _test_distrib_integration(device)
+    _test_macro_distrib_integration(device)
+    _test_micro_distrib_integration(device)
 
 
 def _test_distrib_xla_nprocs(index):
     device = idist.device()
-    _test_distrib_integration(device)
+    _test_macro_distrib_integration(device)
+    _test_micro_distrib_integration(device)
 
 
 @pytest.mark.tpu