Repository: huggingface/datasets
Pull request: Compute WER metric iteratively
Compute WER metric iteratively to avoid MemoryError.

Fix #2078.

Problem statement:
MemoryError when computing WER metric
Hi, I'm trying to follow the ASR example to try Wav2Vec. This is the code that I use for WER calculation:

```
wer = load_metric("wer")
print(wer.compute(predictions=result["predicted"], references=result["target"]))
```

However, I receive the following exception:

`Traceback (most recent call last):
  File "/home/diego/IpGlobal/wav2vec/test_wav2vec.py", line 51, in <module>
    print(wer.compute(predictions=result["predicted"], references=result["target"]))
  File "/home/diego/miniconda3/envs/wav2vec3.6/lib/python3.6/site-packages/datasets/metric.py", line 403, in compute
    output = self._compute(predictions=predictions, references=references, **kwargs)
  File "/home/diego/.cache/huggingface/modules/datasets_modules/metrics/wer/73b2d32b723b7fb8f204d785c00980ae4d937f12a65466f8fdf78706e2951281/wer.py", line 94, in _compute
    return wer(references, predictions)
  File "/home/diego/miniconda3/envs/wav2vec3.6/lib/python3.6/site-packages/jiwer/measures.py", line 81, in wer
    truth, hypothesis, truth_transform, hypothesis_transform, **kwargs
  File "/home/diego/miniconda3/envs/wav2vec3.6/lib/python3.6/site-packages/jiwer/measures.py", line 192, in compute_measures
    H, S, D, I = _get_operation_counts(truth, hypothesis)
  File "/home/diego/miniconda3/envs/wav2vec3.6/lib/python3.6/site-packages/jiwer/measures.py", line 273, in _get_operation_counts
    editops = Levenshtein.editops(source_string, destination_string)
MemoryError`

My system has more than 10GB of available RAM. Looking at the code, I think that it could be related to the way jiwer does the calculation, as it is pasting all the sentences in a single string before calling Levenshtein editops function.

Hints from issue discussion:
Hi ! Thanks for reporting.
We're indeed using `jiwer` to compute the WER.

Maybe instead of calling `jiwer.wer` once for all the preditions/references we can compute the WER iteratively to avoid memory issues ? I'm not too familial with `jiwer` but this must be possible.

Currently the code to compute the WER is defined here:

https://github.com/huggingface/nlp/blob/349ac4398a3bcae6356f14c5754483383a60e8a4/metrics/wer/wer.py#L93-L94

Commit message: Fix style

Patch to review:
diff --git a/metrics/wer/wer.py b/metrics/wer/wer.py
index 624e729f956..0c69f768a41 100644
--- a/metrics/wer/wer.py
+++ b/metrics/wer/wer.py
@@ -14,7 +14,7 @@
 # limitations under the License.
 """ Word Error Ratio (WER) metric. """
 
-from jiwer import wer
+from jiwer import compute_measures
 
 import datasets
 
@@ -91,4 +91,10 @@ def _info(self):
         )
 
     def _compute(self, predictions, references):
-        return wer(references, predictions)
+        incorrect = 0
+        total = 0
+        for prediction, reference in zip(predictions, references):
+            measures = compute_measures(reference, prediction)
+            incorrect += measures["substitutions"] + measures["deletions"] + measures["insertions"]
+            total += measures["substitutions"] + measures["deletions"] + measures["hits"]
+        return incorrect / total


Code review comments:
diff_hunk: @@ -91,4 +91,10 @@ def _info(self):
         )
 
     def _compute(self, predictions, references):
-        return wer(references, predictions)
+        incorrect = 0
+        total = 0
+        for prediction, reference in zip(predictions, references):
+            measures = compute_measures(reference, prediction)
+            incorrect += measures["substitutions"] + measures["deletions"] + measures["insertions"]
+            total += measures["substitutions"] + measures["deletions"] + measures["hits"]
+        return incorrect / total
path: metrics/wer/wer.py
text: @user1:
Just for safety, you may want to handle the case of `total = 0` (e.g., empty set). Unit tests to verify the change would be great as well. Thanks for making this change!

@author:
Thanks @user1 for your comments.

The reasons why I have not explicitly handled the edge case `total = 0` are:
- `compute_measure` does already raise a ValueError exception if any of the `reference`s has length = 0: see https://github.com/jitsi/jiwer/blob/f8e5404e4ddb7259081191443def1b6670480244/jiwer/measures.py#L242
- if `references` and `predictions` have length = 0, the previous behavior was also raising the same ValueError; the current implementation will raise a ZeroDivisionError in this case

In relation to unit tests, currently we do not test scripts. Maybe @user2 can give more insight on this.

@user1:
Sounds good; makes sense!