Repository: mjpost/sacrebleu (language: python)

Issue:
TER between two empty strings is 100
I am not sure whether this is expected behavior, but I just found out that when calculating TER between two empty strings yields a TER score of 100. I would expect 0.

The use case is subtitle correction. If the subtitle system does not generate a subtitle (empty string), and some post-editors correct that with a subtitle and others do not, I would expect that the edited version have a higher TER (insertions) whereas those that did not change the empty string would yield 0.

Is this expected behavior in sacrebleu/the original TERCOM implementation? If so, what is the reasoning behind it? Perhaps that "typically" a reference must always be a non empty-string? I understand that the final TER score is normalized by the number of reference tokens, but it is not intuitive that the score is `100` as a result of dividing by `0`.

```python
from sacrebleu import TER

ter_metric = TER() 

ref = ""
hyp = ""
score = ter_metric.sentence_score(hypothesis=hyp, references=[ref]).score

print(score)
# 100
```
TER between two empty strings is 100
I am not sure whether this is expected behavior, but I just found out that when calculating TER between two empty strings yields a TER score of 100. I would expect 0.

The use case is subtitle correction. If the subtitle system does not generate a subtitle (empty string), and some post-editors correct that with a subtitle and others do not, I would expect that the edited version have a higher TER (insertions) whereas those that did not change the empty string would yield 0.

Is this expected behavior in sacrebleu/the original TERCOM implementation? If so, what is the reasoning behind it? Perhaps that "typically" a reference must always be a non empty-string? I understand that the final TER score is normalized by the number of reference tokens, but it is not intuitive that the score is `100` as a result of dividing by `0`.

```python
from sacrebleu import TER

ter_metric = TER() 

ref = ""
hyp = ""
score = ter_metric.sentence_score(hypothesis=hyp, references=[ref]).score

print(score)
# 100
```


Discussion / hints:
Any ideas @ales-t ?
Good question. I've done a quick check of `tercom` outputs:

* For an empty reference and non-empty hypothesis, the output is `Infinity`. So `sacrebleu` is not consistent with `tercom` here.
* If both are empty, `tercom` outputs `NaN`.

Since we're not consistent with tercom here anyway, maybe it would be worthwhile to change `sacrebleu`'s behavior so that if the hypothesis matches, we would also return TER=0? (Seems like a good choice to me, at least at first glance.)
Any ideas @ales-t ?
Good question. I've done a quick check of `tercom` outputs:

* For an empty reference and non-empty hypothesis, the output is `Infinity`. So `sacrebleu` is not consistent with `tercom` here.
* If both are empty, `tercom` outputs `NaN`.

Since we're not consistent with tercom here anyway, maybe it would be worthwhile to change `sacrebleu`'s behavior so that if the hypothesis matches, we would also return TER=0? (Seems like a good choice to me, at least at first glance.)

Fix patch:
diff --git a/sacrebleu/metrics/ter.py b/sacrebleu/metrics/ter.py
index 3078656..efb45fc 100644
--- a/sacrebleu/metrics/ter.py
+++ b/sacrebleu/metrics/ter.py
@@ -125,7 +125,14 @@ def _compute_score_from_stats(self, stats: List[float]) -> TERScore:
         :return: A `TERScore` object.
         """
         total_edits, sum_ref_lengths = stats[0], stats[1]
-        score = total_edits / sum_ref_lengths if sum_ref_lengths > 0 else 1
+
+        if sum_ref_lengths > 0:
+            score = total_edits / sum_ref_lengths
+        elif total_edits > 0:
+            score = 1.0  # empty reference(s) and non-empty hypothesis
+        else:
+            score = 0.0  # both reference(s) and hypothesis are empty
+
         return TERScore(100 * score, total_edits, sum_ref_lengths)
 
     def _aggregate_and_compute(self, stats: List[List[float]]) -> TERScore:


Test patch:
diff --git a/test/test_ter.py b/test/test_ter.py
index 8457a5c..df650a9 100644
--- a/test/test_ter.py
+++ b/test/test_ter.py
@@ -7,6 +7,8 @@
     (['aaaa bbbb cccc dddd'], ['aaaa bbbb cccc dddd'], 0),  # perfect match
     (['dddd eeee ffff'], ['aaaa bbbb cccc'], 1),  # no overlap
     ([''], ['a'], 1),  # corner case, empty hypothesis
+    (['a'], [''], 1),  # corner case, empty reference
+    ([''], [''], 0),  # corner case, both reference and hypothesis empty, we define it as 0.0
     (['d e f g h a b c'], ['a b c d e f g h'], 1 / 8),  # a single shift fixes MT
     (
         [