Repository: danieldeutsch/repro (language: python)

Issue:
Add MTEQA
https://github.com/ufal/MTEQA


Discussion / hints:

Fix patch:
diff --git a/Changelog.md b/Changelog.md
index 6cef7ea..2444856 100644
--- a/Changelog.md
+++ b/Changelog.md
@@ -9,6 +9,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ## Added
 - Added the [`ParallelModel`](https://repro.readthedocs.io/en/latest/api/repro.models.model.html#repro.models.model.ParallelModel) class as an easy abstraction over the `joblib` library for parallel computation.
 - Added an [`aggregate_parallel_metrics`](https://repro.readthedocs.io/en/latest/api/repro.common.util.html#repro.common.util.aggregate_parallel_metrics) function to make using metrics in parallel easier.
+- Added [MTEQE](https://repro.readthedocs.io/en/latest/models/krubinski2021.html)
 
 ## [v0.1.5](https://github.com/danieldeutsch/repro/releases/tag/v0.1.5) - 2022-03-21
 ## Added
diff --git a/docs/source/papers.md b/docs/source/papers.md
index 7112b21..bd69eab 100644
--- a/docs/source/papers.md
+++ b/docs/source/papers.md
@@ -33,6 +33,7 @@ The following papers have models implemented in Repro.
 |[Automatic Text Evaluation through the Lens of Wasserstein Barycenters](https://arxiv.org/abs/2108.12463)|Pierre Colombo, Guillaume Staerman, Chloe Clavel, and Pablo Piantanida|[Link](colombo2021.md)|
 |[InfoLM: A New Metric to Evaluate Summarization & Data2Text Generation](https://arxiv.org/abs/2112.01589)|Pierre Colombo, Chloe Clavel, and Pablo Piantanida|[Link](colombo2021.md)|
 |[A Pseudo-Metric between Probability Distributions based on Depth-Trimmed Regions](https://arxiv.org/abs/2103.12711)|Guillaume Staerman, Pavlo Mozharovskyi, Pierre Colombo, Stéphan Clémençon, and Florence d'Alché-Buc|[Link](colombo2021.md)|
+|[Just Ask! Evaluating Machine Translation by Asking and Answering Questions](https://www.statmt.org/wmt21/pdf/2021.wmt-1.58.pdf)|Mateusz Krubinski, Erfan Ghadery, Marie-Francine Moens, Pavel Pecina|[Link](krubinski2021.md)|
 
 ## Question Answering
 |Paper|Authors|Docs|
diff --git a/models/krubinski2021/Dockerfile b/models/krubinski2021/Dockerfile
new file mode 100644
index 0000000..6d37588
--- /dev/null
+++ b/models/krubinski2021/Dockerfile
@@ -0,0 +1,22 @@
+FROM danieldeutsch/python:3.8-cuda11.1.1-base
+
+WORKDIR /app
+
+# Download the code
+RUN git clone https://github.com/ufal/MTEQA && \
+    cd MTEQA && \
+    git checkout d1d8b676af4f840ce5a712a41215e880daf81764 && \
+    git submodule init && \
+    git submodule update
+
+# Install the required libraries. The library requires torch==1.8.0
+# but the default CUDA version is incompatible, so we reinstall the right one
+RUN cd MTEQA && \
+    pip install -r requirements.txt --no-cache-dir && \
+    pip install torch==1.8.0+cu111 -f https://download.pytorch.org/whl/torch_stable.html
+
+# Download the models
+RUN cd MTEQA && python download_models.py
+
+# Download NLTK models
+RUN python -c "import nltk; nltk.download('punkt')"
\ No newline at end of file
diff --git a/models/krubinski2021/Readme.md b/models/krubinski2021/Readme.md
new file mode 100644
index 0000000..d57a74d
--- /dev/null
+++ b/models/krubinski2021/Readme.md
@@ -0,0 +1,52 @@
+# Krubinski et al. (2021)
+
+## Publication
+[Just Ask! Evaluating Machine Translation by Asking and Answering Questions](https://www.statmt.org/wmt21/pdf/2021.wmt-1.58.pdf)
+
+## Repositories
+https://github.com/ufal/MTEQA
+
+## Available Models
+- MTEQA:
+  - Description: A QA-based evaluation metric for MT
+  - Name: `krubinski2021-mteqa`
+  - Usage:
+    ```python
+    from repro.models.krubinski2021 import MTEQA
+    model = MTEQA()
+    macro, micro = model.predict_batch(
+        {"candidate": "The candidate translation", "references": ["The references"]}
+    )
+    ```
+    `macro` and `micro` are the average and per-input scores.
+    
+## Implementation Notes
+    
+## Docker Information
+- Image name: `danieldeutsch/krubinski2021`
+- Docker Hub: 
+- Build command:
+  ```shell script
+  repro setup krubinski2021
+  ```
+- Requires network: No
+  
+## Testing
+```shell script
+repro setup krubinski2021
+pytest models/krubinski2021/tests
+```
+
+## Status
+- [x] Regression unit tests pass  
+See [here](https://github.com/danieldeutsch/repro/actions/runs/2276486888) 
+- [ ] Correctness unit tests pass  
+None provided or tested
+- [ ] Model runs on full test dataset  
+Not Tested
+- [ ] Predictions approximately replicate results reported in the paper  
+Not Tested
+- [ ] Predictions exactly replicate results reported in the paper  
+Not Tested
+
+## Changelog
diff --git a/repro/models/krubinski2021/__init__.py b/repro/models/krubinski2021/__init__.py
new file mode 100644
index 0000000..3e64a83
--- /dev/null
+++ b/repro/models/krubinski2021/__init__.py
@@ -0,0 +1,10 @@
+import os
+
+VERSION = "1.0"
+MODEL_NAME = os.path.basename(os.path.dirname(__file__))
+DOCKERHUB_REPO = f"danieldeutsch/{MODEL_NAME}"
+DEFAULT_IMAGE = f"{DOCKERHUB_REPO}:{VERSION}"
+AUTOMATICALLY_PUBLISH = True
+
+from repro.models.krubinski2021.model import MTEQA
+from repro.models.krubinski2021.setup import Krubinski2021SetupSubcommand
diff --git a/repro/models/krubinski2021/model.py b/repro/models/krubinski2021/model.py
new file mode 100644
index 0000000..5533176
--- /dev/null
+++ b/repro/models/krubinski2021/model.py
@@ -0,0 +1,135 @@
+import logging
+from collections import defaultdict
+from typing import Dict, List, Tuple, Union
+
+from repro.common import util
+from repro.common.docker import DockerContainer
+from repro.data.types import MetricsType
+from repro.models import Model
+from repro.models.krubinski2021 import DEFAULT_IMAGE, MODEL_NAME
+
+logger = logging.getLogger(__name__)
+
+
+@Model.register(f"{MODEL_NAME}-mteqa")
+class MTEQA(Model):
+    def __init__(self, image: str = DEFAULT_IMAGE, device: int = 0):
+        self.image = image
+        self.device = device
+
+    def predict(self, candidate: str, references: List[str], **kwargs) -> MetricsType:
+        return self.predict_batch(
+            [{"candidate": candidate, "references": references}], **kwargs
+        )[0]
+
+    def predict_batch(
+        self,
+        inputs: List[Dict[str, Union[str, List[str]]]],
+        gen_from_out: bool = False,
+        **kwargs,
+    ) -> Tuple[MetricsType, List[MetricsType]]:
+        """
+
+        Parameters
+        ----------
+        inputs
+        gen_from_out
+        kwargs
+
+        Returns
+        -------
+
+        """
+        logger.info(f"Calculating MTEQA for {len(inputs)} inputs with model")
+
+        candidates = [inp["candidate"] for inp in inputs]
+        references_list = [inp["references"] for inp in inputs]
+
+        with DockerContainer(self.image) as backend:
+            host_candidate_file = f"{backend.host_dir}/candidates.txt"
+            container_candidate_file = f"{backend.container_dir}/candidates.txt"
+
+            host_reference_file = f"{backend.host_dir}/references.txt"
+            container_reference_file = f"{backend.container_dir}/references.txt"
+
+            indices = []
+            with open(host_candidate_file, "w") as out_cand:
+                with open(host_reference_file, "w") as out_ref:
+                    for i, (candidate, references) in enumerate(
+                        zip(candidates, references_list)
+                    ):
+                        for reference in references:
+                            out_cand.write(candidate + "\n")
+                            out_ref.write(reference + "\n")
+                            indices.append(i)
+
+            host_output_file = f"{backend.host_dir}/output.tsv"
+            container_output_file = f"{backend.container_dir}/output.tsv"
+
+            commands = []
+            cuda = self.device != -1
+            if cuda:
+                commands.append(f"export CUDA_VISIBLE_DEVICES={self.device}")
+
+            # Only English is supported
+            commands.append("cd MTEQA")
+            score_command = (
+                f"python mteqa_score.py"
+                f"  --reference {container_reference_file}"
+                f"  --hypothesis {container_candidate_file}"
+                f"  --lang en"
+                f"  --verbose"
+            )
+            if not cuda:
+                score_command += " --cpu"
+            if gen_from_out:
+                score_command += " --gen_from_out"
+            score_command += f" > {container_output_file}"
+            commands.append(score_command)
+
+            command = " && ".join(commands)
+            backend.run_command(
+                command=command,
+                cuda=cuda,
+                network_disabled=True,
+            )
+
+            # Load the data in the output tsv
+            with open(host_output_file, "r") as f:
+                lines = f.read().splitlines()
+
+            # Format validation
+            header = lines[0].strip().split("\t")
+            if (
+                header[2] != "F1"
+                or header[3] != "EM"
+                or header[4] != "chrf"
+                or header[5] != "bleu"
+            ):
+                raise Exception(f"Unexpected header format: {header}")
+            if len(lines) - 1 != len(indices):
+                raise Exception(
+                    f"Incorrect number of output scores. Expected {len(indices)}, found {len(lines) - 1}"
+                )
+
+            # Group by input candidate index
+            index_to_metrics = defaultdict(list)
+            for index, line in zip(indices, lines[1:]):
+                cols = line.strip().split("\t")
+                index_to_metrics[index].append(
+                    {
+                        "mteqa": {
+                            "f1": float(cols[2]),
+                            "em": float(cols[3]),
+                            "chrf": float(cols[4]),
+                            "bleu": float(cols[5]),
+                        }
+                    }
+                )
+
+            # Average and return
+            micro = []
+            for i in range(len(candidates)):
+                micro.append(util.average_dicts(index_to_metrics[i]))
+            macro = util.average_dicts(micro)
+            return macro, micro
diff --git a/repro/models/krubinski2021/setup.py b/repro/models/krubinski2021/setup.py
new file mode 100644
index 0000000..4ccd179
--- /dev/null
+++ b/repro/models/krubinski2021/setup.py
@@ -0,0 +1,10 @@
+from repro import MODELS_ROOT
+from repro.commands.subcommand import SetupSubcommand
+from repro.common.docker import BuildDockerImageSubcommand
+from repro.models.krubinski2021 import DEFAULT_IMAGE, MODEL_NAME
+
+
+@SetupSubcommand.register(MODEL_NAME)
+class Krubinski2021SetupSubcommand(BuildDockerImageSubcommand):
+    def __init__(self) -> None:
+        super().__init__(f"{MODELS_ROOT}/{MODEL_NAME}", DEFAULT_IMAGE)


Test patch:
diff --git a/models/krubinski2021/tests/__init__.py b/models/krubinski2021/tests/__init__.py
new file mode 100644
index 0000000..e69de29
diff --git a/models/krubinski2021/tests/model_test.py b/models/krubinski2021/tests/model_test.py
new file mode 100644
index 0000000..0f57565
--- /dev/null
+++ b/models/krubinski2021/tests/model_test.py
@@ -0,0 +1,87 @@
+import unittest
+from parameterized import parameterized
+
+from repro.models.krubinski2021 import MTEQA
+from repro.testing import assert_dicts_approx_equal, get_testing_device_parameters
+
+
+class TestKrubinski2021Models(unittest.TestCase):
+    @parameterized.expand(get_testing_device_parameters())
+    def test_mteqa_regression_short(self, device: int):
+        model = MTEQA(device=device)
+        candidate = "It is said that when Richard got sick, Salahuddin sent him some aloof, which was kept in the snow."
+        reference = "It is said, when Richard got sick, Salahudin sent him few Plum fruit which were kept in the snow."
+        expected = {
+            "mteqa": {
+                "f1": 0.21333333333333332,
+                "em": 0.0,
+                "chrf": 48.04483459121631,
+                "bleu": 13.194715521231362,
+            }
+        }
+        actual = model.predict(candidate, [reference])
+        assert_dicts_approx_equal(actual, expected)
+
+        # Include gen_from_out=True
+        expected = {
+            "mteqa": {
+                "f1": 0.24,
+                "em": 0.0,
+                "chrf": 52.15993009975308,
+                "bleu": 14.844054961385282,
+            }
+        }
+        actual = model.predict(candidate, [reference], gen_from_out=True)
+        assert_dicts_approx_equal(actual, expected)
+
+    @parameterized.expand(get_testing_device_parameters())
+    def test_mteqa_regression(self, device: int):
+        model = MTEQA(device=device)
+        inputs = [
+            {
+                "candidate": "The dog bit the man.",
+                "references": ["The dog bit the man.", "The dog had bit the man."],
+            },
+            {
+                "candidate": "It wasn't surprising.",
+                "references": ["It was not unexpected.", "No one was surprised."],
+            },
+            {
+                "candidate": "The man had just bitten him.",
+                "references": ["The man bit him first.", "The man had bitten the dog."],
+            },
+        ]
+        expected_macro = {
+            "mteqa": {
+                "f1": 0.38888888888888884,
+                "em": 0.3333333333333333,
+                "chrf": 41.14592658372042,
+                "bleu": 36.39899534309537,
+            }
+        }
+        expected_micro = [
+            {
+                "mteqa": {
+                    "f1": 0.9166666666666666,
+                    "em": 0.75,
+                    "chrf": 85.93155893536121,
+                    "bleu": 84.1969860292861,
+                }
+            },
+            {"mteqa": {"f1": 0.0, "em": 0.0, "chrf": 2.7027027027027026, "bleu": 0.0}},
+            {
+                "mteqa": {
+                    "f1": 0.25,
+                    "em": 0.25,
+                    "chrf": 34.80351811309734,
+                    "bleu": 25.00000000000001,
+                }
+            },
+        ]
+
+        actual_macro, actual_micro = model.predict_batch(inputs)
+
+        assert_dicts_approx_equal(actual_macro, expected_macro)
+        assert len(actual_micro) == len(expected_micro)
+        for actual, expected in zip(actual_micro, expected_micro):
+            assert_dicts_approx_equal(actual, expected)