Repository: danieldeutsch/repro (language: python)

Issue:
Add BaryScore, DepthScore, and InfoLM
BaryScore: https://arxiv.org/abs/2108.12463
DepthScore: https://arxiv.org/abs/2103.12711
InfoLM: https://arxiv.org/abs/2112.01589
Code repository: https://github.com/PierreColombo/nlg_eval_via_simi_measures/


Discussion / hints:

Fix patch:
diff --git a/Changelog.md b/Changelog.md
index 7655cf0..ce01e91 100644
--- a/Changelog.md
+++ b/Changelog.md
@@ -5,6 +5,9 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
 ## Unreleased
+## Added
+- Added [BaryScore, InfoLM, and DepthScore](models/colombo2021/Readme.md)
+
 ### Changed
 - Changed the backend implementation of MoverScore to use a non-IDF dict based version.
 
diff --git a/Papers.md b/Papers.md
index ac6d85a..10c7da2 100644
--- a/Papers.md
+++ b/Papers.md
@@ -31,6 +31,9 @@ This file contains a list of the papers with models wrapped by this library.
 |[Fill in the BLANC: Human-free quality estimation of document summaries](https://aclanthology.org/2020.eval4nlp-1.2/)|Oleg Vasilyev, Vedant Dharnidharka, and John Bohannon|[Link](models/vasilyev2020/Readme.md)|
 |[Meteor Universal: Language Specific Translation Evaluation for Any Target Language](https://aclanthology.org/W14-3348/)|Michael Denkowski and Alon Lavie|[Link](models/denkowski2014/Readme.md)|
 |[COMET: A Neural Framework for MT Evaluation](https://aclanthology.org/2020.emnlp-main.213/)|Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie|[Link](models/rei2020/Readme.md)|
+|[Automatic Text Evaluation through the Lens of Wasserstein Barycenters](https://arxiv.org/abs/2108.12463)|Pierre Colombo, Guillaume Staerman, Chloe Clavel, and Pablo Piantanida|[Link](models/colombo2021/Readme.md)|
+|[InfoLM: A New Metric to Evaluate Summarization & Data2Text Generation](https://arxiv.org/abs/2112.01589)|Pierre Colombo, Chloe Clavel, and Pablo Piantanida|[Link](models/colombo2021/Readme.md)|
+|[A Pseudo-Metric between Probability Distributions based on Depth-Trimmed Regions](https://arxiv.org/abs/2103.12711)|Guillaume Staerman, Pavlo Mozharovskyi, Pierre Colombo, Stéphan Clémençon, and Florence d'Alché-Buc|[Link](models/colombo2021/Readme.md)|
 
 ### Question Answering
 |Paper|Authors|Link|
diff --git a/models/colombo2021/Dockerfile b/models/colombo2021/Dockerfile
new file mode 100644
index 0000000..481a6bb
--- /dev/null
+++ b/models/colombo2021/Dockerfile
@@ -0,0 +1,26 @@
+FROM danieldeutsch/python:3.7-cuda11.0.3-base
+
+WORKDIR /app
+
+# Get the code
+RUN git clone https://github.com/PierreColombo/nlg_eval_via_simi_measures && \
+    cd nlg_eval_via_simi_measures && \
+    git checkout 44d33f34833193a34407cbe6086418c1374ebce6
+
+# Install dependencies
+RUN pip install --no-cache-dir \
+    numpy==1.21.5 \
+    tqdm==4.62.3 \
+    POT==0.8.1.0 \
+    torch==1.10.2+cu113 \
+    transformers==4.16.1 \
+    sklearn \
+    geomloss==0.2.4 \
+    -f https://download.pytorch.org/whl/cu113/torch_stable.html
+
+# Copy over a new CLI file which has an option for an output file
+COPY src/score_cli.py nlg_eval_via_simi_measures/score_cli.py
+
+# Run warmup queries
+COPY scripts/warmup.sh warmup.sh
+RUN sh warmup.sh
diff --git a/models/colombo2021/Readme.md b/models/colombo2021/Readme.md
new file mode 100644
index 0000000..ee7718f
--- /dev/null
+++ b/models/colombo2021/Readme.md
@@ -0,0 +1,81 @@
+# Colombo et al., (2021, 2022); Staerman et al., (2022)
+
+## Publication
+This Dockerfile corresponds to three different papers:
+- [Automatic Text Evaluation through the Lens of Wasserstein Barycenters](https://arxiv.org/abs/2108.12463)
+- [InfoLM: A New Metric to Evaluate Summarization & Data2Text Generation](https://arxiv.org/abs/2112.01589)
+- [A Pseudo-Metric between Probability Distributions based on Depth-Trimmed Regions](https://arxiv.org/abs/2103.12711)
+
+## Repositories
+All three metrics are implemented in https://github.com/PierreColombo/nlg_eval_via_simi_measures
+
+## Available Models
+- BaryScore
+  - Name: `colombo2021-baryscore`
+  - Usage:
+    ```python
+    from repro.models.colombo2021 import BaryScore
+    model = BaryScore()
+    inputs = [
+        {"candidate": "The candidate", "references": ["The first reference", "The second"]}
+    ]
+    macro, micro = model.predict_batch(inputs)
+    ```
+    `micro` contains the per-input scores and `macro` contains the averaged scores.
+    
+- InfoLM
+  - Name: `colombo2021-infolm`
+  - Usage:
+    ```python
+    from repro.models.colombo2021 import InfoLM
+    model = InfoLM()
+    inputs = [
+        {"candidate": "The candidate", "references": ["The first reference", "The second"]}
+    ]
+    macro, micro = model.predict_batch(inputs)
+    ```
+    `micro` contains the per-input scores and `macro` contains the averaged scores.
+
+- DepthScore
+  - Name: `colombo2021-depthscore`
+  - Usage:
+    ```python
+    from repro.models.colombo2021 import DepthScore
+    model = DepthScore()
+    inputs = [
+        {"candidate": "The candidate", "references": ["The first reference", "The second"]}
+    ]
+    macro, micro = model.predict_batch(inputs)
+    ```
+    `micro` contains the per-input scores and `macro` contains the averaged scores.
+    
+## Implementation Notes
+- For some reason, the unit tests pass on some machines and not on others.
+On one of our dev machines, the CPU and GPU tests pass.
+On another, the CPU pass but the GPU do not.
+On GitHub Actions, the CPU tests do not pass.
+Since they are being run in Docker, I assume there is some difference in hardware causing this, but I do not know what the issue is.
+    
+## Docker Information
+- Image name: `danieldeutsch/colombo2021:1.0`
+- Build command: Provide documentation on how to build the image
+  ```shell script
+  repro setup colombo2021
+  ```
+- Requires network: Yes, it sends a request for resources
+  
+## Testing
+```shell script
+repro setup colombo2021
+pytest models/colombo2021/tests
+```
+
+## Status
+- [ ] Regression unit tests pass  
+See the implementation notes; https://github.com/danieldeutsch/repro/runs/5210482796
+- [ ] Correctness unit tests pass  
+- [ ] Model runs on full test dataset  
+- [ ] Predictions approximately replicate results reported in the paper  
+- [ ] Predictions exactly replicate results reported in the paper  
+
+## Changelog
diff --git a/models/colombo2021/scripts/warmup.sh b/models/colombo2021/scripts/warmup.sh
new file mode 100644
index 0000000..106b4a6
--- /dev/null
+++ b/models/colombo2021/scripts/warmup.sh
@@ -0,0 +1,14 @@
+cd nlg_eval_via_simi_measures
+
+echo "I like my cakes very much" > ref.txt
+echo "I like my cakes very much" > hyp.txt
+
+for metric in infolm baryscore depthscore; do
+  python score_cli.py \
+    --ref ref.txt \
+    --cand hyp.txt \
+    --metric_name=${metric} \
+    --output_file out.txt
+done
+
+rm ref.txt hyp.txt out.txt
\ No newline at end of file
diff --git a/models/colombo2021/src/score_cli.py b/models/colombo2021/src/score_cli.py
new file mode 100644
index 0000000..8be6fc6
--- /dev/null
+++ b/models/colombo2021/src/score_cli.py
@@ -0,0 +1,122 @@
+# fmt: off
+#!/usr/bin/env python
+import os
+import argparse
+import torch
+import json
+
+from bary_score import BaryScoreMetric
+from depth_score import DepthScoreMetric
+from infolm import InfoLM
+from tqdm import tqdm
+import logging
+
+logging.basicConfig(
+    format="%(asctime)s - %(levelname)s - %(name)s -   %(message)s", datefmt="%m/%d/%Y %H:%M:%S",
+    level=logging.INFO)
+
+
+def main():
+    torch.multiprocessing.set_sharing_strategy("file_system")
+    ####################
+    # Common Arguments #
+    ####################
+    parser = argparse.ArgumentParser("Calculate Metrics Based on Statistical Measures of Similarity")
+    parser.add_argument("--metric_name", type=str, default="baryscore",
+                        choices=['depthscore', 'baryscore', 'infolm'], help=" which metric to compute")
+    parser.add_argument(
+        "-m", "--model", default='bert-base-uncased',
+        help="BERT model name (default: bert-base-uncased) or path to a pretrain model",
+    )
+    parser.add_argument("-b", "--batch_size", type=int, default=3, help="batch size (default: 64)")
+    parser.add_argument("-r", "--ref", type=str, required=True, help="reference file path or a string")
+    parser.add_argument(
+        "-c", "--cand", type=str, required=True, help="candidate (system outputs) file path or a string",
+    )
+    parser.add_argument("--idf", action="store_true", help="if true use idf")
+    parser.add_argument("--output_file", required=True)
+    ###################
+    # Argument InfoLM #
+    ###################
+    parser.add_argument("--measure_to_use", type=str, default="fisher_rao",
+                        choices=['kl', 'alpha', 'renyi', 'beta', 'ab', 'l1', "l2", "linf", 'fisher_rao'],
+                        help=" which measure of information to use")
+    parser.add_argument("--temperature", type=float, default=0.5, help=" temperature to calibrate the LM")
+    parser.add_argument("--beta", type=float, default=0.5, help="beta parameter in the ab or beta div")
+    parser.add_argument("--alpha", type=float, default=0.5, help="alpha parameter in the ab, alpha or renyi div")
+
+    ######################
+    # Argument BaryScore #
+    ######################
+    parser.add_argument("--last_layers", type=int, default=5, help="use the last  N layer in BERT")
+    parser.add_argument("--sinkhorn_ref", type=float, default=0.01, help="weight of the KL in the SD")
+
+    #######################
+    # Argument DepthScore #
+    #######################
+    parser.add_argument("--layers_to_consider", type=int, default=9, help="use the N-th layer in BERT")
+    parser.add_argument("--considered_measure", type=str, default='ai_irw',
+                        choices=["irw", "ai_irw", "wasserstein", "sliced", "mmd"], help="")
+    parser.add_argument("--p", type=float, default=5, help="the power of the ground cost")
+    parser.add_argument("--eps", type=float, default=0.3, help="the highest level set")
+    parser.add_argument("--n_alpha", type=float, default=5,
+                        help="the Monte-Carlo parameter for the approximation of the integral  over alpha.")
+
+    args = parser.parse_args()
+    logging.info("Computing Score for {}".format(args.metric_name))
+    if os.path.isfile(args.cand):
+        with open(args.cand) as f:
+            cands = [line.strip() for line in f]
+
+        if os.path.exists(args.ref):
+            with open(args.ref) as f:
+                refs = [line.strip() for line in f]
+        logging.info("Files opened Starting To Process")
+    else:
+        logging.info("Single Sentence Score")
+        cands = [args.cand]
+        refs = [args.ref]
+        assert not args.idf, "do not support idf mode for a single pair of sentences"
+    # do for loops :)
+    if args.metric_name == 'infolm':
+        metric = InfoLM(model_name=args.model, temperature=args.temperature, measure_to_use=args.measure_to_use,
+                        use_idf_weights=args.idf, alpha=args.alpha, beta=args.beta)
+    elif args.metric_name == 'depthscore':
+        metric = DepthScoreMetric(model_name=args.model, layers_to_consider=args.layers_to_consider,
+                                  considered_measure=args.considered_measure, p=args.p,
+                                  eps=args.eps, n_alpha=args.n_alpha)
+    elif args.metric_name == 'baryscore':
+        metric = BaryScoreMetric(model_name=args.model, last_layers=args.last_layers, use_idfs=args.idf,
+                                 sinkhorn_ref=args.sinkhorn_ref)
+    else:
+        raise NotImplementedError
+    all_preds = []
+    metric.prepare_idfs(cands, refs)
+    batched_candidates = [cands[i:i + args.batch_size] for i in range(0, len(cands), args.batch_size)]
+    batched_references = [refs[i:i + args.batch_size] for i in range(0, len(refs), args.batch_size)]
+    for golden_batch, candidate_batch in tqdm(zip(batched_references, batched_candidates), 'Evaluation in Progress'):
+        preds = metric.evaluate_batch(candidate_batch, golden_batch)
+        all_preds.append(preds)
+
+    # Rearrange for per-input dicts
+    per_input_preds = []
+    metric_name = list(preds.keys())[0]
+    for batch in all_preds:
+        num_instances = len(batch[metric_name])
+        for i in range(num_instances):
+            per_input_preds.append(
+                {args.metric_name: {k: batch[k][i] for k in preds.keys()}}
+            )
+
+    with open(args.output_file, "w") as out:
+        out.write(json.dumps(per_input_preds))
+
+    # for k in preds.keys():
+    #     l_preds = []
+    #     for pred in all_preds:
+    #         l_preds.append(pred[k])
+    #     logging.info('Metric {} : {}'.format(k, sum(sum(l_preds, [])) / len(sum(l_preds, []))))
+
+
+if __name__ == "__main__":
+    main()
diff --git a/repro/models/colombo2021/__init__.py b/repro/models/colombo2021/__init__.py
new file mode 100644
index 0000000..60369fc
--- /dev/null
+++ b/repro/models/colombo2021/__init__.py
@@ -0,0 +1,10 @@
+import os
+
+VERSION = "1.0"
+MODEL_NAME = os.path.basename(os.path.dirname(__file__))
+DOCKERHUB_REPO = f"danieldeutsch/{MODEL_NAME}"
+DEFAULT_IMAGE = f"{DOCKERHUB_REPO}:{VERSION}"
+AUTOMATICALLY_PUBLISH = True
+
+from repro.models.colombo2021.models import DepthScore, BaryScore, InfoLM
+from repro.models.colombo2021.setup import Colombo2021SetupSubcommand
diff --git a/repro/models/colombo2021/models.py b/repro/models/colombo2021/models.py
new file mode 100644
index 0000000..2d0dcbe
--- /dev/null
+++ b/repro/models/colombo2021/models.py
@@ -0,0 +1,206 @@
+import json
+import logging
+from typing import Any, Dict, List, Tuple
+
+from overrides import overrides
+
+from repro.common import util
+from repro.common.docker import DockerContainer
+from repro.data.types import MetricsType, TextType
+from repro.models import Model
+from repro.models.colombo2021 import DEFAULT_IMAGE, MODEL_NAME
+
+logger = logging.getLogger(__name__)
+
+
+class _Colombo2021Model(Model):
+    def __init__(self, name: str, image: str, device: int, model: str, batch_size: int):
+        self.name = name
+        self.image = image
+        self.device = device
+        self.model = model
+        self.batch_size = batch_size
+
+    def _get_predict_options(self) -> str:
+        """Returns the command line options for the metric"""
+        raise NotImplementedError
+
+    def predict(self, candidate: TextType, references: List[TextType]) -> MetricsType:
+        return self.predict_batch([{"candidate": candidate, "references": references}])[
+            0
+        ]
+
+    @overrides
+    def predict_batch(
+        self, inputs: List[Dict[str, Any]]
+    ) -> Tuple[MetricsType, List[MetricsType]]:
+        logger.info(f"Predicting scores for {len(inputs)} inputs")
+
+        candidates = [inp["candidate"] for inp in inputs]
+        references_list = [inp["references"] for inp in inputs]
+
+        with DockerContainer(self.image) as backend:
+            host_ref_file = f"{backend.host_dir}/ref.txt"
+            container_ref_file = f"{backend.container_dir}/ref.txt"
+
+            host_cand_file = f"{backend.host_dir}/cand.txt"
+            container_cand_file = f"{backend.container_dir}/cand.txt"
+
+            with open(host_ref_file, "w") as out_ref:
+                with open(host_cand_file, "w") as out_cand:
+                    for candidate, references in zip(candidates, references_list):
+                        # Write the candidate once per reference
+                        for reference in references:
+                            out_ref.write(reference + "\n")
+                            out_cand.write(candidate + "\n")
+
+            host_out_file = f"{backend.host_dir}/out.json"
+            container_out_file = f"{backend.container_dir}/out.json"
+
+            commands = []
+            cuda = self.device != -1
+            if cuda:
+                commands.append(f"export CUDA_VISIBLE_DEVICES={self.device}")
+
+            commands.append("cd nlg_eval_via_simi_measures")
+
+            predict_command = (
+                f"python score_cli.py"
+                f"  --ref {container_ref_file}"
+                f"  --cand {container_cand_file}"
+                f"  --metric_name {self.name}"
+                f"  --output_file {container_out_file}"
+            )
+            if self.model is not None:
+                predict_command += f" --model {self.model}"
+            if self.batch_size is not None:
+                predict_command += f" --batch_size {self.batch_size}"
+
+            predict_command += " " + self._get_predict_options()
+            commands.append(predict_command)
+
+            command = " && ".join(commands)
+            backend.run_command(
+                command=command,
+                cuda=cuda,
+                network_disabled=False,
+            )
+
+            with open(host_out_file, "r") as f:
+                unrolled_micro_metrics = json.load(f)
+
+            # Average over references
+            micro_metrics = []
+            index = 0
+            for references in references_list:
+                metrics = []
+                for _ in references:
+                    metrics.append(unrolled_micro_metrics[index])
+                    index += 1
+                micro_metrics.append(util.average_dicts(metrics))
+
+            macro_metrics = util.average_dicts(micro_metrics)
+            return macro_metrics, micro_metrics
+
+
+@Model.register(f"{MODEL_NAME}-infolm")
+class InfoLM(_Colombo2021Model):
+    def __init__(
+        self,
+        image: str = DEFAULT_IMAGE,
+        device: int = 0,
+        model: str = None,
+        batch_size: int = 8,
+        idf: bool = False,
+        measure: str = None,
+        temperature: float = None,
+        beta: float = None,
+        alpha: float = None,
+    ) -> None:
+        super().__init__("infolm", image, device, model, batch_size)
+        self.idf = idf
+        self.measure = measure
+        self.temperature = temperature
+        self.beta = beta
+        self.alpha = alpha
+
+    @overrides
+    def _get_predict_options(self) -> str:
+        options = []
+        if self.idf:
+            options.append("--idf")
+        if self.measure is not None:
+            options.append(f"--measure_to_use {self.measure}")
+        if self.temperature is not None:
+            options.append(f"--temperature {self.temperature}")
+        if self.beta is not None:
+            options.append(f"--beta {self.beta}")
+        if self.alpha is not None:
+            options.append(f"--alpha {self.alpha}")
+        return " ".join(options)
+
+
+@Model.register(f"{MODEL_NAME}-baryscore")
+class BaryScore(_Colombo2021Model):
+    def __init__(
+        self,
+        image: str = DEFAULT_IMAGE,
+        device: int = 0,
+        model: str = None,
+        batch_size: int = 8,
+        last_layers: int = None,
+        sinkhorn_ref: float = None,
+        idf: bool = False,
+    ) -> None:
+        super().__init__("baryscore", image, device, model, batch_size)
+        self.last_layers = last_layers
+        self.sinkhorn_ref = sinkhorn_ref
+        self.idf = idf
+
+    @overrides
+    def _get_predict_options(self) -> str:
+        options = []
+        if self.last_layers is not None:
+            options.append(f"--last_layers {self.last_layers}")
+        if self.sinkhorn_ref is not None:
+            options.append(f"--sinkhorn_ref {self.sinkhorn_ref}")
+        if self.idf:
+            options.append("--idf")
+        return " ".join(options)
+
+
+@Model.register(f"{MODEL_NAME}-depthscore")
+class DepthScore(_Colombo2021Model):
+    def __init__(
+        self,
+        image: str = DEFAULT_IMAGE,
+        device: int = 0,
+        model: str = None,
+        batch_size: int = 8,
+        layers_to_consider: int = None,
+        considered_measure: str = None,
+        p: float = None,
+        eps: float = None,
+        n_alpha: float = None,
+    ) -> None:
+        super().__init__("depthscore", image, device, model, batch_size)
+        self.layers_to_consider = layers_to_consider
+        self.considered_measure = considered_measure
+        self.p = p
+        self.eps = eps
+        self.n_alpha = n_alpha
+
+    @overrides
+    def _get_predict_options(self) -> str:
+        options = []
+        if self.layers_to_consider is not None:
+            options.append(f"--layers_to_consider {self.layers_to_consider}")
+        if self.considered_measure is not None:
+            options.append(f"--considered_measure {self.considered_measure}")
+        if self.p is not None:
+            options.append(f"--p {self.p}")
+        if self.eps is not None:
+            options.append(f"--eps {self.eps}")
+        if self.n_alpha is not None:
+            options.append(f"--n_alpha {self.n_alpha}")
+        return " ".join(options)
diff --git a/repro/models/colombo2021/setup.py b/repro/models/colombo2021/setup.py
new file mode 100644
index 0000000..725fa8d
--- /dev/null
+++ b/repro/models/colombo2021/setup.py
@@ -0,0 +1,10 @@
+from repro import MODELS_ROOT
+from repro.commands.subcommand import SetupSubcommand
+from repro.common.docker import BuildDockerImageSubcommand
+from repro.models.colombo2021 import DEFAULT_IMAGE, MODEL_NAME
+
+
+@SetupSubcommand.register(MODEL_NAME)
+class Colombo2021SetupSubcommand(BuildDockerImageSubcommand):
+    def __init__(self) -> None:
+        super().__init__(f"{MODELS_ROOT}/{MODEL_NAME}", DEFAULT_IMAGE)


Test patch:
diff --git a/models/colombo2021/tests/__init__.py b/models/colombo2021/tests/__init__.py
new file mode 100644
index 0000000..f10f952
--- /dev/null
+++ b/models/colombo2021/tests/__init__.py
@@ -0,0 +1,3 @@
+import os
+
+FIXTURES_ROOT = f"{os.path.dirname(os.path.abspath(__file__))}/fixtures"
diff --git a/models/colombo2021/tests/fixtures/expected.json b/models/colombo2021/tests/fixtures/expected.json
new file mode 100644
index 0000000..f5a7f9d
--- /dev/null
+++ b/models/colombo2021/tests/fixtures/expected.json
@@ -0,0 +1,477 @@
+{
+  "infolm": {
+    "micro": [
+      {
+        "infolm": {
+          "fisher_rao": 0.474699467420578,
+          "r_fisher_rao": 0.474699467420578,
+          "sim_fisher_rao": 0.474699467420578
+        }
+      },
+      {
+        "infolm": {
+          "fisher_rao": 1.2616801261901855,
+          "r_fisher_rao": 1.2616801261901855,
+          "sim_fisher_rao": 1.2616801261901855
+        }
+      },
+      {
+        "infolm": {
+          "fisher_rao": 1.63699471950531,
+          "r_fisher_rao": 1.63699471950531,
+          "sim_fisher_rao": 1.63699471950531
+        }
+      },
+      {
+        "infolm": {
+          "fisher_rao": 1.277878761291504,
+          "r_fisher_rao": 1.277878761291504,
+          "sim_fisher_rao": 1.277878761291504
+        }
+      },
+      {
+        "infolm": {
+          "fisher_rao": 1.4103935956954956,
+          "r_fisher_rao": 1.4103935956954956,
+          "sim_fisher_rao": 1.4103935956954956
+        }
+      },
+      {
+        "infolm": {
+          "fisher_rao": 0.9920828938484192,
+          "r_fisher_rao": 0.9920828938484192,
+          "sim_fisher_rao": 0.9920828938484192
+        }
+      },
+      {
+        "infolm": {
+          "fisher_rao": 1.0333880186080933,
+          "r_fisher_rao": 1.0333880186080933,
+          "sim_fisher_rao": 1.0333880186080933
+        }
+      },
+      {
+        "infolm": {
+          "fisher_rao": 1.671626329421997,
+          "r_fisher_rao": 1.671626329421997,
+          "sim_fisher_rao": 1.671626329421997
+        }
+      },
+      {
+        "infolm": {
+          "fisher_rao": 1.1372259855270386,
+          "r_fisher_rao": 1.1372259855270386,
+          "sim_fisher_rao": 1.1372259855270386
+        }
+      },
+      {
+        "infolm": {
+          "fisher_rao": 1.757234811782837,
+          "r_fisher_rao": 1.757234811782837,
+          "sim_fisher_rao": 1.757234811782837
+        }
+      }
+    ],
+    "macro": {
+      "infolm": {
+        "fisher_rao": 1.265320470929146,
+        "r_fisher_rao": 1.265320470929146,
+        "sim_fisher_rao": 1.265320470929146
+      }
+    }
+  },
+  "baryscore": {
+    "cpu": {
+      "micro": [
+        {
+          "baryscore": {
+            "baryscore_W": 0.07766812629000892,
+            "baryscore_SD_10": 0.9435403681657479,
+            "baryscore_SD_1": 0.8237248949876972,
+            "baryscore_SD_5": 0.9332710059064289,
+            "baryscore_SD_0.1": 0.09276958362410065,
+            "baryscore_SD_0.5": 0.6270019753138825,
+            "baryscore_SD_0.01": 0.07766849194363867,
+            "baryscore_SD_0.001": 0.07766812629000894
+          }
+        },
+        {
+          "baryscore": {
+            "baryscore_W": 0.2016408689766467,
+            "baryscore_SD_10": 1.0591088887064064,
+            "baryscore_SD_1": 0.9799351483290417,
+            "baryscore_SD_5": 1.0524333510333723,
+            "baryscore_SD_0.1": 0.22141259239333858,
+            "baryscore_SD_0.5": 0.8336105840234236,
+            "baryscore_SD_0.01": 0.20170697594422057,
+            "baryscore_SD_0.001": 1.1883514124962701e-08
+          }
+        },
+        {
+          "baryscore": {
+            "baryscore_W": 0.4381551871071647,
+            "baryscore_SD_10": 1.1192929858806018,
+            "baryscore_SD_1": 1.0693602276847844,
+            "baryscore_SD_5": 1.1149198520955448,
+            "baryscore_SD_0.1": 0.4743048008538526,
+            "baryscore_SD_0.5": 0.9800173130079961,
+            "baryscore_SD_0.01": 0.43832700367555966,
+            "baryscore_SD_0.001": 2.003924421958562e-12
+          }
+        },
+        {
+          "baryscore": {
+            "baryscore_W": 0.2813255022151478,
+            "baryscore_SD_10": 1.0921581506454847,
+            "baryscore_SD_1": 1.0245605381619027,
+            "baryscore_SD_5": 1.0864293581822326,
+            "baryscore_SD_0.1": 0.3166094331337496,
+            "baryscore_SD_0.5": 0.8984449871719464,
+            "baryscore_SD_0.01": 0.2816519305649153,
+            "baryscore_SD_0.001": 2.3780101876663813e-19
+          }
+        },
+        {
+          "baryscore": {
+            "baryscore_W": 0.25661290069938414,
+            "baryscore_SD_10": 1.0060048355610243,
+            "baryscore_SD_1": 0.9225334388167529,
+            "baryscore_SD_5": 0.9988347132027401,
+            "baryscore_SD_0.1": 0.27878960598645464,
+            "baryscore_SD_0.5": 0.7820144826807429,
+            "baryscore_SD_0.01": 0.25674670283613166,
+            "baryscore_SD_0.001": 3.87883155091673e-08
+          }
+        },
+        {
+          "baryscore": {
+            "baryscore_W": 0.19959869602343916,
+            "baryscore_SD_10": 1.1525759996527434,
+            "baryscore_SD_1": 1.0868140244157845,
+            "baryscore_SD_5": 1.147125809727812,
+            "baryscore_SD_0.1": 0.22824258350671353,
+            "baryscore_SD_0.5": 0.9515331204768216,
+            "baryscore_SD_0.01": 0.19977983870139987,
+            "baryscore_SD_0.001": 1.7292058031932015e-07
+          }
+        },
+        {
+          "baryscore": {
+            "baryscore_W": 0.26009874077101536,
+            "baryscore_SD_10": 1.0367049951001042,
+            "baryscore_SD_1": 0.9588764818298308,
+            "baryscore_SD_5": 1.0301041528806354,
+            "baryscore_SD_0.1": 0.2766781812518756,
+            "baryscore_SD_0.5": 0.817987110152457,
+            "baryscore_SD_0.01": 0.2602404284508689,
+            "baryscore_SD_0.001": 1.4347898612872798e-07
+          }
+        },
+        {
+          "baryscore": {
+            "baryscore_W": 0.2550332619106726,
+            "baryscore_SD_10": 1.065608880382261,
+            "baryscore_SD_1": 1.0006220943602468,
+            "baryscore_SD_5": 1.0599375339512176,
+            "baryscore_SD_0.1": 0.28563784324455255,
+            "baryscore_SD_0.5": 0.8836851217094949,
+            "baryscore_SD_0.01": 0.255260254523738,
+            "baryscore_SD_0.001": 5.846539613350077e-08
+          }
+        },
+        {
+          "baryscore": {
+            "baryscore_W": 0.20474150785150907,
+            "baryscore_SD_10": 0.9650287608276645,
+            "baryscore_SD_1": 0.8984869955528119,
+            "baryscore_SD_5": 0.9592218051251002,
+            "baryscore_SD_0.1": 0.22987039940927342,
+            "baryscore_SD_0.5": 0.7836020365106011,
+            "baryscore_SD_0.01": 0.20480956050818433,
+            "baryscore_SD_0.001": 2.827539284869416e-10
+          }
+        },
+        {
+          "baryscore": {
+            "baryscore_W": 0.26533017086138944,
+            "baryscore_SD_10": 1.0612055514689438,
+            "baryscore_SD_1": 0.9929246078512761,
+            "baryscore_SD_5": 1.0555271743239576,
+            "baryscore_SD_0.1": 0.2960996122238252,
+            "baryscore_SD_0.5": 0.8630305814055836,
+            "baryscore_SD_0.01": 0.2656434728981158,
+            "baryscore_SD_0.001": 1.1167973430084266e-07
+          }
+        }
+      ],
+      "macro": {
+        "baryscore": {
+          "baryscore_W": 0.24402049627063777,
+          "baryscore_SD_10": 1.0501229416390983,
+          "baryscore_SD_1": 0.9757838451990128,
+          "baryscore_SD_5": 1.043780475642904,
+          "baryscore_SD_0.1": 0.2700414635627736,
+          "baryscore_SD_0.5": 0.8420927312452949,
+          "baryscore_SD_0.01": 0.24418346600467725,
+          "baryscore_SD_0.001": 0.007766866379129331
+        }
+      }
+    },
+    "gpu": {
+      "micro": [
+        {
+          "baryscore": {
+            "baryscore_W": 0.06075610698482854,
+            "baryscore_SD_10": 0.9566665736645789,
+            "baryscore_SD_1": 0.8327053257686587,
+            "baryscore_SD_5": 0.946014206384727,
+            "baryscore_SD_0.1": 0.07140092714229203,
+            "baryscore_SD_0.5": 0.6274778099202816,
+            "baryscore_SD_0.01": 0.06075610791970633,
+            "baryscore_SD_0.001": 0.060756106984828556
+          }
+        },
+        {
+          "baryscore": {
+            "baryscore_W": 0.21522683242677945,
+            "baryscore_SD_10": 1.0465138976455104,
+            "baryscore_SD_1": 0.9702112685044612,
+            "baryscore_SD_5": 1.0400339952735052,
+            "baryscore_SD_0.1": 0.23922258231524435,
+            "baryscore_SD_0.5": 0.8306662580717632,
+            "baryscore_SD_0.01": 0.21527512761808326,
+            "baryscore_SD_0.001": 4.697930593586469e-13
+          }
+        },
+        {
+          "baryscore": {
+            "baryscore_W": 0.4358260942352773,
+            "baryscore_SD_10": 1.1271078926712006,
+            "baryscore_SD_1": 1.0761172779654367,
+            "baryscore_SD_5": 1.122654562880904,
+            "baryscore_SD_0.1": 0.4730193878365072,
+            "baryscore_SD_0.5": 0.9839934620075355,
+            "baryscore_SD_0.01": 0.4360028677359103,
+            "baryscore_SD_0.001": 2.003970287549622e-12
+          }
+        },
+        {
+          "baryscore": {
+            "baryscore_W": 0.2773132631248679,
+            "baryscore_SD_10": 1.1206847017919714,
+            "baryscore_SD_1": 1.0499791258417068,
+            "baryscore_SD_5": 1.1147019711645716,
+            "baryscore_SD_0.1": 0.3097034053680021,
+            "baryscore_SD_0.5": 0.9174894115604608,
+            "baryscore_SD_0.01": 0.27750219368002466,
+            "baryscore_SD_0.001": 5.894813472187973e-13
+          }
+        },
+        {
+          "baryscore": {
+            "baryscore_W": 0.23976404256396663,
+            "baryscore_SD_10": 1.0222037579173389,
+            "baryscore_SD_1": 0.932632833371444,
+            "baryscore_SD_5": 1.0145535512722,
+            "baryscore_SD_0.1": 0.2595290888664649,
+            "baryscore_SD_0.5": 0.781003330061966,
+            "baryscore_SD_0.01": 0.2398996197805371,
+            "baryscore_SD_0.001": 3.878808957566489e-08
+          }
+        },
+        {
+          "baryscore": {
+            "baryscore_W": 0.18466722724632056,
+            "baryscore_SD_10": 1.1692663047686125,
+            "baryscore_SD_1": 1.10046281690413,
+            "baryscore_SD_5": 1.1636181494427986,
+            "baryscore_SD_0.1": 0.21104931914058828,
+            "baryscore_SD_0.5": 0.9563417360657935,
+            "baryscore_SD_0.01": 0.18483496360530424,
+            "baryscore_SD_0.001": 2.2912007134129803e-07
+          }
+        },
+        {
+          "baryscore": {
+            "baryscore_W": 0.260098833186975,
+            "baryscore_SD_10": 1.0367048778399321,
+            "baryscore_SD_1": 0.9588764225832606,
+            "baryscore_SD_5": 1.0301040402865476,
+            "baryscore_SD_0.1": 0.27667827809668555,
+            "baryscore_SD_0.5": 0.8179871491341018,
+            "baryscore_SD_0.01": 0.26024052060411407,
+            "baryscore_SD_0.001": 1.434770945173599e-07
+          }
+        },
+        {
+          "baryscore": {
+            "baryscore_W": 0.24392037893244028,
+            "baryscore_SD_10": 1.0794176280809022,
+            "baryscore_SD_1": 1.0082520965314263,
+            "baryscore_SD_5": 1.0733175064441536,
+            "baryscore_SD_0.1": 0.27297927108576264,
+            "baryscore_SD_0.5": 0.8799754860772669,
+            "baryscore_SD_0.01": 0.24414780158226812,
+            "baryscore_SD_0.001": 7.052933189447465e-16
+          }
+        },
+        {
+          "baryscore": {
+            "baryscore_W": 0.2558515650915149,
+            "baryscore_SD_10": 0.9045964972939674,
+            "baryscore_SD_1": 0.8474374396964925,
+            "baryscore_SD_5": 0.8994996276376147,
+            "baryscore_SD_0.1": 0.29707305102287407,
+            "baryscore_SD_0.5": 0.7536742602641777,
+            "baryscore_SD_0.01": 0.2559697377459444,
+            "baryscore_SD_0.001": 2.8274128311732745e-10
+          }
+        },
+        {
+          "baryscore": {
+            "baryscore_W": 0.25074680525621296,
+            "baryscore_SD_10": 1.0804328930900553,
+            "baryscore_SD_1": 1.007575565847855,
+            "baryscore_SD_5": 1.0744064713965558,
+            "baryscore_SD_0.1": 0.27964187479247266,
+            "baryscore_SD_0.5": 0.8683185889200877,
+            "baryscore_SD_0.01": 0.2509156656384637,
+            "baryscore_SD_0.001": 1.1167937726480772e-07
+          }
+        }
+      ],
+      "macro": {
+        "baryscore": {
+          "baryscore_W": 0.2424171149049183,
+          "baryscore_SD_10": 1.054359502476407,
+          "baryscore_SD_1": 0.9784250173014872,
+          "baryscore_SD_5": 1.0478904082183578,
+          "baryscore_SD_0.1": 0.26902971856668934,
+          "baryscore_SD_0.5": 0.8416927492083435,
+          "baryscore_SD_0.01": 0.2425544605910356,
+          "baryscore_SD_0.001": 0.006075663033526649
+        }
+      }
+    }
+  },
+  "depthscore": {
+    "cpu": {
+      "micro": [
+        {
+          "depthscore": {
+            "depth_score": 0.12355822859867578
+          }
+        },
+        {
+          "depthscore": {
+            "depth_score": 0.12650006476121495
+          }
+        },
+        {
+          "depthscore": {
+            "depth_score": 0.14299116280021706
+          }
+        },
+        {
+          "depthscore": {
+            "depth_score": 0.11250987366375828
+          }
+        },
+        {
+          "depthscore": {
+            "depth_score": 0.12416236482321114
+          }
+        },
+        {
+          "depthscore": {
+            "depth_score": 0.1359616455787239
+          }
+        },
+        {
+          "depthscore": {
+            "depth_score": 0.11549112226711965
+          }
+        },
+        {
+          "depthscore": {
+            "depth_score": 0.14264698824569153
+          }
+        },
+        {
+          "depthscore": {
+            "depth_score": 0.1233971438754036
+          }
+        },
+        {
+          "depthscore": {
+            "depth_score": 0.10464791475910355
+          }
+        }
+      ],
+      "macro": {
+        "depthscore": {
+          "depth_score": 0.12518665093731193
+        }
+      }
+    },
+    "gpu": {
+      "micro": [
+        {
+          "depthscore": {
+            "depth_score": 0.140606228957553
+          }
+        },
+        {
+          "depthscore": {
+            "depth_score": 0.13433283716956418
+          }
+        },
+        {
+          "depthscore": {
+            "depth_score": 0.1340547153905349
+          }
+        },
+        {
+          "depthscore": {
+            "depth_score": 0.1266320350135981
+          }
+        },
+        {
+          "depthscore": {
+            "depth_score": 0.12889825472303892
+          }
+        },
+        {
+          "depthscore": {
+            "depth_score": 0.10919916321393738
+          }
+        },
+        {
+          "depthscore": {
+            "depth_score": 0.12030093329042388
+          }
+        },
+        {
+          "depthscore": {
+            "depth_score": 0.13452170360274043
+          }
+        },
+        {
+          "depthscore": {
+            "depth_score": 0.12487044901944663
+          }
+        },
+        {
+          "depthscore": {
+            "depth_score": 0.12478174501633163
+          }
+        }
+      ],
+      "macro": {
+        "depthscore": {
+          "depth_score": 0.1278198065397169
+        }
+      }
+    }
+  }
+}
\ No newline at end of file
diff --git a/models/colombo2021/tests/fixtures/hyps.txt b/models/colombo2021/tests/fixtures/hyps.txt
new file mode 100644
index 0000000..41be794
--- /dev/null
+++ b/models/colombo2021/tests/fixtures/hyps.txt
@@ -0,0 +1,10 @@
+28-year-old chef found dead in San Francisco mall
+A 28-year-old chef who recently moved to San Francisco was found dead in the staircase of a local shopping center.
+The victim's brother said he cannot imagine anyone who would want to harm him,"Finally, it went uphill again at him."
+The corpse, found Wednesday morning in the Westfield Mall, was identified as the 28-year-old Frank Galicia from San Francisco, the Justice Department said in San Francisco.
+The San Francisco Police Department said the death was classified as murder and the investigation is on the running.
+The victim's brother, Louis Galicia, told the ABS broadcaster KGO in San Francisco that Frank, who formerly worked as a cook in Boston, had his dream job as a cook at the Sons & Daughters restaurant in San Francisco six months ago.
+A spokesman for the Sons & Daughters said they were "shocked and destroyed on the ground" over his death.
+"We are a small team that works like a close family and we are going to miss him painfully," said the spokesman.
+Our thoughts and condolences are in this difficult time at Franks's family and friends.
+Louis Galicia admitted that Frank initially lived in hostels, but that "things for him finally went uphill."
diff --git a/models/colombo2021/tests/fixtures/refs.txt b/models/colombo2021/tests/fixtures/refs.txt
new file mode 100644
index 0000000..b29adc7
--- /dev/null
+++ b/models/colombo2021/tests/fixtures/refs.txt
@@ -0,0 +1,10 @@
+28-Year-Old Chef Found Dead at San Francisco Mall
+A 28-year-old chef who had recently moved to San Francisco was found dead in the stairwell of a local mall this week.
+But the victim's brother says he can't think of anyone who would want to hurt him, saying, "Things were finally going well for him."
+The body found at the Westfield Mall Wednesday morning was identified as 28-year-old San Francisco resident Frank Galicia, the San Francisco Medical Examiner's Office said.
+The San Francisco Police Department said the death was ruled a homicide and an investigation is ongoing.
+The victim's brother, Louis Galicia, told ABC station KGO in San Francisco that Frank, previously a line cook in Boston, had landed his dream job as line chef at San Francisco's Sons & Daughters restaurant six months ago.
+A spokesperson for Sons & Daughters said they were "shocked and devastated" by his death.
+"We are a small team that operates like a close knit family and he will be dearly missed," the spokesperson said.
+Our thoughts and condolences are with Frank's family and friends at this difficult time.
+Louis Galicia said Frank initially stayed in hostels, but recently, "Things were finally going well for him."
diff --git a/models/colombo2021/tests/models_test.py b/models/colombo2021/tests/models_test.py
new file mode 100644
index 0000000..57e36bf
--- /dev/null
+++ b/models/colombo2021/tests/models_test.py
@@ -0,0 +1,62 @@
+import json
+import unittest
+from parameterized import parameterized
+
+from repro.models.colombo2021 import BaryScore, DepthScore, InfoLM
+from repro.testing import assert_dicts_approx_equal, get_testing_device_parameters
+
+from . import FIXTURES_ROOT
+
+
+class TestColombo2021Models(unittest.TestCase):
+    def setUp(self) -> None:
+        # Load the inputs
+        self.inputs = []
+        with open(f"{FIXTURES_ROOT}/refs.txt", "r") as f_refs:
+            with open(f"{FIXTURES_ROOT}/hyps.txt", "r") as f_hyps:
+                for reference, hyp in zip(f_refs, f_hyps):
+                    self.inputs.append(
+                        {"candidate": hyp.strip(), "references": [reference.strip()]}
+                    )
+
+        # Load the expected outputs
+        with open(f"{FIXTURES_ROOT}/expected.json", "r") as f:
+            self.expected = json.load(f)
+
+    @parameterized.expand(get_testing_device_parameters())
+    def test_infolm(self, device: int):
+        model = InfoLM(device=device)
+        expected_macro = self.expected["infolm"]["macro"]
+        expected_micro = self.expected["infolm"]["micro"]
+        actual_macro, actual_micro = model.predict_batch(self.inputs)
+
+        assert_dicts_approx_equal(expected_macro, actual_macro, abs=1e-4)
+        assert len(expected_micro) == len(actual_micro)
+        for expected, actual in zip(expected_micro, actual_micro):
+            assert_dicts_approx_equal(expected, actual, abs=1e-4)
+
+    @parameterized.expand(get_testing_device_parameters())
+    def test_baryscore(self, device: int):
+        model = BaryScore(device=device)
+        device = "cpu" if device == -1 else "gpu"
+        expected_macro = self.expected["baryscore"][device]["macro"]
+        expected_micro = self.expected["baryscore"][device]["micro"]
+        actual_macro, actual_micro = model.predict_batch(self.inputs)
+
+        assert_dicts_approx_equal(expected_macro, actual_macro, abs=1e-4)
+        assert len(expected_micro) == len(actual_micro)
+        for expected, actual in zip(expected_micro, actual_micro):
+            assert_dicts_approx_equal(expected, actual, abs=1e-4)
+
+    @parameterized.expand(get_testing_device_parameters())
+    def test_depthscore(self, device: int):
+        model = DepthScore(device=device)
+        device = "cpu" if device == -1 else "gpu"
+        expected_macro = self.expected["depthscore"][device]["macro"]
+        expected_micro = self.expected["depthscore"][device]["micro"]
+        actual_macro, actual_micro = model.predict_batch(self.inputs)
+
+        assert_dicts_approx_equal(expected_macro, actual_macro, abs=1e-4)
+        assert len(expected_micro) == len(actual_micro)
+        for expected, actual in zip(expected_micro, actual_micro):
+            assert_dicts_approx_equal(expected, actual, abs=1e-4)