Repository: obss/jury (language: python)

Issue:
Refactor docstrings, typo-check, review imports
Codebase needs a refactor mainly for docstrings and constant strings (like _CITATIONS, etc. used in metric info). Also, import statements needs to be reviewed (e.g.for no-longer supported versions/legacy should be removed). Update README accordingly.


Discussion / hints:

Fix patch:
diff --git a/README.md b/README.md
index 2b30119..e2d4445 100644
--- a/README.md
+++ b/README.md
@@ -15,17 +15,67 @@
 <a href="https://doi.org/10.5281/zenodo.6109838"><img src="https://zenodo.org/badge/DOI/10.5281/zenodo.6109838.svg" alt="DOI"></a>
 </p>
 
-A comprehensive toolkit for evaluating NLP experiments offering various automated metrics. Jury offers a smooth and easy-to-use interface. It uses [datasets](https://github.com/huggingface/datasets/) for underlying metric computation, and hence adding custom metric is easy as extending proper class.
+A comprehensive toolkit for evaluating NLP experiments offering various automated metrics. Jury offers a smooth and easy-to-use interface. It uses a more advanced version of [datasets](https://github.com/huggingface/datasets/) design for underlying metric computation, so that adding custom metric is easy as extending proper class.
 
 Main advantages that Jury offers are:
 
-- Easy to use for any NLG system.
+- Easy to use for any NLP project.
+- Unified structure for computation input across all metrics.
 - Calculate many metrics at once.
-- Metrics calculations are handled concurrently to save processing time.
-- It supports evaluating multiple predictions seamlessly.
+- Metrics calculations can be handled concurrently to save processing time.
+- It seamlessly supports evaluation for multiple predictions/multiple references.
 
 To see more, check the [official Jury blog post](https://medium.com/codable/jury-evaluating-performance-of-nlg-models-730eb9c9999f).
 
+# Available Metrics
+
+The table below shows the current support status for available metrics.
+
+| Metric                                                                        | Jury Support       | HF/datasets Support |
+|-------------------------------------------------------------------------------|--------------------|---------------------|
+| Accuracy-Numeric                                                              | :heavy_check_mark: | :white_check_mark:  |
+| Accuracy-Text                                                                 | :heavy_check_mark: | :x:                 |
+| Bartscore                                                                     | :heavy_check_mark: | :x:                 |
+| Bertscore                                                                     | :heavy_check_mark: | :white_check_mark:  |
+| Bleu                                                                          | :heavy_check_mark: | :white_check_mark:  |
+| Bleurt                                                                        | :heavy_check_mark: | :white_check_mark:  |
+| CER                                                                           | :heavy_check_mark: | :white_check_mark:  |
+| CHRF                                                                          | :heavy_check_mark: | :white_check_mark:  |
+| COMET                                                                         | :heavy_check_mark: | :white_check_mark:  |
+| F1-Numeric                                                                    | :heavy_check_mark: | :white_check_mark:  |
+| F1-Text                                                                       | :heavy_check_mark: | :x:                 |
+| METEOR                                                                        | :heavy_check_mark: | :white_check_mark:  |
+| Precision-Numeric                                                             | :heavy_check_mark: | :white_check_mark:  |
+| Precision-Text                                                                | :heavy_check_mark: | :x:                 |
+| Prism                                                                         | :heavy_check_mark: | :x:                 |
+| Recall-Numeric                                                                | :heavy_check_mark: | :white_check_mark:  |
+| Recall-Text                                                                   | :heavy_check_mark: | :x:                 |
+| ROUGE                                                                         | :heavy_check_mark: | :white_check_mark:  |
+| SacreBleu                                                                     | :heavy_check_mark: | :white_check_mark:  |
+| Seqeval                                                                       | :heavy_check_mark: | :white_check_mark:  |
+| Squad                                                                         | :heavy_check_mark: | :white_check_mark:  |
+| TER                                                                           | :heavy_check_mark: | :white_check_mark:  |
+| WER                                                                           | :heavy_check_mark: | :white_check_mark:  |
+| [Other metrics](https://github.com/huggingface/datasets/tree/master/metrics)* | :white_check_mark: | :white_check_mark:  |
+
+_*_ Placeholder for the rest of the metrics available in `datasets` package apart from those which are present in the 
+table. 
+
+**Notes**
+
+* In the table :white_check_mark: means that this metric is supported (for Jury through the `datasets`), so that it 
+can be used just like the `datasets` metric although unfortunately full Jury support for those metrics are not yet 
+available.
+
+* For metrics marked as :white_check_mark: are available in Jury, and they can (and should) be used as instructed 
+in `datasets` implementation, and there may not be a unified interface that applies for all of these partially 
+supported metrics.
+
+## Request for a New Metric
+
+For the request of a new metric please [open an issue](https://github.com/obss/jury/issues/new?assignees=&labels=&template=new-metric.md&title=) providing the minimum information. Also, PRs addressing new metric 
+supports are welcomed :).
+
 ## <div align="center"> Installation </div>
 
 Through pip,
diff --git a/jury/metrics/_core/auto.py b/jury/metrics/_core/auto.py
index ede7725..6a398ae 100644
--- a/jury/metrics/_core/auto.py
+++ b/jury/metrics/_core/auto.py
@@ -1,5 +1,5 @@
 # coding=utf-8
-# Copyright 2020 Open Business Software Solutions, The HuggingFace Datasets Authors and the TensorFlow Datasets Authors.
+# Copyright 2021 Open Business Software Solutions, The HuggingFace Datasets Authors and the TensorFlow Datasets Authors.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
diff --git a/jury/metrics/_core/base.py b/jury/metrics/_core/base.py
index a4d416c..e572c86 100644
--- a/jury/metrics/_core/base.py
+++ b/jury/metrics/_core/base.py
@@ -1,5 +1,5 @@
 # coding=utf-8
-# Copyright 2020 The HuggingFace Datasets Authors
+# Copyright 2021 Open Business Software Solutions, The HuggingFace Datasets Authors
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -196,7 +196,7 @@ def task(self):
         return self._task
 
 
-class MetricForTask(Metric):
+class MetricForTask(Metric, ABC):
     """
     Base metric class for any task. All metrics must extend this class as metric is required to adopt a task
     inherently. Default task will be language-generation for AutoMetric.
diff --git a/jury/metrics/accuracy/accuracy_for_language_generation.py b/jury/metrics/accuracy/accuracy_for_language_generation.py
index 7b64d9e..e46ebdd 100644
--- a/jury/metrics/accuracy/accuracy_for_language_generation.py
+++ b/jury/metrics/accuracy/accuracy_for_language_generation.py
@@ -29,23 +29,28 @@
 from jury.utils.nlp import normalize_text
 
 _CITATION = """\
-@inproceedings{papineni2002bleu,
-  title={Bleu: a method for automatic evaluation of machine translation},
-  author={Papineni, Kishore and Roukos, Salim and Ward, Todd and Zhu, Wei-Jing},
-  booktitle={Proceedings of the 40th annual meeting of the Association for Computational Linguistics},
-  pages={311--318},
-  year={2002}
+@article{scikit-learn,
+  title={Scikit-learn: Machine Learning in {P}ython},
+  author={Pedregosa, F. and Varoquaux, G. and Gramfort, A. and Michel, V.
+         and Thirion, B. and Grisel, O. and Blondel, M. and Prettenhofer, P.
+         and Weiss, R. and Dubourg, V. and Vanderplas, J. and Passos, A. and
+         Cournapeau, D. and Brucher, M. and Perrot, M. and Duchesnay, E.},
+  journal={Journal of Machine Learning Research},
+  volume={12},
+  pages={2825--2830},
+  year={2011}
 }
 """
 
 _DESCRIPTION = """
-This metric is derived from Modified Unigram Precision as an accuracy metric so that
-it will compute across maximum of reference or prediction. The computation is similar 
-to precision and recall, however, we call this accuracy since there is no measure 
-called "modified unigram accuracy".
-Accuracy is the fraction of the common unigrams between the prediction
-and the references among the max of prediction or reference tokens. It can be computed with:
-Accuracy = # of matching tokens / max(# of prediction tokens, # of reference tokens)
+Accuracy is the proportion of correct predictions among the total number of cases processed. It can be computed with:
+Accuracy = (TP + TN) / (TP + TN + FP + FN)
+TP: True positive
+TN: True negative
+FP: False positive
+FN: False negative
+
+Accuracy for language generation computes token based accuracy.
 """
 
 _KWARGS_DESCRIPTION = """
diff --git a/jury/metrics/accuracy/accuracy_for_sequence_classification.py b/jury/metrics/accuracy/accuracy_for_sequence_classification.py
index 7351c56..ebf66cf 100644
--- a/jury/metrics/accuracy/accuracy_for_sequence_classification.py
+++ b/jury/metrics/accuracy/accuracy_for_sequence_classification.py
@@ -1,5 +1,5 @@
 # coding=utf-8
-# Copyright 2020 The HuggingFace Datasets Authors and the current dataset script contributor.
+# Copyright 2021 Open Business Software Solutions, The HuggingFace Datasets Authors.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -25,44 +25,45 @@
 from jury.metrics._core import MetricForSequenceClassification, SequenceClassificationInstance
 
 _CITATION = """\
-@inproceedings{papineni2002bleu,
-  title={Bleu: a method for automatic evaluation of machine translation},
-  author={Papineni, Kishore and Roukos, Salim and Ward, Todd and Zhu, Wei-Jing},
-  booktitle={Proceedings of the 40th annual meeting of the Association for Computational Linguistics},
-  pages={311--318},
-  year={2002}
+@article{scikit-learn,
+  title={Scikit-learn: Machine Learning in {P}ython},
+  author={Pedregosa, F. and Varoquaux, G. and Gramfort, A. and Michel, V.
+         and Thirion, B. and Grisel, O. and Blondel, M. and Prettenhofer, P.
+         and Weiss, R. and Dubourg, V. and Vanderplas, J. and Passos, A. and
+         Cournapeau, D. and Brucher, M. and Perrot, M. and Duchesnay, E.},
+  journal={Journal of Machine Learning Research},
+  volume={12},
+  pages={2825--2830},
+  year={2011}
 }
 """
 
 _DESCRIPTION = """
-This metric is derived from Modified Unigram Precision as an accuracy metric so that
-it will compute across maximum of reference or prediction. The computation is similar 
-to precision and recall, however, we call this accuracy since there is no measure 
-called "modified unigram accuracy".
-Accuracy is the fraction of the common unigrams between the prediction
-and the references among the max of prediction or reference tokens. It can be computed with:
-Accuracy = # of matching tokens / max(# of prediction tokens, # of reference tokens)
+Accuracy is the proportion of correct predictions among the total number of cases processed. It can be computed with:
+Accuracy = (TP + TN) / (TP + TN + FP + FN)
+TP: True positive
+TN: True negative
+FP: False positive
+FN: False negative
 """
 
 _KWARGS_DESCRIPTION = """
 Args:
-    predictions: list of predictions to score. Each predictions
-        should be a string with tokens separated by spaces.
-    references: list of reference for each prediction. Each
-        reference should be a string with tokens separated by spaces.
+    predictions: Predicted labels, as returned by a model.
+    references: Ground truth labels.
+    normalize: If False, return the number of correctly classified samples.
+        Otherwise, return the fraction of correctly classified samples.
+    sample_weight: Sample weights.
 Returns:
-    'score': Accuracy score.
+    accuracy: Accuracy score.
 Examples:
 
-    >>> accuracy = jury.load_metric("accuracy")
-    >>> predictions = [["the cat is on the mat", "There is cat playing on the mat"], ["Look! a wonderful day."]]
-    >>> references = [
-        ["the cat is playing on the mat.", "The cat plays on the mat."], 
-        ["Today is a wonderful day", "The weather outside is wonderful."]
-    ]
+    >>> accuracy = jury.load_metric("accuracy", task="sequence-classification")
+    >>> predictions = [[0], [2], [1], [0], [0], [1]]
+    >>> references = [[0], [1], [2], [0], [1], [2]]
     >>> results = accuracy.compute(predictions=predictions, references=references)
     >>> print(results)
-    {'accuracy': {'score': 0.7285714285714285}}
+    {'accuracy': {'score': 0.3333333333333333}}
 """
 
 
diff --git a/jury/metrics/bartscore/bartscore_for_language_generation.py b/jury/metrics/bartscore/bartscore_for_language_generation.py
index 0e80a31..b7564a0 100644
--- a/jury/metrics/bartscore/bartscore_for_language_generation.py
+++ b/jury/metrics/bartscore/bartscore_for_language_generation.py
@@ -12,9 +12,11 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-""" Bartscore metric. The part of this file is adapted from metric implementations
+"""
+BARTScore metric. The part of this file is adapted from metric implementations
 of datasets package. See
-https://github.com/huggingface/datasets/blob/master/metrics/ """
+https://github.com/huggingface/datasets/blob/master/metrics/
+"""
 from typing import Callable, Dict, List
 
 import datasets
@@ -65,23 +67,20 @@
 Examples:
 
     >>> bartscore = jury.load_metric("bartscore")
-    >>> predictions = [
-        ["the cat is on the mat", "There is cat playing on mat"],
-        ["Look! what a wonderful day, today.", "Today is a very wonderful day"],
-    ]
+    >>> predictions = [["the cat is on the mat", "There is cat playing on the mat"], ["Look! a wonderful day."]]
     >>> references = [
-        ["the cat is playing on the mat.", "The cat plays on the mat."],
-        ["Today is a wonderful day", "The weather outside is wonderful."],
+        ["the cat is playing on the mat.", "The cat plays on the mat."], 
+        ["Today is a wonderful day", "The weather outside is wonderful."]
     ]
     >>> results = bartscore.compute(predictions=predictions, references=references)
     >>> print(results)
     {
-        'bartscore': {
-            'score': -1.8933048248291016,
-            'model_checkpoint': 'bartscore-large-cnn',
-            'model_weights': None,
-            'segment_scores': False
-        }
+      "bartscore": {
+        "score": -2.201135754585266,
+        "model_checkpoint": "bartscore-large-cnn",
+        "model_weights": null,
+        "segment_scores": false
+      }
     }
 """
 
diff --git a/jury/metrics/bertscore/bertscore_for_language_generation.py b/jury/metrics/bertscore/bertscore_for_language_generation.py
index 5fc168f..680401c 100644
--- a/jury/metrics/bertscore/bertscore_for_language_generation.py
+++ b/jury/metrics/bertscore/bertscore_for_language_generation.py
@@ -12,10 +12,11 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-""" BERTScore metric. The part of this file is adapted from BERTScore implementation
+"""
+BERTScore metric. The part of this file is adapted from BERTScore implementation
 of datasets package. See
-https://github.com/huggingface/datasets/blob/master/metrics/bertscore/bertscore.py """
-
+https://github.com/huggingface/datasets/blob/master/metrics/bertscore/bertscore.py
+"""
 import functools
 from contextlib import contextmanager
 from typing import Dict, List
diff --git a/jury/metrics/bleu/bleu_for_language_generation.py b/jury/metrics/bleu/bleu_for_language_generation.py
index 8437f8c..e019457 100644
--- a/jury/metrics/bleu/bleu_for_language_generation.py
+++ b/jury/metrics/bleu/bleu_for_language_generation.py
@@ -94,9 +94,21 @@
     ]
     >>> results = bleu.compute(predictions=predictions, references=references)
     >>> print(results)
-    {'bleu': {'score': 0.42370250917168295, 
-        'precisions': [0.8823529411764706, 0.6428571428571429, 0.45454545454545453, 0.125], 
-        'brevity_penalty': 1.0, 'length_ratio': 1.0, 'translation_length': 11, 'reference_length': 11}}
+    {
+      "bleu": {
+        "score": 0.42370250917168295,
+        "precisions": [
+          0.8823529411764706,
+          0.6428571428571429,
+          0.45454545454545453,
+          0.125
+        ],
+        "brevity_penalty": 1.0,
+        "length_ratio": 1.0,
+        "translation_length": 11,
+        "reference_length": 11
+      }
+    }
 """
 
 
@@ -164,7 +176,7 @@ def _compute_single_pred_single_ref(
         max_order: int = 4,
         smooth: bool = False,
     ):
-        # _compute_bleu_score expects a slighthly different reference structure
+        # _compute_bleu_score expects a slightly different reference structure
         #   than the one we currently have for single prediction single reference
         references = [[r] for r in references]
         return self._compute_bleu_score(
diff --git a/jury/metrics/bleurt/bleurt_for_language_generation.py b/jury/metrics/bleurt/bleurt_for_language_generation.py
index e8bca96..71fbf42 100644
--- a/jury/metrics/bleurt/bleurt_for_language_generation.py
+++ b/jury/metrics/bleurt/bleurt_for_language_generation.py
@@ -12,9 +12,11 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-""" BLEURT metric. The part of this file is adapted from BLEURT implementation
+"""
+BLEURT metric. The part of this file is adapted from BLEURT implementation
 of datasets package. See
-https://github.com/huggingface/datasets/blob/master/metrics/bleurt/bleurt.py """
+https://github.com/huggingface/datasets/blob/master/metrics/bleurt/bleurt.py
+"""
 
 import os
 from typing import Callable
@@ -61,12 +63,15 @@
     'scores': List of scores.
 Examples:
 
-    >>> predictions = ["hello there", "general kenobi"]
-    >>> references = ["hello there", "general kenobi"]
-    >>> bleurt = jury.load_metric("bleurt")
+    >>> predictions = [["the cat is on the mat", "There is cat playing on the mat"], ["Look! a wonderful day."]]
+    >>> references = [
+        ["the cat is playing on the mat.", "The cat plays on the mat."], 
+        ["Today is a wonderful day", "The weather outside is wonderful."]
+    ]
+    >>> bleurt = jury.load_metric("bleurt", config_name="bleurt-tiny-128")
     >>> results = bleurt.compute(predictions=predictions, references=references)
-    >>> print([round(v, 2) for v in results["scores"]])
-    [1.03, 1.04]
+    >>> print(results)
+    {'bleurt': {'score': 0.25963682122528553, 'scores': [0.47344309091567993, 0.04583055153489113], 'checkpoint': 'bleurt-tiny-128'}}
 """
 
 _LICENSE = """Copyright 2021 Google.
@@ -188,3 +193,13 @@ def _compute_multi_pred_multi_ref(
             scores.append(reduced_score)
 
         return {"score": sum(scores) / len(scores), "scores": scores, "checkpoint": self.config_name}
+
+
+if __name__ == "__main__":
+    import json
+
+    predictions = [["hello there", "general kenobi"]]
+    references = [["hello there", "general kenobi"]]
+    bleurt = BleurtForLanguageGeneration()
+    results = bleurt.compute(predictions=predictions, references=references)
+    print(json.dumps(results, indent=2))
diff --git a/jury/metrics/cer/cer_for_language_generation.py b/jury/metrics/cer/cer_for_language_generation.py
index f250fb7..11e4e29 100644
--- a/jury/metrics/cer/cer_for_language_generation.py
+++ b/jury/metrics/cer/cer_for_language_generation.py
@@ -1,5 +1,5 @@
 # coding=utf-8
-# Copyright 2020 Open Business Software Solutions, The HuggingFace Datasets Authors.
+# Copyright 2021 Open Business Software Solutions, The HuggingFace Datasets Authors.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -62,12 +62,25 @@
 Returns:
     (float): the character error rate
 Examples:
-    >>> predictions = ["this is the prediction", "there is an other sample"]
-    >>> references = ["this is the reference", "there is another one"]
+    >>> predictions = [["the cat is on the mat", "There is cat playing on the mat"], ["Look! a wonderful day."]]
+    >>> references = [
+        ["the cat is playing on the mat.", "The cat plays on the mat."], 
+        ["Today is a wonderful day", "The weather outside is wonderful."]
+    ]
     >>> cer = jury.load_metric("cer")
     >>> cer_score = cer.compute(predictions=predictions, references=references)
     >>> print(cer_score)
-    0.34146341463414637
+    {
+      "cer": {
+        "score": 0.7272727272727273,
+        "overall": {
+          "substitutions": 3.1666666666666665,
+          "deletions": 5.5,
+          "insertions": 2.3333333333333335,
+          "hits": 19.166666666666668
+        }
+      }
+    }
 """
 
 
diff --git a/jury/metrics/chrf/chrf_for_language_generation.py b/jury/metrics/chrf/chrf_for_language_generation.py
index 9ad976c..7f87bac 100644
--- a/jury/metrics/chrf/chrf_for_language_generation.py
+++ b/jury/metrics/chrf/chrf_for_language_generation.py
@@ -1,5 +1,5 @@
 # coding=utf-8
-# Copyright 2020 Open Business Software Solutions, The HuggingFace Datasets Authors.
+# Copyright 2021 Open Business Software Solutions, The HuggingFace Datasets Authors.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -12,7 +12,8 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-""" ChrF(++) (Character n-gram F-score) metric. The part of this file is adapted from HuggingFace's
+"""
+ChrF(++) (Character n-gram F-score) metric. The part of this file is adapted from HuggingFace's
 datasets package implementation of CHRF metric. See
 https://github.com/huggingface/datasets/blob/master/metrics/chrf/chrf.py
 """
@@ -96,12 +97,15 @@
     'word_order': The word n-gram order. If equals to 2, the metric is referred to as chrF++,
     'beta': Determine the importance of recall w.r.t precision
 Examples:
-    >>> prediction = [["The relationship between Obama and Netanyahu is not exactly friendly."]]
-    >>> reference = [["The ties between Obama and Netanyahu are not particularly friendly."]]
+    >>> predictions = [["the cat is on the mat", "There is cat playing on the mat"], ["Look! a wonderful day."]]
+    >>> references = [
+        ["the cat is playing on the mat.", "The cat plays on the mat."], 
+        ["Today is a wonderful day", "The weather outside is wonderful."]
+    ]
     >>> chrf = jury.load_metric("chrf")
     >>> results = chrf.compute(predictions=prediction, references=reference)
     >>> print(results)
-    {'score': 61.576379378113785, 'char_order': 6, 'word_order': 0, 'beta': 2}
+    {'chrf': {'score': 0.29778203723986857, 'char_order': 6, 'word_order': 0, 'beta': 2}}
 """
 
 
diff --git a/jury/metrics/comet/comet_for_language_generation.py b/jury/metrics/comet/comet_for_language_generation.py
index 61fb6e5..137cd44 100644
--- a/jury/metrics/comet/comet_for_language_generation.py
+++ b/jury/metrics/comet/comet_for_language_generation.py
@@ -1,5 +1,5 @@
 # coding=utf-8
-# Copyright 2020 Open Business Software Solutions, The HuggingFace Datasets Authors.
+# Copyright 2021 Open Business Software Solutions, The HuggingFace Datasets Authors.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -77,13 +77,16 @@
     `samples`: List of dictionaries with `src`, `mt`, `ref` and `score`.
     `scores`: List of scores.
 Examples:
-    >>> comet_metric = jury.load_metric('comet')
-    >>> source = ["Dem Feuer konnte Einhalt geboten werden", "Schulen und Kindergärten wurden eröffnet."]
-    >>> hypothesis = ["The fire could be stopped", "Schools and kindergartens were open"]
-    >>> reference = ["They were able to control the fire.", "Schools and kindergartens opened"]
-    >>> results = comet_metric.compute(predictions=hypothesis, references=reference, sources=source)
+    >>> comet_metric = jury.load_metric('comet', config_name="wmt21-cometinho-da")
+    >>> source = ["Die Katze spielt auf der Matte.", "Heute ist ein wunderbarer Tag."]
+    >>> predictions = [["the cat is on the mat", "There is cat playing on the mat"], ["Look! a wonderful day."]]
+    >>> references = [
+        ["the cat is playing on the mat.", "The cat plays on the mat."], 
+        ["Today is a wonderful day", "The weather outside is wonderful."]
+    ]
+    >>> results = comet_metric.compute(sources=source, predictions=hypothesis, references=reference)
     >>> print(results)
-    {'scores': [0.19016975164413452, 0.9156621694564819], 'samples': 0.5529159605503082}
+    {'comet': {'scores': [0.6338749527931213, 0.4925243854522705], 'samples': 0.5631996691226959}}
 """
 
 
diff --git a/jury/metrics/f1/f1_for_sequence_classification.py b/jury/metrics/f1/f1_for_sequence_classification.py
index 94d27c0..38a5aa3 100644
--- a/jury/metrics/f1/f1_for_sequence_classification.py
+++ b/jury/metrics/f1/f1_for_sequence_classification.py
@@ -1,5 +1,5 @@
 # coding=utf-8
-# Copyright 2020 The HuggingFace Datasets Authors and the current dataset script contributor.
+# Copyright 2021 The HuggingFace Datasets Authors and the current dataset script contributor.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -60,25 +60,25 @@
     f1: F1 score.
 Examples:
 
-    >>> f1_metric = datasets.load_metric("f1")
-    >>> results = f1_metric.compute(predictions=[0, 1], references=[0, 1])
+    >>> f1_metric = jury.load_metric("f1", task="sequence-classification")
+    >>> results = f1_metric.compute(predictions=[[0, 1]], references=[[0, 1]])
     >>> print(results)
     {'f1': 1.0}
 
-    >>> predictions = [0, 2, 1, 0, 0, 1]
-    >>> references = [0, 1, 2, 0, 1, 2]
+    >>> predictions = [[0], [2], [1], [0], [0], [1]]
+    >>> references = [[0], [1], [2], [0], [1], [2]]
     >>> results = f1_metric.compute(predictions=predictions, references=references, average="macro")
     >>> print(results)
-    {'f1': 0.26666666666666666}
+    {'f1': {'score': 0.26666666666666666}}
     >>> results = f1_metric.compute(predictions=predictions, references=references, average="micro")
     >>> print(results)
-    {'f1': 0.3333333333333333}
+    {'f1': {'score': 0.3333333333333333}}
     >>> results = f1_metric.compute(predictions=predictions, references=references, average="weighted")
     >>> print(results)
-    {'f1': 0.26666666666666666}
+    {'f1': {'score': 0.26666666666666666}}
     >>> results = f1_metric.compute(predictions=predictions, references=references, average=None)
     >>> print(results)
-    {'f1': array([0.8, 0. , 0. ])}
+    {'f1': {'score': [0.8, 0.0, 0.0]}}
 """
 
 _CITATION = """
diff --git a/jury/metrics/meteor/meteor_for_language_generation.py b/jury/metrics/meteor/meteor_for_language_generation.py
index 0fbbaaf..eb7a5f2 100644
--- a/jury/metrics/meteor/meteor_for_language_generation.py
+++ b/jury/metrics/meteor/meteor_for_language_generation.py
@@ -1,5 +1,5 @@
 # coding=utf-8
-# Copyright 2020 Open Business Software Solutions, The HuggingFace Datasets Authors.
+# Copyright 2021 Open Business Software Solutions, The HuggingFace Datasets Authors.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -12,7 +12,8 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-""" METEOR metric. The part of this file is adapted from HuggingFace's
+"""
+METEOR metric. The part of this file is adapted from HuggingFace's
 datasets package implementation of METEOR metric. See
 https://github.com/huggingface/datasets/blob/master/metrics/meteor/meteor.py
 """
diff --git a/jury/metrics/precision/precision_for_language_generation.py b/jury/metrics/precision/precision_for_language_generation.py
index ef797dd..fc7c200 100644
--- a/jury/metrics/precision/precision_for_language_generation.py
+++ b/jury/metrics/precision/precision_for_language_generation.py
@@ -1,5 +1,5 @@
 # coding=utf-8
-# Copyright 2020 The HuggingFace Datasets Authors and the current dataset script contributor.
+# Copyright 2021 The HuggingFace Datasets Authors and the current dataset script contributor.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
diff --git a/jury/metrics/precision/precision_for_sequence_classification.py b/jury/metrics/precision/precision_for_sequence_classification.py
index 4e88aa3..53e0c0f 100644
--- a/jury/metrics/precision/precision_for_sequence_classification.py
+++ b/jury/metrics/precision/precision_for_sequence_classification.py
@@ -1,5 +1,5 @@
 # coding=utf-8
-# Copyright 2020 The HuggingFace Datasets Authors and the current dataset script contributor.
+# Copyright 2021 The HuggingFace Datasets Authors and the current dataset script contributor.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -61,24 +61,21 @@
 Returns:
     precision: Precision score.
 Examples:
-    >>> precision_metric = datasets.load_metric("precision")
-    >>> results = precision_metric.compute(references=[0, 1], predictions=[0, 1])
-    >>> print(results)
-    {'precision': 1.0}
-    >>> predictions = [0, 2, 1, 0, 0, 1]
-    >>> references = [0, 1, 2, 0, 1, 2]
+    >>> precision_metric = jury.load_metric("precision", task="sequence-classification")
+    >>> predictions = [[0], [2], [1], [0], [0], [1]]
+    >>> references = [[0], [1], [2], [0], [1], [2]]
     >>> results = precision_metric.compute(predictions=predictions, references=references, average='macro')
     >>> print(results)
-    {'precision': 0.2222222222222222}
+    {'precision': {'score': 0.2222222222222222}}
     >>> results = precision_metric.compute(predictions=predictions, references=references, average='micro')
     >>> print(results)
-    {'precision': 0.3333333333333333}
+    {'precision': {'score': 0.3333333333333333}}
     >>> results = precision_metric.compute(predictions=predictions, references=references, average='weighted')
     >>> print(results)
-    {'precision': 0.2222222222222222}
+    {'precision': {'score': 0.2222222222222222}}
     >>> results = precision_metric.compute(predictions=predictions, references=references, average=None)
     >>> print(results)
-    {'precision': array([0.66666667, 0.        , 0.        ])}
+    {'precision': {'score': [0.6666666666666666, 0.0, 0.0]}}
 """
 
 _CITATION = """\
diff --git a/jury/metrics/prism/prism_for_language_generation.py b/jury/metrics/prism/prism_for_language_generation.py
index 2e8e377..6ba7b0d 100644
--- a/jury/metrics/prism/prism_for_language_generation.py
+++ b/jury/metrics/prism/prism_for_language_generation.py
@@ -12,9 +12,11 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-""" Prism metric. The part of this file is adapted from metric implementations
+"""
+Prism metric. The part of this file is adapted from metric implementations
 of datasets package. See
-https://github.com/huggingface/datasets/blob/master/metrics/ """
+https://github.com/huggingface/datasets/blob/master/metrics/
+"""
 import os
 from typing import Callable, Dict, List, Union
 
@@ -68,30 +70,29 @@
 Examples:
 
     >>> prism = jury.load_metric("prism")
-    >>> predictions = [
-        ["the cat is on the mat", "There is cat playing on mat"],
-        ["Look! what a wonderful day, today.", "Today is a very wonderful day"],
-    ]
+    >>> predictions = [["the cat is on the mat", "There is cat playing on the mat"], ["Look! a wonderful day."]]
     >>> references = [
-        ["the cat is playing on the mat.", "The cat plays on the mat."],
-        ["Today is a wonderful day", "The weather outside is wonderful."],
+        ["the cat is playing on the mat.", "The cat plays on the mat."], 
+        ["Today is a wonderful day", "The weather outside is wonderful."]
     ]
     >>> results = prism.compute(predictions=predictions, references=references)
     >>> print(results)
     {
-      "score": -1.1489432752132416,
-      "identifier": {
+      "prism": {
+        "score": -2.3071975708007812,
+        "identifier": {
           "version": "0.1",
           "model": "m39v1",
           "seg_scores": "avg_log_prob",
           "sys_scores": "avg_log_prob",
           "log_base": 2,
           "temperature": 1.0
-      },
-      "model_path_or_url": "http://data.statmt.org/prism/m39v1.tar",
-      "lang": "en",
-      "segment_scores": false,
-      "normalized": false
+        },
+        "model_path_or_url": "default",
+        "lang": "en",
+        "segment_scores": false,
+        "normalized": false
+      }
     }
 """
 
diff --git a/jury/metrics/recall/recall_for_language_generation.py b/jury/metrics/recall/recall_for_language_generation.py
index 9f928b7..fc16a31 100644
--- a/jury/metrics/recall/recall_for_language_generation.py
+++ b/jury/metrics/recall/recall_for_language_generation.py
@@ -1,5 +1,5 @@
 # coding=utf-8
-# Copyright 2020 The HuggingFace Datasets Authors and the current dataset script contributor.
+# Copyright 2021 The HuggingFace Datasets Authors and the current dataset script contributor.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
diff --git a/jury/metrics/recall/recall_for_sequence_classification.py b/jury/metrics/recall/recall_for_sequence_classification.py
index d4728b1..2aa46da 100644
--- a/jury/metrics/recall/recall_for_sequence_classification.py
+++ b/jury/metrics/recall/recall_for_sequence_classification.py
@@ -1,5 +1,5 @@
 # coding=utf-8
-# Copyright 2020 The HuggingFace Datasets Authors and the current dataset script contributor.
+# Copyright 2021 The HuggingFace Datasets Authors and the current dataset script contributor.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -25,10 +25,12 @@
 from jury.metrics._core import MetricForSequenceClassification, SequenceClassificationInstance
 
 _DESCRIPTION = """
-Precision is the fraction of the true examples among the predicted examples. It can be computed with:
-Precision = TP / (TP + FP)
-TP: True positive
-FP: False positive
+This metric is derived from Modified Unigram Precision as a recall metric so that
+it will compute across references and not across predictions unlike precision. The computation is similar, however,
+we call this recall since there is no measure called "modified unigram recall".
+Recall is the fraction of the common unigrams between the prediction
+and the references among the reference tokens. It can be computed with:
+Recall = # of matching tokens / # of reference tokens
 """
 
 _KWARGS_DESCRIPTION = """
@@ -61,24 +63,12 @@
 Returns:
     precision: Precision score.
 Examples:
-    >>> precision_metric = datasets.load_metric("precision")
-    >>> results = precision_metric.compute(references=[0, 1], predictions=[0, 1])
+    >>> precision_metric = jury.load_metric("recall", taks="sequence-classificaiton")
+    >>> predictions = [[0], [2], [1], [0], [0], [1]]
+    >>> references = [[0], [1], [2], [0], [1], [2]]
+    >>> results = precision_metric.compute(predictions=predictions, references=references)
     >>> print(results)
-    {'precision': 1.0}
-    >>> predictions = [0, 2, 1, 0, 0, 1]
-    >>> references = [0, 1, 2, 0, 1, 2]
-    >>> results = precision_metric.compute(predictions=predictions, references=references, average='macro')
-    >>> print(results)
-    {'precision': 0.2222222222222222}
-    >>> results = precision_metric.compute(predictions=predictions, references=references, average='micro')
-    >>> print(results)
-    {'precision': 0.3333333333333333}
-    >>> results = precision_metric.compute(predictions=predictions, references=references, average='weighted')
-    >>> print(results)
-    {'precision': 0.2222222222222222}
-    >>> results = precision_metric.compute(predictions=predictions, references=references, average=None)
-    >>> print(results)
-    {'precision': array([0.66666667, 0.        , 0.        ])}
+    {'recall': {'score': 0.3333333333333333}}
 """
 
 _CITATION = """\
diff --git a/jury/metrics/rouge/rouge_for_language_generation.py b/jury/metrics/rouge/rouge_for_language_generation.py
index eaa7990..473f799 100644
--- a/jury/metrics/rouge/rouge_for_language_generation.py
+++ b/jury/metrics/rouge/rouge_for_language_generation.py
@@ -1,5 +1,5 @@
 # coding=utf-8
-# Copyright 2020 Open Business Software Solutions, The HuggingFace Datasets Authors.
+# Copyright 2021 Open Business Software Solutions, The HuggingFace Datasets Authors.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -83,8 +83,14 @@
     ]
     >>> results = rouge.compute(predictions=predictions, references=references)
     >>> print(results)
-    {'rouge': {'rouge1': 0.7783882783882783, 'rouge2': 0.5925324675324675, 
-        'rougeL': 0.7426739926739926, 'rougeLsum': 0.7426739926739926}}
+    {
+      "rouge": {
+        "rouge1": 0.7783882783882783,
+        "rouge2": 0.5925324675324675,
+        "rougeL": 0.7426739926739926,
+        "rougeLsum": 0.7426739926739926
+      }
+    }
 """
 
 
diff --git a/jury/metrics/sacrebleu/sacrebleu_for_language_generation.py b/jury/metrics/sacrebleu/sacrebleu_for_language_generation.py
index 811066c..dcefa6e 100644
--- a/jury/metrics/sacrebleu/sacrebleu_for_language_generation.py
+++ b/jury/metrics/sacrebleu/sacrebleu_for_language_generation.py
@@ -1,5 +1,5 @@
 # coding=utf-8
-# Copyright 2020 Open Business Software Solutions, The HuggingFace Datasets Authors.
+# Copyright 2021 Open Business Software Solutions, The HuggingFace Datasets Authors.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -93,10 +93,38 @@
     ]
     >>> results = sacrebleu.compute(predictions=predictions, references=references)
     >>> print(results)
-    {'sacrebleu': {'score': 0.32377227131456443, 'counts': [11, 6, 3, 0], 'totals': [13, 11, 9, 7],
-         'precisions': [0.8461538461538461, 0.5454545454545454, 0.33333333333333337, 0.07142857142857144], 
-         'bp': 1.0, 'sys_len': 11, 'ref_len': 12, 
-         'adjusted_precisions': [0.8461538461538461, 0.5454545454545454, 0.33333333333333337, 0.07142857142857144]}}
+    {
+      "sacrebleu": {
+        "score": 0.32377227131456443,
+        "counts": [
+          11,
+          6,
+          3,
+          0
+        ],
+        "totals": [
+          13,
+          11,
+          9,
+          7
+        ],
+        "precisions": [
+          0.8461538461538461,
+          0.5454545454545454,
+          0.33333333333333337,
+          0.07142857142857144
+        ],
+        "bp": 1.0,
+        "sys_len": 11,
+        "ref_len": 12,
+        "adjusted_precisions": [
+          0.8461538461538461,
+          0.5454545454545454,
+          0.33333333333333337,
+          0.07142857142857144
+        ]
+      }
+    }
 """
 
 
diff --git a/jury/metrics/seqeval/seqeval_for_sequence_classification.py b/jury/metrics/seqeval/seqeval_for_sequence_classification.py
index 79149ba..111680b 100644
--- a/jury/metrics/seqeval/seqeval_for_sequence_classification.py
+++ b/jury/metrics/seqeval/seqeval_for_sequence_classification.py
@@ -1,5 +1,5 @@
 # coding=utf-8
-# Copyright 2020 Open Business Software Solutions, The HuggingFace Datasets Authors.
+# Copyright 2021 Open Business Software Solutions, The HuggingFace Datasets Authors.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -95,14 +95,31 @@
 
     >>> predictions = [['O', 'O', 'B-MISC', 'I-MISC', 'I-MISC', 'I-MISC', 'O'], ['B-PER', 'I-PER', 'O']]
     >>> references = [['O', 'O', 'O', 'B-MISC', 'I-MISC', 'I-MISC', 'O'], ['B-PER', 'I-PER', 'O']]
-    >>> seqeval = datasets.load_metric("seqeval")
+    >>> seqeval = jury.load_metric("seqeval")
     >>> results = seqeval.compute(predictions=predictions, references=references)
     >>> print(list(results.keys()))
     ['MISC', 'PER', 'overall_precision', 'overall_recall', 'overall_f1', 'overall_accuracy']
-    >>> print(results["overall_f1"])
-    0.5
-    >>> print(results["PER"]["f1"])
-    1.0
+    >>> print(results)
+    {
+      "seqeval": {
+        "MISC": {
+          "precision": 0.0,
+          "recall": 0.0,
+          "f1": 0.0,
+          "number": 1
+        },
+        "PER": {
+          "precision": 1.0,
+          "recall": 1.0,
+          "f1": 1.0,
+          "number": 1
+        },
+        "overall_precision": 0.5,
+        "overall_recall": 0.5,
+        "overall_f1": 0.5,
+        "overall_accuracy": 0.8
+      }
+    }
 """
 
 
diff --git a/jury/metrics/squad/squad_for_language_generation.py b/jury/metrics/squad/squad_for_language_generation.py
index 0c73063..d9a7044 100644
--- a/jury/metrics/squad/squad_for_language_generation.py
+++ b/jury/metrics/squad/squad_for_language_generation.py
@@ -1,5 +1,5 @@
 # coding=utf-8
-# Copyright 2020 Open Business Software Solutions, The HuggingFace Datasets Authors.
+# Copyright 2021 Open Business Software Solutions, The HuggingFace Datasets Authors.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
diff --git a/jury/metrics/ter/ter_for_language_generation.py b/jury/metrics/ter/ter_for_language_generation.py
index 1dc7986..c26a26c 100644
--- a/jury/metrics/ter/ter_for_language_generation.py
+++ b/jury/metrics/ter/ter_for_language_generation.py
@@ -1,5 +1,5 @@
 # coding=utf-8
-# Copyright 2020 Open Business Software Solutions, The HuggingFace Datasets Authors.
+# Copyright 2021 Open Business Software Solutions, The HuggingFace Datasets Authors.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -12,7 +12,8 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-""" Translation Edit Rate (TER) metric. The part of this file is adapted from HuggingFace's
+"""
+Translation Edit Rate (TER) metric. The part of this file is adapted from HuggingFace's
 datasets package implementation of TER metric. See
 https://github.com/huggingface/datasets/blob/master/metrics/ter/ter.py
 """
@@ -82,12 +83,15 @@
     'num_edits': The cumulative number of edits,
     'ref_length': The cumulative average reference length.
 Examples:
-    >>> predictions = [["hello there general kenobi"], ["foo bar foobar"]]
-    >>> references = [["hello there general kenobi", "hello there !"], ["foo bar foobar", "foo bar foobar"]]
+    >>> predictions = [["the cat is on the mat", "There is cat playing on the mat"], ["Look! a wonderful day."]]
+    >>> references = [
+        ["the cat is playing on the mat.", "The cat plays on the mat."], 
+        ["Today is a wonderful day", "The weather outside is wonderful."]
+    ]
     >>> ter = jury.load_metric("ter")
     >>> results = ter.compute(predictions=predictions, references=references)
     >>> print(results)
-    {'score': 0.0, 'num_edits': 0, 'ref_length': 6.5}
+    {'ter': {'score': 0.5307692307692308, 'avg_num_edits': 2.75, 'avg_ref_length': 5.75}}
 """
 
 
diff --git a/jury/metrics/wer/wer_for_language_generation.py b/jury/metrics/wer/wer_for_language_generation.py
index c4360ed..154f339 100644
--- a/jury/metrics/wer/wer_for_language_generation.py
+++ b/jury/metrics/wer/wer_for_language_generation.py
@@ -1,5 +1,5 @@
 # coding=utf-8
-# Copyright 2020 Open Business Software Solutions, The HuggingFace Datasets Authors.
+# Copyright 2021 Open Business Software Solutions, The HuggingFace Datasets Authors.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -63,18 +63,23 @@
 Returns:
     (float): the word error rate
 Examples:
-    >>> predictions = ["this is the prediction", "there is an other sample"]
-    >>> references = ["this is the reference", "there is another one"]
+    >>> predictions = [["the cat is on the mat", "There is cat playing on the mat"], ["Look! a wonderful day."]]
+    >>> references = [
+        ["the cat is playing on the mat.", "The cat plays on the mat."], 
+        ["Today is a wonderful day", "The weather outside is wonderful."]
+    ]
     >>> wer = jury.load_metric("wer")
     >>> wer_score = wer.compute(predictions=predictions, references=references)
     >>> print(wer_score)
     {
-      "score": 0.5,
-      "overall": {
-        "substitutions": 3,
-        "deletions": 0,
-        "insertions": 1,
-        "hits": 5
+      "wer": {
+        "score": 1.0,
+        "overall": {
+          "substitutions": 2.8333333333333335,
+          "deletions": 0.5,
+          "insertions": 0.16666666666666666,
+          "hits": 2.6666666666666665
+        }
       }
     }
 """
diff --git a/jury/tokenizer.py b/jury/tokenizer.py
index cb36cf4..9bdb645 100644
--- a/jury/tokenizer.py
+++ b/jury/tokenizer.py
@@ -24,8 +24,11 @@ def __init__(self, tokenizer):
     def tokenize(self, predictions: Collator, references: Collator) -> Tuple[Collator, Collator]:
         _predictions = []
         _references = []
-        for preds in predictions:
+        i = 0
+        while i < len(predictions):
+            preds = predictions[i]
+            refs = references[i]
             _predictions.append([self.tokenizer.tokenize(pred) for pred in preds])
-        for refs in references:
             _references.append([self.tokenizer.tokenize(ref) for ref in refs])
+            i += 1
         return Collator(_predictions, keep=True), Collator(_references, keep=True)
diff --git a/requirements.txt b/requirements.txt
index a3a0ca0..29e7303 100755
--- a/requirements.txt
+++ b/requirements.txt
@@ -1,4 +1,4 @@
-datasets>=1.18.3
+datasets>=2.0.0
 fire>=0.4.0
 nltk>=3.6.6,<3.7.1
 numpy>=1.21.0


Test patch:
diff --git a/tests/jury/metrics/test_bartscore.py b/tests/jury/metrics/test_bartscore.py
index edf616e..df1cc83 100644
--- a/tests/jury/metrics/test_bartscore.py
+++ b/tests/jury/metrics/test_bartscore.py
@@ -12,12 +12,6 @@ def jury_bartscore():
     return Jury(metrics=metric)
 
 
-@pytest.fixture(scope="module")
-def jury_bartscore_segment():
-    metric = AutoMetric.load("bartscore", device="cpu", compute_kwargs={"segment_scores": True})
-    return Jury(metrics=metric)
-
-
 @pytest.fixture
 @get_expected_output(prefix="metrics")
 def output_basic():
@@ -47,8 +41,8 @@ def test_basic(predictions, references, jury_bartscore, output_basic):
     assert_almost_equal_dict(actual=scores, desired=output_basic)
 
 
-def test_basic_segment(predictions, references, jury_bartscore_segment, output_basic_segment):
-    scores = jury_bartscore_segment(predictions=predictions, references=references)
+def test_basic_segment(predictions, references, jury_bartscore, output_basic_segment):
+    scores = jury_bartscore(predictions=predictions, references=references, segment_scores=True)
     assert_almost_equal_dict(actual=scores, desired=output_basic_segment)
 
 
diff --git a/tests/jury/metrics/test_cer.py b/tests/jury/metrics/test_cer.py
index 379a69c..c7785e5 100644
--- a/tests/jury/metrics/test_cer.py
+++ b/tests/jury/metrics/test_cer.py
@@ -12,12 +12,6 @@ def jury_cer():
     return Jury(metrics=metric)
 
 
-@pytest.fixture(scope="module")
-def jury_cer_concat():
-    metric = AutoMetric.load("cer", compute_kwargs={"concatenate_texts": True})
-    return Jury(metrics=metric)
-
-
 @pytest.fixture
 @get_expected_output(prefix="metrics")
 def output_basic():
@@ -47,8 +41,8 @@ def test_basic(predictions, references, jury_cer, output_basic):
     assert_almost_equal_dict(actual=scores, desired=output_basic)
 
 
-def test_basic_concat(predictions, references, jury_cer_concat, output_basic_concat):
-    scores = jury_cer_concat(predictions=predictions, references=references)
+def test_basic_concat(predictions, references, jury_cer, output_basic_concat):
+    scores = jury_cer(predictions=predictions, references=references, concatenate_texts=True)
     assert_almost_equal_dict(actual=scores, desired=output_basic_concat)
 
 
diff --git a/tests/jury/metrics/test_wer.py b/tests/jury/metrics/test_wer.py
index 966d5c5..6924a66 100644
--- a/tests/jury/metrics/test_wer.py
+++ b/tests/jury/metrics/test_wer.py
@@ -12,12 +12,6 @@ def jury_wer():
     return Jury(metrics=metric)
 
 
-@pytest.fixture(scope="module")
-def jury_wer_concat():
-    metric = AutoMetric.load("wer", compute_kwargs={"concatenate_texts": True})
-    return Jury(metrics=metric)
-
-
 @pytest.fixture
 @get_expected_output(prefix="metrics")
 def output_basic():
@@ -47,8 +41,8 @@ def test_basic(predictions, references, jury_wer, output_basic):
     assert_almost_equal_dict(actual=scores, desired=output_basic)
 
 
-def test_basic_concat(predictions, references, jury_wer_concat, output_basic_concat):
-    scores = jury_wer_concat(predictions=predictions, references=references)
+def test_basic_concat(predictions, references, jury_wer, output_basic_concat):
+    scores = jury_wer(predictions=predictions, references=references, concatenate_texts=True)
     assert_almost_equal_dict(actual=scores, desired=output_basic_concat)