Repository: obss/jury (language: python)

Issue:
Adopt arrow implementation from datasets
Metric implementation with arrow tables should be adopted for multiple references & multiple predictions cases.


Discussion / hints:

Fix patch:
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index fc5ea27..21a20e2 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -53,7 +53,7 @@ jobs:
 
       - name: Install dependencies
         run: >
-          pip install -e .[develop]
+          pip install -e .[dev]
 
       - name: Lint with flake8 and black
         run: |
diff --git a/README.md b/README.md
index 7048357..45bfaaf 100644
--- a/README.md
+++ b/README.md
@@ -43,19 +43,64 @@ or build from source,
 
 It is only two lines of code to evaluate generated outputs.
 
-    from jury import Jury
-    
-    jury = Jury()
-
-    # Microsoft translator translation for "Yurtta sulh, cihanda sulh." (16.07.2021)
-    predictions = ["Peace in the dormitory, peace in the world."]
-    references = ["Peace at home, peace in the world."]
-    scores = jury.evaluate(predictions, references)
+```python
+from jury import Jury
+
+scorer = Jury()
+predictions = [
+    ["the cat is on the mat", "There is cat playing on the mat"], 
+    ["Look!    a wonderful day."]
+]
+references = [
+    ["the cat is playing on the mat.", "The cat plays on the mat."], 
+    ["Today is a wonderful day", "The weather outside is wonderful."]
+]
+scores = scorer(predictions=predictions, references=references)
+```
 
 Specify metrics you want to use on instantiation.
 
-    jury = Jury(metrics=["bleu", "meteor"])
-    scores = jury.evaluate(predictions, references)
+```python
+scorer = Jury(metrics=["bleu", "meteor"])
+scores = scorer(predictions, references)
+```
+
+#### Use of Metrics standalone
+
+You can directly import metrics from `jury.metrics` as classes, and then instantiate and use as desired.
+
+```python
+from jury.metrics import Bleu
+
+bleu = Bleu()
+score = bleu.compute(predictions=predictions, references=references)
+```
+
+The additional parameters can either be specified on `compute()`
+
+```python
+from jury.metrics import Bleu
+
+bleu = Bleu()
+score = bleu.compute(predictions=predictions, references=references, max_order=4)
+```
+
+, or alternatively on instantiation
+
+```python
+bleu = Bleu(params={"max_order": 1})
+```
+
+Note that you can seemlessly access both `jury` and `datasets` metrics through `jury.load_metric`. 
+
+```python
+import jury
+
+bleu = jury.load_metric("bleu")
+bleu_1 = jury.load_metric("bleu", resulting_name="bleu_1", params={"max_order": 1})
+# metrics not available in `jury` but in `datasets`
+wer = jury.load_metric("wer") # It falls back to `datasets` package with a warning
+```
 
 ### CLI Usage
 
@@ -83,13 +128,36 @@ Then, you can call jury eval with `config` argument.
 
 ### Custom Metrics
 
-You can use custom metrics with inheriting `jury.metrics.Metric`, you can see current metrics on [datasets/metrics](https://github.com/huggingface/datasets/tree/master/metrics). The code snippet below gives a brief explanation.
+You can use custom metrics with inheriting `jury.metrics.Metric`, you can see current metrics implemented on Jury from [jury/metrics](https://github.com/obss/jury/tree/master/jury/metrics). Jury falls back to `datasets` implementation of metrics for the ones that are currently not supported by Jury, you can see the metrics available for `datasets` on [datasets/metrics](https://github.com/huggingface/datasets/tree/master/metrics). 
+
+Jury itself uses `datasets.Metric` as a base class to drive its own base class as `jury.metrics.Metric`. The interface is similar; however, Jury makes the metrics to take a unified input type by handling the inputs for each metrics, and allows supporting several input types as;
+
+- single prediction & single reference
+- single prediction & multiple reference
+- multiple prediction & multiple reference
 
+As a custom metric both base classes can be used; however, we strongly recommend using `jury.metrics.Metric` as it has several advantages such as supporting computations for the input types above or unifying the type of the input.
+
+```python
     from jury.metrics import Metric
+    
+    class CustomMetric(Metric):
+            def _compute_single_pred_single_ref(
+        self, predictions: Collator, references: Collator, reduce_fn: Callable = None, **kwargs
+    ):
+        raise NotImplementedError
+
+    def _compute_single_pred_multi_ref(
+        self, predictions: Collator, references: Collator, reduce_fn: Callable, **kwargs
+    ):
+        raise NotImplementedError
+
+    def _compute_multi_pred_multi_ref(self, predictions: Collator, references: Collator, reduce_fn: Callable, **kwargs
+    ):
+        raise NotImplementedError
+```
 
-    CustomMetric(Metric):
-        def compute(self, predictions, references):
-            pass
+For more details, have a look at base metric implementation [jury.metrics.Metric](./jury/metrics/_base.py)
 
 ## <div align="center"> Contributing </div>
 
@@ -99,7 +167,7 @@ PRs are welcomed as always :)
 
     git clone https://github.com/obss/jury.git
     cd jury
-    pip install -e .[develop]
+    pip install -e .[dev]
 
 ### Tests
 
diff --git a/examples/jury_evaluate.ipynb b/examples/jury_evaluate.ipynb
index 0edeafb..a5d9f0e 100644
--- a/examples/jury_evaluate.ipynb
+++ b/examples/jury_evaluate.ipynb
@@ -54,11 +54,11 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 1,
+   "execution_count": null,
    "metadata": {
     "ExecuteTime": {
-     "end_time": "2021-08-02T22:49:55.831703Z",
-     "start_time": "2021-08-02T22:49:55.817682Z"
+     "end_time": "2021-10-02T19:59:22.515146Z",
+     "start_time": "2021-10-02T19:59:20.716835Z"
     }
    },
    "outputs": [],
@@ -66,14 +66,18 @@
     "import os\n",
     "import json # Just for pretty printing the resulting dict.\n",
     "\n",
-    "from jury import Jury\n",
-    "from jury.metrics import load_metric"
+    "from jury import load_metric, Jury"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 2,
-   "metadata": {},
+   "execution_count": null,
+   "metadata": {
+    "ExecuteTime": {
+     "end_time": "2021-10-02T19:59:22.531153Z",
+     "start_time": "2021-10-02T19:59:22.516139Z"
+    }
+   },
    "outputs": [],
    "source": [
     "from typing import List\n",
@@ -96,11 +100,11 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 3,
+   "execution_count": null,
    "metadata": {
     "ExecuteTime": {
-     "end_time": "2021-08-02T22:49:26.488739Z",
-     "start_time": "2021-08-02T22:49:26.473482Z"
+     "end_time": "2021-10-02T19:59:22.547167Z",
+     "start_time": "2021-10-02T19:59:22.532153Z"
     }
    },
    "outputs": [],
@@ -133,14 +137,19 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 4,
-   "metadata": {},
+   "execution_count": null,
+   "metadata": {
+    "ExecuteTime": {
+     "end_time": "2021-10-02T19:59:22.562180Z",
+     "start_time": "2021-10-02T19:59:22.548168Z"
+    }
+   },
    "outputs": [],
    "source": [
     "MT_METRICS = [\n",
-    "    load_metric(metric_name=\"bleu\", resulting_name=\"bleu_1\", params={\"max_order\": 1}),\n",
-    "    load_metric(metric_name=\"bleu\", resulting_name=\"bleu_2\", params={\"max_order\": 2}),\n",
-    "    load_metric(\"meteor\", \"meteor\"),\n",
+    "    load_metric(\"bleu\", resulting_name=\"bleu_1\", params={\"max_order\": 1}),\n",
+    "    load_metric(\"bleu\", resulting_name=\"bleu_2\", params={\"max_order\": 2}),\n",
+    "    load_metric(\"meteor\"),\n",
     "    load_metric(\"rouge\"),\n",
     "#     load_metric(\"sacrebleu\"),  # (optional)\n",
     "#     load_metric(\"bertscore\"), # (optional) Using default model for lang en\n",
@@ -159,38 +168,15 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 5,
+   "execution_count": null,
    "metadata": {
     "ExecuteTime": {
-     "end_time": "2021-08-02T22:51:23.394106Z",
-     "start_time": "2021-08-02T22:50:52.018613Z"
+     "end_time": "2021-10-02T19:59:26.664594Z",
+     "start_time": "2021-10-02T19:59:22.563182Z"
     },
     "scrolled": true
    },
-   "outputs": [
-    {
-     "name": "stderr",
-     "output_type": "stream",
-     "text": [
-      "100%|██████████████████████████████████████████| 2/2 [00:00<00:00, 58254.22it/s]\n",
-      "  0%|                                                     | 0/2 [00:00<?, ?it/s][nltk_data] Downloading package wordnet to\n",
-      "[nltk_data]     /home/devrimcavusoglu/nltk_data...\n",
-      "[nltk_data]   Package wordnet is already up-to-date!\n",
-      "100%|█████████████████████████████████████████████| 2/2 [00:03<00:00,  1.90s/it]\n",
-      "100%|█████████████████████████████████████████████| 2/2 [00:04<00:00,  2.24s/it]\n",
-      "[nltk_data] Downloading package wordnet to\n",
-      "[nltk_data]     /home/devrimcavusoglu/nltk_data...\n",
-      "[nltk_data]   Package wordnet is already up-to-date!\n",
-      "100%|█████████████████████████████████████████████| 2/2 [00:05<00:00,  2.64s/it]\n",
-      "100%|█████████████████████████████████████████████| 2/2 [00:05<00:00,  2.67s/it]\n",
-      "[nltk_data] Downloading package wordnet to\n",
-      "[nltk_data]     /home/devrimcavusoglu/nltk_data...\n",
-      "[nltk_data]   Package wordnet is already up-to-date!\n",
-      "100%|█████████████████████████████████████████████| 2/2 [00:05<00:00,  2.84s/it]\n",
-      "100%|█████████████████████████████████████████████| 2/2 [00:29<00:00, 14.90s/it]\n"
-     ]
-    }
-   ],
+   "outputs": [],
    "source": [
     "# Compute scores\n",
     "\n",
@@ -200,33 +186,20 @@
     "    os.environ[\"TOKENIZERS_PARALLELISM\"] = \"false\"\n",
     "\n",
     "mt_jury = Jury(metrics=MT_METRICS, run_concurrent=RUN_CONCURRENT)\n",
-    "scores = mt_jury.evaluate(predictions=mt_predictions, references=mt_references)"
+    "scores = mt_jury(predictions=mt_predictions, references=mt_references)"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 6,
+   "execution_count": null,
    "metadata": {
+    "ExecuteTime": {
+     "end_time": "2021-10-02T19:59:26.679608Z",
+     "start_time": "2021-10-02T19:59:26.665594Z"
+    },
     "scrolled": true
    },
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "{\n",
-      "    \"empty_predictions\": 0,\n",
-      "    \"total_items\": 2,\n",
-      "    \"bleu_1\": 0.7920502936517768,\n",
-      "    \"bleu_2\": 0.7225612529515497,\n",
-      "    \"meteor\": 0.5420511682934044,\n",
-      "    \"rougeL\": 0.7948717948717947,\n",
-      "    \"SacreBLEU\": 0.3898310279399514,\n",
-      "    \"BERTScore\": 0.7431023120880127\n",
-      "}\n"
-     ]
-    }
-   ],
+   "outputs": [],
    "source": [
     "# Display results\n",
     "print(json.dumps(scores, indent=4))"
@@ -243,8 +216,13 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 7,
-   "metadata": {},
+   "execution_count": null,
+   "metadata": {
+    "ExecuteTime": {
+     "end_time": "2021-10-02T19:59:29.552689Z",
+     "start_time": "2021-10-02T19:59:29.540678Z"
+    }
+   },
    "outputs": [],
    "source": [
     "qa_predictions = [\"1917\", \"Albert Einstein\", \"foo bar\"]\n",
@@ -257,33 +235,18 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 8,
+   "execution_count": null,
    "metadata": {
+    "ExecuteTime": {
+     "end_time": "2021-10-02T19:59:30.130953Z",
+     "start_time": "2021-10-02T19:59:30.117949Z"
+    },
     "scrolled": true
    },
-   "outputs": [
-    {
-     "name": "stderr",
-     "output_type": "stream",
-     "text": [
-      "100%|██████████████████████████████████████████| 3/3 [00:00<00:00, 79137.81it/s]\n"
-     ]
-    },
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "{\n",
-      "    \"empty_predictions\": 0,\n",
-      "    \"total_items\": 3,\n",
-      "    \"SQUAD\": 0.8222222222222223\n",
-      "}\n"
-     ]
-    }
-   ],
+   "outputs": [],
    "source": [
     "qa_jury = Jury(metrics=QA_METRICS, run_concurrent=False)\n",
-    "scores = qa_jury.evaluate(predictions=qa_predictions, references=qa_references)\n",
+    "scores = qa_jury(predictions=qa_predictions, references=qa_references)\n",
     "print(json.dumps(scores, indent=4))"
    ]
   },
@@ -298,100 +261,135 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 9,
-   "metadata": {},
+   "execution_count": null,
+   "metadata": {
+    "ExecuteTime": {
+     "end_time": "2021-10-02T20:17:28.152162Z",
+     "start_time": "2021-10-02T20:17:28.141152Z"
+    }
+   },
    "outputs": [],
    "source": [
     "from collections import Counter\n",
-    "from typing import Dict\n",
+    "from typing import Callable\n",
     "\n",
+    "import datasets\n",
+    "from jury.collator import Collator\n",
     "from jury.metrics import Metric\n",
+    "from jury.metrics._utils import normalize_text\n",
     "\n",
     "\n",
-    "class Precision(Metric):\n",
+    "class WordMatch(Metric):\n",
     "    \"\"\"\n",
-    "    Compute simple precision as \n",
-    "        Average( # of matching tokens / # of tokens in prediction )\n",
+    "    Compute matching word ratio between prediction and reference \n",
+    "        Average( # of matching tokens / # of unique tokens)\n",
+    "    \n",
+    "    Example:\n",
+    "        pred = [\"foo bar foobar\"]\n",
+    "        ref = [\"foo bar barfoo\"]\n",
+    "        matching_tokens = 2 # 'foo' and 'bar'\n",
+    "        total_tokens = 4 # 'foo', 'bar', 'foobar', and 'barfoo'\n",
+    "        score = 0.5 # (2/4)\n",
     "    \"\"\"\n",
-    "    def __init__(self, metric_name: str = None, resulting_name: str = None, params: Dict = None):\n",
-    "        metric_name = self.__class__.__name__ if metric_name is None else metric_name\n",
-    "        resulting_name = metric_name if resulting_name is None else resulting_name\n",
-    "        super().__init__(metric_name=metric_name, resulting_name=resulting_name, params=params)\n",
-    "        \n",
-    "    def _preprocess(self, predictions, references):\n",
-    "        predictions = [p.split() for p in predictions]\n",
-    "        references = [r.split() for r in references]\n",
+    "    def _info(self):\n",
+    "        return datasets.MetricInfo(\n",
+    "            description=\"Custom metric to compute ratio of matching tokens over unique tokens.\",\n",
+    "            citation=\"Custom metric\",\n",
+    "            inputs_description=\"\",\n",
+    "            features=self.default_features,\n",
+    "        )\n",
+    "    \n",
+    "    def _tokenize(self, predictions: Collator, references: Collator):\n",
+    "        predictions = [normalize_text(p).split() for p in predictions]\n",
+    "        references = [normalize_text(r).split() for r in references]\n",
     "        return predictions, references\n",
     "    \n",
-    "    def _compute(self, predictions, references):\n",
+    "    def _compute_single_pred_single_ref(\n",
+    "        self, predictions: Collator, references: Collator, reduce_fn: Callable = None, **kwargs\n",
+    "    ):\n",
     "        scores = []\n",
+    "        predictions, references = self._tokenize(predictions, references)\n",
     "        for pred, ref in zip(predictions, references):\n",
     "            score = 0\n",
     "            pred_counts = Counter(pred)\n",
     "            ref_counts = Counter(ref)\n",
+    "            total_unique_tokens = len(pred_counts + ref_counts)\n",
     "            for token, pred_count in pred_counts.items():\n",
     "                if token in ref_counts:\n",
     "                    score += min(pred_count, ref_counts[token])  # Intersection count\n",
-    "            scores.append(score /  len(pred))\n",
+    "            score = score / total_unique_tokens\n",
+    "            scores.append(score)\n",
     "        avg_score = sum(scores) / len(scores)\n",
-    "        return {self.resulting_name: avg_score}\n",
-    "    \n",
-    "    def compute(self, predictions, references) -> Dict[str, float]:\n",
-    "        \"\"\"Required to be used by jury.\"\"\"\n",
-    "        predictions, references = self._preprocess(predictions, references)\n",
-    "        return self._compute(predictions, references)\n",
-    "        "
+    "        return {\"score\": avg_score}\n",
+    "\n",
+    "    def _compute_single_pred_multi_ref(\n",
+    "        self, predictions: Collator, references: Collator, reduce_fn: Callable, **kwargs\n",
+    "    ):\n",
+    "        scores = []\n",
+    "        for pred, refs in zip(predictions, references):\n",
+    "            pred_score = [\n",
+    "                self._compute_single_pred_single_ref(Collator([pred], keep=True), Collator([ref], keep=True))\n",
+    "                for ref in refs\n",
+    "            ]\n",
+    "            reduced_score = self._reduce_scores(pred_score, reduce_fn=reduce_fn)\n",
+    "            scores.append(reduced_score)\n",
+    "\n",
+    "        return self._reduce_scores(scores, reduce_fn=np.mean)\n",
+    "\n",
+    "    def _compute_multi_pred_multi_ref(self, predictions: Collator, references: Collator, reduce_fn: Callable, **kwargs):\n",
+    "        scores = []\n",
+    "        for preds, refs in zip(predictions, references):\n",
+    "            pred_scores = []\n",
+    "            for pred in preds:\n",
+    "                pred_score = self._compute_single_pred_multi_ref(\n",
+    "                    Collator([pred], keep=True), Collator([refs], keep=True), reduce_fn=reduce_fn\n",
+    "                )\n",
+    "                pred_scores.append(pred_score)\n",
+    "            reduced_score = self._reduce_scores(pred_scores, reduce_fn=reduce_fn)\n",
+    "            scores.append(reduced_score)\n",
+    "\n",
+    "        return self._reduce_scores(scores, reduce_fn=np.mean)"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 11,
-   "metadata": {},
+   "execution_count": null,
+   "metadata": {
+    "ExecuteTime": {
+     "end_time": "2021-10-02T20:17:28.353117Z",
+     "start_time": "2021-10-02T20:17:28.348113Z"
+    }
+   },
    "outputs": [],
    "source": [
-    "from jury.metrics.squad import SQUAD\n",
+    "from jury.metrics import Squad\n",
     "\n",
     "QA_METRICS = [\n",
-    "    load_metric(\"squad\"),\n",
-    "    Precision()\n",
+    "    Squad(),\n",
+    "    WordMatch()\n",
     "]"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 12,
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stderr",
-     "output_type": "stream",
-     "text": [
-      "100%|██████████████████████████████████████████| 3/3 [00:00<00:00, 75800.67it/s]\n"
-     ]
-    },
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "{\n",
-      "    \"empty_predictions\": 0,\n",
-      "    \"total_items\": 3,\n",
-      "    \"SQUAD\": 0.8222222222222223,\n",
-      "    \"Precision\": 0.8333333333333334\n",
-      "}\n"
-     ]
+   "execution_count": null,
+   "metadata": {
+    "ExecuteTime": {
+     "end_time": "2021-10-02T20:17:28.583593Z",
+     "start_time": "2021-10-02T20:17:28.556577Z"
     }
-   ],
+   },
+   "outputs": [],
    "source": [
     "qa_jury = Jury(metrics=QA_METRICS, run_concurrent=False)\n",
-    "scores = qa_jury.evaluate(predictions=qa_predictions, references=qa_references)\n",
+    "scores = qa_jury(predictions=qa_predictions, references=qa_references)\n",
     "print(json.dumps(scores, indent=4))"
    ]
   }
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "Python 3",
+   "display_name": "Python 3 (ipykernel)",
    "language": "python",
    "name": "python3"
   },
@@ -405,7 +403,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.8.5"
+   "version": "3.8.0"
   },
   "toc": {
    "base_numbering": 1,
@@ -423,4 +421,4 @@
  },
  "nbformat": 4,
  "nbformat_minor": 4
-}
+}
\ No newline at end of file
diff --git a/jury/__init__.py b/jury/__init__.py
index 6408101..3325a10 100644
--- a/jury/__init__.py
+++ b/jury/__init__.py
@@ -1,10 +1,9 @@
 import os
 
 from jury.core import Jury
+from jury.metrics import load_metric
 
 SOURCE_DIR = os.path.abspath(os.path.dirname(__file__))
 PROJECT_ROOT = os.path.dirname(SOURCE_DIR)
-TEST_DIR = os.path.join(PROJECT_ROOT, "tests")
-TEST_DATA_DIR = os.path.join(TEST_DIR, "test_data")
 
-__version__ = "1.1.2"
+__version__ = "2.0.0"
diff --git a/jury/cli.py b/jury/cli.py
index 5791158..f99d0f5 100644
--- a/jury/cli.py
+++ b/jury/cli.py
@@ -40,7 +40,7 @@ def from_file(predictions: str, references: str, reduce_fn: Optional[str] = None
     predictions = read_file(predictions)
     references = read_file(references)
     jury = Jury(metrics=metrics)
-    scores = jury.evaluate(predictions=predictions, references=references, reduce_fn=reduce_fn)
+    scores = jury(predictions=predictions, references=references, reduce_fn=reduce_fn)
     print(json.dumps(scores, default=str, indent=4))
 
 
diff --git a/jury/collator.py b/jury/collator.py
index f7b78d0..d7a9965 100644
--- a/jury/collator.py
+++ b/jury/collator.py
@@ -1,8 +1,7 @@
-from typing import Dict, List, Union
+from typing import List, Union
 
 import numpy as np
 
-from jury.metrics import Metric, load_metric
 from jury.utils import NestedSingleType
 
 
@@ -27,6 +26,10 @@ def nested(self):
 
     def reshape(self, *args):
         _seq = np.array(self, dtype=object)
+        if _seq.shape[:2] == (1, 1):
+            return Collator(_seq.ravel().reshape(1, -1).tolist(), keep=True)
+        elif _seq.ndim == 3 and _seq.shape[1] == 1:
+            args = tuple(list(args) + [-1])
         return Collator(_seq.reshape(args).tolist(), keep=True)
 
     def reshape_len(self, *args):
@@ -34,10 +37,12 @@ def reshape_len(self, *args):
         return self.reshape(_len, *args)
 
     def can_collapse(self):
-        if self.ndim != 2:
-            return False
-
-        return self.shape[1] == 1
+        if self.ndim >= 2:
+            return self.shape[1] == 1
+        if isinstance(self[0], list):
+            n_item = len(self[0])
+            return all([len(items) == n_item for items in self])
+        return True
 
     def to_list(self, collapse=True):
         if collapse:
@@ -63,29 +68,3 @@ def from_list_of_str(seq: List[str]):
     @classmethod
     def from_str(cls, seq: str):
         return [seq]
-
-
-class MetricCollator(list):
-    def __init__(self, metrics: Union[List[str], List[Metric]]):
-        metrics = self._constructor(metrics)
-        super(MetricCollator, self).__init__(metrics)
-
-    def _constructor(self, metrics):
-        _type = NestedSingleType.get_type(metrics)
-        if _type == "list<str>":
-            _metrics = []
-            for metric in metrics:
-                _metrics.append(load_metric(metric))
-            metrics = _metrics
-        return metrics
-
-    def add_metric(self, metric_name: str, resulting_name: str = None, params: Dict = None):
-        metric = load_metric(metric_name, resulting_name=resulting_name, params=params)
-        self.append(metric)
-
-    def remove_metric(self, resulting_name: str):
-        for i, metric in enumerate(self):
-            if metric.resulting_name == resulting_name:
-                self.pop(i)
-                break
-        raise ValueError(f"Metric with resulting name {resulting_name} does not exists.")
diff --git a/jury/core.py b/jury/core.py
index be85067..84aa784 100644
--- a/jury/core.py
+++ b/jury/core.py
@@ -1,13 +1,12 @@
 from concurrent.futures import ProcessPoolExecutor
-from typing import Callable, Dict, List, Mapping, Optional, Union
+from typing import Any, Callable, Dict, List, Mapping, Optional, Union
 
-import numpy as np
-from tqdm import tqdm
-
-from jury.collator import Collator, MetricCollator
+from jury.collator import Collator
 from jury.definitions import DEFAULT_METRICS
 from jury.metrics import Metric, load_metric
-from jury.utils import is_reduce_fn, replace, set_env
+from jury.utils import replace, set_env
+
+MetricParam = Union[str, Metric, Dict[str, Any]]
 
 
 class Jury:
@@ -34,118 +33,125 @@ class Jury:
             ["Today is a wonderful day", "The weather outside is wonderful."]
         ]
         >>> evaluation = Jury()
-        >>> results = evaluation.evaluate(predictions=predictions, references=references)
+        >>> results = evaluation(predictions=predictions, references=references)
         >>> print(results)
         {'bleu_1': 0.6111111111111112, ..., 'rougeL': 0.6470588235294118, ...}
     """
 
     def __init__(
         self,
-        metrics: Optional[Union[List[str], List[Metric]]] = None,
+        metrics: Optional[Union[MetricParam, List[MetricParam]]] = None,
         run_concurrent=False,
     ):
-        if metrics is None:
-            metrics = DEFAULT_METRICS
-        else:
-            for i, m in enumerate(metrics):
-                if isinstance(m, str):
-                    metrics = replace(metrics, load_metric(m.lower()), i)
-
-        self.metrics = MetricCollator(metrics)
+        self.metrics = self._load_metrics(metrics)
         self._concurrent = run_concurrent
 
-    @staticmethod
-    def _compute_metric_for_multiple_items(
-        metric: Metric, predictions: Collator, references: Collator, reduce_fn
+    def __call__(
+        self,
+        *,
+        predictions: Union[List[str], List[List[str]]] = None,
+        references: Union[List[str], List[List[str]]] = None,
+        reduce_fn: Optional[Union[str, Callable]] = None,
     ) -> Dict[str, float]:
-        scores = []
+        """Restricts positional arguments to prevent potential inconsistency between predictions and references."""
+        if predictions is None or references is None:
+            raise TypeError("Both predictions and references have to be passed.")
+        if len(predictions) != len(references):
+            raise ValueError("Lengths of predictions and references must be equal.")
+
+        scores = dict()
+        scores["empty_predictions"] = len([1 for p in predictions if not p])
+        scores["total_items"] = len(references)
 
-        for hyps, refs in tqdm(zip(predictions, references), total=len(references)):
-            if len(hyps) == 0:
-                # scores.append(0)  # Penalize for not generating any question
+        if self._concurrent:
+            inputs_list = self._prepare_concurrent_inputs(predictions, references, reduce_fn)
+            set_env("TOKENIZERS_PARALLELISM", "true")
+            with ProcessPoolExecutor() as executor:
+                for score in executor.map(self._compute_single_score, inputs_list):
+                    scores.update(score)
+        else:
+            for metric in self.metrics:
+                inputs = (metric, predictions, references, reduce_fn)
+                score = self._compute_single_score(inputs)
+                scores.update(score)
+
+        return scores
+
+    def _load_single_metric(self, metric: Union[str, Metric]) -> List[Metric]:
+        if isinstance(metric, str):
+            metric = load_metric(metric)
+        return [metric]
+
+    def _load_multiple_metrics(self, metrics: Union[List[str], List[Dict[str, Any]], List[Metric]]) -> List[Metric]:
+        for i, metric_param in enumerate(metrics):
+            if isinstance(metric_param, str):
+                metric_name = metric_param
+                metrics = replace(metrics, load_metric(metric_name.lower()), i)
+            elif isinstance(metric_param, dict):
+                metric_name = metric_param.pop("metric_name")  # must be given
+                resulting_name = metric_param.pop("resulting_name") if "resulting_name" in metric_param else None
+                params = metric_param
+                metrics = replace(
+                    metrics, load_metric(metric_name=metric_name, resulting_name=resulting_name, params=params), i
+                )
+            elif isinstance(metric_param, Metric):
                 continue
+        return metrics
 
-            if "bleu" in metric.metric_name:
-                score = [
-                    metric.compute(
-                        predictions=Collator([hyp]), references=Collator(refs), return_dict=False, fn_multiple=True
-                    )
-                    for hyp in hyps
-                ]
-            else:
-                score = []
-                for hyp, ref in zip(hyps, refs):
-                    _score = metric.compute(
-                        predictions=Collator([hyp], keep=True),
-                        references=Collator([ref], keep=True),
-                        return_dict=False,
-                        fn_multiple=True,
-                    )
-                    score.append(_score)
-            scores.append(reduce_fn(score))
-
-        return {metric.resulting_name: float(np.mean(scores))}
-
-    def compute_metric(
-        self, metric, predictions: Collator, references: Collator, reduce_fn: Callable
-    ) -> Dict[str, float]:
-        if predictions.can_collapse() and references.can_collapse() and metric.metric_name not in ["bleu", "sacrebleu"]:
-            predictions = Collator(predictions).reshape(-1)
-            references = Collator(references).reshape(-1)
-            result = metric.compute(predictions=predictions, references=references)
-        elif predictions.can_collapse() and metric.metric_name in ["bleu", "sacrebleu", "squad"]:
-            predictions = predictions.reshape_len(-1)
-            result = metric.compute(predictions=predictions, references=references)
+    def _load_metrics(self, metrics: Union[MetricParam, List[MetricParam]]) -> List[Metric]:
+        if metrics is None:
+            metrics = DEFAULT_METRICS
+        elif isinstance(metrics, (str, Metric)):
+            metrics = self._load_single_metric(metrics)
+        elif isinstance(metrics, list):
+            metrics = self._load_multiple_metrics(metrics)
         else:
-            result = self._compute_metric_for_multiple_items(
-                metric, predictions=predictions, references=references, reduce_fn=reduce_fn
-            )
-        return result
+            raise ValueError(f"Unknown input type {type(metrics)}")
+
+        return metrics
+
+    def _score_to_dict(self, score, name: str) -> Dict[str, float]:
+        if isinstance(score, dict):
+            return score
+
+        return {name: score}
 
     def _compute_single_score(self, inputs) -> Mapping[str, float]:
         metric, predictions, references, reduce_fn = inputs
-        score = self.compute_metric(metric=metric, predictions=predictions, references=references, reduce_fn=reduce_fn)
+        if isinstance(metric, Metric):
+            predictions, references = Collator(predictions), Collator(references)
+            score = metric.compute(predictions=predictions, references=references, reduce_fn=reduce_fn)
+        else:
+            metric.resulting_name = metric.name
+            score = metric.compute(predictions=predictions, references=references)
+            score = self._score_to_dict(score, name=metric.name)
         return score
 
     def _prepare_concurrent_inputs(self, predictions, references, reduce_fn):
         inputs = []
         for metric in self.metrics:
-            inputs.append([metric, predictions, references, reduce_fn])
+            inputs.append((metric, predictions, references, reduce_fn))
         return inputs
 
+    def add_metric(self, metric_name: str, resulting_name: str = None, params: Dict = None) -> None:
+        metric = load_metric(metric_name, resulting_name=resulting_name, params=params)
+        self.metrics.append(metric)
+
+    def remove_metric(self, resulting_name: str, error: bool = True) -> None:
+        for i, metric in enumerate(self.metrics):
+            if metric.resulting_name == resulting_name:
+                self.metrics.pop(i)
+                return
+        if error:
+            # raise an error if resulting_metric is not found
+            raise ValueError(f"Metric with resulting name {resulting_name} does not exists.")
+
     def evaluate(
         self,
-        predictions: Union[str, List[str], List[List[str]], List[Dict]],
-        references: Union[str, List[str], List[List[str]], List[Dict]],
+        *,
+        predictions: Union[List[str], List[List[str]]] = None,
+        references: Union[List[str], List[List[str]]] = None,
         reduce_fn: Optional[Union[str, Callable]] = None,
     ) -> Dict[str, float]:
-        if len(predictions) != len(references):
-            raise ValueError("Lengths of predictions and references must be equal.")
-
-        if reduce_fn is None:
-            reduce_fn = "max"
-
-        reduce_fn = getattr(np, reduce_fn) if isinstance(reduce_fn, str) else reduce_fn
-        if not is_reduce_fn(reduce_fn):
-            raise ValueError("'reduce_fn' must be an aggregation function.")
-        predictions = Collator(predictions)
-        references = Collator(references)
-
-        metrics = dict()
-        metrics["empty_predictions"] = len([1 for p in predictions if not p])
-        metrics["total_items"] = len(references)
-
-        if self._concurrent:
-            inputs_list = self._prepare_concurrent_inputs(predictions, references, reduce_fn)
-            set_env("TOKENIZERS_PARALLELISM", "true")
-            with ProcessPoolExecutor() as executor:
-                for score in executor.map(self._compute_single_score, inputs_list):
-                    metrics.update(score)
-        else:
-            for metric in self.metrics:
-                score = self.compute_metric(
-                    metric=metric, predictions=predictions, references=references, reduce_fn=reduce_fn
-                )
-                metrics.update(score)
-
-        return metrics
+        """Returns __call__() method. For backward compatibility."""
+        return self.__call__(predictions=predictions, references=references, reduce_fn=reduce_fn)
diff --git a/jury/definitions.py b/jury/definitions.py
index f0712b8..ab64a83 100644
--- a/jury/definitions.py
+++ b/jury/definitions.py
@@ -1,14 +1,10 @@
-from jury.metrics.bleu import BLEU
-from jury.metrics.meteor import Meteor
-from jury.metrics.rouge import Rouge
-from jury.metrics.sacrebleu import SacreBLEU
+from jury.metrics import Bleu, Meteor, Rouge
 
 DEFAULT_METRICS = [
-    BLEU(resulting_name="bleu_1", params={"max_order": 1}),
-    BLEU(resulting_name="bleu_2", params={"max_order": 2}),
-    BLEU(resulting_name="bleu_3", params={"max_order": 3}),
-    BLEU(resulting_name="bleu_4", params={"max_order": 4}),
+    Bleu(params={"max_order": 1}),
+    Bleu(params={"max_order": 2}),
+    Bleu(params={"max_order": 3}),
+    Bleu(params={"max_order": 4}),
     Meteor(),
     Rouge(),
-    SacreBLEU(),
 ]
diff --git a/jury/metrics/__init__.py b/jury/metrics/__init__.py
index f87d4e4..be6c97a 100644
--- a/jury/metrics/__init__.py
+++ b/jury/metrics/__init__.py
@@ -1,15 +1,25 @@
-import importlib
-from typing import Dict
+from jury.metrics._base import Metric, load_metric
+from jury.metrics.accuracy import Accuracy
+from jury.metrics.bertscore import Bertscore
+from jury.metrics.bleu import Bleu
+from jury.metrics.f1 import F1
+from jury.metrics.meteor import Meteor
+from jury.metrics.precision import Precision
+from jury.metrics.recall import Recall
+from jury.metrics.rouge import Rouge
+from jury.metrics.sacrebleu import Sacrebleu
+from jury.metrics.squad import Squad
 
-from jury.metrics._base import Metric
-
-
-def load_metric(metric_name: str, resulting_name: str = None, params: Dict = None):
-    # load the module, will raise ImportError if module cannot be loaded
-    metric_name = metric_name.lower()
-    base_name = metric_name.split("_")[0]
-    module_name = f"jury.metrics.{base_name}"
-    m = importlib.import_module(module_name)
-    # get the class, will raise AttributeError if class cannot be found
-    c = getattr(m, m.__class_names__.get(metric_name))
-    return c(resulting_name=resulting_name, params=params)
+__all__ = [
+    "Accuracy",
+    "Bertscore",
+    "Bleu",
+    "F1",
+    "Meteor",
+    "Metric",
+    "Precision",
+    "Recall",
+    "Rouge",
+    "Sacrebleu",
+    "Squad",
+]
diff --git a/jury/metrics/_base.py b/jury/metrics/_base.py
index 0ea7f27..158311d 100644
--- a/jury/metrics/_base.py
+++ b/jury/metrics/_base.py
@@ -1,26 +1,210 @@
-from abc import ABC
-from typing import Dict, Union
+# coding=utf-8
+# Copyright 2020 The HuggingFace Datasets Authors
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""
+Metrics base class. The part of this file is adapted from HuggingFace's
+datasets package implementation of Accuracy metric. See
+https://github.com/huggingface/datasets/blob/master/src/datasets/metric.py
+"""
+
+import importlib
+import warnings
+from abc import ABC, abstractmethod
+from typing import Any, Callable, Dict, List, Optional, Tuple, Union
 
 import datasets
+import numpy
+import pandas as pd
+from datasets.utils.logging import get_logger
+
+from jury.collator import Collator
+from jury.metrics._utils import import_module, is_reduce_fn
+
+logger = get_logger(__name__)
+
+
+def load_metric(metric_name: str, resulting_name: str = None, params: Dict = None) -> "Metric":
+    # load the module, will raise ImportError if module cannot be loaded
+    metric_name = metric_name.lower()
+    module_name = f"jury.metrics.{metric_name}"
+    try:
+        m = importlib.import_module(module_name)
+    except ModuleNotFoundError:
+        # Metric not in Jury
+        warnings.warn(
+            f"Metric {metric_name} is not available on Jury, falling back to datasets metric. "
+            f"You may not fully utilize this metric for different input types, e.g multiple predictions"
+            f"or multiple references."
+        )
+        metric = datasets.load_metric(metric_name)
+    else:
+        # get the class, will raise AttributeError if class cannot be found
+        c = getattr(m, m.__class_names__.get(metric_name))
+        metric = c(resulting_name=resulting_name, params=params)
+    return metric
+
+
+class Metric(datasets.Metric, ABC):
+    """Base metric class and common API for all metrics.
+
+    Args:
+        resulting_name (``Optional[str]``): Optional resulting name for :py:class:`jury.Jury` to use. By default, it
+            uses `metric.name` if not given. This is meant to prevent clashes for output dict of
+            :py:method:`jury.Jury.evaluate` such as when bleu-1, and bleu-2 are used together.
+        params (``Optional[Dict[str, Any]]``): These are the parameters to be passed to compute function of the metric.
+            It is meant to ease the support of computation from a jury configuration file, etc.
+        config_name (``str``): This is used to define a hash specific to a metrics computation script and prevents the metric's data
+            to be overridden when the metric loading script is modified.
+        keep_in_memory (``bool``): keep all predictions and references in memory. Not possible in distributed settings.
+        cache_dir (``str``): Path to a directory in which temporary prediction/references data will be stored.
+            The data directory should be located on a shared file-system in distributed setups.
+        num_process (``int``): specify the total number of nodes in a distributed settings.
+            This is useful to compute metrics in distributed setups (in particular non-additive metrics like F1).
+        process_id (``int``): specify the id of the current process in a distributed setup (between 0 and num_process-1)
+            This is useful to compute metrics in distributed setups (in particular non-additive metrics like F1).
+        seed (Optional ``int``): If specified, this will temporarily set numpy's random seed when :func:`datasets.Metric.compute` is run.
+        experiment_id (``str``): A specific experiment id. This is used if several distributed evaluations share the same file system.
+            This is useful to compute metrics in distributed setups (in particular non-additive metrics like F1).
+        max_concurrent_cache_files (``int``): Max number of concurrent metrics cache files (default 10000).
+        timeout (``Union[int, float]``): Timeout in second for distributed setting synchronization.
+    """
+
+    default_features = datasets.Features(
+        {
+            "predictions": datasets.Sequence(datasets.Value("string", id="sequence")),
+            "references": datasets.Sequence(datasets.Value("string", id="sequence")),
+        }
+    )
+
+    def __init__(self, resulting_name: Optional[str] = None, params: Optional[Dict[str, Any]] = None, **kwargs):
+        super().__init__(**kwargs)
+        self.resulting_name = resulting_name if resulting_name is not None else self.name
+        self.params = params or {}
+        if "reduce_fn" not in self.params:
+            self.params.update({"reduce_fn": "max"})
+        self.download_and_prepare()
+
+    def _compute(self, predictions: List[List[str]], references: List[List[str]], **kwargs) -> Dict[str, float]:
+        assert len(predictions) == len(references), "Predictions and references length does not match."
+        reduce_fn = kwargs.get("reduce_fn")
+        reduce_fn = self.params["reduce_fn"] if reduce_fn is None else reduce_fn
+        if isinstance(reduce_fn, str):
+            reduce_fn = getattr(numpy, reduce_fn)
+        elif reduce_fn is not None and not callable(reduce_fn):
+            raise TypeError(f"'reduce_fn' Expected str or callable, got {type(reduce_fn)}")
+        if reduce_fn is not None and not is_reduce_fn(reduce_fn):
+            raise ValueError("'reduce_fn' must be an aggregation function.")
+        eval_params = {**self.params, **kwargs}
+        eval_params.pop("reduce_fn")
+        predictions, references = Collator(predictions), Collator(references)
+        result = self.evaluate(predictions=predictions, references=references, reduce_fn=reduce_fn, **eval_params)
+        return {self.resulting_name: result}
+
+    @abstractmethod
+    def _compute_single_pred_single_ref(
+        self, predictions: Collator, references: Collator, reduce_fn: Callable = None, **kwargs
+    ):
+        """
+        Computes the metric score(s) for single prediction and single reference case.
+
+        Args:
+            predictions: (``List[str]``) Predictions
+            references: (``List[str]``) References
+            reduce_fn: (``str``) Reduce function name.
+            **kwargs: Additional arguments used for the metric computation.
+
+        Returns: score
+        """
+        pass
+
+    @abstractmethod
+    def _compute_single_pred_multi_ref(
+        self, predictions: Collator, references: Collator, reduce_fn: Callable, **kwargs
+    ):
+        """
+        Computes the metric score(s) for single prediction and multiple references case.
+
+        Args:
+            predictions: (``List[str]``) Predictions
+            references: (``List[List[str]]``) References
+            reduce_fn: (``str``) Reduce function name.
+            **kwargs: Additional arguments used for the metric computation.
+
+        Returns: score
+        """
+        pass
+
+    @abstractmethod
+    def _compute_multi_pred_multi_ref(self, predictions: Collator, references: Collator, reduce_fn: Callable, **kwargs):
+        """
+        Computes the metric score(s) for single prediction and multiple references case.
+
+        Args:
+            predictions: (``List[List[str]]``) Predictions
+            references: (``List[List[str]]``) References
+            reduce_fn: (``str``) Reduce function name.
+            **kwargs: Additional arguments used for the metric computation.
+
+        Returns: score
+        """
+        pass
+
+    def _download_and_prepare(self, dl_manager):
+        """Downloads and prepares resources for the metric.
+
+        This is the internal implementation to overwrite called when user calls
+        `download_and_prepare`. It should download all required resources for the metric.
+
+        Args:
+            dl_manager (:class:`DownloadManager`): `DownloadManager` used to download and cache data.
+        """
+        self.external_module_path = None
+        return None
 
+    def _get_external_resource(self, module_name: Optional[str], attr: Optional[str] = None):
+        if self.external_module_path is None:
+            raise AttributeError("'external_module_path' is not defined.")
+        if module_name is None:
+            module_name = "external_module"
+        external_module = import_module(module_name, self.external_module_path)
+        if attr is None:
+            return external_module
+        return getattr(external_module, attr)
 
-class Metric(ABC):
-    def __init__(self, metric_name: str, resulting_name: str = None, params: Dict = None):
-        self.metric_name = metric_name.lower()
-        self.resulting_name = resulting_name if resulting_name is not None else metric_name
-        self.params = params if params is not None else {}
+    @staticmethod
+    def _reduce_scores(scores: Union[List[Dict[str, float]], List[float]], reduce_fn: Callable):
+        if isinstance(scores[0], dict):
+            score = pd.DataFrame(scores).apply(reduce_fn, axis=0).to_dict()
+        else:
+            score = float(reduce_fn(scores))
+        return score
 
-    def compute(self, predictions, references, return_dict: bool = True, fn_multiple: bool = False):
-        predictions, references = self._preprocess(predictions, references, fn_multiple=fn_multiple)
-        metric = datasets.load_metric(self.metric_name)
-        result = metric.compute(predictions=predictions, references=references, **self.params)
-        return self._postprocess(result, return_dict=return_dict)
+    def _preprocess(self, predictions: List[List[str]], references: List[List[str]]) -> Tuple[Collator, Collator]:
+        return Collator(predictions, keep=True), Collator(references, keep=True)
 
-    def _preprocess(self, predictions, references, fn_multiple):
-        return predictions, references
+    def evaluate(
+        self, predictions: Collator, references: Collator, reduce_fn: Optional[Callable] = None, **kwargs
+    ) -> Dict[str, float]:
+        if predictions.can_collapse() and references.can_collapse():
+            predictions = predictions.collapse()
+            references = references.collapse()
+            eval_fn = self._compute_single_pred_single_ref
+        elif predictions.can_collapse() and not references.can_collapse():
+            predictions = predictions.collapse()
+            eval_fn = self._compute_single_pred_multi_ref
+        else:
+            eval_fn = self._compute_multi_pred_multi_ref
 
-    def _postprocess(self, result, return_dict: bool) -> Union[Dict[str, float], float]:
-        score = result[self.metric_name]
-        if not return_dict:
-            return score
-        return {self.resulting_name: float(score)}
+        predictions, references = self._preprocess(predictions, references)
+        return eval_fn(predictions=predictions, references=references, reduce_fn=reduce_fn, **kwargs)
diff --git a/jury/metrics/_utils.py b/jury/metrics/_utils.py
new file mode 100644
index 0000000..82b3016
--- /dev/null
+++ b/jury/metrics/_utils.py
@@ -0,0 +1,62 @@
+import importlib.util
+import os
+import re
+import string
+import warnings
+from typing import Callable, Sequence, Union
+
+import numpy as np
+import requests
+
+
+class PackagePlaceholder:
+    def __init__(self, version: str):
+        self.__version__ = version
+
+
+def requirement_message(metric_name: str, package_name: str) -> str:
+    return (
+        f"In order to use metric '{metric_name}', '{package_name}' is required. "
+        f"You can install the package by `pip install {package_name}`."
+    )
+
+
+def download(source: str, destination: str, overwrite: bool = False, warn: bool = False) -> None:
+    if os.path.exists(destination) and not overwrite:
+        if warn:
+            warnings.warn(
+                f"Path {destination} already exists, not overwriting. To overwrite, speficy " f"'overwrite' parameter."
+            )
+        return
+    r = requests.get(source, allow_redirects=True)
+
+    with open(destination, "wb") as out_file:
+        out_file.write(r.content)
+
+
+def import_module(module_name: str, filepath: str):
+    spec = importlib.util.spec_from_file_location(module_name, filepath)
+    module = importlib.util.module_from_spec(spec)
+    spec.loader.exec_module(module)
+    return module
+
+
+def get_token_lengths(sequences: Sequence[Sequence[str]], reduce_fn: Callable = None) -> Union[int, Sequence[int]]:
+    token_lengths = [len(item) for item in sequences]
+    if reduce_fn is not None:
+        return int(reduce_fn(token_lengths))
+    return token_lengths
+
+
+def normalize_text(text: str) -> str:
+    def remove_punctuations_and_ws(text: str) -> str:
+        pattern = r"[%s]" % re.escape(string.punctuation)
+        text = re.sub(pattern, " ", text)
+        return " ".join(text.split())
+
+    return remove_punctuations_and_ws(text).lower()
+
+
+def is_reduce_fn(fun: Callable) -> bool:
+    result = np.array(fun([1, 2]))
+    return result.size == 1
diff --git a/jury/metrics/accuracy.py b/jury/metrics/accuracy.py
new file mode 100644
index 0000000..14c6df9
--- /dev/null
+++ b/jury/metrics/accuracy.py
@@ -0,0 +1,132 @@
+# coding=utf-8
+# Copyright 2021 Open Business Software Solutions, The HuggingFace Datasets Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""
+Accuracy metric. The part of this file is adapted from HuggingFace's
+datasets package implementation of Accuracy metric. See
+https://github.com/huggingface/datasets/blob/master/metrics/accuracy/accuracy.py
+"""
+from collections import Counter
+from typing import Callable, Dict
+
+import datasets
+import numpy as np
+
+from jury.collator import Collator
+from jury.metrics import Metric
+from jury.metrics._utils import normalize_text
+
+__class_names__ = {"accuracy": "Accuracy"}
+
+_CITATION = """\
+@inproceedings{papineni2002bleu,
+  title={Bleu: a method for automatic evaluation of machine translation},
+  author={Papineni, Kishore and Roukos, Salim and Ward, Todd and Zhu, Wei-Jing},
+  booktitle={Proceedings of the 40th annual meeting of the Association for Computational Linguistics},
+  pages={311--318},
+  year={2002}
+}
+"""
+
+_DESCRIPTION = """
+This metric is derived from Modified Unigram Precision as an accuracy metric so that
+it will compute across maximum of reference or prediction. The computation is similar 
+to precision and recall, however, we call this accuracy since there is no measure 
+called "modified unigram accuracy".
+Accuracy is the fraction of the common unigrams between the prediction
+and the references among the max of prediction or reference tokens. It can be computed with:
+Accuracy = # of matching tokens / max(# of prediction tokens, # of reference tokens)
+"""
+
+_KWARGS_DESCRIPTION = """
+Args:
+    predictions: list of predictions to score. Each predictions
+        should be a string with tokens separated by spaces.
+    references: list of reference for each prediction. Each
+        reference should be a string with tokens separated by spaces.
+Returns:
+    'score': Accuracy score.
+Examples:
+
+    >>> accuracy = jury.load_metric("accuracy")
+    >>> predictions = [["the cat is on the mat", "There is cat playing on the mat"], ["Look! a wonderful day."]]
+    >>> references = [
+        ["the cat is playing on the mat.", "The cat plays on the mat."], 
+        ["Today is a wonderful day", "The weather outside is wonderful."]
+    ]
+    >>> results = accuracy.compute(predictions=predictions, references=references)
+    >>> print(results)
+    {'accuracy': {'score': 0.7285714285714285}}
+"""
+
+
+@datasets.utils.file_utils.add_start_docstrings(_DESCRIPTION, _KWARGS_DESCRIPTION)
+class Accuracy(Metric):
+    def _info(self):
+        return datasets.MetricInfo(
+            description=_DESCRIPTION,
+            citation=_CITATION,
+            inputs_description=_KWARGS_DESCRIPTION,
+            features=self.default_features,
+            reference_urls=["https://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_score.html"],
+        )
+
+    def _tokenize(self, predictions: Collator, references: Collator):
+        predictions = [normalize_text(p).split() for p in predictions]
+        references = [normalize_text(r).split() for r in references]
+        return predictions, references
+
+    def _compute_single_pred_single_ref(
+        self, predictions: Collator, references: Collator, reduce_fn: Callable = None, **kwargs
+    ):
+        scores = []
+        predictions, references = self._tokenize(predictions, references)
+        for pred, ref in zip(predictions, references):
+            score = 0
+            pred_counts = Counter(pred)
+            ref_counts = Counter(ref)
+            for token, pred_count in pred_counts.items():
+                if token in ref_counts:
+                    score += min(pred_count, ref_counts[token])  # Intersection count
+            scores.append(score / max(len(pred), len(ref)))
+        avg_score = sum(scores) / len(scores)
+        return {"score": avg_score}
+
+    def _compute_single_pred_multi_ref(
+        self, predictions: Collator, references: Collator, reduce_fn: Callable, **kwargs
+    ):
+        scores = []
+        for pred, refs in zip(predictions, references):
+            pred_score = [
+                self._compute_single_pred_single_ref(Collator([pred], keep=True), Collator([ref], keep=True))
+                for ref in refs
+            ]
+            reduced_score = self._reduce_scores(pred_score, reduce_fn=reduce_fn)
+            scores.append(reduced_score)
+
+        return self._reduce_scores(scores, reduce_fn=np.mean)
+
+    def _compute_multi_pred_multi_ref(self, predictions: Collator, references: Collator, reduce_fn: Callable, **kwargs):
+        scores = []
+        for preds, refs in zip(predictions, references):
+            pred_scores = []
+            for pred in preds:
+                pred_score = self._compute_single_pred_multi_ref(
+                    Collator([pred], keep=True), Collator([refs], keep=True), reduce_fn=reduce_fn
+                )
+                pred_scores.append(pred_score)
+            reduced_score = self._reduce_scores(pred_scores, reduce_fn=reduce_fn)
+            scores.append(reduced_score)
+
+        return self._reduce_scores(scores, reduce_fn=np.mean)
diff --git a/jury/metrics/bertscore.py b/jury/metrics/bertscore.py
index fdefe27..4cfa0d1 100644
--- a/jury/metrics/bertscore.py
+++ b/jury/metrics/bertscore.py
@@ -1,22 +1,346 @@
-from typing import Dict
+# coding=utf-8
+# Copyright 2021 Open Business Software Solutions, The HuggingFace Datasets Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+""" BERTScore metric. The part of this file is adapted from BERTScore implementation
+of datasets package. See
+https://github.com/huggingface/datasets/blob/master/metrics/bertscore/bertscore.py """
 
-from jury.metrics import Metric
+import functools
+from contextlib import contextmanager
+from typing import Dict, List
 
-__class_names__ = {"bertscore": "BERTScore"}
+import datasets
+import numpy as np
+import pandas as pd
+from packaging import version
 
+from jury.collator import Collator
+from jury.metrics._base import Metric
+from jury.metrics._utils import PackagePlaceholder, requirement_message
 
-class BERTScore(Metric):
-    def __init__(self, metric_name: str = None, resulting_name: str = None, params: Dict = None):
-        metric_name = self.__class__.__name__ if metric_name is None else metric_name
-        resulting_name = metric_name if resulting_name is None else resulting_name
-        params = {"lang": "en"} if params is None else params
-        super().__init__(metric_name=metric_name, resulting_name=resulting_name, params=params)
+# `import bert_score` placeholder
+bert_score = PackagePlaceholder(version="0.3.10")
 
-    def _preprocess(self, predictions, references, fn_multiple):
-        predictions = predictions.to_list()
-        references = references.to_list()
-        return predictions, references
+__class_names__ = {"bertscore": "Bertscore"}
 
-    def _postprocess(self, result, return_dict):
-        result = {self.metric_name: result["f1"][0]}
-        return super()._postprocess(result, return_dict)
+
+@contextmanager
+def filter_logging_context():
+    def filter_log(record):
+        return False if "This IS expected if you are initializing" in record.msg else True
+
+    logger = datasets.utils.logging.get_logger("transformers.modeling_utils")
+    logger.addFilter(filter_log)
+    try:
+        yield
+    finally:
+        logger.removeFilter(filter_log)
+
+
+_CITATION = """\
+@inproceedings{bert-score,
+  title={BERTScore: Evaluating Text Generation with BERT},
+  author={Tianyi Zhang* and Varsha Kishore* and Felix Wu* and Kilian Q. Weinberger and Yoav Artzi},
+  booktitle={International Conference on Learning Representations},
+  year={2020},
+  url={https://openreview.net/forum?id=SkeHuCVFDr}
+}
+"""
+
+_DESCRIPTION = """\
+BERTScore leverages the pre-trained contextual embeddings from BERT and matches words in candidate and reference
+sentences by cosine similarity.
+It has been shown to correlate with human judgment on sentence-level and system-level evaluation.
+Moreover, BERTScore computes precision, recall, and F1 measure, which can be useful for evaluating different language
+generation tasks.
+
+See the `README.md` file at [https://github.com/Tiiiger/bert_score](https://github.com/Tiiiger/bert_score) for more
+information.
+"""
+
+_KWARGS_DESCRIPTION = """
+BERTScore Metrics with the hashcode from a source against one or more references.
+
+Args:
+    predictions (list of str): Prediction/candidate sentences.
+    references (list of str or list of list of str): Reference sentences.
+    lang (str): Language of the sentences; required (e.g. 'en').
+    model_type (str): Bert specification, default using the suggested
+        model for the target language; has to specify at least one of
+        `model_type` or `lang`.
+    num_layers (int): The layer of representation to use,
+        default using the number of layers tuned on WMT16 correlation data.
+    verbose (bool): Turn on intermediate status update.
+    idf (bool or dict): Use idf weighting; can also be a precomputed idf_dict.
+    device (str): On which the contextual embedding model will be allocated on.
+        If this argument is None, the model lives on cuda:0 if cuda is available.
+    nthreads (int): Number of threads.
+    batch_size (int): Bert score processing batch size,
+        at least one of `model_type` or `lang`. `lang` needs to be
+        specified when `rescale_with_baseline` is True.
+    rescale_with_baseline (bool): Rescale bertscore with pre-computed baseline.
+    baseline_path (str): Customized baseline file.
+    use_fast_tokenizer (bool): `use_fast` parameter passed to HF tokenizer. New in version 0.3.10.
+
+Returns:
+    'score': Bertscore f1. This is always the same as 'f1' in cases single-prediction and single-reference, and
+        single-prediction and multiple-references, otherwise it is reduced version of 'f1' by `reduce_fn`. 
+    'precision': Precision.
+    'recall': Recall.
+    'f1': F1 score.
+    'hashcode': Hashcode of the library.
+
+Examples:
+
+    >>> bertscore = jury.load_metric("bertscore")
+    >>> predictions = [["the cat is on the mat", "There is cat playing on the mat"], ["Look! a wonderful day."]]
+    >>> references = [
+        ["the cat is playing on the mat.", "The cat plays on the mat."], 
+        ["Today is a wonderful day", "The weather outside is wonderful."]
+    ]
+    >>> results = bertscore.compute(predictions=predictions, references=references)
+    >>> print(results)
+    {'bertscore': {'score': 0.9473764896392822, 'precision': 0.9467198252677917, 'recall': 0.9480386078357697, 
+        'f1': 0.9473764896392822, 'hashcode': 'roberta-large_L17_no-idf_version=0.3.10(hug_trans=4.9.1)'}}
+"""
+
+
+@datasets.utils.file_utils.add_start_docstrings(_DESCRIPTION, _KWARGS_DESCRIPTION)
+class Bertscore(Metric):
+    def _download_and_prepare(self, dl_manager):
+        global bert_score
+
+        try:
+            import bert_score
+        except ModuleNotFoundError:
+            raise ModuleNotFoundError(requirement_message(metric_name="Bertscore", package_name="bert-score"))
+        else:
+            super(Bertscore, self)._download_and_prepare(dl_manager)
+
+    def _info(self):
+        return datasets.MetricInfo(
+            description=_DESCRIPTION,
+            citation=_CITATION,
+            homepage="https://github.com/Tiiiger/bert_score",
+            inputs_description=_KWARGS_DESCRIPTION,
+            features=datasets.Features(
+                {
+                    "predictions": datasets.Sequence(datasets.Value("string", id="sequence")),
+                    "references": datasets.Sequence(datasets.Value("string", id="sequence"), id="references"),
+                }
+            ),
+            codebase_urls=["https://github.com/Tiiiger/bert_score"],
+            reference_urls=[
+                "https://github.com/Tiiiger/bert_score",
+                "https://arxiv.org/abs/1904.09675",
+            ],
+        )
+
+    def _compute_single_pred_single_ref(
+        self,
+        predictions,
+        references,
+        reduce_fn=None,
+        lang="en",
+        model_type=None,
+        num_layers=None,
+        verbose=False,
+        idf=False,
+        device=None,
+        batch_size=64,
+        nthreads=4,
+        all_layers=False,
+        rescale_with_baseline=False,
+        baseline_path=None,
+        use_fast_tokenizer=False,
+        return_average_scores=False,
+    ):
+        get_hash = bert_score.utils.get_hash
+        scorer = bert_score.BERTScorer
+
+        if version.parse(bert_score.__version__) >= version.parse("0.3.10"):
+            get_hash = functools.partial(get_hash, use_fast_tokenizer=use_fast_tokenizer)
+            scorer = functools.partial(scorer, use_fast_tokenizer=use_fast_tokenizer)
+        elif use_fast_tokenizer:
+            raise ImportWarning(
+                "To use a fast tokenizer, the module `bert-score>=0.3.10` is required, and the current version of `bert-score` doesn't match this condition.\n"
+                'You can install it with `pip install "bert-score>=0.3.10"`.'
+            )
+
+        if model_type is None:
+            assert lang is not None, "either lang or model_type should be specified"
+            model_type = bert_score.utils.lang2model[lang.lower()]
+
+        if num_layers is None:
+            num_layers = bert_score.utils.model2layers[model_type]
+
+        hashcode = get_hash(
+            model=model_type,
+            num_layers=num_layers,
+            idf=idf,
+            rescale_with_baseline=rescale_with_baseline,
+            use_custom_baseline=baseline_path is not None,
+        )
+
+        with filter_logging_context():
+            if not hasattr(self, "cached_bertscorer") or self.cached_bertscorer.hash != hashcode:
+                self.cached_bertscorer = scorer(
+                    model_type=model_type,
+                    num_layers=num_layers,
+                    batch_size=batch_size,
+                    nthreads=nthreads,
+                    all_layers=all_layers,
+                    idf=idf,
+                    device=device,
+                    lang=lang,
+                    rescale_with_baseline=rescale_with_baseline,
+                    baseline_path=baseline_path,
+                )
+
+        (P, R, F) = self.cached_bertscorer.score(
+            cands=predictions,
+            refs=references,
+            verbose=verbose,
+            batch_size=batch_size,
+        )
+
+        P = P.tolist()
+        R = R.tolist()
+        F = F.tolist()
+        score = float(np.mean(F))
+
+        if return_average_scores:
+            P = float(np.mean(P))
+            R = float(np.mean(R))
+            F = float(np.mean(F))
+
+        output_dict = {
+            "score": score,
+            "precision": P,
+            "recall": R,
+            "f1": F,
+            "hashcode": hashcode,
+        }
+        return output_dict
+
+    def _compute_single_pred_multi_ref(
+        self,
+        predictions,
+        references,
+        reduce_fn=None,
+        lang="en",
+        model_type=None,
+        num_layers=None,
+        verbose=False,
+        idf=False,
+        device=None,
+        batch_size=64,
+        nthreads=4,
+        all_layers=False,
+        rescale_with_baseline=False,
+        baseline_path=None,
+        use_fast_tokenizer=False,
+        return_average_scores=False,
+    ):
+        # BERTScore inherently supports multiple references
+        return self._compute_single_pred_single_ref(
+            predictions=predictions,
+            references=references,
+            reduce_fn=reduce_fn,
+            lang=lang,
+            model_type=model_type,
+            num_layers=num_layers,
+            verbose=verbose,
+            idf=idf,
+            device=device,
+            batch_size=batch_size,
+            nthreads=nthreads,
+            all_layers=all_layers,
+            rescale_with_baseline=rescale_with_baseline,
+            baseline_path=baseline_path,
+            use_fast_tokenizer=use_fast_tokenizer,
+            return_average_scores=return_average_scores,
+        )
+
+    def _compute_multi_pred_multi_ref(
+        self,
+        predictions,
+        references,
+        reduce_fn=None,
+        lang="en",
+        model_type=None,
+        num_layers=None,
+        verbose=False,
+        idf=False,
+        device=None,
+        batch_size=64,
+        nthreads=4,
+        all_layers=False,
+        rescale_with_baseline=False,
+        baseline_path=None,
+        use_fast_tokenizer=False,
+    ):
+        scores = []
+        for preds, refs in zip(predictions, references):
+            pred_scores = [
+                self._compute_single_pred_multi_ref(
+                    predictions=Collator([pred], keep=True),
+                    references=Collator([refs], keep=True),
+                    reduce_fn=reduce_fn,
+                    lang=lang,
+                    model_type=model_type,
+                    num_layers=num_layers,
+                    verbose=verbose,
+                    idf=idf,
+                    device=device,
+                    batch_size=batch_size,
+                    nthreads=nthreads,
+                    all_layers=all_layers,
+                    rescale_with_baseline=rescale_with_baseline,
+                    baseline_path=baseline_path,
+                    use_fast_tokenizer=use_fast_tokenizer,
+                    return_average_scores=True,
+                )
+                for pred in preds
+            ]
+            hashcode = pred_scores[0]["hashcode"]
+            reduced_score = self._reduce_multi_pred_scores(pred_scores, reduce_fn=reduce_fn)
+            scores.append(reduced_score)
+
+        # Average reduced scores
+        return self._reduce_multi_pred_scores(scores, reduce_fn=np.mean, hashcode=hashcode)
+
+    def _reduce_multi_pred_scores(self, results: List[Dict], reduce_fn, **kwargs) -> Dict:
+        df = pd.DataFrame(results)
+        if "hashcode" in df:
+            df.drop("hashcode", axis=1, inplace=True)
+        scores = df.apply(reduce_fn, axis=0).to_dict()
+        scores.update(kwargs)
+        return scores
+
+    def add_batch(self, predictions=None, references=None, **kwargs):
+        """Add a batch of predictions and references for the metric's stack."""
+        # References can be strings or lists of strings
+        # Let's change strings to lists of strings with one element
+        if references is not None:
+            references = [[ref] if isinstance(ref, str) else ref for ref in references]
+        super().add_batch(predictions=predictions, references=references, **kwargs)
+
+    def add(self, prediction=None, reference=None, **kwargs):
+        """Add one prediction and reference for the metric's stack."""
+        # References can be strings or lists of strings
+        # Let's change strings to lists of strings with one element
+        if isinstance(reference, str):
+            reference = [reference]
+        super().add(prediction=prediction, reference=reference, **kwargs)
diff --git a/jury/metrics/bleu.py b/jury/metrics/bleu.py
index 4315618..8831164 100644
--- a/jury/metrics/bleu.py
+++ b/jury/metrics/bleu.py
@@ -1,32 +1,238 @@
-from typing import Dict
+# coding=utf-8
+# Copyright 2021 Open Business Software Solutions, The HuggingFace Datasets Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""
+BLEU metric. The part of this file is adapted from HuggingFace's
+datasets package implementation of BLEU metric. See
+https://github.com/huggingface/datasets/blob/master/metrics/bleu/bleu.py
+"""
 
-from jury.metrics import Metric
+import math
+import os
+from typing import Callable, Dict, Tuple
+
+import datasets
+
+from jury.collator import Collator
+from jury.metrics._base import Metric
+from jury.metrics._utils import download, get_token_lengths
 from jury.tokenizer import BLEUDefaultTokenizer, TokenizerWrapper
 
-__class_names__ = {"bleu": "BLEU"}
+__class_names__ = {"bleu": "Bleu"}
+
+
+_CITATION = """\
+@INPROCEEDINGS{Papineni02bleu:a,
+    author = {Kishore Papineni and Salim Roukos and Todd Ward and Wei-jing Zhu},
+    title = {BLEU: a Method for Automatic Evaluation of Machine Translation},
+    booktitle = {},
+    year = {2002},
+    pages = {311--318}
+}
+@inproceedings{lin-och-2004-orange,
+    title = "{ORANGE}: a Method for Evaluating Automatic Evaluation Metrics for Machine Translation",
+    author = "Lin, Chin-Yew  and
+      Och, Franz Josef",
+    booktitle = "{COLING} 2004: Proceedings of the 20th International Conference on Computational Linguistics",
+    month = "aug 23{--}aug 27",
+    year = "2004",
+    address = "Geneva, Switzerland",
+    publisher = "COLING",
+    url = "https://www.aclweb.org/anthology/C04-1072",
+    pages = "501--507",
+}
+"""
+
+_DESCRIPTION = """\
+BLEU (bilingual evaluation understudy) is an algorithm for evaluating the quality of text which has been 
+machine-translated from one natural language to another. Quality is considered to be the correspondence between a 
+machine's output and that of a human: "the closer a machine translation is to a professional human translation, the 
+better it is" – this is the central idea behind BLEU. BLEU was one of the first metrics to claim a high correlation 
+with human judgements of quality, and remains one of the most popular automated and inexpensive metrics.
+
+Scores are calculated for individual translated segments—generally sentences—by comparing them with a set of good 
+quality reference translations. Those scores are then averaged over the whole corpus to reach an estimate of the 
+translation's overall quality. Intelligibility or grammatical correctness are not taken into account[citation needed].
 
+BLEU's output is always a number between 0 and 1. This value indicates how similar the candidate text is to 
+the reference texts, with values closer to 1 representing more similar texts. Few human translations will attain a 
+score of 1, since this would indicate that the candidate is identical to one of the reference translations. For this 
+reason, it is not necessary to attain a score of 1. Because there are more opportunities to match, adding additional
+reference translations will increase the BLEU score.
+"""
 
-class BLEU(Metric):
-    def __init__(self, metric_name: str = None, resulting_name: str = None, params: Dict = None):
-        metric_name = self.__class__.__name__ if metric_name is None else metric_name
-        params = {} if params is None else params
-        tokenizer = params.get("tokenizer", None)
+_KWARGS_DESCRIPTION = """
+Computes BLEU score of translated segments against one or more references.
+Args:
+    predictions: list of translations to score.
+        Each translation should be tokenized into a list of tokens.
+    references: list of lists of references for each translation.
+        Each reference should be tokenized into a list of tokens.
+    max_order: Maximum n-gram order to use when computing BLEU score.
+    smooth: Whether or not to apply Lin et al. 2004 smoothing.
+Returns:
+    'score': BLEU score,
+    'precisions': geometric mean of n-gram precisions,
+    'brevity_penalty': brevity penalty,
+    'length_ratio': ratio of lengths,
+    'translation_length': translation_length,
+    'reference_length': reference_length
+Examples:
+
+    >>> bleu = jury.load_metric("bleu")
+    >>> predictions = [["the cat is on the mat", "There is cat playing on the mat"], ["Look! a wonderful day."]]
+    >>> references = [
+        ["the cat is playing on the mat.", "The cat plays on the mat."], 
+        ["Today is a wonderful day", "The weather outside is wonderful."]
+    ]
+    >>> results = bleu.compute(predictions=predictions, references=references)
+    >>> print(results)
+    {'bleu': {'score': 0.42370250917168295, 
+        'precisions': [0.8823529411764706, 0.6428571428571429, 0.45454545454545453, 0.125], 
+        'brevity_penalty': 1.0, 'length_ratio': 1.0, 'translation_length': 11, 'reference_length': 11}}
+"""
+
+
+@datasets.utils.file_utils.add_start_docstrings(_DESCRIPTION, _KWARGS_DESCRIPTION)
+class Bleu(Metric):
+    def __init__(self, resulting_name: str = None, params: Dict = None):
+        self.should_change_resulting_name = True if resulting_name is None else False
+        tokenizer = params.get("tokenizer", None) if params is not None else None
         self.tokenizer = BLEUDefaultTokenizer() if tokenizer is None else tokenizer
-        super().__init__(metric_name=metric_name, resulting_name=resulting_name, params=params)
+        super().__init__(resulting_name=resulting_name, params=params)
+
+    def _info(self):
+        return datasets.MetricInfo(
+            description=_DESCRIPTION,
+            citation=_CITATION,
+            inputs_description=_KWARGS_DESCRIPTION,
+            features=datasets.Features(
+                {
+                    "predictions": datasets.Sequence(datasets.Value("string", id="tokens"), id="sequence"),
+                    "references": datasets.Sequence(datasets.Value("string", id="tokens"), id="sequence"),
+                }
+            ),
+            codebase_urls=["https://github.com/tensorflow/nmt/blob/master/nmt/scripts/bleu.py"],
+            reference_urls=[
+                "https://en.wikipedia.org/wiki/BLEU",
+                "https://towardsdatascience.com/evaluating-text-output-in-nlp-bleu-at-your-own-risk-e8609665a213",
+            ],
+        )
+
+    def _download_and_prepare(self, dl_manager) -> None:
+        """
+        Downloads and import the computation of bleu score from the implementation
+        of BLEU computation from tensorflow/nmt. See
+        https://github.com/tensorflow/nmt/blob/master/nmt/scripts/bleu.py
+        """
+        nmt_source = "https://raw.githubusercontent.com/tensorflow/nmt/master/nmt/scripts/bleu.py"
+        nmt_dest = os.path.join(self.data_dir, "nmt_bleu.py")
+        download(
+            source=nmt_source,
+            destination=nmt_dest,
+        )
+        self.external_module_path = nmt_dest
 
-    def _preprocess(self, predictions, references, fn_multiple):
+    def _tokenize(self, predictions: Collator, references: Collator) -> Tuple[Collator, Collator]:
         tokenizer_wrapper = TokenizerWrapper(self.tokenizer)
-        predictions, references = tokenizer_wrapper.tokenize(predictions, references)
-        if not fn_multiple:
-            predictions = predictions.collapse()
+        return tokenizer_wrapper.tokenize(predictions, references)
+
+    def _compute_bleu_score(self, predictions: Collator, references: Collator, max_order=4, smooth=False):
+        evaluation_fn = self._get_external_resource("nmt_bleu", attr="compute_bleu")
+        score = evaluation_fn(
+            reference_corpus=references, translation_corpus=predictions, max_order=max_order, smooth=smooth
+        )
+        (bleu, precisions, bp, ratio, translation_length, reference_length) = score
+        return {
+            "score": bleu,
+            "precisions": precisions,
+            "brevity_penalty": bp,
+            "length_ratio": ratio,
+            "translation_length": translation_length,
+            "reference_length": reference_length,
+        }
+
+    def _compute_single_pred_single_ref(
+        self, predictions: Collator, references: Collator, reduce_fn: Callable = None, max_order=4, smooth=False
+    ):
+        predictions = predictions.reshape(
+            len(predictions),
+        )
+        return self._compute_bleu_score(predictions=predictions, references=references)
+
+    def _compute_single_pred_multi_ref(
+        self, predictions: Collator, references: Collator, reduce_fn: Callable = None, max_order=4, smooth=False
+    ):
+        # Bleu score inherently supports multiple references.
+        return self._compute_single_pred_single_ref(
+            predictions=predictions, references=references, reduce_fn=reduce_fn, max_order=max_order, smooth=smooth
+        )
+
+    def _compute_multi_pred_multi_ref(
+        self, predictions: Collator, references: Collator, reduce_fn: Callable = None, max_order=4, smooth=False
+    ):
+        flattened_predictions = []
+        matched_references = []
+        adjusted_reference_length = adjusted_prediction_length = 0
+        for preds, refs in zip(predictions, references):
+            n_preds = len(preds)
+            adjusted_reference_length += get_token_lengths(refs, reduce_fn=min)
+            adjusted_prediction_length += get_token_lengths(preds, reduce_fn=max)
+            flattened_predictions.extend([pred for pred in preds])
+            matched_references.extend([refs] * n_preds)
+
+        flattened_predictions = Collator(flattened_predictions, keep=True)
+        matched_references = Collator(matched_references, keep=True)
+        score = self._compute_single_pred_multi_ref(predictions=flattened_predictions, references=matched_references)
+
+        prediction_length, reference_length = score["translation_length"], score["reference_length"]
+        ratio = prediction_length / reference_length
+        adjusted_ratio = adjusted_prediction_length / adjusted_reference_length
+
+        bleu_score = score["score"]
+        if ratio > 1.0:
+            adjusted_bp = 1.0
+            bleu_score = bleu_score
         else:
-            if predictions.ndim > 2:
-                predictions = predictions.reshape_len(-1)
+            bp = math.exp(1 - 1.0 / ratio)
+            adjusted_bp = math.exp(1 - 1.0 / adjusted_ratio)
+            bleu_score = bleu_score * (adjusted_bp / bp)
+
+        bleu_score *= adjusted_ratio ** 2 / ratio
 
-            if references.ndim == 3:
-                ref_count = references.shape[0]
-                references = references.reshape(1, ref_count, -1)
-            else:
-                references = references.reshape(1, -1)
+        score.update(
+            {
+                "score": bleu_score,
+                "precisions": score["precisions"],
+                "brevity_penalty": adjusted_bp,
+                "length_ratio": adjusted_ratio,
+                "translation_length": adjusted_prediction_length,
+                "reference_length": adjusted_reference_length,
+            }
+        )
 
-        return predictions, references
+        return score
+
+    def evaluate(self, predictions: Collator, references: Collator, reduce_fn: Callable, **kwargs) -> Dict[str, float]:
+        max_order = kwargs.get("max_order")
+        if max_order is not None and self.should_change_resulting_name:
+            self.resulting_name += f"_{max_order}"
+        if predictions.can_collapse() and references.can_collapse():
+            eval_fn = self._compute_single_pred_single_ref
+        elif predictions.can_collapse() and not references.can_collapse():
+            eval_fn = self._compute_single_pred_multi_ref
+        else:
+            eval_fn = self._compute_multi_pred_multi_ref
+        predictions, references = self._tokenize(predictions=predictions, references=references)
+        return eval_fn(predictions=predictions, references=references, reduce_fn=reduce_fn, **kwargs)
diff --git a/jury/metrics/f1.py b/jury/metrics/f1.py
new file mode 100644
index 0000000..4fd4680
--- /dev/null
+++ b/jury/metrics/f1.py
@@ -0,0 +1,131 @@
+# coding=utf-8
+# Copyright 2021 Open Business Software Solutions, The HuggingFace Datasets Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""
+F1  metric. The part of this file is adapted from HuggingFace's
+datasets package implementation of Accuracy metric. See
+https://github.com/huggingface/datasets/blob/master/metrics/f1/f1.py
+"""
+from typing import Callable
+
+import datasets
+import numpy as np
+
+from jury.collator import Collator
+from jury.metrics import Metric
+from jury.metrics._utils import normalize_text
+from jury.metrics.precision import Precision
+from jury.metrics.recall import Recall
+
+__class_names__ = {"f1": "F1"}
+
+_CITATION = """\
+@inproceedings{papineni2002bleu,
+  title={Bleu: a method for automatic evaluation of machine translation},
+  author={Papineni, Kishore and Roukos, Salim and Ward, Todd and Zhu, Wei-Jing},
+  booktitle={Proceedings of the 40th annual meeting of the Association for Computational Linguistics},
+  pages={311--318},
+  year={2002}
+}
+"""
+
+_DESCRIPTION = """
+Harmonic mean of precision and recall metrics. The precision and recall it uses 
+are the implementations of `jury.metrics.precision` and `jury.metrics.recall` respectively.
+"""
+
+_KWARGS_DESCRIPTION = """
+Args:
+    predictions: list of predictions to score. Each predictions
+        should be a string with tokens separated by spaces.
+    references: list of reference for each prediction. Each
+        reference should be a string with tokens separated by spaces.
+Returns:
+    'score': F1 score.
+Examples:
+
+    >>> f1 = jury.load_metric("f1")
+    >>> predictions = [["the cat is on the mat", "There is cat playing on the mat"], ["Look! a wonderful day."]]
+    >>> references = [
+        ["the cat is playing on the mat.", "The cat plays on the mat."], 
+        ["Today is a wonderful day", "The weather outside is wonderful."]
+    ]
+    >>> results = f1.compute(predictions=predictions, references=references)
+    >>> print(results)
+    {'f1': {'score': 0.7948717948717947}}
+"""
+
+
+@datasets.utils.file_utils.add_start_docstrings(_DESCRIPTION, _KWARGS_DESCRIPTION)
+class F1(Metric):
+    def _info(self):
+        return datasets.MetricInfo(
+            description=_DESCRIPTION,
+            citation=_CITATION,
+            inputs_description=_KWARGS_DESCRIPTION,
+            features=datasets.Features(
+                {
+                    "predictions": datasets.Sequence(datasets.Value("string")),
+                    "references": datasets.Sequence(datasets.Value("string")),
+                }
+            ),
+            reference_urls=["https://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_score.html"],
+        )
+
+    def _tokenize(self, predictions: Collator, references: Collator):
+        predictions = [normalize_text(p).split() for p in predictions]
+        references = [normalize_text(r).split() for r in references]
+        return predictions, references
+
+    def _compute_single_pred_single_ref(
+        self, predictions: Collator, references: Collator, reduce_fn: Callable = None, **kwargs
+    ):
+        recall = Recall()
+        precision = Precision()
+        predictions, references = predictions.nested(), references.nested()
+        recall_score = recall.compute(predictions=predictions, references=references)["recall"]["score"]
+        precision_score = precision.compute(predictions=predictions, references=references)["precision"]["score"]
+        try:
+            f1 = (2 * recall_score * precision_score) / (recall_score + precision_score)
+        except ZeroDivisionError:
+            return {"score": 0.0}
+        return {"score": f1}
+
+    def _compute_single_pred_multi_ref(
+        self, predictions: Collator, references: Collator, reduce_fn: Callable, **kwargs
+    ):
+        scores = []
+        for pred, refs in zip(predictions, references):
+            pred_score = [
+                self._compute_single_pred_single_ref(Collator([pred], keep=True), Collator([ref], keep=True))
+                for ref in refs
+            ]
+            reduced_score = self._reduce_scores(pred_score, reduce_fn=reduce_fn)
+            scores.append(reduced_score)
+
+        return self._reduce_scores(scores, reduce_fn=np.mean)
+
+    def _compute_multi_pred_multi_ref(self, predictions: Collator, references: Collator, reduce_fn: Callable, **kwargs):
+        scores = []
+        for preds, refs in zip(predictions, references):
+            pred_scores = []
+            for pred in preds:
+                pred_score = self._compute_single_pred_multi_ref(
+                    Collator([pred], keep=True), Collator([refs], keep=True), reduce_fn=reduce_fn
+                )
+                pred_scores.append(pred_score)
+            reduced_score = self._reduce_scores(pred_scores, reduce_fn=reduce_fn)
+            scores.append(reduced_score)
+
+        return self._reduce_scores(scores, reduce_fn=np.mean)
diff --git a/jury/metrics/meteor.py b/jury/metrics/meteor.py
index 1876ba0..e3732db 100644
--- a/jury/metrics/meteor.py
+++ b/jury/metrics/meteor.py
@@ -1,11 +1,136 @@
-from typing import Dict
+# coding=utf-8
+# Copyright 2020 Open Business Software Solutions, The HuggingFace Datasets Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+""" METEOR metric. The part of this file is adapted from HuggingFace's
+datasets package implementation of METEOR metric. See
+https://github.com/huggingface/datasets/blob/master/metrics/meteor/meteor.py
+"""
+from typing import Dict, Optional
 
-from jury.metrics import Metric
+import datasets
+import numpy as np
+from nltk.translate import meteor_score
+
+from jury.metrics._base import Metric
 
 __class_names__ = {"meteor": "Meteor"}
 
+_CITATION = """\
+@inproceedings{banarjee2005,
+  title     = {{METEOR}: An Automatic Metric for {MT} Evaluation with Improved Correlation with Human Judgments},
+  author    = {Banerjee, Satanjeev  and Lavie, Alon},
+  booktitle = {Proceedings of the {ACL} Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization},
+  month     = jun,
+  year      = {2005},
+  address   = {Ann Arbor, Michigan},
+  publisher = {Association for Computational Linguistics},
+  url       = {https://www.aclweb.org/anthology/W05-0909},
+  pages     = {65--72},
+}
+"""
+
+_DESCRIPTION = """\
+METEOR, an automatic metric for machine translation evaluation
+that is based on a generalized concept of unigram matching between the
+machine-produced translation and human-produced reference translations.
+Unigrams can be matched based on their surface forms, stemmed forms,
+and meanings; furthermore, METEOR can be easily extended to include more
+advanced matching strategies. Once all generalized unigram matches
+between the two strings have been found, METEOR computes a score for
+this matching using a combination of unigram-precision, unigram-recall, and
+a measure of fragmentation that is designed to directly capture how
+well-ordered the matched words in the machine translation are in relation
+to the reference.
+
+METEOR gets an R correlation value of 0.347 with human evaluation on the Arabic
+data and 0.331 on the Chinese data. This is shown to be an improvement on
+using simply unigram-precision, unigram-recall and their harmonic F1
+combination.
+"""
+
+_KWARGS_DESCRIPTION = """
+Computes METEOR score of translated segments against one or more references.
+Args:
+    predictions: list of predictions to score. Each prediction
+        should be a string with tokens separated by spaces.
+    references: list of reference for each prediction. Each
+        reference should be a string with tokens separated by spaces.
+    alpha: Parameter for controlling relative weights of precision and recall. default: 0.9
+    beta: Parameter for controlling shape of penalty as a function of fragmentation. default: 3
+    gamma: Relative weight assigned to fragmentation penalty. default: 0.5
+Returns:
+    'score': meteor score.
+Examples:
+
+    >>> meteor = jury.load_metric("meteor")
+    >>> predictions = [["the cat is on the mat", "There is cat playing on the mat"], ["Look! a wonderful day."]]
+    >>> references = [
+        ["the cat is playing on the mat.", "The cat plays on the mat."], 
+        ["Today is a wonderful day", "The weather outside is wonderful."]
+    ]
+    >>> results = meteor.compute(predictions=predictions, references=references)
+    >>> print(results)
+    {'meteor': {'score': 0.5420511682934044}}
+"""
+
 
+@datasets.utils.file_utils.add_start_docstrings(_DESCRIPTION, _KWARGS_DESCRIPTION)
 class Meteor(Metric):
-    def __init__(self, metric_name: str = None, resulting_name: str = None, params: Dict = None):
-        metric_name = self.__class__.__name__ if metric_name is None else metric_name
-        super().__init__(metric_name=metric_name, resulting_name=resulting_name, params=params)
+    def _info(self):
+        return datasets.MetricInfo(
+            description=_DESCRIPTION,
+            citation=_CITATION,
+            inputs_description=_KWARGS_DESCRIPTION,
+            features=datasets.Features(
+                {
+                    "predictions": datasets.Sequence(datasets.Value("string", id="sequence")),
+                    "references": datasets.Sequence(datasets.Value("string", id="sequence")),
+                }
+            ),
+            codebase_urls=["https://github.com/nltk/nltk/blob/develop/nltk/translate/meteor_score.py"],
+            reference_urls=[
+                "https://www.nltk.org/api/nltk.translate.html#module-nltk.translate.meteor_score",
+                "https://en.wikipedia.org/wiki/METEOR",
+            ],
+        )
+
+    def _download_and_prepare(self, dl_manager):
+        import nltk
+
+        nltk.download("wordnet")
+
+    def _compute_single_pred_single_ref(self, predictions, references, reduce_fn=None, alpha=0.9, beta=3, gamma=0.5):
+        scores = [
+            meteor_score.single_meteor_score(ref, pred, alpha=alpha, beta=beta, gamma=gamma)
+            for ref, pred in zip(references, predictions)
+        ]
+        return {"score": self._reduce_scores(scores, reduce_fn=np.mean)}
+
+    def _compute_single_pred_multi_ref(self, predictions, references, reduce_fn=None, alpha=0.9, beta=3, gamma=0.5):
+        scores = [
+            meteor_score.meteor_score(references=ref, hypothesis=pred, alpha=alpha, beta=beta, gamma=gamma)
+            for ref, pred in zip(references, predictions)
+        ]
+        return {"score": self._reduce_scores(scores, reduce_fn=np.mean)}
+
+    def _compute_multi_pred_multi_ref(self, predictions, references, reduce_fn, alpha=0.9, beta=3, gamma=0.5):
+        scores = []
+        for pred, ref in zip(predictions, references):
+            score = [
+                meteor_score.meteor_score(references=ref, hypothesis=p, alpha=alpha, beta=beta, gamma=gamma)
+                for p in pred
+            ]
+            reduced_score = reduce_fn(score)
+            scores.append(reduce_fn(reduced_score))
+        return {"score": self._reduce_scores(scores, reduce_fn=np.mean)}
diff --git a/jury/metrics/precision.py b/jury/metrics/precision.py
new file mode 100644
index 0000000..85142da
--- /dev/null
+++ b/jury/metrics/precision.py
@@ -0,0 +1,128 @@
+# coding=utf-8
+# Copyright 2020 The HuggingFace Datasets Authors and the current dataset script contributor.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""
+Modified Unigram Precision metric. The part of this file is adapted from HuggingFace's
+datasets package implementation of Accuracy metric. See
+https://github.com/huggingface/datasets/blob/master/metrics/precision/precision.py
+"""
+from collections import Counter
+from typing import Callable
+
+import datasets
+import numpy as np
+
+from jury.collator import Collator
+from jury.metrics import Metric
+from jury.metrics._utils import normalize_text
+
+__class_names__ = {"precision": "Precision"}
+
+_CITATION = """\
+@inproceedings{papineni2002bleu,
+  title={Bleu: a method for automatic evaluation of machine translation},
+  author={Papineni, Kishore and Roukos, Salim and Ward, Todd and Zhu, Wei-Jing},
+  booktitle={Proceedings of the 40th annual meeting of the Association for Computational Linguistics},
+  pages={311--318},
+  year={2002}
+}
+"""
+
+_DESCRIPTION = """
+Modified Unigram Precision is the fraction of the common unigrams between the prediction
+and the references among the prediction tokens. It can be computed with:
+Precision = # of matching tokens / # of prediction tokens
+"""
+
+_KWARGS_DESCRIPTION = """
+Args:
+    predictions: list of predictions to score. Each predictions
+        should be a string with tokens separated by spaces.
+    references: list of reference for each prediction. Each
+        reference should be a string with tokens separated by spaces.
+Returns:
+    'score': Precision score.
+Examples:
+
+    >>> precision = jury.load_metric("precision")
+    >>> predictions = [["the cat is on the mat", "There is cat playing on the mat"], ["Look! a wonderful day."]]
+    >>> references = [
+        ["the cat is playing on the mat.", "The cat plays on the mat."], 
+        ["Today is a wonderful day", "The weather outside is wonderful."]
+    ]
+    >>> results = precision.compute(predictions=predictions, references=references)
+    >>> print(results)
+    {'precision': {'score': 0.875}}
+"""
+
+
+@datasets.utils.file_utils.add_start_docstrings(_DESCRIPTION, _KWARGS_DESCRIPTION)
+class Precision(Metric):
+    def _info(self):
+        return datasets.MetricInfo(
+            description=_DESCRIPTION,
+            citation=_CITATION,
+            inputs_description=_KWARGS_DESCRIPTION,
+            features=self.default_features,
+            reference_urls=["https://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_score.html"],
+        )
+
+    def _tokenize(self, predictions: Collator, references: Collator):
+        predictions = [normalize_text(p).split() for p in predictions]
+        references = [normalize_text(r).split() for r in references]
+        return predictions, references
+
+    def _compute_single_pred_single_ref(
+        self, predictions: Collator, references: Collator, reduce_fn: Callable = None, **kwargs
+    ):
+        scores = []
+        predictions, references = self._tokenize(predictions, references)
+        for pred, ref in zip(predictions, references):
+            score = 0
+            pred_counts = Counter(pred)
+            ref_counts = Counter(ref)
+            for token, pred_count in pred_counts.items():
+                if token in ref_counts:
+                    score += min(pred_count, ref_counts[token])  # Intersection count
+            scores.append(score / len(pred))
+        avg_score = sum(scores) / len(scores)
+        return {"score": avg_score}
+
+    def _compute_single_pred_multi_ref(
+        self, predictions: Collator, references: Collator, reduce_fn: Callable, **kwargs
+    ):
+        scores = []
+        for pred, refs in zip(predictions, references):
+            pred_score = [
+                self._compute_single_pred_single_ref(Collator([pred], keep=True), Collator([ref], keep=True))
+                for ref in refs
+            ]
+            reduced_score = self._reduce_scores(pred_score, reduce_fn=reduce_fn)
+            scores.append(reduced_score)
+
+        return self._reduce_scores(scores, reduce_fn=np.mean)
+
+    def _compute_multi_pred_multi_ref(self, predictions: Collator, references: Collator, reduce_fn: Callable, **kwargs):
+        scores = []
+        for preds, refs in zip(predictions, references):
+            pred_scores = []
+            for pred in preds:
+                pred_score = self._compute_single_pred_multi_ref(
+                    Collator([pred], keep=True), Collator([refs], keep=True), reduce_fn=reduce_fn
+                )
+                pred_scores.append(pred_score)
+            reduced_score = self._reduce_scores(pred_scores, reduce_fn=reduce_fn)
+            scores.append(reduced_score)
+
+        return self._reduce_scores(scores, reduce_fn=np.mean)
diff --git a/jury/metrics/recall.py b/jury/metrics/recall.py
new file mode 100644
index 0000000..a1528d0
--- /dev/null
+++ b/jury/metrics/recall.py
@@ -0,0 +1,131 @@
+# coding=utf-8
+# Copyright 2020 The HuggingFace Datasets Authors and the current dataset script contributor.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""
+Recall metric. The part of this file is adapted from HuggingFace's
+datasets package implementation of Accuracy metric. See
+https://github.com/huggingface/datasets/blob/master/metrics/recall/recall.py
+"""
+from collections import Counter
+from typing import Callable, Dict
+
+import datasets
+import numpy as np
+
+from jury.collator import Collator
+from jury.metrics import Metric
+from jury.metrics._utils import normalize_text
+
+__class_names__ = {"recall": "Recall"}
+
+_CITATION = """\
+@inproceedings{papineni2002bleu,
+  title={Bleu: a method for automatic evaluation of machine translation},
+  author={Papineni, Kishore and Roukos, Salim and Ward, Todd and Zhu, Wei-Jing},
+  booktitle={Proceedings of the 40th annual meeting of the Association for Computational Linguistics},
+  pages={311--318},
+  year={2002}
+}
+"""
+
+_DESCRIPTION = """
+This metric is derived from Modified Unigram Precision as a recall metric so that
+it will compute across references and not across predictions unlike precision. The computation is similar, however,
+we call this recall since there is no measure called "modified unigram recall".
+Recall is the fraction of the common unigrams between the prediction
+and the references among the reference tokens. It can be computed with:
+Recall = # of matching tokens / # of reference tokens
+"""
+
+_KWARGS_DESCRIPTION = """
+Args:
+    predictions: list of predictions to score. Each predictions
+        should be a string with tokens separated by spaces.
+    references: list of reference for each prediction. Each
+        reference should be a string with tokens separated by spaces.
+Returns:
+    'score': Recall score.
+Examples:
+
+    >>> recall = jury.load_metric("recall")
+    >>> predictions = [["the cat is on the mat", "There is cat playing on the mat"], ["Look! a wonderful day."]]
+    >>> references = [
+        ["the cat is playing on the mat.", "The cat plays on the mat."], 
+        ["Today is a wonderful day", "The weather outside is wonderful."]
+    ]
+    >>> results = recall.compute(predictions=predictions, references=references)
+    >>> print(results)
+    {'recall': {'score': 0.7285714285714285}}
+"""
+
+
+@datasets.utils.file_utils.add_start_docstrings(_DESCRIPTION, _KWARGS_DESCRIPTION)
+class Recall(Metric):
+    def _info(self):
+        return datasets.MetricInfo(
+            description=_DESCRIPTION,
+            citation=_CITATION,
+            inputs_description=_KWARGS_DESCRIPTION,
+            features=self.default_features,
+            reference_urls=["https://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_score.html"],
+        )
+
+    def _tokenize(self, predictions: Collator, references: Collator):
+        predictions = [normalize_text(p).split() for p in predictions]
+        references = [normalize_text(r).split() for r in references]
+        return predictions, references
+
+    def _compute_single_pred_single_ref(
+        self, predictions: Collator, references: Collator, reduce_fn: Callable = None, **kwargs
+    ):
+        scores = []
+        predictions, references = self._tokenize(predictions, references)
+        for pred, ref in zip(predictions, references):
+            score = 0
+            pred_counts = Counter(pred)
+            ref_counts = Counter(ref)
+            for token, pred_count in pred_counts.items():
+                if token in ref_counts:
+                    score += min(pred_count, ref_counts[token])  # Intersection count
+            scores.append(score / len(ref))
+        avg_score = sum(scores) / len(scores)
+        return {"score": avg_score}
+
+    def _compute_single_pred_multi_ref(
+        self, predictions: Collator, references: Collator, reduce_fn: Callable, **kwargs
+    ):
+        scores = []
+        for pred, refs in zip(predictions, references):
+            pred_score = [
+                self._compute_single_pred_single_ref(Collator([pred], keep=True), Collator([ref], keep=True))
+                for ref in refs
+            ]
+            reduced_score = self._reduce_scores(pred_score, reduce_fn=reduce_fn)
+            scores.append(reduced_score)
+
+        return self._reduce_scores(scores, reduce_fn=np.mean)
+
+    def _compute_multi_pred_multi_ref(self, predictions: Collator, references: Collator, reduce_fn: Callable, **kwargs):
+        scores = []
+        for preds, refs in zip(predictions, references):
+            pred_scores = []
+            for pred in preds:
+                pred_score = self._compute_single_pred_multi_ref(
+                    Collator([pred], keep=True), Collator([refs], keep=True), reduce_fn=reduce_fn
+                )
+                pred_scores.append(pred_score)
+            reduced_score = self._reduce_scores(pred_scores, reduce_fn=reduce_fn)
+            scores.append(reduced_score)
+
+        return self._reduce_scores(scores, reduce_fn=np.mean)
diff --git a/jury/metrics/rouge.py b/jury/metrics/rouge.py
index d1201c5..e03ff35 100644
--- a/jury/metrics/rouge.py
+++ b/jury/metrics/rouge.py
@@ -1,17 +1,240 @@
-from typing import Dict
+# coding=utf-8
+# Copyright 2020 Open Business Software Solutions, The HuggingFace Datasets Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""
+ROUGE metric. The part of this file is adapted from HuggingFace's
+datasets package implementation of ROUGE metric. See
+https://github.com/huggingface/datasets/blob/master/metrics/rouge/rouge.py
+"""
 
-from jury.metrics import Metric
+from typing import Dict, List, Optional, Union
+
+import datasets
+import pandas as pd
+from rouge_score import rouge_scorer, scoring
+
+from jury.collator import Collator
+from jury.metrics._base import Metric
 
 __class_names__ = {"rouge": "Rouge"}
 
 
+_CITATION = """\
+@inproceedings{lin-2004-rouge,
+    title = "{ROUGE}: A Package for Automatic Evaluation of Summaries",
+    author = "Lin, Chin-Yew",
+    booktitle = "Text Summarization Branches Out",
+    month = jul,
+    year = "2004",
+    address = "Barcelona, Spain",
+    publisher = "Association for Computational Linguistics",
+    url = "https://www.aclweb.org/anthology/W04-1013",
+    pages = "74--81",
+}
+"""
+
+_DESCRIPTION = """\
+ROUGE, or Recall-Oriented Understudy for Gisting Evaluation, is a set of metrics and a software package used for
+evaluating automatic summarization and machine translation software in natural language processing.
+The metrics compare an automatically produced summary or translation against a reference or a set of references 
+(human-produced) summary or translation.
+
+Note that ROUGE is case insensitive, meaning that upper case letters are treated the same way as lower case letters.
+
+This metrics is a wrapper around Google Research reimplementation of ROUGE:
+https://github.com/google-research/google-research/tree/master/rouge
+"""
+
+_KWARGS_DESCRIPTION = """
+Calculates average rouge scores for a list of hypotheses and references
+Args:
+    predictions: list of predictions to score. Each predictions
+        should be a string with tokens separated by spaces.
+    references: list of reference for each prediction. Each
+        reference should be a string with tokens separated by spaces.
+    rouge_types: A list of rouge types to calculate.
+        Valid names:
+        `"rouge{n}"` (e.g. `"rouge1"`, `"rouge2"`) where: {n} is the n-gram based scoring,
+        `"rougeL"`: Longest common subsequence based scoring.
+        `"rougeLSum"`: rougeLsum splits text using `"\n"`.
+        See details in https://github.com/huggingface/datasets/issues/617
+    use_stemmer: Bool indicating whether Porter stemmer should be used to strip word suffixes.
+    use_agregator: Return aggregates if this is set to True
+Returns:
+    rouge1: rouge_1 (precision, recall, f1),
+    rouge2: rouge_2 (precision, recall, f1),
+    rougeL: rouge_l (precision, recall, f1),
+    rougeLsum: rouge_lsum (precision, recall, f1)
+Examples:
+
+    >>> rouge = jury.load_metric("rouge")
+    >>> predictions = [["the cat is on the mat", "There is cat playing on the mat"], ["Look! a wonderful day."]]
+    >>> references = [
+        ["the cat is playing on the mat.", "The cat plays on the mat."], 
+        ["Today is a wonderful day", "The weather outside is wonderful."]
+    ]
+    >>> results = rouge.compute(predictions=predictions, references=references)
+    >>> print(results)
+    {'rouge': {'rouge1': 0.7783882783882783, 'rouge2': 0.5925324675324675, 
+        'rougeL': 0.7426739926739926, 'rougeLsum': 0.7426739926739926}}
+"""
+
+
+@datasets.utils.file_utils.add_start_docstrings(_DESCRIPTION, _KWARGS_DESCRIPTION)
 class Rouge(Metric):
-    def __init__(self, metric_name: str = None, resulting_name: str = None, params: Dict = None):
-        metric_name = self.__class__.__name__ if metric_name is None else metric_name
-        resulting_name = "rougeL" if resulting_name is None else resulting_name
-        params = {"rouge_types": ["rougeL"]} if params is None else params
-        super().__init__(metric_name=metric_name, resulting_name=resulting_name, params=params)
-
-    def _postprocess(self, result, return_dict: bool):
-        result = {self.metric_name: result["rougeL"].mid.fmeasure}
-        return super()._postprocess(result, return_dict)
+    def _info(self):
+        return datasets.MetricInfo(
+            description=_DESCRIPTION,
+            citation=_CITATION,
+            inputs_description=_KWARGS_DESCRIPTION,
+            features=datasets.Features(
+                {
+                    "predictions": datasets.Sequence(datasets.Value("string", id="sequence")),
+                    "references": datasets.Sequence(datasets.Value("string", id="sequence")),
+                }
+            ),
+            codebase_urls=["https://github.com/google-research/google-research/tree/master/rouge"],
+            reference_urls=[
+                "https://en.wikipedia.org/wiki/ROUGE_(metric)",
+                "https://github.com/google-research/google-research/tree/master/rouge",
+            ],
+        )
+
+    @staticmethod
+    def _get_aggregator(use_aggregator: bool) -> Union[List, scoring.BootstrapAggregator]:
+        if use_aggregator:
+            aggregator = scoring.BootstrapAggregator()
+        else:
+            aggregator = []
+        return aggregator
+
+    @staticmethod
+    def _add_score(
+        aggregator: Union[List, scoring.BootstrapAggregator],
+        score: Union[Dict[str, float], Dict[str, scoring.BootstrapAggregator]],
+    ) -> Union[List, scoring.BootstrapAggregator]:
+        if isinstance(aggregator, scoring.BootstrapAggregator):
+            aggregator.add_scores(score)
+        else:
+            aggregator.append(score)
+        return aggregator
+
+    @staticmethod
+    def _aggregate(aggregator) -> Union[Dict[str, float], Dict[str, scoring.AggregateScore]]:
+        if isinstance(aggregator, scoring.BootstrapAggregator):
+            result = aggregator.aggregate()
+        else:
+            result = {}
+            for key in aggregator[0]:
+                result[key] = list(score[key] for score in aggregator)
+        return result
+
+    @staticmethod
+    def _normalize_score_list(
+        score_list: List[Dict[str, scoring.BootstrapAggregator]], metric_to_select: str
+    ) -> List[Dict[str, scoring.BootstrapAggregator]]:
+        for score_dict in score_list:
+            for metric, score in score_dict.items():
+                score_dict[metric] = getattr(score, metric_to_select)
+        return score_list
+
+    @staticmethod
+    def _reduce_dict(score_list: List[Dict[str, scoring.BootstrapAggregator]], reduce_fn: callable) -> Dict[str, float]:
+        return pd.DataFrame(score_list).apply(reduce_fn, axis=0).to_dict()
+
+    @staticmethod
+    def _select_mid_from_aggregation(aggregated_scores: Dict[str, scoring.AggregateScore]) -> Dict[str, float]:
+        return {metric: score.mid for metric, score in aggregated_scores.items()}
+
+    def evaluate(
+        self,
+        *,
+        predictions: Collator = None,
+        references: Collator = None,
+        reduce_fn: callable = None,
+        rouge_types: List[str] = None,
+        use_aggregator: bool = True,
+        use_stemmer: bool = False,
+        metric_to_select: Optional[str] = "fmeasure",
+        **kwargs
+    ):
+        if rouge_types is None:
+            rouge_types = ["rouge1", "rouge2", "rougeL", "rougeLsum"]
+
+        scorer = rouge_scorer.RougeScorer(rouge_types=rouge_types, use_stemmer=use_stemmer)
+
+        if predictions.can_collapse() and references.can_collapse():
+            aggregator = self._get_aggregator(use_aggregator)
+            predictions = predictions.collapse()
+            references = references.collapse()
+            aggregator = self._compute_single_pred_single_ref(
+                predictions, references, reduce_fn, scorer, aggregator, metric_to_select
+            )
+        elif predictions.can_collapse() and not references.can_collapse():
+            # Force to use BootstrapAggregator
+            aggregator = self._get_aggregator(use_aggregator=True)
+            predictions = predictions.collapse()
+            aggregator = self._compute_single_pred_multi_ref(
+                predictions, references, reduce_fn, scorer, aggregator, metric_to_select
+            )
+        else:
+            # Force to use BootstrapAggregator
+            aggregator = self._get_aggregator(use_aggregator=True)
+            aggregator = self._compute_multi_pred_multi_ref(
+                predictions, references, reduce_fn, scorer, aggregator, metric_to_select
+            )
+
+        result = self._aggregate(aggregator)
+
+        if metric_to_select and use_aggregator:
+            result = self._select_mid_from_aggregation(result)
+        elif metric_to_select and not use_aggregator:
+            result = {k: v[0] for k, v in result.items()}
+
+        return result
+
+    def _compute_single_pred_single_ref(
+        self, predictions, references, reduce_fn=None, scorer=None, aggregator=None, metric_to_select=None
+    ):
+        for ref, pred in zip(references, predictions):
+            score = scorer.score(target=ref, prediction=pred)
+            if metric_to_select is not None:
+                score = self._normalize_score_list(score_list=[score], metric_to_select=metric_to_select)[0]
+            aggregator = self._add_score(aggregator, score)
+        return aggregator
+
+    def _compute_single_pred_multi_ref(
+        self, predictions, references, reduce_fn, scorer=None, aggregator=None, metric_to_select=None
+    ):
+        for pred, refs in zip(predictions, references):
+            pred_scores = [scorer.score(target=ref, prediction=pred) for ref in refs]
+            pred_scores = self._normalize_score_list(pred_scores, metric_to_select)
+            score = self._reduce_dict(pred_scores, reduce_fn)
+            aggregator = self._add_score(aggregator, score)
+        return aggregator
+
+    def _compute_multi_pred_multi_ref(
+        self, predictions, references, reduce_fn, scorer=None, aggregator=None, metric_to_select=None
+    ):
+        for preds, refs in zip(predictions, references):
+            multi_aggregator = self._get_aggregator(use_aggregator=True)
+            for pred in preds:
+                pred_score = self._normalize_score_list(
+                    [scorer.score(target=ref, prediction=pred) for ref in refs], metric_to_select
+                )
+                pred_score = self._reduce_dict(pred_score, reduce_fn)
+                multi_aggregator = self._add_score(multi_aggregator, pred_score)
+            score = self._select_mid_from_aggregation(self._aggregate(multi_aggregator))
+            aggregator = self._add_score(aggregator, score)
+        return aggregator
diff --git a/jury/metrics/sacrebleu.py b/jury/metrics/sacrebleu.py
index 70ccbfa..cf9caa6 100644
--- a/jury/metrics/sacrebleu.py
+++ b/jury/metrics/sacrebleu.py
@@ -1,24 +1,276 @@
-from typing import Dict
+# coding=utf-8
+# Copyright 2020 Open Business Software Solutions, The HuggingFace Datasets Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""
+SacreBLEU metric. The part of this file is adapted from SacreBLEU implementation
+of datasets package. See
+https://github.com/huggingface/datasets/blob/master/metrics/sacrebleu/sacrebleu.py
+"""
+import math
+from typing import Callable, Dict, Sequence
 
-from jury.metrics import Metric
+import datasets
+from packaging import version
 
-__class_names__ = {"sacrebleu": "SacreBLEU"}
+from jury.collator import Collator
+from jury.metrics._base import Metric
+from jury.metrics._utils import PackagePlaceholder, get_token_lengths, requirement_message
+from jury.tokenizer import BLEUDefaultTokenizer
 
+# `import sacrebleu as scb` placeholder
+scb = PackagePlaceholder(version="2.0.0")
 
-class SacreBLEU(Metric):
-    def __init__(self, metric_name: str = None, resulting_name: str = None, params: Dict = None):
-        metric_name = self.__class__.__name__ if metric_name is None else metric_name
-        resulting_name = metric_name if resulting_name is None else resulting_name
-        super().__init__(metric_name=metric_name, resulting_name=resulting_name, params=params)
+__class_names__ = {"sacrebleu": "Sacrebleu"}
 
-    def _postprocess(self, result, return_dict):
-        result = {self.metric_name: result["score"] / 100}
-        return super()._postprocess(result, return_dict)
 
-    def _preprocess(self, predictions, references, fn_multiple):
-        if not fn_multiple:
+_CITATION = """\
+@inproceedings{post-2018-call,
+    title = "A Call for Clarity in Reporting {BLEU} Scores",
+    author = "Post, Matt",
+    booktitle = "Proceedings of the Third Conference on Machine Translation: Research Papers",
+    month = oct,
+    year = "2018",
+    address = "Belgium, Brussels",
+    publisher = "Association for Computational Linguistics",
+    url = "https://www.aclweb.org/anthology/W18-6319",
+    pages = "186--191",
+}
+"""
+
+_DESCRIPTION = """\
+SacreBLEU provides hassle-free computation of shareable, comparable, and reproducible BLEU scores.
+Inspired by Rico Sennrich's `multi-bleu-detok.perl`, it produces the official WMT scores but works with plain text.
+It also knows all the standard test sets and handles downloading, processing, and tokenization for you.
+
+See the [README.md] file at https://github.com/mjpost/sacreBLEU for more information.
+"""
+
+_KWARGS_DESCRIPTION = """
+Produces BLEU scores along with its sufficient statistics
+from a source against one or more references.
+
+Args:
+    predictions: list of predictions to score. Each predictions
+        should be a string with tokens separated by spaces.
+    references: list of reference for each prediction. Each
+        reference should be a string with tokens separated by spaces.
+    smooth_method: The smoothing method to use. (Default: 'exp').
+    smooth_value: The smoothing value. Only valid for 'floor' and 'add-k'. (Defaults: floor: 0.1, add-k: 1).
+    tokenize: Tokenization method to use for BLEU. If not provided, defaults to 'zh' for Chinese, 'ja-mecab' for
+        Japanese and '13a' (mteval) otherwise.
+    lowercase: Lowercase the data. If True, enables case-insensitivity. (Default: False).
+    force: Insist that your tokenized input is actually detokenized.
+
+Returns:
+    'score': BLEU score,
+    'counts': Counts,
+    'totals': Totals,
+    'precisions': Precisions,
+    'bp': Brevity penalty,
+    'sys_len': predictions length,
+    'ref_len': reference length,
+    'adjusted_precision': adjusted precisions with corrections for multiple predictions cases.
+
+Examples:
+
+    >>> sacrebleu = jury.load_metric("sacrebleu")
+    >>> predictions = [["the cat is on the mat", "There is cat playing on the mat"], ["Look! a wonderful day."]]
+    >>> references = [
+        ["the cat is playing on the mat.", "The cat plays on the mat."], 
+        ["Today is a wonderful day", "The weather outside is wonderful."]
+    ]
+    >>> results = sacrebleu.compute(predictions=predictions, references=references)
+    >>> print(results)
+    {'sacrebleu': {'score': 0.32377227131456443, 'counts': [11, 6, 3, 0], 'totals': [13, 11, 9, 7],
+         'precisions': [0.8461538461538461, 0.5454545454545454, 0.33333333333333337, 0.07142857142857144], 
+         'bp': 1.0, 'sys_len': 11, 'ref_len': 12, 
+         'adjusted_precisions': [0.8461538461538461, 0.5454545454545454, 0.33333333333333337, 0.07142857142857144]}}
+"""
+
+
+@datasets.utils.file_utils.add_start_docstrings(_DESCRIPTION, _KWARGS_DESCRIPTION)
+class Sacrebleu(Metric):
+    def _download_and_prepare(self, dl_manager):
+        global scb
+        try:
+            import sacrebleu as scb
+        except ModuleNotFoundError:
+            raise ModuleNotFoundError(requirement_message(metric_name="Sacrebleu", package_name="sacrebleu"))
+        else:
+            super(Sacrebleu, self)._download_and_prepare(dl_manager)
+
+    def _info(self):
+        if version.parse(scb.__version__) < version.parse("1.4.12"):
+            raise ImportWarning(
+                "To use `sacrebleu`, the module `sacrebleu>=1.4.12` is required, and the current version of `sacrebleu` doesn't match this condition.\n"
+                'You can install it with `pip install "sacrebleu>=1.4.12"`.'
+            )
+        return datasets.MetricInfo(
+            description=_DESCRIPTION,
+            citation=_CITATION,
+            homepage="https://github.com/mjpost/sacreBLEU",
+            inputs_description=_KWARGS_DESCRIPTION,
+            features=datasets.Features(
+                {
+                    "predictions": datasets.Sequence(datasets.Value("string", id="sequence"), id="predictions"),
+                    "references": datasets.Sequence(datasets.Value("string", id="sequence"), id="references"),
+                }
+            ),
+            codebase_urls=["https://github.com/mjpost/sacreBLEU"],
+            reference_urls=[
+                "https://github.com/mjpost/sacreBLEU",
+                "https://en.wikipedia.org/wiki/BLEU",
+                "https://towardsdatascience.com/evaluating-text-output-in-nlp-bleu-at-your-own-risk-e8609665a213",
+            ],
+        )
+
+    def _tokenize(self, seq: Sequence[str]) -> Sequence[Sequence[str]]:
+        tokenizer = BLEUDefaultTokenizer()
+        return [tokenizer.tokenize(s) for s in seq]
+
+    def _validate_references(self, references: Collator) -> None:
+        references_per_prediction = len(references[0])
+        if any(len(refs) != references_per_prediction for refs in references):
+            raise ValueError("Sacrebleu requires the same number of references for each prediction")
+
+    def _compute_single_pred_single_ref(
+        self,
+        predictions: Collator,
+        references: Collator,
+        reduce_fn: Callable = None,
+        smooth_method="exp",
+        smooth_value=None,
+        force=False,
+        lowercase=False,
+        tokenize=None,
+        use_effective_order=False,
+    ):
+        output = scb.corpus_bleu(
+            predictions,
+            references,
+            smooth_method=smooth_method,
+            smooth_value=smooth_value,
+            force=force,
+            lowercase=lowercase,
+            use_effective_order=use_effective_order,
+            **(dict(tokenize=tokenize) if tokenize else {}),
+        )
+        output_dict = {
+            "score": output.score / 100,
+            "counts": output.counts,
+            "totals": output.totals,
+            "precisions": [p / 100 for p in output.precisions],
+            "bp": output.bp,
+            "sys_len": output.sys_len,
+            "ref_len": output.ref_len,
+        }
+        return output_dict
+
+    def _compute_single_pred_multi_ref(
+        self,
+        predictions: Collator,
+        references: Collator,
+        reduce_fn: Callable,
+        smooth_method="exp",
+        smooth_value=None,
+        force=False,
+        lowercase=False,
+        tokenize=None,
+        use_effective_order=False,
+    ):
+        # SacreBleu inherently supports multiple references.
+        return self._compute_single_pred_single_ref(
+            predictions=predictions,
+            references=references,
+            reduce_fn=reduce_fn,
+            smooth_method=smooth_method,
+            smooth_value=smooth_value,
+            force=force,
+            lowercase=lowercase,
+            tokenize=tokenize,
+            use_effective_order=use_effective_order,
+        )
+
+    def _compute_multi_pred_multi_ref(
+        self,
+        predictions: Collator,
+        references: Collator,
+        reduce_fn: Callable,
+        smooth_method="exp",
+        smooth_value=None,
+        force=False,
+        lowercase=False,
+        tokenize=None,
+        use_effective_order=False,
+    ):
+        flattened_predictions = []
+        matched_references = []
+        adjusted_prediction_length = 0
+        for preds, refs in zip(predictions, references):
+            n_preds = len(preds)
+            tokenized_preds = self._tokenize(preds)
+            adjusted_prediction_length += get_token_lengths(tokenized_preds, reduce_fn=max)
+            flattened_predictions.extend([pred for pred in preds])
+            matched_references.extend([refs] * n_preds)
+        flattened_predictions = Collator(flattened_predictions, keep=True)
+        matched_references = Collator(matched_references, keep=True)
+        score = self._compute_single_pred_multi_ref(
+            predictions=flattened_predictions,
+            references=matched_references,
+            reduce_fn=reduce_fn,
+            smooth_method=smooth_method,
+            smooth_value=smooth_value,
+            force=force,
+            lowercase=lowercase,
+            tokenize=tokenize,
+            use_effective_order=use_effective_order,
+        )
+        prediction_length, reference_length = score["sys_len"], score["ref_len"]
+        ratio = prediction_length / reference_length
+        adjusted_ratio = adjusted_prediction_length / reference_length
+        if ratio > 1.0:
+            adjusted_bp = 1.0
+            scb_score = score["score"]
+        else:
+            bp = math.exp(1 - 1.0 / ratio)
+            adjusted_bp = math.exp(1 - 1.0 / adjusted_ratio)
+            scb_score = score["score"] * (adjusted_bp / bp)
+
+        scb_score = scb_score * n_preds
+        precisions = [p * n_preds for p in score["precisions"]]
+
+        score.update(
+            {
+                "score": scb_score,
+                "adjusted_precisions": precisions,
+                "bp": adjusted_bp,
+                "sys_len": adjusted_prediction_length,
+            }
+        )
+
+        return score
+
+    def evaluate(
+        self, predictions: Collator, references: Collator, reduce_fn: Callable = None, **kwargs
+    ) -> Dict[str, float]:
+        if predictions.can_collapse() and references.can_collapse():
+            predictions = predictions.collapse()
+            eval_fn = self._compute_single_pred_single_ref
+        elif predictions.can_collapse() and not references.can_collapse():
             predictions = predictions.collapse()
+            eval_fn = self._compute_single_pred_multi_ref
         else:
-            predictions = predictions.reshape(-1)
-            references = references.reshape(1, -1)
-        return predictions, references
+            eval_fn = self._compute_multi_pred_multi_ref
+        self._validate_references(references)
+        return eval_fn(predictions=predictions, references=references, reduce_fn=reduce_fn, **kwargs)
diff --git a/jury/metrics/squad.py b/jury/metrics/squad.py
index 33b67d8..d33d9d5 100644
--- a/jury/metrics/squad.py
+++ b/jury/metrics/squad.py
@@ -1,22 +1,128 @@
-from typing import Dict
+# coding=utf-8
+# Copyright 2020 Open Business Software Solutions, The HuggingFace Datasets Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+""" SQuAD metric. The part of this file is adapted from SacreBLEU implementation
+of datasets package. See
+https://github.com/huggingface/datasets/blob/master/metrics/squad/squad.py"""
+import copyreg
+import os
+import types
+from typing import Callable, Dict, List
+
+import datasets
+import numpy as np
+import pandas as pd
 
 from jury.collator import Collator
-from jury.metrics import Metric
+from jury.metrics._base import Metric
+from jury.metrics._utils import download, import_module
+from jury.utils import NestedSingleType
 
-__class_names__ = {"squad": "SQUAD_F1", "squad_f1": "SQUAD_F1", "squad_em": "SQUAD_EM"}
+__class_names__ = {"squad": "Squad"}
 
-from jury.utils import NestedSingleType
 
+_CITATION = """\
+@inproceedings{Rajpurkar2016SQuAD10,
+  title={SQuAD: 100, 000+ Questions for Machine Comprehension of Text},
+  author={Pranav Rajpurkar and Jian Zhang and Konstantin Lopyrev and Percy Liang},
+  booktitle={EMNLP},
+  year={2016}
+}
+"""
+
+_DESCRIPTION = """
+This metric wrap the official scoring script for version 1 of the Stanford Question Answering Dataset (SQuAD).
+
+Stanford Question Answering Dataset (SQuAD) is a reading comprehension dataset, consisting of questions posed by
+crowdworkers on a set of Wikipedia articles, where the answer to every question is a segment of text, or span,
+from the corresponding reading passage, or the question might be unanswerable.
+"""
 
-class SQUAD(Metric):
-    def __init__(self, metric_name: str = None, resulting_name: str = None, params: Dict = None):
-        metric_name = self.__class__.__name__ if metric_name is None else metric_name
-        resulting_name = metric_name if resulting_name is None else resulting_name
-        super().__init__(metric_name=metric_name, resulting_name=resulting_name, params=params)
+_KWARGS_DESCRIPTION = """
+Computes SQuAD scores (F1 and EM).
+Args:
+    predictions: list of predictions to score. Each predictions
+        should be a string with tokens separated by spaces. Optionally,
+        List of question-answers dictionaries with the following key-values:
+        - 'id': id of the question-answer pair as given in the references (see below)
+        - 'prediction_text': the text of the answer
+    references: list of reference for each prediction. Each
+        reference should be a string with tokens separated by spaces. Optionally,
+        List of question-answers dictionaries with the following key-values:
+        - 'id': id of the question-answer pair (see above),
+        - 'answers': a Dict in the SQuAD dataset format
+            {
+                'text': list of possible texts for the answer, as a list of strings
+                'answer_start': list of start positions for the answer, as a list of ints
+            }
+            Note that answer_start values are not taken into account to compute the metric.
+Returns:
+    'exact_match': Exact match (the normalized answer exactly match the gold answer)
+    'f1': The F-score of predicted tokens versus the gold answer
+Examples:
 
-    def _preprocess(self, predictions, references, fn_multiple):
-        if NestedSingleType.get_type(predictions, order=-1) == "str":
+    >>> squad = jury.load_metric("squad")
+    >>> predictions = [["the cat is on the mat", "There is cat playing on the mat"], ["Look! a wonderful day."]]
+    >>> references = [
+        ["the cat is playing on the mat.", "The cat plays on the mat."], 
+        ["Today is a wonderful day", "The weather outside is wonderful."]
+    ]
+    >>> results = squad.compute(predictions=predictions, references=references)
+    >>> print(results)
+    {'squad': {'exact_match': 0.0, 'f1': 74.02597402597402}}
+"""
+
+
+@datasets.utils.file_utils.add_start_docstrings(_DESCRIPTION, _KWARGS_DESCRIPTION)
+class Squad(Metric):
+    def _download_and_prepare(self, dl_manager) -> None:
+        """
+        Downloads and import the computation of squad score from the implementation
+        of Squad computation from huggingface/datasets. See
+        https://github.com/huggingface/datasets/blob/master/metrics/squad/evaluate.py
+        """
+        squad_source = "https://raw.githubusercontent.com/huggingface/datasets/master/metrics/squad/evaluate.py"
+        squad_dest = os.path.join(self.data_dir, "squad_evaluate.py")
+        download(
+            source=squad_source,
+            destination=squad_dest,
+        )
+        self.external_module_path = squad_dest
+
+    def _info(self):
+        return datasets.MetricInfo(
+            description=_DESCRIPTION,
+            citation=_CITATION,
+            inputs_description=_KWARGS_DESCRIPTION,
+            features=datasets.Features(
+                {
+                    "predictions": datasets.Sequence(datasets.Value("string")),
+                    "references": datasets.Sequence(datasets.Value("string")),
+                }
+            ),
+            codebase_urls=["https://rajpurkar.github.io/SQuAD-explorer/"],
+            reference_urls=["https://rajpurkar.github.io/SQuAD-explorer/"],
+        )
+
+    def _preprocess(self, predictions: Collator, references: Collator):
+        if NestedSingleType.get_type(predictions, order=-1) == "str" and predictions.can_collapse():
             predictions = [{"prediction_text": pred, "id": str(i)} for i, pred in enumerate(predictions.collapse())]
+        elif NestedSingleType.get_type(predictions, order=-1) == "str" and not predictions.can_collapse():
+            _predictions = []
+            for i, preds in enumerate(predictions):
+                _predictions.append([{"prediction_text": pred, "id": str(i)} for pred in preds])
+            predictions = _predictions
         if NestedSingleType.get_type(references, order=-1) == "str":
             references = [
                 {"answers": {"answer_start": [-1], "text": Collator(ref).collapse()}, "id": str(i)}
@@ -24,24 +130,50 @@ def _preprocess(self, predictions, references, fn_multiple):
             ]
         return predictions, references
 
+    def _compute_single_pred_single_ref(self, predictions: Collator, references: Collator, **kwargs):
+        pred_dict = {prediction["id"]: prediction["prediction_text"] for prediction in predictions}
+        dataset = [
+            {
+                "paragraphs": [
+                    {
+                        "qas": [
+                            {
+                                "answers": [{"text": answer_text} for answer_text in ref["answers"]["text"]],
+                                "id": ref["id"],
+                            }
+                            for ref in references
+                        ]
+                    }
+                ]
+            }
+        ]
+        evaluation_fn = self._get_external_resource("squad_evaluate", attr="evaluate")
+        score = evaluation_fn(dataset=dataset, predictions=pred_dict)
+        for metric_, score_ in score.items():
+            score[metric_] = score_ / 100
+        return score
 
-class SQUAD_EM(SQUAD):
-    def __init__(self, resulting_name: str = None, params: Dict = None):
-        metric_name = "squad"
-        resulting_name = self.__class__.__name__.lower() if resulting_name is None else resulting_name
-        super().__init__(metric_name=metric_name, resulting_name=resulting_name, params=params)
-
-    def _postprocess(self, result, return_dict):
-        result = {self.metric_name: result["exact_match"] / 100}
-        return super()._postprocess(result, return_dict)
+    def _compute_single_pred_multi_ref(self, predictions: Collator, references: Collator, reduce_fn=None, **kwargs):
+        return self._compute_single_pred_single_ref(predictions=predictions, references=references, reduce_fn=reduce_fn)
 
+    def _compute_multi_pred_multi_ref(self, predictions: Collator, references: Collator, reduce_fn: Callable, **kwargs):
+        scores = []
+        for preds, refs in zip(predictions, references):
+            pred_scores = []
+            for pred in preds:
+                pred_scores.append(
+                    self._compute_single_pred_multi_ref(
+                        predictions=Collator([pred], keep=True),
+                        references=Collator([refs], keep=True),
+                        reduce_fn=reduce_fn,
+                    )
+                )
+            reduced_score = self._reduce_multi_pred_scores(pred_scores, reduce_fn)
+            scores.append(reduced_score)
 
-class SQUAD_F1(SQUAD):
-    def __init__(self, resulting_name: str = None, params: Dict = None):
-        metric_name = "squad"
-        resulting_name = self.__class__.__name__.lower() if resulting_name is None else resulting_name
-        super().__init__(metric_name=metric_name, resulting_name=resulting_name, params=params)
+        # Average reduced scores
+        return self._reduce_multi_pred_scores(scores, np.mean)
 
-    def _postprocess(self, result, return_dict):
-        result = {self.metric_name: result["f1"] / 100}
-        return super()._postprocess(result, return_dict)
+    def _reduce_multi_pred_scores(self, results: List[Dict], reduce_fn) -> Dict:
+        df = pd.DataFrame(results)
+        return df.apply(reduce_fn, axis=0).to_dict()
diff --git a/jury/tokenizer.py b/jury/tokenizer.py
index 2e436d7..f6aa4c9 100644
--- a/jury/tokenizer.py
+++ b/jury/tokenizer.py
@@ -14,6 +14,14 @@ def tokenize(self, text: str) -> List[str]:
 
 
 class TokenizerWrapper:
+    """
+    Wraps the tokenizer object to adapt tokenize method such that it returns
+    a tuple of jury.Collator object instead of list.
+
+    Args:
+        tokenizer: Tokenizer object that implements `tokenize` method.
+    """
+
     def __init__(self, tokenizer):
         self.tokenizer = tokenizer
 
diff --git a/jury/utils.py b/jury/utils.py
index 165ad46..aa8bac4 100644
--- a/jury/utils.py
+++ b/jury/utils.py
@@ -1,9 +1,7 @@
 import os
 import re
 import string
-from typing import Callable, Dict, List, Optional
-
-import numpy as np
+from typing import Dict, List, Optional
 
 
 class NestedSingleType:
@@ -50,11 +48,6 @@ def bulk_remove_keys(obj: Dict, keys: List[str]) -> Dict:
     return {k: v for k, v in obj.items() if k not in keys}
 
 
-def is_reduce_fn(fun: Callable) -> bool:
-    result = np.array(fun([1, 2]))
-    return result.size == 1
-
-
 def set_env(name: str, value: str):
     if not isinstance(value, str):
         raise ValueError(f"Expected type str for 'value', got {type(value)}.")
diff --git a/pyproject.toml b/pyproject.toml
index 3aa87ab..776c7cc 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -9,3 +9,10 @@ exclude = '''
   )/
 )
 '''
+
+[tool.pytest.ini_options]
+minversion = "6.2"
+timeout = 150
+timeout_method = "thread"
+testpaths = "tests"
+addopts = "-vvv"
diff --git a/requirements.txt b/requirements.txt
index 0663410..63f917a 100755
--- a/requirements.txt
+++ b/requirements.txt
@@ -3,4 +3,4 @@ fire>=0.4.0
 nltk==3.6.2
 numpy>=1.21.0
 rouge-score==0.0.4
-tqdm
\ No newline at end of file
+tqdm
diff --git a/setup.py b/setup.py
index f96e2b6..6f8b8d7 100644
--- a/setup.py
+++ b/setup.py
@@ -1,6 +1,8 @@
 import io
 import os
+import platform
 import re
+from typing import List
 
 import setuptools
 
@@ -23,21 +25,32 @@ def get_version():
         return re.search(r'^__version__ = [\'"]([^\'"]*)[\'"]', f.read(), re.M).group(1)
 
 
+def add_pywin(reqs: List[str]) -> None:
+    if platform.system() == "Windows":
+        # Latest PyWin32 build (301) fails, required for sacrebleu
+        ext_package = ["pywin32==228"]
+    else:
+        ext_package = []
+    reqs.extend(ext_package)
+
+
 _DEV_REQUIREMENTS = [
     "black==21.7b0",
     "deepdiff==5.5.0",
     "flake8==3.9.2",
     "isort==5.9.2",
+    "jiwer>=2.2.0",
     "pytest>=6.2.4",
     "pytest-cov>=2.12.1",
+    "pytest-timeout>=1.4.2",
 ]
 
-_METRIC_REQUIREMENTS = ["sacrebleu==1.5.1", "bert_score==0.3.9"]
+_METRIC_REQUIREMENTS = ["sacrebleu>=2.0.0", "bert_score==0.3.10"]
+add_pywin(_METRIC_REQUIREMENTS)
 
 extras = {
-    "tests": _DEV_REQUIREMENTS,
     "metrics": _METRIC_REQUIREMENTS,
-    "develop": _DEV_REQUIREMENTS + _METRIC_REQUIREMENTS,
+    "dev": _DEV_REQUIREMENTS + _METRIC_REQUIREMENTS,
 }
 
 


Test patch:
diff --git a/tests/jury/__init__.py b/tests/jury/__init__.py
index e69de29..f891209 100644
--- a/tests/jury/__init__.py
+++ b/tests/jury/__init__.py
@@ -0,0 +1,5 @@
+import os.path
+
+TEST_DIR = os.path.abspath(os.path.dirname(__file__))
+TEST_DATA_DIR = os.path.join(TEST_DIR, "test_data")
+EXPECTED_OUTPUTS = os.path.join(TEST_DATA_DIR, "expected_outputs")
diff --git a/tests/jury/conftest.py b/tests/jury/conftest.py
index 1e10b40..22d344e 100644
--- a/tests/jury/conftest.py
+++ b/tests/jury/conftest.py
@@ -1,50 +1,98 @@
+import inspect
+import json
+import os
+from typing import Optional
+
 import pytest
 
-from jury import Jury
-from jury.metrics import load_metric
+from jury import Jury, load_metric
+from tests.jury import EXPECTED_OUTPUTS
 
 _TEST_METRICS = [
+    load_metric("accuracy"),
+    load_metric("bertscore", params={"model_type": "albert-base-v1", "device": "cpu"}),
     load_metric("bleu"),
+    load_metric("f1"),
     load_metric("meteor"),
+    load_metric("precision"),
+    load_metric("recall"),
     load_metric("rouge"),
     load_metric("sacrebleu"),
-    load_metric("bertscore", params={"model_type": "albert-base-v1", "device": "cpu"}),
     load_metric("squad"),
 ]
 
-_CONCURRENT_TEST_METRICS = [
-    load_metric("bleu"),
-    load_metric("meteor"),
-    load_metric("rouge"),
-    # load_metric("sacrebleu"),
-    # load_metric("bertscore", params={"model_type": "albert-base-v1", "device": "cpu"}),  # Memory allocation
-    load_metric("squad"),
+_STR_TEST_METRIC = "bleu"
+
+_LIST_STR_TEST_METRICS = [
+    "accuracy",
+    "bertscore",
+    "bleu",
+    "f1",
+    "meteor",
+    "precision",
+    "recall",
+    "rouge",
+    "sacrebleu",
+    "squad",
+]
+
+_LIST_DICT_TEST_METRICS = [
+    {"metric_name": "accuracy"},
+    {"metric_name": "bertscore", "model_type": "albert-base-v1"},
+    {"metric_name": "bleu", "resulting_name": "bleu-1", "max_order": 1},
+    {"metric_name": "bleu", "resulting_name": "bleu-2", "max_order": 2},
+    {"metric_name": "f1", "resulting_name": "F1"},
+    {"metric_name": "meteor", "resulting_name": "METEOR"},
+    {"metric_name": "precision"},
+    {"metric_name": "recall"},
+    {"metric_name": "rouge"},
+    {"metric_name": "sacrebleu"},
+    {"metric_name": "squad"},
 ]
 
-_STR_TEST_METRICS = ["bleu", "meteor", "rouge", "sacrebleu", "squad"]
+_LIST_MIXED_TEST_METRICS = [
+    "accuracy",
+    "bertscore",
+    "bleu",
+    {"metric_name": "f1"},
+    {"metric_name": "meteor"},
+    {"metric_name": "precision"},
+    "recall",
+    "rouge",
+    {"metric_name": "sacrebleu"},
+    {"metric_name": "squad"},
+]
+
+
+_DATASETS_METRICS = ["wer"]
 
 
-@pytest.fixture(scope="session")
+@pytest.fixture(scope="package")
 def predictions():
     return ["There is a cat on the mat.", "Look! a wonderful day."]
 
 
-@pytest.fixture(scope="session")
+@pytest.fixture(scope="package")
 def references():
     return ["The cat is playing on the mat.", "Today is a wonderful day"]
 
 
-@pytest.fixture(scope="session")
+@pytest.fixture(scope="function")
+def inconsistent_predictions():
+    return ["There is a cat on the mat."]
+
+
+@pytest.fixture(scope="function")
 def single_prediction_array():
     return [["the cat is on the mat"], ["Look! a wonderful day."]]
 
 
-@pytest.fixture(scope="session")
+@pytest.fixture(scope="package")
 def multiple_predictions():
     return [["the cat is on the mat", "There is cat playing on the mat"], ["Look! a wonderful day."]]
 
 
-@pytest.fixture(scope="session")
+@pytest.fixture(scope="package")
 def multiple_references():
     return [
         ["the cat is playing on the mat.", "The cat plays on the mat."],
@@ -52,16 +100,55 @@ def multiple_references():
     ]
 
 
-@pytest.fixture(scope="package")
+@pytest.fixture(scope="module")
 def jury():
     return Jury(metrics=_TEST_METRICS)
 
 
-@pytest.fixture(scope="package")
+@pytest.fixture(scope="function")
+def jury_concurrent():
+    return Jury(metrics=_TEST_METRICS, run_concurrent=True)
+
+
+@pytest.fixture(scope="function")
 def jury_str():
-    return Jury(metrics=_STR_TEST_METRICS)
+    return Jury(metrics=_STR_TEST_METRIC)
 
 
-@pytest.fixture(scope="package")
-def jury_concurrent():
-    return Jury(metrics=_CONCURRENT_TEST_METRICS, run_concurrent=True)
+@pytest.fixture(scope="function")
+def jury_list_str():
+    return Jury(metrics=_LIST_STR_TEST_METRICS)
+
+
+@pytest.fixture(scope="function")
+def jury_list_dict():
+    return Jury(metrics=_LIST_DICT_TEST_METRICS)
+
+
+@pytest.fixture(scope="function")
+def jury_list_mixed():
+    return Jury(metrics=_LIST_MIXED_TEST_METRICS)
+
+
+@pytest.fixture(scope="function")
+def jury_datasets():
+    return Jury(metrics=_DATASETS_METRICS)
+
+
+def json_load(path: str):
+    with open(path, "r") as jf:
+        content = json.load(jf)
+    return content
+
+
+def get_expected_output(prefix: Optional[str] = None):
+    def wrapper(fn, *args, **kwargs):
+        module_name = os.path.basename(inspect.getfile(fn)).replace(".py", "")
+        path = os.path.join(EXPECTED_OUTPUTS, prefix, f"{module_name}.json")
+        test_name = fn.__name__.replace("output_", "")
+        fn.output = json_load(path)[test_name]
+        return fn
+
+    if prefix is None:
+        prefix = ""
+    return wrapper
diff --git a/tests/jury/metrics/test_accuracy.py b/tests/jury/metrics/test_accuracy.py
new file mode 100644
index 0000000..1811d1f
--- /dev/null
+++ b/tests/jury/metrics/test_accuracy.py
@@ -0,0 +1,44 @@
+import pytest
+
+from jury import Jury
+from jury.metrics import Accuracy
+from tests.jury.conftest import get_expected_output
+from tests.utils import assert_almost_equal_dict
+
+
+@pytest.fixture(scope="module")
+def jury():
+    return Jury(metrics=[Accuracy()])
+
+
+@pytest.fixture
+@get_expected_output(prefix="metrics")
+def output_basic():
+    return output_basic.output
+
+
+@pytest.fixture
+@get_expected_output(prefix="metrics")
+def output_multiple_ref():
+    return output_multiple_ref.output
+
+
+@pytest.fixture
+@get_expected_output(prefix="metrics")
+def output_multiple_pred_multiple_ref():
+    return output_multiple_pred_multiple_ref.output
+
+
+def test_basic(predictions, references, jury, output_basic):
+    scores = jury(predictions=predictions, references=references)
+    assert_almost_equal_dict(actual=scores, desired=output_basic)
+
+
+def test_multiple_ref(predictions, multiple_references, jury, output_multiple_ref):
+    scores = jury(predictions=predictions, references=multiple_references)
+    assert_almost_equal_dict(actual=scores, desired=output_multiple_ref)
+
+
+def test_multiple_pred_multiple_ref(multiple_predictions, multiple_references, jury, output_multiple_pred_multiple_ref):
+    scores = jury(predictions=multiple_predictions, references=multiple_references)
+    assert_almost_equal_dict(actual=scores, desired=output_multiple_pred_multiple_ref)
diff --git a/tests/jury/metrics/test_bertscore.py b/tests/jury/metrics/test_bertscore.py
index e22da7c..b32d6db 100644
--- a/tests/jury/metrics/test_bertscore.py
+++ b/tests/jury/metrics/test_bertscore.py
@@ -1,31 +1,46 @@
 import pytest
 
 from jury import Jury
-from jury.metrics.bertscore import BERTScore
+from jury.metrics.bertscore import Bertscore
+from tests.jury.conftest import get_expected_output
 from tests.utils import assert_almost_equal_dict
 
 
 @pytest.fixture(scope="module")
 def jury():
-    return Jury(metrics=[BERTScore(params={"model_type": "albert-base-v1"})])
+    return Jury(metrics=[Bertscore(params={"model_type": "albert-base-v1", "device": "cpu"})])
 
 
-def test_basic(predictions, references, jury):
-    _EXPECTED_RESULT = {"empty_predictions": 0, "total_items": 2, "BERTScore": 0.8293454647064209}
+@pytest.fixture
+@get_expected_output(prefix="metrics")
+def output_basic():
+    return output_basic.output
 
-    scores = jury.evaluate(predictions, references)
-    assert_almost_equal_dict(actual=scores, desired=_EXPECTED_RESULT)
 
+@pytest.fixture
+@get_expected_output(prefix="metrics")
+def output_multiple_ref():
+    return output_multiple_ref.output
 
-def test_multiple_ref(predictions, multiple_references, jury):
-    _EXPECTED_RESULT = {"empty_predictions": 0, "total_items": 2, "BERTScore": 0.7350329756736755}
 
-    scores = jury.evaluate(predictions, multiple_references)
-    assert_almost_equal_dict(actual=scores, desired=_EXPECTED_RESULT)
+@pytest.fixture
+@get_expected_output(prefix="metrics")
+def output_multiple_pred_multiple_ref():
+    return output_multiple_pred_multiple_ref.output
 
 
-def test_multiple_pred_multiple_ref(multiple_predictions, multiple_references, jury):
-    _EXPECTED_RESULT = {"empty_predictions": 0, "total_items": 2, "BERTScore": 0.7431023120880127}
+def test_basic(predictions, references, jury, output_basic):
+    scores = jury(predictions=predictions, references=references)
+    assert_almost_equal_dict(actual=scores, desired=output_basic, exclude_paths="root['bertscore']['hashcode']")
 
-    scores = jury.evaluate(multiple_predictions, multiple_references)
-    assert_almost_equal_dict(actual=scores, desired=_EXPECTED_RESULT)
+
+def test_multiple_ref(predictions, multiple_references, jury, output_multiple_ref):
+    scores = jury(predictions=predictions, references=multiple_references)
+    assert_almost_equal_dict(actual=scores, desired=output_multiple_ref, exclude_paths="root['bertscore']['hashcode']")
+
+
+def test_multiple_pred_multiple_ref(multiple_predictions, multiple_references, jury, output_multiple_pred_multiple_ref):
+    scores = jury(predictions=multiple_predictions, references=multiple_references)
+    assert_almost_equal_dict(
+        actual=scores, desired=output_multiple_pred_multiple_ref, exclude_paths="root['bertscore']['hashcode']"
+    )
diff --git a/tests/jury/metrics/test_bleu.py b/tests/jury/metrics/test_bleu.py
index 11dddd2..18045d9 100644
--- a/tests/jury/metrics/test_bleu.py
+++ b/tests/jury/metrics/test_bleu.py
@@ -1,31 +1,44 @@
 import pytest
 
 from jury import Jury
-from jury.metrics.bleu import BLEU
+from jury.metrics.bleu import Bleu
+from tests.jury.conftest import get_expected_output
 from tests.utils import assert_almost_equal_dict
 
 
 @pytest.fixture(scope="module")
 def jury():
-    return Jury(metrics=[BLEU()])
+    return Jury(metrics=[Bleu()])
 
 
-def test_basic(predictions, references, jury):
-    _EXPECTED_RESULT = {"BLEU": 0.0, "empty_predictions": 0, "total_items": 2}
+@pytest.fixture
+@get_expected_output(prefix="metrics")
+def output_basic():
+    return output_basic.output
 
-    scores = jury.evaluate(predictions, references)
-    assert_almost_equal_dict(actual=scores, desired=_EXPECTED_RESULT)
 
+@pytest.fixture
+@get_expected_output(prefix="metrics")
+def output_multiple_ref():
+    return output_multiple_ref.output
 
-def test_multiple_ref(predictions, multiple_references, jury):
-    _EXPECTED_RESULT = {"BLEU": 0.0, "empty_predictions": 0, "total_items": 2}
 
-    scores = jury.evaluate(predictions, multiple_references)
-    assert_almost_equal_dict(actual=scores, desired=_EXPECTED_RESULT)
+@pytest.fixture
+@get_expected_output(prefix="metrics")
+def output_multiple_pred_multiple_ref():
+    return output_multiple_pred_multiple_ref.output
 
 
-def test_multiple_pred_multiple_ref(multiple_predictions, multiple_references, jury):
-    _EXPECTED_RESULT = {"empty_predictions": 0, "total_items": 2, "BLEU": 0.22749707020240187}
+def test_basic(predictions, references, jury, output_basic):
+    scores = jury(predictions=predictions, references=references)
+    assert_almost_equal_dict(actual=scores, desired=output_basic)
 
-    scores = jury.evaluate(multiple_predictions, multiple_references)
-    assert_almost_equal_dict(actual=scores, desired=_EXPECTED_RESULT)
+
+def test_multiple_ref(predictions, multiple_references, jury, output_multiple_ref):
+    scores = jury(predictions=predictions, references=multiple_references)
+    assert_almost_equal_dict(actual=scores, desired=output_multiple_ref)
+
+
+def test_multiple_pred_multiple_ref(multiple_predictions, multiple_references, jury, output_multiple_pred_multiple_ref):
+    scores = jury(predictions=multiple_predictions, references=multiple_references)
+    assert_almost_equal_dict(actual=scores, desired=output_multiple_pred_multiple_ref)
diff --git a/tests/jury/metrics/test_f1.py b/tests/jury/metrics/test_f1.py
new file mode 100644
index 0000000..129cfa9
--- /dev/null
+++ b/tests/jury/metrics/test_f1.py
@@ -0,0 +1,44 @@
+import pytest
+
+from jury import Jury
+from jury.metrics import F1
+from tests.jury.conftest import get_expected_output
+from tests.utils import assert_almost_equal_dict
+
+
+@pytest.fixture(scope="module")
+def jury():
+    return Jury(metrics=[F1()])
+
+
+@pytest.fixture
+@get_expected_output(prefix="metrics")
+def output_basic():
+    return output_basic.output
+
+
+@pytest.fixture
+@get_expected_output(prefix="metrics")
+def output_multiple_ref():
+    return output_multiple_ref.output
+
+
+@pytest.fixture
+@get_expected_output(prefix="metrics")
+def output_multiple_pred_multiple_ref():
+    return output_multiple_pred_multiple_ref.output
+
+
+def test_basic(predictions, references, jury, output_basic):
+    scores = jury(predictions=predictions, references=references)
+    assert_almost_equal_dict(actual=scores, desired=output_basic)
+
+
+def test_multiple_ref(predictions, multiple_references, jury, output_multiple_ref):
+    scores = jury(predictions=predictions, references=multiple_references)
+    assert_almost_equal_dict(actual=scores, desired=output_multiple_ref)
+
+
+def test_multiple_pred_multiple_ref(multiple_predictions, multiple_references, jury, output_multiple_pred_multiple_ref):
+    scores = jury(predictions=multiple_predictions, references=multiple_references)
+    assert_almost_equal_dict(actual=scores, desired=output_multiple_pred_multiple_ref)
diff --git a/tests/jury/metrics/test_meteor.py b/tests/jury/metrics/test_meteor.py
index ecd47c1..ed5203c 100644
--- a/tests/jury/metrics/test_meteor.py
+++ b/tests/jury/metrics/test_meteor.py
@@ -2,6 +2,7 @@
 
 from jury import Jury
 from jury.metrics.meteor import Meteor
+from tests.jury.conftest import get_expected_output
 from tests.utils import assert_almost_equal_dict
 
 
@@ -10,22 +11,34 @@ def jury():
     return Jury(metrics=[Meteor()])
 
 
-def test_basic(predictions, references, jury):
-    _EXPECTED_RESULT = {"empty_predictions": 0, "total_items": 2, "Meteor": 0.5098979591836734}
+@pytest.fixture
+@get_expected_output(prefix="metrics")
+def output_basic():
+    return output_basic.output
 
-    scores = jury.evaluate(predictions, references)
-    assert_almost_equal_dict(actual=scores, desired=_EXPECTED_RESULT)
 
+@pytest.fixture
+@get_expected_output(prefix="metrics")
+def output_multiple_ref():
+    return output_multiple_ref.output
 
-def test_multiple_ref(predictions, multiple_references, jury):
-    _EXPECTED_RESULT = {"empty_predictions": 0, "total_items": 2, "Meteor": 0.5098979591836734}
 
-    scores = jury.evaluate(predictions, multiple_references)
-    assert_almost_equal_dict(actual=scores, desired=_EXPECTED_RESULT)
+@pytest.fixture
+@get_expected_output(prefix="metrics")
+def output_multiple_pred_multiple_ref():
+    return output_multiple_pred_multiple_ref.output
 
 
-def test_multiple_pred_multiple_ref(multiple_predictions, multiple_references, jury):
-    _EXPECTED_RESULT = {"empty_predictions": 0, "total_items": 2, "Meteor": 0.5420511682934044}
+def test_basic(predictions, references, jury, output_basic):
+    scores = jury(predictions=predictions, references=references)
+    assert_almost_equal_dict(actual=scores, desired=output_basic)
 
-    scores = jury.evaluate(multiple_predictions, multiple_references)
-    assert_almost_equal_dict(actual=scores, desired=_EXPECTED_RESULT)
+
+def test_multiple_ref(predictions, multiple_references, jury, output_multiple_ref):
+    scores = jury(predictions=predictions, references=multiple_references)
+    assert_almost_equal_dict(actual=scores, desired=output_multiple_ref)
+
+
+def test_multiple_pred_multiple_ref(multiple_predictions, multiple_references, jury, output_multiple_pred_multiple_ref):
+    scores = jury(predictions=multiple_predictions, references=multiple_references)
+    assert_almost_equal_dict(actual=scores, desired=output_multiple_pred_multiple_ref)
diff --git a/tests/jury/metrics/test_precision.py b/tests/jury/metrics/test_precision.py
new file mode 100644
index 0000000..1515142
--- /dev/null
+++ b/tests/jury/metrics/test_precision.py
@@ -0,0 +1,44 @@
+import pytest
+
+from jury import Jury
+from jury.metrics.precision import Precision
+from tests.jury.conftest import get_expected_output
+from tests.utils import assert_almost_equal_dict
+
+
+@pytest.fixture(scope="module")
+def jury():
+    return Jury(metrics=[Precision()])
+
+
+@pytest.fixture
+@get_expected_output(prefix="metrics")
+def output_basic():
+    return output_basic.output
+
+
+@pytest.fixture
+@get_expected_output(prefix="metrics")
+def output_multiple_ref():
+    return output_multiple_ref.output
+
+
+@pytest.fixture
+@get_expected_output(prefix="metrics")
+def output_multiple_pred_multiple_ref():
+    return output_multiple_pred_multiple_ref.output
+
+
+def test_basic(predictions, references, jury, output_basic):
+    scores = jury(predictions=predictions, references=references)
+    assert_almost_equal_dict(actual=scores, desired=output_basic)
+
+
+def test_multiple_ref(predictions, multiple_references, jury, output_multiple_ref):
+    scores = jury(predictions=predictions, references=multiple_references)
+    assert_almost_equal_dict(actual=scores, desired=output_multiple_ref)
+
+
+def test_multiple_pred_multiple_ref(multiple_predictions, multiple_references, jury, output_multiple_pred_multiple_ref):
+    scores = jury(predictions=multiple_predictions, references=multiple_references)
+    assert_almost_equal_dict(actual=scores, desired=output_multiple_pred_multiple_ref)
diff --git a/tests/jury/metrics/test_recall.py b/tests/jury/metrics/test_recall.py
new file mode 100644
index 0000000..aea2e7d
--- /dev/null
+++ b/tests/jury/metrics/test_recall.py
@@ -0,0 +1,44 @@
+import pytest
+
+from jury import Jury
+from jury.metrics import Recall
+from tests.jury.conftest import get_expected_output
+from tests.utils import assert_almost_equal_dict
+
+
+@pytest.fixture(scope="module")
+def jury():
+    return Jury(metrics=[Recall()])
+
+
+@pytest.fixture
+@get_expected_output(prefix="metrics")
+def output_basic():
+    return output_basic.output
+
+
+@pytest.fixture
+@get_expected_output(prefix="metrics")
+def output_multiple_ref():
+    return output_multiple_ref.output
+
+
+@pytest.fixture
+@get_expected_output(prefix="metrics")
+def output_multiple_pred_multiple_ref():
+    return output_multiple_pred_multiple_ref.output
+
+
+def test_basic(predictions, references, jury, output_basic):
+    scores = jury(predictions=predictions, references=references)
+    assert_almost_equal_dict(actual=scores, desired=output_basic)
+
+
+def test_multiple_ref(predictions, multiple_references, jury, output_multiple_ref):
+    scores = jury(predictions=predictions, references=multiple_references)
+    assert_almost_equal_dict(actual=scores, desired=output_multiple_ref)
+
+
+def test_multiple_pred_multiple_ref(multiple_predictions, multiple_references, jury, output_multiple_pred_multiple_ref):
+    scores = jury(predictions=multiple_predictions, references=multiple_references)
+    assert_almost_equal_dict(actual=scores, desired=output_multiple_pred_multiple_ref)
diff --git a/tests/jury/metrics/test_rouge.py b/tests/jury/metrics/test_rouge.py
index 760c93b..2a4713e 100644
--- a/tests/jury/metrics/test_rouge.py
+++ b/tests/jury/metrics/test_rouge.py
@@ -2,6 +2,7 @@
 
 from jury import Jury
 from jury.metrics.rouge import Rouge
+from tests.jury.conftest import get_expected_output
 from tests.utils import assert_almost_equal_dict
 
 
@@ -10,22 +11,34 @@ def jury():
     return Jury(metrics=[Rouge()])
 
 
-def test_basic(predictions, references, jury):
-    _EXPECTED_RESULT = {"empty_predictions": 0, "total_items": 2, "rougeL": 0.6190476190476191}
+@pytest.fixture
+@get_expected_output(prefix="metrics")
+def output_basic():
+    return output_basic.output
 
-    scores = jury.evaluate(predictions, references)
-    assert_almost_equal_dict(actual=scores, desired=_EXPECTED_RESULT)
 
+@pytest.fixture
+@get_expected_output(prefix="metrics")
+def output_multiple_ref():
+    return output_multiple_ref.output
 
-def test_multiple_ref(predictions, multiple_references, jury):
-    _EXPECTED_RESULT = {"empty_predictions": 0, "total_items": 2, "rougeL": 0.6190476190476191}
 
-    scores = jury.evaluate(predictions, multiple_references)
-    assert_almost_equal_dict(actual=scores, desired=_EXPECTED_RESULT)
+@pytest.fixture
+@get_expected_output(prefix="metrics")
+def output_multiple_pred_multiple_ref():
+    return output_multiple_pred_multiple_ref.output
 
 
-def test_multiple_pred_multiple_ref(multiple_predictions, multiple_references, jury):
-    _EXPECTED_RESULT = {"empty_predictions": 0, "total_items": 2, "rougeL": 0.7948717948717948}
+def test_basic(predictions, references, jury, output_basic):
+    scores = jury(predictions=predictions, references=references)
+    assert_almost_equal_dict(actual=scores, desired=output_basic)
 
-    scores = jury.evaluate(multiple_predictions, multiple_references)
-    assert_almost_equal_dict(actual=scores, desired=_EXPECTED_RESULT)
+
+def test_multiple_ref(predictions, multiple_references, jury, output_multiple_ref):
+    scores = jury(predictions=predictions, references=multiple_references)
+    assert_almost_equal_dict(actual=scores, desired=output_multiple_ref)
+
+
+def test_multiple_pred_multiple_ref(multiple_predictions, multiple_references, jury, output_multiple_pred_multiple_ref):
+    scores = jury(predictions=multiple_predictions, references=multiple_references)
+    assert_almost_equal_dict(actual=scores, desired=output_multiple_pred_multiple_ref)
diff --git a/tests/jury/metrics/test_sacrebleu.py b/tests/jury/metrics/test_sacrebleu.py
index 5ad2c95..4e8d952 100644
--- a/tests/jury/metrics/test_sacrebleu.py
+++ b/tests/jury/metrics/test_sacrebleu.py
@@ -1,31 +1,44 @@
 import pytest
 
 from jury import Jury
-from jury.metrics.sacrebleu import SacreBLEU
+from jury.metrics.sacrebleu import Sacrebleu
+from tests.jury.conftest import get_expected_output
 from tests.utils import assert_almost_equal_dict
 
 
 @pytest.fixture(scope="module")
 def jury():
-    return Jury(metrics=[SacreBLEU()])
+    return Jury(metrics=[Sacrebleu()])
 
 
-def test_basic(predictions, references, jury):
-    _EXPECTED_RESULT = {"SacreBLEU": 0.3165801094780895, "empty_predictions": 0, "total_items": 2}
+@pytest.fixture
+@get_expected_output(prefix="metrics")
+def output_basic():
+    return output_basic.output
 
-    scores = jury.evaluate(predictions, references)
-    assert_almost_equal_dict(actual=scores, desired=_EXPECTED_RESULT)
 
+@pytest.fixture
+@get_expected_output(prefix="metrics")
+def output_multiple_ref():
+    return output_multiple_ref.output
 
-def test_multiple_ref(predictions, multiple_references, jury):
-    _EXPECTED_RESULT = {"SacreBLEU": 0.3250296630171846, "empty_predictions": 0, "total_items": 2}
 
-    scores = jury.evaluate(predictions, multiple_references)
-    assert_almost_equal_dict(actual=scores, desired=_EXPECTED_RESULT)
+@pytest.fixture
+@get_expected_output(prefix="metrics")
+def output_multiple_pred_multiple_ref():
+    return output_multiple_pred_multiple_ref.output
 
 
-def test_multiple_pred_multiple_ref(multiple_predictions, multiple_references, jury):
-    _EXPECTED_RESULT = {"empty_predictions": 0, "total_items": 2, "SacreBLEU": 0.3898310279399514}
+def test_basic(predictions, references, jury, output_basic):
+    scores = jury(predictions=predictions, references=references)
+    assert_almost_equal_dict(actual=scores, desired=output_basic)
 
-    scores = jury.evaluate(multiple_predictions, multiple_references)
-    assert_almost_equal_dict(actual=scores, desired=_EXPECTED_RESULT)
+
+def test_multiple_ref(predictions, multiple_references, jury, output_multiple_ref):
+    scores = jury(predictions=predictions, references=multiple_references)
+    assert_almost_equal_dict(actual=scores, desired=output_multiple_ref)
+
+
+def test_multiple_pred_multiple_ref(multiple_predictions, multiple_references, jury, output_multiple_pred_multiple_ref):
+    scores = jury(predictions=multiple_predictions, references=multiple_references)
+    assert_almost_equal_dict(actual=scores, desired=output_multiple_pred_multiple_ref)
diff --git a/tests/jury/metrics/test_squad.py b/tests/jury/metrics/test_squad.py
index 7e71c42..44723a1 100644
--- a/tests/jury/metrics/test_squad.py
+++ b/tests/jury/metrics/test_squad.py
@@ -1,13 +1,14 @@
 import pytest
 
 from jury import Jury
-from jury.metrics.squad import SQUAD_F1
+from jury.metrics.squad import Squad
+from tests.jury.conftest import get_expected_output
 from tests.utils import assert_almost_equal_dict
 
 
 @pytest.fixture(scope="module")
 def jury():
-    return Jury(metrics=[SQUAD_F1()])
+    return Jury(metrics=[Squad()])
 
 
 @pytest.fixture
@@ -22,51 +23,42 @@ def references():
 
 @pytest.fixture
 def multiple_references():
-    return ["1917", ["Einstein", "Albert Einstein"]]
+    return [["1917"], ["Einstein", "Albert Einstein"]]
 
 
 @pytest.fixture
 def multiple_predictions():
-    return [["1917", "November 1917"], "Albert Einstein"]
-
-
-@pytest.fixture()
-def squad_style_predictions():
-    return [{"prediction_text": "1976", "id": "56e10a3be3433e1400422b22"}]
+    return [["1917", "November 1917"], ["Albert Einstein"]]
 
 
 @pytest.fixture
-def squad_style_references():
-    return [{"answers": {"answer_start": [97], "text": ["1976"]}, "id": "56e10a3be3433e1400422b22"}]
-
+@get_expected_output(prefix="metrics")
+def output_basic():
+    return output_basic.output
 
-def test_basic_str_input(predictions, references, jury):
-    _EXPECTED_RESULT = {
-        "empty_predictions": 0,
-        "total_items": 2,
-        "squad_f1": 0.8333333333333333,
-    }
-
-    scores = jury.evaluate(predictions, references)
-    assert_almost_equal_dict(_EXPECTED_RESULT, scores)
 
+@pytest.fixture
+@get_expected_output(prefix="metrics")
+def output_multiple_ref():
+    return output_multiple_ref.output
 
-def test_basic_dict_input(squad_style_predictions, squad_style_references, jury):
-    _EXPECTED_RESULT = {"empty_predictions": 0, "total_items": 1, "squad_f1": 1.0}
 
-    scores = jury.evaluate(squad_style_predictions, squad_style_references)
-    assert_almost_equal_dict(actual=scores, desired=_EXPECTED_RESULT)
+@pytest.fixture
+@get_expected_output(prefix="metrics")
+def output_multiple_pred_multiple_ref():
+    return output_multiple_pred_multiple_ref.output
 
 
-def test_multiple_ref(predictions, multiple_references, jury):
-    _EXPECTED_RESULT = {"empty_predictions": 0, "total_items": 2, "squad_f1": 1.0}
+def test_basic(predictions, references, jury, output_basic):
+    scores = jury(predictions=predictions, references=references)
+    assert_almost_equal_dict(actual=scores, desired=output_basic)
 
-    scores = jury.evaluate(predictions, multiple_references)
-    assert_almost_equal_dict(actual=scores, desired=_EXPECTED_RESULT)
 
+def test_multiple_ref(predictions, multiple_references, jury, output_multiple_ref):
+    scores = jury(predictions=predictions, references=multiple_references)
+    assert_almost_equal_dict(actual=scores, desired=output_multiple_ref)
 
-def test_multiple_pred_multiple_ref(multiple_predictions, multiple_references, jury):
-    _EXPECTED_RESULT = {"empty_predictions": 0, "total_items": 2, "squad_f1": 0.5}
 
-    scores = jury.evaluate(multiple_predictions, multiple_references)
-    assert_almost_equal_dict(actual=scores, desired=_EXPECTED_RESULT)
+def test_multiple_pred_multiple_ref(multiple_predictions, multiple_references, jury, output_multiple_pred_multiple_ref):
+    scores = jury(predictions=multiple_predictions, references=multiple_references)
+    assert_almost_equal_dict(actual=scores, desired=output_multiple_pred_multiple_ref)
diff --git a/tests/jury/test_cli.py b/tests/jury/test_cli.py
index 1627862..f79df89 100644
--- a/tests/jury/test_cli.py
+++ b/tests/jury/test_cli.py
@@ -2,30 +2,22 @@
 
 import pytest
 
-from jury import TEST_DATA_DIR
+from tests.jury import TEST_DATA_DIR
+from tests.jury.conftest import get_expected_output
 from tests.utils import assert_almost_equal_dict, shell_capture
 
 
 @pytest.fixture
-def desired_output():
-    return {
-        "empty_predictions": 0,
-        "total_items": 2,
-        "bleu_1": 0.6666666666666666,
-        "bleu_2": 0.5063696835418333,
-        "bleu_3": 0.4119912453171404,
-        "bleu_4": 0.29689669509442307,
-        "Meteor": 0.6594164989939638,
-        "rougeL": 0.6190476190476191,
-        "SacreBLEU": 0.4765798792330085,
-    }
-
-
-def test_cli(desired_output):
-    predictions_file = Path(TEST_DATA_DIR) / "predictions.txt"
-    references_file = Path(TEST_DATA_DIR) / "references.txt"
+@get_expected_output(prefix=None)
+def output_cli():
+    return output_cli.output
+
+
+def test_cli(output_cli):
+    predictions_file = Path(TEST_DATA_DIR) / "cli" / "predictions.txt"
+    references_file = Path(TEST_DATA_DIR) / "cli" / "references.txt"
 
     cmd = f"jury eval --predictions {predictions_file} --references {references_file}"
 
     out = shell_capture(cmd)
-    assert_almost_equal_dict(desired_output, out)
+    assert_almost_equal_dict(actual=out, desired=output_cli)
diff --git a/tests/test_data/predictions.txt b/tests/jury/test_data/cli/predictions.txt
similarity index 100%
rename from tests/test_data/predictions.txt
rename to tests/jury/test_data/cli/predictions.txt
diff --git a/tests/test_data/references.txt b/tests/jury/test_data/cli/references.txt
similarity index 100%
rename from tests/test_data/references.txt
rename to tests/jury/test_data/cli/references.txt
diff --git a/tests/jury/test_data/expected_outputs/metrics/test_accuracy.json b/tests/jury/test_data/expected_outputs/metrics/test_accuracy.json
new file mode 100644
index 0000000..01c0e1d
--- /dev/null
+++ b/tests/jury/test_data/expected_outputs/metrics/test_accuracy.json
@@ -0,0 +1,23 @@
+{
+  "basic": {
+    "empty_predictions": 0,
+    "total_items": 2,
+    "accuracy": {
+      "score": 0.6571428571428571
+    }
+  },
+  "multiple_ref": {
+    "empty_predictions": 0,
+    "total_items": 2,
+    "accuracy": {
+      "score": 0.6571428571428571
+    }
+  },
+  "multiple_pred_multiple_ref": {
+    "empty_predictions": 0,
+    "total_items": 2,
+    "accuracy": {
+      "score": 0.7285714285714285
+    }
+  }
+}
diff --git a/tests/jury/test_data/expected_outputs/metrics/test_bertscore.json b/tests/jury/test_data/expected_outputs/metrics/test_bertscore.json
new file mode 100644
index 0000000..8a8e990
--- /dev/null
+++ b/tests/jury/test_data/expected_outputs/metrics/test_bertscore.json
@@ -0,0 +1,53 @@
+{
+  "basic": {
+    "empty_predictions": 0,
+    "total_items": 2,
+    "bertscore": {
+      "score": 0.7350329756736755,
+      "precision": [
+        0.832823634147644,
+        0.6074944734573364
+      ],
+      "recall": [
+        0.8258962631225586,
+        0.6777912378311157
+      ],
+      "f1": [
+        0.8293454647064209,
+        0.6407204866409302
+      ],
+      "hashcode": "albert-base-v1_L10_no-idf_version=0.3.9(hug_trans=4.9.1)"
+    }
+  },
+  "multiple_ref": {
+    "empty_predictions": 0,
+    "total_items": 2,
+    "bertscore": {
+      "score": 0.7350329756736755,
+      "precision": [
+        0.832823634147644,
+        0.6074944734573364
+      ],
+      "recall": [
+        0.8300701379776001,
+        0.6777912378311157
+      ],
+      "f1": [
+        0.8293454647064209,
+        0.6407204866409302
+      ],
+      "hashcode": "albert-base-v1_L10_no-idf_version=0.3.9(hug_trans=4.9.1)"
+    }
+  },
+  "multiple_pred_multiple_ref": {
+    "empty_predictions": 0,
+    "total_items": 2,
+    "bertscore": {
+      "score": 0.7485644221305847,
+      "precision": 0.7493639290332794,
+      "recall": 0.7577464580535889,
+      "f1": 0.7485644221305847,
+      "hashcode": "albert-base-v1_L10_no-idf_version=0.3.9(hug_trans=4.9.1)"
+    }
+  }
+}
diff --git a/tests/jury/test_data/expected_outputs/metrics/test_bleu.json b/tests/jury/test_data/expected_outputs/metrics/test_bleu.json
new file mode 100644
index 0000000..559b3d5
--- /dev/null
+++ b/tests/jury/test_data/expected_outputs/metrics/test_bleu.json
@@ -0,0 +1,53 @@
+{
+  "basic": {
+    "empty_predictions": 0,
+    "total_items": 2,
+    "bleu": {
+      "score": 0.0,
+      "precisions": [
+        0.7272727272727273,
+        0.4444444444444444,
+        0.2857142857142857,
+        0.0
+      ],
+      "brevity_penalty": 0.9131007162822622,
+      "length_ratio": 0.9166666666666666,
+      "translation_length": 11,
+      "reference_length": 12
+    }
+  },
+  "multiple_ref": {
+    "empty_predictions": 0,
+    "total_items": 2,
+    "bleu": {
+      "score": 0.0,
+      "precisions": [
+        0.7272727272727273,
+        0.4444444444444444,
+        0.2857142857142857,
+        0.0
+      ],
+      "brevity_penalty": 1.0,
+      "length_ratio": 1.0,
+      "translation_length": 11,
+      "reference_length": 11
+    }
+  },
+  "multiple_pred_multiple_ref": {
+    "empty_predictions": 0,
+    "total_items": 2,
+    "bleu": {
+      "score": 0.42370250917168295,
+      "precisions": [
+        0.8823529411764706,
+        0.6428571428571429,
+        0.45454545454545453,
+        0.125
+      ],
+      "brevity_penalty": 1.0,
+      "length_ratio": 1.0,
+      "translation_length": 11,
+      "reference_length": 11
+    }
+  }
+}
diff --git a/tests/jury/test_data/expected_outputs/metrics/test_f1.json b/tests/jury/test_data/expected_outputs/metrics/test_f1.json
new file mode 100644
index 0000000..6377b83
--- /dev/null
+++ b/tests/jury/test_data/expected_outputs/metrics/test_f1.json
@@ -0,0 +1,23 @@
+{
+  "basic": {
+    "empty_predictions": 0,
+    "total_items": 2,
+    "f1": {
+      "score": 0.6926184355490269
+    }
+  },
+  "multiple_ref": {
+    "empty_predictions": 0,
+    "total_items": 2,
+    "f1": {
+      "score": 0.6904761904761905
+    }
+  },
+  "multiple_pred_multiple_ref": {
+    "empty_predictions": 0,
+    "total_items": 2,
+    "f1": {
+      "score": 0.7948717948717947
+    }
+  }
+}
diff --git a/tests/jury/test_data/expected_outputs/metrics/test_meteor.json b/tests/jury/test_data/expected_outputs/metrics/test_meteor.json
new file mode 100644
index 0000000..baeebd9
--- /dev/null
+++ b/tests/jury/test_data/expected_outputs/metrics/test_meteor.json
@@ -0,0 +1,23 @@
+{
+  "basic": {
+    "empty_predictions": 0,
+    "total_items": 2,
+    "meteor": {
+      "score": 0.5098979591836734
+    }
+  },
+  "multiple_ref": {
+    "empty_predictions": 0,
+    "total_items": 2,
+    "meteor": {
+      "score": 0.5098979591836734
+    }
+  },
+  "multiple_pred_multiple_ref": {
+    "empty_predictions": 0,
+    "total_items": 2,
+    "meteor": {
+      "score": 0.5420511682934044
+    }
+  }
+}
diff --git a/tests/jury/test_data/expected_outputs/metrics/test_precision.json b/tests/jury/test_data/expected_outputs/metrics/test_precision.json
new file mode 100644
index 0000000..832c20d
--- /dev/null
+++ b/tests/jury/test_data/expected_outputs/metrics/test_precision.json
@@ -0,0 +1,23 @@
+{
+  "basic": {
+    "empty_predictions": 0,
+    "total_items": 2,
+    "precision": {
+      "score": 0.7321428571428572
+    }
+  },
+  "multiple_ref": {
+    "empty_predictions": 0,
+    "total_items": 2,
+    "precision": {
+      "score": 0.7321428571428572
+    }
+  },
+  "multiple_pred_multiple_ref": {
+    "empty_predictions": 0,
+    "total_items": 2,
+    "precision": {
+      "score": 0.875
+    }
+  }
+}
diff --git a/tests/jury/test_data/expected_outputs/metrics/test_recall.json b/tests/jury/test_data/expected_outputs/metrics/test_recall.json
new file mode 100644
index 0000000..2805a5b
--- /dev/null
+++ b/tests/jury/test_data/expected_outputs/metrics/test_recall.json
@@ -0,0 +1,23 @@
+{
+  "basic": {
+    "empty_predictions": 0,
+    "total_items": 2,
+    "recall": {
+      "score": 0.6571428571428571
+    }
+  },
+  "multiple_ref": {
+    "empty_predictions": 0,
+    "total_items": 2,
+    "recall": {
+      "score": 0.6571428571428571
+    }
+  },
+  "multiple_pred_multiple_ref": {
+    "empty_predictions": 0,
+    "total_items": 2,
+    "recall": {
+      "score": 0.7285714285714285
+    }
+  }
+}
diff --git a/tests/jury/test_data/expected_outputs/metrics/test_rouge.json b/tests/jury/test_data/expected_outputs/metrics/test_rouge.json
new file mode 100644
index 0000000..ae74ce4
--- /dev/null
+++ b/tests/jury/test_data/expected_outputs/metrics/test_rouge.json
@@ -0,0 +1,32 @@
+{
+  "basic": {
+    "empty_predictions": 0,
+    "total_items": 2,
+    "rouge": {
+      "rouge1": 0.6904761904761905,
+      "rouge2": 0.45238095238095244,
+      "rougeL": 0.619047619047619,
+      "rougeLsum": 0.619047619047619
+    }
+  },
+  "multiple_ref": {
+    "empty_predictions": 0,
+    "total_items": 2,
+    "rouge": {
+      "rouge1": 0.6904761904761905,
+      "rouge2": 0.4675324675324676,
+      "rougeL": 0.641025641025641,
+      "rougeLsum": 0.641025641025641
+    }
+  },
+  "multiple_pred_multiple_ref": {
+    "empty_predictions": 0,
+    "total_items": 2,
+    "rouge": {
+      "rouge1": 0.7783882783882783,
+      "rouge2": 0.5925324675324675,
+      "rougeL": 0.7426739926739926,
+      "rougeLsum": 0.7426739926739926
+    }
+  }
+}
diff --git a/tests/jury/test_data/expected_outputs/metrics/test_sacrebleu.json b/tests/jury/test_data/expected_outputs/metrics/test_sacrebleu.json
new file mode 100644
index 0000000..78f87dd
--- /dev/null
+++ b/tests/jury/test_data/expected_outputs/metrics/test_sacrebleu.json
@@ -0,0 +1,92 @@
+{
+  "basic": {
+    "empty_predictions": 0,
+    "total_items": 2,
+    "sacrebleu": {
+      "score": 0.42728700639623396,
+      "counts": [
+        7,
+        4,
+        2,
+        1
+      ],
+      "totals": [
+        8,
+        7,
+        6,
+        5
+      ],
+      "precisions": [
+        0.875,
+        0.5714285714285715,
+        0.33333333333333337,
+        0.2
+      ],
+      "bp": 1.0,
+      "sys_len": 8,
+      "ref_len": 8
+    }
+  },
+  "multiple_ref": {
+    "empty_predictions": 0,
+    "total_items": 2,
+    "sacrebleu": {
+      "score": 0.2705411345269698,
+      "counts": [
+        9,
+        4,
+        2,
+        1
+      ],
+      "totals": [
+        14,
+        12,
+        10,
+        8
+      ],
+      "precisions": [
+        0.6428571428571429,
+        0.33333333333333337,
+        0.2,
+        0.125
+      ],
+      "bp": 1.0,
+      "sys_len": 14,
+      "ref_len": 14
+    }
+  },
+  "multiple_pred_multiple_ref": {
+    "empty_predictions": 0,
+    "total_items": 2,
+    "sacrebleu": {
+      "score": 0.32377227131456443,
+      "counts": [
+        11,
+        6,
+        3,
+        0
+      ],
+      "totals": [
+        13,
+        11,
+        9,
+        7
+      ],
+      "precisions": [
+        0.8461538461538461,
+        0.5454545454545454,
+        0.33333333333333337,
+        0.07142857142857144
+      ],
+      "bp": 1.0,
+      "sys_len": 11,
+      "ref_len": 12,
+      "adjusted_precisions": [
+        0.8461538461538461,
+        0.5454545454545454,
+        0.33333333333333337,
+        0.07142857142857144
+      ]
+    }
+  }
+}
diff --git a/tests/jury/test_data/expected_outputs/metrics/test_squad.json b/tests/jury/test_data/expected_outputs/metrics/test_squad.json
new file mode 100644
index 0000000..c447952
--- /dev/null
+++ b/tests/jury/test_data/expected_outputs/metrics/test_squad.json
@@ -0,0 +1,26 @@
+{
+  "basic": {
+    "empty_predictions": 0,
+    "squad": {
+      "exact_match": 0.5,
+      "f1": 0.8333333333333333
+    },
+    "total_items": 2
+  },
+  "multiple_ref": {
+    "empty_predictions": 0,
+    "squad": {
+      "exact_match": 1.0,
+      "f1": 1.0
+    },
+    "total_items": 2
+  },
+  "multiple_pred_multiple_ref": {
+    "empty_predictions": 0,
+    "squad": {
+      "exact_match": 1.0,
+      "f1": 1.0
+    },
+    "total_items": 2
+  }
+}
diff --git a/tests/jury/test_data/expected_outputs/test_cli.json b/tests/jury/test_data/expected_outputs/test_cli.json
new file mode 100644
index 0000000..93a11a1
--- /dev/null
+++ b/tests/jury/test_data/expected_outputs/test_cli.json
@@ -0,0 +1,67 @@
+{
+  "cli": {
+    "empty_predictions": 0,
+    "total_items": 2,
+    "bleu_1": {
+      "score": 0.29689669509442307,
+      "precisions": [
+        0.6666666666666666,
+        0.38461538461538464,
+        0.2727272727272727,
+        0.1111111111111111
+      ],
+      "brevity_penalty": 1.0,
+      "length_ratio": 1.0714285714285714,
+      "translation_length": 15,
+      "reference_length": 14
+    },
+    "bleu_2": {
+      "score": 0.29689669509442307,
+      "precisions": [
+        0.6666666666666666,
+        0.38461538461538464,
+        0.2727272727272727,
+        0.1111111111111111
+      ],
+      "brevity_penalty": 1.0,
+      "length_ratio": 1.0714285714285714,
+      "translation_length": 15,
+      "reference_length": 14
+    },
+    "bleu_3": {
+      "score": 0.29689669509442307,
+      "precisions": [
+        0.6666666666666666,
+        0.38461538461538464,
+        0.2727272727272727,
+        0.1111111111111111
+      ],
+      "brevity_penalty": 1.0,
+      "length_ratio": 1.0714285714285714,
+      "translation_length": 15,
+      "reference_length": 14
+    },
+    "bleu_4": {
+      "score": 0.29689669509442307,
+      "precisions": [
+        0.6666666666666666,
+        0.38461538461538464,
+        0.2727272727272727,
+        0.1111111111111111
+      ],
+      "brevity_penalty": 1.0,
+      "length_ratio": 1.0714285714285714,
+      "translation_length": 15,
+      "reference_length": 14
+    },
+    "meteor": {
+      "score": 0.6594164989939638
+    },
+    "rouge": {
+      "rouge1": 0.6904761904761905,
+      "rouge2": 0.3974358974358974,
+      "rougeL": 0.6190476190476191,
+      "rougeLsum": 0.6190476190476191
+    }
+  }
+}
\ No newline at end of file
diff --git a/tests/jury/test_data/expected_outputs/test_jury.json b/tests/jury/test_data/expected_outputs/test_jury.json
new file mode 100644
index 0000000..cc0924e
--- /dev/null
+++ b/tests/jury/test_data/expected_outputs/test_jury.json
@@ -0,0 +1,608 @@
+{
+  "evaluate": {
+    "empty_predictions": 0,
+    "total_items": 2,
+    "accuracy": {
+      "score": 0.6571428571428571
+    },
+    "bertscore": {
+      "score": 0.7350330352783203,
+      "precision": [
+        0.8328237533569336,
+        0.6074944734573364
+      ],
+      "recall": [
+        0.8258965015411377,
+        0.6777911186218262
+      ],
+      "f1": [
+        0.8293456435203552,
+        0.6407204270362854
+      ],
+      "hashcode": "albert-base-v1_L10_no-idf_version=0.3.10(hug_trans=4.9.1)"
+    },
+    "bleu": {
+      "score": 0.0,
+      "precisions": [
+        0.7272727272727273,
+        0.4444444444444444,
+        0.2857142857142857,
+        0.0
+      ],
+      "brevity_penalty": 0.9131007162822622,
+      "length_ratio": 0.9166666666666666,
+      "translation_length": 11,
+      "reference_length": 12
+    },
+    "f1": {
+      "score": 0.6926184355490269
+    },
+    "meteor": {
+      "score": 0.5098979591836734
+    },
+    "precision": {
+      "score": 0.7321428571428572
+    },
+    "recall": {
+      "score": 0.6571428571428571
+    },
+    "rouge": {
+      "rouge1": 0.6904761904761905,
+      "rouge2": 0.45238095238095244,
+      "rougeL": 0.619047619047619,
+      "rougeLsum": 0.619047619047619
+    },
+    "sacrebleu": {
+      "score": 0.42728700639623396,
+      "counts": [
+        7,
+        4,
+        2,
+        1
+      ],
+      "totals": [
+        8,
+        7,
+        6,
+        5
+      ],
+      "precisions": [
+        0.875,
+        0.5714285714285715,
+        0.33333333333333337,
+        0.2
+      ],
+      "bp": 1.0,
+      "sys_len": 8,
+      "ref_len": 8
+    },
+    "squad": {
+      "exact_match": 0.0,
+      "f1": 0.6857142857142858
+    }
+  },
+  "evaluate_concurrent": {
+    "empty_predictions": 0,
+    "total_items": 2,
+    "accuracy": {
+      "score": 0.6571428571428571
+    },
+    "bertscore": {
+      "score": 0.7350330352783203,
+      "precision": [
+        0.8328237533569336,
+        0.6074944734573364
+      ],
+      "recall": [
+        0.8258965015411377,
+        0.6777911186218262
+      ],
+      "f1": [
+        0.8293456435203552,
+        0.6407204270362854
+      ],
+      "hashcode": "albert-base-v1_L10_no-idf_version=0.3.10(hug_trans=4.9.1)"
+    },
+    "bleu": {
+      "score": 0.0,
+      "precisions": [
+        0.7272727272727273,
+        0.4444444444444444,
+        0.2857142857142857,
+        0.0
+      ],
+      "brevity_penalty": 0.9131007162822622,
+      "length_ratio": 0.9166666666666666,
+      "translation_length": 11,
+      "reference_length": 12
+    },
+    "f1": {
+      "score": 0.6926184355490269
+    },
+    "meteor": {
+      "score": 0.5098979591836734
+    },
+    "precision": {
+      "score": 0.7321428571428572
+    },
+    "recall": {
+      "score": 0.6571428571428571
+    },
+    "rouge": {
+      "rouge1": 0.6904761904761905,
+      "rouge2": 0.45238095238095244,
+      "rougeL": 0.619047619047619,
+      "rougeLsum": 0.619047619047619
+    },
+    "sacrebleu": {
+      "score": 0.42728700639623396,
+      "counts": [
+        7,
+        4,
+        2,
+        1
+      ],
+      "totals": [
+        8,
+        7,
+        6,
+        5
+      ],
+      "precisions": [
+        0.875,
+        0.5714285714285715,
+        0.33333333333333337,
+        0.2
+      ],
+      "bp": 1.0,
+      "sys_len": 8,
+      "ref_len": 8
+    },
+    "squad": {
+      "exact_match": 0.0,
+      "f1": 0.6857142857142858
+    }
+  },
+  "evaluate_str_input": {
+    "empty_predictions": 0,
+    "total_items": 2,
+    "bleu": {
+      "score": 0.0,
+      "precisions": [
+        0.7272727272727273,
+        0.4444444444444444,
+        0.2857142857142857,
+        0.0
+      ],
+      "brevity_penalty": 0.9131007162822622,
+      "length_ratio": 0.9166666666666666,
+      "translation_length": 11,
+      "reference_length": 12
+    }
+  },
+  "evaluate_list_str_input": {
+    "empty_predictions": 0,
+    "total_items": 2,
+    "accuracy": {
+      "score": 0.6571428571428571
+    },
+    "bertscore": {
+      "score": 0.9396075010299683,
+      "precision": [
+        0.9543163180351257,
+        0.9277863502502441
+      ],
+      "recall": [
+        0.9502041935920715,
+        0.9261335134506226
+      ],
+      "f1": [
+        0.9522558450698853,
+        0.9269591569900513
+      ],
+      "hashcode": "roberta-large_L17_no-idf_version=0.3.10(hug_trans=4.9.1)"
+    },
+    "bleu": {
+      "score": 0.0,
+      "precisions": [
+        0.7272727272727273,
+        0.4444444444444444,
+        0.2857142857142857,
+        0.0
+      ],
+      "brevity_penalty": 0.9131007162822622,
+      "length_ratio": 0.9166666666666666,
+      "translation_length": 11,
+      "reference_length": 12
+    },
+    "f1": {
+      "score": 0.6926184355490269
+    },
+    "meteor": {
+      "score": 0.5098979591836734
+    },
+    "precision": {
+      "score": 0.7321428571428572
+    },
+    "recall": {
+      "score": 0.6571428571428571
+    },
+    "rouge": {
+      "rouge1": 0.6904761904761905,
+      "rouge2": 0.45238095238095244,
+      "rougeL": 0.619047619047619,
+      "rougeLsum": 0.619047619047619
+    },
+    "sacrebleu": {
+      "score": 0.42728700639623396,
+      "counts": [
+        7,
+        4,
+        2,
+        1
+      ],
+      "totals": [
+        8,
+        7,
+        6,
+        5
+      ],
+      "precisions": [
+        0.875,
+        0.5714285714285715,
+        0.33333333333333337,
+        0.2
+      ],
+      "bp": 1.0,
+      "sys_len": 8,
+      "ref_len": 8
+    },
+    "squad": {
+      "exact_match": 0.0,
+      "f1": 0.6857142857142858
+    }
+  },
+  "evaluate_list_dict_input": {
+    "empty_predictions": 0,
+    "total_items": 2,
+    "accuracy": {
+      "score": 0.6571428571428571
+    },
+    "bertscore": {
+      "score": 0.7350329756736755,
+      "precision": [
+        0.832823634147644,
+        0.6074944734573364
+      ],
+      "recall": [
+        0.8258962631225586,
+        0.6777912378311157
+      ],
+      "f1": [
+        0.8293454647064209,
+        0.6407204866409302
+      ],
+      "hashcode": "albert-base-v1_L10_no-idf_version=0.3.10(hug_trans=4.9.1)"
+    },
+    "bleu-1": {
+      "score": 0.0,
+      "precisions": [
+        0.7272727272727273,
+        0.4444444444444444,
+        0.2857142857142857,
+        0.0
+      ],
+      "brevity_penalty": 0.9131007162822622,
+      "length_ratio": 0.9166666666666666,
+      "translation_length": 11,
+      "reference_length": 12
+    },
+    "bleu-2": {
+      "score": 0.0,
+      "precisions": [
+        0.7272727272727273,
+        0.4444444444444444,
+        0.2857142857142857,
+        0.0
+      ],
+      "brevity_penalty": 0.9131007162822622,
+      "length_ratio": 0.9166666666666666,
+      "translation_length": 11,
+      "reference_length": 12
+    },
+    "F1": {
+      "score": 0.6926184355490269
+    },
+    "METEOR": {
+      "score": 0.5098979591836734
+    },
+    "precision": {
+      "score": 0.7321428571428572
+    },
+    "recall": {
+      "score": 0.6571428571428571
+    },
+    "rouge": {
+      "rouge1": 0.6904761904761905,
+      "rouge2": 0.45238095238095244,
+      "rougeL": 0.619047619047619,
+      "rougeLsum": 0.619047619047619
+    },
+    "sacrebleu": {
+      "score": 0.42728700639623396,
+      "counts": [
+        7,
+        4,
+        2,
+        1
+      ],
+      "totals": [
+        8,
+        7,
+        6,
+        5
+      ],
+      "precisions": [
+        0.875,
+        0.5714285714285715,
+        0.33333333333333337,
+        0.2
+      ],
+      "bp": 1.0,
+      "sys_len": 8,
+      "ref_len": 8
+    },
+    "squad": {
+      "exact_match": 0.0,
+      "f1": 0.6857142857142858
+    }
+  },
+  "evaluate_list_mixed_input": {
+    "empty_predictions": 0,
+    "total_items": 2,
+    "accuracy": {
+      "score": 0.6571428571428571
+    },
+    "bertscore": {
+      "score": 0.9396075010299683,
+      "precision": [
+        0.9543163180351257,
+        0.9277863502502441
+      ],
+      "recall": [
+        0.9502041935920715,
+        0.9261335134506226
+      ],
+      "f1": [
+        0.9522558450698853,
+        0.9269591569900513
+      ],
+      "hashcode": "roberta-large_L17_no-idf_version=0.3.10(hug_trans=4.9.1)"
+    },
+    "bleu": {
+      "score": 0.0,
+      "precisions": [
+        0.7272727272727273,
+        0.4444444444444444,
+        0.2857142857142857,
+        0.0
+      ],
+      "brevity_penalty": 0.9131007162822622,
+      "length_ratio": 0.9166666666666666,
+      "translation_length": 11,
+      "reference_length": 12
+    },
+    "f1": {
+      "score": 0.6926184355490269
+    },
+    "meteor": {
+      "score": 0.5098979591836734
+    },
+    "precision": {
+      "score": 0.7321428571428572
+    },
+    "recall": {
+      "score": 0.6571428571428571
+    },
+    "rouge": {
+      "rouge1": 0.6904761904761905,
+      "rouge2": 0.45238095238095244,
+      "rougeL": 0.619047619047619,
+      "rougeLsum": 0.619047619047619
+    },
+    "sacrebleu": {
+      "score": 0.42728700639623396,
+      "counts": [
+        7,
+        4,
+        2,
+        1
+      ],
+      "totals": [
+        8,
+        7,
+        6,
+        5
+      ],
+      "precisions": [
+        0.875,
+        0.5714285714285715,
+        0.33333333333333337,
+        0.2
+      ],
+      "bp": 1.0,
+      "sys_len": 8,
+      "ref_len": 8
+    },
+    "squad": {
+      "exact_match": 0.0,
+      "f1": 0.6857142857142858
+    }
+  },
+  "evaluate_datasets_metric": {
+    "empty_predictions": 0,
+    "total_items": 2,
+    "wer": 0.5833333333333334
+  },
+  "evaluate_corpus": {
+    "empty_predictions": 0,
+    "total_items": 2,
+    "accuracy": {
+      "score": 0.7285714285714285
+    },
+    "bertscore": {
+      "score": 0.7485643923282623,
+      "precision": [
+        0.8912334442138672,
+        0.6074945330619812
+      ],
+      "recall": [
+        0.837701678276062,
+        0.6777911186218262
+      ],
+      "f1": [
+        0.8564083576202393,
+        0.6407204270362854
+      ],
+      "hashcode": "albert-base-v1_L10_no-idf_version=0.3.10(hug_trans=4.9.1)"
+    },
+    "bleu": {
+      "score": 0.0,
+      "precisions": [
+        0.9,
+        0.75,
+        0.5,
+        0.0
+      ],
+      "brevity_penalty": 0.9048374180359595,
+      "length_ratio": 0.9090909090909091,
+      "translation_length": 10,
+      "reference_length": 11
+    },
+    "f1": {
+      "score": 0.7948717948717947
+    },
+    "meteor": {
+      "score": 0.5420511682934044
+    },
+    "precision": {
+      "score": 0.875
+    },
+    "recall": {
+      "score": 0.7285714285714285
+    },
+    "rouge": {
+      "rouge1": 0.7948717948717947,
+      "rouge2": 0.6493506493506493,
+      "rougeL": 0.7948717948717947,
+      "rougeLsum": 0.7948717948717947
+    },
+    "sacrebleu": {
+      "score": 0.2730120862709067,
+      "counts": [
+        8,
+        4,
+        2,
+        0
+      ],
+      "totals": [
+        12,
+        10,
+        8,
+        6
+      ],
+      "precisions": [
+        0.6666666666666667,
+        0.4,
+        0.25,
+        0.08333333333333334
+      ],
+      "bp": 1.0,
+      "sys_len": 12,
+      "ref_len": 11
+    },
+    "squad": {
+      "exact_match": 0.0,
+      "f1": 0.7301587301587302
+    }
+  },
+  "evaluate_multiple_predictions": {
+    "empty_predictions": 0,
+    "total_items": 2,
+    "accuracy": {
+      "score": 0.7285714285714285
+    },
+    "bertscore": {
+      "score": 0.7485643923282623,
+      "precision": 0.7493639886379242,
+      "recall": 0.7577463984489441,
+      "f1": 0.7485643923282623,
+      "hashcode": "albert-base-v1_L10_no-idf_version=0.3.10(hug_trans=4.9.1)"
+    },
+    "bleu": {
+      "score": 0.42370250917168295,
+      "precisions": [
+        0.8823529411764706,
+        0.6428571428571429,
+        0.45454545454545453,
+        0.125
+      ],
+      "brevity_penalty": 1.0,
+      "length_ratio": 1.0,
+      "translation_length": 11,
+      "reference_length": 11
+    },
+    "f1": {
+      "score": 0.7948717948717947
+    },
+    "meteor": {
+      "score": 0.5420511682934044
+    },
+    "precision": {
+      "score": 0.875
+    },
+    "recall": {
+      "score": 0.7285714285714285
+    },
+    "rouge": {
+      "rouge1": 0.7783882783882783,
+      "rouge2": 0.5925324675324675,
+      "rougeL": 0.7426739926739926,
+      "rougeLsum": 0.7426739926739926
+    },
+    "sacrebleu": {
+      "score": 0.32377227131456443,
+      "counts": [
+        11,
+        6,
+        3,
+        0
+      ],
+      "totals": [
+        13,
+        11,
+        9,
+        7
+      ],
+      "precisions": [
+        0.8461538461538461,
+        0.5454545454545454,
+        0.33333333333333337,
+        0.07142857142857144
+      ],
+      "bp": 1.0,
+      "sys_len": 11,
+      "ref_len": 12,
+      "adjusted_precisions": [
+        0.8461538461538461,
+        0.5454545454545454,
+        0.33333333333333337,
+        0.07142857142857144
+      ]
+    },
+    "squad": {
+      "exact_match": 0.0,
+      "f1": 0.7402597402597403
+    }
+  }
+}
\ No newline at end of file
diff --git a/tests/jury/test_jury.py b/tests/jury/test_jury.py
index 26977ed..d8c08fd 100644
--- a/tests/jury/test_jury.py
+++ b/tests/jury/test_jury.py
@@ -1,43 +1,154 @@
+import datasets
 import numpy as np
 import pytest
 
+from tests.jury.conftest import get_expected_output
+from tests.utils import assert_almost_equal_dict
 
-def test_evaluate_basic(predictions, references, jury):
-    scores = jury.evaluate(predictions, references)
 
-    assert all([scores[metric.resulting_name] is not None for metric in jury.metrics])
+@pytest.fixture
+@get_expected_output(prefix=None)
+def output_evaluate_concurrent():
+    return output_evaluate_concurrent.output
 
 
-def test_evaluate_basic_str_input(predictions, references, jury_str):
-    scores = jury_str.evaluate(predictions, references)
+@pytest.fixture
+@get_expected_output(prefix=None)
+def output_evaluate():
+    return output_evaluate.output
 
-    assert all([scores[metric.resulting_name] is not None for metric in jury_str.metrics])
 
+@pytest.fixture
+@get_expected_output(prefix=None)
+def output_evaluate_str_input():
+    return output_evaluate_str_input.output
 
-def test_evaluate_corpus(single_prediction_array, multiple_references, jury):
-    scores = jury.evaluate(single_prediction_array, multiple_references)
 
-    assert all([scores[metric.resulting_name] is not None for metric in jury.metrics])
+@pytest.fixture
+@get_expected_output(prefix=None)
+def output_evaluate_list_str_input():
+    return output_evaluate_list_str_input.output
 
 
-def test_evaluate_multiple_predictions(multiple_predictions, multiple_references, jury):
-    scores = jury.evaluate(predictions=multiple_predictions, references=multiple_references)
+@pytest.fixture
+@get_expected_output(prefix=None)
+def output_evaluate_list_dict_input():
+    return output_evaluate_list_dict_input.output
 
-    assert all([scores[metric.resulting_name] is not None for metric in jury.metrics])
+
+@pytest.fixture
+@get_expected_output(prefix=None)
+def output_evaluate_list_mixed_input():
+    return output_evaluate_list_mixed_input.output
+
+
+@pytest.fixture
+@get_expected_output(prefix=None)
+def output_evaluate_datasets_metric():
+    return output_evaluate_datasets_metric.output
+
+
+@pytest.fixture
+@get_expected_output(prefix=None)
+def output_evaluate_corpus():
+    return output_evaluate_corpus.output
+
+
+@pytest.fixture
+@get_expected_output(prefix=None)
+def output_evaluate_multiple_predictions():
+    return output_evaluate_multiple_predictions.output
+
+
+def test_evaluate_concurrent(predictions, references, jury_concurrent, output_evaluate_concurrent):
+    scores = jury_concurrent(predictions=predictions, references=references)
+    assert_almost_equal_dict(
+        actual=scores, desired=output_evaluate_concurrent, exclude_paths="root['bertscore']['hashcode']"
+    )
+
+
+def test_evaluate_no_input(predictions, references, jury):
+    with pytest.raises(TypeError):
+        jury(predictions=predictions)
+        jury(references=references)
+        jury()
+
+
+def test_evaluate_inconsistent_input(inconsistent_predictions, references, jury):
+    # Different length
+    with pytest.raises(ValueError):
+        jury(predictions=inconsistent_predictions, references=references)
 
 
-def test_evaluate_concurrent(predictions, references, jury_concurrent):
-    scores = jury_concurrent.evaluate(predictions, references)
+def test_evaluate(predictions, references, jury, output_evaluate):
+    scores = jury(predictions=predictions, references=references)
+    assert_almost_equal_dict(actual=scores, desired=output_evaluate, exclude_paths="root['bertscore']['hashcode']")
 
-    assert all([scores[metric.resulting_name] is not None for metric in jury_concurrent.metrics])
+
+def test_evaluate_str_input(predictions, references, jury_str, output_evaluate_str_input):
+    scores = jury_str(predictions=predictions, references=references)
+    assert_almost_equal_dict(actual=scores, desired=output_evaluate_str_input)
+
+
+def test_evaluate_list_str_input(predictions, references, jury_list_str, output_evaluate_list_str_input):
+    scores = jury_list_str(predictions=predictions, references=references)
+    assert_almost_equal_dict(
+        actual=scores, desired=output_evaluate_list_str_input, exclude_paths="root['bertscore']['hashcode']"
+    )
+
+
+def test_evaluate_list_dict_input(predictions, references, jury_list_dict, output_evaluate_list_dict_input):
+    scores = jury_list_dict(predictions=predictions, references=references)
+    assert_almost_equal_dict(
+        actual=scores, desired=output_evaluate_list_dict_input, exclude_paths="root['bertscore']['hashcode']"
+    )
+
+
+def test_evaluate_list_mixed_input(predictions, references, jury_list_mixed, output_evaluate_list_mixed_input):
+    scores = jury_list_mixed(predictions=predictions, references=references)
+    assert_almost_equal_dict(
+        actual=scores, desired=output_evaluate_list_mixed_input, exclude_paths="root['bertscore']['hashcode']"
+    )
+
+
+def test_evaluate_datasets_metric(predictions, references, jury_datasets, output_evaluate_datasets_metric):
+    scores = jury_datasets(predictions=predictions, references=references)
+    assert_almost_equal_dict(actual=scores, desired=output_evaluate_datasets_metric)
+
+
+def test_evaluate_corpus(single_prediction_array, multiple_references, jury, output_evaluate_corpus):
+    scores = jury(predictions=single_prediction_array, references=multiple_references)
+    assert_almost_equal_dict(
+        actual=scores, desired=output_evaluate_corpus, exclude_paths="root['bertscore']['hashcode']"
+    )
+
+
+def test_evaluate_multiple_predictions(
+    multiple_predictions, multiple_references, jury, output_evaluate_multiple_predictions
+):
+    scores = jury(predictions=multiple_predictions, references=multiple_references)
+    assert_almost_equal_dict(
+        actual=scores, desired=output_evaluate_multiple_predictions, exclude_paths="root['bertscore']['hashcode']"
+    )
 
 
 def test_reduce_fn(predictions, references, jury):
     _reduce_fn = np.mean
     _non_reduce_fn = np.exp
-    scores = jury.evaluate(predictions, references, reduce_fn=_reduce_fn)
+    scores = jury(predictions=predictions, references=references, reduce_fn=_reduce_fn)
 
     assert all([scores[metric.resulting_name] is not None for metric in jury.metrics])
 
     with pytest.raises(ValueError):
-        jury.evaluate(predictions, references, reduce_fn=_non_reduce_fn)
+        jury(predictions=predictions, references=references, reduce_fn=_non_reduce_fn)
+
+
+def test_load_metric():
+    from jury import load_metric
+    from jury.metrics import Metric as JuryMetric
+
+    assert isinstance(load_metric("squad"), JuryMetric)
+    assert isinstance(load_metric("squad_v2"), datasets.Metric)
+
+    with pytest.raises(FileNotFoundError):
+        load_metric("abcdefgh")
diff --git a/tests/jury/test_utils.py b/tests/jury/test_utils.py
index f71fd51..582b309 100644
--- a/tests/jury/test_utils.py
+++ b/tests/jury/test_utils.py
@@ -1,17 +1,18 @@
 import numpy as np
 import pytest
 
-from jury.utils import bulk_remove_keys, is_reduce_fn, remove_punctuations
+from jury.metrics._utils import is_reduce_fn
+from jury.utils import bulk_remove_keys, remove_punctuations
 
 
 @pytest.fixture
 def raw_text():
-    return '!*This, a (test); text. Testing: 123 alpha123.  Multiple    spaces     "uneven sizes", and-also-this.'
+    return '!*This\'s, a (test); text. Testing: 123 alpha123.  Multiple    spaces     "uneven sizes", and-also-this.'
 
 
 @pytest.fixture
 def preprocessed_text():
-    return "This a test text Testing 123 alpha123 Multiple spaces uneven sizes and also this"
+    return "This s a test text Testing 123 alpha123 Multiple spaces uneven sizes and also this"
 
 
 def test_remove_punctuations(raw_text, preprocessed_text):
diff --git a/tests/run_tests.py b/tests/run_tests.py
index 0f0ee49..5da2cef 100644
--- a/tests/run_tests.py
+++ b/tests/run_tests.py
@@ -1,5 +1,5 @@
 from tests.utils import shell, validate_and_exit
 
 if __name__ == "__main__":
-    sts_tests = shell("pytest --cov jury --cov-report term-missing --cov-report xml -vvv tests")
+    sts_tests = shell("pytest --cov jury --cov-report term-missing --cov-report xml")
     validate_and_exit(tests=sts_tests)
diff --git a/tests/utils.py b/tests/utils.py
index 8e37e6b..c0e42fd 100644
--- a/tests/utils.py
+++ b/tests/utils.py
@@ -46,8 +46,8 @@ def print_console_centered(text: str, fill_char="="):
     print(f" {text} ".center(w, fill_char))
 
 
-def assert_almost_equal_dict(actual: Dict, desired: Dict, decimal=5):
-    assert DeepDiff(actual, desired, significant_digits=decimal) == {}
+def assert_almost_equal_dict(actual: Dict, desired: Dict, decimal=5, exclude_paths=None):
+    assert DeepDiff(actual, desired, significant_digits=decimal, exclude_paths=exclude_paths) == {}
 
 
 def shell_capture(command, out_json=True):