Repository: obss/jury (language: python)

Issue:
Move infrastructure to new `evaluate` package (from `datasets`)
[datasets'](https://github.com/huggingface/datasets) metrics will be deprecated in favor of the new [evaluate](https://github.com/huggingface/evaluate) package


Discussion / hints:

Fix patch:
diff --git a/.github/ISSUE_TEMPLATE/bug_report.md b/.github/ISSUE_TEMPLATE/bug_report.md
index 95609f1..9011a2d 100644
--- a/.github/ISSUE_TEMPLATE/bug_report.md
+++ b/.github/ISSUE_TEMPLATE/bug_report.md
@@ -22,4 +22,5 @@ If applicable, add full traceback to help explain your problem.
 **Environment Information:**
  - OS: (e.g. Ubuntu 20.04 LTS)
  - jury version: (`$ jury version`)
+ - evaluate version: (e.g. `$ pip list | grep evaluate` or for windows (`> pip list | findstr evaluate`))
  - datasets version: (e.g. `$ pip list | grep datasets` or for windows (`> pip list | findstr datasets`))
diff --git a/README.md b/README.md
index 6e07d0c..e242a56 100644
--- a/README.md
+++ b/README.md
@@ -15,7 +15,7 @@
 <a href="https://doi.org/10.5281/zenodo.6109838"><img src="https://zenodo.org/badge/DOI/10.5281/zenodo.6109838.svg" alt="DOI"></a>
 </p>
 
-A comprehensive toolkit for evaluating NLP experiments offering various automated metrics. Jury offers a smooth and easy-to-use interface. It uses a more advanced version of [datasets](https://github.com/huggingface/datasets/) design for underlying metric computation, so that adding custom metric is easy as extending proper class.
+A comprehensive toolkit for evaluating NLP experiments offering various automated metrics. Jury offers a smooth and easy-to-use interface. It uses a more advanced version of [evaluate](https://github.com/huggingface/evaluate/) design for underlying metric computation, so that adding custom metric is easy as extending proper class.
 
 Main advantages that Jury offers are:
 
@@ -31,7 +31,7 @@ To see more, check the [official Jury blog post](https://medium.com/codable/jury
 
 The table below shows the current support status for available metrics.
 
-| Metric                                                                        | Jury Support       | HF/datasets Support |
+| Metric                                                                        | Jury Support       | HF/evaluate Support |
 |-------------------------------------------------------------------------------|--------------------|---------------------|
 | Accuracy-Numeric                                                              | :heavy_check_mark: | :white_check_mark:  |
 | Accuracy-Text                                                                 | :heavy_check_mark: | :x:                 |
@@ -56,9 +56,9 @@ The table below shows the current support status for available metrics.
 | Squad                                                                         | :heavy_check_mark: | :white_check_mark:  |
 | TER                                                                           | :heavy_check_mark: | :white_check_mark:  |
 | WER                                                                           | :heavy_check_mark: | :white_check_mark:  |
-| [Other metrics](https://github.com/huggingface/datasets/tree/master/metrics)* | :white_check_mark: | :white_check_mark:  |
+| [Other metrics](https://github.com/huggingface/evaluate/tree/master/metrics)* | :white_check_mark: | :white_check_mark:  |
 
-_*_ Placeholder for the rest of the metrics available in `datasets` package apart from those which are present in the 
+_*_ Placeholder for the rest of the metrics available in `evaluate` package apart from those which are present in the 
 table. 
 
 **Notes**
@@ -67,8 +67,8 @@ table.
 types (single prediction & single reference, single prediction & multiple references, multiple predictions & multiple 
 references) are supported
 
-* The entry :white_check_mark: means that this metric is supported (for Jury through the `datasets`), so that it 
-can (and should) be used just like the `datasets` metric as instructed in `datasets` implementation although 
+* The entry :white_check_mark: means that this metric is supported (for Jury through the `evaluate`), so that it 
+can (and should) be used just like the `evaluate` metric as instructed in `evaluate` implementation although 
 unfortunately full Jury support for those metrics are not yet available.
 
 ## Request for a New Metric
@@ -149,15 +149,15 @@ bleu = Bleu.construct(compute_kwargs={"max_order": 1})
 score = bleu.compute(predictions=predictions, references=references)
 ```
 
-Note that you can seemlessly access both `jury` and `datasets` metrics through `jury.load_metric`. 
+Note that you can seemlessly access both `jury` and `evaluate` metrics through `jury.load_metric`. 
 
 ```python
 import jury
 
 bleu = jury.load_metric("bleu")
 bleu_1 = jury.load_metric("bleu", resulting_name="bleu_1", compute_kwargs={"max_order": 1})
-# metrics not available in `jury` but in `datasets`
-wer = jury.load_metric("competition_math") # It falls back to `datasets` package with a warning
+# metrics not available in `jury` but in `evaluate`
+wer = jury.load_metric("competition_math") # It falls back to `evaluate` package with a warning
 ```
 
 ### CLI Usage
@@ -190,9 +190,9 @@ Then, you can call jury eval with `config` argument.
 
 ### Custom Metrics
 
-You can use custom metrics with inheriting `jury.metrics.Metric`, you can see current metrics implemented on Jury from [jury/metrics](https://github.com/obss/jury/tree/master/jury/metrics). Jury falls back to `datasets` implementation of metrics for the ones that are currently not supported by Jury, you can see the metrics available for `datasets` on [datasets/metrics](https://github.com/huggingface/datasets/tree/master/metrics). 
+You can use custom metrics with inheriting `jury.metrics.Metric`, you can see current metrics implemented on Jury from [jury/metrics](https://github.com/obss/jury/tree/master/jury/metrics). Jury falls back to `evaluate` implementation of metrics for the ones that are currently not supported by Jury, you can see the metrics available for `evaluate` on [evaluate/metrics](https://github.com/huggingface/evaluate/tree/master/metrics). 
 
-Jury itself uses `datasets.Metric` as a base class to drive its own base class as `jury.metrics.Metric`. The interface is similar; however, Jury makes the metrics to take a unified input type by handling the inputs for each metrics, and allows supporting several input types as;
+Jury itself uses `evaluate.Metric` as a base class to drive its own base class as `jury.metrics.Metric`. The interface is similar; however, Jury makes the metrics to take a unified input type by handling the inputs for each metrics, and allows supporting several input types as;
 
 - single prediction & single reference
 - single prediction & multiple reference
diff --git a/examples/environment.yml b/examples/environment.yml
deleted file mode 100644
index b84809e..0000000
--- a/examples/environment.yml
+++ /dev/null
@@ -1,13 +0,0 @@
-name: jury
-channels:
-  - anaconda
-dependencies:
-  - pip
-  - python=3.8
-  - pip:
-    - ipywidgets==7.6.3
-    - jury==1.0.0
-    - jupyter==1.0.0
-    - notebook>=6.4.0
-    - sacrebleu==1.5.1
-    - bert-score==0.3.9
\ No newline at end of file
diff --git a/examples/jury_evaluate.ipynb b/examples/jury_evaluate.ipynb
index a49f5b9..d83d3e4 100644
--- a/examples/jury_evaluate.ipynb
+++ b/examples/jury_evaluate.ipynb
@@ -56,14 +56,31 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 1,
    "metadata": {
     "ExecuteTime": {
      "end_time": "2021-10-02T19:59:22.515146Z",
      "start_time": "2021-10-02T19:59:20.716835Z"
     }
    },
-   "outputs": [],
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "/home/devrim/anaconda3/envs/jury/lib/python3.9/site-packages/tqdm/auto.py:22: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
+      "  from .autonotebook import tqdm as notebook_tqdm\n",
+      "2022-09-30 17:48:30.188018: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA\n",
+      "To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.\n",
+      "2022-09-30 17:48:30.361067: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory\n",
+      "2022-09-30 17:48:30.361090: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.\n",
+      "2022-09-30 17:48:30.396709: E tensorflow/stream_executor/cuda/cuda_blas.cc:2981] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered\n",
+      "2022-09-30 17:48:30.939257: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory\n",
+      "2022-09-30 17:48:30.939339: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory\n",
+      "2022-09-30 17:48:30.939347: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.\n"
+     ]
+    }
+   ],
    "source": [
     "import os\n",
     "import json # Just for pretty printing the resulting dict.\n",
@@ -73,7 +90,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 2,
    "metadata": {
     "ExecuteTime": {
      "end_time": "2021-10-02T19:59:22.531153Z",
@@ -102,7 +119,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 3,
    "metadata": {
     "ExecuteTime": {
      "end_time": "2021-10-02T19:59:22.547167Z",
@@ -139,7 +156,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 4,
    "metadata": {
     "ExecuteTime": {
      "end_time": "2021-10-02T19:59:22.562180Z",
@@ -170,7 +187,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 5,
    "metadata": {
     "ExecuteTime": {
      "end_time": "2021-10-02T19:59:26.664594Z",
@@ -187,7 +204,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 6,
    "metadata": {
     "ExecuteTime": {
      "end_time": "2021-10-02T19:59:26.679608Z",
@@ -195,7 +212,48 @@
     },
     "scrolled": true
    },
-   "outputs": [],
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "{\n",
+      "    \"total_items\": 2,\n",
+      "    \"empty_items\": 0,\n",
+      "    \"bleu_1\": {\n",
+      "        \"score\": 0.8823529411764706,\n",
+      "        \"precisions\": [\n",
+      "            0.8823529411764706\n",
+      "        ],\n",
+      "        \"brevity_penalty\": 1.0,\n",
+      "        \"length_ratio\": 1.0,\n",
+      "        \"translation_length\": 11,\n",
+      "        \"reference_length\": 11\n",
+      "    },\n",
+      "    \"bleu_2\": {\n",
+      "        \"score\": 0.7531446678801508,\n",
+      "        \"precisions\": [\n",
+      "            0.8823529411764706,\n",
+      "            0.6428571428571429\n",
+      "        ],\n",
+      "        \"brevity_penalty\": 1.0,\n",
+      "        \"length_ratio\": 1.0,\n",
+      "        \"translation_length\": 11,\n",
+      "        \"reference_length\": 11\n",
+      "    },\n",
+      "    \"meteor\": {\n",
+      "        \"score\": 0.727184593644221\n",
+      "    },\n",
+      "    \"rouge\": {\n",
+      "        \"rouge1\": 0.7783882783882783,\n",
+      "        \"rouge2\": 0.5925324675324675,\n",
+      "        \"rougeL\": 0.7426739926739926,\n",
+      "        \"rougeLsum\": 0.7426739926739926\n",
+      "    }\n",
+      "}\n"
+     ]
+    }
+   ],
    "source": [
     "# Display results\n",
     "print(json.dumps(scores, indent=4))"
@@ -212,7 +270,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 7,
    "metadata": {
     "ExecuteTime": {
      "end_time": "2021-10-02T19:59:29.552689Z",
@@ -231,7 +289,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 8,
    "metadata": {
     "ExecuteTime": {
      "end_time": "2021-10-02T19:59:30.130953Z",
@@ -239,7 +297,22 @@
     },
     "scrolled": true
    },
-   "outputs": [],
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "{\n",
+      "    \"total_items\": 3,\n",
+      "    \"empty_items\": 0,\n",
+      "    \"squad\": {\n",
+      "        \"exact_match\": 0.33333333333333337,\n",
+      "        \"f1\": 0.8222222222222223\n",
+      "    }\n",
+      "}\n"
+     ]
+    }
+   ],
    "source": [
     "qa_jury = Jury(metrics=QA_METRICS, run_concurrent=False)\n",
     "scores = qa_jury(predictions=qa_predictions, references=qa_references)\n",
@@ -257,7 +330,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 9,
    "metadata": {
     "ExecuteTime": {
      "end_time": "2021-10-02T20:17:28.152162Z",
@@ -269,7 +342,7 @@
     "from collections import Counter\n",
     "from typing import Callable\n",
     "\n",
-    "import datasets\n",
+    "import evaluate\n",
     "from jury.collator import Collator\n",
     "from jury.metrics import MetricForLanguageGeneration\n",
     "from jury.utils.nlp import normalize_text\n",
@@ -288,7 +361,7 @@
     "        score = 0.5 # (2/4)\n",
     "    \"\"\"\n",
     "    def _info(self):\n",
-    "        return datasets.MetricInfo(\n",
+    "        return evaluate.MetricInfo(\n",
     "            description=\"Custom metric to compute ratio of matching tokens over unique tokens.\",\n",
     "            citation=\"Custom metric\",\n",
     "            inputs_description=\"\",\n",
@@ -349,7 +422,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 10,
    "metadata": {
     "ExecuteTime": {
      "end_time": "2021-10-02T20:17:28.353117Z",
@@ -366,15 +439,33 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 11,
    "metadata": {
     "ExecuteTime": {
      "end_time": "2021-10-02T20:17:28.583593Z",
      "start_time": "2021-10-02T20:17:28.556577Z"
     },
-    "scrolled": false
+    "scrolled": true
    },
-   "outputs": [],
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "{\n",
+      "    \"total_items\": 3,\n",
+      "    \"empty_items\": 0,\n",
+      "    \"squad\": {\n",
+      "        \"exact_match\": 0.33333333333333337,\n",
+      "        \"f1\": 0.8222222222222223\n",
+      "    },\n",
+      "    \"word_match\": {\n",
+      "        \"score\": 0.7222222222222222\n",
+      "    }\n",
+      "}\n"
+     ]
+    }
+   ],
    "source": [
     "qa_jury = Jury(metrics=QA_METRICS, run_concurrent=False)\n",
     "scores = qa_jury(predictions=qa_predictions, references=qa_references)\n",
@@ -398,7 +489,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.8.5"
+   "version": "3.9.13"
   },
   "toc": {
    "base_numbering": 1,
diff --git a/jury/metrics/_core/auto.py b/jury/metrics/_core/auto.py
index 6a398ae..728a843 100644
--- a/jury/metrics/_core/auto.py
+++ b/jury/metrics/_core/auto.py
@@ -1,5 +1,5 @@
 # coding=utf-8
-# Copyright 2021 Open Business Software Solutions, The HuggingFace Datasets Authors and the TensorFlow Datasets Authors.
+# Copyright 2021 Open Business Software Solutions, The HuggingFace evaluate Authors and the TensorFlow evaluate Authors.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -18,7 +18,7 @@
 import warnings
 from typing import Any, Dict, NamedTuple, Optional
 
-import datasets
+import evaluate
 
 from jury.metrics._core.base import Metric
 from jury.metrics._core.utils import import_module, list_metrics
@@ -40,17 +40,17 @@ def load_metric(
             path to the metric processing script with the metric builder. Can be either:
                 - a local absolute or relative path to processing script or the directory containing the script,
                     e.g. ``'./metrics/rogue/rouge.py'``
-                - a metric identifier on the HuggingFace datasets repo (list all available metrics with
+                - a metric identifier on the HuggingFace evaluate repo (list all available metrics with
                     ``jury.list_metrics()``) e.g. ``'rouge'`` or ``'bleu'``
         resulting_name (Optional ``str``): Resulting name of the computed score returned.
         task (Optional ``str``): Task name for the metric. "language-generation" by default.
         compute_kwargs (Optional ``Dict[str, Any]``): Arguments to be passed to `compute()` method of metric at
             computation.
         use_jury_only (``bool``): Whether to use jury metrics only or not. False by default.
-        kwargs (Optional): Additional keyword arguments to be passed to :py:func:`datasets.load_metric`.
+        kwargs (Optional): Additional keyword arguments to be passed to :py:func:`evaluate.load_metric`.
 
     Returns:
-        `datasets.Metric`
+        `evaluate.Metric`
     """
     return AutoMetric.load(
         path=path,
@@ -101,14 +101,14 @@ def load(
             if use_jury_only:
                 raise ValueError(
                     f"Metric {resolved_path.path} is not available on jury, set use_jury_only=False to use"
-                    f"additional metrics (e.g datasets metrics)."
+                    f"additional metrics (e.g evaluate metrics)."
                 )
             warnings.warn(
-                f"Metric {resolved_path.path} is not available on jury, falling back to datasets metric. "
+                f"Metric {resolved_path.path} is not available on jury, falling back to evaluate metric. "
                 f"You may not fully utilize this metric for different input types, e.g multiple predictions "
                 f"or multiple references."
             )
-            metric = datasets.load_metric(resolved_path.path, **kwargs)
+            metric = evaluate.load(resolved_path.path, **kwargs)
         else:
             # get the class, will raise AttributeError if class cannot be found
             factory_class = module.__main_class__
@@ -140,4 +140,4 @@ class ResolvedName(NamedTuple):
                 raise FileNotFoundError(f"File {path} does not exists.")
             return ResolvedName(path=path, resolution="external-module")
         else:
-            return ResolvedName(path=path, resolution="datasets")
+            return ResolvedName(path=path, resolution="evaluate")
diff --git a/jury/metrics/_core/base.py b/jury/metrics/_core/base.py
index e572c86..ac39a1f 100644
--- a/jury/metrics/_core/base.py
+++ b/jury/metrics/_core/base.py
@@ -1,5 +1,5 @@
 # coding=utf-8
-# Copyright 2021 Open Business Software Solutions, The HuggingFace Datasets Authors
+# Copyright 2021 Open Business Software Solutions, The HuggingFace evaluate Authors
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -14,17 +14,18 @@
 # limitations under the License.
 """
 Metrics base class. The part of this file is adapted from HuggingFace's
-datasets package implementation of Accuracy metric. See
-https://github.com/huggingface/datasets/blob/master/src/datasets/metric.py
+evaluate package implementation of Accuracy metric. See
+https://github.com/huggingface/evaluate/blob/master/src/evaluate/metric.py
 """
 from abc import ABC, abstractmethod
 from typing import Any, Callable, Dict, List, Optional, Tuple, Union
 
 import datasets
+import evaluate
 import numpy
 import numpy as np
 import pandas as pd
-from datasets.utils.logging import get_logger
+from evaluate.utils.logging import get_logger
 
 from jury.collator import Collator
 from jury.metrics._core.utils import import_module, is_reduce_fn
@@ -38,7 +39,7 @@
 logger = get_logger(__name__)
 
 
-class Metric(datasets.Metric, ABC):
+class Metric(evaluate.Metric, ABC):
     """
     Base metric class and common API for all metrics.
 
@@ -59,7 +60,7 @@ class Metric(datasets.Metric, ABC):
         process_id (``int``): specify the id of the current process in a distributed setup (between 0 and num_process-1)
             This is useful to compute metrics in distributed setups (in particular non-additive metrics like F1).
         seed (Optional ``int``): If specified, this will temporarily set numpy's random seed when
-        :func:`datasets.Metric.compute` is run.
+        :func:`evaluate.Metric.compute` is run.
         experiment_id (Optional ``str``): A specific experiment id. This is used if several distributed evaluations
             share the same file system. This is useful to compute metrics in distributed setups (in particular
             non-additive metrics like F1).
@@ -204,7 +205,7 @@ class MetricForTask(Metric, ABC):
     All metrics extending :py:class:`jury.metrics._core.base.MetricForTask` must implement the following:
 
         - _task (``[str]``): Task name for the base task metric.
-        - _default_features() (``datasets.Features``): Task input as a :py:class:`datasets.Features`.
+        - _default_features() (``datasets.Features``): Task input as a :py:class:`evaluate.Features`.
 
      Args:
         resulting_name (Optional ``[str]``): Optional resulting name for :py:class:`jury.Jury` to use. By default, it
diff --git a/jury/metrics/accuracy/accuracy_for_language_generation.py b/jury/metrics/accuracy/accuracy_for_language_generation.py
index e46ebdd..dd04be5 100644
--- a/jury/metrics/accuracy/accuracy_for_language_generation.py
+++ b/jury/metrics/accuracy/accuracy_for_language_generation.py
@@ -1,5 +1,5 @@
 # coding=utf-8
-# Copyright 2021 Open Business Software Solutions, The HuggingFace Datasets Authors.
+# Copyright 2021 Open Business Software Solutions, The HuggingFace evaluate Authors.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -14,14 +14,14 @@
 # limitations under the License.
 """
 Accuracy metric. The part of this file is adapted from HuggingFace's
-datasets package implementation of Accuracy metric. See
-https://github.com/huggingface/datasets/blob/master/metrics/accuracy/accuracy.py
+evaluate package implementation of Accuracy metric. See
+https://github.com/huggingface/evaluate/blob/master/metrics/accuracy/accuracy.py
 """
 
 from collections import Counter
 from typing import Callable
 
-import datasets
+import evaluate
 import numpy as np
 
 from jury.collator import Collator
@@ -75,10 +75,10 @@
 """
 
 
-@datasets.utils.file_utils.add_start_docstrings(_DESCRIPTION, _KWARGS_DESCRIPTION)
+@evaluate.utils.file_utils.add_start_docstrings(_DESCRIPTION, _KWARGS_DESCRIPTION)
 class AccuracyForLanguageGeneration(MetricForLanguageGeneration):
     def _info(self):
-        return datasets.MetricInfo(
+        return evaluate.MetricInfo(
             description=_DESCRIPTION,
             citation=_CITATION,
             inputs_description=_KWARGS_DESCRIPTION,
diff --git a/jury/metrics/accuracy/accuracy_for_sequence_classification.py b/jury/metrics/accuracy/accuracy_for_sequence_classification.py
index ebf66cf..2c6f045 100644
--- a/jury/metrics/accuracy/accuracy_for_sequence_classification.py
+++ b/jury/metrics/accuracy/accuracy_for_sequence_classification.py
@@ -1,5 +1,5 @@
 # coding=utf-8
-# Copyright 2021 Open Business Software Solutions, The HuggingFace Datasets Authors.
+# Copyright 2021 Open Business Software Solutions, The HuggingFace evaluate Authors.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -14,12 +14,12 @@
 # limitations under the License.
 """
 Accuracy metric. The part of this file is adapted from HuggingFace's
-datasets package implementation of Accuracy metric. See
-https://github.com/huggingface/datasets/blob/master/metrics/accuracy/accuracy.py
+evaluate package implementation of Accuracy metric. See
+https://github.com/huggingface/evaluate/blob/master/metrics/accuracy/accuracy.py
 """
 from typing import Callable
 
-import datasets
+import evaluate
 from sklearn.metrics import accuracy_score
 
 from jury.metrics._core import MetricForSequenceClassification, SequenceClassificationInstance
@@ -67,10 +67,10 @@
 """
 
 
-@datasets.utils.file_utils.add_start_docstrings(_DESCRIPTION, _KWARGS_DESCRIPTION)
+@evaluate.utils.file_utils.add_start_docstrings(_DESCRIPTION, _KWARGS_DESCRIPTION)
 class AccuracyForSequenceClassification(MetricForSequenceClassification):
     def _info(self):
-        return datasets.MetricInfo(
+        return evaluate.MetricInfo(
             description=_DESCRIPTION,
             citation=_CITATION,
             inputs_description=_KWARGS_DESCRIPTION,
diff --git a/jury/metrics/bartscore/bartscore_for_language_generation.py b/jury/metrics/bartscore/bartscore_for_language_generation.py
index b7564a0..89944db 100644
--- a/jury/metrics/bartscore/bartscore_for_language_generation.py
+++ b/jury/metrics/bartscore/bartscore_for_language_generation.py
@@ -1,5 +1,5 @@
 # coding=utf-8
-# Copyright 2021 Open Business Software Solutions, The HuggingFace Datasets Authors.
+# Copyright 2021 Open Business Software Solutions, The HuggingFace evaluate Authors.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -14,12 +14,12 @@
 # limitations under the License.
 """
 BARTScore metric. The part of this file is adapted from metric implementations
-of datasets package. See
-https://github.com/huggingface/datasets/blob/master/metrics/
+of evaluate package. See
+https://github.com/huggingface/evaluate/blob/master/metrics/
 """
 from typing import Callable, Dict, List
 
-import datasets
+import evaluate
 import numpy as np
 
 from jury.metrics import LanguageGenerationInstance
@@ -108,7 +108,7 @@
 }
 
 
-@datasets.utils.file_utils.add_start_docstrings(_DESCRIPTION, _KWARGS_DESCRIPTION)
+@evaluate.utils.file_utils.add_start_docstrings(_DESCRIPTION, _KWARGS_DESCRIPTION)
 class BartscoreForLanguageGeneration(MetricForLanguageGeneration):
     def __init__(
         self,
@@ -169,7 +169,7 @@ def _download_and_prepare(self, dl_manager) -> None:
             self.scorer.load(path=model_dest)
 
     def _info(self):
-        return datasets.MetricInfo(
+        return evaluate.MetricInfo(
             description=_DESCRIPTION,
             citation=_CITATION,
             homepage="https://github.com/neulab/BARTScore",
diff --git a/jury/metrics/bertscore/bertscore_for_language_generation.py b/jury/metrics/bertscore/bertscore_for_language_generation.py
index 680401c..1f432df 100644
--- a/jury/metrics/bertscore/bertscore_for_language_generation.py
+++ b/jury/metrics/bertscore/bertscore_for_language_generation.py
@@ -1,5 +1,5 @@
 # coding=utf-8
-# Copyright 2021 Open Business Software Solutions, The HuggingFace Datasets Authors.
+# Copyright 2021 Open Business Software Solutions, The HuggingFace evaluate Authors.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -14,14 +14,14 @@
 # limitations under the License.
 """
 BERTScore metric. The part of this file is adapted from BERTScore implementation
-of datasets package. See
-https://github.com/huggingface/datasets/blob/master/metrics/bertscore/bertscore.py
+of evaluate package. See
+https://github.com/huggingface/evaluate/blob/master/metrics/bertscore/bertscore.py
 """
 import functools
 from contextlib import contextmanager
 from typing import Dict, List
 
-import datasets
+import evaluate
 import numpy as np
 import pandas as pd
 from packaging import version
@@ -39,7 +39,7 @@ def filter_logging_context():
     def filter_log(record):
         return False if "This IS expected if you are initializing" in record.msg else True
 
-    logger = datasets.utils.logging.get_logger("transformers.modeling_utils")
+    logger = evaluate.utils.logging.get_logger("transformers.modeling_utils")
     logger.addFilter(filter_log)
     try:
         yield
@@ -115,7 +115,7 @@ def filter_log(record):
 """
 
 
-@datasets.utils.file_utils.add_start_docstrings(_DESCRIPTION, _KWARGS_DESCRIPTION)
+@evaluate.utils.file_utils.add_start_docstrings(_DESCRIPTION, _KWARGS_DESCRIPTION)
 class BertscoreForLanguageGeneration(MetricForLanguageGeneration):
     def _download_and_prepare(self, dl_manager):
         global bert_score
@@ -128,7 +128,7 @@ def _download_and_prepare(self, dl_manager):
             super(BertscoreForLanguageGeneration, self)._download_and_prepare(dl_manager)
 
     def _info(self):
-        return datasets.MetricInfo(
+        return evaluate.MetricInfo(
             description=_DESCRIPTION,
             citation=_CITATION,
             homepage="https://github.com/Tiiiger/bert_score",
diff --git a/jury/metrics/bleu/bleu_for_language_generation.py b/jury/metrics/bleu/bleu_for_language_generation.py
index e019457..5487a58 100644
--- a/jury/metrics/bleu/bleu_for_language_generation.py
+++ b/jury/metrics/bleu/bleu_for_language_generation.py
@@ -1,5 +1,5 @@
 # coding=utf-8
-# Copyright 2021 Open Business Software Solutions, The HuggingFace Datasets Authors.
+# Copyright 2021 Open Business Software Solutions, The HuggingFace evaluate Authors.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -14,14 +14,15 @@
 # limitations under the License.
 """
 BLEU metric. The part of this file is adapted from HuggingFace's
-datasets package implementation of BLEU metric. See
-https://github.com/huggingface/datasets/blob/master/metrics/bleu/bleu.py
+evaluate package implementation of BLEU metric. See
+https://github.com/huggingface/evaluate/blob/master/metrics/bleu/bleu.py
 """
 
 import math
 from typing import Callable, Dict, Tuple
 
 import datasets
+import evaluate
 
 from jury.collator import Collator
 from jury.metrics._core import MetricForLanguageGeneration
@@ -112,7 +113,7 @@
 """
 
 
-@datasets.utils.file_utils.add_start_docstrings(_DESCRIPTION, _KWARGS_DESCRIPTION)
+@evaluate.utils.file_utils.add_start_docstrings(_DESCRIPTION, _KWARGS_DESCRIPTION)
 class BleuForLanguageGeneration(MetricForLanguageGeneration):
     def __init__(self, resulting_name: str = None, compute_kwargs: Dict = None, **kwargs):
         self.should_change_resulting_name = resulting_name is None
@@ -120,7 +121,7 @@ def __init__(self, resulting_name: str = None, compute_kwargs: Dict = None, **kw
         super().__init__(resulting_name=resulting_name, compute_kwargs=compute_kwargs, **kwargs)
 
     def _info(self):
-        return datasets.MetricInfo(
+        return evaluate.MetricInfo(
             description=_DESCRIPTION,
             citation=_CITATION,
             inputs_description=_KWARGS_DESCRIPTION,
diff --git a/jury/metrics/bleurt/bleurt_for_language_generation.py b/jury/metrics/bleurt/bleurt_for_language_generation.py
index 2210183..0f3b133 100644
--- a/jury/metrics/bleurt/bleurt_for_language_generation.py
+++ b/jury/metrics/bleurt/bleurt_for_language_generation.py
@@ -1,5 +1,5 @@
 # coding=utf-8
-# Copyright 2021 Open Business Software Solutions, The HuggingFace Datasets Authors.
+# Copyright 2021 Open Business Software Solutions, The HuggingFace evaluate Authors.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -14,14 +14,14 @@
 # limitations under the License.
 """
 BLEURT metric. The part of this file is adapted from BLEURT implementation
-of datasets package. See
-https://github.com/huggingface/datasets/blob/master/metrics/bleurt/bleurt.py
+of evaluate package. See
+https://github.com/huggingface/evaluate/blob/master/metrics/bleurt/bleurt.py
 """
 
 import os
 from typing import Callable
 
-import datasets
+import evaluate
 
 from jury.metrics import LanguageGenerationInstance, MetricForLanguageGeneration
 from jury.metrics._core.utils import PackagePlaceholder, requirement_message
@@ -29,7 +29,7 @@
 # `import bleurt` placeholder
 bleurt = PackagePlaceholder(version="1.2.2")
 
-logger = datasets.logging.get_logger(__name__)
+logger = evaluate.logging.get_logger(__name__)
 
 
 _CITATION = """\
@@ -102,10 +102,10 @@
 }
 
 
-@datasets.utils.file_utils.add_start_docstrings(_DESCRIPTION, _KWARGS_DESCRIPTION)
+@evaluate.utils.file_utils.add_start_docstrings(_DESCRIPTION, _KWARGS_DESCRIPTION)
 class BleurtForLanguageGeneration(MetricForLanguageGeneration):
     def _info(self):
-        return datasets.MetricInfo(
+        return evaluate.MetricInfo(
             description=_DESCRIPTION,
             citation=_CITATION,
             homepage="https://github.com/google-research/bleurt",
diff --git a/jury/metrics/cer/cer_for_language_generation.py b/jury/metrics/cer/cer_for_language_generation.py
index 11e4e29..725128a 100644
--- a/jury/metrics/cer/cer_for_language_generation.py
+++ b/jury/metrics/cer/cer_for_language_generation.py
@@ -1,5 +1,5 @@
 # coding=utf-8
-# Copyright 2021 Open Business Software Solutions, The HuggingFace Datasets Authors.
+# Copyright 2021 Open Business Software Solutions, The HuggingFace evaluate Authors.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -13,13 +13,13 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 """ CER metric. The part of this file is adapted from HuggingFace's
-datasets package implementation of CER metric. See
-https://github.com/huggingface/datasets/blob/master/metrics/cer/cer.py
+evaluate package implementation of CER metric. See
+https://github.com/huggingface/evaluate/blob/master/metrics/cer/cer.py
 """
 import warnings
 from typing import Callable, List, Tuple, Union
 
-import datasets
+import evaluate
 
 from jury.metrics import LanguageGenerationInstance, MetricForLanguageGeneration
 from jury.metrics._core.utils import PackagePlaceholder, requirement_message
@@ -84,10 +84,10 @@
 """
 
 
-@datasets.utils.file_utils.add_start_docstrings(_DESCRIPTION, _KWARGS_DESCRIPTION)
+@evaluate.utils.file_utils.add_start_docstrings(_DESCRIPTION, _KWARGS_DESCRIPTION)
 class CERForLanguageGeneration(MetricForLanguageGeneration):
     def _info(self):
-        return datasets.MetricInfo(
+        return evaluate.MetricInfo(
             description=_DESCRIPTION,
             citation=_CITATION,
             inputs_description=_KWARGS_DESCRIPTION,
diff --git a/jury/metrics/chrf/chrf_for_language_generation.py b/jury/metrics/chrf/chrf_for_language_generation.py
index 7f87bac..2181ced 100644
--- a/jury/metrics/chrf/chrf_for_language_generation.py
+++ b/jury/metrics/chrf/chrf_for_language_generation.py
@@ -1,5 +1,5 @@
 # coding=utf-8
-# Copyright 2021 Open Business Software Solutions, The HuggingFace Datasets Authors.
+# Copyright 2021 Open Business Software Solutions, The HuggingFace evaluate Authors.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -14,12 +14,12 @@
 # limitations under the License.
 """
 ChrF(++) (Character n-gram F-score) metric. The part of this file is adapted from HuggingFace's
-datasets package implementation of CHRF metric. See
-https://github.com/huggingface/datasets/blob/master/metrics/chrf/chrf.py
+evaluate package implementation of CHRF metric. See
+https://github.com/huggingface/evaluate/blob/master/metrics/chrf/chrf.py
 """
 from typing import Callable, Dict, List, Tuple, Union
 
-import datasets
+import evaluate
 from packaging import version
 
 from jury.collator import Collator
@@ -74,7 +74,7 @@
 that is already present in sacrebleu.
 The implementation here is slightly different from sacrebleu in terms of the required input format. The length of
 the references and hypotheses lists need to be the same, so you may need to transpose your references compared to
-sacrebleu's required input format. See https://github.com/huggingface/datasets/issues/3154#issuecomment-950746534
+sacrebleu's required input format. See https://github.com/huggingface/evaluate/issues/3154#issuecomment-950746534
 See the README.md file at https://github.com/mjpost/sacreBLEU#chrf--chrf for more information.
 """
 
@@ -109,7 +109,7 @@
 """
 
 
-@datasets.utils.file_utils.add_start_docstrings(_DESCRIPTION, _KWARGS_DESCRIPTION)
+@evaluate.utils.file_utils.add_start_docstrings(_DESCRIPTION, _KWARGS_DESCRIPTION)
 class CHRFForLanguageGeneration(MetricForLanguageGeneration):
     def _info(self):
         if version.parse(scb.__version__) < version.parse("1.4.12"):
@@ -117,7 +117,7 @@ def _info(self):
                 "To use `sacrebleu`, the module `sacrebleu>=1.4.12` is required, and the current version of "
                 "`sacrebleu` doesn't match this condition.\nYou can install it with `pip install sacrebleu>=1.4.12`."
             )
-        return datasets.MetricInfo(
+        return evaluate.MetricInfo(
             description=_DESCRIPTION,
             citation=_CITATION,
             homepage="https://github.com/mjpost/sacreBLEU#chrf--chrf",
diff --git a/jury/metrics/comet/comet_for_language_generation.py b/jury/metrics/comet/comet_for_language_generation.py
index 137cd44..681e590 100644
--- a/jury/metrics/comet/comet_for_language_generation.py
+++ b/jury/metrics/comet/comet_for_language_generation.py
@@ -1,5 +1,5 @@
 # coding=utf-8
-# Copyright 2021 Open Business Software Solutions, The HuggingFace Datasets Authors.
+# Copyright 2021 Open Business Software Solutions, The HuggingFace evaluate Authors.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -14,13 +14,13 @@
 # limitations under the License.
 """
 Comet cross-lingual MT evaluation metric. The part of this file is adapted from
-Comet implementation of datasets package. See
-https://github.com/huggingface/datasets/blob/master/metrics/comet/comet.py
+Comet implementation of evaluate package. See
+https://github.com/huggingface/evaluate/blob/master/metrics/comet/comet.py
 """
 
 from typing import Callable, Union
 
-import datasets
+import evaluate
 
 from jury.metrics import LanguageGenerationInstance
 from jury.metrics._core import MetricForCrossLingualEvaluation
@@ -90,7 +90,7 @@
 """
 
 
-@datasets.utils.file_utils.add_start_docstrings(_DESCRIPTION, _KWARGS_DESCRIPTION)
+@evaluate.utils.file_utils.add_start_docstrings(_DESCRIPTION, _KWARGS_DESCRIPTION)
 class CometForCrossLingualEvaluation(MetricForCrossLingualEvaluation):
     def _download_and_prepare(self, dl_manager):
         global comet
@@ -108,7 +108,7 @@ def _download_and_prepare(self, dl_manager):
         self.scorer = comet.load_from_checkpoint(checkpoint_path)
 
     def _info(self):
-        return datasets.MetricInfo(
+        return evaluate.MetricInfo(
             description=_DESCRIPTION,
             citation=_CITATION,
             homepage="https://unbabel.github.io/COMET/html/index.html",
diff --git a/jury/metrics/f1/f1_for_language_generation.py b/jury/metrics/f1/f1_for_language_generation.py
index 6c1b517..bcbb588 100644
--- a/jury/metrics/f1/f1_for_language_generation.py
+++ b/jury/metrics/f1/f1_for_language_generation.py
@@ -1,5 +1,5 @@
 # coding=utf-8
-# Copyright 2021 Open Business Software Solutions, The HuggingFace Datasets Authors.
+# Copyright 2021 Open Business Software Solutions, The HuggingFace evaluate Authors.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -14,13 +14,13 @@
 # limitations under the License.
 """
 F1 metric. The part of this file is adapted from HuggingFace's
-datasets package implementation of F1 metric. See
-https://github.com/huggingface/datasets/blob/master/metrics/f1/f1.py
+evaluate package implementation of F1 metric. See
+https://github.com/huggingface/evaluate/blob/master/metrics/f1/f1.py
 """
 
 from typing import Callable
 
-import datasets
+import evaluate
 import numpy as np
 
 from jury.collator import Collator
@@ -64,10 +64,10 @@
 """
 
 
-@datasets.utils.file_utils.add_start_docstrings(_DESCRIPTION, _KWARGS_DESCRIPTION)
+@evaluate.utils.file_utils.add_start_docstrings(_DESCRIPTION, _KWARGS_DESCRIPTION)
 class F1ForLanguageGeneration(MetricForLanguageGeneration):
     def _info(self):
-        return datasets.MetricInfo(
+        return evaluate.MetricInfo(
             description=_DESCRIPTION,
             citation=_CITATION,
             inputs_description=_KWARGS_DESCRIPTION,
diff --git a/jury/metrics/f1/f1_for_sequence_classification.py b/jury/metrics/f1/f1_for_sequence_classification.py
index 38a5aa3..23e29c2 100644
--- a/jury/metrics/f1/f1_for_sequence_classification.py
+++ b/jury/metrics/f1/f1_for_sequence_classification.py
@@ -1,5 +1,5 @@
 # coding=utf-8
-# Copyright 2021 The HuggingFace Datasets Authors and the current dataset script contributor.
+# Copyright 2021 The HuggingFace evaluate Authors and the current dataset script contributor.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -14,12 +14,12 @@
 # limitations under the License.
 """
 F1 metric. The part of this file is adapted from HuggingFace's
-datasets package implementation of F1 metric. See
-https://github.com/huggingface/datasets/blob/master/metrics/f1/f1.py
+evaluate package implementation of F1 metric. See
+https://github.com/huggingface/evaluate/blob/master/metrics/f1/f1.py
 """
 from typing import List, Union
 
-import datasets
+import evaluate
 from sklearn.metrics import f1_score
 
 from jury.metrics._core import MetricForSequenceClassification, SequenceClassificationInstance, load_metric
@@ -96,13 +96,13 @@
 """
 
 
-@datasets.utils.file_utils.add_start_docstrings(_DESCRIPTION, _KWARGS_DESCRIPTION)
+@evaluate.utils.file_utils.add_start_docstrings(_DESCRIPTION, _KWARGS_DESCRIPTION)
 class F1ForSequenceClassification(MetricForSequenceClassification):
     _precision = load_metric("precision", task="sequence-classification")
     _recall = load_metric("recall", task="sequence-classification")
 
     def _info(self):
-        return datasets.MetricInfo(
+        return evaluate.MetricInfo(
             description=_DESCRIPTION,
             citation=_CITATION,
             inputs_description=_KWARGS_DESCRIPTION,
diff --git a/jury/metrics/meteor/meteor_for_language_generation.py b/jury/metrics/meteor/meteor_for_language_generation.py
index eb7a5f2..44c601e 100644
--- a/jury/metrics/meteor/meteor_for_language_generation.py
+++ b/jury/metrics/meteor/meteor_for_language_generation.py
@@ -1,5 +1,5 @@
 # coding=utf-8
-# Copyright 2021 Open Business Software Solutions, The HuggingFace Datasets Authors.
+# Copyright 2021 Open Business Software Solutions, The HuggingFace evaluate Authors.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -14,13 +14,13 @@
 # limitations under the License.
 """
 METEOR metric. The part of this file is adapted from HuggingFace's
-datasets package implementation of METEOR metric. See
-https://github.com/huggingface/datasets/blob/master/metrics/meteor/meteor.py
+evaluate package implementation of METEOR metric. See
+https://github.com/huggingface/evaluate/blob/master/metrics/meteor/meteor.py
 """
 
 from typing import Callable, Dict, Tuple
 
-import datasets
+import evaluate
 import numpy as np
 from nltk import __version__ as NLTK_VERSION
 from nltk.translate import meteor_score
@@ -94,7 +94,7 @@
 """
 
 
-@datasets.utils.file_utils.add_start_docstrings(_DESCRIPTION, _KWARGS_DESCRIPTION)
+@evaluate.utils.file_utils.add_start_docstrings(_DESCRIPTION, _KWARGS_DESCRIPTION)
 class MeteorForLanguageGeneration(MetricForLanguageGeneration):
     def __init__(self, resulting_name: str = None, compute_kwargs: Dict = None, **kwargs):
         self.should_change_resulting_name = resulting_name is None
@@ -102,7 +102,7 @@ def __init__(self, resulting_name: str = None, compute_kwargs: Dict = None, **kw
         super().__init__(resulting_name=resulting_name, compute_kwargs=compute_kwargs, **kwargs)
 
     def _info(self):
-        return datasets.MetricInfo(
+        return evaluate.MetricInfo(
             description=_DESCRIPTION,
             citation=_CITATION,
             inputs_description=_KWARGS_DESCRIPTION,
diff --git a/jury/metrics/precision/precision_for_language_generation.py b/jury/metrics/precision/precision_for_language_generation.py
index b21f815..bbf9621 100644
--- a/jury/metrics/precision/precision_for_language_generation.py
+++ b/jury/metrics/precision/precision_for_language_generation.py
@@ -1,5 +1,5 @@
 # coding=utf-8
-# Copyright 2021 The HuggingFace Datasets Authors and the current dataset script contributor.
+# Copyright 2021 The HuggingFace evaluate Authors and the current dataset script contributor.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -14,13 +14,13 @@
 # limitations under the License.
 """
 Modified Unigram Precision metric. The part of this file is adapted from HuggingFace's
-datasets package implementation of Precision metric. See
-https://github.com/huggingface/datasets/blob/master/metrics/precision/precision.py
+evaluate package implementation of Precision metric. See
+https://github.com/huggingface/evaluate/blob/master/metrics/precision/precision.py
 """
 from collections import Counter
 from typing import Callable
 
-import datasets
+import evaluate
 import numpy as np
 
 from jury.collator import Collator
@@ -65,10 +65,10 @@
 """
 
 
-@datasets.utils.file_utils.add_start_docstrings(_DESCRIPTION, _KWARGS_DESCRIPTION)
+@evaluate.utils.file_utils.add_start_docstrings(_DESCRIPTION, _KWARGS_DESCRIPTION)
 class PrecisionForLanguageGeneration(MetricForLanguageGeneration):
     def _info(self):
-        return datasets.MetricInfo(
+        return evaluate.MetricInfo(
             description=_DESCRIPTION,
             citation=_CITATION,
             inputs_description=_KWARGS_DESCRIPTION,
diff --git a/jury/metrics/precision/precision_for_sequence_classification.py b/jury/metrics/precision/precision_for_sequence_classification.py
index 53e0c0f..58de787 100644
--- a/jury/metrics/precision/precision_for_sequence_classification.py
+++ b/jury/metrics/precision/precision_for_sequence_classification.py
@@ -1,5 +1,5 @@
 # coding=utf-8
-# Copyright 2021 The HuggingFace Datasets Authors and the current dataset script contributor.
+# Copyright 2021 The HuggingFace evaluate Authors and the current dataset script contributor.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -14,12 +14,12 @@
 # limitations under the License.
 """
 Precision metric. The part of this file is adapted from HuggingFace's
-datasets package implementation of Precision metric. See
-https://github.com/huggingface/datasets/blob/master/metrics/precision/precision.py
+evaluate package implementation of Precision metric. See
+https://github.com/huggingface/evaluate/blob/master/metrics/precision/precision.py
 """
 from typing import Callable
 
-import datasets
+import evaluate
 from sklearn.metrics import precision_score
 
 from jury.metrics._core import MetricForSequenceClassification, SequenceClassificationInstance
@@ -93,10 +93,10 @@
 """
 
 
-@datasets.utils.file_utils.add_start_docstrings(_DESCRIPTION, _KWARGS_DESCRIPTION)
+@evaluate.utils.file_utils.add_start_docstrings(_DESCRIPTION, _KWARGS_DESCRIPTION)
 class PrecisionForSequenceClassification(MetricForSequenceClassification):
     def _info(self):
-        return datasets.MetricInfo(
+        return evaluate.MetricInfo(
             description=_DESCRIPTION,
             citation=_CITATION,
             inputs_description=_KWARGS_DESCRIPTION,
diff --git a/jury/metrics/prism/prism_for_language_generation.py b/jury/metrics/prism/prism_for_language_generation.py
index 6ba7b0d..95be92a 100644
--- a/jury/metrics/prism/prism_for_language_generation.py
+++ b/jury/metrics/prism/prism_for_language_generation.py
@@ -1,5 +1,5 @@
 # coding=utf-8
-# Copyright 2021 Open Business Software Solutions, The HuggingFace Datasets Authors.
+# Copyright 2021 Open Business Software Solutions, The HuggingFace evaluate Authors.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -14,13 +14,13 @@
 # limitations under the License.
 """
 Prism metric. The part of this file is adapted from metric implementations
-of datasets package. See
-https://github.com/huggingface/datasets/blob/master/metrics/
+of evaluate package. See
+https://github.com/huggingface/evaluate/blob/master/metrics/
 """
 import os
 from typing import Callable, Dict, List, Union
 
-import datasets
+import evaluate
 import validators
 
 from jury.metrics import LanguageGenerationInstance
@@ -130,7 +130,7 @@
 }
 
 
-@datasets.utils.file_utils.add_start_docstrings(_DESCRIPTION, _KWARGS_DESCRIPTION)
+@evaluate.utils.file_utils.add_start_docstrings(_DESCRIPTION, _KWARGS_DESCRIPTION)
 class PrismForLanguageGeneration(MetricForLanguageGeneration):
     def __init__(
         self,
@@ -188,7 +188,7 @@ def _download_and_prepare(self, dl_manager) -> None:
         )
 
     def _info(self):
-        return datasets.MetricInfo(
+        return evaluate.MetricInfo(
             description=_DESCRIPTION,
             citation=_CITATION,
             homepage="https://github.com/thompsonb/prism",
diff --git a/jury/metrics/recall/recall_for_language_generation.py b/jury/metrics/recall/recall_for_language_generation.py
index 86d6dc9..8a842f1 100644
--- a/jury/metrics/recall/recall_for_language_generation.py
+++ b/jury/metrics/recall/recall_for_language_generation.py
@@ -1,5 +1,5 @@
 # coding=utf-8
-# Copyright 2021 The HuggingFace Datasets Authors and the current dataset script contributor.
+# Copyright 2021 The HuggingFace evaluate Authors and the current dataset script contributor.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -14,14 +14,14 @@
 # limitations under the License.
 """
 Recall metric. The part of this file is adapted from HuggingFace's
-datasets package implementation of Recall metric. See
-https://github.com/huggingface/datasets/blob/master/metrics/recall/recall.py
+evaluate package implementation of Recall metric. See
+https://github.com/huggingface/evaluate/blob/master/metrics/recall/recall.py
 """
 
 from collections import Counter
 from typing import Callable
 
-import datasets
+import evaluate
 import numpy as np
 
 from jury.collator import Collator
@@ -69,10 +69,10 @@
 """
 
 
-@datasets.utils.file_utils.add_start_docstrings(_DESCRIPTION, _KWARGS_DESCRIPTION)
+@evaluate.utils.file_utils.add_start_docstrings(_DESCRIPTION, _KWARGS_DESCRIPTION)
 class RecallForLanguageGeneration(MetricForLanguageGeneration):
     def _info(self):
-        return datasets.MetricInfo(
+        return evaluate.MetricInfo(
             description=_DESCRIPTION,
             citation=_CITATION,
             inputs_description=_KWARGS_DESCRIPTION,
diff --git a/jury/metrics/recall/recall_for_sequence_classification.py b/jury/metrics/recall/recall_for_sequence_classification.py
index 2aa46da..f181de7 100644
--- a/jury/metrics/recall/recall_for_sequence_classification.py
+++ b/jury/metrics/recall/recall_for_sequence_classification.py
@@ -1,5 +1,5 @@
 # coding=utf-8
-# Copyright 2021 The HuggingFace Datasets Authors and the current dataset script contributor.
+# Copyright 2021 The HuggingFace evaluate Authors and the current dataset script contributor.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -14,12 +14,12 @@
 # limitations under the License.
 """
 Recall metric. The part of this file is adapted from HuggingFace's
-datasets package implementation of Recall metric. See
-https://github.com/huggingface/datasets/blob/master/metrics/recall/recall.py
+evaluate package implementation of Recall metric. See
+https://github.com/huggingface/evaluate/blob/master/metrics/recall/recall.py
 """
 from typing import Callable
 
-import datasets
+import evaluate
 from sklearn.metrics import recall_score
 
 from jury.metrics._core import MetricForSequenceClassification, SequenceClassificationInstance
@@ -86,10 +86,10 @@
 """
 
 
-@datasets.utils.file_utils.add_start_docstrings(_DESCRIPTION, _KWARGS_DESCRIPTION)
+@evaluate.utils.file_utils.add_start_docstrings(_DESCRIPTION, _KWARGS_DESCRIPTION)
 class RecallForSequenceClassification(MetricForSequenceClassification):
     def _info(self):
-        return datasets.MetricInfo(
+        return evaluate.MetricInfo(
             description=_DESCRIPTION,
             citation=_CITATION,
             inputs_description=_KWARGS_DESCRIPTION,
diff --git a/jury/metrics/rouge/rouge_for_language_generation.py b/jury/metrics/rouge/rouge_for_language_generation.py
index 473f799..b2b8b80 100644
--- a/jury/metrics/rouge/rouge_for_language_generation.py
+++ b/jury/metrics/rouge/rouge_for_language_generation.py
@@ -1,5 +1,5 @@
 # coding=utf-8
-# Copyright 2021 Open Business Software Solutions, The HuggingFace Datasets Authors.
+# Copyright 2021 Open Business Software Solutions, The HuggingFace evaluate Authors.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -14,13 +14,13 @@
 # limitations under the License.
 """
 ROUGE metric. The part of this file is adapted from HuggingFace's
-datasets package implementation of ROUGE metric. See
-https://github.com/huggingface/datasets/blob/master/metrics/rouge/rouge.py
+evaluate package implementation of ROUGE metric. See
+https://github.com/huggingface/evaluate/blob/master/metrics/rouge/rouge.py
 """
 
 from typing import Callable, Dict, List, Optional, Union
 
-import datasets
+import evaluate
 import pandas as pd
 from rouge_score import rouge_scorer, scoring
 
@@ -65,7 +65,7 @@
         `"rouge{n}"` (e.g. `"rouge1"`, `"rouge2"`) where: {n} is the n-gram based scoring,
         `"rougeL"`: Longest common subsequence based scoring.
         `"rougeLSum"`: rougeLsum splits text using `"\n"`.
-        See details in https://github.com/huggingface/datasets/issues/617
+        See details in https://github.com/huggingface/evaluate/issues/617
     use_stemmer: Bool indicating whether Porter stemmer should be used to strip word suffixes.
     use_agregator: Return aggregates if this is set to True
 Returns:
@@ -94,10 +94,10 @@
 """
 
 
-@datasets.utils.file_utils.add_start_docstrings(_DESCRIPTION, _KWARGS_DESCRIPTION)
+@evaluate.utils.file_utils.add_start_docstrings(_DESCRIPTION, _KWARGS_DESCRIPTION)
 class RougeForLanguageGeneration(MetricForLanguageGeneration):
     def _info(self):
-        return datasets.MetricInfo(
+        return evaluate.MetricInfo(
             description=_DESCRIPTION,
             citation=_CITATION,
             inputs_description=_KWARGS_DESCRIPTION,
diff --git a/jury/metrics/sacrebleu/sacrebleu_for_language_generation.py b/jury/metrics/sacrebleu/sacrebleu_for_language_generation.py
index dcefa6e..b45cc3f 100644
--- a/jury/metrics/sacrebleu/sacrebleu_for_language_generation.py
+++ b/jury/metrics/sacrebleu/sacrebleu_for_language_generation.py
@@ -1,5 +1,5 @@
 # coding=utf-8
-# Copyright 2021 Open Business Software Solutions, The HuggingFace Datasets Authors.
+# Copyright 2021 Open Business Software Solutions, The HuggingFace evaluate Authors.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -14,14 +14,14 @@
 # limitations under the License.
 """
 SacreBLEU metric. The part of this file is adapted from SacreBLEU implementation
-of datasets package. See
-https://github.com/huggingface/datasets/blob/master/metrics/sacrebleu/sacrebleu.py
+of evaluate package. See
+https://github.com/huggingface/evaluate/blob/master/metrics/sacrebleu/sacrebleu.py
 """
 
 import math
 from typing import Callable, Dict, Sequence
 
-import datasets
+import evaluate
 from packaging import version
 
 from jury.collator import Collator
@@ -128,7 +128,7 @@
 """
 
 
-@datasets.utils.file_utils.add_start_docstrings(_DESCRIPTION, _KWARGS_DESCRIPTION)
+@evaluate.utils.file_utils.add_start_docstrings(_DESCRIPTION, _KWARGS_DESCRIPTION)
 class SacrebleuForLanguageGeneration(MetricForLanguageGeneration):
     def _download_and_prepare(self, dl_manager):
         global scb
@@ -145,7 +145,7 @@ def _info(self):
                 "To use `sacrebleu`, the module `sacrebleu>=1.4.12` is required, and the current version of `sacrebleu` doesn't match this condition.\n"
                 'You can install it with `pip install "sacrebleu>=1.4.12"`.'
             )
-        return datasets.MetricInfo(
+        return evaluate.MetricInfo(
             description=_DESCRIPTION,
             citation=_CITATION,
             homepage="https://github.com/mjpost/sacreBLEU",
diff --git a/jury/metrics/seqeval/seqeval_for_sequence_classification.py b/jury/metrics/seqeval/seqeval_for_sequence_classification.py
index 111680b..460eb79 100644
--- a/jury/metrics/seqeval/seqeval_for_sequence_classification.py
+++ b/jury/metrics/seqeval/seqeval_for_sequence_classification.py
@@ -1,5 +1,5 @@
 # coding=utf-8
-# Copyright 2021 Open Business Software Solutions, The HuggingFace Datasets Authors.
+# Copyright 2021 Open Business Software Solutions, The HuggingFace evaluate Authors.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -14,13 +14,13 @@
 # limitations under the License.
 """
 SeqEval metric. The part of this file is adapted from seqeval implementation
-of datasets package. See
-https://github.com/huggingface/datasets/blob/master/metrics/seqeval/seqeval.py
+of evaluate package. See
+https://github.com/huggingface/evaluate/blob/master/metrics/seqeval/seqeval.py
 """
 import importlib
 from typing import Dict, List, Optional, Union
 
-import datasets
+import evaluate
 
 from jury.metrics._core import MetricForSequenceLabeling, SequenceLabelingInstance
 from jury.metrics._core.utils import PackagePlaceholder, requirement_message
@@ -123,7 +123,7 @@
 """
 
 
-@datasets.utils.file_utils.add_start_docstrings(_DESCRIPTION, _KWARGS_DESCRIPTION)
+@evaluate.utils.file_utils.add_start_docstrings(_DESCRIPTION, _KWARGS_DESCRIPTION)
 class SeqevalForLanguageGeneration(MetricForSequenceLabeling):
     def _download_and_prepare(self, dl_manager):
         global seqeval
@@ -135,7 +135,7 @@ def _download_and_prepare(self, dl_manager):
             super(SeqevalForLanguageGeneration, self)._download_and_prepare(dl_manager)
 
     def _info(self):
-        return datasets.MetricInfo(
+        return evaluate.MetricInfo(
             description=_DESCRIPTION,
             citation=_CITATION,
             homepage="https://github.com/chakki-works/seqeval",
diff --git a/jury/metrics/squad/squad_for_language_generation.py b/jury/metrics/squad/squad_for_language_generation.py
index d9a7044..4c480f5 100644
--- a/jury/metrics/squad/squad_for_language_generation.py
+++ b/jury/metrics/squad/squad_for_language_generation.py
@@ -1,5 +1,5 @@
 # coding=utf-8
-# Copyright 2021 Open Business Software Solutions, The HuggingFace Datasets Authors.
+# Copyright 2021 Open Business Software Solutions, The HuggingFace evaluate Authors.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -13,11 +13,11 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 """ SQuAD metric. The part of this file is adapted from SacreBLEU implementation
-of datasets package. See
-https://github.com/huggingface/datasets/blob/master/metrics/squad/squad.py"""
+of evaluate package. See
+https://github.com/huggingface/evaluate/blob/master/metrics/squad/squad.py"""
 from typing import Callable, Dict, List
 
-import datasets
+import evaluate
 import numpy as np
 import pandas as pd
 
@@ -77,19 +77,19 @@
 """
 
 
-@datasets.utils.file_utils.add_start_docstrings(_DESCRIPTION, _KWARGS_DESCRIPTION)
+@evaluate.utils.file_utils.add_start_docstrings(_DESCRIPTION, _KWARGS_DESCRIPTION)
 class SquadForLanguageGeneration(MetricForLanguageGeneration):
     def _download_and_prepare(self, dl_manager) -> None:
         """
         Downloads and import the computation of squad score from the implementation
-        of Squad computation from huggingface/datasets. See
-        https://github.com/huggingface/datasets/blob/master/metrics/squad/evaluate.py
+        of Squad computation from huggingface/evaluate. See
+        https://github.com/huggingface/evaluate/blob/master/metrics/squad/evaluate.py
         """
-        squad_source = "https://raw.githubusercontent.com/huggingface/datasets/master/metrics/squad/evaluate.py"
+        squad_source = f"https://raw.githubusercontent.com/huggingface/evaluate/v{evaluate.__version__}/metrics/squad/compute_score.py"
         self.external_module_path = dl_manager.download(squad_source)
 
     def _info(self):
-        return datasets.MetricInfo(
+        return evaluate.MetricInfo(
             description=_DESCRIPTION,
             citation=_CITATION,
             inputs_description=_KWARGS_DESCRIPTION,
@@ -130,7 +130,7 @@ def _compute_single_pred_single_ref(self, predictions: Collator, references: Col
                 ]
             }
         ]
-        evaluation_fn = self._get_external_resource("squad_evaluate", attr="evaluate")
+        evaluation_fn = self._get_external_resource("squad_compute_score", attr="compute_score")
         score = evaluation_fn(dataset=dataset, predictions=pred_dict)
         for metric_, score_ in score.items():
             score[metric_] = score_ / 100
diff --git a/jury/metrics/ter/ter_for_language_generation.py b/jury/metrics/ter/ter_for_language_generation.py
index c26a26c..054a1d8 100644
--- a/jury/metrics/ter/ter_for_language_generation.py
+++ b/jury/metrics/ter/ter_for_language_generation.py
@@ -1,5 +1,5 @@
 # coding=utf-8
-# Copyright 2021 Open Business Software Solutions, The HuggingFace Datasets Authors.
+# Copyright 2021 Open Business Software Solutions, The HuggingFace evaluate Authors.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -14,12 +14,12 @@
 # limitations under the License.
 """
 Translation Edit Rate (TER) metric. The part of this file is adapted from HuggingFace's
-datasets package implementation of TER metric. See
-https://github.com/huggingface/datasets/blob/master/metrics/ter/ter.py
+evaluate package implementation of TER metric. See
+https://github.com/huggingface/evaluate/blob/master/metrics/ter/ter.py
 """
 from typing import Callable, Dict, Sequence
 
-import datasets
+import evaluate
 
 from jury.collator import Collator
 from jury.metrics import LanguageGenerationInstance, MetricForLanguageGeneration
@@ -65,7 +65,7 @@
 here: https://github.com/jhclark/tercom.
 The implementation here is slightly different from sacrebleu in terms of the required input format. The length of
 the references and hypotheses lists need to be the same, so you may need to transpose your references compared to
-sacrebleu's required input format. See https://github.com/huggingface/datasets/issues/3154#issuecomment-950746534
+sacrebleu's required input format. See https://github.com/huggingface/evaluate/issues/3154#issuecomment-950746534
 See the README.md file at https://github.com/mjpost/sacreBLEU#ter for more information.
 """
 
@@ -95,10 +95,10 @@
 """
 
 
-@datasets.utils.file_utils.add_start_docstrings(_DESCRIPTION, _KWARGS_DESCRIPTION)
+@evaluate.utils.file_utils.add_start_docstrings(_DESCRIPTION, _KWARGS_DESCRIPTION)
 class TERForLanguageGeneration(MetricForLanguageGeneration):
     def _info(self):
-        return datasets.MetricInfo(
+        return evaluate.MetricInfo(
             description=_DESCRIPTION,
             citation=_CITATION,
             homepage="http://www.cs.umd.edu/~snover/tercom/",
diff --git a/jury/metrics/wer/wer_for_language_generation.py b/jury/metrics/wer/wer_for_language_generation.py
index 154f339..8a90a92 100644
--- a/jury/metrics/wer/wer_for_language_generation.py
+++ b/jury/metrics/wer/wer_for_language_generation.py
@@ -1,5 +1,5 @@
 # coding=utf-8
-# Copyright 2021 Open Business Software Solutions, The HuggingFace Datasets Authors.
+# Copyright 2021 Open Business Software Solutions, The HuggingFace evaluate Authors.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -13,13 +13,13 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 """ Word Error Ratio (WER) metric. The part of this file is adapted from HuggingFace's
-datasets package implementation of CER metric. See
-https://github.com/huggingface/datasets/blob/master/metrics/wer/wer.py
+evaluate package implementation of CER metric. See
+https://github.com/huggingface/evaluate/blob/master/metrics/wer/wer.py
 """
 import warnings
 from typing import Callable, List, Tuple, Union
 
-import datasets
+import evaluate
 
 from jury.metrics import LanguageGenerationInstance, MetricForLanguageGeneration
 from jury.metrics._core.utils import PackagePlaceholder, requirement_message
@@ -85,10 +85,10 @@
 """
 
 
-@datasets.utils.file_utils.add_start_docstrings(_DESCRIPTION, _KWARGS_DESCRIPTION)
+@evaluate.utils.file_utils.add_start_docstrings(_DESCRIPTION, _KWARGS_DESCRIPTION)
 class WERForLanguageGeneration(MetricForLanguageGeneration):
     def _info(self):
-        return datasets.MetricInfo(
+        return evaluate.MetricInfo(
             description=_DESCRIPTION,
             citation=_CITATION,
             inputs_description=_KWARGS_DESCRIPTION,
diff --git a/requirements-dev.txt b/requirements-dev.txt
index c0aed03..4a55099 100644
--- a/requirements-dev.txt
+++ b/requirements-dev.txt
@@ -2,5 +2,5 @@
 # Falling back to fork for Windows related issue: https://github.com/google-research/bleurt/issues/49
 bleurt @ git+https://github.com/devrimcavusoglu/bleurt.git
 
-# for datasets metric test
+# for evaluate metric test
 math_equivalence @ git+https://github.com/hendrycks/math.git
diff --git a/requirements.txt b/requirements.txt
index ab82fca..2d92c26 100755
--- a/requirements.txt
+++ b/requirements.txt
@@ -1,9 +1,7 @@
 click==8.0.4
-datasets>=2.0.0
+evaluate>=0.2.2,<0.3
 fire>=0.4.0
 nltk>=3.6.6,<3.7.1
-numpy>=1.21.0
-pandas>=1.1.5
 rouge-score==0.0.4
 sklearn
 tqdm


Test patch:
diff --git a/tests/jury/conftest.py b/tests/jury/conftest.py
index fc92be4..7465ae7 100644
--- a/tests/jury/conftest.py
+++ b/tests/jury/conftest.py
@@ -64,7 +64,7 @@
     {"path": "squad"},
 ]
 
-_DATASETS_METRICS = "competition_math"
+_HF_METRICS = "competition_math"
 
 _TEST_METRICS_SEQUENCE_CLASSIFICATION = [
     {"path": "accuracy", "task": "sequence-classification"},
@@ -177,8 +177,8 @@ def jury_list_mixed():
 
 
 @pytest.fixture(scope="function")
-def jury_datasets():
-    return Jury(metrics=_DATASETS_METRICS)
+def jury_hf():
+    return Jury(metrics=_HF_METRICS)
 
 
 @pytest.fixture(scope="function")
diff --git a/tests/jury/test_jury.py b/tests/jury/test_jury.py
index 6388b9f..110918b 100644
--- a/tests/jury/test_jury.py
+++ b/tests/jury/test_jury.py
@@ -1,4 +1,4 @@
-import datasets
+import evaluate
 import numpy as np
 import pytest
 
@@ -44,8 +44,8 @@ def output_evaluate_list_mixed_input():
 
 @pytest.fixture
 @get_expected_output(prefix=None)
-def output_evaluate_datasets_metric():
-    return output_evaluate_datasets_metric.output
+def output_evaluate_hf_metric():
+    return output_evaluate_hf_metric.output
 
 
 @pytest.fixture
@@ -121,9 +121,9 @@ def test_evaluate_list_mixed_input(predictions, references, jury_list_mixed, out
     assert_almost_equal_dict(actual=scores, desired=output_evaluate_list_mixed_input)
 
 
-def test_evaluate_datasets_metric(predictions, references, jury_datasets, output_evaluate_datasets_metric):
-    scores = jury_datasets(predictions=predictions, references=references)
-    assert_almost_equal_dict(actual=scores, desired=output_evaluate_datasets_metric)
+def test_evaluate_hf_metric(predictions, references, jury_hf, output_evaluate_hf_metric):
+    scores = jury_hf(predictions=predictions, references=references)
+    assert_almost_equal_dict(actual=scores, desired=output_evaluate_hf_metric)
 
 
 def test_evaluate_corpus(single_prediction_array, multiple_references, jury_base, output_evaluate_corpus):
@@ -173,7 +173,7 @@ def test_load_metric():
     from jury.metrics._core import Metric as JuryMetric
 
     assert isinstance(load_metric("squad"), JuryMetric)
-    assert isinstance(load_metric("squad_v2"), datasets.Metric)
+    assert isinstance(load_metric("squad_v2"), evaluate.Metric)
 
     with pytest.raises(FileNotFoundError):
         load_metric("abcdefgh")
diff --git a/tests/test_data/custom_bleu/custom_bleu_for_language_generation.py b/tests/test_data/custom_bleu/custom_bleu_for_language_generation.py
index 2a67ca0..393ecc1 100644
--- a/tests/test_data/custom_bleu/custom_bleu_for_language_generation.py
+++ b/tests/test_data/custom_bleu/custom_bleu_for_language_generation.py
@@ -1,5 +1,5 @@
 # coding=utf-8
-# Copyright 2021 Open Business Software Solutions, The HuggingFace Datasets Authors.
+# Copyright 2021 Open Business Software Solutions, The HuggingFace Evaluate Authors.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -14,14 +14,15 @@
 # limitations under the License.
 """
 BLEU metric. The part of this file is adapted from HuggingFace's
-datasets package implementation of BLEU metric. See
-https://github.com/huggingface/datasets/blob/master/metrics/bleu/bleu.py
+evaluate package implementation of BLEU metric. See
+https://github.com/huggingface/evaluate/blob/master/metrics/bleu/bleu.py
 """
 
 import math
 from typing import Callable, Dict, Tuple
 
 import datasets
+import evaluate
 
 from jury.collator import Collator
 from jury.metrics._core import MetricForLanguageGeneration
@@ -100,7 +101,7 @@
 """
 
 
-@datasets.utils.file_utils.add_start_docstrings(_DESCRIPTION, _KWARGS_DESCRIPTION)
+@evaluate.utils.file_utils.add_start_docstrings(_DESCRIPTION, _KWARGS_DESCRIPTION)
 class CustomBleuForLanguageGeneration(MetricForLanguageGeneration):
     def __init__(self, resulting_name: str = None, compute_kwargs: Dict = None, **kwargs):
         self.should_change_resulting_name = resulting_name is None
@@ -108,7 +109,7 @@ def __init__(self, resulting_name: str = None, compute_kwargs: Dict = None, **kw
         super().__init__(resulting_name=resulting_name, compute_kwargs=compute_kwargs, **kwargs)
 
     def _info(self):
-        return datasets.MetricInfo(
+        return evaluate.MetricInfo(
             description=_DESCRIPTION,
             citation=_CITATION,
             inputs_description=_KWARGS_DESCRIPTION,
diff --git a/tests/test_data/expected_outputs/test_jury.json b/tests/test_data/expected_outputs/test_jury.json
index 40dc864..d9e4f4f 100644
--- a/tests/test_data/expected_outputs/test_jury.json
+++ b/tests/test_data/expected_outputs/test_jury.json
@@ -354,7 +354,7 @@
       "f1": 0.6857142857142858
     }
   },
-  "evaluate_datasets_metric": {
+  "evaluate_hf_metric": {
     "total_items": 2,
     "empty_items": 0,
     "accuracy": 0.0