Repository: iKernels/transformers-lightning (language: python)

Issue:
Remove metrics package



Discussion / hints:

Fix patch:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 236971a..b8231de 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -5,6 +5,11 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/).
 
 
+## 0.7.7
+
+- Completely removed metrics package, now all metrics available in `torchmetrics` library.
+
+
 ## 0.7.6
 
 - Package publication fixed
diff --git a/README.md b/README.md
index 894e418..1fb0aea 100644
--- a/README.md
+++ b/README.md
@@ -1,6 +1,6 @@
 # transformers-lightning
 
-A collection of `adapters`, `callbacks`, `datamodules`, `datasets`, `language-modeling`, `metrics`, `models`, `schedulers` and `optimizers` to better integrate the [PyTorch Lightning](https://pytorch-lightning.readthedocs.io/en/stable/lightning-module.html) and the [Transformers](https://huggingface.co/transformers/) libraries.
+A collection of `adapters`, `callbacks`, `datamodules`, `datasets`, `language-modeling`, `models`, `schedulers` and `optimizers` to better integrate the [PyTorch Lightning](https://pytorch-lightning.readthedocs.io/en/stable/lightning-module.html) and the [Transformers](https://huggingface.co/transformers/) libraries.
 
 I'm happy to announce that all the `metrics` contained in this package has been successfully integrated into [torchmetrics](https://github.com/PyTorchLightning/metrics/tree/master/torchmetrics/retrieval).
 
@@ -17,7 +17,6 @@ I'm happy to announce that all the `metrics` contained in this package has been
   * [2.6. Language modeling](transformers_lightning/language_modeling)
   * [2.7. Schedulers](transformers_lightning/schedulers)
   * [2.8. Optimizers](transformers_lightning/optimizers)
-  * [2.9. Metrics](transformers_lightning/metrics)
 
 **[3. Main file](#main)**
 
diff --git a/transformers_lightning/__init__.py b/transformers_lightning/__init__.py
index 677d8d4..ec73a57 100644
--- a/transformers_lightning/__init__.py
+++ b/transformers_lightning/__init__.py
@@ -4,7 +4,6 @@
 import transformers_lightning.datasets  # noqa: F401
 import transformers_lightning.defaults  # noqa: F401
 import transformers_lightning.language_modeling  # noqa: F401
-import transformers_lightning.metrics  # noqa: F401
 import transformers_lightning.models  # noqa: F401
 import transformers_lightning.optimizers  # noqa: F401
 import transformers_lightning.schedulers  # noqa: F401
diff --git a/transformers_lightning/metrics/README.md b/transformers_lightning/metrics/README.md
deleted file mode 100644
index 5bf3f58..0000000
--- a/transformers_lightning/metrics/README.md
+++ /dev/null
@@ -1,4 +0,0 @@
-# Metrics
-
-All metrics were integrated successfully in the [torchmetrics](https://github.com/PyTorchLightning/metrics/tree/master/torchmetrics/retrieval) library.
-Now this package contains only the new `HitRate@K` metric, which will be likely moved to `torchmetrics` in the next release. `HitRate@K` is the fraction of queries for which at least a positive candidate was found in the first `K` positions.
diff --git a/transformers_lightning/metrics/__init__.py b/transformers_lightning/metrics/__init__.py
deleted file mode 100644
index df23b61..0000000
--- a/transformers_lightning/metrics/__init__.py
+++ /dev/null
@@ -1,1 +0,0 @@
-from transformers_lightning.metrics.hit_rate import HitRate  # noqa: F401
diff --git a/transformers_lightning/metrics/functional/__init__.py b/transformers_lightning/metrics/functional/__init__.py
deleted file mode 100644
index e69de29..0000000
diff --git a/transformers_lightning/metrics/functional/hit_rate.py b/transformers_lightning/metrics/functional/hit_rate.py
deleted file mode 100644
index b0dadc1..0000000
--- a/transformers_lightning/metrics/functional/hit_rate.py
+++ /dev/null
@@ -1,38 +0,0 @@
-import torch
-from torch import Tensor, tensor
-from torchmetrics.utilities.checks import _check_retrieval_functional_inputs
-
-
-def retrieval_hit_rate(preds: Tensor, target: Tensor, k: int = None) -> Tensor:
-    r"""
-    Computes the hit rate (for information retrieval).
-    The hit rate is 1.0 if there is at least one relevant document among all the top `k` retrieved documents.
-
-    ``preds`` and ``target`` should be of the same shape and live on the same device. If no ``target`` is ``True``,
-    ``0`` is returned. ``target`` must be either `bool` or `integers` and ``preds`` must be `float`,
-    otherwise an error is raised. If you want to measure Precision@K, ``k`` must be a positive integer.
-
-    Args:
-        preds: estimated probabilities of each document to be relevant.
-        target: ground truth about each document being relevant or not.
-        k: consider only the top k elements (default: None)
-
-    Returns:
-        a single-value tensor with the precision (at ``k``) of the predictions ``preds`` w.r.t. the labels ``target``.
-
-    Example:
-        >>> preds = tensor([0.2, 0.3, 0.5])
-        >>> target = tensor([True, False, True])
-        >>> retrieval_hit_rate(preds, target, k=2)
-        tensor(0.5000)
-    """
-    preds, target = _check_retrieval_functional_inputs(preds, target)
-
-    if not (isinstance(k, int) and k > 0):
-        raise ValueError("`k` has to be a positive integer")
-
-    if not target.sum():
-        return tensor(0.0, device=preds.device)
-
-    relevant = target[torch.argsort(preds, dim=-1, descending=True)][:k].sum()
-    return (relevant > 0).float()
diff --git a/transformers_lightning/metrics/hit_rate.py b/transformers_lightning/metrics/hit_rate.py
deleted file mode 100644
index bb6bed3..0000000
--- a/transformers_lightning/metrics/hit_rate.py
+++ /dev/null
@@ -1,81 +0,0 @@
-from typing import Any, Callable, Optional
-
-from torch import Tensor
-from torchmetrics.retrieval.retrieval_metric import RetrievalMetric
-
-from transformers_lightning.metrics.functional.hit_rate import retrieval_hit_rate
-
-
-class HitRate(RetrievalMetric):
-    r"""
-    Computes Hit Rate @ k, where 1.0 is given to queries where a relevant document is in the top `k`.
-
-    Works with binary target data. Accepts float predictions from a model output.
-
-    Forward accepts:
-
-    - ``preds`` (float tensor): ``(N, ...)``
-    - ``target`` (long or bool tensor): ``(N, ...)``
-    - ``indexes`` (long tensor): ``(N, ...)``
-
-    ``indexes``, ``preds`` and ``target`` must have the same dimension.
-    ``indexes`` indicate to which query a prediction belongs.
-    Predictions will be first grouped by ``indexes`` and then `Precision` will be computed as the mean
-    of the `Precision` over each query.
-
-    Args:
-        empty_target_action:
-            Specify what to do with queries that do not have at least a positive ``target``. Choose from:
-
-            - ``'neg'``: those queries count as ``0.0`` (default)
-            - ``'pos'``: those queries count as ``1.0``
-            - ``'skip'``: skip those queries; if all queries are skipped, ``0.0`` is returned
-            - ``'error'``: raise a ``ValueError``
-
-        compute_on_step:
-            Forward only calls ``update()`` and return None if this is set to False. default: True
-        dist_sync_on_step:
-            Synchronize metric state across processes at each ``forward()``
-            before returning the value at the step. default: False
-        process_group:
-            Specify the process group on which synchronization is called. default: None (which selects
-            the entire world)
-        dist_sync_fn:
-            Callback that performs the allgather operation on the metric state. When `None`, DDP
-            will be used to perform the allgather. default: None
-        k: consider only the top k elements for each query. default: None
-
-    Example:
-        >>> from torchmetrics import WBQARetrievalPrecision
-        >>> indexes = tensor([0, 0, 0, 1, 1, 1, 1])
-        >>> preds = tensor([0.2, 0.3, 0.5, 0.1, 0.3, 0.5, 0.2])
-        >>> target = tensor([False, False, True, False, True, False, True])
-        >>> p2 = WBQARetrievalPrecision(k=2)
-        >>> p2(preds, target, indexes=indexes)
-        tensor(0.5000)
-    """
-
-    def __init__(
-        self,
-        empty_target_action: str = 'neg',
-        compute_on_step: bool = True,
-        dist_sync_on_step: bool = False,
-        process_group: Optional[Any] = None,
-        dist_sync_fn: Callable = None,
-        k: int = None
-    ):
-        super().__init__(
-            empty_target_action=empty_target_action,
-            compute_on_step=compute_on_step,
-            dist_sync_on_step=dist_sync_on_step,
-            process_group=process_group,
-            dist_sync_fn=dist_sync_fn
-        )
-
-        if k is None:
-            raise ValueError("`k` has to be a positive integer")
-
-        self.k = k
-
-    def _metric(self, preds: Tensor, target: Tensor) -> Tensor:
-        return retrieval_hit_rate(preds=preds, target=target, k=self.k)


Test patch:
diff --git a/tests/test_metrics.py b/tests/test_metrics.py
deleted file mode 100644
index 97ce6a5..0000000
--- a/tests/test_metrics.py
+++ /dev/null
@@ -1,61 +0,0 @@
-import pytest
-import torch
-
-from transformers_lightning.metrics import HitRate
-
-
-@pytest.mark.parametrize(
-    "metric_class, metric_kwargs, preds, target, idx, expected_result", (
-        [
-            HitRate,
-            {'k': 1},
-            torch.tensor([0.3, 0.5, 0.4, 0.9, 0.2], dtype=torch.float),
-            torch.tensor([False, False, True, True, False], dtype=torch.bool),
-            torch.tensor([0, 0, 1, 1, 1], dtype=torch.long),
-            torch.tensor(0.5)
-        ],
-        [
-            HitRate,
-            {'k': 1},
-            torch.tensor([0.5, 0.2, 0.1, -0.6, 0.8], dtype=torch.float),
-            torch.tensor([False, False, True, True, False], dtype=torch.bool),
-            torch.tensor([0, 0, 1, 1, 0], dtype=torch.long),
-            torch.tensor(0.5)
-        ],
-        [
-            HitRate,
-            {'k': 2},
-            torch.tensor([0.3, 0.7, 0.5, 0.4, 0.9, 0.2], dtype=torch.float),
-            torch.tensor([False, True, False, False, True, False], dtype=torch.bool),
-            torch.tensor([0, 0, 1, 1, 1, 1], dtype=torch.long),
-            torch.tensor(1.0)
-        ],
-        [
-            HitRate,
-            {'k': 1},
-            torch.tensor([0.3, 0.7, 0.5, 0.4, 0.9, 0.2], dtype=torch.float),
-            torch.tensor([False, False, True, True, False, False], dtype=torch.bool),
-            torch.tensor([0, 0, 0, 1, 1, 1], dtype=torch.long),
-            torch.tensor(0.0)
-        ],
-        [
-            HitRate,
-            {'k': 2},
-            torch.tensor([0.3, 0.7, 0.5, 0.4, 0.9, 0.2, 0.4, 0.5, 0.0], dtype=torch.float),
-            torch.tensor([True, False, True, False, False, False, False, True, False], dtype=torch.bool),
-            torch.tensor([0, 0, 0, 1, 1, 1, 2, 2, 2], dtype=torch.long),
-            torch.tensor(2 / 3)
-        ],
-    )
-)
-def test_metrics(metric_class, metric_kwargs, preds, target, idx, expected_result):
-
-    metric = metric_class(**metric_kwargs)
-    metric.update(preds, target, idx)
-
-    value = metric.compute()
-
-    assert torch.allclose(
-        expected_result,
-        value,
-    ), f"{expected_result} vs {value}"