Repository: catalyst-team/catalyst (language: python)

Issue:
SklearnMetric integration
## 🚀 Feature Request
<!-- A clear and concise description of the feature proposal. -->

Let's add official support for all sklearn metrics to our Metric API.


### Motivation
<!-- Please outline the motivation for the proposal. Is your feature request related to a problem? e.g., I'm always frustrated when [...]. If this is related to another GitHub issue, please link here too -->

Sklearn metrics play an essential role in ml models benchmarking.

### Proposal
<!-- 
A clear and concise description of what you want to happen.
You could use minimal examples - https://github.com/catalyst-team/catalyst#minimal-examples. 
-->

Possible usage

```python
import torch
from torch.utils.data import DataLoader, TensorDataset
from catalyst import dl

# sample data
num_samples, num_features, num_classes = int(1e4), int(1e1), 4
X = torch.rand(num_samples, num_features)
y = (torch.rand(num_samples, num_classes) > 0.5).to(torch.float32)

# pytorch loaders
dataset = TensorDataset(X, y)
loader = DataLoader(dataset, batch_size=32, num_workers=1)
loaders = {"train": loader, "valid": loader}

# model, criterion, optimizer, scheduler
model = torch.nn.Linear(num_features, num_classes)
criterion = torch.nn.BCEWithLogitsLoss()
optimizer = torch.optim.Adam(model.parameters())
scheduler = torch.optim.lr_scheduler.MultiStepLR(optimizer, [2])

# model training
runner = dl.SupervisedRunner(
    input_key="features", output_key="logits", target_key="targets", loss_key="loss"
)
runner.train(
    model=model,
    criterion=criterion,
    optimizer=optimizer,
    scheduler=scheduler,
    loaders=loaders,
    logdir="./logdir",
    num_epochs=3,
    valid_loader="valid",
    valid_metric="accuracy",
    minimize_valid_metric=False,
    verbose=True,
    callbacks=[
        dl.BatchTransformCallback(
            transform=torch.sigmoid,
            scope="on_batch_end",
            input_key="logits",
            output_key="scores"
        ),
        dl.AUCCallback(input_key="logits", target_key="targets"),
        dl.MultilabelAccuracyCallback(input_key="logits", target_key="targets", threshold=0.5),
        # here is the trick
        dl.SklearnCallback(
            keys={"y_score": "scores", "y_true": "targets"}, 
            metirc_fn=sklearn.metrics.coverage_error, 
            metric_key="coverage_error",
        )
    ]
)
```

We also should add sklearn as an extra module to the Catalyst [like `ml` one](https://github.com/catalyst-team/catalyst/blob/master/catalyst/settings.py#L154).

### Alternatives
<!-- A clear and concise description of any alternative solutions or features you've considered. -->

We could implement everything with PyTorch, which we should do next. Nevertheless, sklearn full support is still required as a great integration.


### Additional context
<!-- Add any other context or screenshots about the feature request here. -->


### Checklist
- [x] feature proposal description
- [x] motivation
- [x] extra proposal context / proposal alternatives review
  

### FAQ
Please review the FAQ before submitting an issue:
- [x] I have read the [documentation and FAQ](https://catalyst-team.github.io/catalyst/)
- [x] I have reviewed the [minimal examples section](https://github.com/catalyst-team/catalyst#minimal-examples)
- [x] I have checked the [changelog](https://github.com/catalyst-team/catalyst/blob/master/CHANGELOG.md) for main framework updates
- [x] I have read the [contribution guide](https://github.com/catalyst-team/catalyst/blob/master/CONTRIBUTING.md)
- [x] I have joined [Catalyst slack (#__questions channel)](https://join.slack.com/t/catalyst-team-core/shared_invite/zt-d9miirnn-z86oKDzFMKlMG4fgFdZafw) for issue discussion



Discussion / hints:
I could handle it:)

Fix patch:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 92b0c87c85..4b79f2f58f 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -13,6 +13,7 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/).
 - FAQ docs: multiple input and output keys, engine tutorial ([#1202](https://github.com/catalyst-team/catalyst/pull/1202))
 - minimal Config API example ([#1215](https://github.com/catalyst-team/catalyst/pull/1215))
 - Distributed RL example (Catalyst.RL 2.0 concepts) ([#1224](https://github.com/catalyst-team/catalyst/pull/1224))
+- SklearnCallback as integration of sklearn metrics ([#1198](https://github.com/catalyst-team/catalyst/pull/1198))
 
 ### Changed
 
diff --git a/catalyst/callbacks/metrics/__init__.py b/catalyst/callbacks/metrics/__init__.py
index 7194f27e51..b9331bea67 100644
--- a/catalyst/callbacks/metrics/__init__.py
+++ b/catalyst/callbacks/metrics/__init__.py
@@ -29,3 +29,6 @@
     IOUCallback,
     TrevskyCallback,
 )
+
+if SETTINGS.ml_required:
+    from catalyst.callbacks.metrics.scikit_learn import SklearnBatchCallback, SklearnLoaderCallback
diff --git a/catalyst/callbacks/metrics/scikit_learn.py b/catalyst/callbacks/metrics/scikit_learn.py
new file mode 100644
index 0000000000..61abba5c43
--- /dev/null
+++ b/catalyst/callbacks/metrics/scikit_learn.py
@@ -0,0 +1,267 @@
+from typing import Any, Callable, Dict, Mapping, Union
+from functools import partial
+
+import torch
+
+from catalyst.callbacks.metric import FunctionalBatchMetricCallback, LoaderMetricCallback
+from catalyst.core.runner import IRunner
+from catalyst.metrics import FunctionalBatchMetric, FunctionalLoaderMetric
+from catalyst.settings import SETTINGS
+
+if SETTINGS.ml_required:
+    import sklearn
+
+
+class SklearnBatchCallback(FunctionalBatchMetricCallback):
+    """SklearnBatchCallback implements an integration of **batch-based** Sklearn metrics
+
+    Args:
+        keys: a dictionary containing:
+            a mapping between ``metric_fn`` arguments and keys in ``runner.batch``
+            other arguments needed for ``metric_fn``
+        metric_fn: metric function that gets outputs, targets, and other arguments given
+            in ``keys`` and returns score
+        metric_key: key to store computed metric in ``runner.batch_metrics`` dictionary
+        log_on_batch: boolean flag to log computed metrics every batch
+        metric_kwargs: additional parameters for ``metric_fn``
+
+    .. note::
+        catalyst[ml] required for this callback
+
+    Examples:
+
+    .. code-block:: python
+
+        import sklearn
+        import torch
+        from torch.utils.data import DataLoader, TensorDataset
+        from catalyst import dl
+        from functools import partial
+
+        # sample data
+        num_samples, num_features, num_classes = int(1e4), int(1e1), 4
+        X = torch.rand(num_samples, num_features)
+        y = (torch.rand(num_samples, num_classes) > 0.5).to(torch.float32)
+
+        # pytorch loaders
+        dataset = TensorDataset(X, y)
+        loader = DataLoader(dataset, batch_size=32, num_workers=1)
+        loaders = {"train": loader, "valid": loader}
+
+        # model, criterion, optimizer, scheduler
+        model = torch.nn.Linear(num_features, num_classes)
+        criterion = torch.nn.BCEWithLogitsLoss()
+        optimizer = torch.optim.Adam(model.parameters())
+        scheduler = torch.optim.lr_scheduler.MultiStepLR(optimizer, [2])
+
+        # model training
+        runner = dl.SupervisedRunner(
+            input_key="features", output_key="logits", target_key="targets", loss_key="loss"
+        )
+        runner.train(
+            model=model,
+            criterion=criterion,
+            optimizer=optimizer,
+            scheduler=scheduler,
+            loaders=loaders,
+            num_epochs=3,
+            verbose=True,
+            callbacks=[
+                dl.BatchTransformCallback(
+                    input_key="targets",
+                    output_key="labels",
+                    transform=partial(torch.argmax, dim=1),
+                    scope="on_batch_end",
+                ),
+                dl.BatchTransformCallback(
+                    input_key="logits",
+                    output_key="scores",
+                    transform=partial(torch.softmax, dim=1),
+                    scope="on_batch_end",
+                ),
+                dl.BatchTransformCallback(
+                    input_key="scores",
+                    output_key="preds",
+                    transform=partial(torch.argmax, dim=1),
+                    scope="on_batch_end",
+                ),
+                dl.MultilabelAccuracyCallback(
+                    input_key="logits", target_key="targets", threshold=0.5
+                ),
+                dl.SklearnBatchCallback(
+                    keys={"y_pred": "preds", "y_true": "labels"},
+                    metric_fn="f1_score",
+                    metric_key="sk_f1",
+                    average="macro",
+                    zero_division=1,
+                )
+            ]
+        )
+
+    .. note::
+        Please follow the `minimal examples`_ sections for more use cases.
+
+        .. _`minimal examples`: https://github.com/catalyst-team/catalyst#minimal-examples
+    """
+
+    def __init__(
+        self,
+        keys: Mapping[str, Any],
+        metric_fn: Union[Callable, str],
+        metric_key: str,
+        log_on_batch: bool = True,
+        **metric_kwargs
+    ):
+        """Init."""
+        if isinstance(metric_fn, str):
+            metric_fn = sklearn.metrics.__dict__[metric_fn]
+        metric_fn = partial(metric_fn, **metric_kwargs)
+
+        super().__init__(
+            metric=FunctionalBatchMetric(metric_fn=metric_fn, metric_key=metric_key),
+            input_key=keys,
+            target_key=keys,
+            log_on_batch=log_on_batch,
+        )
+
+    def _get_key_value_inputs(self, runner: "IRunner") -> Dict[str, torch.Tensor]:
+        """Get data from batch in a format suitable for the Sklearn metrics calculation
+
+        Args:
+            runner: current runner
+
+        Returns:
+            dict of inputs and targets tensors
+        """
+        kv_inputs = {}
+        for key, value in self._keys.items():
+            kv_inputs[key] = runner.batch[value].cpu().detach()
+        kv_inputs["batch_size"] = runner.batch_size
+        return kv_inputs
+
+
+class SklearnLoaderCallback(LoaderMetricCallback):
+    """SklearnLoaderCallback implements an integration of **loader-based** Sklearn metrics
+
+    Args:
+        keys: a mapping between ``metric_fn`` arguments and keys in ``runner.batch``
+        metric_fn: metric function that gets outputs, targets, and other arguments given
+            in ``keys`` and returns score
+        metric_key: key to store computed metric in ``runner.batch_metrics`` dictionary
+        metric_kwargs: additional parameters for ``metric_fn``
+
+    .. note::
+        catalyst[ml] required for this callback
+
+    Examples:
+
+    .. code-block:: python
+
+        import sklearn
+        import torch
+        from torch.utils.data import DataLoader, TensorDataset
+        from catalyst import dl
+        from functools import partial
+
+        # sample data
+        num_samples, num_features, num_classes = int(1e4), int(1e1), 4
+        X = torch.rand(num_samples, num_features)
+        y = (torch.rand(num_samples, num_classes) > 0.5).to(torch.float32)
+
+        # pytorch loaders
+        dataset = TensorDataset(X, y)
+        loader = DataLoader(dataset, batch_size=32, num_workers=1)
+        loaders = {"train": loader, "valid": loader}
+
+        # model, criterion, optimizer, scheduler
+        model = torch.nn.Linear(num_features, num_classes)
+        criterion = torch.nn.BCEWithLogitsLoss()
+        optimizer = torch.optim.Adam(model.parameters())
+        scheduler = torch.optim.lr_scheduler.MultiStepLR(optimizer, [2])
+
+        # model training
+        runner = dl.SupervisedRunner(
+            input_key="features", output_key="logits", target_key="targets", loss_key="loss"
+        )
+        runner.train(
+            model=model,
+            criterion=criterion,
+            optimizer=optimizer,
+            scheduler=scheduler,
+            loaders=loaders,
+            num_epochs=3,
+            verbose=True,
+            callbacks=[
+                dl.BatchTransformCallback(
+                    input_key="targets",
+                    output_key="labels",
+                    transform=partial(torch.argmax, dim=1),
+                    scope="on_batch_end",
+                ),
+                dl.BatchTransformCallback(
+                    input_key="logits",
+                    output_key="scores",
+                    transform=partial(torch.softmax, dim=1),
+                    scope="on_batch_end",
+                ),
+                dl.BatchTransformCallback(
+                    input_key="scores",
+                    output_key="preds",
+                    transform=partial(torch.argmax, dim=1),
+                    scope="on_batch_end",
+                ),
+                dl.MultilabelAccuracyCallback(
+                    input_key="logits", target_key="targets", threshold=0.5
+                ),
+                dl.SklearnLoaderCallback(
+                    keys={"y_score": "scores", "y_true": "labels"},
+                    metric_fn="roc_auc_score",
+                    metric_key="roc_auc_score",
+                    average="macro",
+                    multi_class="ovo"
+                )
+            ]
+        )
+
+    .. note::
+        Please follow the `minimal examples`_ sections for more use cases.
+
+        .. _`minimal examples`: https://github.com/catalyst-team/catalyst#minimal-examples
+    """
+
+    def __init__(
+        self,
+        keys: Mapping[str, Any],
+        metric_fn: Union[Callable, str],
+        metric_key: str,
+        **metric_kwargs
+    ):
+        """Init."""
+        if isinstance(metric_fn, str):
+            metric_fn = sklearn.metrics.__dict__[metric_fn]
+        metric_fn = partial(metric_fn, **metric_kwargs)
+
+        super().__init__(
+            metric=FunctionalLoaderMetric(
+                metric_fn=metric_fn, metric_key=metric_key, accumulative_fields=keys
+            ),
+            input_key=keys,
+            target_key=keys,
+        )
+
+    def _get_key_value_inputs(self, runner: "IRunner") -> Dict[str, torch.Tensor]:
+        """Get data from batch in a format suitable for the Sklearn metrics calculation
+
+        Args:
+            runner: current runner
+
+        Returns:
+            dict of inputs and targets tensors
+        """
+        kv_inputs = {}
+        for key, value in self._keys.items():
+            kv_inputs[key] = runner.batch[value].cpu().detach()
+        return kv_inputs
+
+
+__all__ = ["SklearnBatchCallback", "SklearnLoaderCallback"]
diff --git a/catalyst/metrics/__init__.py b/catalyst/metrics/__init__.py
index efe58c5eec..8d68bf6591 100644
--- a/catalyst/metrics/__init__.py
+++ b/catalyst/metrics/__init__.py
@@ -14,7 +14,7 @@
 )
 from catalyst.metrics._additive import AdditiveValueMetric
 from catalyst.metrics._confusion_matrix import ConfusionMatrixMetric
-from catalyst.metrics._functional_metric import FunctionalBatchMetric
+from catalyst.metrics._functional_metric import FunctionalBatchMetric, FunctionalLoaderMetric
 
 from catalyst.metrics._accuracy import AccuracyMetric, MultilabelAccuracyMetric
 from catalyst.metrics._auc import AUCMetric
diff --git a/catalyst/metrics/_functional_metric.py b/catalyst/metrics/_functional_metric.py
index 9ac7915758..9c3d2c8eb5 100644
--- a/catalyst/metrics/_functional_metric.py
+++ b/catalyst/metrics/_functional_metric.py
@@ -1,13 +1,13 @@
-from typing import Callable, Dict
+from typing import Callable, Dict, Iterable
 
 import torch
 
-from catalyst.metrics import ICallbackBatchMetric
+from catalyst.metrics import AccumulationMetric, ICallbackBatchMetric, ICallbackLoaderMetric
 from catalyst.metrics._additive import AdditiveValueMetric
 
 
 class FunctionalBatchMetric(ICallbackBatchMetric):
-    """Class for custom metrics in a functional way.
+    """Class for custom **batch-based** metrics in a functional way.
 
     Args:
         metric_fn: metric function, that get outputs, targets and return score as torch.Tensor
@@ -120,4 +120,106 @@ def compute_key_value(self) -> Dict[str, torch.Tensor]:
         }
 
 
-__all__ = ["FunctionalBatchMetric"]
+class FunctionalLoaderMetric(ICallbackLoaderMetric):
+    """Class for custom **loader-based** metrics in a functional way.
+
+    Args:
+        metric_fn: metric function, that get outputs, targets and return score as torch.Tensor
+        metric_key: metric name
+        accumulative_fields: list of keys to accumulate data from batch
+        compute_on_call: if True, allows compute metric's value on call
+        prefix: metric prefix
+        suffix: metric suffix
+
+    .. note::
+
+        Metrics are calculated over all samples.
+
+    Examples:
+
+    .. code-block:: python
+
+        from functools import partial
+        import torch
+        from catalyst import metrics
+        import sklearn.metrics
+
+        targets = torch.tensor([3, 0, 2, 2, 1])
+        outputs = torch.rand((len(targets), targets.max()+1)).softmax(1)
+
+        metric = metrics.FunctionalLoaderMetric(
+            metric_fn=partial(
+                sklearn.metrics.roc_auc_score, average="macro", multi_class="ovr"
+            ),
+            metric_key="sk_auc",
+            accumulative_fields=['y_score','y_true'],
+
+        )
+        metric.reset(len(outputs), len(outputs))
+
+        metric.update(y_score=outputs, y_true=targets)
+        metric.compute()
+        # ...
+
+        metric.compute_key_value()
+        # {'sk_auc': ...}
+
+    """
+
+    def __init__(
+        self,
+        metric_fn: Callable,
+        metric_key: str,
+        accumulative_fields: Iterable[str] = None,
+        compute_on_call: bool = True,
+        prefix: str = None,
+        suffix: str = None,
+    ):
+        """Init"""
+        super().__init__(compute_on_call=compute_on_call, prefix=prefix, suffix=suffix)
+        self.metric_fn = metric_fn
+        self.metric_name = f"{self.prefix}{metric_key}{self.suffix}"
+        self.accumulative_metric = AccumulationMetric(
+            accumulative_fields=accumulative_fields, compute_on_call=compute_on_call
+        )
+
+    def reset(self, num_batches: int, num_samples: int) -> None:
+        """
+        Reset metrics fields
+
+        Args:
+            num_batches: expected number of batches
+            num_samples: expected number of samples to accumulate
+        """
+        self.accumulative_metric.reset(num_batches, num_samples)
+
+    def update(self, **kwargs) -> None:
+        """
+        Update storage
+
+        Args:
+            **kwargs: ``self.metric_fn`` inputs to store
+        """
+        self.accumulative_metric.update(**kwargs)
+
+    def compute(self) -> torch.Tensor:
+        """
+        Get metric for the whole loader
+
+        Returns:
+            custom metric
+        """
+        stored_values = self.accumulative_metric.compute()
+        return self.metric_fn(**stored_values)
+
+    def compute_key_value(self) -> Dict[str, torch.Tensor]:
+        """
+        Get metric for the whole loader
+
+        Returns:
+            Dict with one element-custom metric
+        """
+        return {self.metric_name: self.compute()}
+
+
+__all__ = ["FunctionalBatchMetric", "FunctionalLoaderMetric"]
diff --git a/docs/api/callbacks.rst b/docs/api/callbacks.rst
index aa0eac7109..2a650707cf 100644
--- a/docs/api/callbacks.rst
+++ b/docs/api/callbacks.rst
@@ -281,3 +281,18 @@ Segmentation – TrevskyCallback
     :members:
     :exclude-members: __init__, on_experiment_start, on_stage_start, on_epoch_start, on_loader_start, on_batch_start, on_batch_end, on_loader_end, on_epoch_end, on_stage_end, on_experiment_end
     :show-inheritance:
+
+SklearnBatchCallback
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+.. autoclass:: catalyst.callbacks.metrics.scikit_learn.SklearnBatchCallback
+    :members:
+    :exclude-members: __init__, on_experiment_start, on_stage_start, on_epoch_start, on_loader_start, on_batch_start, on_batch_end, on_loader_end, on_epoch_end, on_stage_end, on_experiment_end
+    :show-inheritance:
+
+SklearnLoaderCallback
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+.. autoclass:: catalyst.callbacks.metrics.scikit_learn.SklearnLoaderCallback
+    :members:
+    :exclude-members: __init__, on_experiment_start, on_stage_start, on_epoch_start, on_loader_start, on_batch_start, on_batch_end, on_loader_end, on_epoch_end, on_stage_end, on_experiment_end
+    :show-inheritance:
+
diff --git a/docs/api/metrics.rst b/docs/api/metrics.rst
index 9bb77d976b..26231fd9e2 100644
--- a/docs/api/metrics.rst
+++ b/docs/api/metrics.rst
@@ -68,6 +68,13 @@ FunctionalBatchMetric
     :undoc-members:
     :show-inheritance:
 
+FunctionalLoaderMetric
+~~~~~~~~~~~~~~~~~~~~~~~~~
+.. autoclass:: catalyst.metrics._functional_metric.FunctionalLoaderMetric
+    :exclude-members: __init__
+    :undoc-members:
+    :show-inheritance:
+
 
 Runner Metrics
 ----------------------


Test patch:
diff --git a/tests/pipelines/test_finetune2.py b/tests/pipelines/test_finetune2.py
index 88b79d672d..849b624885 100644
--- a/tests/pipelines/test_finetune2.py
+++ b/tests/pipelines/test_finetune2.py
@@ -1,9 +1,11 @@
 # flake8: noqa
 
+from functools import partial
 import os
 from tempfile import TemporaryDirectory
 
 from pytest import mark
+import torch
 from torch import nn, optim
 from torch.utils.data import DataLoader
 
@@ -92,6 +94,18 @@ def get_scheduler(self, stage: str, optimizer):
 
     def get_callbacks(self, stage: str):
         callbacks = {
+            "scores": dl.BatchTransformCallback(
+                input_key="logits",
+                output_key="scores",
+                transform=partial(torch.softmax, dim=1),
+                scope="on_batch_end",
+            ),
+            "labels": dl.BatchTransformCallback(
+                input_key="scores",
+                output_key="labels",
+                transform=partial(torch.argmax, dim=1),
+                scope="on_batch_end",
+            ),
             "criterion": dl.CriterionCallback(
                 metric_key="loss", input_key="logits", target_key="targets"
             ),
@@ -115,6 +129,13 @@ def get_callbacks(self, stage: str):
             callbacks["confusion_matrix"] = dl.ConfusionMatrixCallback(
                 input_key="logits", target_key="targets", num_classes=10
             )
+            callbacks["f1_score"] = dl.SklearnBatchCallback(
+                keys={"y_pred": "labels", "y_true": "targets"},
+                metric_fn="f1_score",
+                metric_key="sk_f1",
+                average="macro",
+                zero_division=1,
+            )
         return callbacks
 
     def handle_batch(self, batch):