Repository: mlflow/mlflow

Issue:
[FR] Improve binary classification metric support in `eval_and_log_metrics`
Thank you for submitting a feature request. **Before proceeding, please review MLflow's [Issue Policy for feature requests](https://www.github.com/mlflow/mlflow/blob/master/ISSUE_POLICY.md#feature-requests) and the [MLflow Contributing Guide](https://github.com/mlflow/mlflow/blob/master/CONTRIBUTING.rst)**.

**Please fill in this feature request template to ensure a timely and thorough response.**

## Willingness to contribute
The MLflow Community encourages new feature contributions. Would you or another member of your organization be willing to contribute an implementation of this feature (either as an MLflow Plugin or an enhancement to the MLflow code base)?

- [ ] Yes. I can contribute this feature independently.
- [ ] Yes. I would be willing to contribute this feature with guidance from the MLflow community.
- [ ] No. I cannot contribute this feature at this time.

## Proposal Summary

(In a few sentences, provide a clear, high-level description of the feature request)

## Motivation
- What is the use case for this feature?
- Why is this use case valuable to support for MLflow users in general?
- Why is this use case valuable to support for your project(s) or organization?
- Why is it currently difficult to achieve this use case? (please be as specific as possible about why related MLflow features and components are insufficient)

### What component(s), interfaces, languages, and integrations does this feature affect?
Components 
- [ ] `area/artifacts`: Artifact stores and artifact logging
- [ ] `area/build`: Build and test infrastructure for MLflow
- [ ] `area/docs`: MLflow documentation pages
- [ ] `area/examples`: Example code
- [ ] `area/model-registry`: Model Registry service, APIs, and the fluent client calls for Model Registry
- [ ] `area/models`: MLmodel format, model serialization/deserialization, flavors
- [ ] `area/projects`: MLproject format, project running backends
- [ ] `area/scoring`: MLflow Model server, model deployment tools, Spark UDFs
- [ ] `area/server-infra`: MLflow Tracking server backend
- [x] `area/tracking`: Tracking Service, tracking client APIs, autologging

Interfaces
- [ ] `area/uiux`: Front-end, user experience, plotting, JavaScript, JavaScript dev server
- [ ] `area/docker`: Docker use across MLflow's components, such as MLflow Projects and MLflow Models
- [ ] `area/sqlalchemy`: Use of SQLAlchemy in the Tracking Service or Model Registry
- [ ] `area/windows`: Windows support

Languages 
- [ ] `language/r`: R APIs and clients
- [ ] `language/java`: Java APIs and clients
- [ ] `language/new`: Proposals for new client languages

Integrations
- [ ] `integrations/azure`: Azure and Azure ML integrations
- [ ] `integrations/sagemaker`: SageMaker integrations
- [ ] `integrations/databricks`: Databricks integrations

## Details

(Use this section to include any additional information about the feature. If you have a proposal for how to implement this feature, please include it here. For implementation guidelines, please refer to the [Contributing Guide](https://github.com/mlflow/mlflow/blob/master/CONTRIBUTING.rst#contribution-guidelines).)

Currently in `mlflow.sklearn.eval_and_log_metrics` function [code](https://github.com/mlflow/mlflow/blob/41deb9ee5705d2727dc3df7fe60ce2c09231c8cc/mlflow/sklearn/__init__.py#L1722), when the model is a classifier, the function computes precision/recall/f1 metrics as multi-class classification with the `average="weighted"` label: [code](https://github.com/mlflow/mlflow/blob/41deb9ee5705d2727dc3df7fe60ce2c09231c8cc/mlflow/sklearn/utils.py#L192-L213). This could produce drastically different metric when the classifier is doing binary classification, in which case we should supply a `pos_label` parameter (sklearn [doc](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.recall_score.html). We should expose the interface of `eval_and_log_users` function for user to optionally fill in the `pos_label` if they are doing binary classification.



Discussion:
Can pick this up if no one else is looking

Resolving patch:
diff --git a/mlflow/sklearn/__init__.py b/mlflow/sklearn/__init__.py
--- a/mlflow/sklearn/__init__.py
+++ b/mlflow/sklearn/__init__.py
@@ -1719,7 +1719,7 @@ def patched_fn_with_autolog_disabled(original, *args, **kwargs):
         )
 
 
-def eval_and_log_metrics(model, X, y_true, *, prefix, sample_weight=None):
+def eval_and_log_metrics(model, X, y_true, *, prefix, sample_weight=None, pos_label=None):
     """
     Computes and logs metrics (and artifacts) for the given model and labeled dataset.
     The metrics/artifacts mirror what is auto-logged when training a model
@@ -1730,6 +1730,13 @@ def eval_and_log_metrics(model, X, y_true, *, prefix, sample_weight=None):
     :param y_true: The labels for the evaluation dataset.
     :param prefix: Prefix used to name metrics and artifacts.
     :param sample_weight: Per-sample weights to apply in the computation of metrics/artifacts.
+    :param pos_label: The positive label used to compute binary classification metrics such as
+        precision, recall, f1, etc. This parameter is only used for binary classification model
+        - if used on multi-label model, the evaluation will fail;
+        - if used for regression model, the parameter will be ignored.
+        For multi-label classification, keep `pos_label` unset (or set to `None`), and the
+        function will calculate metrics for each label and find their average weighted by support
+        (number of true instances for each label).
     :return: The dict of logged metrics. Artifacts can be retrieved by inspecting the run.
 
     ** Example **
@@ -1769,11 +1776,11 @@ def eval_and_log_metrics(model, X, y_true, *, prefix, sample_weight=None):
     metrics_manager = _AUTOLOGGING_METRICS_MANAGER
     with metrics_manager.disable_log_post_training_metrics():
         return _eval_and_log_metrics_impl(
-            model, X, y_true, prefix=prefix, sample_weight=sample_weight
+            model, X, y_true, prefix=prefix, sample_weight=sample_weight, pos_label=pos_label
         )
 
 
-def _eval_and_log_metrics_impl(model, X, y_true, *, prefix, sample_weight=None):
+def _eval_and_log_metrics_impl(model, X, y_true, *, prefix, sample_weight, pos_label):
     from mlflow.sklearn.utils import _log_estimator_content
     from sklearn.base import BaseEstimator
 
@@ -1804,6 +1811,7 @@ def _eval_and_log_metrics_impl(model, X, y_true, *, prefix, sample_weight=None):
             X=X,
             y_true=y_true,
             sample_weight=sample_weight,
+            pos_label=pos_label,
         )
 
     return metrics
diff --git a/mlflow/sklearn/utils.py b/mlflow/sklearn/utils.py
--- a/mlflow/sklearn/utils.py
+++ b/mlflow/sklearn/utils.py
@@ -147,7 +147,7 @@ def _get_metrics_value_dict(metrics_list):
     return metric_value_dict
 
 
-def _get_classifier_metrics(fitted_estimator, prefix, X, y_true, sample_weight):
+def _get_classifier_metrics(fitted_estimator, prefix, X, y_true, sample_weight, pos_label):
     """
     Compute and record various common metrics for classifiers
 
@@ -157,8 +157,9 @@ def _get_classifier_metrics(fitted_estimator, prefix, X, y_true, sample_weight):
     https://scikit-learn.org/stable/modules/generated/sklearn.metrics.recall_score.html
     (3) f1_score:
     https://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html
-    By default, we choose the parameter `labels` to be `None`, `pos_label` to be `1`,
-    `average` to be `weighted` to compute the weighted precision score.
+    By default, when `pos_label` is not specified (passed in as `None`), we set `average`
+    to `weighted` to compute the weighted score of theese metrics.
+    When the `pos_label` is specified (not `None`), we set `average` to `binary`.
 
     For (4) accuracy score:
     https://scikit-learn.org/stable/modules/generated/sklearn.metrics.accuracy_score.html
@@ -187,6 +188,7 @@ def _get_classifier_metrics(fitted_estimator, prefix, X, y_true, sample_weight):
     """
     import sklearn
 
+    average = "weighted" if pos_label is None else "binary"
     y_pred = fitted_estimator.predict(X)
 
     classifier_metrics = [
@@ -194,21 +196,33 @@ def _get_classifier_metrics(fitted_estimator, prefix, X, y_true, sample_weight):
             name=prefix + "precision_score",
             function=sklearn.metrics.precision_score,
             arguments=dict(
-                y_true=y_true, y_pred=y_pred, average="weighted", sample_weight=sample_weight
+                y_true=y_true,
+                y_pred=y_pred,
+                pos_label=pos_label,
+                average=average,
+                sample_weight=sample_weight,
             ),
         ),
         _SklearnMetric(
             name=prefix + "recall_score",
             function=sklearn.metrics.recall_score,
             arguments=dict(
-                y_true=y_true, y_pred=y_pred, average="weighted", sample_weight=sample_weight
+                y_true=y_true,
+                y_pred=y_pred,
+                pos_label=pos_label,
+                average=average,
+                sample_weight=sample_weight,
             ),
         ),
         _SklearnMetric(
             name=prefix + "f1_score",
             function=sklearn.metrics.f1_score,
             arguments=dict(
-                y_true=y_true, y_pred=y_pred, average="weighted", sample_weight=sample_weight
+                y_true=y_true,
+                y_pred=y_pred,
+                pos_label=pos_label,
+                average=average,
+                sample_weight=sample_weight,
             ),
         ),
         _SklearnMetric(
@@ -468,7 +482,7 @@ def _log_warning_for_artifacts(func_name, func_call, err):
 
 
 def _log_specialized_estimator_content(
-    autologging_client, fitted_estimator, run_id, prefix, X, y_true=None, sample_weight=None
+    autologging_client, fitted_estimator, run_id, prefix, X, y_true, sample_weight, pos_label
 ):
     import sklearn
 
@@ -478,7 +492,7 @@ def _log_specialized_estimator_content(
         try:
             if sklearn.base.is_classifier(fitted_estimator):
                 metrics = _get_classifier_metrics(
-                    fitted_estimator, prefix, X, y_true, sample_weight
+                    fitted_estimator, prefix, X, y_true, sample_weight, pos_label
                 )
             elif sklearn.base.is_regressor(fitted_estimator):
                 metrics = _get_regressor_metrics(fitted_estimator, prefix, X, y_true, sample_weight)
@@ -528,7 +542,14 @@ def _log_specialized_estimator_content(
 
 
 def _log_estimator_content(
-    autologging_client, estimator, run_id, prefix, X, y_true=None, sample_weight=None
+    autologging_client,
+    estimator,
+    run_id,
+    prefix,
+    X,
+    y_true=None,
+    sample_weight=None,
+    pos_label=None,
 ):
     """
     Logs content for the given estimator, which includes metrics and artifacts that might be
@@ -544,6 +565,10 @@ def _log_estimator_content(
     :param X: The data samples.
     :param y_true: Labels.
     :param sample_weight: Per-sample weights used in the computation of metrics and artifacts.
+    :param pos_label: The positive label used to compute binary classification metrics such as
+        precision, recall, f1, etc. This parameter is only used for classification metrics.
+        If set to `None`, the function will calculate metrics for each label and find their
+        average weighted by support (number of true instances for each label).
     :return: A dict of the computed metrics.
     """
     metrics = _log_specialized_estimator_content(
@@ -554,6 +579,7 @@ def _log_estimator_content(
         X=X,
         y_true=y_true,
         sample_weight=sample_weight,
+        pos_label=pos_label,
     )
 
     if hasattr(estimator, "score") and y_true is not None:


Test patch:
diff --git a/tests/sklearn/test_sklearn_autolog.py b/tests/sklearn/test_sklearn_autolog.py
--- a/tests/sklearn/test_sklearn_autolog.py
+++ b/tests/sklearn/test_sklearn_autolog.py
@@ -1194,6 +1194,63 @@ def test_eval_and_log_metrics_for_binary_classifier():
     assert sorted(artifacts) == sorted(eval_artifacts)
 
 
+def test_eval_and_log_metrics_for_binary_classifier_with_pos_label():
+    # disable autologging so that we can check for the sole existence of eval-time metrics
+    mlflow.sklearn.autolog(disable=True)
+
+    import sklearn.ensemble
+
+    model = sklearn.ensemble.RandomForestClassifier(max_depth=2, random_state=0, n_estimators=10)
+    X, y = sklearn.datasets.load_breast_cancer(return_X_y=True)
+    X_eval = X[:-1, :]
+    y_eval = y[:-1]
+
+    with mlflow.start_run() as run:
+        model = fit_model(model, X, y, "fit")
+        eval_metrics = mlflow.sklearn.eval_and_log_metrics(
+            model=model, X=X_eval, y_true=y_eval, prefix="val_", pos_label=1
+        )
+
+    y_pred = model.predict(X_eval)
+    y_pred_prob = model.predict_proba(X_eval)
+    # For binary classification, y_score only accepts the probability of greater label
+    y_pred_prob_roc = y_pred_prob[:, 1]
+
+    expected_metrics = {
+        "val_score": model.score(X_eval, y_eval),
+        "val_accuracy_score": sklearn.metrics.accuracy_score(y_eval, y_pred),
+        "val_precision_score": sklearn.metrics.precision_score(y_eval, y_pred, pos_label=1),
+        "val_recall_score": sklearn.metrics.recall_score(y_eval, y_pred, pos_label=1),
+        "val_f1_score": sklearn.metrics.f1_score(y_eval, y_pred, pos_label=1),
+        "val_log_loss": sklearn.metrics.log_loss(y_eval, y_pred_prob),
+    }
+    if _is_metric_supported("roc_auc_score"):
+        expected_metrics["val_roc_auc_score"] = sklearn.metrics.roc_auc_score(
+            y_eval,
+            y_score=y_pred_prob_roc,
+            average="weighted",
+            multi_class="ovo",
+        )
+    assert eval_metrics == pytest.approx(expected_metrics)
+
+    eval_artifacts = []
+    if _is_plotting_supported():
+        eval_artifacts.extend(
+            [
+                "{}.png".format("val_confusion_matrix"),
+                "{}.png".format("val_roc_curve"),
+                "{}.png".format("val_precision_recall_curve"),
+            ]
+        )
+
+    # Check that logged artifacts/metrics are the same as the ones returned by the method
+    run_id = run.info.run_id
+    _, metrics, _, artifacts = get_run_data(run_id)
+
+    assert metrics == eval_metrics
+    assert sorted(artifacts) == sorted(eval_artifacts)
+
+
 def test_eval_and_log_metrics_matches_training_metrics():
     mlflow.sklearn.autolog()