Repository: scikit-learn/scikit-learn
Pull request: FEA add TunedThresholdClassifier meta-estimator to post-tune the cut-off threshold
superseded #16525
closes #16525 
closes https://github.com/scikit-learn/scikit-learn/issues/8614
closes https://github.com/scikit-learn/scikit-learn/pull/10117
supersedes https://github.com/scikit-learn/scikit-learn/pull/10117

build upon https://github.com/scikit-learn/scikit-learn/pull/26037

relates to #4813

### Summary

We introduce a `TunedThresholdClassifier` that intends to post-tune the cut-off points to convert a soft decision of the `decision_function` or `predict_proba` to a hard decision provided by `predict`.

### Important features to have in mind:

**`objective_metric`**: the objective metric is set to either a metric to be maximized or a pair of metrics, one to be optimized under the constraint of the other (to find a trade-off). Additionally, we can pass a cost/gain-matrix that could be used to optimize a business metric. For this case, we are limited to constant cost/gain. In the future, we can think of cost/gain that depends on the matrix `X` but we would need to be able to forward meta-data to the scorer (a good additional use case for SLEP006 @adrinjalali).

**`cv` and `refit`**: we provide some flexibility to pass refitted model and single train/test split. We add limitations and documentation to caveats with an example.

### Point to discuss

- Are we fine with the name `TunedThresholdClassifier`? Shall instead have something about "threshold" (e.g. `ThresholdTuner`)?
- We are using the term `objective_metric`, `constraint_value` and `objective_score`. Is the naming fine? An alternative to "objective" might be "utility"?

### Further work

I implemented currently a single example that shows the feature in the context of post-tuning of the decision threshold.

The current example is using a single train/test split for the figure and I think it would be nice to have some ROC/precision-recall curve obtained from cross-validation to be complete. However, we need some new features to be implemented.

I also am planning to analyse the usage of this feature on the problem of calibration on imbalanced classification problems. The feeling on this topic is that resampling strategies involved an implicit tuning of the decision threshold at the cost of a badly calibrated model. It might be better to learn a model on the imbalanced problem directly, making sure that it is well calibrated and then post-tune the decision threshold for "hard" prediction. In this case, you get the best of two worlds: a calibrated model if the output of `predict_proba` is important to you and an optimum hard predictor for your specific utility metric. However, this is going to need some investigation and will be better suited for another PR.

Problem statement:
Add wrapper class that changes threshold value for predict
This was discussed before, but not sure if there's an issue.
We should have a wrapper class that changes the decision threshold based on a cross-validation (or hold-out, pre-fit) estimate.
This is very very common so I think we should have a built-in solution.
Simple rules for selecting a new threshold are:
- picking the point on the roc curve that's closest to the ideal corner
- picking the point on the precision-recall curve that's closest to the ideal corner
- optimize one metric while holding another one constant: find the best threshold to yield the best recall with a precision of at least 10% for example. We could also make this slightly less flexible and just say "the median of the largest threshold over cross-validation that yields at least X precision".

Hints from issue discussion:
Belongs in sklearn.calibration?

On 20 March 2017 at 09:47, Andreas Mueller <notifications@github.com> wrote:

> This was discussed before, but not sure if there's an issue.
> We should have a wrapper class that changes the decision threshold based
> on a cross-validation (or hold-out, pre-fit) estimate.
> This is very very common so I think we should have a built-in solution.
> Simple rules for selecting a new threshold are:
>
>    - picking the point on the roc curve that's closest to the ideal corner
>    - picking the point on the precision-recall curve that's closest to
>    the ideal corner
>    - optimize one metric while holding another one constant: find the
>    best threshold to yield the best recall with a precision of at least 10%
>    for example. We could also make this slightly less flexible and just say
>    "the median of the largest threshold over cross-validation that yields at
>    least X precision".
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/scikit-learn/scikit-learn/issues/8614>, or mute the
> thread
> <https://github.com/notifications/unsubscribe-auth/AAEz61b8TPwZTL-6YRneiF08me7tE6cdks5rnbCagaJpZM4Mh6Hm>
> .
>

I wouldn't object to that.
There was also a recent paper by some colleagues from nyu how to properly control for different kinds of errors but I need to search for that...
Related to #6663 from @betatim ?
Yes, definitely related to #6663. Though I think I would implement it as a meta-estimator and not a transformer and I would add cross-validation to adjust the threshold using mentioned strategies.
Hey, it seems like no one is working on that; can I give it a try? 
you're welcome to give it a go, though we often encourage new contributors
to start with something smaller to become familiar with the contribution
process. If I were you, I'd start by writing the class docstring and then
tests.

@jnothman that sounds good. Indeed I wouldn't try to commit the whole feature at once. I also see it as a way to become more familiar with the library. I first need to get a good grip of the requested feature though. @amueller did you maybe find the paper you mentioned?
thnx in advance
to give an update. I went through the mentioned issues and got a better understanding of what the task is. Now I think I can start implementing the api of the class. But I could still use some help on what the exact threshold decision should be based on. Maybe as @amueller said provide all of three options and let the user decide which one to use? 
Start with one?

Commit message: Update sklearn/metrics/_scorer.py

Co-authored-by: Christian Lorentzen <lorentzen.ch@gmail.com>

Patch to review:
diff --git a/doc/model_selection.rst b/doc/model_selection.rst
index 25cd2b655ccc5..3a46315e65156 100644
--- a/doc/model_selection.rst
+++ b/doc/model_selection.rst
@@ -14,5 +14,6 @@ Model selection and evaluation
 
     modules/cross_validation
     modules/grid_search
+    modules/prediction
     modules/model_evaluation
     modules/learning_curve
diff --git a/doc/modules/classes.rst b/doc/modules/classes.rst
index dc13c292cb94d..e8e4a3671a542 100644
--- a/doc/modules/classes.rst
+++ b/doc/modules/classes.rst
@@ -1218,6 +1218,16 @@ Hyper-parameter optimizers
    model_selection.RandomizedSearchCV
    model_selection.HalvingRandomSearchCV
 
+Model post-fit tuning
+---------------------
+
+.. currentmodule:: sklearn
+
+.. autosummary::
+   :toctree: generated/
+   :template: class.rst
+
+   model_selection.CutOffClassifier
 
 Model validation
 ----------------
diff --git a/doc/modules/prediction.rst b/doc/modules/prediction.rst
new file mode 100644
index 0000000000000..d990c242aceee
--- /dev/null
+++ b/doc/modules/prediction.rst
@@ -0,0 +1,155 @@
+.. currentmodule:: sklearn.model_selection
+
+.. _cutoffclassifier:
+
+========================================================
+Tuning cut-off decision threshold for classes prediction
+========================================================
+
+Classifiers are predictive models: they use statistical learning to predict
+outcomes. The outcomes of a classifier takes two forms: a "soft" score for each
+sample in relation to each class, and a "hard" categorical prediction (i.e.
+class label). Soft predictions are obtained using :term:`predict_proba` or
+:term:`decision_function` while hard predictions are obtained using
+:term:`predict`.
+
+In scikit-learn, there is a connection between soft and hard prediction. In the
+case of a binary classification, hard predictions are obtained by associating
+the positive class with probability value greater than 0.5 (obtained with
+:term:`predict_proba`) or decision function value greater than 0 (obtained with
+:term:`decision_function`).
+
+    >>> from sklearn.datasets import make_classification
+    >>> from sklearn.tree import DecisionTreeClassifier
+    >>> X, y = make_classification(random_state=0)
+    >>> classifier = DecisionTreeClassifier(max_depth=2, random_state=0).fit(X, y)
+    >>> classifier.predict_proba(X[:4])
+    array([[0.94   , 0.06   ],
+           [0.94   , 0.06   ],
+           [0.04..., 0.95...],
+           [0.04..., 0.95...]])
+    >>> classifier.predict(X[:4])
+    array([0, 0, 1, 1])
+
+
+Similar rules apply for other classification problems.
+
+While these approaches are reasonable as default behaviors, they might not be
+adapted to some cases. The context and nature of the use case define the
+expected behavior of the classifier and thus the strategy to convert soft
+predictions into hard predictions. We illustrate this point with an example.
+
+Let's imagine the deployment of a predictive model helping medical doctors to
+detect cancers. In a setting where this model would be a tool to discard
+obvious cases, doctors might be interested to have a high recall (all cancers
+cases should be tagged as such) to not miss any patient with a cancer. However,
+it will be at the cost of having more false positive predictions (i.e. lower
+precision). Thus, in terms of decision threshold, it would be better to
+classify a patient having a cancer for a lower probability than 0.5.
+
+Post-tuning of the decision threshold
+=====================================
+
+One solution to address the problem stated in the introduction is to tune the decision
+threshold of the classifier once this model has been trained. The
+:class:`~sklearn.model_selection.CutOffClassifier` allows to tune this threshold using
+an internal cross-validation. The optimum threshold is tuned to maximize a given metric
+with or without constraints.
+
+The following image illustrate the tuning of the cut-off point for a gradient
+boosting classifier. While the vanilla and tuned classifiers provide the same
+Receiver Operating Characteristic (ROC) and Precision-Recall curves, and thus
+the same :term:`predict_proba` outputs, the "hard" predictions defer because of
+the tuned cut-off point. The vanilla classifier predicts the class of interest
+for a probability greater than 0.5 while the tuned classifier predicts the
+class of interest for a very low probability (around 0.02). This cut-off point
+is maximizes a utility metric defined by the business case (in this case an
+insurance company).
+
+.. figure:: ../auto_examples/model_selection/images/sphx_glr_plot_cutoff_tuning_002.png
+   :target: ../auto_examples/model_selection/plot_cutoff_tuning.html
+   :align: center
+
+Available options to tune the cut-off point
+-------------------------------------------
+
+The cut-off point can be tuned with different strategies controlled by the parameter
+`objective_metric`.
+
+A straightforward use case is to maximize a pre-defined scikit-learn metric. These
+metrics can be found by calling the function :func:`~sklearn.metrics.get_scorer_names`.
+We provide an example where we maximize the balanced accuracy.
+
+.. note::
+
+    It is important to notice that these metrics comes with default parameter, notably
+    the label of the class of interested (i.e. `pos_label`). Thus, if this label is not
+    the right one for your application, you need to define a scorer and pass the right
+    `pos_label` (and additional parameters) using the
+    :func:`~sklearn.metrics.make_scorer`. You should refer to :ref:`scoring` to get all
+    information to define your own scoring function. For instance, we show how to pass
+    the information to the scorer that the label of interest is `0` when maximizing the
+    :func:`~sklearn.metrics.f1_score`:
+
+        >>> from sklearn.linear_model import LogisticRegression
+        >>> from sklearn.model_selection import CutOffClassifier, train_test_split
+        >>> from sklearn.metrics import make_scorer, f1_score
+        >>> X, y = make_classification(
+        ...    n_samples=1_000, weights=[0.1, 0.9], random_state=0)
+        >>> X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)
+        >>> pos_label = 0
+        >>> scorer = make_scorer(f1_score, pos_label=pos_label)
+        >>> model = CutOffClassifier(LogisticRegression(), objective_metric=scorer).fit(
+        ...     X_train, y_train)
+        >>> scorer(model, X_test, y_test)
+        0.82...
+        >>> # compare it with the internal score found by cross-validation
+        >>> model.objective_score_
+        0.86...
+
+A second strategy aims at maximizing a metric while imposing constraints on another
+metric. Four pre-defined options exist, 2 that uses the Receiver Operating
+Characteristic (ROC) statistics and 2 that uses the Precision-Recall statistics.
+
+- `"max_tpr_at_tnr_constraint"`: maximizes the True Positive Rate (TPR) such that the
+  True Negative Rate (TNR) is the closest to a given value.
+- `"max_tnr_at_tpr_constraint"`: maximizes the TNR such that the TPR is the closest to
+  a given value.
+- `"max_precision_at_recall_constraint"`: maximizes the precision such that the recall
+  is the closest to a given value.
+- `"max_recall_at_precision_constraint"`: maximizes the recall such that the precision
+  is the closest to a given value.
+
+For these options, the `constraint_value` parameter needs to be defined. In addition,
+you can use the `pos_label` parameter to indicate the label of the class of interest.
+
+The final strategy maximizes a custom utility function. This problem is also known as
+cost-sensitive learning. The utility function is defined by providing dictionary
+containing the cost-gain associated with the entries of the confusion matrix. The keys
+are defined as `{"tn", "fp", "fn", "tp"}`. The class of interest is defined using the
+`pos_label` parameter. Refer to :ref:`cost_sensitive_learning_example` for an example
+depicting the use of such a utility function.
+
+Important notes regarding the internal cross-validation
+-------------------------------------------------------
+
+By default :class:`~sklearn.model_selection.CutOffClassifier` uses a 5-fold stratified
+cross-validation to tune the cut-off point. The parameter `cv` allows to control the
+cross-validation strategy. It is possible to go around cross-validation by passing
+`cv="prefit"` and provide an already fitted classifier. In this case, the cut-off point
+is tuned on the data provided to the `fit` method.
+
+However, you should be extremely careful when using this option. You should never use
+the same data for training the classifier and tuning the cut-off point at the risk of
+overfitting. Refer to :ref:`cutoffclassifier_no_cv` that shows such overfitting. If
+you are in a situation where you have limited resources, you should can consider using
+a float number that will use a single split internally.
+
+The option `cv="prefit"` should only be used when the provided classifier was already
+trained on some data and you want to tune (or re-tune) on a new validation set.
+
+Examples
+--------
+
+- See :ref:`sphx_glr_auto_examples_model_selection_plot_cutoff_tuning.py` example for
+  an example of tuning the decision threshold of a classifier.
diff --git a/doc/whats_new/v1.3.rst b/doc/whats_new/v1.3.rst
index 41d5d1fdeeabd..c783d1f70da16 100644
--- a/doc/whats_new/v1.3.rst
+++ b/doc/whats_new/v1.3.rst
@@ -413,6 +413,11 @@ Changelog
   `return_indices` to return the train-test indices of each cv split.
   :pr:`25659` by :user:`Guillaume Lemaitre <glemaitre>`.
 
+- |MajorFeature| :class:`model_selection.CutOffClassifier` calibrates decision threshold
+  function of a binary classifier by maximizing a classification metric through
+  cross-validation.
+  :pr:`26120` by :user:`Guillaume Lemaitre <glemaitre>`.
+
 :mod:`sklearn.naive_bayes`
 ..........................
 
diff --git a/examples/model_selection/plot_cutoff_tuning.py b/examples/model_selection/plot_cutoff_tuning.py
new file mode 100644
index 0000000000000..c5f15a18a270b
--- /dev/null
+++ b/examples/model_selection/plot_cutoff_tuning.py
@@ -0,0 +1,566 @@
+"""
+==================================================
+Post-tuning the cut-off point of decision function
+==================================================
+
+Once a classifier is trained, the output of the :term:`predict` method output hard
+predictions corresponding to a thresholding of either the :term:`decision function`
+or the :term:`predict_proba` output. For binary classifier, the default threshold is
+defined as a probability score of 0.5 or a decision function value of 0.0.
+
+However, it happens that this default strategy is not optimized for the task at hand.
+Here, we use the "Statlog" German credit dataset [1]_ to illustrate such an application.
+In this dataset, the task is to predict whether a person is "good" or "bad" credit.
+In addition, cost-matrix is provided where it is defined that classifying a "bad"
+credit as "good" is 5 times more costly than classifying a "good" credit as "bad".
+
+We use the :class:`~sklearn.model_selection.CutOffClassifier` to tune the
+cut-off point of the decision function to minimize the business cost provided to us.
+
+.. topic:: References
+
+    .. [1] "Statlog (German Credit Data) Data Set", UCI Machine Learning Repository,
+       `Link <https://archive.ics.uci.edu/ml/datasets/Statlog+%28German+Credit+Data%29>`_.  # noqa
+"""
+
+# %%
+# "Statlog" German credit dataset
+# -------------------------------
+#
+# We fetch the German credit dataset from OpenML.
+import sklearn
+from sklearn.datasets import fetch_openml
+
+sklearn.set_config(transform_output="pandas")
+german_credit = fetch_openml(data_id=31, as_frame=True, parser="pandas")
+X, y = german_credit.data, german_credit.target
+
+# %%
+# We check the feature types available in `X`.
+X.info()
+
+# %%
+X.head()
+
+# %%
+# Many features are categorical and usually string-encoded. We need to encode
+# these categories when we develop ou predictive model. Let's check the targets.
+y.value_counts()
+
+# %%
+# A first observation is that the dataset is imbalanced. We would need to be careful
+# when evaluating our predictive model and use a family of metrics that are adapted
+# to this setting.
+#
+# In addition, we observe that the target is string-encoded. Some metrics
+# (e.g. precision and recall) require to provide the label of interest also called
+# the "positive label". Here, we define that our goal is to predict whether or not
+# a sample is a "bad" credit.
+pos_label = "bad"
+
+# %%
+# To carry our analysis, we split our dataset using a single stratified split.
+from sklearn.model_selection import train_test_split
+
+X_train, X_test, y_train, y_test = train_test_split(X, y, stratify=y, random_state=0)
+
+# %%
+# We are ready to design our predictive model and the associated evaluation strategy.
+#
+# Evaluation metrics
+# ------------------
+#
+# In this section, we define a set of metrics that we use later. To see
+# the effect of tuning the cut-off point, we evaluate the predictive model using
+# the Receiver Operating Characteristic (ROC) curve and the Precision-Recall curve.
+# The values reported on these plots are therefore the true positive rate (TPR) and
+# the false positive rate (FPR) for the ROC curve and the precision and recall for the
+# Precision-Recall curve.
+#
+# From these four metrics, scikit-learn does not provide a scorer for the FPR. We
+# therefore need to define a small custom function to compute it.
+import numpy as np
+from sklearn.metrics import confusion_matrix, make_scorer, precision_score, recall_score
+
+
+def fpr_score(y, y_pred, **kwargs):
+    cm = confusion_matrix(y, y_pred)
+    classes = np.unique(y)
+    pos_label = kwargs.get("pos_label", classes[-1])
+    pos_label_idx = np.searchsorted(classes, pos_label)
+    if pos_label_idx == 0:
+        cm = cm[::-1, ::-1]
+    tn, fp, _, _ = cm.ravel()
+    tnr = tn / (tn + fp)
+    return 1 - tnr
+
+
+# %%
+# As previously stated, the "positive label" is not defined as the value "1" and calling
+# some of the metrics with this non-standard value raise an error. We need to
+# provide the indication of the "positive label" to the metrics.
+#
+# We therefore need to define a scikit-learn scorer using
+# :func:`~sklearn.metrics.make_scorer` where the information is passed. We store all
+# created scorer in a dictionary. To use them, we need to pass the fitted model and
+# the data and target on which we want to evaluate the predictive model.
+tpr_score = recall_score  # TPR and recall are the same metric
+scoring = {
+    "precision": make_scorer(precision_score, pos_label=pos_label),
+    "recall": make_scorer(recall_score, pos_label=pos_label),
+    "fpr": make_scorer(fpr_score, pos_label=pos_label),
+    "tpr": make_scorer(tpr_score, pos_label=pos_label),
+}
+
+# %%
+# In addition, the original research [1]_ defines a business metric. They provide a
+# cost-matrix where they define that predicting a "bad" credit as "good" is 5 times more
+# costly than the opposite. We define a dictionary containing this information and a
+# score function that computes the cost.
+cost_gain_matrix = {"tp": 0, "tn": 0, "fp": -1, "fn": -5}
+
+
+def gain_cost_score(y, y_pred, **kwargs):
+    cm = confusion_matrix(y, y_pred)
+    classes = np.unique(y)
+    pos_label = kwargs.get("pos_label", classes[-1])
+    pos_label_idx = np.searchsorted(classes, pos_label)
+    if pos_label_idx == 0:
+        cm = cm[::-1, ::-1]
+    costs_and_gain = np.array(
+        [
+            [kwargs["cost_gain_matrix"]["tn"], kwargs["cost_gain_matrix"]["fp"]],
+            [kwargs["cost_gain_matrix"]["fn"], kwargs["cost_gain_matrix"]["tp"]],
+        ]
+    )
+    return (costs_and_gain * cm).sum()
+
+
+scoring["cost_gain"] = make_scorer(
+    gain_cost_score, pos_label=pos_label, cost_gain_matrix=cost_gain_matrix
+)
+# %%
+# Vanilla predictive model
+# ------------------------
+#
+# Design of the predictive model
+# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+#
+# In this section, we design our predictive model. We use a gradient boosting classifier
+# using :class:`~sklearn.ensemble.HistGradientBoostingClassifier`. However, we need to
+# encode the categorical features with numerical values and we therefore use an
+# :class:`~sklearn.preprocessing.OrdinalEncoder` to do so. The numerical features are
+# kept as-is. To recover the name of the categorical columns, we use the helper function
+# :func:`~sklearn.compose.make_column_selector` and the fact that the categorical
+# features are stored as `category` dtype.
+from sklearn.compose import ColumnTransformer, make_column_selector as selector
+from sklearn.ensemble import HistGradientBoostingClassifier
+from sklearn.pipeline import Pipeline
+from sklearn.preprocessing import OrdinalEncoder
+
+categorical_columns = selector(dtype_include="category")(X)
+numerical_columns = selector(dtype_exclude="category")(X)
+
+preprocessor = ColumnTransformer(
+    [
+        (
+            "categorical",
+            OrdinalEncoder(handle_unknown="use_encoded_value", unknown_value=-1),
+            categorical_columns,
+        ),
+        ("numerical", "passthrough", numerical_columns),
+    ],
+    verbose_feature_names_out=False,
+)
+model = Pipeline(
+    [
+        ("preprocessor", preprocessor),
+        (
+            "classifier",
+            HistGradientBoostingClassifier(
+                categorical_features=categorical_columns, random_state=0
+            ),
+        ),
+    ]
+)
+
+model.fit(X_train, y_train)
+
+# %%
+# Evaluation of the predictive model
+# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+#
+# We evaluate the performance of our predictive model using the ROC and Precision-Recall
+# curves.
+import matplotlib.pyplot as plt
+from sklearn.metrics import PrecisionRecallDisplay, RocCurveDisplay
+
+fig, axs = plt.subplots(nrows=1, ncols=2, figsize=(14, 6))
+
+PrecisionRecallDisplay.from_estimator(
+    model, X_test, y_test, pos_label=pos_label, ax=axs[0], name="GBDT"
+)
+axs[0].plot(
+    scoring["recall"](model, X_test, y_test),
+    scoring["precision"](model, X_test, y_test),
+    marker="o",
+    markersize=10,
+    color="tab:blue",
+    label="Default cut-off point at a probability of 0.5",
+)
+axs[0].set_title("Precision-Recall curve")
+axs[0].legend()
+
+RocCurveDisplay.from_estimator(
+    model,
+    X_test,
+    y_test,
+    pos_label=pos_label,
+    ax=axs[1],
+    name="GBDT",
+    plot_chance_level=True,
+)
+axs[1].plot(
+    scoring["fpr"](model, X_test, y_test),
+    scoring["tpr"](model, X_test, y_test),
+    marker="o",
+    markersize=10,
+    color="tab:blue",
+    label="Default cut-off point at a probability of 0.5",
+)
+axs[1].set_title("ROC curve")
+axs[1].legend()
+_ = fig.suptitle("Evaluation of the vanilla GBDT model")
+
+# %%
+# We recall that these curves corresponds show the statistical performance of the
+# predictive model for different cut-off points. For the Precision-Recall curve, the
+# reported metrics are the precision and recall and for the ROC curve, the reported
+# metrics are the TPR (same as recall) and FPR.
+#
+# Here, the different cut-off points corresponds at different level of probability
+# scores ranging between 0 and 1. By default, `model.predict` uses a cut-off point at
+# a probability of 0.5. Thus, the metrics for this cut-off point is reported with the
+# blue dot on the curves: it corresponds to the statistical performance of the model
+# when using `model.predict`.
+#
+# However, we recall that the original aim was to minimize the cost (or maximize the
+# gain) by the business metric. We can compute the value of the business metric:
+scoring["cost_gain"](model, X_test, y_test)
+
+# %%
+# At this stage, we don't know if some other cut-off points could lead to a greater
+# gain. To be able to decide, if another cut-off point is better, we would need to
+# compute the cost-gain using the business metric for all possible cut-off points and
+# choose the optimal one. This strategy is quite tedious to implement and the
+# :class:`~sklearn.metrics.CutOffClassifier` class is here to help us. It automatically
+# compute the cost-gain for all possible cut-off points and choose the optimal one.
+#
+# .. _cost_sensitive_learning_example:
+#
+# Tuning the cut-off point
+# ------------------------
+#
+# We use :class:`~sklearn.metrics.CutOffClassifier` to tune the cut-off point. We need
+# to provide the business metric to optimize as well as the positive label. Internally,
+# the optimum cut-off point is chosen such that it maximized the business metric
+# via cross-validation. By default a 5-fold stratified cross-validation is used.
+from sklearn.model_selection import CutOffClassifier
+
+model_tuned = CutOffClassifier(
+    estimator=model,
+    pos_label=pos_label,
+    objective_metric=cost_gain_matrix,
+)
+model_tuned.fit(X_train, y_train)
+
+# %%
+# We plot the ROC and Precision-Recall curves for the vanilla model and the tuned model.
+# Also we plot the cut-off points that would be used by each model.
+fig, axs = plt.subplots(nrows=1, ncols=3, figsize=(21, 6))
+
+linestyles = ("dashed", "dotted")
+markerstyles = ("o", ">")
+colors = ("tab:blue", "tab:orange")
+names = ("Vanilla GBDT", "Tuned GBDT")
+for idx, (est, linestyle, marker, color, name) in enumerate(
+    zip((model, model_tuned), linestyles, markerstyles, colors, names)
+):
+    decision_threshold = getattr(est, "decision_threshold_", 0.5)
+    PrecisionRecallDisplay.from_estimator(
+        est,
+        X_test,
+        y_test,
+        pos_label=pos_label,
+        linestyle=linestyle,
+        color=color,
+        ax=axs[0],
+        name=name,
+    )
+    axs[0].plot(
+        scoring["recall"](est, X_test, y_test),
+        scoring["precision"](est, X_test, y_test),
+        marker,
+        markersize=10,
+        color=color,
+        label=f"Cut-off point at probability of {decision_threshold:.2f}",
+    )
+    RocCurveDisplay.from_estimator(
+        est,
+        X_test,
+        y_test,
+        pos_label=pos_label,
+        linestyle=linestyle,
+        color=color,
+        ax=axs[1],
+        name=name,
+        plot_chance_level=idx == 1,
+    )
+    axs[1].plot(
+        scoring["fpr"](est, X_test, y_test),
+        scoring["tpr"](est, X_test, y_test),
+        marker,
+        markersize=10,
+        color=color,
+        label=f"Cut-off point at probability of {decision_threshold:.2f}",
+    )
+
+axs[0].set_title("Precision-Recall curve")
+axs[0].legend()
+axs[1].set_title("ROC curve")
+axs[1].legend()
+
+axs[2].plot(
+    model_tuned.decision_thresholds_, model_tuned.objective_scores_, color="tab:orange"
+)
+axs[2].plot(
+    model_tuned.decision_threshold_,
+    model_tuned.objective_score_,
+    "o",
+    markersize=10,
+    color="tab:orange",
+    label="Optimal cut-off point for the business metric",
+)
+axs[2].legend()
+axs[2].set_xlabel("Decision threshold (probability)")
+axs[2].set_ylabel("Objective score (using cost-matrix)")
+axs[2].set_title("Objective score as a function of the decision threshold")
+
+_ = fig.suptitle("Comparison of the cut-off point for the vanilla and tuned GBDT model")
+
+# %%
+# The first remark is that both classifiers have exactly the same ROC and
+# Precision-Recall curves. It is expected because by default, the classifier is fitted
+# on the same training data. In a later section, we discuss more in details the
+# available options regarding model refitting and cross-validation.
+#
+# The second remark is that the cut-off points of the vanilla and tuned model are
+# different. To understand why the tuned model has chosen this cut-off point, we can
+# look at the right-hand side plot that plots the objective score that is our exactly
+# the same as our business metric. We see that the optimum threshold corresponds to the
+# maximum of the objective score.
+#
+# We can now check if choosing this cut-off point leads to a better score on the testing
+# set:
+scoring["cost_gain"](model_tuned, X_test, y_test)
+
+# %%
+# We observe that the decision generalized on the testing set leading to a better
+# business score.
+#
+# .. _cutoffclassifier_no_cv:
+#
+# Consideration regarding model refitting and cross-validation
+# ------------------------------------------------------------
+#
+# In the above experiment, we use the default parameter of the
+# :class:`~sklearn.model_selection.CutOffClassifier`. By default, the cut-off point is
+# tuned using a 5-fold stratified cross-validation. Also, the underlying predictive
+# model is refitted on the entire training data once the cut-off point is chosen.
+#
+# These two strategies can be changed by providing the `refit` and `cv` parameters.
+# For instance, one could provide a fitted `estimator` and set `cv="prefit"`, in which
+# case the cut-off point is found on the entire dataset provided at fitting time.
+# Also, the underlying classifier is not be refitted. Here, we can try to do such
+# experiment.
+model.fit(X_train, y_train)
+model_tuned.set_params(cv="prefit").fit(X_train, y_train)
+
+
+# %%
+# Then, we evaluate our model with the same approach as before:
+fig, axs = plt.subplots(nrows=1, ncols=3, figsize=(21, 6))
+
+linestyles = ("dashed", "dotted")
+markerstyles = ("o", ">")
+colors = ("tab:blue", "tab:orange")
+names = ("Vanilla GBDT", "Tuned GBDT")
+for idx, (est, linestyle, marker, color, name) in enumerate(
+    zip((model, model_tuned), linestyles, markerstyles, colors, names)
+):
+    decision_threshold = getattr(est, "decision_threshold_", 0.5)
+    PrecisionRecallDisplay.from_estimator(
+        est,
+        X_test,
+        y_test,
+        pos_label=pos_label,
+        linestyle=linestyle,
+        color=color,
+        ax=axs[0],
+        name=name,
+    )
+    axs[0].plot(
+        scoring["recall"](est, X_test, y_test),
+        scoring["precision"](est, X_test, y_test),
+        marker,
+        markersize=10,
+        color=color,
+        label=f"Cut-off point at probability of {decision_threshold:.2f}",
+    )
+    RocCurveDisplay.from_estimator(
+        est,
+        X_test,
+        y_test,
+        pos_label=pos_label,
+        linestyle=linestyle,
+        color=color,
+        ax=axs[1],
+        name=name,
+        plot_chance_level=idx == 1,
+    )
+    axs[1].plot(
+        scoring["fpr"](est, X_test, y_test),
+        scoring["tpr"](est, X_test, y_test),
+        marker,
+        markersize=10,
+        color=color,
+        label=f"Cut-off point at probability of {decision_threshold:.2f}",
+    )
+
+axs[0].set_title("Precision-Recall curve")
+axs[0].legend()
+axs[1].set_title("ROC curve")
+axs[1].legend()
+
+axs[2].plot(
+    model_tuned.decision_thresholds_, model_tuned.objective_scores_, color="tab:orange"
+)
+axs[2].plot(
+    model_tuned.decision_threshold_,
+    model_tuned.objective_score_,
+    "o",
+    markersize=10,
+    color="tab:orange",
+    label="Optimal cut-off point for the business metric",
+)
+axs[2].legend()
+axs[2].set_xlabel("Decision threshold (probability)")
+axs[2].set_ylabel("Objective score (using cost-matrix)")
+axs[2].set_title("Objective score as a function of the decision threshold")
+
+_ = fig.suptitle("Tuned GBDT model without refitting and using the entire dataset")
+
+# %%
+# We observe the that the optimum cut-off point is different than in the previous
+# experiment. If we look at the right-hand side plot, we observe that the objective
+# score has large plateau with a minimum cost (around 0). This behavior is symptomatic
+# of an overfitting. Because we disable cross-validation, we tuned the cut-off point on
+# the same set as the model was trained on, and this is the reason for the observed
+# overfitting.
+#
+# This option should therefore be used with caution. One needs to make sure that the
+# data providing at fitting time to the
+# :class:`~sklearn.model_selection.CutOffClassifier` is not the same as the data used to
+# train the underlying classifier. This could happen sometimes when the idea is just
+# to tune the predictive model on a completely new validation set without a costly
+# complete refit.
+#
+# In the case that cross-validation is too costly, a potential alternative is to use
+# a single train-test split by providing a floating number in range `[0, 1]` to the
+# `cv` parameter. It splits the data into a training and testing set. Let's
+# explore this option:
+model_tuned.set_params(cv=0.75).fit(X_train, y_train)
+
+# %%
+fig, axs = plt.subplots(nrows=1, ncols=3, figsize=(21, 6))
+
+linestyles = ("dashed", "dotted")
+markerstyles = ("o", ">")
+colors = ("tab:blue", "tab:orange")
+names = ("Vanilla GBDT", "Tuned GBDT")
+for idx, (est, linestyle, marker, color, name) in enumerate(
+    zip((model, model_tuned), linestyles, markerstyles, colors, names)
+):
+    decision_threshold = getattr(est, "decision_threshold_", 0.5)
+    PrecisionRecallDisplay.from_estimator(
+        est,
+        X_test,
+        y_test,
+        pos_label=pos_label,
+        linestyle=linestyle,
+        color=color,
+        ax=axs[0],
+        name=name,
+    )
+    axs[0].plot(
+        scoring["recall"](est, X_test, y_test),
+        scoring["precision"](est, X_test, y_test),
+        marker,
+        markersize=10,
+        color=color,
+        label=f"Cut-off point at probability of {decision_threshold:.2f}",
+    )
+    RocCurveDisplay.from_estimator(
+        est,
+        X_test,
+        y_test,
+        pos_label=pos_label,
+        linestyle=linestyle,
+        color=color,
+        ax=axs[1],
+        name=name,
+        plot_chance_level=idx == 1,
+    )
+    axs[1].plot(
+        scoring["fpr"](est, X_test, y_test),
+        scoring["tpr"](est, X_test, y_test),
+        marker,
+        markersize=10,
+        color=color,
+        label=f"Cut-off point at probability of {decision_threshold:.2f}",
+    )
+
+axs[0].set_title("Precision-Recall curve")
+axs[0].legend()
+axs[1].set_title("ROC curve")
+axs[1].legend()
+
+axs[2].plot(
+    model_tuned.decision_thresholds_, model_tuned.objective_scores_, color="tab:orange"
+)
+axs[2].plot(
+    model_tuned.decision_threshold_,
+    model_tuned.objective_score_,
+    "o",
+    markersize=10,
+    color="tab:orange",
+    label="Optimal cut-off point for the business metric",
+)
+axs[2].legend()
+axs[2].set_xlabel("Decision threshold (probability)")
+axs[2].set_ylabel("Objective score (using cost-matrix)")
+axs[2].set_title("Objective score as a function of the decision threshold")
+
+_ = fig.suptitle("Tuned GBDT model without refitting and using the entire dataset")
+
+# %%
+# Regarding the cut-off point, we observe that the optimum is similar to the multiple
+# repeated cross-validation case. However, be aware that a single split does not account
+# for the variability of the fit/predict process and thus we are enable to know if there
+# is any variance in the cut-off point. The repeated cross-validation averages this
+# effect.
+#
+# The second observation is about the ROC and Precision-Recall curves the tuned model.
+# We observe that the curve is now different from the vanilla model. It is not
+# surprising because we train the underlying classifier on a subset of the data provided
+# at `fit` while keeping a validation set to tune the cut-off point.
diff --git a/sklearn/metrics/_scorer.py b/sklearn/metrics/_scorer.py
index d868295edd42b..38c87cc4a9969 100644
--- a/sklearn/metrics/_scorer.py
+++ b/sklearn/metrics/_scorer.py
@@ -220,6 +220,20 @@ def _factory_args(self):
         """Return non-default make_scorer arguments for repr."""
         return ""
 
+    def _from_scores_to_class_labels(self, y_score, threshold, classes):
+        """Threshold `y_score` and return the associated class labels."""
+        pos_label = self._get_pos_label()
+        if pos_label is None:
+            map_thresholded_score_to_label = np.array([0, 1])
+        else:
+            pos_label_idx = np.flatnonzero(classes == pos_label)[0]
+            neg_label_idx = np.flatnonzero(classes != pos_label)[0]
+            map_thresholded_score_to_label = np.array([neg_label_idx, pos_label_idx])
+
+        return classes[
+            map_thresholded_score_to_label[(y_score >= threshold).astype(int)]
+        ]
+
 
 class _PredictScorer(_BaseScorer):
     def _score(self, method_caller, estimator, X, y_true, sample_weight=None):
@@ -363,6 +377,62 @@ def _factory_args(self):
         return ", needs_threshold=True"
 
 
+class _ContinuousScorer(_BaseScorer):
+    """Scorer taking a continuous response and output a score for each threshold."""
+
+    def __init__(self, score_func, sign, response_method, kwargs):
+        super().__init__(score_func=score_func, sign=sign, kwargs=kwargs)
+        self.response_method = response_method
+
+    def _score(self, method_caller, estimator, X, y_true, sample_weight=None):
+        """Evaluate predicted target values for X relative to y_true.
+
+        Parameters
+        ----------
+        estimator : object
+            Trained estimator to use for scoring.
+
+        X : {array-like, sparse matrix} of shape (n_samples, n_features)
+            Test data that will be fed to estimator.predict.
+
+        y_true : array-like of shape (n_samples,)
+            Gold standard target values for X.
+
+        sample_weight : array-like of shape (n_samples,), default=None
+            Sample weights.
+
+        Returns
+        -------
+        score : float
+            Score function applied to prediction of estimator on X.
+        """
+        pos_label = self._get_pos_label()
+        y_score = method_caller(estimator, self.response_method, X, pos_label=pos_label)
+
+        if sample_weight is not None:
+            score_func = partial(self._score_func, sample_weight=sample_weight)
+        else:
+            score_func = self._score_func
+
+        # TODO: this is pretty slow if we have a lot of `potential_thresholds`
+        # We could parallelize but then we are inside a nested parallel loop where the
+        # external parallelism is on the CV.
+        # Another guess would be to interpolate the potential thresholds at this moment.
+        # Easy for the probability case but not for the decision function case since it
+        # is not bounded.
+        potential_thresholds = np.unique(y_score)
+        score_thresholds = [
+            self._sign
+            * score_func(
+                y_true,
+                self._from_scores_to_class_labels(y_score, th, estimator.classes_),
+                **self._kwargs,
+            )
+            for th in potential_thresholds
+        ]
+        return potential_thresholds, np.array(score_thresholds)
+
+
 @validate_params(
     {
         "scoring": [str, callable, None],
diff --git a/sklearn/metrics/tests/test_score_objects.py b/sklearn/metrics/tests/test_score_objects.py
index aa63994bb5e95..0caaaa7a23ae5 100644
--- a/sklearn/metrics/tests/test_score_objects.py
+++ b/sklearn/metrics/tests/test_score_objects.py
@@ -40,6 +40,7 @@
     _passthrough_scorer,
     _MultimetricScorer,
     _check_multimetric_scoring,
+    _ContinuousScorer,
 )
 from sklearn.metrics import make_scorer, get_scorer, SCORERS, get_scorer_names
 from sklearn.neighbors import KNeighborsClassifier
@@ -1192,3 +1193,82 @@ def test_scorer_no_op_multiclass_select_proba():
         labels=lr.classes_,
     )
     scorer(lr, X_test, y_test)
+
+
+def test_continuous_scorer():
+    """Check the behaviour of the `_ContinuousScorer` class."""
+    X, y = make_classification(random_state=0)
+    estimator = LogisticRegression().fit(X, y)
+    scorer = _ContinuousScorer(
+        balanced_accuracy_score, sign=1, response_method="predict_proba", kwargs={}
+    )
+    thresholds, scores = scorer(estimator, X, y)
+
+    assert thresholds.shape == scores.shape
+    # check that the thresholds are probability with extreme values close to 0 and 1
+    assert 0 <= thresholds.min() <= 0.01
+    assert 0.99 <= thresholds.max() <= 1
+    # balanced accuracy should be between 0.5 and 1 when it is not adjusted
+    assert 0.5 <= scores.min() <= 1
+
+    # check that passing kwargs to the scorer works
+    scorer = _ContinuousScorer(
+        balanced_accuracy_score,
+        sign=1,
+        response_method="predict_proba",
+        kwargs={"adjusted": True},
+    )
+    thresholds, scores = scorer(estimator, X, y)
+
+    # balanced accuracy should be between 0.5 and 1 when it is not adjusted
+    assert 0 <= scores.min() <= 0.5
+
+    # check that we can inverse the sign of the score when dealing with `neg_*` scorer
+    scorer = _ContinuousScorer(
+        balanced_accuracy_score,
+        sign=-1,
+        response_method="predict_proba",
+        kwargs={"adjusted": True},
+    )
+    thresholds, scores = scorer(estimator, X, y)
+
+    assert all(scores <= 0)
+
+
+def test_continuous_scorer_pos_label(global_random_seed):
+    """Check that we propagate properly the `pos_label` parameter to the scorer."""
+    n_samples = 30
+    X, y = make_classification(
+        n_samples=n_samples, weights=[0.9, 0.1], random_state=global_random_seed
+    )
+    estimator = LogisticRegression().fit(X, y)
+
+    scorer = _ContinuousScorer(
+        recall_score,
+        sign=1,
+        response_method="predict_proba",
+        kwargs={"pos_label": 1},
+    )
+    thresholds_pos_label_1, scores_pos_label_1 = scorer(estimator, X, y)
+
+    scorer = _ContinuousScorer(
+        recall_score,
+        sign=1,
+        response_method="predict_proba",
+        kwargs={"pos_label": 0},
+    )
+    thresholds_pos_label_0, scores_pos_label_0 = scorer(estimator, X, y)
+
+    # If `pos_label` is not forwarded to the scorer, the thresholds will be equal.
+    # Make sure that this is not the case.
+    # assert not (thresholds_pos_label_1 == thresholds_pos_label_0).all()
+    # Since we have an imbalanced problem, the thresholds should represent higher
+    # probabilities level when `pos_label=0` than with `pos_label=1`.
+    assert np.sum(thresholds_pos_label_1 < 0.15) > 2 / 3 * n_samples
+    assert np.sum(thresholds_pos_label_0 > 0.85) > 2 / 3 * n_samples
+
+    # The recall cannot be negative and `pos_label=1` should have a higher recall
+    # since there is less samples to be considered.
+    assert 0.0 < scores_pos_label_0.min() < scores_pos_label_1.min()
+    assert scores_pos_label_0.max() == pytest.approx(1.0)
+    assert scores_pos_label_1.max() == pytest.approx(1.0)
diff --git a/sklearn/model_selection/__init__.py b/sklearn/model_selection/__init__.py
index 76dc02e625408..d55d981e5a76c 100644
--- a/sklearn/model_selection/__init__.py
+++ b/sklearn/model_selection/__init__.py
@@ -1,5 +1,7 @@
 import typing
 
+from ._prediction import CutOffClassifier
+
 from ._split import BaseCrossValidator
 from ._split import BaseShuffleSplit
 from ._split import KFold
@@ -65,6 +67,7 @@
     "StratifiedKFold",
     "StratifiedGroupKFold",
     "StratifiedShuffleSplit",
+    "CutOffClassifier",
     "check_cv",
     "cross_val_predict",
     "cross_val_score",
diff --git a/sklearn/model_selection/_prediction.py b/sklearn/model_selection/_prediction.py
new file mode 100644
index 0000000000000..e0946e0e77569
--- /dev/null
+++ b/sklearn/model_selection/_prediction.py
@@ -0,0 +1,801 @@
+from collections.abc import MutableMapping
+from inspect import signature
+from numbers import Integral, Real
+
+import numpy as np
+
+from ..base import BaseEstimator, ClassifierMixin, MetaEstimatorMixin, clone
+from ..exceptions import NotFittedError
+from ..metrics import (
+    check_scoring,
+    confusion_matrix,
+    get_scorer_names,
+    make_scorer,
+    precision_recall_curve,
+    roc_curve,
+)
+from ..metrics._scorer import _ContinuousScorer
+from ..utils import _safe_indexing
+from ..utils._param_validation import HasMethods, Interval, RealNotInt, StrOptions
+from ..utils._response import _get_response_values_binary
+from ..utils.metaestimators import available_if
+from ..utils.multiclass import type_of_target
+from ..utils.parallel import Parallel, delayed
+from ..utils.validation import (
+    _check_fit_params,
+    _check_sample_weight,
+    _check_pos_label_consistency,
+    _num_samples,
+    check_consistent_length,
+    check_is_fitted,
+    indexable,
+)
+
+from ._split import check_cv, StratifiedShuffleSplit
+
+
+def _estimator_has(attr):
+    """Check if we can delegate a method to the underlying estimator.
+
+    First, we check the first fitted estimator if available, otherwise we
+    check the unfitted estimator.
+    """
+    return lambda self: (
+        hasattr(self.estimator_, attr)
+        if hasattr(self, "estimator_")
+        else hasattr(self.estimator, attr)
+    )
+
+
+def _fit_and_score(
+    classifier,
+    X,
+    y,
+    sample_weight,
+    fit_params,
+    train_idx,
+    val_idx,
+    scorer,
+    score_method,
+):
+    """Fit a classifier and compute the scores for different decision thresholds.
+
+    Parameters
+    ----------
+    classifier : estimator instance
+        The classifier to fit and used for scoring. If `classifier` is already fitted,
+        it will be used as is.
+
+    X : {array-like, sparse matrix} of shape (n_samples, n_features)
+        The entire dataset.
+
+    y : array-like of shape (n_samples,)
+        The entire target vector.
+
+    sample_weight : array-like of shape (n_samples,)
+        Some optional associated sample weights.
+
+    fit_params : dict
+        Parameters to pass to the `fit` method of the underlying classifier.
+
+    train_idx : ndarray of shape (n_train_samples,) or None
+        The indices of the training set. If `None`, `classifier` is expected to be
+        already fitted.
+
+    val_idx : ndarray of shape (n_val_samples,)
+        The indices of the validation set used to score `classifier`. If `train_idx`,
+        the entire set will be used.
+
+    scorer : scorer instance
+        The scorer taking `classifier` and the validation set as input and outputting
+        decision thresholds and scores.
+
+    score_method : str or callable
+        The scoring method to use. Used to detect if we compute TPR/TNR or precision/
+        recall.
+
+    Returns
+    -------
+    thresholds : ndarray of shape (n_thresholds,)
+        The decision thresholds used to compute the scores. They are returned in
+        ascending order.
+
+    scores : ndarray of shape (n_thresholds,) or tuple os such arrays
+        The scores computed for each decision threshold. When TPR/TNR or precision/
+        recall are computed, `scores` is a tuple of two arrays.
+    """
+    arrays = (X, y) if sample_weight is None else (X, y, sample_weight)
+    check_consistent_length(*arrays)
+
+    fit_parameters = signature(classifier.fit).parameters
+    supports_sw = "sample_weight" in fit_parameters
+
+    if train_idx is not None:
+        X_train, X_val = _safe_indexing(X, train_idx), _safe_indexing(X, val_idx)
+        y_train, y_val = _safe_indexing(y, train_idx), _safe_indexing(y, val_idx)
+        if sample_weight is not None:
+            sw_train, sw_val = (
+                _safe_indexing(sample_weight, train_idx),
+                _safe_indexing(sample_weight, val_idx),
+            )
+        else:
+            sw_train, sw_val = None, None
+        fit_params_train = _check_fit_params(X, fit_params, indices=train_idx)
+        if supports_sw:
+            classifier.fit(X_train, y_train, sample_weight=sw_train, **fit_params_train)
+        else:
+            classifier.fit(X_train, y_train, **fit_params_train)
+    else:  # prefit estimator, only a validation set is provided
+        X_val, y_val, sw_val = X, y, sample_weight
+        check_is_fitted(classifier, "classes_")
+
+    if isinstance(score_method, str):
+        if score_method in {"max_tpr_at_tnr_constraint", "max_tnr_at_tpr_constraint"}:
+            fpr, tpr, potential_thresholds = scorer(
+                classifier, X_val, y_val, sample_weight=sw_val
+            )
+            # thresholds are in decreasing order
+            return potential_thresholds[::-1], ((1 - fpr)[::-1], tpr[::-1])
+        elif score_method in {
+            "max_precision_at_recall_constraint",
+            "max_recall_at_precision_constraint",
+        }:
+            precision, recall, potential_thresholds = scorer(
+                classifier, X_val, y_val, sample_weight=sw_val
+            )
+            # thresholds are in increasing order
+            # the last element of the precision and recall is not associated with any
+            # threshold and should be discarded
+            return potential_thresholds, (precision[:-1], recall[:-1])
+    return scorer(classifier, X_val, y_val, sample_weight=sw_val)
+
+
+class CutOffClassifier(ClassifierMixin, MetaEstimatorMixin, BaseEstimator):
+    """Decision threshold tuning for binary classification.
+
+    This estimator post-tunes the decision threshold (cut-off point) that is
+    used for converting probabilities (i.e. output of `predict_proba`) or
+    decision function (i.e. output of `decision_function`) into a predicted
+    class. The tuning is done by maximizing a binary metric, potentially
+    constrained by a another metric.
+
+    Read more in the :ref:`User Guide <cutoffclassifier>`.
+
+    .. versionadded:: 1.3
+
+    Parameters
+    ----------
+    estimator : estimator instance
+        The classifier, fitted or not fitted, for which we want to optimize
+        the decision threshold used during `predict`.
+
+    objective_metric : {"max_tpr_at_tnr_constraint", "max_tnr_at_tpr_constraint", \
+            "max_precision_at_recall_constraint, "max_recall_at_precision_constraint"} \
+            , str, dict or callable, default="balanced_accuracy"
+        The objective metric to be optimized. Can be one of:
+
+        * a string associated to a scoring function (see model evaluation
+          documentation);
+        * a scorer callable object created with :func:`~sklearn.metrics.make_scorer`;
+        * `"max_tnr_at_tpr_constraint"`: find the decision threshold for a true
+          positive ratio (TPR) of `constraint_value`;
+        * `"max_tpr_at_tnr_constraint"`: find the decision threshold for a true
+          negative ratio (TNR) of `constraint_value`.
+        * `"max_precision_at_recall_constraint"`: find the decision threshold for a
+          recall of `constraint_value`;
+        * `"max_recall_at_precision_constraint"`: find the decision threshold for a
+          precision of `constraint_value`.
+        * a dictionary to be used as cost-sensitive matrix. The keys of the
+          dictionary should be: `("tp", "fp", "tn", "fn")`. The values of the
+          dictionary corresponds costs (negative values) and gains (positive
+          values).
+
+    constraint_value : float, default=None
+        The value associated with the `objective_metric` metric for which we
+        want to find the decision threshold when `objective_metric` is equal one of
+        `"max_tnr_at_tpr_constraint"`, `"max_tpr_at_tnr_constraint"`,
+        `"max_precision_at_recall_constraint"`, or
+        `"max_recall_at_precision_constraint"`.
+
+    pos_label : int, float, bool or str, default=None
+        The label of the positive class. Used when `objective_metric` is
+        `"max_tnr_at_tpr_constraint"`"`, `"max_tpr_at_tnr_constraint"`, or a dictionary.
+        When `pos_label=None`, if `y_true` is in `{-1, 1}` or `{0, 1}`,
+        `pos_label` is set to 1, otherwise an error will be raised. When using a
+        scorer, `pos_label` can be passed as a keyword argument to
+        :func:`~sklearn.metrics.make_scorer`.
+
+    response_method : {"auto", "decision_function", "predict_proba"}, default="auto"
+        Methods by the classifier `base_estimator` corresponding to the
+        decision function for which we want to find a threshold. It can be:
+
+        * if `"auto"`, it will try to invoke, for each classifier,
+          `"predict_proba"` or `"decision_function"` in that order.
+        * otherwise, one of `"predict_proba"` or `"decision_function"`.
+          If the method is not implemented by the classifier, it will raise an
+          error.
+
+    n_thresholds : int, default=100
+        The number of decision threshold to use when discretizing the output
+        of the classifier `method`.
+
+    cv : int, float, cross-validation generator, iterable or "prefit", default=None
+        Determines the cross-validation splitting strategy to train classifier.
+        Possible inputs for cv are:
+
+        * `None`, to use the default 5-fold stratified K-fold cross validation;
+        * An integer number, to specify the number of folds in a stratified k-fold;
+        * A float number, to specify a single shuffle split. The floating number should
+          be in (0, 1) and represent the size of the validation set;
+        * An object to be used as a cross-validation generator;
+        * An iterable yielding train, test splits;
+        * `"prefit"`, to bypass the cross-validation.
+
+        Refer :ref:`User Guide <cross_validation>` for the various
+        cross-validation strategies that can be used here.
+
+        .. warning::
+            Using `cv="prefit"` and passing the same dataset for fitting `estimator`
+            and tuning the cut-off point is subject to undesired overfitting. You can
+            refer to :ref:`cutoffclassifier_no_cv` for an example.
+
+            This option should only be used when the set used to fit `estimator` is
+            different from the one used to tune the cut-off point (by calling
+            :meth:`CutOffClassifier.fit`).
+
+    refit : "auto" or bool, default="auto"
+        Whether or not to refit the classifier on the entire training set once
+        the decision threshold has been found. By default, `refit="auto"` is
+        equivalent to `refit=False` when `cv` is a float number using a single
+        shuffle split or `cv="prefit"` otherwise `refit=True` in all other
+        cases. Note that forcing `refit=False` on cross-validation having more
+        than a single split will raise an error. Similarly, `refit=True` in
+        conjunction with `cv="prefit"` will raise an error.
+
+    n_jobs : int, default=None
+        The number of jobs to run in parallel. When `cv` represents a
+        cross-validation strategy, the fitting and scoring on each data split
+        is done in parallel. ``None`` means 1 unless in a
+        :obj:`joblib.parallel_backend` context. ``-1`` means using all
+        processors. See :term:`Glossary <n_jobs>` for more details.
+
+    random_state : int, RandomState instance or None, default=None
+        Controls the randomness of cross-validation when `cv` is a float.
+        See :term:`Glossary <random_state>`.
+
+    Attributes
+    ----------
+    estimator_ : estimator instance
+        The fitted classifier used when predicting.
+
+    decision_threshold_ : float
+        The new decision threshold.
+
+    decision_thresholds_ : ndarray of shape (n_thresholds,)
+        All decision thresholds that were evaluated.
+
+    objective_score_ : float or tuple of floats
+        The score of the objective metric associated with the decision threshold found.
+        When `objective_metric` is one of `"max_tpr_at_tnr_constraint"`,
+        `"max_tnr_at_tpr_constraint"`, `"max_precision_at_recall_constraint"`,
+        `"max_recall_at_precision_constraint"`, it will corresponds to a tuple of
+        two float values: the first one is the score of the metric which is constrained
+        and the second one is the score of the maximized metric.
+
+    objective_scores_ : ndarray of shape (n_thresholds,)
+        The scores of the objective metric associated with the decision thresholds.
+
+    classes_ : ndarray of shape (n_classes,)
+        The class labels.
+
+    n_features_in_ : int
+        Number of features seen during :term:`fit`. Only defined if the
+        underlying estimator exposes such an attribute when fit.
+
+    feature_names_in_ : ndarray of shape (`n_features_in_`,)
+        Names of features seen during :term:`fit`. Only defined if the
+        underlying estimator exposes such an attribute when fit.
+
+    See Also
+    --------
+    sklearn.calibration.CalibratedClassifierCV : Estimator that calibrates
+        probabilities.
+
+    Examples
+    --------
+    >>> from sklearn.datasets import make_classification
+    >>> from sklearn.ensemble import RandomForestClassifier
+    >>> from sklearn.metrics import classification_report
+    >>> from sklearn.model_selection import CutOffClassifier, train_test_split
+    >>> X, y = make_classification(
+    ...     n_samples=1_000, weights=[0.9, 0.1], class_sep=0.8, random_state=42
+    ... )
+    >>> X_train, X_test, y_train, y_test = train_test_split(
+    ...     X, y, stratify=y, random_state=42
+    ... )
+    >>> classifier = RandomForestClassifier(random_state=0).fit(X_train, y_train)
+    >>> print(classification_report(y_test, classifier.predict(X_test)))
+                  precision    recall  f1-score   support
+    <BLANKLINE>
+               0       0.94      0.99      0.96       224
+               1       0.80      0.46      0.59        26
+    <BLANKLINE>
+        accuracy                           0.93       250
+       macro avg       0.87      0.72      0.77       250
+    weighted avg       0.93      0.93      0.92       250
+    <BLANKLINE>
+    >>> classifier_tuned = CutOffClassifier(
+    ...     classifier, objective_metric="max_precision_at_recall_constraint",
+    ...     constraint_value=0.7,
+    ... ).fit(X_train, y_train)
+    >>> print(
+    ...     f"Cut-off point found at {classifier_tuned.decision_threshold_:.3f} for a "
+    ...     f"recall of {classifier_tuned.objective_score_[0]:.3f} and a precision of "
+    ...     f"{classifier_tuned.objective_score_[1]:.3f}."
+    ... )
+    Cut-off point found at 0.3... for a recall of 0.7... and a precision of 0.7...
+    >>> print(classification_report(y_test, classifier_tuned.predict(X_test)))
+                  precision    recall  f1-score   support
+    <BLANKLINE>
+               0       0.96      0.96      0.96       224
+               1       0.68      0.65      0.67        26
+    <BLANKLINE>
+        accuracy                           0.93       250
+       macro avg       0.82      0.81      0.81       250
+    weighted avg       0.93      0.93      0.93       250
+    <BLANKLINE>
+    """
+
+    _parameter_constraints: dict = {
+        "estimator": [
+            HasMethods(["fit", "predict_proba"]),
+            HasMethods(["fit", "decision_function"]),
+        ],
+        "objective_metric": [
+            StrOptions(
+                set(get_scorer_names())
+                | {
+                    "max_tnr_at_tpr_constraint",
+                    "max_tpr_at_tnr_constraint",
+                    "max_precision_at_recall_constraint",
+                    "max_recall_at_precision_constraint",
+                }
+            ),
+            callable,
+            MutableMapping,
+        ],
+        "constraint_value": [Real, None],
+        "pos_label": [Real, str, "boolean", None],
+        "response_method": [StrOptions({"auto", "predict_proba", "decision_function"})],
+        "n_thresholds": [Interval(Integral, 1, None, closed="left")],
+        "cv": [
+            "cv_object",
+            StrOptions({"prefit"}),
+            Interval(RealNotInt, 0.0, 1.0, closed="right"),
+        ],
+        "refit": ["boolean", StrOptions({"auto"})],
+        "n_jobs": [Integral, None],
+        "random_state": ["random_state"],
+    }
+
+    def __init__(
+        self,
+        estimator,
+        *,
+        objective_metric="balanced_accuracy",
+        constraint_value=None,
+        pos_label=None,
+        response_method="auto",
+        n_thresholds=100,
+        cv=None,
+        refit="auto",
+        n_jobs=None,
+        random_state=None,
+    ):
+        self.estimator = estimator
+        self.objective_metric = objective_metric
+        self.constraint_value = constraint_value
+        self.pos_label = pos_label
+        self.response_method = response_method
+        self.n_thresholds = n_thresholds
+        self.cv = cv
+        self.refit = refit
+        self.n_jobs = n_jobs
+        self.random_state = random_state
+
+    def fit(self, X, y, sample_weight=None, **fit_params):
+        """Fit the classifier and post-tune the decision threshold.
+
+        Parameters
+        ----------
+        X : {array-like, sparse matrix} of shape (n_samples, n_features)
+            Training data.
+
+        y : array-like of shape (n_samples,)
+            Target values.
+
+        sample_weight : array-like of shape (n_samples,), default=None
+            Sample weights. If `None`, then samples are equally weighted.
+
+        **fit_params : dict
+            Parameters to pass to the `fit` method of the underlying
+            classifier.
+
+        Returns
+        -------
+        self : object
+            Returns an instance of self.
+        """
+        self._validate_params()
+        X, y = indexable(X, y)
+
+        y_type = type_of_target(y, input_name="y")
+        if y_type != "binary":
+            raise ValueError(
+                f"Only binary classification is supported. Unknown label type: {y_type}"
+            )
+
+        if sample_weight is not None:
+            sample_weight = _check_sample_weight(sample_weight, X)
+
+        if isinstance(self.cv, Real) and 0 < self.cv <= 1:
+            cv = StratifiedShuffleSplit(
+                n_splits=1, test_size=self.cv, random_state=self.random_state
+            )
+            refit = False if self.refit == "auto" else self.refit
+        elif self.cv == "prefit":
+            if self.refit is True:
+                raise ValueError("When cv='prefit', refit cannot be True.")
+            try:
+                check_is_fitted(self.estimator, "classes_")
+            except NotFittedError as exc:
+                raise NotFittedError(
+                    """When cv='prefit', `estimator` must be fitted."""
+                ) from exc
+            cv, refit = self.cv, False
+        else:
+            cv = check_cv(self.cv, y=y, classifier=True)
+            if self.refit is False and cv.get_n_splits() > 1:
+                raise ValueError("When cv has several folds, refit cannot be False.")
+            if self.refit == "auto":
+                refit = cv.get_n_splits() > 1
+            else:
+                refit = self.refit
+
+        if self.response_method == "auto":
+            self._response_method = ["predict_proba", "decision_function"]
+        else:
+            self._response_method = self.response_method
+
+        if isinstance(self.objective_metric, str) and self.objective_metric in {
+            "max_tpr_at_tnr_constraint",
+            "max_tnr_at_tpr_constraint",
+            "max_precision_at_recall_constraint",
+            "max_recall_at_precision_constraint",
+        }:
+            if self.constraint_value is None:
+                raise ValueError(
+                    "When `objective_metric` is 'max_tpr_at_tnr_constraint', "
+                    "'max_tnr_at_tpr_constraint', 'max_precision_at_recall_constraint',"
+                    " or 'max_recall_at_precision_constraint', `constraint_value` must "
+                    "be provided. Got None instead."
+                )
+            constraint_value = self.constraint_value
+        else:
+            constraint_value = "highest"
+
+        fit_parameters = signature(self.estimator.fit).parameters
+        supports_sw = "sample_weight" in fit_parameters
+
+        # in the following block, we:
+        # - define the final classifier `self.estimator_` and train it if necessary
+        # - define `classifier` to be used to post-tune the decision threshold
+        # - define `split` to be used to fit/score `classifier`
+        if cv == "prefit":
+            self.estimator_ = self.estimator
+            classifier = self.estimator_
+            splits = ([None, range(_num_samples(X))],)
+        else:
+            self.estimator_ = clone(self.estimator)
+            classifier = clone(self.estimator)
+            splits = cv.split(X, y)
+
+            if refit:
+                # train on the whole dataset
+                X_train, y_train, sw_train = X, y, sample_weight
+                fit_params_train = _check_fit_params(X, fit_params, indices=None)
+            else:
+                # single split cross-validation
+                train_idx, _ = next(cv.split(X, y))
+                X_train = _safe_indexing(X, train_idx)
+                y_train = _safe_indexing(y, train_idx)
+                if sample_weight is not None:
+                    sw_train = _safe_indexing(sample_weight, train_idx)
+                else:
+                    sw_train = None
+                fit_params_train = _check_fit_params(X, fit_params, indices=train_idx)
+
+            if sw_train is not None and supports_sw:
+                self.estimator_.fit(
+                    X_train, y_train, sample_weight=sw_train, **fit_params_train
+                )
+            else:
+                self.estimator_.fit(X_train, y_train, **fit_params_train)
+
+        if isinstance(self.objective_metric, MutableMapping):
+            keys = set(self.objective_metric.keys())
+            if not keys == {"tp", "tn", "fp", "fn"}:
+                raise ValueError(
+                    "Invalid keys in `objective_metric`. Valid keys are "
+                    f"'tp', 'tn', 'fp', and 'fn'. Got {keys} instead."
+                )
+            pos_label = _check_pos_label_consistency(self.pos_label, y)
+
+            def cost_sensitive_score_func(y_true, y_pred, **kwargs):
+                costs_and_gain = np.array(
+                    [
+                        [kwargs["tn"], kwargs["fp"]],
+                        [kwargs["fn"], kwargs["tp"]],
+                    ]
+                )
+
+                sample_weight = kwargs.get("sample_weight", None)
+                cm = confusion_matrix(y_true, y_pred, sample_weight=sample_weight)
+
+                pos_label, classes = kwargs["pos_label"], np.unique(y_true)
+                pos_label_idx = np.searchsorted(classes, pos_label)
+                if pos_label_idx == 0:
+                    # reorder the confusion matrix to be aligned with the cost-matrix
+                    cm = cm[::-1, ::-1]
+
+                return (costs_and_gain * cm).sum()
+
+            self._scorer = _ContinuousScorer(
+                score_func=cost_sensitive_score_func,
+                sign=1,
+                response_method=self._response_method,
+                kwargs={
+                    **self.objective_metric,
+                    "pos_label": pos_label,
+                },
+            )
+        elif self.objective_metric in {
+            "max_tnr_at_tpr_constraint",
+            "max_tpr_at_tnr_constraint",
+            "max_precision_at_recall_constraint",
+            "max_recall_at_precision_constraint",
+        }:
+            if self._response_method == "predict_proba":
+                params_scorer = {"needs_proba": True, "pos_label": self.pos_label}
+            elif (
+                isinstance(self._response_method, list)
+                and self._response_method[0] == "predict_proba"
+                and hasattr(classifier, "predict_proba")
+            ):
+                # TODO: this is due to a limitation in `make_scorer`: ideally, we should
+                # be able to pass a list of response methods to `make_scorer` and give
+                # priority to `predict_proba` other `decision_function`.
+                # Here, we manually check if the classifier provide `predict_proba` to
+                # use `needs_proba` instead and ensure that no error will be raised.
+                params_scorer = {"needs_proba": True, "pos_label": self.pos_label}
+            else:
+                params_scorer = {"needs_threshold": True, "pos_label": self.pos_label}
+
+            if "tpr" in self.objective_metric:  # tpr/tnr
+                score_func = roc_curve
+            else:  # precision/recall
+                score_func = precision_recall_curve
+            self._scorer = make_scorer(score_func, **params_scorer)
+        else:
+            scoring = check_scoring(classifier, scoring=self.objective_metric)
+            # add `pos_label` if requested by the scorer function
+            scorer_kwargs = {**scoring._kwargs}
+            signature_scoring_func = signature(scoring._score_func)
+            if (
+                "pos_label" in signature_scoring_func.parameters
+                and "pos_label" not in scorer_kwargs
+            ):
+                if self.pos_label is None:
+                    # Since the provided `pos_label` is the default, we need to
+                    # use the default value of the scoring function that can be either
+                    # `None` or `1`.
+                    scorer_kwargs["pos_label"] = signature_scoring_func.parameters[
+                        "pos_label"
+                    ].default
+                else:
+                    scorer_kwargs["pos_label"] = self.pos_label
+            # transform a binary metric into a curve metric for all possible decision
+            # thresholds
+            self._scorer = _ContinuousScorer(
+                score_func=scoring._score_func,
+                sign=scoring._sign,
+                response_method=self._response_method,
+                kwargs=scorer_kwargs,
+            )
+
+        cv_thresholds, cv_scores = zip(
+            *Parallel(n_jobs=self.n_jobs)(
+                delayed(_fit_and_score)(
+                    classifier,
+                    X,
+                    y,
+                    sample_weight,
+                    fit_params,
+                    train_idx,
+                    val_idx,
+                    self._scorer,
+                    self.objective_metric,
+                )
+                for train_idx, val_idx in splits
+            )
+        )
+
+        if any(len(th) == 1 for th in cv_thresholds):
+            raise ValueError(
+                "The provided estimator makes constant predictions. Therefore, it is "
+                "impossible to optimize the decision threshold."
+            )
+
+        # find the global min and max thresholds across all folds
+        min_threshold = np.min([th.min() for th in cv_thresholds])
+        max_threshold = np.max([th.max() for th in cv_thresholds])
+        self.decision_thresholds_ = np.linspace(
+            min_threshold, max_threshold, num=self.n_thresholds
+        )
+
+        def _mean_interpolated_score(threshold_interpolated, cv_thresholds, cv_scores):
+            return np.mean(
+                [
+                    np.interp(threshold_interpolated, th, sc)
+                    for th, sc in zip(cv_thresholds, cv_scores)
+                ],
+                axis=0,
+            )
+
+        if constraint_value == "highest":  # find best score
+            self.objective_scores_ = _mean_interpolated_score(
+                self.decision_thresholds_, cv_thresholds, cv_scores
+            )
+            best_idx = self.objective_scores_.argmax()
+            self.objective_score_ = self.objective_scores_[best_idx]
+            self.decision_threshold_ = self.decision_thresholds_[best_idx]
+        else:
+            if "tpr" in self.objective_metric:  # tpr/tnr
+                mean_tnr, mean_tpr = [
+                    _mean_interpolated_score(
+                        self.decision_thresholds_, cv_thresholds, sc
+                    )
+                    for sc in zip(*cv_scores)
+                ]
+            else:  # precision/recall
+                mean_precision, mean_recall = [
+                    _mean_interpolated_score(
+                        self.decision_thresholds_, cv_thresholds, sc
+                    )
+                    for sc in zip(*cv_scores)
+                ]
+
+            def _get_best_idx(constrained_score, maximized_score):
+                """Find the index of the best score constrained by another score."""
+                indices = np.arange(len(constrained_score))
+                mask = constrained_score >= constraint_value
+                mask_idx = maximized_score[mask].argmax()
+                return indices[mask][mask_idx]
+
+            if self.objective_metric == "max_tpr_at_tnr_constraint":
+                constrained_score, maximized_score = mean_tnr, mean_tpr
+            elif self.objective_metric == "max_tnr_at_tpr_constraint":
+                constrained_score, maximized_score = mean_tpr, mean_tnr
+            elif self.objective_metric == "max_precision_at_recall_constraint":
+                constrained_score, maximized_score = mean_recall, mean_precision
+            else:  # max_recall_at_precision_constraint
+                constrained_score, maximized_score = mean_precision, mean_recall
+
+            self.objective_scores_ = (constrained_score, maximized_score)
+            best_idx = _get_best_idx(constrained_score, maximized_score)
+            self.objective_score_ = (
+                constrained_score[best_idx],
+                maximized_score[best_idx],
+            )
+            self.decision_threshold_ = self.decision_thresholds_[best_idx]
+
+        if hasattr(self.estimator_, "n_features_in_"):
+            self.n_features_in_ = self.estimator_.n_features_in_
+        if hasattr(self.estimator_, "feature_names_in_"):
+            self.feature_names_in_ = self.estimator_.feature_names_in_
+
+        return self
+
+    @property
+    def classes_(self):
+        """Classes labels."""
+        return self.estimator_.classes_
+
+    def predict(self, X):
+        """Predict the target of new samples.
+
+        Parameters
+        ----------
+        X : {array-like, sparse matrix} of shape (n_samples, n_features)
+            The samples, as accepted by `estimator.predict`.
+
+        Returns
+        -------
+        C : ndarray of shape (n_samples,)
+            The predicted class.
+        """
+        check_is_fitted(self, "estimator_")
+        pos_label = self._scorer._get_pos_label()
+        y_score, _ = _get_response_values_binary(
+            self.estimator_, X, self._response_method, pos_label=pos_label
+        )
+        return self._scorer._from_scores_to_class_labels(
+            y_score, self.decision_threshold_, self.classes_
+        )
+
+    @available_if(_estimator_has("predict_proba"))
+    def predict_proba(self, X):
+        """Predict class probabilities for `X` using the fitted estimator.
+
+        Parameters
+        ----------
+        X : {array-like, sparse matrix} of shape (n_samples, n_features)
+            Training vectors, where `n_samples` is the number of samples and
+            `n_features` is the number of features.
+
+        Returns
+        -------
+        probabilities : ndarray of shape (n_samples, n_classes)
+            The class probabilities of the input samples.
+        """
+        check_is_fitted(self, "estimator_")
+        return self.estimator_.predict_proba(X)
+
+    @available_if(_estimator_has("predict_log_proba"))
+    def predict_log_proba(self, X):
+        """Predict logarithm class probabilities for `X` using the fitted estimator.
+
+        Parameters
+        ----------
+        X : {array-like, sparse matrix} of shape (n_samples, n_features)
+            Training vectors, where `n_samples` is the number of samples and
+            `n_features` is the number of features.
+
+        Returns
+        -------
+        log_probabilities : ndarray of shape (n_samples, n_classes)
+            The logarithm class probabilities of the input samples.
+        """
+        check_is_fitted(self, "estimator_")
+        return self.estimator_.predict_log_proba(X)
+
+    @available_if(_estimator_has("decision_function"))
+    def decision_function(self, X):
+        """Decision function for samples in `X` using the fitted estimator.
+
+        Parameters
+        ----------
+        X : {array-like, sparse matrix} of shape (n_samples, n_features)
+            Training vectors, where `n_samples` is the number of samples and
+            `n_features` is the number of features.
+
+        Returns
+        -------
+        decisions : ndarray of shape (n_samples,)
+            The decision function computed the fitted estimator.
+        """
+        check_is_fitted(self, "estimator_")
+        return self.estimator_.decision_function(X)
+
+    def _more_tags(self):
+        return {
+            "binary_only": True,
+            "_xfail_checks": {
+                "check_classifiers_train": "Threshold at probability 0.5 does not hold",
+                "check_sample_weights_invariance": (
+                    "Due to the cross-validation and sample ordering, removing a sample"
+                    " is not strictly equal to putting is weight to zero. Specific unit"
+                    " tests are added for CutOffClassifier specifically."
+                ),
+            },
+        }
diff --git a/sklearn/model_selection/tests/test_prediction.py b/sklearn/model_selection/tests/test_prediction.py
new file mode 100644
index 0000000000000..afbfa62401634
--- /dev/null
+++ b/sklearn/model_selection/tests/test_prediction.py
@@ -0,0 +1,833 @@
+import numpy as np
+import pytest
+
+from sklearn.base import clone
+from sklearn.datasets import load_breast_cancer, load_iris, make_classification
+from sklearn.dummy import DummyClassifier
+from sklearn.ensemble import GradientBoostingClassifier
+from sklearn.exceptions import NotFittedError
+from sklearn.linear_model import LogisticRegression
+from sklearn.metrics import (
+    balanced_accuracy_score,
+    confusion_matrix,
+    fbeta_score,
+    f1_score,
+    make_scorer,
+    precision_recall_curve,
+    precision_score,
+    recall_score,
+    roc_curve,
+)
+from sklearn.metrics._scorer import _ContinuousScorer
+from sklearn.pipeline import make_pipeline
+from sklearn.preprocessing import StandardScaler
+from sklearn.svm import SVC
+from sklearn.tree import DecisionTreeClassifier
+from sklearn.utils._mocking import CheckingClassifier
+from sklearn.utils._testing import (
+    _convert_container,
+    assert_allclose,
+    assert_array_equal,
+)
+
+from sklearn.model_selection import CutOffClassifier
+from sklearn.model_selection._prediction import _fit_and_score
+
+
+@pytest.mark.parametrize(
+    "scorer, score_method",
+    [
+        (
+            _ContinuousScorer(
+                score_func=balanced_accuracy_score,
+                sign=1,
+                response_method="predict_proba",
+                kwargs={},
+            ),
+            "balanced_accuracy",
+        ),
+        (
+            make_scorer(roc_curve, needs_proba=True),
+            "max_tnr_at_tpr_constraint",
+        ),
+        (
+            make_scorer(roc_curve, needs_proba=True),
+            "max_tpr_at_tnr_constraint",
+        ),
+        (
+            make_scorer(precision_recall_curve, needs_proba=True),
+            "max_precision_at_recall_constraint",
+        ),
+        (
+            make_scorer(precision_recall_curve, needs_proba=True),
+            "max_recall_at_precision_constraint",
+        ),
+    ],
+)
+def test_fit_and_score_scorers(scorer, score_method):
+    """Check that `_fit_and_score` returns thresholds in ascending order for the
+    different accepted scorers."""
+    X, y = make_classification(n_samples=100, random_state=0)
+    train_idx, val_idx = np.arange(50), np.arange(50, 100)
+    classifier = LogisticRegression()
+
+    thresholds, scores = _fit_and_score(
+        classifier,
+        X,
+        y,
+        sample_weight=None,
+        fit_params={},
+        train_idx=train_idx,
+        val_idx=val_idx,
+        scorer=scorer,
+        score_method=score_method,
+    )
+
+    if score_method.startswith("max_"):
+        assert_array_equal(np.argsort(thresholds), np.arange(len(thresholds)))
+        assert isinstance(scores, tuple) and len(scores) == 2
+        for sc in scores:
+            assert np.logical_and(sc >= 0, sc <= 1).all()
+    else:
+        assert_array_equal(np.argsort(thresholds), np.arange(len(thresholds)))
+        assert isinstance(scores, np.ndarray)
+        assert np.logical_and(scores >= 0, scores <= 1).all()
+
+
+@pytest.mark.parametrize(
+    "scorer, score_method, expected_score",
+    [
+        (
+            _ContinuousScorer(
+                score_func=balanced_accuracy_score,
+                sign=1,
+                response_method="predict_proba",
+                kwargs={},
+            ),
+            "balanced_accuracy",
+            [0.5, 1.0],
+        ),
+        (
+            make_scorer(roc_curve, needs_proba=True),
+            "max_tnr_at_tpr_constraint",
+            [[0.0, 1.0, 1.0], [1.0, 1.0, 0.0]],
+        ),
+        (
+            make_scorer(roc_curve, needs_proba=True),
+            "max_tpr_at_tnr_constraint",
+            [[0.0, 1.0, 1.0], [1.0, 1.0, 0.0]],
+        ),
+        (
+            make_scorer(precision_recall_curve, needs_proba=True),
+            "max_precision_at_recall_constraint",
+            [[0.5, 1.0], [1.0, 1.0]],
+        ),
+        (
+            make_scorer(precision_recall_curve, needs_proba=True),
+            "max_recall_at_precision_constraint",
+            [[0.5, 1.0], [1.0, 1.0]],
+        ),
+    ],
+)
+def test_fit_and_score_prefit(scorer, score_method, expected_score):
+    """Check the behaviour with a prefit classifier."""
+    X, y = make_classification(n_samples=100, random_state=0)
+
+    # `train_idx is None` to indicate that the classifier is prefit
+    train_idx, val_idx = None, np.arange(50, 100)
+    classifier = DecisionTreeClassifier(random_state=0)
+
+    with pytest.raises(NotFittedError):
+        _fit_and_score(
+            classifier,
+            X,
+            y,
+            sample_weight=None,
+            fit_params={},
+            train_idx=train_idx,
+            val_idx=val_idx,
+            scorer=scorer,
+            score_method=score_method,
+        )
+
+    classifier.fit(X, y)
+    # make sure that the classifier memorized the full dataset such that
+    # we get perfect predictions and thus match the expected score
+    assert classifier.score(X[val_idx], y[val_idx]) == pytest.approx(1.0)
+
+    thresholds, scores = _fit_and_score(
+        classifier,
+        X,
+        y,
+        sample_weight=None,
+        fit_params={},
+        train_idx=train_idx,
+        val_idx=val_idx,
+        scorer=scorer,
+        score_method=score_method,
+    )
+    assert_array_equal(np.argsort(thresholds), np.arange(len(thresholds)))
+    assert_allclose(scores, expected_score)
+
+
+@pytest.mark.parametrize(
+    "scorer, score_method",
+    [
+        (
+            _ContinuousScorer(
+                score_func=balanced_accuracy_score,
+                sign=1,
+                response_method="predict_proba",
+                kwargs={},
+            ),
+            "balanced_accuracy",
+        ),
+        (
+            make_scorer(roc_curve, needs_proba=True),
+            "max_tnr_at_tpr_constraint",
+        ),
+        (
+            make_scorer(roc_curve, needs_proba=True),
+            "max_tpr_at_tnr_constraint",
+        ),
+        (
+            make_scorer(precision_recall_curve, needs_proba=True),
+            "max_precision_at_recall_constraint",
+        ),
+        (
+            make_scorer(precision_recall_curve, needs_proba=True),
+            "max_recall_at_precision_constraint",
+        ),
+    ],
+)
+def test_fit_and_score_sample_weight(scorer, score_method):
+    """Check that we dispatch the sample-weight to fit and score the classifier."""
+    X, y = load_iris(return_X_y=True)
+    X, y = X[:100], y[:100]  # only 2 classes
+
+    # create a dataset and repeat twice the sample of class #0
+    X_repeated, y_repeated = np.vstack([X, X[y == 0]]), np.hstack([y, y[y == 0]])
+    # create a sample weight vector that is equivalent to the repeated dataset
+    sample_weight = np.ones_like(y)
+    sample_weight[:50] *= 2
+
+    classifier = LogisticRegression()
+    train_repeated_idx = np.arange(X_repeated.shape[0])
+    val_repeated_idx = np.arange(X_repeated.shape[0])
+    thresholds_repeated, scores_repeated = _fit_and_score(
+        classifier,
+        X_repeated,
+        y_repeated,
+        sample_weight=None,
+        fit_params={},
+        train_idx=train_repeated_idx,
+        val_idx=val_repeated_idx,
+        scorer=scorer,
+        score_method=score_method,
+    )
+
+    train_idx, val_idx = np.arange(X.shape[0]), np.arange(X.shape[0])
+    thresholds, scores = _fit_and_score(
+        classifier,
+        X,
+        y,
+        sample_weight=sample_weight,
+        fit_params={},
+        train_idx=train_idx,
+        val_idx=val_idx,
+        scorer=scorer,
+        score_method=score_method,
+    )
+
+    assert_allclose(thresholds_repeated, thresholds)
+    assert_allclose(scores_repeated, scores)
+
+
+@pytest.mark.parametrize(
+    "scorer, score_method",
+    [
+        (
+            _ContinuousScorer(
+                score_func=balanced_accuracy_score,
+                sign=1,
+                response_method="predict_proba",
+                kwargs={},
+            ),
+            "balanced_accuracy",
+        ),
+        (
+            make_scorer(roc_curve, needs_proba=True),
+            "max_tnr_at_tpr_constraint",
+        ),
+        (
+            make_scorer(roc_curve, needs_proba=True),
+            "max_tpr_at_tnr_constraint",
+        ),
+        (
+            make_scorer(precision_recall_curve, needs_proba=True),
+            "max_precision_at_recall_constraint",
+        ),
+        (
+            make_scorer(precision_recall_curve, needs_proba=True),
+            "max_recall_at_precision_constraint",
+        ),
+    ],
+)
+@pytest.mark.parametrize("fit_params_type", ["list", "array"])
+def test_fit_and_score_fit_params(scorer, score_method, fit_params_type):
+    """Check that we pass `fit_params` to the classifier when calling `fit`."""
+    X, y = make_classification(n_samples=100, random_state=0)
+    fit_params = {
+        "a": _convert_container(y, fit_params_type),
+        "b": _convert_container(y, fit_params_type),
+    }
+
+    classifier = CheckingClassifier(expected_fit_params=["a", "b"], random_state=0)
+    train_idx, val_idx = np.arange(50), np.arange(50, 100)
+
+    _fit_and_score(
+        classifier,
+        X,
+        y,
+        sample_weight=None,
+        fit_params=fit_params,
+        train_idx=train_idx,
+        val_idx=val_idx,
+        scorer=scorer,
+        score_method=score_method,
+    )
+
+
+def test_cutoffclassifier_no_binary():
+    """Check that we raise an informative error message for non-binary problem."""
+    X, y = make_classification(n_classes=3, n_clusters_per_class=1)
+    err_msg = "Only binary classification is supported."
+    with pytest.raises(ValueError, match=err_msg):
+        CutOffClassifier(LogisticRegression()).fit(X, y)
+
+
+@pytest.mark.parametrize(
+    "params, err_type, err_msg",
+    [
+        (
+            {"cv": "prefit", "refit": True},
+            ValueError,
+            "When cv='prefit', refit cannot be True.",
+        ),
+        (
+            {"cv": 10, "refit": False},
+            ValueError,
+            "When cv has several folds, refit cannot be False.",
+        ),
+        (
+            {"cv": "prefit", "refit": False},
+            NotFittedError,
+            "`estimator` must be fitted.",
+        ),
+    ],
+)
+def test_cutoffclassifier_conflict_cv_refit(params, err_type, err_msg):
+    """Check that we raise an informative error message when `cv` and `refit`
+    cannot be used together.
+    """
+    X, y = make_classification(n_samples=100, random_state=0)
+    with pytest.raises(err_type, match=err_msg):
+        CutOffClassifier(LogisticRegression(), **params).fit(X, y)
+
+
+@pytest.mark.parametrize(
+    "estimator",
+    [LogisticRegression(), SVC(), GradientBoostingClassifier(n_estimators=4)],
+)
+@pytest.mark.parametrize(
+    "response_method", ["predict_proba", "predict_log_proba", "decision_function"]
+)
+def test_cutoffclassifier_estimator_response_methods(estimator, response_method):
+    """Check that `CutOffClassifier` exposes the same response methods as the
+    underlying estimator.
+    """
+    X, y = make_classification(n_samples=100, random_state=0)
+
+    model = CutOffClassifier(estimator)
+    assert hasattr(model, response_method) == hasattr(estimator, response_method)
+
+    model.fit(X, y)
+    assert hasattr(model, response_method) == hasattr(estimator, response_method)
+
+    if hasattr(model, response_method):
+        y_pred_cutoff = getattr(model, response_method)(X)
+        y_pred_underlying_estimator = getattr(model.estimator_, response_method)(X)
+
+        assert_allclose(y_pred_cutoff, y_pred_underlying_estimator)
+
+
+@pytest.mark.parametrize(
+    "response_method", ["auto", "decision_function", "predict_proba"]
+)
+def test_cutoffclassifier_with_constraint_value(response_method):
+    """Check that `CutOffClassifier` is optimizing a given objective metric."""
+    X, y = load_breast_cancer(return_X_y=True)
+    # remove feature to degrade performances
+    X = X[:, :5]
+
+    # make the problem completely imbalanced such that the balanced accuracy is low
+    indices_pos = np.flatnonzero(y == 1)
+    indices_pos = indices_pos[: indices_pos.size // 50]
+    indices_neg = np.flatnonzero(y == 0)
+
+    X = np.vstack([X[indices_neg], X[indices_pos]])
+    y = np.hstack([y[indices_neg], y[indices_pos]])
+
+    lr = make_pipeline(StandardScaler(), LogisticRegression()).fit(X, y)
+    n_thresholds = 100
+    model = CutOffClassifier(
+        estimator=lr,
+        objective_metric="balanced_accuracy",
+        response_method=response_method,
+        n_thresholds=n_thresholds,
+    )
+    score_optimized = balanced_accuracy_score(y, model.fit(X, y).predict(X))
+    score_baseline = balanced_accuracy_score(y, lr.predict(X))
+    assert score_optimized > score_baseline
+    assert model.decision_thresholds_.shape == (n_thresholds,)
+    assert model.objective_scores_.shape == (n_thresholds,)
+
+
+@pytest.mark.parametrize(
+    "metrics",
+    [
+        ("max_tpr_at_tnr_constraint", "max_tnr_at_tpr_constraint"),
+        ("max_tnr_at_tpr_constraint", "max_tpr_at_tnr_constraint"),
+    ],
+)
+def test_cutoffclassifier_limit_metric_tradeoff(metrics):
+    """Check that an objective value of 0 give opposite predictions with tnr/tpr and
+    precision/recall.
+    """
+    X, y = load_breast_cancer(return_X_y=True)
+    estimator = make_pipeline(StandardScaler(), LogisticRegression())
+    model = CutOffClassifier(
+        estimator=estimator,
+        objective_metric=metrics[0],
+        constraint_value=0,
+    )
+    y_pred_1 = model.fit(X, y).predict(X)
+    model.set_params(objective_metric=metrics[1])
+    y_pred_2 = (~model.fit(X, y).predict(X).astype(bool)).astype(int)
+    assert np.mean(y_pred_1 == y_pred_2) > 0.98
+
+
+def test_cutoffclassifier_metric_with_parameter():
+    """Check that we can pass a metric with a parameter in addition check that
+    `f_beta with beta=1` is equivalent to `f1`.
+    """
+    X, y = load_breast_cancer(return_X_y=True)
+    lr = make_pipeline(StandardScaler(), LogisticRegression()).fit(X, y)
+    model_fbeta = CutOffClassifier(
+        estimator=lr, objective_metric=make_scorer(fbeta_score, beta=1)
+    ).fit(X, y)
+    model_f1 = CutOffClassifier(
+        estimator=lr, objective_metric=make_scorer(f1_score)
+    ).fit(X, y)
+
+    assert model_fbeta.decision_threshold_ == pytest.approx(
+        model_f1.decision_threshold_
+    )
+
+
+@pytest.mark.parametrize(
+    "response_method", ["auto", "decision_function", "predict_proba"]
+)
+@pytest.mark.parametrize(
+    "metric",
+    [
+        "max_tnr_at_tpr_constraint",
+        "max_tpr_at_tnr_constraint",
+        "max_precision_at_recall_constraint",
+        "max_recall_at_precision_constraint",
+        make_scorer(balanced_accuracy_score),
+        make_scorer(f1_score, pos_label="cancer"),
+        {"tp": 5, "tn": 1, "fp": -1, "fn": -1},
+    ],
+)
+def test_cutoffclassifier_with_string_targets(response_method, metric):
+    """Check that targets represented by str are properly managed.
+    Also, check with several metrics to be sure that `pos_label` is properly
+    dispatched.
+    """
+    X, y = load_breast_cancer(return_X_y=True)
+    # Encode numeric targets by meaningful strings. We purposely designed the class
+    # names such that the `pos_label` is the first alphabetically sorted class and thus
+    # encoded as 0.
+    classes = np.array(["cancer", "healthy"], dtype=object)
+    y = classes[y]
+    model = CutOffClassifier(
+        estimator=make_pipeline(StandardScaler(), LogisticRegression()),
+        objective_metric=metric,
+        constraint_value=0.9,
+        pos_label="cancer",
+        response_method=response_method,
+        n_thresholds=100,
+    ).fit(X, y)
+    assert_array_equal(model.classes_, np.sort(classes))
+    y_pred = model.predict(X)
+    assert_array_equal(np.sort(np.unique(y_pred)), np.sort(classes))
+
+
+@pytest.mark.parametrize("with_sample_weight", [True, False])
+def test_cutoffclassifier_refit(with_sample_weight, global_random_seed):
+    """Check the behaviour of the `refit` parameter."""
+    rng = np.random.RandomState(global_random_seed)
+    X, y = make_classification(n_samples=100, random_state=0)
+    if with_sample_weight:
+        sample_weight = rng.randn(X.shape[0])
+        sample_weight = np.abs(sample_weight, out=sample_weight)
+    else:
+        sample_weight = None
+
+    # check that `estimator_` if fitted on the full dataset when `refit=True`
+    estimator = LogisticRegression()
+    model = CutOffClassifier(estimator, refit=True).fit(
+        X, y, sample_weight=sample_weight
+    )
+
+    assert model.estimator_ is not estimator
+    estimator.fit(X, y, sample_weight=sample_weight)
+    assert_allclose(model.estimator_.coef_, estimator.coef_)
+    assert_allclose(model.estimator_.intercept_, estimator.intercept_)
+
+    # check that `estimator_` was not altered when `refit=False` and `cv="prefit"`
+    estimator = LogisticRegression().fit(X, y, sample_weight=sample_weight)
+    coef = estimator.coef_.copy()
+    model = CutOffClassifier(estimator, cv="prefit", refit=False).fit(
+        X, y, sample_weight=sample_weight
+    )
+
+    assert model.estimator_ is estimator
+    assert_allclose(model.estimator_.coef_, coef)
+
+    # check that we train `estimator_` on the training split of a given cross-validation
+    estimator = LogisticRegression()
+    cv = [
+        (np.arange(50), np.arange(50, 100)),
+    ]  # single split
+    model = CutOffClassifier(estimator, cv=cv, refit=False).fit(
+        X, y, sample_weight=sample_weight
+    )
+
+    assert model.estimator_ is not estimator
+    if with_sample_weight:
+        sw_train = sample_weight[cv[0][0]]
+    else:
+        sw_train = None
+    estimator.fit(X[cv[0][0]], y[cv[0][0]], sample_weight=sw_train)
+    assert_allclose(model.estimator_.coef_, estimator.coef_)
+
+
+@pytest.mark.parametrize(
+    "objective_metric",
+    [
+        "max_tnr_at_tpr_constraint",
+        "max_tpr_at_tnr_constraint",
+        "max_precision_at_recall_constraint",
+        "max_recall_at_precision_constraint",
+        "balanced_accuracy",
+    ],
+)
+@pytest.mark.parametrize("fit_params_type", ["list", "array"])
+def test_cutoffclassifier_fit_params(objective_metric, fit_params_type):
+    """Check that we pass `fit_params` to the classifier when calling `fit`."""
+    X, y = make_classification(n_samples=100, random_state=0)
+    fit_params = {
+        "a": _convert_container(y, fit_params_type),
+        "b": _convert_container(y, fit_params_type),
+    }
+
+    classifier = CheckingClassifier(expected_fit_params=["a", "b"], random_state=0)
+    model = CutOffClassifier(
+        classifier, objective_metric=objective_metric, constraint_value=0.5
+    )
+    model.fit(X, y, **fit_params)
+
+
+@pytest.mark.parametrize(
+    "objective_metric, constraint_value",
+    [
+        ("max_tnr_at_tpr_constraint", 0.5),
+        ("max_tpr_at_tnr_constraint", 0.5),
+        ("max_precision_at_recall_constraint", 0.5),
+        ("max_recall_at_precision_constraint", 0.5),
+    ],
+)
+@pytest.mark.parametrize(
+    "response_method", ["auto", "decision_function", "predict_proba"]
+)
+def test_cutoffclassifier_response_method_scorer_with_constraint_metric(
+    objective_metric, constraint_value, response_method, global_random_seed
+):
+    """Check that we use the proper scorer and forwarding the requested response method
+    for TNR/TPR and precision/recall metrics.
+    """
+    X, y = make_classification(n_samples=100, random_state=global_random_seed)
+    classifier = LogisticRegression()
+
+    n_thresholds = 100
+    model = CutOffClassifier(
+        classifier,
+        objective_metric=objective_metric,
+        constraint_value=constraint_value,
+        response_method=response_method,
+        n_thresholds=n_thresholds,
+    )
+    model.fit(X, y)
+    assert model.decision_thresholds_.shape == (n_thresholds,)
+    assert all(score.shape == (n_thresholds,) for score in model.objective_scores_)
+
+    if response_method in ("auto", "predict_proba"):
+        # "auto" will fall back  in priority on `predict_proba` if `estimator`
+        # supports it.
+        # we expect the decision threshold to be in [0, 1]
+        if objective_metric in (
+            "max_tnr_at_tpr_constraint",
+            "max_precision_at_recall_constraint",
+        ):
+            assert 0.5 <= model.decision_threshold_ <= 1
+        else:  # "max_tpr_at_tnr_constraint" or "max_recall_at_precision_constraint"
+            assert 0 <= model.decision_threshold_ <= 0.5
+    else:  # "decision_function"
+        # we expect the decision function to be centered in 0.0 and to be larger than
+        # -1 and 1.
+        if objective_metric in (
+            "max_tnr_at_tpr_constraint",
+            "max_precision_at_recall_constraint",
+        ):
+            assert 0 < model.decision_threshold_ < 20
+        else:  # "max_tpr_at_tnr_constraint" or "max_recall_at_precision_constraint"
+            assert -20 < model.decision_threshold_ < 0
+
+
+def test_cutoffclassifier_objective_metric_dict(global_random_seed):
+    """Check that we can pass a custom objective metric."""
+    X, y = make_classification(n_samples=500, random_state=global_random_seed)
+    classifier = LogisticRegression()
+
+    # we need to set a small number of thresholds to avoid ties and picking a too low
+    # threshold.
+    n_thresholds = 5
+
+    # affect a high gain to true negative and force the classifier to mainly
+    # predict the negative class.
+    costs_and_again = {"tp": 0, "tn": 10, "fp": 0, "fn": 0}
+    model = CutOffClassifier(
+        classifier, objective_metric=costs_and_again, n_thresholds=n_thresholds
+    )
+    model.fit(X, y)
+
+    assert model.decision_thresholds_.shape == (n_thresholds,)
+    assert model.objective_scores_.shape == (n_thresholds,)
+
+    assert model.decision_threshold_ > 0.99
+    assert np.mean(model.predict(X) == 0) > 0.9
+
+    # use the true positive now
+    costs_and_again = {"tp": 10, "tn": 0, "fp": 0, "fn": 0}
+    model = CutOffClassifier(
+        classifier, objective_metric=costs_and_again, n_thresholds=n_thresholds
+    )
+    model.fit(X, y)
+
+    assert model.decision_thresholds_.shape == (n_thresholds,)
+    assert model.objective_scores_.shape == (n_thresholds,)
+
+    assert model.decision_threshold_ < 0.01
+    assert np.mean(model.predict(X) == 1) > 0.9
+
+    # flipping the `pos_label` to zero should force the classifier to always predict 0
+    # and thus have a low threshold
+    pos_label = 0
+    model = CutOffClassifier(
+        classifier,
+        objective_metric=costs_and_again,
+        n_thresholds=n_thresholds,
+        pos_label=pos_label,
+    )
+    model.fit(X, y)
+
+    assert model.decision_thresholds_.shape == (n_thresholds,)
+    assert model.objective_scores_.shape == (n_thresholds,)
+
+    assert model.decision_threshold_ < 0.01
+    assert np.mean(model.predict(X) == 0) > 0.9
+
+
+def test_cutoffclassifier_sample_weight_costs_and_again():
+    """Check that we dispatch the `sample_weight` to the scorer when computing the
+    confusion matrix."""
+    X, y = load_iris(return_X_y=True)
+    X, y = X[:100], y[:100]  # only 2 classes
+
+    # create a dataset and repeat twice the sample of class #0
+    X_repeated, y_repeated = np.vstack([X, X[y == 0]]), np.hstack([y, y[y == 0]])
+    # create a sample weight vector that is equivalent to the repeated dataset
+    sample_weight = np.ones_like(y)
+    sample_weight[:50] *= 2
+
+    # we use a prefit classifier to simplify the test
+    cv = "prefit"
+    estimator = LogisticRegression().fit(X, y)
+    costs_and_again = {"tp": 1, "tn": 1, "fp": -1, "fn": -1}
+
+    model_repeat = CutOffClassifier(estimator, cv=cv, objective_metric=costs_and_again)
+    model_repeat.fit(X_repeated, y_repeated, sample_weight=None)
+
+    model_sw = CutOffClassifier(estimator, cv=cv, objective_metric=costs_and_again)
+    model_sw.fit(X, y, sample_weight=sample_weight)
+
+    assert model_repeat.objective_score_ == pytest.approx(model_sw.objective_score_)
+
+
+def test_cutoffclassifier_cv_zeros_sample_weights_equivalence():
+    """Check that passing removing some sample from the dataset `X` is
+    equivalent to passing a `sample_weight` with a factor 0."""
+    X, y = load_iris(return_X_y=True)
+    # Scale the data to avoid any convergence issue
+    X = StandardScaler().fit_transform(X)
+    # Only use 2 classes and select samples such that 2-fold cross-validation
+    # split will lead to an equivalence with a `sample_weight` of 0
+    X = np.vstack((X[:40], X[50:90]))
+    y = np.hstack((y[:40], y[50:90]))
+    sample_weight = np.zeros_like(y)
+    sample_weight[::2] = 1
+
+    estimator = LogisticRegression()
+    model_without_weights = CutOffClassifier(estimator, cv=2)
+    model_with_weights = clone(model_without_weights)
+
+    model_with_weights.fit(X, y, sample_weight=sample_weight)
+    model_without_weights.fit(X[::2], y[::2])
+
+    assert_allclose(
+        model_with_weights.estimator_.coef_, model_without_weights.estimator_.coef_
+    )
+
+    y_pred_with_weights = model_with_weights.predict_proba(X)
+    y_pred_without_weights = model_without_weights.predict_proba(X)
+    assert_allclose(y_pred_with_weights, y_pred_without_weights)
+
+
+def test_cutoffclassifier_error_constant_learner():
+    """Check that we raise an error message when providing an estimator that predicts
+    only a single class."""
+    X, y = make_classification(random_state=0)
+    estimator = DummyClassifier(strategy="constant", constant=1)
+    err_msg = "The provided estimator makes constant predictions."
+    with pytest.raises(ValueError, match=err_msg):
+        CutOffClassifier(estimator).fit(X, y)
+
+
+@pytest.mark.parametrize(
+    "objective_metric",
+    ["max_precision_at_recall_constraint", "max_recall_at_precision_constraint"],
+)
+@pytest.mark.parametrize("pos_label", [0, 1])
+def test_cutoffclassifier_pos_label_precision_recall(objective_metric, pos_label):
+    """Check that `pos_label` is dispatched correctly by checking the precision and
+    recall score found during the optimization and the one found at `predict` time."""
+    X, y = make_classification(n_samples=5_000, weights=[0.6, 0.4], random_state=42)
+
+    # prefit the estimator to avoid variability due to the cross-validation
+    estimator = LogisticRegression().fit(X, y)
+
+    constraint_value = 0.7
+    model = CutOffClassifier(
+        estimator,
+        objective_metric=objective_metric,
+        constraint_value=constraint_value,
+        cv="prefit",
+        pos_label=pos_label,
+    ).fit(X, y)
+
+    precision = precision_score(y, model.predict(X), pos_label=pos_label)
+    recall = recall_score(y, model.predict(X), pos_label=pos_label)
+
+    # due to internal interpolation, the scores will vary slightly
+    if objective_metric == "max_precision_at_recall_constraint":
+        assert recall == pytest.approx(model.objective_score_[0], abs=1e-3)
+        assert precision == pytest.approx(model.objective_score_[1], abs=1e-3)
+    else:
+        assert precision == pytest.approx(model.objective_score_[0], abs=1e-3)
+        assert recall == pytest.approx(model.objective_score_[1], abs=1e-3)
+
+
+@pytest.mark.parametrize(
+    "objective_metric", ["max_tnr_at_tpr_constraint", "max_tpr_at_tnr_constraint"]
+)
+@pytest.mark.parametrize("pos_label", [0, 1])
+def test_cutoffclassifier_pos_label_tnr_tpr(objective_metric, pos_label):
+    """Check that `pos_label` is dispatched correctly by checking the TNR and TPR
+    score found during the optimization and the one found at `predict` time."""
+    X, y = make_classification(n_samples=5_000, weights=[0.6, 0.4], random_state=42)
+
+    # prefit the estimator to avoid variability due to the cross-validation
+    estimator = LogisticRegression().fit(X, y)
+
+    constraint_value = 0.7
+    model = CutOffClassifier(
+        estimator,
+        objective_metric=objective_metric,
+        constraint_value=constraint_value,
+        cv="prefit",
+        pos_label=pos_label,
+    ).fit(X, y)
+
+    def tnr_tpr_score(y_true, y_pred, pos_label=pos_label):
+        cm = confusion_matrix(y_true, y_pred)
+        if pos_label == 0:
+            cm = cm[::-1, ::-1]
+        tn, fp, fn, tp = cm.ravel()
+        tnr = tn / (tn + fp)
+        tpr = tp / (tp + fn)
+        return tnr, tpr
+
+    tnr, tpr = tnr_tpr_score(y, model.predict(X), pos_label=pos_label)
+    # due to internal interpolation, the scores will vary slightly
+    if objective_metric == "max_tnr_at_tpr_constraint":
+        assert tpr == pytest.approx(model.objective_score_[0], abs=0.05)
+        assert tnr == pytest.approx(model.objective_score_[1], abs=0.05)
+    else:
+        assert tnr == pytest.approx(model.objective_score_[0], abs=0.05)
+        assert tpr == pytest.approx(model.objective_score_[1], abs=0.05)
+
+
+@pytest.mark.parametrize(
+    "metric_type",
+    ["string", "scorer_without_pos_label", "scorer_with_pos_label"],
+)
+@pytest.mark.parametrize("pos_label", [0, 1])
+def test_cutoffclassifier_pos_label_single_metric(pos_label, metric_type):
+    """Check that `pos_label` is dispatched correctly when getting a scorer linked to
+    a known metric. By default, the scorer in scikit-learn only have a default value
+    for `pos_label` which is 1.
+    """
+    X, y = make_classification(n_samples=100, weights=[0.6, 0.4], random_state=42)
+
+    # prefit the estimator to avoid variability due to the cross-validation
+    estimator = LogisticRegression().fit(X, y)
+
+    if metric_type == "string":
+        objective_metric = "precision"
+    elif metric_type == "scorer_without_pos_label":
+        objective_metric = make_scorer(precision_score)
+    else:  # metric_type == "scorer_with_pos_label"
+        objective_metric = make_scorer(precision_score, pos_label=pos_label)
+
+    model = CutOffClassifier(
+        estimator,
+        objective_metric=objective_metric,
+        cv="prefit",
+        pos_label=pos_label,
+        n_thresholds=500,
+    ).fit(X, y)
+
+    precision = precision_score(y, model.predict(X), pos_label=pos_label)
+    assert precision == pytest.approx(model.objective_score_, abs=1e-3)
diff --git a/sklearn/utils/_mocking.py b/sklearn/utils/_mocking.py
index 688bfb68ed484..7feb85568ae07 100644
--- a/sklearn/utils/_mocking.py
+++ b/sklearn/utils/_mocking.py
@@ -3,7 +3,7 @@
 from ..base import BaseEstimator, ClassifierMixin
 from .metaestimators import available_if
 from .validation import _check_sample_weight, _num_samples, check_array
-from .validation import check_is_fitted
+from .validation import check_is_fitted, check_random_state
 
 
 class ArraySlicingWrapper:
@@ -131,6 +131,7 @@ def __init__(
         foo_param=0,
         expected_sample_weight=None,
         expected_fit_params=None,
+        random_state=None,
     ):
         self.check_y = check_y
         self.check_y_params = check_y_params
@@ -140,6 +141,7 @@ def __init__(
         self.foo_param = foo_param
         self.expected_sample_weight = expected_sample_weight
         self.expected_fit_params = expected_fit_params
+        self.random_state = random_state
 
     def _check_X_y(self, X, y=None, should_be_fitted=True):
         """Validate X and y and make extra check.
@@ -239,7 +241,8 @@ def predict(self, X):
         """
         if self.methods_to_check == "all" or "predict" in self.methods_to_check:
             X, y = self._check_X_y(X)
-        return self.classes_[np.zeros(_num_samples(X), dtype=int)]
+        rng = check_random_state(self.random_state)
+        return rng.choice(self.classes_, size=_num_samples(X))
 
     def predict_proba(self, X):
         """Predict probabilities for each class.
@@ -259,8 +262,10 @@ def predict_proba(self, X):
         """
         if self.methods_to_check == "all" or "predict_proba" in self.methods_to_check:
             X, y = self._check_X_y(X)
-        proba = np.zeros((_num_samples(X), len(self.classes_)))
-        proba[:, 0] = 1
+        rng = check_random_state(self.random_state)
+        proba = rng.randn(_num_samples(X), len(self.classes_))
+        proba = np.abs(proba, out=proba)
+        proba /= np.sum(proba, axis=1)[:, np.newaxis]
         return proba
 
     def decision_function(self, X):
@@ -282,14 +287,13 @@ def decision_function(self, X):
             or "decision_function" in self.methods_to_check
         ):
             X, y = self._check_X_y(X)
+        rng = check_random_state(self.random_state)
         if len(self.classes_) == 2:
             # for binary classifier, the confidence score is related to
             # classes_[1] and therefore should be null.
-            return np.zeros(_num_samples(X))
+            return rng.randn(_num_samples(X))
         else:
-            decision = np.zeros((_num_samples(X), len(self.classes_)))
-            decision[:, 0] = 1
-            return decision
+            return rng.randn(_num_samples(X), len(self.classes_))
 
     def score(self, X=None, Y=None):
         """Fake score.
diff --git a/sklearn/utils/tests/test_mocking.py b/sklearn/utils/tests/test_mocking.py
index 718c62d5cc83b..b24dd13a3bbd7 100644
--- a/sklearn/utils/tests/test_mocking.py
+++ b/sklearn/utils/tests/test_mocking.py
@@ -3,7 +3,6 @@
 from scipy import sparse
 
 from numpy.testing import assert_array_equal
-from numpy.testing import assert_allclose
 
 from sklearn.datasets import load_iris
 from sklearn.utils import check_array
@@ -93,7 +92,7 @@ def test_checking_classifier(iris, input_type):
     assert clf.n_features_in_ == 4
 
     y_pred = clf.predict(X)
-    assert_array_equal(y_pred, np.zeros(y_pred.size, dtype=int))
+    assert all(pred in clf.classes_ for pred in y_pred)
 
     assert clf.score(X) == pytest.approx(0)
     clf.set_params(foo_param=10)
@@ -101,13 +100,10 @@ def test_checking_classifier(iris, input_type):
 
     y_proba = clf.predict_proba(X)
     assert y_proba.shape == (150, 3)
-    assert_allclose(y_proba[:, 0], 1)
-    assert_allclose(y_proba[:, 1:], 0)
+    assert np.logical_and(y_proba >= 0, y_proba <= 1).all()
 
     y_decision = clf.decision_function(X)
     assert y_decision.shape == (150, 3)
-    assert_allclose(y_decision[:, 0], 1)
-    assert_allclose(y_decision[:, 1:], 0)
 
     # check the shape in case of binary classification
     first_2_classes = np.logical_or(y == 0, y == 1)
@@ -117,12 +113,10 @@ def test_checking_classifier(iris, input_type):
 
     y_proba = clf.predict_proba(X)
     assert y_proba.shape == (100, 2)
-    assert_allclose(y_proba[:, 0], 1)
-    assert_allclose(y_proba[:, 1], 0)
+    assert np.logical_and(y_proba >= 0, y_proba <= 1).all()
 
     y_decision = clf.decision_function(X)
     assert y_decision.shape == (100,)
-    assert_allclose(y_decision, 0)
 
 
 def test_checking_classifier_with_params(iris):


Code review comments:
diff_hunk: @@ -363,6 +377,62 @@ def _factory_args(self):
         return ", needs_threshold=True"
 
 
+class _ContinuousScorer(_BaseScorer):
+    """Scorer taking a continuous response and output a score for each threshold."""
+
+    def __init__(self, score_func, sign, response_method, kwargs):
+        super().__init__(score_func=score_func, sign=sign, kwargs=kwargs)
+        self.response_method = response_method
+
+    def _score(self, method_caller, estimator, X, y_true, sample_weight=None):
+        """Evaluate predicted target values for X relative to y_true.
+
+        Parameters
+        ----------
+        estimator : object
+            Trained estimator to use for scoring.
+
+        X : {array-like, sparse matrix} of shape (n_samples, n_features)
+            Test data that will be fed to estimator.predict.
+
+        y_true : array-like of shape (n_samples,)
+            Gold standard target values for X.
+
+        sample_weight : array-like of shape (n_samples,), default=None
+            Sample weights.
+
+        Returns
+        -------
+        score : float
+            Score function applied to prediction of estimator on X.
+        """
+        pos_label = self._get_pos_label()
+        y_score = method_caller(estimator, self.response_method, X, pos_label=pos_label)
+
+        if sample_weight is not None:
+            score_func = partial(self._score_func, sample_weight=sample_weight)
+        else:
+            score_func = self._score_func
+
+        # TODO: this is pretty slow if we have a lot of `potential_thresholds`
path: sklearn/metrics/_scorer.py
text: @author:
Actually, the overhead here is due to the parameter validation.
Once we remove the nested parameter validation, we would need to have any parallelism.

diff_hunk: @@ -0,0 +1,566 @@
+"""
+==================================================
+Post-tuning the cut-off point of decision function
+==================================================
+
+Once a classifier is trained, the output of the :term:`predict` method output hard
+predictions corresponding to a thresholding of either the :term:`decision function`
+or the :term:`predict_proba` output. For binary classifier, the default threshold is
+defined as a probability score of 0.5 or a decision function value of 0.0.
+
+However, it happens that this default strategy is not optimized for the task at hand.
+Here, we use the "Statlog" German credit dataset [1]_ to illustrate such an application.
+In this dataset, the task is to predict whether a person is "good" or "bad" credit.
+In addition, cost-matrix is provided where it is defined that classifying a "bad"
+credit as "good" is 5 times more costly than classifying a "good" credit as "bad".
+
+We use the :class:`~sklearn.model_selection.CutOffClassifier` to tune the
+cut-off point of the decision function to minimize the business cost provided to us.
+
+.. topic:: References
+
+    .. [1] "Statlog (German Credit Data) Data Set", UCI Machine Learning Repository,
+       `Link <https://archive.ics.uci.edu/ml/datasets/Statlog+%28German+Credit+Data%29>`_.  # noqa
+"""
+
+# %%
+# "Statlog" German credit dataset
+# -------------------------------
+#
+# We fetch the German credit dataset from OpenML.
+import sklearn
+from sklearn.datasets import fetch_openml
+
+sklearn.set_config(transform_output="pandas")
+german_credit = fetch_openml(data_id=31, as_frame=True, parser="pandas")
+X, y = german_credit.data, german_credit.target
+
+# %%
+# We check the feature types available in `X`.
+X.info()
+
+# %%
+X.head()
+
+# %%
+# Many features are categorical and usually string-encoded. We need to encode
+# these categories when we develop ou predictive model. Let's check the targets.
+y.value_counts()
+
+# %%
+# A first observation is that the dataset is imbalanced. We would need to be careful
+# when evaluating our predictive model and use a family of metrics that are adapted
+# to this setting.
+#
+# In addition, we observe that the target is string-encoded. Some metrics
+# (e.g. precision and recall) require to provide the label of interest also called
+# the "positive label". Here, we define that our goal is to predict whether or not
+# a sample is a "bad" credit.
+pos_label = "bad"
+
+# %%
+# To carry our analysis, we split our dataset using a single stratified split.
+from sklearn.model_selection import train_test_split
+
+X_train, X_test, y_train, y_test = train_test_split(X, y, stratify=y, random_state=0)
+
+# %%
+# We are ready to design our predictive model and the associated evaluation strategy.
+#
+# Evaluation metrics
+# ------------------
+#
+# In this section, we define a set of metrics that we use later. To see
+# the effect of tuning the cut-off point, we evaluate the predictive model using
+# the Receiver Operating Characteristic (ROC) curve and the Precision-Recall curve.
+# The values reported on these plots are therefore the true positive rate (TPR) and
+# the false positive rate (FPR) for the ROC curve and the precision and recall for the
+# Precision-Recall curve.
+#
+# From these four metrics, scikit-learn does not provide a scorer for the FPR. We
+# therefore need to define a small custom function to compute it.
+import numpy as np
+from sklearn.metrics import confusion_matrix, make_scorer, precision_score, recall_score
+
+
+def fpr_score(y, y_pred, **kwargs):
+    cm = confusion_matrix(y, y_pred)
+    classes = np.unique(y)
+    pos_label = kwargs.get("pos_label", classes[-1])
+    pos_label_idx = np.searchsorted(classes, pos_label)
+    if pos_label_idx == 0:
+        cm = cm[::-1, ::-1]
+    tn, fp, _, _ = cm.ravel()
+    tnr = tn / (tn + fp)
+    return 1 - tnr
+
+
+# %%
+# As previously stated, the "positive label" is not defined as the value "1" and calling
+# some of the metrics with this non-standard value raise an error. We need to
+# provide the indication of the "positive label" to the metrics.
+#
+# We therefore need to define a scikit-learn scorer using
+# :func:`~sklearn.metrics.make_scorer` where the information is passed. We store all
+# created scorer in a dictionary. To use them, we need to pass the fitted model and
+# the data and target on which we want to evaluate the predictive model.
+tpr_score = recall_score  # TPR and recall are the same metric
+scoring = {
+    "precision": make_scorer(precision_score, pos_label=pos_label),
+    "recall": make_scorer(recall_score, pos_label=pos_label),
+    "fpr": make_scorer(fpr_score, pos_label=pos_label),
+    "tpr": make_scorer(tpr_score, pos_label=pos_label),
+}
+
+# %%
+# In addition, the original research [1]_ defines a business metric. They provide a
+# cost-matrix where they define that predicting a "bad" credit as "good" is 5 times more
+# costly than the opposite. We define a dictionary containing this information and a
+# score function that computes the cost.
+cost_gain_matrix = {"tp": 0, "tn": 0, "fp": -1, "fn": -5}
+
+
+def gain_cost_score(y, y_pred, **kwargs):
+    cm = confusion_matrix(y, y_pred)
+    classes = np.unique(y)
+    pos_label = kwargs.get("pos_label", classes[-1])
+    pos_label_idx = np.searchsorted(classes, pos_label)
+    if pos_label_idx == 0:
+        cm = cm[::-1, ::-1]
+    costs_and_gain = np.array(
+        [
+            [kwargs["cost_gain_matrix"]["tn"], kwargs["cost_gain_matrix"]["fp"]],
+            [kwargs["cost_gain_matrix"]["fn"], kwargs["cost_gain_matrix"]["tp"]],
+        ]
+    )
+    return (costs_and_gain * cm).sum()
+
+
+scoring["cost_gain"] = make_scorer(
+    gain_cost_score, pos_label=pos_label, cost_gain_matrix=cost_gain_matrix
+)
+# %%
+# Vanilla predictive model
+# ------------------------
+#
+# Design of the predictive model
+# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+#
+# In this section, we design our predictive model. We use a gradient boosting classifier
+# using :class:`~sklearn.ensemble.HistGradientBoostingClassifier`. However, we need to
+# encode the categorical features with numerical values and we therefore use an
+# :class:`~sklearn.preprocessing.OrdinalEncoder` to do so. The numerical features are
+# kept as-is. To recover the name of the categorical columns, we use the helper function
+# :func:`~sklearn.compose.make_column_selector` and the fact that the categorical
+# features are stored as `category` dtype.
+from sklearn.compose import ColumnTransformer, make_column_selector as selector
+from sklearn.ensemble import HistGradientBoostingClassifier
+from sklearn.pipeline import Pipeline
+from sklearn.preprocessing import OrdinalEncoder
+
+categorical_columns = selector(dtype_include="category")(X)
+numerical_columns = selector(dtype_exclude="category")(X)
+
+preprocessor = ColumnTransformer(
+    [
+        (
+            "categorical",
+            OrdinalEncoder(handle_unknown="use_encoded_value", unknown_value=-1),
+            categorical_columns,
+        ),
+        ("numerical", "passthrough", numerical_columns),
+    ],
+    verbose_feature_names_out=False,
+)
+model = Pipeline(
+    [
+        ("preprocessor", preprocessor),
+        (
+            "classifier",
+            HistGradientBoostingClassifier(
+                categorical_features=categorical_columns, random_state=0
+            ),
+        ),
+    ]
+)
+
+model.fit(X_train, y_train)
+
+# %%
+# Evaluation of the predictive model
+# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+#
+# We evaluate the performance of our predictive model using the ROC and Precision-Recall
+# curves.
+import matplotlib.pyplot as plt
+from sklearn.metrics import PrecisionRecallDisplay, RocCurveDisplay
+
+fig, axs = plt.subplots(nrows=1, ncols=2, figsize=(14, 6))
+
+PrecisionRecallDisplay.from_estimator(
+    model, X_test, y_test, pos_label=pos_label, ax=axs[0], name="GBDT"
+)
+axs[0].plot(
+    scoring["recall"](model, X_test, y_test),
+    scoring["precision"](model, X_test, y_test),
+    marker="o",
+    markersize=10,
+    color="tab:blue",
+    label="Default cut-off point at a probability of 0.5",
+)
+axs[0].set_title("Precision-Recall curve")
+axs[0].legend()
+
+RocCurveDisplay.from_estimator(
+    model,
+    X_test,
+    y_test,
+    pos_label=pos_label,
+    ax=axs[1],
+    name="GBDT",
+    plot_chance_level=True,
+)
+axs[1].plot(
+    scoring["fpr"](model, X_test, y_test),
+    scoring["tpr"](model, X_test, y_test),
+    marker="o",
+    markersize=10,
+    color="tab:blue",
+    label="Default cut-off point at a probability of 0.5",
+)
+axs[1].set_title("ROC curve")
+axs[1].legend()
+_ = fig.suptitle("Evaluation of the vanilla GBDT model")
+
+# %%
+# We recall that these curves corresponds show the statistical performance of the
+# predictive model for different cut-off points. For the Precision-Recall curve, the
+# reported metrics are the precision and recall and for the ROC curve, the reported
+# metrics are the TPR (same as recall) and FPR.
+#
+# Here, the different cut-off points corresponds at different level of probability
+# scores ranging between 0 and 1. By default, `model.predict` uses a cut-off point at
+# a probability of 0.5. Thus, the metrics for this cut-off point is reported with the
+# blue dot on the curves: it corresponds to the statistical performance of the model
+# when using `model.predict`.
+#
+# However, we recall that the original aim was to minimize the cost (or maximize the
+# gain) by the business metric. We can compute the value of the business metric:
+scoring["cost_gain"](model, X_test, y_test)
+
+# %%
+# At this stage, we don't know if some other cut-off points could lead to a greater
+# gain. To be able to decide, if another cut-off point is better, we would need to
+# compute the cost-gain using the business metric for all possible cut-off points and
+# choose the optimal one. This strategy is quite tedious to implement and the
+# :class:`~sklearn.metrics.CutOffClassifier` class is here to help us. It automatically
+# compute the cost-gain for all possible cut-off points and choose the optimal one.
+#
+# .. _cost_sensitive_learning_example:
+#
+# Tuning the cut-off point
+# ------------------------
+#
+# We use :class:`~sklearn.metrics.CutOffClassifier` to tune the cut-off point. We need
+# to provide the business metric to optimize as well as the positive label. Internally,
+# the optimum cut-off point is chosen such that it maximized the business metric
+# via cross-validation. By default a 5-fold stratified cross-validation is used.
+from sklearn.model_selection import CutOffClassifier
+
+model_tuned = CutOffClassifier(
+    estimator=model,
+    pos_label=pos_label,
+    objective_metric=cost_gain_matrix,
+)
+model_tuned.fit(X_train, y_train)
+
+# %%
+# We plot the ROC and Precision-Recall curves for the vanilla model and the tuned model.
+# Also we plot the cut-off points that would be used by each model.
+fig, axs = plt.subplots(nrows=1, ncols=3, figsize=(21, 6))
+
+linestyles = ("dashed", "dotted")
+markerstyles = ("o", ">")
+colors = ("tab:blue", "tab:orange")
+names = ("Vanilla GBDT", "Tuned GBDT")
+for idx, (est, linestyle, marker, color, name) in enumerate(
+    zip((model, model_tuned), linestyles, markerstyles, colors, names)
+):
+    decision_threshold = getattr(est, "decision_threshold_", 0.5)
+    PrecisionRecallDisplay.from_estimator(
+        est,
+        X_test,
+        y_test,
+        pos_label=pos_label,
+        linestyle=linestyle,
+        color=color,
+        ax=axs[0],
+        name=name,
+    )
+    axs[0].plot(
+        scoring["recall"](est, X_test, y_test),
+        scoring["precision"](est, X_test, y_test),
+        marker,
+        markersize=10,
+        color=color,
+        label=f"Cut-off point at probability of {decision_threshold:.2f}",
+    )
+    RocCurveDisplay.from_estimator(
+        est,
+        X_test,
+        y_test,
+        pos_label=pos_label,
+        linestyle=linestyle,
+        color=color,
+        ax=axs[1],
+        name=name,
+        plot_chance_level=idx == 1,
+    )
+    axs[1].plot(
+        scoring["fpr"](est, X_test, y_test),
+        scoring["tpr"](est, X_test, y_test),
+        marker,
+        markersize=10,
+        color=color,
+        label=f"Cut-off point at probability of {decision_threshold:.2f}",
+    )
+
+axs[0].set_title("Precision-Recall curve")
+axs[0].legend()
+axs[1].set_title("ROC curve")
+axs[1].legend()
+
+axs[2].plot(
+    model_tuned.decision_thresholds_, model_tuned.objective_scores_, color="tab:orange"
+)
+axs[2].plot(
+    model_tuned.decision_threshold_,
+    model_tuned.objective_score_,
+    "o",
+    markersize=10,
+    color="tab:orange",
path: examples/model_selection/plot_cutoff_tuning.py
text: @user1:
Maybe here we can plot the point we had for the default cut-off so we can use the same `markerstyles` and colors in the 3 plots.

diff_hunk: @@ -0,0 +1,566 @@
+"""
+==================================================
+Post-tuning the cut-off point of decision function
+==================================================
+
+Once a classifier is trained, the output of the :term:`predict` method output hard
+predictions corresponding to a thresholding of either the :term:`decision function`
+or the :term:`predict_proba` output. For binary classifier, the default threshold is
+defined as a probability score of 0.5 or a decision function value of 0.0.
+
+However, it happens that this default strategy is not optimized for the task at hand.
+Here, we use the "Statlog" German credit dataset [1]_ to illustrate such an application.
+In this dataset, the task is to predict whether a person is "good" or "bad" credit.
+In addition, cost-matrix is provided where it is defined that classifying a "bad"
+credit as "good" is 5 times more costly than classifying a "good" credit as "bad".
+
+We use the :class:`~sklearn.model_selection.CutOffClassifier` to tune the
+cut-off point of the decision function to minimize the business cost provided to us.
+
+.. topic:: References
+
+    .. [1] "Statlog (German Credit Data) Data Set", UCI Machine Learning Repository,
+       `Link <https://archive.ics.uci.edu/ml/datasets/Statlog+%28German+Credit+Data%29>`_.  # noqa
+"""
+
+# %%
+# "Statlog" German credit dataset
+# -------------------------------
+#
+# We fetch the German credit dataset from OpenML.
+import sklearn
+from sklearn.datasets import fetch_openml
+
+sklearn.set_config(transform_output="pandas")
+german_credit = fetch_openml(data_id=31, as_frame=True, parser="pandas")
+X, y = german_credit.data, german_credit.target
+
+# %%
+# We check the feature types available in `X`.
+X.info()
+
+# %%
+X.head()
+
+# %%
+# Many features are categorical and usually string-encoded. We need to encode
+# these categories when we develop ou predictive model. Let's check the targets.
+y.value_counts()
+
+# %%
+# A first observation is that the dataset is imbalanced. We would need to be careful
+# when evaluating our predictive model and use a family of metrics that are adapted
+# to this setting.
+#
+# In addition, we observe that the target is string-encoded. Some metrics
+# (e.g. precision and recall) require to provide the label of interest also called
+# the "positive label". Here, we define that our goal is to predict whether or not
+# a sample is a "bad" credit.
+pos_label = "bad"
+
+# %%
+# To carry our analysis, we split our dataset using a single stratified split.
+from sklearn.model_selection import train_test_split
+
+X_train, X_test, y_train, y_test = train_test_split(X, y, stratify=y, random_state=0)
+
+# %%
+# We are ready to design our predictive model and the associated evaluation strategy.
+#
+# Evaluation metrics
+# ------------------
+#
+# In this section, we define a set of metrics that we use later. To see
+# the effect of tuning the cut-off point, we evaluate the predictive model using
+# the Receiver Operating Characteristic (ROC) curve and the Precision-Recall curve.
+# The values reported on these plots are therefore the true positive rate (TPR) and
+# the false positive rate (FPR) for the ROC curve and the precision and recall for the
+# Precision-Recall curve.
+#
+# From these four metrics, scikit-learn does not provide a scorer for the FPR. We
+# therefore need to define a small custom function to compute it.
+import numpy as np
+from sklearn.metrics import confusion_matrix, make_scorer, precision_score, recall_score
+
+
+def fpr_score(y, y_pred, **kwargs):
+    cm = confusion_matrix(y, y_pred)
+    classes = np.unique(y)
+    pos_label = kwargs.get("pos_label", classes[-1])
+    pos_label_idx = np.searchsorted(classes, pos_label)
+    if pos_label_idx == 0:
+        cm = cm[::-1, ::-1]
+    tn, fp, _, _ = cm.ravel()
+    tnr = tn / (tn + fp)
+    return 1 - tnr
+
+
+# %%
+# As previously stated, the "positive label" is not defined as the value "1" and calling
+# some of the metrics with this non-standard value raise an error. We need to
+# provide the indication of the "positive label" to the metrics.
+#
+# We therefore need to define a scikit-learn scorer using
+# :func:`~sklearn.metrics.make_scorer` where the information is passed. We store all
+# created scorer in a dictionary. To use them, we need to pass the fitted model and
+# the data and target on which we want to evaluate the predictive model.
+tpr_score = recall_score  # TPR and recall are the same metric
+scoring = {
+    "precision": make_scorer(precision_score, pos_label=pos_label),
+    "recall": make_scorer(recall_score, pos_label=pos_label),
+    "fpr": make_scorer(fpr_score, pos_label=pos_label),
+    "tpr": make_scorer(tpr_score, pos_label=pos_label),
+}
+
+# %%
+# In addition, the original research [1]_ defines a business metric. They provide a
+# cost-matrix where they define that predicting a "bad" credit as "good" is 5 times more
+# costly than the opposite. We define a dictionary containing this information and a
+# score function that computes the cost.
+cost_gain_matrix = {"tp": 0, "tn": 0, "fp": -1, "fn": -5}
+
+
+def gain_cost_score(y, y_pred, **kwargs):
+    cm = confusion_matrix(y, y_pred)
+    classes = np.unique(y)
+    pos_label = kwargs.get("pos_label", classes[-1])
+    pos_label_idx = np.searchsorted(classes, pos_label)
+    if pos_label_idx == 0:
+        cm = cm[::-1, ::-1]
+    costs_and_gain = np.array(
+        [
+            [kwargs["cost_gain_matrix"]["tn"], kwargs["cost_gain_matrix"]["fp"]],
+            [kwargs["cost_gain_matrix"]["fn"], kwargs["cost_gain_matrix"]["tp"]],
+        ]
+    )
+    return (costs_and_gain * cm).sum()
+
+
+scoring["cost_gain"] = make_scorer(
+    gain_cost_score, pos_label=pos_label, cost_gain_matrix=cost_gain_matrix
+)
+# %%
+# Vanilla predictive model
+# ------------------------
+#
+# Design of the predictive model
+# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+#
+# In this section, we design our predictive model. We use a gradient boosting classifier
+# using :class:`~sklearn.ensemble.HistGradientBoostingClassifier`. However, we need to
+# encode the categorical features with numerical values and we therefore use an
+# :class:`~sklearn.preprocessing.OrdinalEncoder` to do so. The numerical features are
+# kept as-is. To recover the name of the categorical columns, we use the helper function
+# :func:`~sklearn.compose.make_column_selector` and the fact that the categorical
+# features are stored as `category` dtype.
+from sklearn.compose import ColumnTransformer, make_column_selector as selector
+from sklearn.ensemble import HistGradientBoostingClassifier
+from sklearn.pipeline import Pipeline
+from sklearn.preprocessing import OrdinalEncoder
+
+categorical_columns = selector(dtype_include="category")(X)
+numerical_columns = selector(dtype_exclude="category")(X)
+
+preprocessor = ColumnTransformer(
+    [
+        (
+            "categorical",
+            OrdinalEncoder(handle_unknown="use_encoded_value", unknown_value=-1),
+            categorical_columns,
+        ),
+        ("numerical", "passthrough", numerical_columns),
+    ],
+    verbose_feature_names_out=False,
+)
+model = Pipeline(
+    [
+        ("preprocessor", preprocessor),
+        (
+            "classifier",
+            HistGradientBoostingClassifier(
+                categorical_features=categorical_columns, random_state=0
+            ),
+        ),
+    ]
+)
+
+model.fit(X_train, y_train)
+
+# %%
+# Evaluation of the predictive model
+# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+#
+# We evaluate the performance of our predictive model using the ROC and Precision-Recall
+# curves.
+import matplotlib.pyplot as plt
+from sklearn.metrics import PrecisionRecallDisplay, RocCurveDisplay
+
+fig, axs = plt.subplots(nrows=1, ncols=2, figsize=(14, 6))
+
+PrecisionRecallDisplay.from_estimator(
+    model, X_test, y_test, pos_label=pos_label, ax=axs[0], name="GBDT"
+)
+axs[0].plot(
+    scoring["recall"](model, X_test, y_test),
+    scoring["precision"](model, X_test, y_test),
+    marker="o",
+    markersize=10,
+    color="tab:blue",
+    label="Default cut-off point at a probability of 0.5",
+)
+axs[0].set_title("Precision-Recall curve")
+axs[0].legend()
+
+RocCurveDisplay.from_estimator(
+    model,
+    X_test,
+    y_test,
+    pos_label=pos_label,
+    ax=axs[1],
+    name="GBDT",
+    plot_chance_level=True,
+)
+axs[1].plot(
+    scoring["fpr"](model, X_test, y_test),
+    scoring["tpr"](model, X_test, y_test),
+    marker="o",
+    markersize=10,
+    color="tab:blue",
+    label="Default cut-off point at a probability of 0.5",
+)
+axs[1].set_title("ROC curve")
+axs[1].legend()
+_ = fig.suptitle("Evaluation of the vanilla GBDT model")
+
+# %%
+# We recall that these curves corresponds show the statistical performance of the
+# predictive model for different cut-off points. For the Precision-Recall curve, the
+# reported metrics are the precision and recall and for the ROC curve, the reported
+# metrics are the TPR (same as recall) and FPR.
+#
+# Here, the different cut-off points corresponds at different level of probability
+# scores ranging between 0 and 1. By default, `model.predict` uses a cut-off point at
+# a probability of 0.5. Thus, the metrics for this cut-off point is reported with the
+# blue dot on the curves: it corresponds to the statistical performance of the model
+# when using `model.predict`.
+#
+# However, we recall that the original aim was to minimize the cost (or maximize the
+# gain) by the business metric. We can compute the value of the business metric:
+scoring["cost_gain"](model, X_test, y_test)
+
+# %%
+# At this stage, we don't know if some other cut-off points could lead to a greater
+# gain. To be able to decide, if another cut-off point is better, we would need to
+# compute the cost-gain using the business metric for all possible cut-off points and
+# choose the optimal one. This strategy is quite tedious to implement and the
+# :class:`~sklearn.metrics.CutOffClassifier` class is here to help us. It automatically
+# compute the cost-gain for all possible cut-off points and choose the optimal one.
+#
+# .. _cost_sensitive_learning_example:
+#
+# Tuning the cut-off point
+# ------------------------
+#
+# We use :class:`~sklearn.metrics.CutOffClassifier` to tune the cut-off point. We need
+# to provide the business metric to optimize as well as the positive label. Internally,
+# the optimum cut-off point is chosen such that it maximized the business metric
+# via cross-validation. By default a 5-fold stratified cross-validation is used.
+from sklearn.model_selection import CutOffClassifier
+
+model_tuned = CutOffClassifier(
+    estimator=model,
+    pos_label=pos_label,
+    objective_metric=cost_gain_matrix,
+)
+model_tuned.fit(X_train, y_train)
+
+# %%
+# We plot the ROC and Precision-Recall curves for the vanilla model and the tuned model.
+# Also we plot the cut-off points that would be used by each model.
+fig, axs = plt.subplots(nrows=1, ncols=3, figsize=(21, 6))
+
+linestyles = ("dashed", "dotted")
+markerstyles = ("o", ">")
+colors = ("tab:blue", "tab:orange")
+names = ("Vanilla GBDT", "Tuned GBDT")
+for idx, (est, linestyle, marker, color, name) in enumerate(
+    zip((model, model_tuned), linestyles, markerstyles, colors, names)
+):
+    decision_threshold = getattr(est, "decision_threshold_", 0.5)
+    PrecisionRecallDisplay.from_estimator(
+        est,
+        X_test,
+        y_test,
+        pos_label=pos_label,
+        linestyle=linestyle,
+        color=color,
+        ax=axs[0],
+        name=name,
+    )
+    axs[0].plot(
+        scoring["recall"](est, X_test, y_test),
+        scoring["precision"](est, X_test, y_test),
+        marker,
+        markersize=10,
+        color=color,
+        label=f"Cut-off point at probability of {decision_threshold:.2f}",
+    )
+    RocCurveDisplay.from_estimator(
+        est,
+        X_test,
+        y_test,
+        pos_label=pos_label,
+        linestyle=linestyle,
+        color=color,
+        ax=axs[1],
+        name=name,
+        plot_chance_level=idx == 1,
+    )
+    axs[1].plot(
+        scoring["fpr"](est, X_test, y_test),
+        scoring["tpr"](est, X_test, y_test),
+        marker,
+        markersize=10,
+        color=color,
+        label=f"Cut-off point at probability of {decision_threshold:.2f}",
+    )
+
+axs[0].set_title("Precision-Recall curve")
+axs[0].legend()
+axs[1].set_title("ROC curve")
+axs[1].legend()
+
+axs[2].plot(
+    model_tuned.decision_thresholds_, model_tuned.objective_scores_, color="tab:orange"
+)
+axs[2].plot(
+    model_tuned.decision_threshold_,
+    model_tuned.objective_score_,
+    "o",
+    markersize=10,
+    color="tab:orange",
+    label="Optimal cut-off point for the business metric",
+)
+axs[2].legend()
+axs[2].set_xlabel("Decision threshold (probability)")
+axs[2].set_ylabel("Objective score (using cost-matrix)")
+axs[2].set_title("Objective score as a function of the decision threshold")
+
+_ = fig.suptitle("Comparison of the cut-off point for the vanilla and tuned GBDT model")
+
+# %%
+# The first remark is that both classifiers have exactly the same ROC and
+# Precision-Recall curves. It is expected because by default, the classifier is fitted
+# on the same training data. In a later section, we discuss more in details the
+# available options regarding model refitting and cross-validation.
+#
+# The second remark is that the cut-off points of the vanilla and tuned model are
+# different. To understand why the tuned model has chosen this cut-off point, we can
+# look at the right-hand side plot that plots the objective score that is our exactly
+# the same as our business metric. We see that the optimum threshold corresponds to the
+# maximum of the objective score.
+#
+# We can now check if choosing this cut-off point leads to a better score on the testing
+# set:
+scoring["cost_gain"](model_tuned, X_test, y_test)
+
+# %%
+# We observe that the decision generalized on the testing set leading to a better
+# business score.
+#
+# .. _cutoffclassifier_no_cv:
+#
+# Consideration regarding model refitting and cross-validation
+# ------------------------------------------------------------
+#
+# In the above experiment, we use the default parameter of the
+# :class:`~sklearn.model_selection.CutOffClassifier`. By default, the cut-off point is
+# tuned using a 5-fold stratified cross-validation. Also, the underlying predictive
+# model is refitted on the entire training data once the cut-off point is chosen.
path: examples/model_selection/plot_cutoff_tuning.py
text: @user1:
```suggestion
# In the above experiment, we use the default setting of the
# :class:`~sklearn.model_selection.CutOffClassifier`. In particular, the cut-off
# point is tuned using a 5-fold stratified cross-validation. Also, the
# underlying predictive model is refitted on the entire training data once the
# cut-off point is chosen.
```

diff_hunk: @@ -0,0 +1,566 @@
+"""
+==================================================
+Post-tuning the cut-off point of decision function
+==================================================
+
+Once a classifier is trained, the output of the :term:`predict` method output hard
+predictions corresponding to a thresholding of either the :term:`decision function`
+or the :term:`predict_proba` output. For binary classifier, the default threshold is
+defined as a probability score of 0.5 or a decision function value of 0.0.
+
+However, it happens that this default strategy is not optimized for the task at hand.
+Here, we use the "Statlog" German credit dataset [1]_ to illustrate such an application.
+In this dataset, the task is to predict whether a person is "good" or "bad" credit.
+In addition, cost-matrix is provided where it is defined that classifying a "bad"
+credit as "good" is 5 times more costly than classifying a "good" credit as "bad".
+
+We use the :class:`~sklearn.model_selection.CutOffClassifier` to tune the
+cut-off point of the decision function to minimize the business cost provided to us.
+
+.. topic:: References
+
+    .. [1] "Statlog (German Credit Data) Data Set", UCI Machine Learning Repository,
+       `Link <https://archive.ics.uci.edu/ml/datasets/Statlog+%28German+Credit+Data%29>`_.  # noqa
+"""
+
+# %%
+# "Statlog" German credit dataset
+# -------------------------------
+#
+# We fetch the German credit dataset from OpenML.
+import sklearn
+from sklearn.datasets import fetch_openml
+
+sklearn.set_config(transform_output="pandas")
+german_credit = fetch_openml(data_id=31, as_frame=True, parser="pandas")
+X, y = german_credit.data, german_credit.target
+
+# %%
+# We check the feature types available in `X`.
+X.info()
+
+# %%
+X.head()
+
+# %%
+# Many features are categorical and usually string-encoded. We need to encode
+# these categories when we develop ou predictive model. Let's check the targets.
+y.value_counts()
+
+# %%
+# A first observation is that the dataset is imbalanced. We would need to be careful
+# when evaluating our predictive model and use a family of metrics that are adapted
+# to this setting.
+#
+# In addition, we observe that the target is string-encoded. Some metrics
+# (e.g. precision and recall) require to provide the label of interest also called
+# the "positive label". Here, we define that our goal is to predict whether or not
+# a sample is a "bad" credit.
+pos_label = "bad"
+
+# %%
+# To carry our analysis, we split our dataset using a single stratified split.
+from sklearn.model_selection import train_test_split
+
+X_train, X_test, y_train, y_test = train_test_split(X, y, stratify=y, random_state=0)
+
+# %%
+# We are ready to design our predictive model and the associated evaluation strategy.
+#
+# Evaluation metrics
+# ------------------
+#
+# In this section, we define a set of metrics that we use later. To see
+# the effect of tuning the cut-off point, we evaluate the predictive model using
+# the Receiver Operating Characteristic (ROC) curve and the Precision-Recall curve.
+# The values reported on these plots are therefore the true positive rate (TPR) and
+# the false positive rate (FPR) for the ROC curve and the precision and recall for the
+# Precision-Recall curve.
+#
+# From these four metrics, scikit-learn does not provide a scorer for the FPR. We
+# therefore need to define a small custom function to compute it.
+import numpy as np
+from sklearn.metrics import confusion_matrix, make_scorer, precision_score, recall_score
+
+
+def fpr_score(y, y_pred, **kwargs):
+    cm = confusion_matrix(y, y_pred)
+    classes = np.unique(y)
+    pos_label = kwargs.get("pos_label", classes[-1])
+    pos_label_idx = np.searchsorted(classes, pos_label)
+    if pos_label_idx == 0:
+        cm = cm[::-1, ::-1]
+    tn, fp, _, _ = cm.ravel()
+    tnr = tn / (tn + fp)
+    return 1 - tnr
+
+
+# %%
+# As previously stated, the "positive label" is not defined as the value "1" and calling
+# some of the metrics with this non-standard value raise an error. We need to
+# provide the indication of the "positive label" to the metrics.
+#
+# We therefore need to define a scikit-learn scorer using
+# :func:`~sklearn.metrics.make_scorer` where the information is passed. We store all
+# created scorer in a dictionary. To use them, we need to pass the fitted model and
+# the data and target on which we want to evaluate the predictive model.
+tpr_score = recall_score  # TPR and recall are the same metric
+scoring = {
+    "precision": make_scorer(precision_score, pos_label=pos_label),
+    "recall": make_scorer(recall_score, pos_label=pos_label),
+    "fpr": make_scorer(fpr_score, pos_label=pos_label),
+    "tpr": make_scorer(tpr_score, pos_label=pos_label),
+}
+
+# %%
+# In addition, the original research [1]_ defines a business metric. They provide a
+# cost-matrix where they define that predicting a "bad" credit as "good" is 5 times more
+# costly than the opposite. We define a dictionary containing this information and a
+# score function that computes the cost.
+cost_gain_matrix = {"tp": 0, "tn": 0, "fp": -1, "fn": -5}
+
+
+def gain_cost_score(y, y_pred, **kwargs):
+    cm = confusion_matrix(y, y_pred)
+    classes = np.unique(y)
+    pos_label = kwargs.get("pos_label", classes[-1])
+    pos_label_idx = np.searchsorted(classes, pos_label)
+    if pos_label_idx == 0:
+        cm = cm[::-1, ::-1]
+    costs_and_gain = np.array(
+        [
+            [kwargs["cost_gain_matrix"]["tn"], kwargs["cost_gain_matrix"]["fp"]],
+            [kwargs["cost_gain_matrix"]["fn"], kwargs["cost_gain_matrix"]["tp"]],
+        ]
+    )
+    return (costs_and_gain * cm).sum()
+
+
+scoring["cost_gain"] = make_scorer(
+    gain_cost_score, pos_label=pos_label, cost_gain_matrix=cost_gain_matrix
+)
+# %%
+# Vanilla predictive model
+# ------------------------
+#
+# Design of the predictive model
+# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+#
+# In this section, we design our predictive model. We use a gradient boosting classifier
+# using :class:`~sklearn.ensemble.HistGradientBoostingClassifier`. However, we need to
+# encode the categorical features with numerical values and we therefore use an
+# :class:`~sklearn.preprocessing.OrdinalEncoder` to do so. The numerical features are
+# kept as-is. To recover the name of the categorical columns, we use the helper function
+# :func:`~sklearn.compose.make_column_selector` and the fact that the categorical
+# features are stored as `category` dtype.
+from sklearn.compose import ColumnTransformer, make_column_selector as selector
+from sklearn.ensemble import HistGradientBoostingClassifier
+from sklearn.pipeline import Pipeline
+from sklearn.preprocessing import OrdinalEncoder
+
+categorical_columns = selector(dtype_include="category")(X)
+numerical_columns = selector(dtype_exclude="category")(X)
+
+preprocessor = ColumnTransformer(
+    [
+        (
+            "categorical",
+            OrdinalEncoder(handle_unknown="use_encoded_value", unknown_value=-1),
+            categorical_columns,
+        ),
+        ("numerical", "passthrough", numerical_columns),
+    ],
+    verbose_feature_names_out=False,
+)
+model = Pipeline(
+    [
+        ("preprocessor", preprocessor),
+        (
+            "classifier",
+            HistGradientBoostingClassifier(
+                categorical_features=categorical_columns, random_state=0
+            ),
+        ),
+    ]
+)
+
+model.fit(X_train, y_train)
+
+# %%
+# Evaluation of the predictive model
+# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+#
+# We evaluate the performance of our predictive model using the ROC and Precision-Recall
+# curves.
+import matplotlib.pyplot as plt
+from sklearn.metrics import PrecisionRecallDisplay, RocCurveDisplay
+
+fig, axs = plt.subplots(nrows=1, ncols=2, figsize=(14, 6))
+
+PrecisionRecallDisplay.from_estimator(
+    model, X_test, y_test, pos_label=pos_label, ax=axs[0], name="GBDT"
+)
+axs[0].plot(
+    scoring["recall"](model, X_test, y_test),
+    scoring["precision"](model, X_test, y_test),
+    marker="o",
+    markersize=10,
+    color="tab:blue",
+    label="Default cut-off point at a probability of 0.5",
+)
+axs[0].set_title("Precision-Recall curve")
+axs[0].legend()
+
+RocCurveDisplay.from_estimator(
+    model,
+    X_test,
+    y_test,
+    pos_label=pos_label,
+    ax=axs[1],
+    name="GBDT",
+    plot_chance_level=True,
+)
+axs[1].plot(
+    scoring["fpr"](model, X_test, y_test),
+    scoring["tpr"](model, X_test, y_test),
+    marker="o",
+    markersize=10,
+    color="tab:blue",
+    label="Default cut-off point at a probability of 0.5",
+)
+axs[1].set_title("ROC curve")
+axs[1].legend()
+_ = fig.suptitle("Evaluation of the vanilla GBDT model")
+
+# %%
+# We recall that these curves corresponds show the statistical performance of the
+# predictive model for different cut-off points. For the Precision-Recall curve, the
+# reported metrics are the precision and recall and for the ROC curve, the reported
+# metrics are the TPR (same as recall) and FPR.
+#
+# Here, the different cut-off points corresponds at different level of probability
+# scores ranging between 0 and 1. By default, `model.predict` uses a cut-off point at
+# a probability of 0.5. Thus, the metrics for this cut-off point is reported with the
+# blue dot on the curves: it corresponds to the statistical performance of the model
+# when using `model.predict`.
+#
+# However, we recall that the original aim was to minimize the cost (or maximize the
+# gain) by the business metric. We can compute the value of the business metric:
+scoring["cost_gain"](model, X_test, y_test)
+
+# %%
+# At this stage, we don't know if some other cut-off points could lead to a greater
+# gain. To be able to decide, if another cut-off point is better, we would need to
+# compute the cost-gain using the business metric for all possible cut-off points and
+# choose the optimal one. This strategy is quite tedious to implement and the
+# :class:`~sklearn.metrics.CutOffClassifier` class is here to help us. It automatically
+# compute the cost-gain for all possible cut-off points and choose the optimal one.
+#
+# .. _cost_sensitive_learning_example:
+#
+# Tuning the cut-off point
+# ------------------------
+#
+# We use :class:`~sklearn.metrics.CutOffClassifier` to tune the cut-off point. We need
+# to provide the business metric to optimize as well as the positive label. Internally,
+# the optimum cut-off point is chosen such that it maximized the business metric
+# via cross-validation. By default a 5-fold stratified cross-validation is used.
+from sklearn.model_selection import CutOffClassifier
+
+model_tuned = CutOffClassifier(
+    estimator=model,
+    pos_label=pos_label,
+    objective_metric=cost_gain_matrix,
+)
+model_tuned.fit(X_train, y_train)
+
+# %%
+# We plot the ROC and Precision-Recall curves for the vanilla model and the tuned model.
+# Also we plot the cut-off points that would be used by each model.
+fig, axs = plt.subplots(nrows=1, ncols=3, figsize=(21, 6))
+
+linestyles = ("dashed", "dotted")
+markerstyles = ("o", ">")
+colors = ("tab:blue", "tab:orange")
+names = ("Vanilla GBDT", "Tuned GBDT")
+for idx, (est, linestyle, marker, color, name) in enumerate(
+    zip((model, model_tuned), linestyles, markerstyles, colors, names)
+):
+    decision_threshold = getattr(est, "decision_threshold_", 0.5)
+    PrecisionRecallDisplay.from_estimator(
+        est,
+        X_test,
+        y_test,
+        pos_label=pos_label,
+        linestyle=linestyle,
+        color=color,
+        ax=axs[0],
+        name=name,
+    )
+    axs[0].plot(
+        scoring["recall"](est, X_test, y_test),
+        scoring["precision"](est, X_test, y_test),
+        marker,
+        markersize=10,
+        color=color,
+        label=f"Cut-off point at probability of {decision_threshold:.2f}",
+    )
+    RocCurveDisplay.from_estimator(
+        est,
+        X_test,
+        y_test,
+        pos_label=pos_label,
+        linestyle=linestyle,
+        color=color,
+        ax=axs[1],
+        name=name,
+        plot_chance_level=idx == 1,
+    )
+    axs[1].plot(
+        scoring["fpr"](est, X_test, y_test),
+        scoring["tpr"](est, X_test, y_test),
+        marker,
+        markersize=10,
+        color=color,
+        label=f"Cut-off point at probability of {decision_threshold:.2f}",
+    )
+
+axs[0].set_title("Precision-Recall curve")
+axs[0].legend()
+axs[1].set_title("ROC curve")
+axs[1].legend()
+
+axs[2].plot(
+    model_tuned.decision_thresholds_, model_tuned.objective_scores_, color="tab:orange"
+)
+axs[2].plot(
+    model_tuned.decision_threshold_,
+    model_tuned.objective_score_,
+    "o",
+    markersize=10,
+    color="tab:orange",
+    label="Optimal cut-off point for the business metric",
+)
+axs[2].legend()
+axs[2].set_xlabel("Decision threshold (probability)")
+axs[2].set_ylabel("Objective score (using cost-matrix)")
+axs[2].set_title("Objective score as a function of the decision threshold")
+
+_ = fig.suptitle("Comparison of the cut-off point for the vanilla and tuned GBDT model")
+
+# %%
+# The first remark is that both classifiers have exactly the same ROC and
+# Precision-Recall curves. It is expected because by default, the classifier is fitted
+# on the same training data. In a later section, we discuss more in details the
+# available options regarding model refitting and cross-validation.
+#
+# The second remark is that the cut-off points of the vanilla and tuned model are
+# different. To understand why the tuned model has chosen this cut-off point, we can
+# look at the right-hand side plot that plots the objective score that is our exactly
+# the same as our business metric. We see that the optimum threshold corresponds to the
+# maximum of the objective score.
+#
+# We can now check if choosing this cut-off point leads to a better score on the testing
+# set:
+scoring["cost_gain"](model_tuned, X_test, y_test)
+
+# %%
+# We observe that the decision generalized on the testing set leading to a better
+# business score.
+#
+# .. _cutoffclassifier_no_cv:
+#
+# Consideration regarding model refitting and cross-validation
+# ------------------------------------------------------------
+#
+# In the above experiment, we use the default parameter of the
+# :class:`~sklearn.model_selection.CutOffClassifier`. By default, the cut-off point is
+# tuned using a 5-fold stratified cross-validation. Also, the underlying predictive
+# model is refitted on the entire training data once the cut-off point is chosen.
+#
+# These two strategies can be changed by providing the `refit` and `cv` parameters.
+# For instance, one could provide a fitted `estimator` and set `cv="prefit"`, in which
+# case the cut-off point is found on the entire dataset provided at fitting time.
+# Also, the underlying classifier is not be refitted. Here, we can try to do such
+# experiment.
+model.fit(X_train, y_train)
+model_tuned.set_params(cv="prefit").fit(X_train, y_train)
+
+
+# %%
+# Then, we evaluate our model with the same approach as before:
+fig, axs = plt.subplots(nrows=1, ncols=3, figsize=(21, 6))
+
+linestyles = ("dashed", "dotted")
+markerstyles = ("o", ">")
+colors = ("tab:blue", "tab:orange")
+names = ("Vanilla GBDT", "Tuned GBDT")
+for idx, (est, linestyle, marker, color, name) in enumerate(
+    zip((model, model_tuned), linestyles, markerstyles, colors, names)
+):
+    decision_threshold = getattr(est, "decision_threshold_", 0.5)
+    PrecisionRecallDisplay.from_estimator(
+        est,
+        X_test,
+        y_test,
+        pos_label=pos_label,
+        linestyle=linestyle,
+        color=color,
+        ax=axs[0],
+        name=name,
+    )
+    axs[0].plot(
+        scoring["recall"](est, X_test, y_test),
+        scoring["precision"](est, X_test, y_test),
+        marker,
+        markersize=10,
+        color=color,
+        label=f"Cut-off point at probability of {decision_threshold:.2f}",
+    )
+    RocCurveDisplay.from_estimator(
+        est,
+        X_test,
+        y_test,
+        pos_label=pos_label,
+        linestyle=linestyle,
+        color=color,
+        ax=axs[1],
+        name=name,
+        plot_chance_level=idx == 1,
+    )
+    axs[1].plot(
+        scoring["fpr"](est, X_test, y_test),
+        scoring["tpr"](est, X_test, y_test),
+        marker,
+        markersize=10,
+        color=color,
+        label=f"Cut-off point at probability of {decision_threshold:.2f}",
+    )
+
+axs[0].set_title("Precision-Recall curve")
+axs[0].legend()
+axs[1].set_title("ROC curve")
+axs[1].legend()
+
+axs[2].plot(
+    model_tuned.decision_thresholds_, model_tuned.objective_scores_, color="tab:orange"
+)
+axs[2].plot(
+    model_tuned.decision_threshold_,
+    model_tuned.objective_score_,
+    "o",
+    markersize=10,
+    color="tab:orange",
+    label="Optimal cut-off point for the business metric",
+)
+axs[2].legend()
+axs[2].set_xlabel("Decision threshold (probability)")
+axs[2].set_ylabel("Objective score (using cost-matrix)")
+axs[2].set_title("Objective score as a function of the decision threshold")
+
+_ = fig.suptitle("Tuned GBDT model without refitting and using the entire dataset")
+
+# %%
+# We observe the that the optimum cut-off point is different than in the previous
+# experiment. If we look at the right-hand side plot, we observe that the objective
+# score has large plateau with a minimum cost (around 0). This behavior is symptomatic
+# of an overfitting. Because we disable cross-validation, we tuned the cut-off point on
+# the same set as the model was trained on, and this is the reason for the observed
+# overfitting.
+#
+# This option should therefore be used with caution. One needs to make sure that the
+# data providing at fitting time to the
+# :class:`~sklearn.model_selection.CutOffClassifier` is not the same as the data used to
+# train the underlying classifier. This could happen sometimes when the idea is just
+# to tune the predictive model on a completely new validation set without a costly
+# complete refit.
+#
+# In the case that cross-validation is too costly, a potential alternative is to use
+# a single train-test split by providing a floating number in range `[0, 1]` to the
+# `cv` parameter. It splits the data into a training and testing set. Let's
+# explore this option:
+model_tuned.set_params(cv=0.75).fit(X_train, y_train)
+
+# %%
+fig, axs = plt.subplots(nrows=1, ncols=3, figsize=(21, 6))
+
+linestyles = ("dashed", "dotted")
+markerstyles = ("o", ">")
+colors = ("tab:blue", "tab:orange")
+names = ("Vanilla GBDT", "Tuned GBDT")
+for idx, (est, linestyle, marker, color, name) in enumerate(
+    zip((model, model_tuned), linestyles, markerstyles, colors, names)
+):
+    decision_threshold = getattr(est, "decision_threshold_", 0.5)
+    PrecisionRecallDisplay.from_estimator(
+        est,
+        X_test,
+        y_test,
+        pos_label=pos_label,
+        linestyle=linestyle,
+        color=color,
+        ax=axs[0],
+        name=name,
+    )
+    axs[0].plot(
+        scoring["recall"](est, X_test, y_test),
+        scoring["precision"](est, X_test, y_test),
+        marker,
+        markersize=10,
+        color=color,
+        label=f"Cut-off point at probability of {decision_threshold:.2f}",
+    )
+    RocCurveDisplay.from_estimator(
+        est,
+        X_test,
+        y_test,
+        pos_label=pos_label,
+        linestyle=linestyle,
+        color=color,
+        ax=axs[1],
+        name=name,
+        plot_chance_level=idx == 1,
+    )
+    axs[1].plot(
+        scoring["fpr"](est, X_test, y_test),
+        scoring["tpr"](est, X_test, y_test),
+        marker,
+        markersize=10,
+        color=color,
+        label=f"Cut-off point at probability of {decision_threshold:.2f}",
+    )
+
+axs[0].set_title("Precision-Recall curve")
+axs[0].legend()
+axs[1].set_title("ROC curve")
+axs[1].legend()
+
+axs[2].plot(
+    model_tuned.decision_thresholds_, model_tuned.objective_scores_, color="tab:orange"
+)
+axs[2].plot(
+    model_tuned.decision_threshold_,
+    model_tuned.objective_score_,
+    "o",
+    markersize=10,
+    color="tab:orange",
+    label="Optimal cut-off point for the business metric",
+)
+axs[2].legend()
+axs[2].set_xlabel("Decision threshold (probability)")
+axs[2].set_ylabel("Objective score (using cost-matrix)")
+axs[2].set_title("Objective score as a function of the decision threshold")
+
+_ = fig.suptitle("Tuned GBDT model without refitting and using the entire dataset")
+
+# %%
+# Regarding the cut-off point, we observe that the optimum is similar to the multiple
+# repeated cross-validation case. However, be aware that a single split does not account
+# for the variability of the fit/predict process and thus we are enable to know if there
path: examples/model_selection/plot_cutoff_tuning.py
text: @user1:
```suggestion
# for the variability of the fit/predict process and thus we are unable to know if there
```

diff_hunk: @@ -0,0 +1,566 @@
+"""
+==================================================
+Post-tuning the cut-off point of decision function
+==================================================
+
+Once a classifier is trained, the output of the :term:`predict` method output hard
+predictions corresponding to a thresholding of either the :term:`decision function`
+or the :term:`predict_proba` output. For binary classifier, the default threshold is
+defined as a probability score of 0.5 or a decision function value of 0.0.
+
+However, it happens that this default strategy is not optimized for the task at hand.
+Here, we use the "Statlog" German credit dataset [1]_ to illustrate such an application.
+In this dataset, the task is to predict whether a person is "good" or "bad" credit.
+In addition, cost-matrix is provided where it is defined that classifying a "bad"
+credit as "good" is 5 times more costly than classifying a "good" credit as "bad".
+
+We use the :class:`~sklearn.model_selection.CutOffClassifier` to tune the
+cut-off point of the decision function to minimize the business cost provided to us.
+
+.. topic:: References
+
+    .. [1] "Statlog (German Credit Data) Data Set", UCI Machine Learning Repository,
+       `Link <https://archive.ics.uci.edu/ml/datasets/Statlog+%28German+Credit+Data%29>`_.  # noqa
+"""
+
+# %%
+# "Statlog" German credit dataset
+# -------------------------------
+#
+# We fetch the German credit dataset from OpenML.
+import sklearn
+from sklearn.datasets import fetch_openml
+
+sklearn.set_config(transform_output="pandas")
+german_credit = fetch_openml(data_id=31, as_frame=True, parser="pandas")
+X, y = german_credit.data, german_credit.target
+
+# %%
+# We check the feature types available in `X`.
+X.info()
+
+# %%
+X.head()
+
+# %%
+# Many features are categorical and usually string-encoded. We need to encode
+# these categories when we develop ou predictive model. Let's check the targets.
+y.value_counts()
+
+# %%
+# A first observation is that the dataset is imbalanced. We would need to be careful
+# when evaluating our predictive model and use a family of metrics that are adapted
+# to this setting.
+#
+# In addition, we observe that the target is string-encoded. Some metrics
+# (e.g. precision and recall) require to provide the label of interest also called
+# the "positive label". Here, we define that our goal is to predict whether or not
+# a sample is a "bad" credit.
+pos_label = "bad"
+
+# %%
+# To carry our analysis, we split our dataset using a single stratified split.
+from sklearn.model_selection import train_test_split
+
+X_train, X_test, y_train, y_test = train_test_split(X, y, stratify=y, random_state=0)
+
+# %%
+# We are ready to design our predictive model and the associated evaluation strategy.
+#
+# Evaluation metrics
+# ------------------
+#
+# In this section, we define a set of metrics that we use later. To see
+# the effect of tuning the cut-off point, we evaluate the predictive model using
+# the Receiver Operating Characteristic (ROC) curve and the Precision-Recall curve.
+# The values reported on these plots are therefore the true positive rate (TPR) and
+# the false positive rate (FPR) for the ROC curve and the precision and recall for the
+# Precision-Recall curve.
+#
+# From these four metrics, scikit-learn does not provide a scorer for the FPR. We
+# therefore need to define a small custom function to compute it.
+import numpy as np
+from sklearn.metrics import confusion_matrix, make_scorer, precision_score, recall_score
+
+
+def fpr_score(y, y_pred, **kwargs):
+    cm = confusion_matrix(y, y_pred)
+    classes = np.unique(y)
+    pos_label = kwargs.get("pos_label", classes[-1])
+    pos_label_idx = np.searchsorted(classes, pos_label)
+    if pos_label_idx == 0:
+        cm = cm[::-1, ::-1]
+    tn, fp, _, _ = cm.ravel()
+    tnr = tn / (tn + fp)
+    return 1 - tnr
+
+
+# %%
+# As previously stated, the "positive label" is not defined as the value "1" and calling
+# some of the metrics with this non-standard value raise an error. We need to
+# provide the indication of the "positive label" to the metrics.
+#
+# We therefore need to define a scikit-learn scorer using
+# :func:`~sklearn.metrics.make_scorer` where the information is passed. We store all
+# created scorer in a dictionary. To use them, we need to pass the fitted model and
+# the data and target on which we want to evaluate the predictive model.
+tpr_score = recall_score  # TPR and recall are the same metric
+scoring = {
+    "precision": make_scorer(precision_score, pos_label=pos_label),
+    "recall": make_scorer(recall_score, pos_label=pos_label),
+    "fpr": make_scorer(fpr_score, pos_label=pos_label),
+    "tpr": make_scorer(tpr_score, pos_label=pos_label),
+}
+
+# %%
+# In addition, the original research [1]_ defines a business metric. They provide a
+# cost-matrix where they define that predicting a "bad" credit as "good" is 5 times more
+# costly than the opposite. We define a dictionary containing this information and a
+# score function that computes the cost.
+cost_gain_matrix = {"tp": 0, "tn": 0, "fp": -1, "fn": -5}
+
+
+def gain_cost_score(y, y_pred, **kwargs):
+    cm = confusion_matrix(y, y_pred)
+    classes = np.unique(y)
+    pos_label = kwargs.get("pos_label", classes[-1])
+    pos_label_idx = np.searchsorted(classes, pos_label)
+    if pos_label_idx == 0:
+        cm = cm[::-1, ::-1]
+    costs_and_gain = np.array(
+        [
+            [kwargs["cost_gain_matrix"]["tn"], kwargs["cost_gain_matrix"]["fp"]],
+            [kwargs["cost_gain_matrix"]["fn"], kwargs["cost_gain_matrix"]["tp"]],
+        ]
+    )
+    return (costs_and_gain * cm).sum()
+
+
+scoring["cost_gain"] = make_scorer(
+    gain_cost_score, pos_label=pos_label, cost_gain_matrix=cost_gain_matrix
+)
+# %%
+# Vanilla predictive model
+# ------------------------
+#
+# Design of the predictive model
+# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+#
+# In this section, we design our predictive model. We use a gradient boosting classifier
+# using :class:`~sklearn.ensemble.HistGradientBoostingClassifier`. However, we need to
+# encode the categorical features with numerical values and we therefore use an
+# :class:`~sklearn.preprocessing.OrdinalEncoder` to do so. The numerical features are
+# kept as-is. To recover the name of the categorical columns, we use the helper function
+# :func:`~sklearn.compose.make_column_selector` and the fact that the categorical
+# features are stored as `category` dtype.
+from sklearn.compose import ColumnTransformer, make_column_selector as selector
+from sklearn.ensemble import HistGradientBoostingClassifier
+from sklearn.pipeline import Pipeline
+from sklearn.preprocessing import OrdinalEncoder
+
+categorical_columns = selector(dtype_include="category")(X)
+numerical_columns = selector(dtype_exclude="category")(X)
+
+preprocessor = ColumnTransformer(
+    [
+        (
+            "categorical",
+            OrdinalEncoder(handle_unknown="use_encoded_value", unknown_value=-1),
+            categorical_columns,
+        ),
+        ("numerical", "passthrough", numerical_columns),
+    ],
+    verbose_feature_names_out=False,
+)
+model = Pipeline(
+    [
+        ("preprocessor", preprocessor),
+        (
+            "classifier",
+            HistGradientBoostingClassifier(
+                categorical_features=categorical_columns, random_state=0
+            ),
+        ),
+    ]
+)
+
+model.fit(X_train, y_train)
+
+# %%
+# Evaluation of the predictive model
+# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+#
+# We evaluate the performance of our predictive model using the ROC and Precision-Recall
+# curves.
+import matplotlib.pyplot as plt
+from sklearn.metrics import PrecisionRecallDisplay, RocCurveDisplay
+
+fig, axs = plt.subplots(nrows=1, ncols=2, figsize=(14, 6))
+
+PrecisionRecallDisplay.from_estimator(
+    model, X_test, y_test, pos_label=pos_label, ax=axs[0], name="GBDT"
+)
+axs[0].plot(
+    scoring["recall"](model, X_test, y_test),
+    scoring["precision"](model, X_test, y_test),
+    marker="o",
+    markersize=10,
+    color="tab:blue",
+    label="Default cut-off point at a probability of 0.5",
+)
+axs[0].set_title("Precision-Recall curve")
+axs[0].legend()
+
+RocCurveDisplay.from_estimator(
+    model,
+    X_test,
+    y_test,
+    pos_label=pos_label,
+    ax=axs[1],
+    name="GBDT",
+    plot_chance_level=True,
+)
+axs[1].plot(
+    scoring["fpr"](model, X_test, y_test),
+    scoring["tpr"](model, X_test, y_test),
+    marker="o",
+    markersize=10,
+    color="tab:blue",
+    label="Default cut-off point at a probability of 0.5",
+)
+axs[1].set_title("ROC curve")
+axs[1].legend()
+_ = fig.suptitle("Evaluation of the vanilla GBDT model")
+
+# %%
+# We recall that these curves corresponds show the statistical performance of the
+# predictive model for different cut-off points. For the Precision-Recall curve, the
+# reported metrics are the precision and recall and for the ROC curve, the reported
+# metrics are the TPR (same as recall) and FPR.
+#
+# Here, the different cut-off points corresponds at different level of probability
+# scores ranging between 0 and 1. By default, `model.predict` uses a cut-off point at
+# a probability of 0.5. Thus, the metrics for this cut-off point is reported with the
+# blue dot on the curves: it corresponds to the statistical performance of the model
+# when using `model.predict`.
+#
+# However, we recall that the original aim was to minimize the cost (or maximize the
+# gain) by the business metric. We can compute the value of the business metric:
+scoring["cost_gain"](model, X_test, y_test)
+
+# %%
+# At this stage, we don't know if some other cut-off points could lead to a greater
+# gain. To be able to decide, if another cut-off point is better, we would need to
+# compute the cost-gain using the business metric for all possible cut-off points and
+# choose the optimal one. This strategy is quite tedious to implement and the
+# :class:`~sklearn.metrics.CutOffClassifier` class is here to help us. It automatically
+# compute the cost-gain for all possible cut-off points and choose the optimal one.
+#
+# .. _cost_sensitive_learning_example:
+#
+# Tuning the cut-off point
+# ------------------------
+#
+# We use :class:`~sklearn.metrics.CutOffClassifier` to tune the cut-off point. We need
+# to provide the business metric to optimize as well as the positive label. Internally,
+# the optimum cut-off point is chosen such that it maximized the business metric
+# via cross-validation. By default a 5-fold stratified cross-validation is used.
+from sklearn.model_selection import CutOffClassifier
+
+model_tuned = CutOffClassifier(
+    estimator=model,
+    pos_label=pos_label,
+    objective_metric=cost_gain_matrix,
+)
+model_tuned.fit(X_train, y_train)
+
+# %%
+# We plot the ROC and Precision-Recall curves for the vanilla model and the tuned model.
+# Also we plot the cut-off points that would be used by each model.
+fig, axs = plt.subplots(nrows=1, ncols=3, figsize=(21, 6))
+
+linestyles = ("dashed", "dotted")
+markerstyles = ("o", ">")
+colors = ("tab:blue", "tab:orange")
+names = ("Vanilla GBDT", "Tuned GBDT")
+for idx, (est, linestyle, marker, color, name) in enumerate(
+    zip((model, model_tuned), linestyles, markerstyles, colors, names)
+):
+    decision_threshold = getattr(est, "decision_threshold_", 0.5)
+    PrecisionRecallDisplay.from_estimator(
+        est,
+        X_test,
+        y_test,
+        pos_label=pos_label,
+        linestyle=linestyle,
+        color=color,
+        ax=axs[0],
+        name=name,
+    )
+    axs[0].plot(
+        scoring["recall"](est, X_test, y_test),
+        scoring["precision"](est, X_test, y_test),
+        marker,
+        markersize=10,
+        color=color,
+        label=f"Cut-off point at probability of {decision_threshold:.2f}",
+    )
+    RocCurveDisplay.from_estimator(
+        est,
+        X_test,
+        y_test,
+        pos_label=pos_label,
+        linestyle=linestyle,
+        color=color,
+        ax=axs[1],
+        name=name,
+        plot_chance_level=idx == 1,
+    )
+    axs[1].plot(
+        scoring["fpr"](est, X_test, y_test),
+        scoring["tpr"](est, X_test, y_test),
+        marker,
+        markersize=10,
+        color=color,
+        label=f"Cut-off point at probability of {decision_threshold:.2f}",
+    )
+
+axs[0].set_title("Precision-Recall curve")
+axs[0].legend()
+axs[1].set_title("ROC curve")
+axs[1].legend()
+
+axs[2].plot(
+    model_tuned.decision_thresholds_, model_tuned.objective_scores_, color="tab:orange"
+)
+axs[2].plot(
+    model_tuned.decision_threshold_,
+    model_tuned.objective_score_,
+    "o",
+    markersize=10,
+    color="tab:orange",
+    label="Optimal cut-off point for the business metric",
+)
+axs[2].legend()
+axs[2].set_xlabel("Decision threshold (probability)")
+axs[2].set_ylabel("Objective score (using cost-matrix)")
+axs[2].set_title("Objective score as a function of the decision threshold")
+
+_ = fig.suptitle("Comparison of the cut-off point for the vanilla and tuned GBDT model")
+
+# %%
+# The first remark is that both classifiers have exactly the same ROC and
+# Precision-Recall curves. It is expected because by default, the classifier is fitted
+# on the same training data. In a later section, we discuss more in details the
+# available options regarding model refitting and cross-validation.
+#
+# The second remark is that the cut-off points of the vanilla and tuned model are
+# different. To understand why the tuned model has chosen this cut-off point, we can
+# look at the right-hand side plot that plots the objective score that is our exactly
+# the same as our business metric. We see that the optimum threshold corresponds to the
+# maximum of the objective score.
+#
+# We can now check if choosing this cut-off point leads to a better score on the testing
+# set:
+scoring["cost_gain"](model_tuned, X_test, y_test)
+
+# %%
+# We observe that the decision generalized on the testing set leading to a better
+# business score.
+#
+# .. _cutoffclassifier_no_cv:
+#
+# Consideration regarding model refitting and cross-validation
+# ------------------------------------------------------------
+#
+# In the above experiment, we use the default parameter of the
+# :class:`~sklearn.model_selection.CutOffClassifier`. By default, the cut-off point is
+# tuned using a 5-fold stratified cross-validation. Also, the underlying predictive
+# model is refitted on the entire training data once the cut-off point is chosen.
+#
+# These two strategies can be changed by providing the `refit` and `cv` parameters.
+# For instance, one could provide a fitted `estimator` and set `cv="prefit"`, in which
+# case the cut-off point is found on the entire dataset provided at fitting time.
+# Also, the underlying classifier is not be refitted. Here, we can try to do such
+# experiment.
+model.fit(X_train, y_train)
+model_tuned.set_params(cv="prefit").fit(X_train, y_train)
+
+
+# %%
+# Then, we evaluate our model with the same approach as before:
+fig, axs = plt.subplots(nrows=1, ncols=3, figsize=(21, 6))
+
+linestyles = ("dashed", "dotted")
+markerstyles = ("o", ">")
+colors = ("tab:blue", "tab:orange")
+names = ("Vanilla GBDT", "Tuned GBDT")
+for idx, (est, linestyle, marker, color, name) in enumerate(
+    zip((model, model_tuned), linestyles, markerstyles, colors, names)
+):
+    decision_threshold = getattr(est, "decision_threshold_", 0.5)
+    PrecisionRecallDisplay.from_estimator(
+        est,
+        X_test,
+        y_test,
+        pos_label=pos_label,
+        linestyle=linestyle,
+        color=color,
+        ax=axs[0],
+        name=name,
+    )
+    axs[0].plot(
+        scoring["recall"](est, X_test, y_test),
+        scoring["precision"](est, X_test, y_test),
+        marker,
+        markersize=10,
+        color=color,
+        label=f"Cut-off point at probability of {decision_threshold:.2f}",
+    )
+    RocCurveDisplay.from_estimator(
+        est,
+        X_test,
+        y_test,
+        pos_label=pos_label,
+        linestyle=linestyle,
+        color=color,
+        ax=axs[1],
+        name=name,
+        plot_chance_level=idx == 1,
+    )
+    axs[1].plot(
+        scoring["fpr"](est, X_test, y_test),
+        scoring["tpr"](est, X_test, y_test),
+        marker,
+        markersize=10,
+        color=color,
+        label=f"Cut-off point at probability of {decision_threshold:.2f}",
+    )
+
+axs[0].set_title("Precision-Recall curve")
+axs[0].legend()
+axs[1].set_title("ROC curve")
+axs[1].legend()
+
+axs[2].plot(
+    model_tuned.decision_thresholds_, model_tuned.objective_scores_, color="tab:orange"
+)
+axs[2].plot(
+    model_tuned.decision_threshold_,
+    model_tuned.objective_score_,
+    "o",
+    markersize=10,
+    color="tab:orange",
+    label="Optimal cut-off point for the business metric",
+)
+axs[2].legend()
+axs[2].set_xlabel("Decision threshold (probability)")
+axs[2].set_ylabel("Objective score (using cost-matrix)")
+axs[2].set_title("Objective score as a function of the decision threshold")
+
+_ = fig.suptitle("Tuned GBDT model without refitting and using the entire dataset")
+
+# %%
+# We observe the that the optimum cut-off point is different than in the previous
+# experiment. If we look at the right-hand side plot, we observe that the objective
+# score has large plateau with a minimum cost (around 0). This behavior is symptomatic
+# of an overfitting. Because we disable cross-validation, we tuned the cut-off point on
+# the same set as the model was trained on, and this is the reason for the observed
+# overfitting.
+#
+# This option should therefore be used with caution. One needs to make sure that the
+# data providing at fitting time to the
+# :class:`~sklearn.model_selection.CutOffClassifier` is not the same as the data used to
+# train the underlying classifier. This could happen sometimes when the idea is just
+# to tune the predictive model on a completely new validation set without a costly
+# complete refit.
+#
+# In the case that cross-validation is too costly, a potential alternative is to use
+# a single train-test split by providing a floating number in range `[0, 1]` to the
+# `cv` parameter. It splits the data into a training and testing set. Let's
+# explore this option:
+model_tuned.set_params(cv=0.75).fit(X_train, y_train)
+
+# %%
+fig, axs = plt.subplots(nrows=1, ncols=3, figsize=(21, 6))
+
+linestyles = ("dashed", "dotted")
+markerstyles = ("o", ">")
+colors = ("tab:blue", "tab:orange")
+names = ("Vanilla GBDT", "Tuned GBDT")
+for idx, (est, linestyle, marker, color, name) in enumerate(
+    zip((model, model_tuned), linestyles, markerstyles, colors, names)
+):
+    decision_threshold = getattr(est, "decision_threshold_", 0.5)
+    PrecisionRecallDisplay.from_estimator(
+        est,
+        X_test,
+        y_test,
+        pos_label=pos_label,
+        linestyle=linestyle,
+        color=color,
+        ax=axs[0],
+        name=name,
+    )
+    axs[0].plot(
+        scoring["recall"](est, X_test, y_test),
+        scoring["precision"](est, X_test, y_test),
+        marker,
+        markersize=10,
+        color=color,
+        label=f"Cut-off point at probability of {decision_threshold:.2f}",
+    )
+    RocCurveDisplay.from_estimator(
+        est,
+        X_test,
+        y_test,
+        pos_label=pos_label,
+        linestyle=linestyle,
+        color=color,
+        ax=axs[1],
+        name=name,
+        plot_chance_level=idx == 1,
+    )
+    axs[1].plot(
+        scoring["fpr"](est, X_test, y_test),
+        scoring["tpr"](est, X_test, y_test),
+        marker,
+        markersize=10,
+        color=color,
+        label=f"Cut-off point at probability of {decision_threshold:.2f}",
+    )
+
+axs[0].set_title("Precision-Recall curve")
+axs[0].legend()
+axs[1].set_title("ROC curve")
+axs[1].legend()
+
+axs[2].plot(
+    model_tuned.decision_thresholds_, model_tuned.objective_scores_, color="tab:orange"
+)
+axs[2].plot(
+    model_tuned.decision_threshold_,
+    model_tuned.objective_score_,
+    "o",
+    markersize=10,
+    color="tab:orange",
+    label="Optimal cut-off point for the business metric",
+)
+axs[2].legend()
+axs[2].set_xlabel("Decision threshold (probability)")
+axs[2].set_ylabel("Objective score (using cost-matrix)")
+axs[2].set_title("Objective score as a function of the decision threshold")
+
+_ = fig.suptitle("Tuned GBDT model without refitting and using the entire dataset")
+
+# %%
+# Regarding the cut-off point, we observe that the optimum is similar to the multiple
+# repeated cross-validation case. However, be aware that a single split does not account
+# for the variability of the fit/predict process and thus we are enable to know if there
+# is any variance in the cut-off point. The repeated cross-validation averages this
+# effect.
path: examples/model_selection/plot_cutoff_tuning.py
text: @user1:
```suggestion
# is any variance in the cut-off point. The repeated cross-validation averages out
# this effect.
```

diff_hunk: @@ -0,0 +1,566 @@
+"""
+==================================================
+Post-tuning the cut-off point of decision function
+==================================================
+
+Once a classifier is trained, the output of the :term:`predict` method output hard
+predictions corresponding to a thresholding of either the :term:`decision function`
+or the :term:`predict_proba` output. For binary classifier, the default threshold is
+defined as a probability score of 0.5 or a decision function value of 0.0.
+
+However, it happens that this default strategy is not optimized for the task at hand.
+Here, we use the "Statlog" German credit dataset [1]_ to illustrate such an application.
+In this dataset, the task is to predict whether a person is "good" or "bad" credit.
+In addition, cost-matrix is provided where it is defined that classifying a "bad"
+credit as "good" is 5 times more costly than classifying a "good" credit as "bad".
+
+We use the :class:`~sklearn.model_selection.CutOffClassifier` to tune the
+cut-off point of the decision function to minimize the business cost provided to us.
+
+.. topic:: References
+
+    .. [1] "Statlog (German Credit Data) Data Set", UCI Machine Learning Repository,
+       `Link <https://archive.ics.uci.edu/ml/datasets/Statlog+%28German+Credit+Data%29>`_.  # noqa
+"""
+
+# %%
+# "Statlog" German credit dataset
+# -------------------------------
+#
+# We fetch the German credit dataset from OpenML.
+import sklearn
+from sklearn.datasets import fetch_openml
+
+sklearn.set_config(transform_output="pandas")
+german_credit = fetch_openml(data_id=31, as_frame=True, parser="pandas")
+X, y = german_credit.data, german_credit.target
+
+# %%
+# We check the feature types available in `X`.
+X.info()
+
+# %%
+X.head()
+
+# %%
+# Many features are categorical and usually string-encoded. We need to encode
+# these categories when we develop ou predictive model. Let's check the targets.
+y.value_counts()
+
+# %%
+# A first observation is that the dataset is imbalanced. We would need to be careful
+# when evaluating our predictive model and use a family of metrics that are adapted
+# to this setting.
+#
+# In addition, we observe that the target is string-encoded. Some metrics
+# (e.g. precision and recall) require to provide the label of interest also called
+# the "positive label". Here, we define that our goal is to predict whether or not
+# a sample is a "bad" credit.
+pos_label = "bad"
+
+# %%
+# To carry our analysis, we split our dataset using a single stratified split.
+from sklearn.model_selection import train_test_split
+
+X_train, X_test, y_train, y_test = train_test_split(X, y, stratify=y, random_state=0)
+
+# %%
+# We are ready to design our predictive model and the associated evaluation strategy.
+#
+# Evaluation metrics
+# ------------------
+#
+# In this section, we define a set of metrics that we use later. To see
+# the effect of tuning the cut-off point, we evaluate the predictive model using
+# the Receiver Operating Characteristic (ROC) curve and the Precision-Recall curve.
+# The values reported on these plots are therefore the true positive rate (TPR) and
+# the false positive rate (FPR) for the ROC curve and the precision and recall for the
+# Precision-Recall curve.
+#
+# From these four metrics, scikit-learn does not provide a scorer for the FPR. We
+# therefore need to define a small custom function to compute it.
+import numpy as np
+from sklearn.metrics import confusion_matrix, make_scorer, precision_score, recall_score
+
+
+def fpr_score(y, y_pred, **kwargs):
+    cm = confusion_matrix(y, y_pred)
+    classes = np.unique(y)
+    pos_label = kwargs.get("pos_label", classes[-1])
+    pos_label_idx = np.searchsorted(classes, pos_label)
+    if pos_label_idx == 0:
+        cm = cm[::-1, ::-1]
+    tn, fp, _, _ = cm.ravel()
+    tnr = tn / (tn + fp)
+    return 1 - tnr
+
+
+# %%
+# As previously stated, the "positive label" is not defined as the value "1" and calling
+# some of the metrics with this non-standard value raise an error. We need to
+# provide the indication of the "positive label" to the metrics.
+#
+# We therefore need to define a scikit-learn scorer using
+# :func:`~sklearn.metrics.make_scorer` where the information is passed. We store all
+# created scorer in a dictionary. To use them, we need to pass the fitted model and
+# the data and target on which we want to evaluate the predictive model.
+tpr_score = recall_score  # TPR and recall are the same metric
+scoring = {
+    "precision": make_scorer(precision_score, pos_label=pos_label),
+    "recall": make_scorer(recall_score, pos_label=pos_label),
+    "fpr": make_scorer(fpr_score, pos_label=pos_label),
+    "tpr": make_scorer(tpr_score, pos_label=pos_label),
+}
+
+# %%
+# In addition, the original research [1]_ defines a business metric. They provide a
+# cost-matrix where they define that predicting a "bad" credit as "good" is 5 times more
+# costly than the opposite. We define a dictionary containing this information and a
+# score function that computes the cost.
+cost_gain_matrix = {"tp": 0, "tn": 0, "fp": -1, "fn": -5}
+
+
+def gain_cost_score(y, y_pred, **kwargs):
+    cm = confusion_matrix(y, y_pred)
+    classes = np.unique(y)
+    pos_label = kwargs.get("pos_label", classes[-1])
+    pos_label_idx = np.searchsorted(classes, pos_label)
+    if pos_label_idx == 0:
+        cm = cm[::-1, ::-1]
+    costs_and_gain = np.array(
+        [
+            [kwargs["cost_gain_matrix"]["tn"], kwargs["cost_gain_matrix"]["fp"]],
+            [kwargs["cost_gain_matrix"]["fn"], kwargs["cost_gain_matrix"]["tp"]],
+        ]
+    )
+    return (costs_and_gain * cm).sum()
+
+
+scoring["cost_gain"] = make_scorer(
+    gain_cost_score, pos_label=pos_label, cost_gain_matrix=cost_gain_matrix
+)
+# %%
+# Vanilla predictive model
+# ------------------------
+#
+# Design of the predictive model
+# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+#
+# In this section, we design our predictive model. We use a gradient boosting classifier
+# using :class:`~sklearn.ensemble.HistGradientBoostingClassifier`. However, we need to
+# encode the categorical features with numerical values and we therefore use an
+# :class:`~sklearn.preprocessing.OrdinalEncoder` to do so. The numerical features are
+# kept as-is. To recover the name of the categorical columns, we use the helper function
+# :func:`~sklearn.compose.make_column_selector` and the fact that the categorical
+# features are stored as `category` dtype.
+from sklearn.compose import ColumnTransformer, make_column_selector as selector
+from sklearn.ensemble import HistGradientBoostingClassifier
+from sklearn.pipeline import Pipeline
+from sklearn.preprocessing import OrdinalEncoder
+
+categorical_columns = selector(dtype_include="category")(X)
+numerical_columns = selector(dtype_exclude="category")(X)
+
+preprocessor = ColumnTransformer(
+    [
+        (
+            "categorical",
+            OrdinalEncoder(handle_unknown="use_encoded_value", unknown_value=-1),
+            categorical_columns,
+        ),
+        ("numerical", "passthrough", numerical_columns),
+    ],
+    verbose_feature_names_out=False,
+)
+model = Pipeline(
+    [
+        ("preprocessor", preprocessor),
+        (
+            "classifier",
+            HistGradientBoostingClassifier(
+                categorical_features=categorical_columns, random_state=0
+            ),
+        ),
+    ]
+)
+
+model.fit(X_train, y_train)
+
+# %%
+# Evaluation of the predictive model
+# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+#
+# We evaluate the performance of our predictive model using the ROC and Precision-Recall
+# curves.
+import matplotlib.pyplot as plt
+from sklearn.metrics import PrecisionRecallDisplay, RocCurveDisplay
+
+fig, axs = plt.subplots(nrows=1, ncols=2, figsize=(14, 6))
+
+PrecisionRecallDisplay.from_estimator(
+    model, X_test, y_test, pos_label=pos_label, ax=axs[0], name="GBDT"
+)
+axs[0].plot(
+    scoring["recall"](model, X_test, y_test),
+    scoring["precision"](model, X_test, y_test),
+    marker="o",
+    markersize=10,
+    color="tab:blue",
+    label="Default cut-off point at a probability of 0.5",
+)
+axs[0].set_title("Precision-Recall curve")
+axs[0].legend()
+
+RocCurveDisplay.from_estimator(
+    model,
+    X_test,
+    y_test,
+    pos_label=pos_label,
+    ax=axs[1],
+    name="GBDT",
+    plot_chance_level=True,
+)
+axs[1].plot(
+    scoring["fpr"](model, X_test, y_test),
+    scoring["tpr"](model, X_test, y_test),
+    marker="o",
+    markersize=10,
+    color="tab:blue",
+    label="Default cut-off point at a probability of 0.5",
+)
+axs[1].set_title("ROC curve")
+axs[1].legend()
+_ = fig.suptitle("Evaluation of the vanilla GBDT model")
+
+# %%
+# We recall that these curves corresponds show the statistical performance of the
+# predictive model for different cut-off points. For the Precision-Recall curve, the
+# reported metrics are the precision and recall and for the ROC curve, the reported
+# metrics are the TPR (same as recall) and FPR.
+#
+# Here, the different cut-off points corresponds at different level of probability
+# scores ranging between 0 and 1. By default, `model.predict` uses a cut-off point at
+# a probability of 0.5. Thus, the metrics for this cut-off point is reported with the
+# blue dot on the curves: it corresponds to the statistical performance of the model
+# when using `model.predict`.
+#
+# However, we recall that the original aim was to minimize the cost (or maximize the
+# gain) by the business metric. We can compute the value of the business metric:
+scoring["cost_gain"](model, X_test, y_test)
+
+# %%
+# At this stage, we don't know if some other cut-off points could lead to a greater
+# gain. To be able to decide, if another cut-off point is better, we would need to
+# compute the cost-gain using the business metric for all possible cut-off points and
+# choose the optimal one. This strategy is quite tedious to implement and the
+# :class:`~sklearn.metrics.CutOffClassifier` class is here to help us. It automatically
+# compute the cost-gain for all possible cut-off points and choose the optimal one.
path: examples/model_selection/plot_cutoff_tuning.py
text: @user1:
```suggestion
# At this stage we don't know if any other cut-off can lead to a greater gain.
# To find the optimal one, we need to compute the cost-gain using the business
# metric for all possible cut-off points and choose the best. This strategy can
# be quite tedious to implement by hand, but the
# :class:`~sklearn.metrics.CutOffClassifier` class is here to help us. It
# automatically computes the cost-gain for all possible cut-off points and
# optimizes for the `objective_metric`.
```

diff_hunk: @@ -0,0 +1,566 @@
+"""
+==================================================
+Post-tuning the cut-off point of decision function
+==================================================
+
+Once a classifier is trained, the output of the :term:`predict` method output hard
+predictions corresponding to a thresholding of either the :term:`decision function`
+or the :term:`predict_proba` output. For binary classifier, the default threshold is
+defined as a probability score of 0.5 or a decision function value of 0.0.
+
+However, it happens that this default strategy is not optimized for the task at hand.
+Here, we use the "Statlog" German credit dataset [1]_ to illustrate such an application.
+In this dataset, the task is to predict whether a person is "good" or "bad" credit.
+In addition, cost-matrix is provided where it is defined that classifying a "bad"
+credit as "good" is 5 times more costly than classifying a "good" credit as "bad".
+
+We use the :class:`~sklearn.model_selection.CutOffClassifier` to tune the
+cut-off point of the decision function to minimize the business cost provided to us.
+
+.. topic:: References
+
+    .. [1] "Statlog (German Credit Data) Data Set", UCI Machine Learning Repository,
+       `Link <https://archive.ics.uci.edu/ml/datasets/Statlog+%28German+Credit+Data%29>`_.  # noqa
+"""
+
+# %%
+# "Statlog" German credit dataset
+# -------------------------------
+#
+# We fetch the German credit dataset from OpenML.
+import sklearn
+from sklearn.datasets import fetch_openml
+
+sklearn.set_config(transform_output="pandas")
+german_credit = fetch_openml(data_id=31, as_frame=True, parser="pandas")
+X, y = german_credit.data, german_credit.target
+
+# %%
+# We check the feature types available in `X`.
+X.info()
+
+# %%
+X.head()
+
+# %%
+# Many features are categorical and usually string-encoded. We need to encode
+# these categories when we develop ou predictive model. Let's check the targets.
+y.value_counts()
+
+# %%
+# A first observation is that the dataset is imbalanced. We would need to be careful
+# when evaluating our predictive model and use a family of metrics that are adapted
+# to this setting.
+#
+# In addition, we observe that the target is string-encoded. Some metrics
+# (e.g. precision and recall) require to provide the label of interest also called
+# the "positive label". Here, we define that our goal is to predict whether or not
+# a sample is a "bad" credit.
+pos_label = "bad"
+
+# %%
+# To carry our analysis, we split our dataset using a single stratified split.
+from sklearn.model_selection import train_test_split
+
+X_train, X_test, y_train, y_test = train_test_split(X, y, stratify=y, random_state=0)
+
+# %%
+# We are ready to design our predictive model and the associated evaluation strategy.
+#
+# Evaluation metrics
+# ------------------
+#
+# In this section, we define a set of metrics that we use later. To see
+# the effect of tuning the cut-off point, we evaluate the predictive model using
+# the Receiver Operating Characteristic (ROC) curve and the Precision-Recall curve.
+# The values reported on these plots are therefore the true positive rate (TPR) and
+# the false positive rate (FPR) for the ROC curve and the precision and recall for the
+# Precision-Recall curve.
+#
+# From these four metrics, scikit-learn does not provide a scorer for the FPR. We
+# therefore need to define a small custom function to compute it.
+import numpy as np
+from sklearn.metrics import confusion_matrix, make_scorer, precision_score, recall_score
+
+
+def fpr_score(y, y_pred, **kwargs):
+    cm = confusion_matrix(y, y_pred)
+    classes = np.unique(y)
+    pos_label = kwargs.get("pos_label", classes[-1])
+    pos_label_idx = np.searchsorted(classes, pos_label)
+    if pos_label_idx == 0:
+        cm = cm[::-1, ::-1]
+    tn, fp, _, _ = cm.ravel()
+    tnr = tn / (tn + fp)
+    return 1 - tnr
+
+
+# %%
+# As previously stated, the "positive label" is not defined as the value "1" and calling
+# some of the metrics with this non-standard value raise an error. We need to
+# provide the indication of the "positive label" to the metrics.
+#
+# We therefore need to define a scikit-learn scorer using
+# :func:`~sklearn.metrics.make_scorer` where the information is passed. We store all
+# created scorer in a dictionary. To use them, we need to pass the fitted model and
+# the data and target on which we want to evaluate the predictive model.
+tpr_score = recall_score  # TPR and recall are the same metric
+scoring = {
+    "precision": make_scorer(precision_score, pos_label=pos_label),
+    "recall": make_scorer(recall_score, pos_label=pos_label),
+    "fpr": make_scorer(fpr_score, pos_label=pos_label),
+    "tpr": make_scorer(tpr_score, pos_label=pos_label),
+}
+
+# %%
+# In addition, the original research [1]_ defines a business metric. They provide a
+# cost-matrix where they define that predicting a "bad" credit as "good" is 5 times more
+# costly than the opposite. We define a dictionary containing this information and a
+# score function that computes the cost.
+cost_gain_matrix = {"tp": 0, "tn": 0, "fp": -1, "fn": -5}
+
+
+def gain_cost_score(y, y_pred, **kwargs):
+    cm = confusion_matrix(y, y_pred)
+    classes = np.unique(y)
+    pos_label = kwargs.get("pos_label", classes[-1])
+    pos_label_idx = np.searchsorted(classes, pos_label)
+    if pos_label_idx == 0:
+        cm = cm[::-1, ::-1]
+    costs_and_gain = np.array(
+        [
+            [kwargs["cost_gain_matrix"]["tn"], kwargs["cost_gain_matrix"]["fp"]],
+            [kwargs["cost_gain_matrix"]["fn"], kwargs["cost_gain_matrix"]["tp"]],
+        ]
+    )
+    return (costs_and_gain * cm).sum()
+
+
+scoring["cost_gain"] = make_scorer(
+    gain_cost_score, pos_label=pos_label, cost_gain_matrix=cost_gain_matrix
+)
+# %%
+# Vanilla predictive model
+# ------------------------
+#
+# Design of the predictive model
+# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+#
+# In this section, we design our predictive model. We use a gradient boosting classifier
+# using :class:`~sklearn.ensemble.HistGradientBoostingClassifier`. However, we need to
+# encode the categorical features with numerical values and we therefore use an
+# :class:`~sklearn.preprocessing.OrdinalEncoder` to do so. The numerical features are
+# kept as-is. To recover the name of the categorical columns, we use the helper function
+# :func:`~sklearn.compose.make_column_selector` and the fact that the categorical
+# features are stored as `category` dtype.
+from sklearn.compose import ColumnTransformer, make_column_selector as selector
+from sklearn.ensemble import HistGradientBoostingClassifier
+from sklearn.pipeline import Pipeline
+from sklearn.preprocessing import OrdinalEncoder
+
+categorical_columns = selector(dtype_include="category")(X)
+numerical_columns = selector(dtype_exclude="category")(X)
+
+preprocessor = ColumnTransformer(
+    [
+        (
+            "categorical",
+            OrdinalEncoder(handle_unknown="use_encoded_value", unknown_value=-1),
+            categorical_columns,
+        ),
+        ("numerical", "passthrough", numerical_columns),
+    ],
+    verbose_feature_names_out=False,
+)
+model = Pipeline(
+    [
+        ("preprocessor", preprocessor),
+        (
+            "classifier",
+            HistGradientBoostingClassifier(
+                categorical_features=categorical_columns, random_state=0
+            ),
+        ),
+    ]
+)
+
+model.fit(X_train, y_train)
+
+# %%
+# Evaluation of the predictive model
+# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+#
+# We evaluate the performance of our predictive model using the ROC and Precision-Recall
+# curves.
+import matplotlib.pyplot as plt
+from sklearn.metrics import PrecisionRecallDisplay, RocCurveDisplay
+
+fig, axs = plt.subplots(nrows=1, ncols=2, figsize=(14, 6))
+
+PrecisionRecallDisplay.from_estimator(
+    model, X_test, y_test, pos_label=pos_label, ax=axs[0], name="GBDT"
+)
+axs[0].plot(
+    scoring["recall"](model, X_test, y_test),
+    scoring["precision"](model, X_test, y_test),
+    marker="o",
+    markersize=10,
+    color="tab:blue",
+    label="Default cut-off point at a probability of 0.5",
+)
+axs[0].set_title("Precision-Recall curve")
+axs[0].legend()
+
+RocCurveDisplay.from_estimator(
+    model,
+    X_test,
+    y_test,
+    pos_label=pos_label,
+    ax=axs[1],
+    name="GBDT",
+    plot_chance_level=True,
+)
+axs[1].plot(
+    scoring["fpr"](model, X_test, y_test),
+    scoring["tpr"](model, X_test, y_test),
+    marker="o",
+    markersize=10,
+    color="tab:blue",
+    label="Default cut-off point at a probability of 0.5",
+)
+axs[1].set_title("ROC curve")
+axs[1].legend()
+_ = fig.suptitle("Evaluation of the vanilla GBDT model")
+
+# %%
+# We recall that these curves corresponds show the statistical performance of the
+# predictive model for different cut-off points. For the Precision-Recall curve, the
+# reported metrics are the precision and recall and for the ROC curve, the reported
+# metrics are the TPR (same as recall) and FPR.
+#
+# Here, the different cut-off points corresponds at different level of probability
+# scores ranging between 0 and 1. By default, `model.predict` uses a cut-off point at
+# a probability of 0.5. Thus, the metrics for this cut-off point is reported with the
+# blue dot on the curves: it corresponds to the statistical performance of the model
+# when using `model.predict`.
+#
+# However, we recall that the original aim was to minimize the cost (or maximize the
+# gain) by the business metric. We can compute the value of the business metric:
+scoring["cost_gain"](model, X_test, y_test)
+
+# %%
+# At this stage, we don't know if some other cut-off points could lead to a greater
+# gain. To be able to decide, if another cut-off point is better, we would need to
+# compute the cost-gain using the business metric for all possible cut-off points and
+# choose the optimal one. This strategy is quite tedious to implement and the
+# :class:`~sklearn.metrics.CutOffClassifier` class is here to help us. It automatically
+# compute the cost-gain for all possible cut-off points and choose the optimal one.
+#
+# .. _cost_sensitive_learning_example:
+#
+# Tuning the cut-off point
+# ------------------------
+#
+# We use :class:`~sklearn.metrics.CutOffClassifier` to tune the cut-off point. We need
+# to provide the business metric to optimize as well as the positive label. Internally,
+# the optimum cut-off point is chosen such that it maximized the business metric
path: examples/model_selection/plot_cutoff_tuning.py
text: @user1:
```suggestion
# the optimum cut-off point is chosen such that it maximizes the business metric
```

diff_hunk: @@ -0,0 +1,155 @@
+.. currentmodule:: sklearn.model_selection
+
+.. _cutoffclassifier:
+
+========================================================
+Tuning cut-off decision threshold for classes prediction
+========================================================
+
+Classifiers are predictive models: they use statistical learning to predict
+outcomes. The outcomes of a classifier takes two forms: a "soft" score for each
+sample in relation to each class, and a "hard" categorical prediction (i.e.
+class label). Soft predictions are obtained using :term:`predict_proba` or
+:term:`decision_function` while hard predictions are obtained using
+:term:`predict`.
+
+In scikit-learn, there is a connection between soft and hard prediction. In the
+case of a binary classification, hard predictions are obtained by associating
+the positive class with probability value greater than 0.5 (obtained with
+:term:`predict_proba`) or decision function value greater than 0 (obtained with
+:term:`decision_function`).
+
+    >>> from sklearn.datasets import make_classification
+    >>> from sklearn.tree import DecisionTreeClassifier
+    >>> X, y = make_classification(random_state=0)
+    >>> classifier = DecisionTreeClassifier(max_depth=2, random_state=0).fit(X, y)
+    >>> classifier.predict_proba(X[:4])
+    array([[0.94   , 0.06   ],
+           [0.94   , 0.06   ],
+           [0.04..., 0.95...],
+           [0.04..., 0.95...]])
+    >>> classifier.predict(X[:4])
+    array([0, 0, 1, 1])
+
+
+Similar rules apply for other classification problems.
+
+While these approaches are reasonable as default behaviors, they might not be
+adapted to some cases. The context and nature of the use case define the
+expected behavior of the classifier and thus the strategy to convert soft
+predictions into hard predictions. We illustrate this point with an example.
+
+Let's imagine the deployment of a predictive model helping medical doctors to
+detect cancers. In a setting where this model would be a tool to discard
+obvious cases, doctors might be interested to have a high recall (all cancers
+cases should be tagged as such) to not miss any patient with a cancer. However,
+it will be at the cost of having more false positive predictions (i.e. lower
+precision). Thus, in terms of decision threshold, it would be better to
+classify a patient having a cancer for a lower probability than 0.5.
path: doc/modules/prediction.rst
text: @user1:
I am adding a small interpolation on avoiding "potentially harmful treatments". If you find it too distracting feel free to omit it and just retain the grammar tweaks:

```suggestion
Let's imagine the deployment of a predictive model helping medical doctors to
detect cancers. In a setting where this model was a tool to discard obvious
cases and false positives don't lead to potentially harmful treatments, doctors
might be interested in having a high recall (all cancers cases should be tagged
as such) to not miss any patient with a cancer. However, that is at the cost of
having more false positive predictions (i.e. lower precision). Thus, in terms of
decision threshold, it may be better to classify a patient as having a cancer
for a probability lower than 0.5.
```

diff_hunk: @@ -0,0 +1,155 @@
+.. currentmodule:: sklearn.model_selection
+
+.. _cutoffclassifier:
+
+========================================================
+Tuning cut-off decision threshold for classes prediction
+========================================================
+
+Classifiers are predictive models: they use statistical learning to predict
+outcomes. The outcomes of a classifier takes two forms: a "soft" score for each
+sample in relation to each class, and a "hard" categorical prediction (i.e.
+class label). Soft predictions are obtained using :term:`predict_proba` or
+:term:`decision_function` while hard predictions are obtained using
+:term:`predict`.
+
+In scikit-learn, there is a connection between soft and hard prediction. In the
+case of a binary classification, hard predictions are obtained by associating
+the positive class with probability value greater than 0.5 (obtained with
+:term:`predict_proba`) or decision function value greater than 0 (obtained with
+:term:`decision_function`).
+
+    >>> from sklearn.datasets import make_classification
+    >>> from sklearn.tree import DecisionTreeClassifier
+    >>> X, y = make_classification(random_state=0)
+    >>> classifier = DecisionTreeClassifier(max_depth=2, random_state=0).fit(X, y)
+    >>> classifier.predict_proba(X[:4])
+    array([[0.94   , 0.06   ],
+           [0.94   , 0.06   ],
+           [0.04..., 0.95...],
+           [0.04..., 0.95...]])
+    >>> classifier.predict(X[:4])
+    array([0, 0, 1, 1])
+
+
+Similar rules apply for other classification problems.
+
+While these approaches are reasonable as default behaviors, they might not be
+adapted to some cases. The context and nature of the use case define the
+expected behavior of the classifier and thus the strategy to convert soft
+predictions into hard predictions. We illustrate this point with an example.
+
+Let's imagine the deployment of a predictive model helping medical doctors to
+detect cancers. In a setting where this model would be a tool to discard
+obvious cases, doctors might be interested to have a high recall (all cancers
+cases should be tagged as such) to not miss any patient with a cancer. However,
+it will be at the cost of having more false positive predictions (i.e. lower
+precision). Thus, in terms of decision threshold, it would be better to
+classify a patient having a cancer for a lower probability than 0.5.
+
+Post-tuning of the decision threshold
+=====================================
+
+One solution to address the problem stated in the introduction is to tune the decision
+threshold of the classifier once this model has been trained. The
+:class:`~sklearn.model_selection.CutOffClassifier` allows to tune this threshold using
+an internal cross-validation. The optimum threshold is tuned to maximize a given metric
+with or without constraints.
+
+The following image illustrate the tuning of the cut-off point for a gradient
+boosting classifier. While the vanilla and tuned classifiers provide the same
+Receiver Operating Characteristic (ROC) and Precision-Recall curves, and thus
+the same :term:`predict_proba` outputs, the "hard" predictions defer because of
path: doc/modules/prediction.rst
text: @user1:
```suggestion
the same :term:`predict_proba` outputs, the "hard" predictions differ because of
```

diff_hunk: @@ -0,0 +1,155 @@
+.. currentmodule:: sklearn.model_selection
+
+.. _cutoffclassifier:
+
+========================================================
+Tuning cut-off decision threshold for classes prediction
+========================================================
+
+Classifiers are predictive models: they use statistical learning to predict
+outcomes. The outcomes of a classifier takes two forms: a "soft" score for each
+sample in relation to each class, and a "hard" categorical prediction (i.e.
+class label). Soft predictions are obtained using :term:`predict_proba` or
+:term:`decision_function` while hard predictions are obtained using
+:term:`predict`.
+
+In scikit-learn, there is a connection between soft and hard prediction. In the
+case of a binary classification, hard predictions are obtained by associating
+the positive class with probability value greater than 0.5 (obtained with
+:term:`predict_proba`) or decision function value greater than 0 (obtained with
+:term:`decision_function`).
+
+    >>> from sklearn.datasets import make_classification
+    >>> from sklearn.tree import DecisionTreeClassifier
+    >>> X, y = make_classification(random_state=0)
+    >>> classifier = DecisionTreeClassifier(max_depth=2, random_state=0).fit(X, y)
+    >>> classifier.predict_proba(X[:4])
+    array([[0.94   , 0.06   ],
+           [0.94   , 0.06   ],
+           [0.04..., 0.95...],
+           [0.04..., 0.95...]])
+    >>> classifier.predict(X[:4])
+    array([0, 0, 1, 1])
+
+
+Similar rules apply for other classification problems.
+
+While these approaches are reasonable as default behaviors, they might not be
+adapted to some cases. The context and nature of the use case define the
+expected behavior of the classifier and thus the strategy to convert soft
+predictions into hard predictions. We illustrate this point with an example.
+
+Let's imagine the deployment of a predictive model helping medical doctors to
+detect cancers. In a setting where this model would be a tool to discard
+obvious cases, doctors might be interested to have a high recall (all cancers
+cases should be tagged as such) to not miss any patient with a cancer. However,
+it will be at the cost of having more false positive predictions (i.e. lower
+precision). Thus, in terms of decision threshold, it would be better to
+classify a patient having a cancer for a lower probability than 0.5.
+
+Post-tuning of the decision threshold
+=====================================
+
+One solution to address the problem stated in the introduction is to tune the decision
+threshold of the classifier once this model has been trained. The
+:class:`~sklearn.model_selection.CutOffClassifier` allows to tune this threshold using
+an internal cross-validation. The optimum threshold is tuned to maximize a given metric
+with or without constraints.
+
+The following image illustrate the tuning of the cut-off point for a gradient
+boosting classifier. While the vanilla and tuned classifiers provide the same
+Receiver Operating Characteristic (ROC) and Precision-Recall curves, and thus
+the same :term:`predict_proba` outputs, the "hard" predictions defer because of
+the tuned cut-off point. The vanilla classifier predicts the class of interest
+for a probability greater than 0.5 while the tuned classifier predicts the
+class of interest for a very low probability (around 0.02). This cut-off point
+is maximizes a utility metric defined by the business case (in this case an
path: doc/modules/prediction.rst
text: @user1:
```suggestion
optimizes a utility metric defined by the business case (in this case an
```

diff_hunk: @@ -0,0 +1,155 @@
+.. currentmodule:: sklearn.model_selection
+
+.. _cutoffclassifier:
+
+========================================================
+Tuning cut-off decision threshold for classes prediction
+========================================================
+
+Classifiers are predictive models: they use statistical learning to predict
+outcomes. The outcomes of a classifier takes two forms: a "soft" score for each
+sample in relation to each class, and a "hard" categorical prediction (i.e.
+class label). Soft predictions are obtained using :term:`predict_proba` or
+:term:`decision_function` while hard predictions are obtained using
+:term:`predict`.
+
+In scikit-learn, there is a connection between soft and hard prediction. In the
+case of a binary classification, hard predictions are obtained by associating
+the positive class with probability value greater than 0.5 (obtained with
+:term:`predict_proba`) or decision function value greater than 0 (obtained with
+:term:`decision_function`).
+
+    >>> from sklearn.datasets import make_classification
+    >>> from sklearn.tree import DecisionTreeClassifier
+    >>> X, y = make_classification(random_state=0)
+    >>> classifier = DecisionTreeClassifier(max_depth=2, random_state=0).fit(X, y)
+    >>> classifier.predict_proba(X[:4])
+    array([[0.94   , 0.06   ],
+           [0.94   , 0.06   ],
+           [0.04..., 0.95...],
+           [0.04..., 0.95...]])
+    >>> classifier.predict(X[:4])
+    array([0, 0, 1, 1])
+
+
+Similar rules apply for other classification problems.
+
+While these approaches are reasonable as default behaviors, they might not be
+adapted to some cases. The context and nature of the use case define the
+expected behavior of the classifier and thus the strategy to convert soft
+predictions into hard predictions. We illustrate this point with an example.
+
+Let's imagine the deployment of a predictive model helping medical doctors to
+detect cancers. In a setting where this model would be a tool to discard
+obvious cases, doctors might be interested to have a high recall (all cancers
+cases should be tagged as such) to not miss any patient with a cancer. However,
+it will be at the cost of having more false positive predictions (i.e. lower
+precision). Thus, in terms of decision threshold, it would be better to
+classify a patient having a cancer for a lower probability than 0.5.
+
+Post-tuning of the decision threshold
+=====================================
+
+One solution to address the problem stated in the introduction is to tune the decision
+threshold of the classifier once this model has been trained. The
+:class:`~sklearn.model_selection.CutOffClassifier` allows to tune this threshold using
+an internal cross-validation. The optimum threshold is tuned to maximize a given metric
+with or without constraints.
+
+The following image illustrate the tuning of the cut-off point for a gradient
+boosting classifier. While the vanilla and tuned classifiers provide the same
+Receiver Operating Characteristic (ROC) and Precision-Recall curves, and thus
+the same :term:`predict_proba` outputs, the "hard" predictions defer because of
+the tuned cut-off point. The vanilla classifier predicts the class of interest
+for a probability greater than 0.5 while the tuned classifier predicts the
+class of interest for a very low probability (around 0.02). This cut-off point
+is maximizes a utility metric defined by the business case (in this case an
+insurance company).
+
+.. figure:: ../auto_examples/model_selection/images/sphx_glr_plot_cutoff_tuning_002.png
+   :target: ../auto_examples/model_selection/plot_cutoff_tuning.html
+   :align: center
+
+Available options to tune the cut-off point
+-------------------------------------------
+
+The cut-off point can be tuned with different strategies controlled by the parameter
+`objective_metric`.
+
+A straightforward use case is to maximize a pre-defined scikit-learn metric. These
+metrics can be found by calling the function :func:`~sklearn.metrics.get_scorer_names`.
+We provide an example where we maximize the balanced accuracy.
+
+.. note::
+
+    It is important to notice that these metrics comes with default parameter, notably
+    the label of the class of interested (i.e. `pos_label`). Thus, if this label is not
+    the right one for your application, you need to define a scorer and pass the right
+    `pos_label` (and additional parameters) using the
+    :func:`~sklearn.metrics.make_scorer`. You should refer to :ref:`scoring` to get all
+    information to define your own scoring function. For instance, we show how to pass
+    the information to the scorer that the label of interest is `0` when maximizing the
+    :func:`~sklearn.metrics.f1_score`:
+
+        >>> from sklearn.linear_model import LogisticRegression
+        >>> from sklearn.model_selection import CutOffClassifier, train_test_split
+        >>> from sklearn.metrics import make_scorer, f1_score
+        >>> X, y = make_classification(
+        ...    n_samples=1_000, weights=[0.1, 0.9], random_state=0)
+        >>> X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)
+        >>> pos_label = 0
+        >>> scorer = make_scorer(f1_score, pos_label=pos_label)
+        >>> model = CutOffClassifier(LogisticRegression(), objective_metric=scorer).fit(
+        ...     X_train, y_train)
+        >>> scorer(model, X_test, y_test)
+        0.82...
+        >>> # compare it with the internal score found by cross-validation
+        >>> model.objective_score_
+        0.86...
+
+A second strategy aims at maximizing a metric while imposing constraints on another
+metric. Four pre-defined options exist, 2 that uses the Receiver Operating
+Characteristic (ROC) statistics and 2 that uses the Precision-Recall statistics.
+
+- `"max_tpr_at_tnr_constraint"`: maximizes the True Positive Rate (TPR) such that the
+  True Negative Rate (TNR) is the closest to a given value.
+- `"max_tnr_at_tpr_constraint"`: maximizes the TNR such that the TPR is the closest to
+  a given value.
+- `"max_precision_at_recall_constraint"`: maximizes the precision such that the recall
+  is the closest to a given value.
+- `"max_recall_at_precision_constraint"`: maximizes the recall such that the precision
+  is the closest to a given value.
+
+For these options, the `constraint_value` parameter needs to be defined. In addition,
+you can use the `pos_label` parameter to indicate the label of the class of interest.
+
+The final strategy maximizes a custom utility function. This problem is also known as
+cost-sensitive learning. The utility function is defined by providing dictionary
+containing the cost-gain associated with the entries of the confusion matrix. The keys
+are defined as `{"tn", "fp", "fn", "tp"}`. The class of interest is defined using the
+`pos_label` parameter. Refer to :ref:`cost_sensitive_learning_example` for an example
+depicting the use of such a utility function.
+
+Important notes regarding the internal cross-validation
+-------------------------------------------------------
+
+By default :class:`~sklearn.model_selection.CutOffClassifier` uses a 5-fold stratified
+cross-validation to tune the cut-off point. The parameter `cv` allows to control the
+cross-validation strategy. It is possible to go around cross-validation by passing
+`cv="prefit"` and provide an already fitted classifier. In this case, the cut-off point
+is tuned on the data provided to the `fit` method.
+
+However, you should be extremely careful when using this option. You should never use
+the same data for training the classifier and tuning the cut-off point at the risk of
+overfitting. Refer to :ref:`cutoffclassifier_no_cv` that shows such overfitting. If
+you are in a situation where you have limited resources, you should can consider using
path: doc/modules/prediction.rst
text: @user1:
```suggestion
you are in a situation where you have limited resources, you should consider using
```

diff_hunk: @@ -0,0 +1,566 @@
+"""
+==================================================
+Post-tuning the cut-off point of decision function
+==================================================
+
+Once a classifier is trained, the output of the :term:`predict` method output hard
+predictions corresponding to a thresholding of either the :term:`decision function`
+or the :term:`predict_proba` output. For binary classifier, the default threshold is
+defined as a probability score of 0.5 or a decision function value of 0.0.
+
+However, it happens that this default strategy is not optimized for the task at hand.
+Here, we use the "Statlog" German credit dataset [1]_ to illustrate such an application.
+In this dataset, the task is to predict whether a person is "good" or "bad" credit.
+In addition, cost-matrix is provided where it is defined that classifying a "bad"
+credit as "good" is 5 times more costly than classifying a "good" credit as "bad".
+
+We use the :class:`~sklearn.model_selection.CutOffClassifier` to tune the
+cut-off point of the decision function to minimize the business cost provided to us.
+
+.. topic:: References
+
+    .. [1] "Statlog (German Credit Data) Data Set", UCI Machine Learning Repository,
+       `Link <https://archive.ics.uci.edu/ml/datasets/Statlog+%28German+Credit+Data%29>`_.  # noqa
+"""
+
+# %%
+# "Statlog" German credit dataset
+# -------------------------------
+#
+# We fetch the German credit dataset from OpenML.
+import sklearn
+from sklearn.datasets import fetch_openml
+
+sklearn.set_config(transform_output="pandas")
+german_credit = fetch_openml(data_id=31, as_frame=True, parser="pandas")
+X, y = german_credit.data, german_credit.target
+
+# %%
+# We check the feature types available in `X`.
+X.info()
+
+# %%
+X.head()
+
+# %%
+# Many features are categorical and usually string-encoded. We need to encode
+# these categories when we develop ou predictive model. Let's check the targets.
+y.value_counts()
+
+# %%
+# A first observation is that the dataset is imbalanced. We would need to be careful
+# when evaluating our predictive model and use a family of metrics that are adapted
+# to this setting.
+#
+# In addition, we observe that the target is string-encoded. Some metrics
+# (e.g. precision and recall) require to provide the label of interest also called
+# the "positive label". Here, we define that our goal is to predict whether or not
+# a sample is a "bad" credit.
+pos_label = "bad"
+
+# %%
+# To carry our analysis, we split our dataset using a single stratified split.
+from sklearn.model_selection import train_test_split
+
+X_train, X_test, y_train, y_test = train_test_split(X, y, stratify=y, random_state=0)
+
+# %%
+# We are ready to design our predictive model and the associated evaluation strategy.
+#
+# Evaluation metrics
+# ------------------
+#
+# In this section, we define a set of metrics that we use later. To see
+# the effect of tuning the cut-off point, we evaluate the predictive model using
+# the Receiver Operating Characteristic (ROC) curve and the Precision-Recall curve.
+# The values reported on these plots are therefore the true positive rate (TPR) and
+# the false positive rate (FPR) for the ROC curve and the precision and recall for the
+# Precision-Recall curve.
+#
+# From these four metrics, scikit-learn does not provide a scorer for the FPR. We
+# therefore need to define a small custom function to compute it.
+import numpy as np
+from sklearn.metrics import confusion_matrix, make_scorer, precision_score, recall_score
+
+
+def fpr_score(y, y_pred, **kwargs):
+    cm = confusion_matrix(y, y_pred)
+    classes = np.unique(y)
+    pos_label = kwargs.get("pos_label", classes[-1])
+    pos_label_idx = np.searchsorted(classes, pos_label)
+    if pos_label_idx == 0:
+        cm = cm[::-1, ::-1]
+    tn, fp, _, _ = cm.ravel()
+    tnr = tn / (tn + fp)
+    return 1 - tnr
+
+
+# %%
+# As previously stated, the "positive label" is not defined as the value "1" and calling
+# some of the metrics with this non-standard value raise an error. We need to
+# provide the indication of the "positive label" to the metrics.
+#
+# We therefore need to define a scikit-learn scorer using
+# :func:`~sklearn.metrics.make_scorer` where the information is passed. We store all
+# created scorer in a dictionary. To use them, we need to pass the fitted model and
+# the data and target on which we want to evaluate the predictive model.
+tpr_score = recall_score  # TPR and recall are the same metric
+scoring = {
+    "precision": make_scorer(precision_score, pos_label=pos_label),
+    "recall": make_scorer(recall_score, pos_label=pos_label),
+    "fpr": make_scorer(fpr_score, pos_label=pos_label),
+    "tpr": make_scorer(tpr_score, pos_label=pos_label),
+}
+
+# %%
+# In addition, the original research [1]_ defines a business metric. They provide a
+# cost-matrix where they define that predicting a "bad" credit as "good" is 5 times more
+# costly than the opposite. We define a dictionary containing this information and a
+# score function that computes the cost.
+cost_gain_matrix = {"tp": 0, "tn": 0, "fp": -1, "fn": -5}
+
+
+def gain_cost_score(y, y_pred, **kwargs):
+    cm = confusion_matrix(y, y_pred)
+    classes = np.unique(y)
+    pos_label = kwargs.get("pos_label", classes[-1])
+    pos_label_idx = np.searchsorted(classes, pos_label)
+    if pos_label_idx == 0:
+        cm = cm[::-1, ::-1]
+    costs_and_gain = np.array(
+        [
+            [kwargs["cost_gain_matrix"]["tn"], kwargs["cost_gain_matrix"]["fp"]],
+            [kwargs["cost_gain_matrix"]["fn"], kwargs["cost_gain_matrix"]["tp"]],
+        ]
+    )
+    return (costs_and_gain * cm).sum()
+
+
+scoring["cost_gain"] = make_scorer(
+    gain_cost_score, pos_label=pos_label, cost_gain_matrix=cost_gain_matrix
+)
+# %%
+# Vanilla predictive model
+# ------------------------
+#
+# Design of the predictive model
+# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+#
+# In this section, we design our predictive model. We use a gradient boosting classifier
+# using :class:`~sklearn.ensemble.HistGradientBoostingClassifier`. However, we need to
+# encode the categorical features with numerical values and we therefore use an
+# :class:`~sklearn.preprocessing.OrdinalEncoder` to do so. The numerical features are
+# kept as-is. To recover the name of the categorical columns, we use the helper function
+# :func:`~sklearn.compose.make_column_selector` and the fact that the categorical
+# features are stored as `category` dtype.
+from sklearn.compose import ColumnTransformer, make_column_selector as selector
+from sklearn.ensemble import HistGradientBoostingClassifier
+from sklearn.pipeline import Pipeline
+from sklearn.preprocessing import OrdinalEncoder
+
+categorical_columns = selector(dtype_include="category")(X)
+numerical_columns = selector(dtype_exclude="category")(X)
+
+preprocessor = ColumnTransformer(
+    [
+        (
+            "categorical",
+            OrdinalEncoder(handle_unknown="use_encoded_value", unknown_value=-1),
+            categorical_columns,
+        ),
+        ("numerical", "passthrough", numerical_columns),
+    ],
+    verbose_feature_names_out=False,
+)
+model = Pipeline(
+    [
+        ("preprocessor", preprocessor),
+        (
+            "classifier",
+            HistGradientBoostingClassifier(
+                categorical_features=categorical_columns, random_state=0
+            ),
+        ),
+    ]
+)
+
+model.fit(X_train, y_train)
+
+# %%
+# Evaluation of the predictive model
+# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+#
+# We evaluate the performance of our predictive model using the ROC and Precision-Recall
+# curves.
+import matplotlib.pyplot as plt
+from sklearn.metrics import PrecisionRecallDisplay, RocCurveDisplay
+
+fig, axs = plt.subplots(nrows=1, ncols=2, figsize=(14, 6))
+
+PrecisionRecallDisplay.from_estimator(
+    model, X_test, y_test, pos_label=pos_label, ax=axs[0], name="GBDT"
+)
+axs[0].plot(
+    scoring["recall"](model, X_test, y_test),
+    scoring["precision"](model, X_test, y_test),
+    marker="o",
+    markersize=10,
+    color="tab:blue",
+    label="Default cut-off point at a probability of 0.5",
+)
+axs[0].set_title("Precision-Recall curve")
+axs[0].legend()
+
+RocCurveDisplay.from_estimator(
+    model,
+    X_test,
+    y_test,
+    pos_label=pos_label,
+    ax=axs[1],
+    name="GBDT",
+    plot_chance_level=True,
+)
+axs[1].plot(
+    scoring["fpr"](model, X_test, y_test),
+    scoring["tpr"](model, X_test, y_test),
+    marker="o",
+    markersize=10,
+    color="tab:blue",
+    label="Default cut-off point at a probability of 0.5",
+)
+axs[1].set_title("ROC curve")
+axs[1].legend()
+_ = fig.suptitle("Evaluation of the vanilla GBDT model")
+
+# %%
+# We recall that these curves corresponds show the statistical performance of the
+# predictive model for different cut-off points. For the Precision-Recall curve, the
+# reported metrics are the precision and recall and for the ROC curve, the reported
+# metrics are the TPR (same as recall) and FPR.
+#
+# Here, the different cut-off points corresponds at different level of probability
+# scores ranging between 0 and 1. By default, `model.predict` uses a cut-off point at
+# a probability of 0.5. Thus, the metrics for this cut-off point is reported with the
+# blue dot on the curves: it corresponds to the statistical performance of the model
+# when using `model.predict`.
+#
+# However, we recall that the original aim was to minimize the cost (or maximize the
+# gain) by the business metric. We can compute the value of the business metric:
+scoring["cost_gain"](model, X_test, y_test)
+
+# %%
+# At this stage, we don't know if some other cut-off points could lead to a greater
+# gain. To be able to decide, if another cut-off point is better, we would need to
+# compute the cost-gain using the business metric for all possible cut-off points and
+# choose the optimal one. This strategy is quite tedious to implement and the
+# :class:`~sklearn.metrics.CutOffClassifier` class is here to help us. It automatically
+# compute the cost-gain for all possible cut-off points and choose the optimal one.
+#
+# .. _cost_sensitive_learning_example:
+#
+# Tuning the cut-off point
+# ------------------------
+#
+# We use :class:`~sklearn.metrics.CutOffClassifier` to tune the cut-off point. We need
path: examples/model_selection/plot_cutoff_tuning.py
text: @user1:
```suggestion
# We use :class:`~sklearn.model_selection.CutOffClassifier` to tune the cut-off point. We need
```

diff_hunk: @@ -0,0 +1,566 @@
+"""
+==================================================
+Post-tuning the cut-off point of decision function
+==================================================
+
+Once a classifier is trained, the output of the :term:`predict` method output hard
+predictions corresponding to a thresholding of either the :term:`decision function`
+or the :term:`predict_proba` output. For binary classifier, the default threshold is
+defined as a probability score of 0.5 or a decision function value of 0.0.
+
+However, it happens that this default strategy is not optimized for the task at hand.
+Here, we use the "Statlog" German credit dataset [1]_ to illustrate such an application.
+In this dataset, the task is to predict whether a person is "good" or "bad" credit.
+In addition, cost-matrix is provided where it is defined that classifying a "bad"
+credit as "good" is 5 times more costly than classifying a "good" credit as "bad".
+
+We use the :class:`~sklearn.model_selection.CutOffClassifier` to tune the
+cut-off point of the decision function to minimize the business cost provided to us.
+
+.. topic:: References
+
+    .. [1] "Statlog (German Credit Data) Data Set", UCI Machine Learning Repository,
+       `Link <https://archive.ics.uci.edu/ml/datasets/Statlog+%28German+Credit+Data%29>`_.  # noqa
+"""
+
+# %%
+# "Statlog" German credit dataset
+# -------------------------------
+#
+# We fetch the German credit dataset from OpenML.
+import sklearn
+from sklearn.datasets import fetch_openml
+
+sklearn.set_config(transform_output="pandas")
+german_credit = fetch_openml(data_id=31, as_frame=True, parser="pandas")
+X, y = german_credit.data, german_credit.target
+
+# %%
+# We check the feature types available in `X`.
+X.info()
+
+# %%
+X.head()
+
+# %%
+# Many features are categorical and usually string-encoded. We need to encode
+# these categories when we develop ou predictive model. Let's check the targets.
+y.value_counts()
+
+# %%
+# A first observation is that the dataset is imbalanced. We would need to be careful
+# when evaluating our predictive model and use a family of metrics that are adapted
+# to this setting.
+#
+# In addition, we observe that the target is string-encoded. Some metrics
+# (e.g. precision and recall) require to provide the label of interest also called
+# the "positive label". Here, we define that our goal is to predict whether or not
+# a sample is a "bad" credit.
+pos_label = "bad"
+
+# %%
+# To carry our analysis, we split our dataset using a single stratified split.
+from sklearn.model_selection import train_test_split
+
+X_train, X_test, y_train, y_test = train_test_split(X, y, stratify=y, random_state=0)
+
+# %%
+# We are ready to design our predictive model and the associated evaluation strategy.
+#
+# Evaluation metrics
+# ------------------
+#
+# In this section, we define a set of metrics that we use later. To see
+# the effect of tuning the cut-off point, we evaluate the predictive model using
+# the Receiver Operating Characteristic (ROC) curve and the Precision-Recall curve.
+# The values reported on these plots are therefore the true positive rate (TPR) and
+# the false positive rate (FPR) for the ROC curve and the precision and recall for the
+# Precision-Recall curve.
+#
+# From these four metrics, scikit-learn does not provide a scorer for the FPR. We
+# therefore need to define a small custom function to compute it.
+import numpy as np
+from sklearn.metrics import confusion_matrix, make_scorer, precision_score, recall_score
+
+
+def fpr_score(y, y_pred, **kwargs):
+    cm = confusion_matrix(y, y_pred)
+    classes = np.unique(y)
+    pos_label = kwargs.get("pos_label", classes[-1])
+    pos_label_idx = np.searchsorted(classes, pos_label)
+    if pos_label_idx == 0:
+        cm = cm[::-1, ::-1]
+    tn, fp, _, _ = cm.ravel()
+    tnr = tn / (tn + fp)
+    return 1 - tnr
+
+
+# %%
+# As previously stated, the "positive label" is not defined as the value "1" and calling
+# some of the metrics with this non-standard value raise an error. We need to
+# provide the indication of the "positive label" to the metrics.
+#
+# We therefore need to define a scikit-learn scorer using
+# :func:`~sklearn.metrics.make_scorer` where the information is passed. We store all
+# created scorer in a dictionary. To use them, we need to pass the fitted model and
+# the data and target on which we want to evaluate the predictive model.
+tpr_score = recall_score  # TPR and recall are the same metric
+scoring = {
+    "precision": make_scorer(precision_score, pos_label=pos_label),
+    "recall": make_scorer(recall_score, pos_label=pos_label),
+    "fpr": make_scorer(fpr_score, pos_label=pos_label),
+    "tpr": make_scorer(tpr_score, pos_label=pos_label),
+}
+
+# %%
+# In addition, the original research [1]_ defines a business metric. They provide a
+# cost-matrix where they define that predicting a "bad" credit as "good" is 5 times more
+# costly than the opposite. We define a dictionary containing this information and a
+# score function that computes the cost.
+cost_gain_matrix = {"tp": 0, "tn": 0, "fp": -1, "fn": -5}
+
+
+def gain_cost_score(y, y_pred, **kwargs):
+    cm = confusion_matrix(y, y_pred)
+    classes = np.unique(y)
+    pos_label = kwargs.get("pos_label", classes[-1])
+    pos_label_idx = np.searchsorted(classes, pos_label)
+    if pos_label_idx == 0:
+        cm = cm[::-1, ::-1]
+    costs_and_gain = np.array(
+        [
+            [kwargs["cost_gain_matrix"]["tn"], kwargs["cost_gain_matrix"]["fp"]],
+            [kwargs["cost_gain_matrix"]["fn"], kwargs["cost_gain_matrix"]["tp"]],
+        ]
+    )
+    return (costs_and_gain * cm).sum()
+
+
+scoring["cost_gain"] = make_scorer(
+    gain_cost_score, pos_label=pos_label, cost_gain_matrix=cost_gain_matrix
+)
+# %%
+# Vanilla predictive model
+# ------------------------
+#
+# Design of the predictive model
+# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+#
+# In this section, we design our predictive model. We use a gradient boosting classifier
+# using :class:`~sklearn.ensemble.HistGradientBoostingClassifier`. However, we need to
+# encode the categorical features with numerical values and we therefore use an
+# :class:`~sklearn.preprocessing.OrdinalEncoder` to do so. The numerical features are
+# kept as-is. To recover the name of the categorical columns, we use the helper function
+# :func:`~sklearn.compose.make_column_selector` and the fact that the categorical
+# features are stored as `category` dtype.
+from sklearn.compose import ColumnTransformer, make_column_selector as selector
+from sklearn.ensemble import HistGradientBoostingClassifier
+from sklearn.pipeline import Pipeline
+from sklearn.preprocessing import OrdinalEncoder
+
+categorical_columns = selector(dtype_include="category")(X)
+numerical_columns = selector(dtype_exclude="category")(X)
+
+preprocessor = ColumnTransformer(
+    [
+        (
+            "categorical",
+            OrdinalEncoder(handle_unknown="use_encoded_value", unknown_value=-1),
+            categorical_columns,
+        ),
+        ("numerical", "passthrough", numerical_columns),
+    ],
+    verbose_feature_names_out=False,
+)
+model = Pipeline(
+    [
+        ("preprocessor", preprocessor),
+        (
+            "classifier",
+            HistGradientBoostingClassifier(
+                categorical_features=categorical_columns, random_state=0
+            ),
+        ),
+    ]
+)
+
+model.fit(X_train, y_train)
+
+# %%
+# Evaluation of the predictive model
+# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+#
+# We evaluate the performance of our predictive model using the ROC and Precision-Recall
+# curves.
+import matplotlib.pyplot as plt
+from sklearn.metrics import PrecisionRecallDisplay, RocCurveDisplay
+
+fig, axs = plt.subplots(nrows=1, ncols=2, figsize=(14, 6))
+
+PrecisionRecallDisplay.from_estimator(
+    model, X_test, y_test, pos_label=pos_label, ax=axs[0], name="GBDT"
+)
+axs[0].plot(
+    scoring["recall"](model, X_test, y_test),
+    scoring["precision"](model, X_test, y_test),
+    marker="o",
+    markersize=10,
+    color="tab:blue",
+    label="Default cut-off point at a probability of 0.5",
+)
+axs[0].set_title("Precision-Recall curve")
+axs[0].legend()
+
+RocCurveDisplay.from_estimator(
+    model,
+    X_test,
+    y_test,
+    pos_label=pos_label,
+    ax=axs[1],
+    name="GBDT",
+    plot_chance_level=True,
+)
+axs[1].plot(
+    scoring["fpr"](model, X_test, y_test),
+    scoring["tpr"](model, X_test, y_test),
+    marker="o",
+    markersize=10,
+    color="tab:blue",
+    label="Default cut-off point at a probability of 0.5",
+)
+axs[1].set_title("ROC curve")
+axs[1].legend()
+_ = fig.suptitle("Evaluation of the vanilla GBDT model")
+
+# %%
+# We recall that these curves corresponds show the statistical performance of the
+# predictive model for different cut-off points. For the Precision-Recall curve, the
+# reported metrics are the precision and recall and for the ROC curve, the reported
+# metrics are the TPR (same as recall) and FPR.
+#
+# Here, the different cut-off points corresponds at different level of probability
+# scores ranging between 0 and 1. By default, `model.predict` uses a cut-off point at
+# a probability of 0.5. Thus, the metrics for this cut-off point is reported with the
+# blue dot on the curves: it corresponds to the statistical performance of the model
+# when using `model.predict`.
+#
+# However, we recall that the original aim was to minimize the cost (or maximize the
+# gain) by the business metric. We can compute the value of the business metric:
+scoring["cost_gain"](model, X_test, y_test)
+
+# %%
+# At this stage, we don't know if some other cut-off points could lead to a greater
+# gain. To be able to decide, if another cut-off point is better, we would need to
+# compute the cost-gain using the business metric for all possible cut-off points and
+# choose the optimal one. This strategy is quite tedious to implement and the
+# :class:`~sklearn.metrics.CutOffClassifier` class is here to help us. It automatically
+# compute the cost-gain for all possible cut-off points and choose the optimal one.
+#
+# .. _cost_sensitive_learning_example:
+#
+# Tuning the cut-off point
+# ------------------------
+#
+# We use :class:`~sklearn.metrics.CutOffClassifier` to tune the cut-off point. We need
+# to provide the business metric to optimize as well as the positive label. Internally,
+# the optimum cut-off point is chosen such that it maximized the business metric
+# via cross-validation. By default a 5-fold stratified cross-validation is used.
+from sklearn.model_selection import CutOffClassifier
+
+model_tuned = CutOffClassifier(
+    estimator=model,
+    pos_label=pos_label,
+    objective_metric=cost_gain_matrix,
+)
+model_tuned.fit(X_train, y_train)
+
+# %%
+# We plot the ROC and Precision-Recall curves for the vanilla model and the tuned model.
+# Also we plot the cut-off points that would be used by each model.
+fig, axs = plt.subplots(nrows=1, ncols=3, figsize=(21, 6))
path: examples/model_selection/plot_cutoff_tuning.py
text: @user1:
In the rendering this figure is too small, can we make it a bit bigger?