Repository: scikit-learn/scikit-learn
Pull request: ENH Add CalibrationDisplay plotting class
<!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#pull-request-checklist
-->

#### Reference Issues/PRs
closes #8425

#### What does this implement/fix? Explain your changes.
Adds `CalibrationDisplay` for binary classifiers with visualization API

#### Any other comments?
Plot currently looks like this:
![image](https://user-images.githubusercontent.com/23182829/83751061-836c2880-a666-11ea-95d2-3d1c20ccb0a5.png)

Yet to add tests and update examples as I am unsure about API:

Should a histogram be added? If so should the histogram be a separate plot (e.g., [here](https://scikit-learn.org/stable/auto_examples/calibration/plot_compare_calibration.html)) or on the same plot (as suggested by @amueller: https://github.com/scikit-learn/scikit-learn/issues/8425#issuecomment-281485024).
If we put it on the same plot I'm worried it will be too crowded. If we want 2 separate plots, the API becomes difficult as we should have 2 different plot `**kwargs` parameters, for both plots so people can amend them separately. You also couldn't use `ax = plt.gca()` to get the current axis when you want to add lines to an existing plot (like for the current plots, e.g., [here](https://scikit-learn.org/dev/visualizations.html)). I think you could use `CalibrationDisplay.ax_` though.


Problem statement:
Add calibration curve to plotting module
I think it would be nice to have something like

```python
def plot_calibration_curve(y_true, prob_y_pred, n_bins=5, ax=None):
    prop_true, prop_pred = calibration_curve(y_true, prob_y_pred, n_bins=n_bins)
    if ax is None:
        ax = plt.gca()
    ax.plot([0, 1], [0, 1], ':', c='k')
    ax.plot(prop_pred, prop_true)

    ax.set_xlabel("predicted probability")
    ax.set_ylabel("fraction of positive samples")
    ax.set_aspect("equal")
```

I got confused for a bit because the order of the return arguments is not the order of the arguments to plot. I think a plot function even just labeling the axes right would decrease mental effort (yes, I'm lazy).

Hints from issue discussion:
should plot functions return the artist? or the axes? hm...
Maybe adding a histogram as in http://scikit-learn.org/stable/auto_examples/calibration/plot_compare_calibration.html would also be nice
```python
def plot_calibration_curve(y_true, y_prob, n_bins=5, ax=None, hist=True, normalize=False):
    prob_true, prob_pred = calibration_curve(y_true, y_prob, n_bins=n_bins, normalize=normalize)
    if ax is None:
        ax = plt.gca()
    if hist:
        if normalize:
            y_prob = (y_prob - y_prob.min()) / (y_prob.max() - y_prob.min())
        ax.hist(y_prob, weights=np.ones_like(y_prob) / len(y_prob), alpha=.4,
               bins=np.maximum(10, n_bins))
    ax.plot([0, 1], [0, 1], ':', c='k')
    curve = ax.plot(prob_pred, prob_true, marker="o")

    ax.set_xlabel("predicted probability")
    ax.set_ylabel("fraction of positive samples")

    ax.set(aspect='equal')
    return curve

```
wow this took me an hour... and I'm not friends with ``aspect='equal'`` any more
Like to work on it but bit confused with `plotting module` . Is there any separate module for plotting in sklearn or it just about an example like this : http://scikit-learn.org/stable/auto_examples/calibration/plot_calibration_curve.html#sphx-glr-auto-examples-calibration-plot-calibration-curve-py
> should plot functions return the artist? or the axes? hm...

I wasn't aware of this until recently but this is the recommended signature from http://matplotlib.org/faq/usage_faq.html#coding-styles:

```py
def my_plotter(ax, data1, data2, param_dict):
    """
    A helper function to make a graph

    Parameters
    ----------
    ax : Axes
        The axes to draw to

    data1 : array
       The x data

    data2 : array
       The y data

    param_dict : dict
       Dictionary of kwargs to pass to ax.plot

    Returns
    -------
    out : list
        list of artists added
    """
    out = ax.plot(data1, data2, **param_dict)
    return out
```

I guess this is close enough to what you are doing in https://github.com/scikit-learn/scikit-learn/issues/8425#issuecomment-281485024.
There is (or will be if none of the PRs have been merge yet) ``sklearn.plot``
@amueller May be it's `will be` because I haven't found `plot` directory inside `sklearn`.
hm yeah I'm behind on everything :-/
Hi I am new to contributing to scikit-learn but I would like to work on this! Is anyone else working on this?
I don't think a PR has been opened, though @rishikksh20 has clearly expressed some interest

Commit message: add reliability diagram

Patch to review:
diff --git a/doc/modules/calibration.rst b/doc/modules/calibration.rst
index cfc185c854edb..912920e495587 100644
--- a/doc/modules/calibration.rst
+++ b/doc/modules/calibration.rst
@@ -30,11 +30,21 @@ approximately 80% actually belong to the positive class.
 Calibration curves
 ------------------
 
-The following plot compares how well the probabilistic predictions of
-different classifiers are calibrated, using :func:`calibration_curve`.
+Calibration curves (also known as reliability diagrams) compare how well the
+probabilistic predictions of a binary classifier are calibrated. It plots
+the true frequency of the positive label against its predicted probability,
+for binned predictions.
 The x axis represents the average predicted probability in each bin. The
 y axis is the *fraction of positives*, i.e. the proportion of samples whose
-class is the positive class (in each bin).
+class is the positive class (in each bin). The top calibration curve plot
+is created with :func:`plot_calibration_curve`, which uses
+:func:`calibration_curve` to calculate the per bin average predicted
+probabilities and fraction of positives. :func:`plot_calibration_curve`
+takes as input a fitted classifier, used to calculate the predicted
+probabilities. The classifier thus must have :term:`predict_proba` method. For
+the few classifiers that do not have a :term:`predict_proba` method, it is
+possible to use :class:`CalibratedClassifierCV` to calibrate the classifier
+outputs to probabilities.
 
 .. figure:: ../auto_examples/calibration/images/sphx_glr_plot_compare_calibration_001.png
    :target: ../auto_examples/calibration/plot_compare_calibration.html
diff --git a/doc/modules/classes.rst b/doc/modules/classes.rst
index 2e54d000a13aa..4e07ac0bcc2a5 100644
--- a/doc/modules/classes.rst
+++ b/doc/modules/classes.rst
@@ -77,6 +77,19 @@ Functions
 
    calibration.calibration_curve
 
+.. autosummary::
+   :toctree: generated/
+   :template: class.rst
+
+   calibration.CalibrationDisplay
+
+
+.. autosummary::
+   :toctree: generated/
+   :template: function.rst
+
+   calibration.plot_calibration_curve
+
 .. _cluster_ref:
 
 :mod:`sklearn.cluster`: Clustering
diff --git a/doc/visualizations.rst b/doc/visualizations.rst
index ebb98700d9e08..ff2bd1505c05b 100644
--- a/doc/visualizations.rst
+++ b/doc/visualizations.rst
@@ -61,6 +61,7 @@ values of the curves.
     * :ref:`sphx_glr_auto_examples_miscellaneous_plot_roc_curve_visualization_api.py`
     * :ref:`sphx_glr_auto_examples_miscellaneous_plot_partial_dependence_visualization_api.py`
     * :ref:`sphx_glr_auto_examples_miscellaneous_plot_display_object_visualization.py`
+    * :ref:`sphx_glr_auto_examples_calibration_plot_compare_calibration.py`
 
 Available Plotting Utilities
 ============================
@@ -72,6 +73,7 @@ Functions
 
 .. autosummary::
 
+   calibration.plot_calibration_curve
    inspection.plot_partial_dependence
    metrics.plot_confusion_matrix
    metrics.plot_precision_recall_curve
@@ -85,6 +87,7 @@ Display Objects
 
 .. autosummary::
 
+   calibration.CalibrationDisplay
    inspection.PartialDependenceDisplay
    metrics.ConfusionMatrixDisplay
    metrics.PrecisionRecallDisplay
diff --git a/doc/whats_new/v0.24.rst b/doc/whats_new/v0.24.rst
index 53cbe13161151..c90e442ee575c 100644
--- a/doc/whats_new/v0.24.rst
+++ b/doc/whats_new/v0.24.rst
@@ -48,6 +48,9 @@ Changelog
 :mod:`sklearn.calibration`
 ..........................
 
+- |Feature| :func:`calibration.plot_calibration_curve` added to plot
+  calibration curves. :pr:`17443` by :user:`Lucy Liu <lucyleeow>`.
+
 - |Efficiency| :class:`calibration.CalibratedClassifierCV.fit` now supports
   parallelization via `joblib.Parallel` using argument `n_jobs`.
   :pr:`17107` by :user:`Julien Jerphanion <jjerphan>`.
diff --git a/examples/calibration/plot_calibration_curve.py b/examples/calibration/plot_calibration_curve.py
index 3218e5c223343..ae335dc43e54d 100644
--- a/examples/calibration/plot_calibration_curve.py
+++ b/examples/calibration/plot_calibration_curve.py
@@ -5,130 +5,290 @@
 
 When performing classification one often wants to predict not only the class
 label, but also the associated probability. This probability gives some
-kind of confidence on the prediction. This example demonstrates how to display
-how well calibrated the predicted probabilities are and how to calibrate an
-uncalibrated classifier.
-
-The experiment is performed on an artificial dataset for binary classification
-with 100,000 samples (1,000 of them are used for model fitting) with 20
-features. Of the 20 features, only 2 are informative and 10 are redundant. The
-first figure shows the estimated probabilities obtained with logistic
-regression, Gaussian naive Bayes, and Gaussian naive Bayes with both isotonic
-calibration and sigmoid calibration. The calibration performance is evaluated
-with Brier score, reported in the legend (the smaller the better). One can
-observe here that logistic regression is well calibrated while raw Gaussian
-naive Bayes performs very badly. This is because of the redundant features
-which violate the assumption of feature-independence and result in an overly
-confident classifier, which is indicated by the typical transposed-sigmoid
-curve.
-
-Calibration of the probabilities of Gaussian naive Bayes with isotonic
-regression can fix this issue as can be seen from the nearly diagonal
-calibration curve. Sigmoid calibration also improves the brier score slightly,
-albeit not as strongly as the non-parametric isotonic regression. This can be
-attributed to the fact that we have plenty of calibration data such that the
-greater flexibility of the non-parametric model can be exploited.
-
-The second figure shows the calibration curve of a linear support-vector
-classifier (LinearSVC). LinearSVC shows the opposite behavior as Gaussian
-naive Bayes: the calibration curve has a sigmoid curve, which is typical for
-an under-confident classifier. In the case of LinearSVC, this is caused by the
-margin property of the hinge loss, which lets the model focus on hard samples
-that are close to the decision boundary (the support vectors).
-
-Both kinds of calibration can fix this issue and yield nearly identical
-results. This shows that sigmoid calibration can deal with situations where
-the calibration curve of the base classifier is sigmoid (e.g., for LinearSVC)
-but not where it is transposed-sigmoid (e.g., Gaussian naive Bayes).
+kind of confidence on the prediction. This example demonstrates how to
+visualize how well calibrated the predicted probabilities are using calibration
+curves, also known as reliability diagrams. Calibration of an uncalibrated
+classifier will also be demonstrated.
 """
-print(__doc__)
+
+# %%
 
 # Author: Alexandre Gramfort <alexandre.gramfort@telecom-paristech.fr>
 #         Jan Hendrik Metzen <jhm@informatik.uni-bremen.de>
 # License: BSD Style.
 
 import matplotlib.pyplot as plt
+from matplotlib.gridspec import GridSpec
+import pandas as pd
+import numpy as np
 
-from sklearn import datasets
+from sklearn.datasets import make_classification
+from sklearn.model_selection import train_test_split
 from sklearn.naive_bayes import GaussianNB
 from sklearn.svm import LinearSVC
 from sklearn.linear_model import LogisticRegression
-from sklearn.metrics import (brier_score_loss, precision_score, recall_score,
-                             f1_score)
-from sklearn.calibration import CalibratedClassifierCV, calibration_curve
-from sklearn.model_selection import train_test_split
+from sklearn.metrics import (precision_score, recall_score, f1_score,
+                             brier_score_loss, log_loss)
+from sklearn.calibration import CalibratedClassifierCV, plot_calibration_curve
 
+# %%
+# Dataset
+# -------
+#
+# We will use a synthetic binary classification dataset with 100,000 samples
+# and 20 features. Of the 20 features, only 2 are informative, 10 are
+# redundant (random combinations of the informative features) and the
+# remaining 8 are 'useless' (random numbers). Of the 100,000 samples, 1,000
+# will be used for model fitting and the rest for testing.
 
-# Create dataset of classification task with many redundant and few
-# informative features
-X, y = datasets.make_classification(n_samples=100000, n_features=20,
-                                    n_informative=2, n_redundant=10,
-                                    random_state=42)
+X, y = make_classification(n_samples=100000, n_features=20, n_informative=2,
+                           n_redundant=10, random_state=42)
 
 X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.99,
                                                     random_state=42)
 
+# %%
+# Calibration curves
+# ------------------
+#
+# First, we will compare:
+#
+# * :class:`~sklearn.linear_model.LogisticRegression` (used as baseline
+#   since very often, properly regularized logistic regression is well
+#   calibrated by default thanks to the use of log-loss)
+# * Raw :class:`~sklearn.naive_bayes.GaussianNB`
+# * :class:`~sklearn.naive_bayes.GaussianNB` with isotonic and sigmoid
+#   calibration (see :ref:`User Guide <calibration>`)
+#
+# Calibration curves for all 4 conditions are plotted below, with the average
+# predicted probability for each bin on the x-axis and the fraction of positive
+# classes in each bin on the y-axis.
+#
+
+lr = LogisticRegression(C=1.)
+gnb = GaussianNB()
+gnb_isotonic = CalibratedClassifierCV(gnb, cv=2, method='isotonic')
+gnb_sigmoid = CalibratedClassifierCV(gnb, cv=2, method='sigmoid')
+
+clf_list = [(lr, 'Logistic'),
+            (gnb, 'Naive Bayes'),
+            (gnb_isotonic, 'Naive Bayes + Isotonic'),
+            (gnb_sigmoid, 'Naive Bayes + Sigmoid')]
+
+fig = plt.figure(figsize=(10, 10))
+gs = GridSpec(4, 2)
+colors = plt.cm.get_cmap('Dark2')
+
+ax1 = fig.add_subplot(gs[:2, :2])
+viz_objects = {}
+for i, (clf, name) in enumerate(clf_list):
+    clf.fit(X_train, y_train)
+    viz = plot_calibration_curve(
+        clf, X_test, y_test, n_bins=10, name=name, ax=ax1, color=colors(i)
+    )
+    viz_objects[name] = viz
+
+ax1.grid()
+ax1.set_title('Calibration plots (Naive Bayes)')
+
+# Add histogram
+grid_positions = [(2, 0), (2, 1), (3, 0), (3, 1)]
+for i, (_, name) in enumerate(clf_list):
+    row, col = grid_positions[i]
+    ax = fig.add_subplot(gs[row, col])
+
+    ax.hist(
+        viz_objects[name].y_prob, range=(0, 1), bins=10, label=name,
+        color=colors(i)
+    )
+    ax.set(title=name, xlabel="Mean predicted probability", ylabel="Count")
+
+plt.tight_layout()
+plt.show()
+
+# %%
+# Raw :class:`~sklearn.naive_bayes.GaussianNB` performs very badly because of
+# the redundant features which violate the assumption of feature-independence
+# and result in an overly confident classifier, which is indicated by the
+# typical transposed-sigmoid curve. Calibration of the probabilities of
+# :class:`~sklearn.naive_bayes.GaussianNB` with isotonic regression can fix
+# this issue as can be seen from the nearly diagonal calibration curve. Sigmoid
+# regression also improves calibration slightly,
+# albeit not as strongly as the non-parametric isotonic regression. This can be
+# attributed to the fact that we have plenty of calibration data such that the
+# greater flexibility of the non-parametric model can be exploited.
+#
+# Below we show the Brier score, log loss, precision, recall and F1 score (see
+# :ref:`User Guide <precision_recall_f_measure_metrics>`). Notice that
+# although calibration improves the Brier score (a metric composed of
+# calibration loss and refinement loss) and :ref:`log_loss`, it does not
+# significantly alter the prediction accuracy measures (precision, recall and
+# F1 score).
+# This is because calibration should not significantly change prediction
+# probabilities at the location of the decision threshold (at x = 0.5 on the
+# graph). Calibration should however, make the predicted probabilities more
+# accurate and thus more useful for making allocation decisions under
+# uncertainty.
+
+index = []
+brier = []
+logloss = []
+precision = []
+recall = []
+f1 = []
+
+for i, (clf, name) in enumerate(clf_list):
+    clf.fit(X_train, y_train)
+    y_proba = clf.predict_proba(X_test)
+    y_pred = clf.predict(X_test)
+
+    # Create DataFrame index
+    index.append(name)
+    # Store column data
+    brier.append(brier_score_loss(y_test, y_proba[:, 1]))
+    logloss.append(log_loss(y_test, y_proba[:, 1]))
+    precision.append(precision_score(y_test, y_pred))
+    recall.append(recall_score(y_test, y_pred))
+    f1.append(f1_score(y_test, y_pred))
+
+score_df = pd.DataFrame(
+    data={'Brier score': brier, 'Log loss': logloss, 'Precision': precision,
+          'Recall': recall, 'F1': f1},
+    index=index,
+)
+score_df.round(3)
+
+# %%
+# Next, we will compare:
+#
+# * :class:`~sklearn.linear_model.LogisticRegression` (baseline)
+# * :class:`~sklearn.svm.LinearSVC` (:term:`decision_function` scores
+#   min-max scaled to [0,1], since they are not proper probabilities)
+# * :class:`~sklearn.svm.LinearSVC` with isotonic and sigmoid
+#   calibration (see :ref:`User Guide <calibration>`)
+#
 
-def plot_calibration_curve(est, name, fig_index):
-    """Plot calibration curve for est w/o and with calibration. """
-    # Calibrated with isotonic calibration
-    isotonic = CalibratedClassifierCV(est, cv=2, method='isotonic')
 
-    # Calibrated with sigmoid calibration
-    sigmoid = CalibratedClassifierCV(est, cv=2, method='sigmoid')
+class NaivelyCalibratedLinearSVC(LinearSVC):
+    """LinearSVC with `predict_proba` method that naively scales
+    `decision_function` output for binary classification."""
 
-    # Logistic regression with no calibration as baseline
-    lr = LogisticRegression(C=1.)
+    def fit(self, X, y):
+        super().fit(X, y)
+        df = self.decision_function(X)
+        self.df_min_ = df.min()
+        self.df_max_ = df.max()
 
-    fig = plt.figure(fig_index, figsize=(10, 10))
-    ax1 = plt.subplot2grid((3, 1), (0, 0), rowspan=2)
-    ax2 = plt.subplot2grid((3, 1), (2, 0))
+    def predict_proba(self, X):
+        """Min-max scale output of `decision_function` to [0,1].
 
-    ax1.plot([0, 1], [0, 1], "k:", label="Perfectly calibrated")
-    for clf, name in [(lr, 'Logistic'),
-                      (est, name),
-                      (isotonic, name + ' + Isotonic'),
-                      (sigmoid, name + ' + Sigmoid')]:
-        clf.fit(X_train, y_train)
-        y_pred = clf.predict(X_test)
-        if hasattr(clf, "predict_proba"):
-            prob_pos = clf.predict_proba(X_test)[:, 1]
-        else:  # use decision function
-            prob_pos = clf.decision_function(X_test)
-            prob_pos = \
-                (prob_pos - prob_pos.min()) / (prob_pos.max() - prob_pos.min())
+        Parameters
+        ----------
+        X : ndarray of shape (n_samples, n_features)
+            The data.
 
-        clf_score = brier_score_loss(y_test, prob_pos, pos_label=y.max())
-        print("%s:" % name)
-        print("\tBrier: %1.3f" % (clf_score))
-        print("\tPrecision: %1.3f" % precision_score(y_test, y_pred))
-        print("\tRecall: %1.3f" % recall_score(y_test, y_pred))
-        print("\tF1: %1.3f\n" % f1_score(y_test, y_pred))
+        Returns
+        -------
+        proba : ndarray of shape (n_samples, 2)
+            Predicted probabilities of the 2 classes.
+        """
+        df = self.decision_function(X)
+        calibrated_df = (df - self.df_min_) / (self.df_max_ - self.df_min_)
+        proba_class1 = np.clip(calibrated_df, 0, 1)
+        proba_class0 = 1 - proba_class1
+        proba = np.c_[proba_class0, proba_class1]
+        return proba
 
-        fraction_of_positives, mean_predicted_value = \
-            calibration_curve(y_test, prob_pos, n_bins=10)
 
-        ax1.plot(mean_predicted_value, fraction_of_positives, "s-",
-                 label="%s (%1.3f)" % (name, clf_score))
+lr = LogisticRegression(C=1.)
+svc = NaivelyCalibratedLinearSVC(max_iter=10000)
+svc_isotonic = CalibratedClassifierCV(svc, cv=2, method='isotonic')
+svc_sigmoid = CalibratedClassifierCV(svc, cv=2, method='sigmoid')
 
-        ax2.hist(prob_pos, range=(0, 1), bins=10, label=name,
-                 histtype="step", lw=2)
+clf_list = [(lr, 'Logistic'),
+            (svc, 'SVC'),
+            (svc_isotonic, 'SVC + Isotonic'),
+            (svc_sigmoid, 'SVC + Sigmoid')]
 
-    ax1.set_ylabel("Fraction of positives")
-    ax1.set_ylim([-0.05, 1.05])
-    ax1.legend(loc="lower right")
-    ax1.set_title('Calibration plots  (reliability curve)')
+fig = plt.figure(figsize=(10, 10))
+gs = GridSpec(4, 2)
 
-    ax2.set_xlabel("Mean predicted value")
-    ax2.set_ylabel("Count")
-    ax2.legend(loc="upper center", ncol=2)
+ax1 = fig.add_subplot(gs[:2, :2])
+viz_objects = {}
+for i, (clf, name) in enumerate(clf_list):
+    clf.fit(X_train, y_train)
+    viz = plot_calibration_curve(
+        clf, X_test, y_test, n_bins=10, name=name, ax=ax1, color=colors(i)
+    )
+    viz_objects[name] = viz
 
-    plt.tight_layout()
+ax1.grid()
+ax1.set_title('Calibration plots (SVC)')
 
-# Plot calibration curve for Gaussian Naive Bayes
-plot_calibration_curve(GaussianNB(), "Naive Bayes", 1)
+# Add histogram
+grid_positions = [(2, 0), (2, 1), (3, 0), (3, 1)]
+for i, (_, name) in enumerate(clf_list):
+    row, col = grid_positions[i]
+    ax = fig.add_subplot(gs[row, col])
 
-# Plot calibration curve for Linear SVC
-plot_calibration_curve(LinearSVC(max_iter=10000), "SVC", 2)
+    ax.hist(
+        viz_objects[name].y_prob, range=(0, 1), bins=10, label=name,
+        color=colors(i)
+    )
+    ax.set(title=name, xlabel="Mean predicted probability", ylabel="Count")
 
+plt.tight_layout()
 plt.show()
+
+# %%
+# :class:`~sklearn.svm.LinearSVC` shows the opposite
+# behavior to :class:`~sklearn.naive_bayes.GaussianNB`; the calibration
+# curve has a sigmoid shape, which is typical for an under-confident
+# classifier. In the case of :class:`~sklearn.svm.LinearSVC`, this is caused
+# by the margin property of the hinge loss, which lets the model focus on
+# difficult samples that are close to the decision boundary (the support
+# vectors).
+#
+# Both kinds of calibration (sigmoid and isotonic) can fix this issue and
+# yield nearly identical results.
+#
+# As before, we show the Brier score, precision, recall and F1 score below.
+
+index = []
+brier = []
+logloss = []
+precision = []
+recall = []
+f1 = []
+
+for i, (clf, name) in enumerate(clf_list):
+    clf.fit(X_train, y_train)
+    y_proba = clf.predict_proba(X_test)
+    y_pred = clf.predict(X_test)
+
+    # Create DataFrame index
+    index.append(name)
+    # Store column data
+    brier.append(brier_score_loss(y_test, y_proba[:, 1]))
+    logloss.append(log_loss(y_test, y_proba[:, 1]))
+    precision.append(precision_score(y_test, y_pred))
+    recall.append(recall_score(y_test, y_pred))
+    f1.append(f1_score(y_test, y_pred))
+
+score_df = pd.DataFrame(
+    data={'Brier score': brier, 'Log loss': logloss, 'Precision': precision,
+          'Recall': recall, 'F1': f1},
+    index=index,
+)
+score_df.round(3)
+
+# %%
+# Summary
+# -------
+#
+# Parametric sigmoid calibration can deal with situations where the calibration
+# curve of the base classifier is sigmoid (e.g., for
+# :class:`~sklearn.svm.LinearSVC`) but not where it is transposed-sigmoid
+# (e.g., :class:`~sklearn.naive_bayes.GaussianNB`). Non-parametric
+# isotonic calibration can deal with both situations but may require more
+# data to produce good results.
diff --git a/examples/calibration/plot_compare_calibration.py b/examples/calibration/plot_compare_calibration.py
index a8599aecc16af..6951fa44d214c 100644
--- a/examples/calibration/plot_compare_calibration.py
+++ b/examples/calibration/plot_compare_calibration.py
@@ -4,119 +4,155 @@
 ========================================
 
 Well calibrated classifiers are probabilistic classifiers for which the output
-of the predict_proba method can be directly interpreted as a confidence level.
-For instance a well calibrated (binary) classifier should classify the samples
-such that among the samples to which it gave a predict_proba value close to
-0.8, approx. 80% actually belong to the positive class.
-
-LogisticRegression returns well calibrated predictions as it directly
-optimizes log-loss. In contrast, the other methods return biased probabilities,
-with different biases per method:
-
-* GaussianNaiveBayes tends to push probabilities to 0 or 1 (note the counts in
-  the histograms). This is mainly because it makes the assumption that features
-  are conditionally independent given the class, which is not the case in this
-  dataset which contains 2 redundant features.
-
-* RandomForestClassifier shows the opposite behavior: the histograms show
-  peaks at approx. 0.2 and 0.9 probability, while probabilities close to 0 or 1
-  are very rare. An explanation for this is given by Niculescu-Mizil and Caruana
-  [1]_: "Methods such as bagging and random forests that average predictions
-  from a base set of models can have difficulty making predictions near 0 and 1
-  because variance in the underlying base models will bias predictions that
-  should be near zero or one away from these values. Because predictions are
-  restricted to the interval [0,1], errors caused by variance tend to be one-
-  sided near zero and one. For example, if a model should predict p = 0 for a
-  case, the only way bagging can achieve this is if all bagged trees predict
-  zero. If we add noise to the trees that bagging is averaging over, this noise
-  will cause some trees to predict values larger than 0 for this case, thus
-  moving the average prediction of the bagged ensemble away from 0. We observe
-  this effect most strongly with random forests because the base-level trees
-  trained with random forests have relatively high variance due to feature
-  subsetting." As a result, the calibration curve shows a characteristic
-  sigmoid shape, indicating that the classifier could trust its "intuition"
-  more and return probabilities closer to 0 or 1 typically.
-
-* Support Vector Classification (SVC) shows an even more sigmoid curve as
-  the  RandomForestClassifier, which is typical for maximum-margin methods
-  (compare Niculescu-Mizil and Caruana [1]_), which focus on hard samples
-  that are close to the decision boundary (the support vectors).
-
-.. topic:: References:
-
-    .. [1] Predicting Good Probabilities with Supervised Learning,
-          A. Niculescu-Mizil & R. Caruana, ICML 2005
+of :term:`predict_proba` can be directly interpreted as a confidence level.
+For instance, a well calibrated (binary) classifier should classify the samples
+such that for the samples to which it gave a :term:`predict_proba` value close
+to 0.8, approximately 80% actually belong to the positive class.
 """
-print(__doc__)
+
+# %%
 
 # Author: Jan Hendrik Metzen <jhm@informatik.uni-bremen.de>
 # License: BSD Style.
 
-import numpy as np
-np.random.seed(0)
-
 import matplotlib.pyplot as plt
+from matplotlib.gridspec import GridSpec
+import numpy as np
 
-from sklearn import datasets
+from sklearn.datasets import make_classification
 from sklearn.naive_bayes import GaussianNB
 from sklearn.linear_model import LogisticRegression
 from sklearn.ensemble import RandomForestClassifier
 from sklearn.svm import LinearSVC
-from sklearn.calibration import calibration_curve
-
-X, y = datasets.make_classification(n_samples=100000, n_features=20,
-                                    n_informative=2, n_redundant=2)
+from sklearn.calibration import plot_calibration_curve
+
+# %%
+# Dataset
+# -------
+#
+# We will use a synthetic binary classification dataset with 100,000 samples
+# and 20 features. Of the 20 features, only 2 are informative, 2 are
+# redundant (random combinations of the informative features) and the
+# remaining 16 are 'useless' (random numbers). Of the 100,000 samples,
+# 100 will be used for model fitting and the remaining for testing.
+
+X, y = make_classification(
+  n_samples=100000, n_features=20, n_informative=2, n_redundant=2,
+  random_state=42
+)
 
 train_samples = 100  # Samples used for training the models
-
 X_train = X[:train_samples]
 X_test = X[train_samples:]
 y_train = y[:train_samples]
 y_test = y[train_samples:]
 
+# %%
+# Calibration curves
+# ------------------
+#
+# :class:`~sklearn.linear_model.LogisticRegression` returns well calibrated
+# predictions as it directly optimizes log-loss. In contrast, the other methods
+# return biased probabilities, with different biases per method:
+#
+# * :class:`~sklearn.naive_bayes.GaussianNB` tends to push
+#   probabilities to 0 or 1 (see histogram). This is mainly
+#   because it makes the assumption that features are conditionally independent
+#   given the class, which is not the case in this dataset which contains 2
+#   redundant features.
+#
+# * :class:`~sklearn.ensemble.RandomForestClassifier` shows the opposite
+#   behavior: the histograms show peaks at approx. 0.2 and 0.9 probability,
+#   while probabilities close to 0 or 1 are very rare. An explanation for this
+#   is given by Niculescu-Mizil and Caruana [1]_: "Methods such as bagging and
+#   random forests that average predictions from a base set of models can have
+#   difficulty making predictions near 0 and 1 because variance in the
+#   underlying base models will bias predictions that should be near zero or
+#   one away from these values. Because predictions are restricted to the
+#   interval [0,1], errors caused by variance tend to be one- sided near zero
+#   and one. For example, if a model should predict p = 0 for a case, the only
+#   way bagging can achieve this is if all bagged trees predict zero. If we add
+#   noise to the trees that bagging is averaging over, this noise will cause
+#   some trees to predict values larger than 0 for this case, thus moving the
+#   average prediction of the bagged ensemble away from 0. We observe this
+#   effect most strongly with random forests because the base-level trees
+#   trained with random forests have relatively high variance due to feature
+#   subsetting." As a result, the calibration curve shows a characteristic
+#   sigmoid shape, indicating that the classifier could trust its "intuition"
+#   more and return probabilities closer to 0 or 1 typically.
+#
+# * we use naively calibrated (min-max scaled to [0,1])
+#   :term:`decision_function` values to show the performance of
+#   :class:`~sklearn.svm.LinearSVC`. :class:`~sklearn.svm.LinearSVC` shows an
+#   even more sigmoid curve than the
+#   :class:`~sklearn.ensemble.RandomForestClassifier`, which is typical for
+#   maximum-margin methods [1]_ as they focus on difficult to classify samples
+#   that are close to the decision boundary (the support vectors).
+
+
+class NaivelyCalibratedLinearSVC(LinearSVC):
+    """LinearSVC with `predict_proba` method that naively scales
+    `decision_function` output."""
+
+    def fit(self, X, y):
+        super().fit(X, y)
+        df = self.decision_function(X)
+        self.df_min_ = df.min()
+        self.df_max_ = df.max()
+
+    def predict_proba(self, X):
+        """Min-max scale output of `decision_function` to [0,1]."""
+        df = self.decision_function(X)
+        calibrated_df = (df - self.df_min_) / (self.df_max_ - self.df_min_)
+        return np.clip(calibrated_df, 0, 1)
+
+
 # Create classifiers
 lr = LogisticRegression()
 gnb = GaussianNB()
-svc = LinearSVC(C=1.0)
+svc = NaivelyCalibratedLinearSVC(C=1.0)
 rfc = RandomForestClassifier()
 
+clf_list = [(lr, 'Logistic'),
+            (gnb, 'Naive Bayes'),
+            (svc, 'SVC'),
+            (rfc, 'Random forest')]
 
-# #############################################################################
-# Plot calibration plots
+fig = plt.figure(figsize=(10, 10))
+gs = GridSpec(4, 2)
+colors = plt.cm.get_cmap('Dark2')
 
-plt.figure(figsize=(10, 10))
-ax1 = plt.subplot2grid((3, 1), (0, 0), rowspan=2)
-ax2 = plt.subplot2grid((3, 1), (2, 0))
-
-ax1.plot([0, 1], [0, 1], "k:", label="Perfectly calibrated")
-for clf, name in [(lr, 'Logistic'),
-                  (gnb, 'Naive Bayes'),
-                  (svc, 'Support Vector Classification'),
-                  (rfc, 'Random Forest')]:
+ax1 = fig.add_subplot(gs[:2, :2])
+viz_objects = {}
+for i, (clf, name) in enumerate(clf_list):
     clf.fit(X_train, y_train)
-    if hasattr(clf, "predict_proba"):
-        prob_pos = clf.predict_proba(X_test)[:, 1]
-    else:  # use decision function
-        prob_pos = clf.decision_function(X_test)
-        prob_pos = \
-            (prob_pos - prob_pos.min()) / (prob_pos.max() - prob_pos.min())
-    fraction_of_positives, mean_predicted_value = \
-        calibration_curve(y_test, prob_pos, n_bins=10)
-
-    ax1.plot(mean_predicted_value, fraction_of_positives, "s-",
-             label="%s" % (name, ))
-
-    ax2.hist(prob_pos, range=(0, 1), bins=10, label=name,
-             histtype="step", lw=2)
-
-ax1.set_ylabel("Fraction of positives")
-ax1.set_ylim([-0.05, 1.05])
-ax1.legend(loc="lower right")
-ax1.set_title('Calibration plots  (reliability curve)')
-
-ax2.set_xlabel("Mean predicted value")
-ax2.set_ylabel("Count")
-ax2.legend(loc="upper center", ncol=2)
+    viz = plot_calibration_curve(
+        clf, X_test, y_test, n_bins=10, name=name, ax=ax1, color=colors(i)
+    )
+    viz_objects[name] = viz
+
+ax1.grid()
+ax1.set_title('Calibration plots')
+
+# Add histogram
+grid_positions = [(2, 0), (2, 1), (3, 0), (3, 1)]
+for i, (_, name) in enumerate(clf_list):
+    row, col = grid_positions[i]
+    ax = fig.add_subplot(gs[row, col])
+
+    ax.hist(
+        viz_objects[name].y_prob, range=(0, 1), bins=10, label=name,
+        color=colors(i)
+    )
+    ax.set(title=name, xlabel="Mean predicted probability", ylabel="Count")
 
 plt.tight_layout()
 plt.show()
+
+# %%
+# References
+# ----------
+#
+# .. [1] `Predicting Good Probabilities with Supervised Learning
+#        <https://dl.acm.org/doi/pdf/10.1145/1102351.1102430>`_,
+#        A. Niculescu-Mizil & R. Caruana, ICML 2005
diff --git a/sklearn/calibration.py b/sklearn/calibration.py
index abea91e7e53df..fda0b4cabd61e 100644
--- a/sklearn/calibration.py
+++ b/sklearn/calibration.py
@@ -21,9 +21,11 @@
 from .preprocessing import LabelEncoder
 
 from .base import (BaseEstimator, ClassifierMixin, RegressorMixin, clone,
-                   MetaEstimatorMixin)
+                   MetaEstimatorMixin, is_classifier)
+from .metrics._plot.base import _check_classifier_response_method
 from .preprocessing import label_binarize, LabelBinarizer
-from .utils import check_array, indexable, column_or_1d
+from .utils import (check_array, indexable, column_or_1d,
+                    check_matplotlib_support)
 from .utils.validation import check_is_fitted, check_consistent_length
 from .utils.validation import _check_sample_weight
 from .pipeline import Pipeline
@@ -753,3 +755,208 @@ def calibration_curve(y_true, y_prob, *, normalize=False, n_bins=5,
     prob_pred = bin_sums[nonzero] / bin_total[nonzero]
 
     return prob_true, prob_pred
+
+
+class CalibrationDisplay:
+    """Calibration visualization.
+
+    It is recommend to use :func:`~sklearn.calibration.plot_calibration_curve`
+    to create a visualizer. All parameters are stored as attributes.
+
+    Read more in the :ref:`User Guide <calibration>`.
+
+    Parameters
+    -----------
+    prob_true : ndarray
+        The proportion of samples whose class is the positive class (fraction
+        of positives), in each bin.
+
+    prob_pred : ndarray
+        The mean predicted probability in each bin.
+
+    y_prob : ndarray of shape (n_samples,)
+        Probability estimates for the positive class, for each sample.
+
+    estimator_name : str, default=None
+        Name of estimator. If `None`, then the estimator name is not shown.
+
+    Attributes
+    ----------
+    line_ : matplotlib Artist
+        Calibration curve.
+
+    ax_ : matplotlib Axes
+        Axes with calibration curve.
+
+    figure_ : matplotlib Figure
+        Figure containing the curve.
+    """
+    def __init__(self, prob_true, prob_pred, y_prob, *, estimator_name=None):
+        self.prob_true = prob_true
+        self.prob_pred = prob_pred
+        self.y_prob = y_prob
+        self.estimator_name = estimator_name
+
+    def plot(self, ax=None, *, name=None, ref_line=True, **kwargs):
+        """Plot visualization.
+
+        Extra keyword arguments will be passed to
+        :func:`matplotlib.pyplot.plot`.
+
+        Parameters
+        ----------
+        ax : Matplotlib Axes, default=None
+            Axes object to plot on. If `None`, a new figure and axes is
+            created.
+
+        name : str, default=None
+            Name of calibration curve for labeling. If `None`, use the
+            name of the estimator.
+
+        ref_line : bool, default=True
+            If `True`, plots a reference line representing a perfectly
+            calibrated classifier.
+
+        **kwargs : dict
+            Keyword arguments to be passed to :func:`matplotlib.pyplot.plot`.
+
+        Returns
+        -------
+        display : :class:`~sklearn.calibration.CalibrationDisplay`
+            Object that stores computed values.
+        """
+        check_matplotlib_support("CalibrationDisplay.plot")
+        import matplotlib.pyplot as plt
+
+        if ax is None:
+            fig, ax = plt.subplots()
+
+        name = self.estimator_name if name is None else name
+
+        line_kwargs = {}
+        if name is not None:
+            line_kwargs["label"] = name
+        line_kwargs.update(**kwargs)
+
+        existing_ref_line = ('Perfectly calibrated' in
+                             ax.get_legend_handles_labels()[1])
+        if ref_line and not existing_ref_line:
+            ax.plot([0, 1], [0, 1], "k:", label="Perfectly calibrated")
+        self.line_ = ax.plot(self.prob_pred, self.prob_true, "s-",
+                             **line_kwargs)[0]
+
+        if "label" in line_kwargs:
+            ax.legend(loc="lower right")
+
+        ax.set(xlabel="Mean predicted probability",
+               ylabel="Fraction of positives")
+
+        self.ax_ = ax
+        self.figure_ = ax.figure
+        return self
+
+
+def plot_calibration_curve(estimator, X, y, *,
+                           n_bins=5, strategy='uniform',
+                           name=None, ref_line=True, ax=None, **kwargs):
+    """Plot calibration curve, also known as reliability diagrams, for binary
+    classifiers.
+
+    The average predicted probability for each bin is plotted on the x-axis
+    and the fraction of positive classes in each bin is plotted on the y-axis.
+
+    Extra keyword arguments will be passed to :func:`matplotlib.pyplot.plot`.
+
+    Read more in the :ref:`User Guide <calibration>`.
+
+    Parameters
+    ----------
+    estimator : estimator instance
+        Fitted classifier or a fitted :class:`~sklearn.pipeline.Pipeline` in
+        which the last estimator is a classifier. The classifier must
+        have a :term:`predict_proba` method.
+
+    X : {array-like, sparse matrix} of shape (n_samples, n_features)
+        Input values.
+
+    y : array-like of shape (n_samples,)
+        Binary target values.
+
+    n_bins : int, default=5
+        Number of bins to discretize the [0, 1] interval into when calculating
+        the calibration curve. A bigger number requires more data.
+
+    strategy : {'uniform', 'quantile'}, default='uniform'
+        Strategy used to define the widths of the bins.
+
+        - `'uniform'`: The bins have identical widths.
+        - `'quantile'`: The bins have the same number of samples and depend on
+          predicted probabilities.
+
+    name : str, default=None
+        Name for labeling curve. If `None`, the name of the estimator is used.
+
+    ref_line : bool, default=True
+        If `True`, plots a reference line representing a perfectly calibrated
+        classifier.
+
+    ax : matplotlib axes, default=None
+        Axes object to plot on. If `None`, a new figure and axes is created.
+
+    **kwargs : dict
+        Keyword arguments to be passed to :func:`matplotlib.pyplot.plot`.
+
+    Returns
+    -------
+    display : :class:`~sklearn.calibration.CalibrationDisplay`.
+        Object that stores computed values.
+
+    Examples
+    --------
+    >>> import matplotlib.pyplot as plt                     # doctest: +SKIP
+    >>> from sklearn import (datasets, calibration, model_selection,
+    ...                      linear_model)
+    >>> X, y = datasets.make_classification(random_state=0)
+    >>> X_train, X_test, y_train, y_test = model_selection.train_test_split(
+    ...     X, y, random_state=0)
+    >>> clf = linear_model.LogisticRegression(random_state=0)
+    >>> clf.fit(X_train, y_train)
+    LogisticRegression(random_state=0)
+    >>> metrics.plot_calibration_curve(clf, X_test, y_test)  # doctest: +SKIP
+    >>> plt.show()                                           # doctest: +SKIP
+    """
+    check_matplotlib_support("plot_calibration_curve")
+    binary_error = "Only binary classification is supported."
+
+    if not is_classifier(estimator):
+        raise ValueError("The 'estimator' parameter should be a fitted binary "
+                         "classifier")
+
+    try:
+        prediction_method = _check_classifier_response_method(
+            estimator, response_method='predict_proba'
+        )
+    except ValueError:
+        raise ValueError("Response method 'predict_proba' not defined in "
+                         f"{estimator.__class__.__name__}")
+
+    y_prob = prediction_method(X)
+
+    if not len(estimator.classes_) == 2:
+        raise ValueError(binary_error)
+    if y_prob.ndim != 1:
+        if y_prob.shape[1] != 2:
+            raise ValueError(binary_error)
+        else:
+            y_prob = y_prob[:, 1]
+
+    prob_true, prob_pred = calibration_curve(
+        y, y_prob, n_bins=n_bins, strategy=strategy
+    )
+
+    name = name if name is not None else estimator.__class__.__name__
+    viz = CalibrationDisplay(
+        prob_true=prob_true, prob_pred=prob_pred, y_prob=y_prob,
+        estimator_name=name
+    )
+    return viz.plot(ax=ax, name=name, ref_line=ref_line, **kwargs)
diff --git a/sklearn/tests/test_calibration.py b/sklearn/tests/test_calibration.py
index f4e6f6d62d938..aa799b311192d 100644
--- a/sklearn/tests/test_calibration.py
+++ b/sklearn/tests/test_calibration.py
@@ -8,25 +8,28 @@
 
 from sklearn.base import BaseEstimator
 from sklearn.model_selection import LeaveOneOut, train_test_split
-
 from sklearn.utils._testing import (assert_array_almost_equal,
                                     assert_almost_equal,
                                     assert_array_equal,
                                     assert_raises, ignore_warnings)
 from sklearn.exceptions import NotFittedError
-from sklearn.datasets import make_classification, make_blobs
+from sklearn.datasets import make_classification, make_blobs, load_iris
 from sklearn.preprocessing import LabelBinarizer
 from sklearn.model_selection import KFold
 from sklearn.naive_bayes import MultinomialNB
 from sklearn.ensemble import RandomForestClassifier, RandomForestRegressor
+from sklearn.tree import DecisionTreeClassifier
+from sklearn.linear_model import LogisticRegression
 from sklearn.svm import LinearSVC
+from sklearn.pipeline import Pipeline, make_pipeline
+from sklearn.preprocessing import StandardScaler
 from sklearn.feature_extraction import DictVectorizer
-from sklearn.pipeline import Pipeline
 from sklearn.impute import SimpleImputer
 from sklearn.metrics import brier_score_loss, log_loss
 from sklearn.calibration import CalibratedClassifierCV
 from sklearn.calibration import _sigmoid_calibration, _SigmoidCalibration
-from sklearn.calibration import calibration_curve
+from sklearn.calibration import (calibration_curve, plot_calibration_curve,
+                                 CalibrationDisplay)
 
 
 def test_calibration():
@@ -455,3 +458,136 @@ def test_calibration_attributes(clf, cv):
         classes = LabelBinarizer().fit(y).classes_
         assert_array_equal(calib_clf.classes_, classes)
         assert calib_clf.n_features_in_ == X.shape[1]
+
+
+@pytest.fixture(scope="module")
+def data():
+    return load_iris(return_X_y=True)
+
+
+@pytest.fixture(scope="module")
+def data_binary(data):
+    X, y = data
+    return X[y < 2], y[y < 2]
+
+
+@pytest.mark.parametrize(
+    "estimator_name, expected_label",
+    [(None, "_line1"),
+     ("my_est", "my_est"),
+     ("my_est2", "my_est2")]
+)
+def test_calibration_display_default_labels(pyplot, estimator_name,
+                                            expected_label):
+    prob_true = np.array([0, 1, 1, 0])
+    prob_pred = np.array([0.2, 0.8, 0.8, 0.4])
+    y_prob = np.array([])
+
+    viz = CalibrationDisplay(prob_true, prob_pred, y_prob,
+                             estimator_name=estimator_name)
+    viz.plot()
+    assert viz.line_.get_label() == expected_label
+
+
+def test_plot_calibration_curve_error_non_binary(pyplot, data):
+    X, y = data
+    clf = DecisionTreeClassifier()
+    clf.fit(X, y)
+
+    msg = "Only binary classification is supported."
+    with pytest.raises(ValueError, match=msg):
+        plot_calibration_curve(clf, X, y)
+
+
+def test_plot_calibration_curve_no_predict_proba(pyplot, data_binary):
+    X, y = data_binary
+    clf = LinearSVC().fit(X, y)
+
+    msg = "Response method 'predict_proba' not defined in"
+    with pytest.raises(ValueError, match=msg):
+        plot_calibration_curve(clf, X, y)
+
+
+def test_plot_calibration_curve_not_fitted(pyplot, data_binary):
+    X, y = data_binary
+    clf = LogisticRegression()
+
+    with pytest.raises(NotFittedError):
+        plot_calibration_curve(clf, X, y)
+
+
+@pytest.mark.parametrize("n_bins", [5, 10])
+@pytest.mark.parametrize("strategy", ["uniform", "quantile"])
+def test_plot_calibration_curve(pyplot, data_binary, n_bins, strategy):
+    # Ensure `plot_calibration_curve` and `calibration_curve` compute the same
+    # results. Also checks attributes of the CalibrationDisplay object.
+    X, y = data_binary
+
+    lr = LogisticRegression().fit(X, y)
+
+    viz = plot_calibration_curve(
+        lr, X, y, n_bins=n_bins, strategy=strategy, alpha=0.8
+    )
+
+    y_prob = lr.predict_proba(X)[:, 1]
+    prob_true, prob_pred = calibration_curve(
+        y, y_prob, n_bins=n_bins, strategy=strategy
+    )
+
+    assert_allclose(viz.prob_true, prob_true)
+    assert_allclose(viz.prob_pred, prob_pred)
+    assert_allclose(viz.y_prob, y_prob)
+
+    assert viz.estimator_name == "LogisticRegression"
+
+    # cannot fail thanks to pyplot fixture
+    import matplotlib as mpl  # noqa
+    assert isinstance(viz.line_, mpl.lines.Line2D)
+    assert viz.line_.get_alpha() == 0.8
+    assert isinstance(viz.ax_, mpl.axes.Axes)
+    assert isinstance(viz.figure_, mpl.figure.Figure)
+
+    assert viz.ax_.get_xlabel() == "Mean predicted probability"
+    assert viz.ax_.get_ylabel() == "Fraction of positives"
+    assert viz.line_.get_label() == "LogisticRegression"
+
+
+def test_plot_calibration_curve_pipeline(pyplot, data_binary):
+    # Ensure pipelines are supported by plot_calibration_curve
+    X, y = data_binary
+    clf = make_pipeline(StandardScaler(), LogisticRegression())
+    clf.fit(X, y)
+    viz = plot_calibration_curve(clf, X, y)
+    assert clf.__class__.__name__ in viz.line_.get_label()
+    assert viz.estimator_name == clf.__class__.__name__
+
+
+def test_plot_calibration_curve_estimator_name_multiple_calls(pyplot,
+                                                              data_binary):
+    # Check that the `name` used when calling `plot_calibration_curve` is
+    # used when multiple `viz.plot()` calls are made.
+    X, y = data_binary
+    clf_name = "my hand-crafted name"
+    clf = LogisticRegression().fit(X, y)
+    viz = plot_calibration_curve(clf, X, y, name=clf_name)
+    assert viz.estimator_name == clf_name
+    pyplot.close("all")
+    viz.plot()
+    assert clf_name == viz.line_.get_label()
+    pyplot.close("all")
+    clf_name = "another_name"
+    viz.plot(name=clf_name)
+    assert clf_name == viz.line_.get_label()
+
+
+def test_plot_calibration_curve_ref_line(pyplot, data_binary):
+    # Check that `ref_line` only appears once
+    X, y = data_binary
+    lr = LogisticRegression().fit(X, y)
+    dt = DecisionTreeClassifier().fit(X, y)
+
+    viz = plot_calibration_curve(lr, X, y)
+    viz2 = plot_calibration_curve(dt, X, y, ax=viz.ax_)
+
+    labels = viz2.ax_.get_legend_handles_labels()[1]
+    assert labels.count('Perfectly calibrated') == 1


Code review comments:
diff_hunk: @@ -5,130 +5,290 @@
 
 When performing classification one often wants to predict not only the class
 label, but also the associated probability. This probability gives some
-kind of confidence on the prediction. This example demonstrates how to display
-how well calibrated the predicted probabilities are and how to calibrate an
-uncalibrated classifier.
-
-The experiment is performed on an artificial dataset for binary classification
-with 100,000 samples (1,000 of them are used for model fitting) with 20
-features. Of the 20 features, only 2 are informative and 10 are redundant. The
-first figure shows the estimated probabilities obtained with logistic
-regression, Gaussian naive Bayes, and Gaussian naive Bayes with both isotonic
-calibration and sigmoid calibration. The calibration performance is evaluated
-with Brier score, reported in the legend (the smaller the better). One can
-observe here that logistic regression is well calibrated while raw Gaussian
-naive Bayes performs very badly. This is because of the redundant features
-which violate the assumption of feature-independence and result in an overly
-confident classifier, which is indicated by the typical transposed-sigmoid
-curve.
-
-Calibration of the probabilities of Gaussian naive Bayes with isotonic
-regression can fix this issue as can be seen from the nearly diagonal
-calibration curve. Sigmoid calibration also improves the brier score slightly,
-albeit not as strongly as the non-parametric isotonic regression. This can be
-attributed to the fact that we have plenty of calibration data such that the
-greater flexibility of the non-parametric model can be exploited.
-
-The second figure shows the calibration curve of a linear support-vector
-classifier (LinearSVC). LinearSVC shows the opposite behavior as Gaussian
-naive Bayes: the calibration curve has a sigmoid curve, which is typical for
-an under-confident classifier. In the case of LinearSVC, this is caused by the
-margin property of the hinge loss, which lets the model focus on hard samples
-that are close to the decision boundary (the support vectors).
-
-Both kinds of calibration can fix this issue and yield nearly identical
-results. This shows that sigmoid calibration can deal with situations where
-the calibration curve of the base classifier is sigmoid (e.g., for LinearSVC)
-but not where it is transposed-sigmoid (e.g., Gaussian naive Bayes).
+kind of confidence on the prediction. This example demonstrates how to
+visualize how well calibrated the predicted probabilities are using calibration
+curves, also known as reliability diagrams. Calibration of an uncalibrated
+classifier will also be demonstrated.
 """
-print(__doc__)
+
+# %%
 
 # Author: Alexandre Gramfort <alexandre.gramfort@telecom-paristech.fr>
 #         Jan Hendrik Metzen <jhm@informatik.uni-bremen.de>
 # License: BSD Style.
 
 import matplotlib.pyplot as plt
+from matplotlib.gridspec import GridSpec
+import pandas as pd
+import numpy as np
 
-from sklearn import datasets
+from sklearn.datasets import make_classification
+from sklearn.model_selection import train_test_split
 from sklearn.naive_bayes import GaussianNB
 from sklearn.svm import LinearSVC
 from sklearn.linear_model import LogisticRegression
-from sklearn.metrics import (brier_score_loss, precision_score, recall_score,
-                             f1_score)
-from sklearn.calibration import CalibratedClassifierCV, calibration_curve
-from sklearn.model_selection import train_test_split
+from sklearn.metrics import (precision_score, recall_score, f1_score,
+                             brier_score_loss, log_loss)
+from sklearn.calibration import CalibratedClassifierCV, plot_calibration_curve
 
+# %%
+# Dataset
+# -------
+#
+# We will use a synthetic binary classification dataset with 100,000 samples
+# and 20 features. Of the 20 features, only 2 are informative, 10 are
+# redundant (random combinations of the informative features) and the
+# remaining 8 are 'useless' (random numbers). Of the 100,000 samples, 1,000
path: examples/calibration/plot_calibration_curve.py
text: @user1:
useless -> uninformative?

diff_hunk: @@ -5,130 +5,290 @@
 
 When performing classification one often wants to predict not only the class
 label, but also the associated probability. This probability gives some
-kind of confidence on the prediction. This example demonstrates how to display
-how well calibrated the predicted probabilities are and how to calibrate an
-uncalibrated classifier.
-
-The experiment is performed on an artificial dataset for binary classification
-with 100,000 samples (1,000 of them are used for model fitting) with 20
-features. Of the 20 features, only 2 are informative and 10 are redundant. The
-first figure shows the estimated probabilities obtained with logistic
-regression, Gaussian naive Bayes, and Gaussian naive Bayes with both isotonic
-calibration and sigmoid calibration. The calibration performance is evaluated
-with Brier score, reported in the legend (the smaller the better). One can
-observe here that logistic regression is well calibrated while raw Gaussian
-naive Bayes performs very badly. This is because of the redundant features
-which violate the assumption of feature-independence and result in an overly
-confident classifier, which is indicated by the typical transposed-sigmoid
-curve.
-
-Calibration of the probabilities of Gaussian naive Bayes with isotonic
-regression can fix this issue as can be seen from the nearly diagonal
-calibration curve. Sigmoid calibration also improves the brier score slightly,
-albeit not as strongly as the non-parametric isotonic regression. This can be
-attributed to the fact that we have plenty of calibration data such that the
-greater flexibility of the non-parametric model can be exploited.
-
-The second figure shows the calibration curve of a linear support-vector
-classifier (LinearSVC). LinearSVC shows the opposite behavior as Gaussian
-naive Bayes: the calibration curve has a sigmoid curve, which is typical for
-an under-confident classifier. In the case of LinearSVC, this is caused by the
-margin property of the hinge loss, which lets the model focus on hard samples
-that are close to the decision boundary (the support vectors).
-
-Both kinds of calibration can fix this issue and yield nearly identical
-results. This shows that sigmoid calibration can deal with situations where
-the calibration curve of the base classifier is sigmoid (e.g., for LinearSVC)
-but not where it is transposed-sigmoid (e.g., Gaussian naive Bayes).
+kind of confidence on the prediction. This example demonstrates how to
+visualize how well calibrated the predicted probabilities are using calibration
+curves, also known as reliability diagrams. Calibration of an uncalibrated
+classifier will also be demonstrated.
 """
-print(__doc__)
+
+# %%
 
 # Author: Alexandre Gramfort <alexandre.gramfort@telecom-paristech.fr>
 #         Jan Hendrik Metzen <jhm@informatik.uni-bremen.de>
 # License: BSD Style.
 
 import matplotlib.pyplot as plt
+from matplotlib.gridspec import GridSpec
+import pandas as pd
+import numpy as np
 
-from sklearn import datasets
+from sklearn.datasets import make_classification
+from sklearn.model_selection import train_test_split
 from sklearn.naive_bayes import GaussianNB
 from sklearn.svm import LinearSVC
 from sklearn.linear_model import LogisticRegression
-from sklearn.metrics import (brier_score_loss, precision_score, recall_score,
-                             f1_score)
-from sklearn.calibration import CalibratedClassifierCV, calibration_curve
-from sklearn.model_selection import train_test_split
+from sklearn.metrics import (precision_score, recall_score, f1_score,
+                             brier_score_loss, log_loss)
+from sklearn.calibration import CalibratedClassifierCV, plot_calibration_curve
 
+# %%
+# Dataset
+# -------
+#
+# We will use a synthetic binary classification dataset with 100,000 samples
+# and 20 features. Of the 20 features, only 2 are informative, 10 are
+# redundant (random combinations of the informative features) and the
+# remaining 8 are 'useless' (random numbers). Of the 100,000 samples, 1,000
+# will be used for model fitting and the rest for testing.
 
-# Create dataset of classification task with many redundant and few
-# informative features
-X, y = datasets.make_classification(n_samples=100000, n_features=20,
-                                    n_informative=2, n_redundant=10,
-                                    random_state=42)
+X, y = make_classification(n_samples=100000, n_features=20, n_informative=2,
+                           n_redundant=10, random_state=42)
 
 X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.99,
                                                     random_state=42)
 
+# %%
+# Calibration curves
+# ------------------
+#
+# First, we will compare:
+#
+# * :class:`~sklearn.linear_model.LogisticRegression` (used as baseline
+#   since very often, properly regularized logistic regression is well
+#   calibrated by default thanks to the use of log-loss)
+# * Raw :class:`~sklearn.naive_bayes.GaussianNB`
+# * :class:`~sklearn.naive_bayes.GaussianNB` with isotonic and sigmoid
+#   calibration (see :ref:`User Guide <calibration>`)
+#
+# Calibration curves for all 4 conditions are plotted below, with the average
+# predicted probability for each bin on the x-axis and the fraction of positive
+# classes in each bin on the y-axis.
+#
+
+lr = LogisticRegression(C=1.)
+gnb = GaussianNB()
+gnb_isotonic = CalibratedClassifierCV(gnb, cv=2, method='isotonic')
+gnb_sigmoid = CalibratedClassifierCV(gnb, cv=2, method='sigmoid')
+
+clf_list = [(lr, 'Logistic'),
+            (gnb, 'Naive Bayes'),
+            (gnb_isotonic, 'Naive Bayes + Isotonic'),
+            (gnb_sigmoid, 'Naive Bayes + Sigmoid')]
+
+fig = plt.figure(figsize=(10, 10))
+gs = GridSpec(4, 2)
+colors = plt.cm.get_cmap('Dark2')
+
+ax1 = fig.add_subplot(gs[:2, :2])
+viz_objects = {}
+for i, (clf, name) in enumerate(clf_list):
+    clf.fit(X_train, y_train)
+    viz = plot_calibration_curve(
+        clf, X_test, y_test, n_bins=10, name=name, ax=ax1, color=colors(i)
+    )
+    viz_objects[name] = viz
+
+ax1.grid()
+ax1.set_title('Calibration plots (Naive Bayes)')
+
+# Add histogram
+grid_positions = [(2, 0), (2, 1), (3, 0), (3, 1)]
+for i, (_, name) in enumerate(clf_list):
+    row, col = grid_positions[i]
+    ax = fig.add_subplot(gs[row, col])
+
+    ax.hist(
+        viz_objects[name].y_prob, range=(0, 1), bins=10, label=name,
+        color=colors(i)
+    )
+    ax.set(title=name, xlabel="Mean predicted probability", ylabel="Count")
+
+plt.tight_layout()
+plt.show()
+
+# %%
+# Raw :class:`~sklearn.naive_bayes.GaussianNB` performs very badly because of
+# the redundant features which violate the assumption of feature-independence
+# and result in an overly confident classifier, which is indicated by the
+# typical transposed-sigmoid curve. Calibration of the probabilities of
+# :class:`~sklearn.naive_bayes.GaussianNB` with isotonic regression can fix
+# this issue as can be seen from the nearly diagonal calibration curve. Sigmoid
+# regression also improves calibration slightly,
+# albeit not as strongly as the non-parametric isotonic regression. This can be
+# attributed to the fact that we have plenty of calibration data such that the
+# greater flexibility of the non-parametric model can be exploited.
+#
+# Below we show the Brier score, log loss, precision, recall and F1 score (see
+# :ref:`User Guide <precision_recall_f_measure_metrics>`). Notice that
+# although calibration improves the Brier score (a metric composed of
+# calibration loss and refinement loss) and :ref:`log_loss`, it does not
+# significantly alter the prediction accuracy measures (precision, recall and
+# F1 score).
+# This is because calibration should not significantly change prediction
+# probabilities at the location of the decision threshold (at x = 0.5 on the
+# graph). Calibration should however, make the predicted probabilities more
+# accurate and thus more useful for making allocation decisions under
+# uncertainty.
path: examples/calibration/plot_calibration_curve.py
text: @user1:
These are great insights!
I would suggest to also add the ROC AUC in the table so that we can mention that ranking metrics are not impacte, because calibration is a monotonic transformation.

@author:
Great point! Should I make the caveat about isotonic not (yet) being strictly monotonic or ignore as #17790 will be hopefully in soon :smile:  ?

@user1:
no need to care IMO. if you want to be extra careful you can say "up to tie breaks" or something similar

diff_hunk: @@ -5,130 +5,290 @@
 
 When performing classification one often wants to predict not only the class
 label, but also the associated probability. This probability gives some
-kind of confidence on the prediction. This example demonstrates how to display
-how well calibrated the predicted probabilities are and how to calibrate an
-uncalibrated classifier.
-
-The experiment is performed on an artificial dataset for binary classification
-with 100,000 samples (1,000 of them are used for model fitting) with 20
-features. Of the 20 features, only 2 are informative and 10 are redundant. The
-first figure shows the estimated probabilities obtained with logistic
-regression, Gaussian naive Bayes, and Gaussian naive Bayes with both isotonic
-calibration and sigmoid calibration. The calibration performance is evaluated
-with Brier score, reported in the legend (the smaller the better). One can
-observe here that logistic regression is well calibrated while raw Gaussian
-naive Bayes performs very badly. This is because of the redundant features
-which violate the assumption of feature-independence and result in an overly
-confident classifier, which is indicated by the typical transposed-sigmoid
-curve.
-
-Calibration of the probabilities of Gaussian naive Bayes with isotonic
-regression can fix this issue as can be seen from the nearly diagonal
-calibration curve. Sigmoid calibration also improves the brier score slightly,
-albeit not as strongly as the non-parametric isotonic regression. This can be
-attributed to the fact that we have plenty of calibration data such that the
-greater flexibility of the non-parametric model can be exploited.
-
-The second figure shows the calibration curve of a linear support-vector
-classifier (LinearSVC). LinearSVC shows the opposite behavior as Gaussian
-naive Bayes: the calibration curve has a sigmoid curve, which is typical for
-an under-confident classifier. In the case of LinearSVC, this is caused by the
-margin property of the hinge loss, which lets the model focus on hard samples
-that are close to the decision boundary (the support vectors).
-
-Both kinds of calibration can fix this issue and yield nearly identical
-results. This shows that sigmoid calibration can deal with situations where
-the calibration curve of the base classifier is sigmoid (e.g., for LinearSVC)
-but not where it is transposed-sigmoid (e.g., Gaussian naive Bayes).
+kind of confidence on the prediction. This example demonstrates how to
+visualize how well calibrated the predicted probabilities are using calibration
+curves, also known as reliability diagrams. Calibration of an uncalibrated
+classifier will also be demonstrated.
 """
-print(__doc__)
+
+# %%
 
 # Author: Alexandre Gramfort <alexandre.gramfort@telecom-paristech.fr>
 #         Jan Hendrik Metzen <jhm@informatik.uni-bremen.de>
 # License: BSD Style.
 
 import matplotlib.pyplot as plt
+from matplotlib.gridspec import GridSpec
+import pandas as pd
+import numpy as np
 
-from sklearn import datasets
+from sklearn.datasets import make_classification
+from sklearn.model_selection import train_test_split
 from sklearn.naive_bayes import GaussianNB
 from sklearn.svm import LinearSVC
 from sklearn.linear_model import LogisticRegression
-from sklearn.metrics import (brier_score_loss, precision_score, recall_score,
-                             f1_score)
-from sklearn.calibration import CalibratedClassifierCV, calibration_curve
-from sklearn.model_selection import train_test_split
+from sklearn.metrics import (precision_score, recall_score, f1_score,
+                             brier_score_loss, log_loss)
+from sklearn.calibration import CalibratedClassifierCV, plot_calibration_curve
 
+# %%
+# Dataset
+# -------
+#
+# We will use a synthetic binary classification dataset with 100,000 samples
+# and 20 features. Of the 20 features, only 2 are informative, 10 are
+# redundant (random combinations of the informative features) and the
+# remaining 8 are 'useless' (random numbers). Of the 100,000 samples, 1,000
+# will be used for model fitting and the rest for testing.
 
-# Create dataset of classification task with many redundant and few
-# informative features
-X, y = datasets.make_classification(n_samples=100000, n_features=20,
-                                    n_informative=2, n_redundant=10,
-                                    random_state=42)
+X, y = make_classification(n_samples=100000, n_features=20, n_informative=2,
+                           n_redundant=10, random_state=42)
 
 X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.99,
                                                     random_state=42)
 
+# %%
+# Calibration curves
+# ------------------
+#
+# First, we will compare:
+#
+# * :class:`~sklearn.linear_model.LogisticRegression` (used as baseline
+#   since very often, properly regularized logistic regression is well
+#   calibrated by default thanks to the use of log-loss)
+# * Raw :class:`~sklearn.naive_bayes.GaussianNB`
+# * :class:`~sklearn.naive_bayes.GaussianNB` with isotonic and sigmoid
+#   calibration (see :ref:`User Guide <calibration>`)
+#
+# Calibration curves for all 4 conditions are plotted below, with the average
+# predicted probability for each bin on the x-axis and the fraction of positive
+# classes in each bin on the y-axis.
+#
+
+lr = LogisticRegression(C=1.)
+gnb = GaussianNB()
+gnb_isotonic = CalibratedClassifierCV(gnb, cv=2, method='isotonic')
+gnb_sigmoid = CalibratedClassifierCV(gnb, cv=2, method='sigmoid')
+
+clf_list = [(lr, 'Logistic'),
+            (gnb, 'Naive Bayes'),
+            (gnb_isotonic, 'Naive Bayes + Isotonic'),
+            (gnb_sigmoid, 'Naive Bayes + Sigmoid')]
+
+fig = plt.figure(figsize=(10, 10))
+gs = GridSpec(4, 2)
+colors = plt.cm.get_cmap('Dark2')
+
+ax1 = fig.add_subplot(gs[:2, :2])
+viz_objects = {}
+for i, (clf, name) in enumerate(clf_list):
+    clf.fit(X_train, y_train)
+    viz = plot_calibration_curve(
+        clf, X_test, y_test, n_bins=10, name=name, ax=ax1, color=colors(i)
+    )
+    viz_objects[name] = viz
+
+ax1.grid()
+ax1.set_title('Calibration plots (Naive Bayes)')
+
+# Add histogram
+grid_positions = [(2, 0), (2, 1), (3, 0), (3, 1)]
+for i, (_, name) in enumerate(clf_list):
+    row, col = grid_positions[i]
+    ax = fig.add_subplot(gs[row, col])
+
+    ax.hist(
+        viz_objects[name].y_prob, range=(0, 1), bins=10, label=name,
+        color=colors(i)
+    )
+    ax.set(title=name, xlabel="Mean predicted probability", ylabel="Count")
+
+plt.tight_layout()
+plt.show()
+
+# %%
+# Raw :class:`~sklearn.naive_bayes.GaussianNB` performs very badly because of
+# the redundant features which violate the assumption of feature-independence
+# and result in an overly confident classifier, which is indicated by the
+# typical transposed-sigmoid curve. Calibration of the probabilities of
+# :class:`~sklearn.naive_bayes.GaussianNB` with isotonic regression can fix
+# this issue as can be seen from the nearly diagonal calibration curve. Sigmoid
+# regression also improves calibration slightly,
+# albeit not as strongly as the non-parametric isotonic regression. This can be
+# attributed to the fact that we have plenty of calibration data such that the
+# greater flexibility of the non-parametric model can be exploited.
+#
+# Below we show the Brier score, log loss, precision, recall and F1 score (see
+# :ref:`User Guide <precision_recall_f_measure_metrics>`). Notice that
+# although calibration improves the Brier score (a metric composed of
+# calibration loss and refinement loss) and :ref:`log_loss`, it does not
+# significantly alter the prediction accuracy measures (precision, recall and
+# F1 score).
+# This is because calibration should not significantly change prediction
+# probabilities at the location of the decision threshold (at x = 0.5 on the
+# graph). Calibration should however, make the predicted probabilities more
+# accurate and thus more useful for making allocation decisions under
+# uncertainty.
+
+index = []
+brier = []
+logloss = []
+precision = []
+recall = []
+f1 = []
+
+for i, (clf, name) in enumerate(clf_list):
+    clf.fit(X_train, y_train)
+    y_proba = clf.predict_proba(X_test)
+    y_pred = clf.predict(X_test)
+
+    # Create DataFrame index
+    index.append(name)
+    # Store column data
+    brier.append(brier_score_loss(y_test, y_proba[:, 1]))
+    logloss.append(log_loss(y_test, y_proba[:, 1]))
+    precision.append(precision_score(y_test, y_pred))
+    recall.append(recall_score(y_test, y_pred))
+    f1.append(f1_score(y_test, y_pred))
+
+score_df = pd.DataFrame(
+    data={'Brier score': brier, 'Log loss': logloss, 'Precision': precision,
+          'Recall': recall, 'F1': f1},
+    index=index,
+)
+score_df.round(3)
+
+# %%
+# Next, we will compare:
+#
+# * :class:`~sklearn.linear_model.LogisticRegression` (baseline)
+# * :class:`~sklearn.svm.LinearSVC` (:term:`decision_function` scores
+#   min-max scaled to [0,1], since they are not proper probabilities)
path: examples/calibration/plot_calibration_curve.py
text: @user1:
Let's be a bit more verbose:

```suggestion
# * Uncalibrated :class:`~sklearn.svm.LinearSVC`. Since SVC does not output probabilities by default, we naively scale the output of the :term:`decision_function` into [0, 1] by applying min-max scaling.
```

diff_hunk: @@ -5,130 +5,290 @@
 
 When performing classification one often wants to predict not only the class
 label, but also the associated probability. This probability gives some
-kind of confidence on the prediction. This example demonstrates how to display
-how well calibrated the predicted probabilities are and how to calibrate an
-uncalibrated classifier.
-
-The experiment is performed on an artificial dataset for binary classification
-with 100,000 samples (1,000 of them are used for model fitting) with 20
-features. Of the 20 features, only 2 are informative and 10 are redundant. The
-first figure shows the estimated probabilities obtained with logistic
-regression, Gaussian naive Bayes, and Gaussian naive Bayes with both isotonic
-calibration and sigmoid calibration. The calibration performance is evaluated
-with Brier score, reported in the legend (the smaller the better). One can
-observe here that logistic regression is well calibrated while raw Gaussian
-naive Bayes performs very badly. This is because of the redundant features
-which violate the assumption of feature-independence and result in an overly
-confident classifier, which is indicated by the typical transposed-sigmoid
-curve.
-
-Calibration of the probabilities of Gaussian naive Bayes with isotonic
-regression can fix this issue as can be seen from the nearly diagonal
-calibration curve. Sigmoid calibration also improves the brier score slightly,
-albeit not as strongly as the non-parametric isotonic regression. This can be
-attributed to the fact that we have plenty of calibration data such that the
-greater flexibility of the non-parametric model can be exploited.
-
-The second figure shows the calibration curve of a linear support-vector
-classifier (LinearSVC). LinearSVC shows the opposite behavior as Gaussian
-naive Bayes: the calibration curve has a sigmoid curve, which is typical for
-an under-confident classifier. In the case of LinearSVC, this is caused by the
-margin property of the hinge loss, which lets the model focus on hard samples
-that are close to the decision boundary (the support vectors).
-
-Both kinds of calibration can fix this issue and yield nearly identical
-results. This shows that sigmoid calibration can deal with situations where
-the calibration curve of the base classifier is sigmoid (e.g., for LinearSVC)
-but not where it is transposed-sigmoid (e.g., Gaussian naive Bayes).
+kind of confidence on the prediction. This example demonstrates how to
+visualize how well calibrated the predicted probabilities are using calibration
+curves, also known as reliability diagrams. Calibration of an uncalibrated
+classifier will also be demonstrated.
 """
-print(__doc__)
+
+# %%
 
 # Author: Alexandre Gramfort <alexandre.gramfort@telecom-paristech.fr>
 #         Jan Hendrik Metzen <jhm@informatik.uni-bremen.de>
 # License: BSD Style.
 
 import matplotlib.pyplot as plt
+from matplotlib.gridspec import GridSpec
+import pandas as pd
+import numpy as np
 
-from sklearn import datasets
+from sklearn.datasets import make_classification
+from sklearn.model_selection import train_test_split
 from sklearn.naive_bayes import GaussianNB
 from sklearn.svm import LinearSVC
 from sklearn.linear_model import LogisticRegression
-from sklearn.metrics import (brier_score_loss, precision_score, recall_score,
-                             f1_score)
-from sklearn.calibration import CalibratedClassifierCV, calibration_curve
-from sklearn.model_selection import train_test_split
+from sklearn.metrics import (precision_score, recall_score, f1_score,
+                             brier_score_loss, log_loss)
+from sklearn.calibration import CalibratedClassifierCV, plot_calibration_curve
 
+# %%
+# Dataset
+# -------
+#
+# We will use a synthetic binary classification dataset with 100,000 samples
+# and 20 features. Of the 20 features, only 2 are informative, 10 are
+# redundant (random combinations of the informative features) and the
+# remaining 8 are 'useless' (random numbers). Of the 100,000 samples, 1,000
+# will be used for model fitting and the rest for testing.
 
-# Create dataset of classification task with many redundant and few
-# informative features
-X, y = datasets.make_classification(n_samples=100000, n_features=20,
-                                    n_informative=2, n_redundant=10,
-                                    random_state=42)
+X, y = make_classification(n_samples=100000, n_features=20, n_informative=2,
+                           n_redundant=10, random_state=42)
 
 X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.99,
                                                     random_state=42)
 
+# %%
+# Calibration curves
+# ------------------
+#
+# First, we will compare:
+#
+# * :class:`~sklearn.linear_model.LogisticRegression` (used as baseline
+#   since very often, properly regularized logistic regression is well
+#   calibrated by default thanks to the use of log-loss)
+# * Raw :class:`~sklearn.naive_bayes.GaussianNB`
+# * :class:`~sklearn.naive_bayes.GaussianNB` with isotonic and sigmoid
+#   calibration (see :ref:`User Guide <calibration>`)
+#
+# Calibration curves for all 4 conditions are plotted below, with the average
+# predicted probability for each bin on the x-axis and the fraction of positive
+# classes in each bin on the y-axis.
+#
+
+lr = LogisticRegression(C=1.)
+gnb = GaussianNB()
+gnb_isotonic = CalibratedClassifierCV(gnb, cv=2, method='isotonic')
+gnb_sigmoid = CalibratedClassifierCV(gnb, cv=2, method='sigmoid')
+
+clf_list = [(lr, 'Logistic'),
+            (gnb, 'Naive Bayes'),
+            (gnb_isotonic, 'Naive Bayes + Isotonic'),
+            (gnb_sigmoid, 'Naive Bayes + Sigmoid')]
+
+fig = plt.figure(figsize=(10, 10))
+gs = GridSpec(4, 2)
+colors = plt.cm.get_cmap('Dark2')
+
+ax1 = fig.add_subplot(gs[:2, :2])
+viz_objects = {}
+for i, (clf, name) in enumerate(clf_list):
+    clf.fit(X_train, y_train)
+    viz = plot_calibration_curve(
+        clf, X_test, y_test, n_bins=10, name=name, ax=ax1, color=colors(i)
+    )
+    viz_objects[name] = viz
+
+ax1.grid()
+ax1.set_title('Calibration plots (Naive Bayes)')
+
+# Add histogram
+grid_positions = [(2, 0), (2, 1), (3, 0), (3, 1)]
+for i, (_, name) in enumerate(clf_list):
+    row, col = grid_positions[i]
+    ax = fig.add_subplot(gs[row, col])
+
+    ax.hist(
+        viz_objects[name].y_prob, range=(0, 1), bins=10, label=name,
+        color=colors(i)
+    )
+    ax.set(title=name, xlabel="Mean predicted probability", ylabel="Count")
+
+plt.tight_layout()
+plt.show()
+
+# %%
+# Raw :class:`~sklearn.naive_bayes.GaussianNB` performs very badly because of
+# the redundant features which violate the assumption of feature-independence
+# and result in an overly confident classifier, which is indicated by the
+# typical transposed-sigmoid curve. Calibration of the probabilities of
+# :class:`~sklearn.naive_bayes.GaussianNB` with isotonic regression can fix
+# this issue as can be seen from the nearly diagonal calibration curve. Sigmoid
+# regression also improves calibration slightly,
+# albeit not as strongly as the non-parametric isotonic regression. This can be
+# attributed to the fact that we have plenty of calibration data such that the
+# greater flexibility of the non-parametric model can be exploited.
+#
+# Below we show the Brier score, log loss, precision, recall and F1 score (see
+# :ref:`User Guide <precision_recall_f_measure_metrics>`). Notice that
+# although calibration improves the Brier score (a metric composed of
+# calibration loss and refinement loss) and :ref:`log_loss`, it does not
+# significantly alter the prediction accuracy measures (precision, recall and
+# F1 score).
+# This is because calibration should not significantly change prediction
+# probabilities at the location of the decision threshold (at x = 0.5 on the
+# graph). Calibration should however, make the predicted probabilities more
+# accurate and thus more useful for making allocation decisions under
+# uncertainty.
+
+index = []
+brier = []
+logloss = []
+precision = []
+recall = []
+f1 = []
+
+for i, (clf, name) in enumerate(clf_list):
+    clf.fit(X_train, y_train)
+    y_proba = clf.predict_proba(X_test)
+    y_pred = clf.predict(X_test)
+
+    # Create DataFrame index
+    index.append(name)
+    # Store column data
+    brier.append(brier_score_loss(y_test, y_proba[:, 1]))
+    logloss.append(log_loss(y_test, y_proba[:, 1]))
+    precision.append(precision_score(y_test, y_pred))
+    recall.append(recall_score(y_test, y_pred))
+    f1.append(f1_score(y_test, y_pred))
+
+score_df = pd.DataFrame(
+    data={'Brier score': brier, 'Log loss': logloss, 'Precision': precision,
+          'Recall': recall, 'F1': f1},
+    index=index,
+)
+score_df.round(3)
+
+# %%
+# Next, we will compare:
+#
+# * :class:`~sklearn.linear_model.LogisticRegression` (baseline)
+# * :class:`~sklearn.svm.LinearSVC` (:term:`decision_function` scores
+#   min-max scaled to [0,1], since they are not proper probabilities)
+# * :class:`~sklearn.svm.LinearSVC` with isotonic and sigmoid
+#   calibration (see :ref:`User Guide <calibration>`)
+#
 
-def plot_calibration_curve(est, name, fig_index):
-    """Plot calibration curve for est w/o and with calibration. """
-    # Calibrated with isotonic calibration
-    isotonic = CalibratedClassifierCV(est, cv=2, method='isotonic')
 
-    # Calibrated with sigmoid calibration
-    sigmoid = CalibratedClassifierCV(est, cv=2, method='sigmoid')
+class NaivelyCalibratedLinearSVC(LinearSVC):
+    """LinearSVC with `predict_proba` method that naively scales
+    `decision_function` output for binary classification."""
 
-    # Logistic regression with no calibration as baseline
-    lr = LogisticRegression(C=1.)
+    def fit(self, X, y):
+        super().fit(X, y)
+        df = self.decision_function(X)
+        self.df_min_ = df.min()
+        self.df_max_ = df.max()
 
-    fig = plt.figure(fig_index, figsize=(10, 10))
-    ax1 = plt.subplot2grid((3, 1), (0, 0), rowspan=2)
-    ax2 = plt.subplot2grid((3, 1), (2, 0))
+    def predict_proba(self, X):
+        """Min-max scale output of `decision_function` to [0,1].
 
-    ax1.plot([0, 1], [0, 1], "k:", label="Perfectly calibrated")
-    for clf, name in [(lr, 'Logistic'),
-                      (est, name),
-                      (isotonic, name + ' + Isotonic'),
-                      (sigmoid, name + ' + Sigmoid')]:
-        clf.fit(X_train, y_train)
-        y_pred = clf.predict(X_test)
-        if hasattr(clf, "predict_proba"):
-            prob_pos = clf.predict_proba(X_test)[:, 1]
-        else:  # use decision function
-            prob_pos = clf.decision_function(X_test)
-            prob_pos = \
-                (prob_pos - prob_pos.min()) / (prob_pos.max() - prob_pos.min())
+        Parameters
+        ----------
+        X : ndarray of shape (n_samples, n_features)
+            The data.
 
-        clf_score = brier_score_loss(y_test, prob_pos, pos_label=y.max())
-        print("%s:" % name)
-        print("\tBrier: %1.3f" % (clf_score))
-        print("\tPrecision: %1.3f" % precision_score(y_test, y_pred))
-        print("\tRecall: %1.3f" % recall_score(y_test, y_pred))
-        print("\tF1: %1.3f\n" % f1_score(y_test, y_pred))
+        Returns
+        -------
+        proba : ndarray of shape (n_samples, 2)
+            Predicted probabilities of the 2 classes.
+        """
+        df = self.decision_function(X)
+        calibrated_df = (df - self.df_min_) / (self.df_max_ - self.df_min_)
+        proba_class1 = np.clip(calibrated_df, 0, 1)
+        proba_class0 = 1 - proba_class1
path: examples/calibration/plot_calibration_curve.py
text: @user1:
nit: we usually call these pos_class and neg_class

diff_hunk: @@ -5,130 +5,290 @@
 
 When performing classification one often wants to predict not only the class
 label, but also the associated probability. This probability gives some
-kind of confidence on the prediction. This example demonstrates how to display
-how well calibrated the predicted probabilities are and how to calibrate an
-uncalibrated classifier.
-
-The experiment is performed on an artificial dataset for binary classification
-with 100,000 samples (1,000 of them are used for model fitting) with 20
-features. Of the 20 features, only 2 are informative and 10 are redundant. The
-first figure shows the estimated probabilities obtained with logistic
-regression, Gaussian naive Bayes, and Gaussian naive Bayes with both isotonic
-calibration and sigmoid calibration. The calibration performance is evaluated
-with Brier score, reported in the legend (the smaller the better). One can
-observe here that logistic regression is well calibrated while raw Gaussian
-naive Bayes performs very badly. This is because of the redundant features
-which violate the assumption of feature-independence and result in an overly
-confident classifier, which is indicated by the typical transposed-sigmoid
-curve.
-
-Calibration of the probabilities of Gaussian naive Bayes with isotonic
-regression can fix this issue as can be seen from the nearly diagonal
-calibration curve. Sigmoid calibration also improves the brier score slightly,
-albeit not as strongly as the non-parametric isotonic regression. This can be
-attributed to the fact that we have plenty of calibration data such that the
-greater flexibility of the non-parametric model can be exploited.
-
-The second figure shows the calibration curve of a linear support-vector
-classifier (LinearSVC). LinearSVC shows the opposite behavior as Gaussian
-naive Bayes: the calibration curve has a sigmoid curve, which is typical for
-an under-confident classifier. In the case of LinearSVC, this is caused by the
-margin property of the hinge loss, which lets the model focus on hard samples
-that are close to the decision boundary (the support vectors).
-
-Both kinds of calibration can fix this issue and yield nearly identical
-results. This shows that sigmoid calibration can deal with situations where
-the calibration curve of the base classifier is sigmoid (e.g., for LinearSVC)
-but not where it is transposed-sigmoid (e.g., Gaussian naive Bayes).
+kind of confidence on the prediction. This example demonstrates how to
+visualize how well calibrated the predicted probabilities are using calibration
+curves, also known as reliability diagrams. Calibration of an uncalibrated
+classifier will also be demonstrated.
 """
-print(__doc__)
+
+# %%
 
 # Author: Alexandre Gramfort <alexandre.gramfort@telecom-paristech.fr>
 #         Jan Hendrik Metzen <jhm@informatik.uni-bremen.de>
 # License: BSD Style.
 
 import matplotlib.pyplot as plt
+from matplotlib.gridspec import GridSpec
+import pandas as pd
+import numpy as np
 
-from sklearn import datasets
+from sklearn.datasets import make_classification
+from sklearn.model_selection import train_test_split
 from sklearn.naive_bayes import GaussianNB
 from sklearn.svm import LinearSVC
 from sklearn.linear_model import LogisticRegression
-from sklearn.metrics import (brier_score_loss, precision_score, recall_score,
-                             f1_score)
-from sklearn.calibration import CalibratedClassifierCV, calibration_curve
-from sklearn.model_selection import train_test_split
+from sklearn.metrics import (precision_score, recall_score, f1_score,
+                             brier_score_loss, log_loss)
+from sklearn.calibration import CalibratedClassifierCV, plot_calibration_curve
 
+# %%
+# Dataset
+# -------
+#
+# We will use a synthetic binary classification dataset with 100,000 samples
+# and 20 features. Of the 20 features, only 2 are informative, 10 are
+# redundant (random combinations of the informative features) and the
+# remaining 8 are 'useless' (random numbers). Of the 100,000 samples, 1,000
+# will be used for model fitting and the rest for testing.
 
-# Create dataset of classification task with many redundant and few
-# informative features
-X, y = datasets.make_classification(n_samples=100000, n_features=20,
-                                    n_informative=2, n_redundant=10,
-                                    random_state=42)
+X, y = make_classification(n_samples=100000, n_features=20, n_informative=2,
+                           n_redundant=10, random_state=42)
 
 X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.99,
                                                     random_state=42)
 
+# %%
+# Calibration curves
+# ------------------
+#
+# First, we will compare:
+#
+# * :class:`~sklearn.linear_model.LogisticRegression` (used as baseline
+#   since very often, properly regularized logistic regression is well
+#   calibrated by default thanks to the use of log-loss)
+# * Raw :class:`~sklearn.naive_bayes.GaussianNB`
+# * :class:`~sklearn.naive_bayes.GaussianNB` with isotonic and sigmoid
+#   calibration (see :ref:`User Guide <calibration>`)
+#
+# Calibration curves for all 4 conditions are plotted below, with the average
+# predicted probability for each bin on the x-axis and the fraction of positive
+# classes in each bin on the y-axis.
+#
+
+lr = LogisticRegression(C=1.)
+gnb = GaussianNB()
+gnb_isotonic = CalibratedClassifierCV(gnb, cv=2, method='isotonic')
+gnb_sigmoid = CalibratedClassifierCV(gnb, cv=2, method='sigmoid')
+
+clf_list = [(lr, 'Logistic'),
+            (gnb, 'Naive Bayes'),
+            (gnb_isotonic, 'Naive Bayes + Isotonic'),
+            (gnb_sigmoid, 'Naive Bayes + Sigmoid')]
+
+fig = plt.figure(figsize=(10, 10))
+gs = GridSpec(4, 2)
+colors = plt.cm.get_cmap('Dark2')
+
+ax1 = fig.add_subplot(gs[:2, :2])
+viz_objects = {}
+for i, (clf, name) in enumerate(clf_list):
+    clf.fit(X_train, y_train)
+    viz = plot_calibration_curve(
+        clf, X_test, y_test, n_bins=10, name=name, ax=ax1, color=colors(i)
+    )
+    viz_objects[name] = viz
+
+ax1.grid()
+ax1.set_title('Calibration plots (Naive Bayes)')
+
+# Add histogram
+grid_positions = [(2, 0), (2, 1), (3, 0), (3, 1)]
+for i, (_, name) in enumerate(clf_list):
+    row, col = grid_positions[i]
+    ax = fig.add_subplot(gs[row, col])
+
+    ax.hist(
+        viz_objects[name].y_prob, range=(0, 1), bins=10, label=name,
+        color=colors(i)
+    )
+    ax.set(title=name, xlabel="Mean predicted probability", ylabel="Count")
+
+plt.tight_layout()
+plt.show()
+
+# %%
+# Raw :class:`~sklearn.naive_bayes.GaussianNB` performs very badly because of
+# the redundant features which violate the assumption of feature-independence
+# and result in an overly confident classifier, which is indicated by the
+# typical transposed-sigmoid curve. Calibration of the probabilities of
+# :class:`~sklearn.naive_bayes.GaussianNB` with isotonic regression can fix
+# this issue as can be seen from the nearly diagonal calibration curve. Sigmoid
+# regression also improves calibration slightly,
+# albeit not as strongly as the non-parametric isotonic regression. This can be
+# attributed to the fact that we have plenty of calibration data such that the
+# greater flexibility of the non-parametric model can be exploited.
+#
+# Below we show the Brier score, log loss, precision, recall and F1 score (see
path: examples/calibration/plot_calibration_curve.py
text: @user1:
I would call it brier score **loss** here and in the table. Otherwise it's not really obvious that lower means better.

@author:
Fixed for all occurances

diff_hunk: @@ -5,130 +5,290 @@
 
 When performing classification one often wants to predict not only the class
 label, but also the associated probability. This probability gives some
-kind of confidence on the prediction. This example demonstrates how to display
-how well calibrated the predicted probabilities are and how to calibrate an
-uncalibrated classifier.
-
-The experiment is performed on an artificial dataset for binary classification
-with 100,000 samples (1,000 of them are used for model fitting) with 20
-features. Of the 20 features, only 2 are informative and 10 are redundant. The
-first figure shows the estimated probabilities obtained with logistic
-regression, Gaussian naive Bayes, and Gaussian naive Bayes with both isotonic
-calibration and sigmoid calibration. The calibration performance is evaluated
-with Brier score, reported in the legend (the smaller the better). One can
-observe here that logistic regression is well calibrated while raw Gaussian
-naive Bayes performs very badly. This is because of the redundant features
-which violate the assumption of feature-independence and result in an overly
-confident classifier, which is indicated by the typical transposed-sigmoid
-curve.
-
-Calibration of the probabilities of Gaussian naive Bayes with isotonic
-regression can fix this issue as can be seen from the nearly diagonal
-calibration curve. Sigmoid calibration also improves the brier score slightly,
-albeit not as strongly as the non-parametric isotonic regression. This can be
-attributed to the fact that we have plenty of calibration data such that the
-greater flexibility of the non-parametric model can be exploited.
-
-The second figure shows the calibration curve of a linear support-vector
-classifier (LinearSVC). LinearSVC shows the opposite behavior as Gaussian
-naive Bayes: the calibration curve has a sigmoid curve, which is typical for
-an under-confident classifier. In the case of LinearSVC, this is caused by the
-margin property of the hinge loss, which lets the model focus on hard samples
-that are close to the decision boundary (the support vectors).
-
-Both kinds of calibration can fix this issue and yield nearly identical
-results. This shows that sigmoid calibration can deal with situations where
-the calibration curve of the base classifier is sigmoid (e.g., for LinearSVC)
-but not where it is transposed-sigmoid (e.g., Gaussian naive Bayes).
+kind of confidence on the prediction. This example demonstrates how to
+visualize how well calibrated the predicted probabilities are using calibration
+curves, also known as reliability diagrams. Calibration of an uncalibrated
+classifier will also be demonstrated.
 """
-print(__doc__)
+
+# %%
 
 # Author: Alexandre Gramfort <alexandre.gramfort@telecom-paristech.fr>
 #         Jan Hendrik Metzen <jhm@informatik.uni-bremen.de>
 # License: BSD Style.
 
 import matplotlib.pyplot as plt
+from matplotlib.gridspec import GridSpec
+import pandas as pd
+import numpy as np
 
-from sklearn import datasets
+from sklearn.datasets import make_classification
+from sklearn.model_selection import train_test_split
 from sklearn.naive_bayes import GaussianNB
 from sklearn.svm import LinearSVC
 from sklearn.linear_model import LogisticRegression
-from sklearn.metrics import (brier_score_loss, precision_score, recall_score,
-                             f1_score)
-from sklearn.calibration import CalibratedClassifierCV, calibration_curve
-from sklearn.model_selection import train_test_split
+from sklearn.metrics import (precision_score, recall_score, f1_score,
+                             brier_score_loss, log_loss)
+from sklearn.calibration import CalibratedClassifierCV, plot_calibration_curve
 
+# %%
+# Dataset
+# -------
+#
+# We will use a synthetic binary classification dataset with 100,000 samples
+# and 20 features. Of the 20 features, only 2 are informative, 10 are
+# redundant (random combinations of the informative features) and the
+# remaining 8 are 'useless' (random numbers). Of the 100,000 samples, 1,000
+# will be used for model fitting and the rest for testing.
 
-# Create dataset of classification task with many redundant and few
-# informative features
-X, y = datasets.make_classification(n_samples=100000, n_features=20,
-                                    n_informative=2, n_redundant=10,
-                                    random_state=42)
+X, y = make_classification(n_samples=100000, n_features=20, n_informative=2,
+                           n_redundant=10, random_state=42)
 
 X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.99,
                                                     random_state=42)
 
+# %%
+# Calibration curves
+# ------------------
+#
+# First, we will compare:
+#
+# * :class:`~sklearn.linear_model.LogisticRegression` (used as baseline
+#   since very often, properly regularized logistic regression is well
+#   calibrated by default thanks to the use of log-loss)
+# * Raw :class:`~sklearn.naive_bayes.GaussianNB`
+# * :class:`~sklearn.naive_bayes.GaussianNB` with isotonic and sigmoid
+#   calibration (see :ref:`User Guide <calibration>`)
+#
+# Calibration curves for all 4 conditions are plotted below, with the average
+# predicted probability for each bin on the x-axis and the fraction of positive
+# classes in each bin on the y-axis.
+#
+
+lr = LogisticRegression(C=1.)
+gnb = GaussianNB()
+gnb_isotonic = CalibratedClassifierCV(gnb, cv=2, method='isotonic')
+gnb_sigmoid = CalibratedClassifierCV(gnb, cv=2, method='sigmoid')
+
+clf_list = [(lr, 'Logistic'),
+            (gnb, 'Naive Bayes'),
+            (gnb_isotonic, 'Naive Bayes + Isotonic'),
+            (gnb_sigmoid, 'Naive Bayes + Sigmoid')]
+
+fig = plt.figure(figsize=(10, 10))
+gs = GridSpec(4, 2)
+colors = plt.cm.get_cmap('Dark2')
+
+ax1 = fig.add_subplot(gs[:2, :2])
+viz_objects = {}
+for i, (clf, name) in enumerate(clf_list):
+    clf.fit(X_train, y_train)
+    viz = plot_calibration_curve(
+        clf, X_test, y_test, n_bins=10, name=name, ax=ax1, color=colors(i)
+    )
+    viz_objects[name] = viz
+
+ax1.grid()
+ax1.set_title('Calibration plots (Naive Bayes)')
+
+# Add histogram
+grid_positions = [(2, 0), (2, 1), (3, 0), (3, 1)]
+for i, (_, name) in enumerate(clf_list):
+    row, col = grid_positions[i]
+    ax = fig.add_subplot(gs[row, col])
+
+    ax.hist(
+        viz_objects[name].y_prob, range=(0, 1), bins=10, label=name,
+        color=colors(i)
+    )
+    ax.set(title=name, xlabel="Mean predicted probability", ylabel="Count")
+
+plt.tight_layout()
+plt.show()
+
+# %%
+# Raw :class:`~sklearn.naive_bayes.GaussianNB` performs very badly because of
+# the redundant features which violate the assumption of feature-independence
+# and result in an overly confident classifier, which is indicated by the
+# typical transposed-sigmoid curve. Calibration of the probabilities of
+# :class:`~sklearn.naive_bayes.GaussianNB` with isotonic regression can fix
+# this issue as can be seen from the nearly diagonal calibration curve. Sigmoid
+# regression also improves calibration slightly,
+# albeit not as strongly as the non-parametric isotonic regression. This can be
+# attributed to the fact that we have plenty of calibration data such that the
+# greater flexibility of the non-parametric model can be exploited.
+#
+# Below we show the Brier score, log loss, precision, recall and F1 score (see
+# :ref:`User Guide <precision_recall_f_measure_metrics>`). Notice that
+# although calibration improves the Brier score (a metric composed of
+# calibration loss and refinement loss) and :ref:`log_loss`, it does not
+# significantly alter the prediction accuracy measures (precision, recall and
+# F1 score).
+# This is because calibration should not significantly change prediction
+# probabilities at the location of the decision threshold (at x = 0.5 on the
+# graph). Calibration should however, make the predicted probabilities more
+# accurate and thus more useful for making allocation decisions under
+# uncertainty.
+
+index = []
+brier = []
+logloss = []
+precision = []
+recall = []
+f1 = []
+
+for i, (clf, name) in enumerate(clf_list):
+    clf.fit(X_train, y_train)
+    y_proba = clf.predict_proba(X_test)
+    y_pred = clf.predict(X_test)
+
+    # Create DataFrame index
+    index.append(name)
+    # Store column data
+    brier.append(brier_score_loss(y_test, y_proba[:, 1]))
+    logloss.append(log_loss(y_test, y_proba[:, 1]))
+    precision.append(precision_score(y_test, y_pred))
+    recall.append(recall_score(y_test, y_pred))
+    f1.append(f1_score(y_test, y_pred))
+
+score_df = pd.DataFrame(
+    data={'Brier score': brier, 'Log loss': logloss, 'Precision': precision,
+          'Recall': recall, 'F1': f1},
+    index=index,
+)
+score_df.round(3)
+
+# %%
+# Next, we will compare:
+#
+# * :class:`~sklearn.linear_model.LogisticRegression` (baseline)
+# * :class:`~sklearn.svm.LinearSVC` (:term:`decision_function` scores
+#   min-max scaled to [0,1], since they are not proper probabilities)
+# * :class:`~sklearn.svm.LinearSVC` with isotonic and sigmoid
+#   calibration (see :ref:`User Guide <calibration>`)
+#
 
-def plot_calibration_curve(est, name, fig_index):
-    """Plot calibration curve for est w/o and with calibration. """
-    # Calibrated with isotonic calibration
-    isotonic = CalibratedClassifierCV(est, cv=2, method='isotonic')
 
-    # Calibrated with sigmoid calibration
-    sigmoid = CalibratedClassifierCV(est, cv=2, method='sigmoid')
+class NaivelyCalibratedLinearSVC(LinearSVC):
+    """LinearSVC with `predict_proba` method that naively scales
+    `decision_function` output for binary classification."""
 
-    # Logistic regression with no calibration as baseline
-    lr = LogisticRegression(C=1.)
+    def fit(self, X, y):
+        super().fit(X, y)
+        df = self.decision_function(X)
+        self.df_min_ = df.min()
+        self.df_max_ = df.max()
 
-    fig = plt.figure(fig_index, figsize=(10, 10))
-    ax1 = plt.subplot2grid((3, 1), (0, 0), rowspan=2)
-    ax2 = plt.subplot2grid((3, 1), (2, 0))
+    def predict_proba(self, X):
+        """Min-max scale output of `decision_function` to [0,1].
 
-    ax1.plot([0, 1], [0, 1], "k:", label="Perfectly calibrated")
-    for clf, name in [(lr, 'Logistic'),
-                      (est, name),
-                      (isotonic, name + ' + Isotonic'),
-                      (sigmoid, name + ' + Sigmoid')]:
-        clf.fit(X_train, y_train)
-        y_pred = clf.predict(X_test)
-        if hasattr(clf, "predict_proba"):
-            prob_pos = clf.predict_proba(X_test)[:, 1]
-        else:  # use decision function
-            prob_pos = clf.decision_function(X_test)
-            prob_pos = \
-                (prob_pos - prob_pos.min()) / (prob_pos.max() - prob_pos.min())
+        Parameters
+        ----------
+        X : ndarray of shape (n_samples, n_features)
+            The data.
 
-        clf_score = brier_score_loss(y_test, prob_pos, pos_label=y.max())
-        print("%s:" % name)
-        print("\tBrier: %1.3f" % (clf_score))
-        print("\tPrecision: %1.3f" % precision_score(y_test, y_pred))
-        print("\tRecall: %1.3f" % recall_score(y_test, y_pred))
-        print("\tF1: %1.3f\n" % f1_score(y_test, y_pred))
+        Returns
+        -------
+        proba : ndarray of shape (n_samples, 2)
+            Predicted probabilities of the 2 classes.
+        """
+        df = self.decision_function(X)
+        calibrated_df = (df - self.df_min_) / (self.df_max_ - self.df_min_)
+        proba_class1 = np.clip(calibrated_df, 0, 1)
+        proba_class0 = 1 - proba_class1
+        proba = np.c_[proba_class0, proba_class1]
+        return proba
 
-        fraction_of_positives, mean_predicted_value = \
-            calibration_curve(y_test, prob_pos, n_bins=10)
 
-        ax1.plot(mean_predicted_value, fraction_of_positives, "s-",
-                 label="%s (%1.3f)" % (name, clf_score))
+lr = LogisticRegression(C=1.)
+svc = NaivelyCalibratedLinearSVC(max_iter=10000)
path: examples/calibration/plot_calibration_curve.py
text: @user1:
```suggestion
svc = NaivelyCalibratedLinearSVC(max_iter=10_000)
```

diff_hunk: @@ -4,119 +4,155 @@
 ========================================
 
 Well calibrated classifiers are probabilistic classifiers for which the output
-of the predict_proba method can be directly interpreted as a confidence level.
-For instance a well calibrated (binary) classifier should classify the samples
-such that among the samples to which it gave a predict_proba value close to
-0.8, approx. 80% actually belong to the positive class.
-
-LogisticRegression returns well calibrated predictions as it directly
-optimizes log-loss. In contrast, the other methods return biased probabilities,
-with different biases per method:
-
-* GaussianNaiveBayes tends to push probabilities to 0 or 1 (note the counts in
-  the histograms). This is mainly because it makes the assumption that features
-  are conditionally independent given the class, which is not the case in this
-  dataset which contains 2 redundant features.
-
-* RandomForestClassifier shows the opposite behavior: the histograms show
-  peaks at approx. 0.2 and 0.9 probability, while probabilities close to 0 or 1
-  are very rare. An explanation for this is given by Niculescu-Mizil and Caruana
-  [1]_: "Methods such as bagging and random forests that average predictions
-  from a base set of models can have difficulty making predictions near 0 and 1
-  because variance in the underlying base models will bias predictions that
-  should be near zero or one away from these values. Because predictions are
-  restricted to the interval [0,1], errors caused by variance tend to be one-
-  sided near zero and one. For example, if a model should predict p = 0 for a
-  case, the only way bagging can achieve this is if all bagged trees predict
-  zero. If we add noise to the trees that bagging is averaging over, this noise
-  will cause some trees to predict values larger than 0 for this case, thus
-  moving the average prediction of the bagged ensemble away from 0. We observe
-  this effect most strongly with random forests because the base-level trees
-  trained with random forests have relatively high variance due to feature
-  subsetting." As a result, the calibration curve shows a characteristic
-  sigmoid shape, indicating that the classifier could trust its "intuition"
-  more and return probabilities closer to 0 or 1 typically.
-
-* Support Vector Classification (SVC) shows an even more sigmoid curve as
-  the  RandomForestClassifier, which is typical for maximum-margin methods
-  (compare Niculescu-Mizil and Caruana [1]_), which focus on hard samples
-  that are close to the decision boundary (the support vectors).
-
-.. topic:: References:
-
-    .. [1] Predicting Good Probabilities with Supervised Learning,
-          A. Niculescu-Mizil & R. Caruana, ICML 2005
+of :term:`predict_proba` can be directly interpreted as a confidence level.
+For instance, a well calibrated (binary) classifier should classify the samples
+such that for the samples to which it gave a :term:`predict_proba` value close
+to 0.8, approximately 80% actually belong to the positive class.
 """
-print(__doc__)
+
+# %%
 
 # Author: Jan Hendrik Metzen <jhm@informatik.uni-bremen.de>
 # License: BSD Style.
 
-import numpy as np
-np.random.seed(0)
-
 import matplotlib.pyplot as plt
+from matplotlib.gridspec import GridSpec
+import numpy as np
 
-from sklearn import datasets
+from sklearn.datasets import make_classification
 from sklearn.naive_bayes import GaussianNB
 from sklearn.linear_model import LogisticRegression
 from sklearn.ensemble import RandomForestClassifier
 from sklearn.svm import LinearSVC
-from sklearn.calibration import calibration_curve
-
-X, y = datasets.make_classification(n_samples=100000, n_features=20,
-                                    n_informative=2, n_redundant=2)
+from sklearn.calibration import plot_calibration_curve
+
+# %%
+# Dataset
+# -------
+#
+# We will use a synthetic binary classification dataset with 100,000 samples
+# and 20 features. Of the 20 features, only 2 are informative, 2 are
+# redundant (random combinations of the informative features) and the
+# remaining 16 are 'useless' (random numbers). Of the 100,000 samples,
path: examples/calibration/plot_compare_calibration.py
text: @user1:
uninformative, 100_000, etc. ;)

diff_hunk: @@ -753,3 +755,208 @@ def calibration_curve(y_true, y_prob, *, normalize=False, n_bins=5,
     prob_pred = bin_sums[nonzero] / bin_total[nonzero]
 
     return prob_true, prob_pred
+
+
+class CalibrationDisplay:
+    """Calibration visualization.
+
+    It is recommend to use :func:`~sklearn.calibration.plot_calibration_curve`
+    to create a visualizer. All parameters are stored as attributes.
+
+    Read more in the :ref:`User Guide <calibration>`.
+
+    Parameters
+    -----------
+    prob_true : ndarray
+        The proportion of samples whose class is the positive class (fraction
+        of positives), in each bin.
+
+    prob_pred : ndarray
+        The mean predicted probability in each bin.
+
+    y_prob : ndarray of shape (n_samples,)
+        Probability estimates for the positive class, for each sample.
+
+    estimator_name : str, default=None
+        Name of estimator. If `None`, then the estimator name is not shown.
+
+    Attributes
+    ----------
+    line_ : matplotlib Artist
+        Calibration curve.
+
+    ax_ : matplotlib Axes
+        Axes with calibration curve.
+
+    figure_ : matplotlib Figure
+        Figure containing the curve.
+    """
+    def __init__(self, prob_true, prob_pred, y_prob, *, estimator_name=None):
+        self.prob_true = prob_true
+        self.prob_pred = prob_pred
+        self.y_prob = y_prob
+        self.estimator_name = estimator_name
+
+    def plot(self, ax=None, *, name=None, ref_line=True, **kwargs):
+        """Plot visualization.
+
+        Extra keyword arguments will be passed to
+        :func:`matplotlib.pyplot.plot`.
+
+        Parameters
+        ----------
+        ax : Matplotlib Axes, default=None
+            Axes object to plot on. If `None`, a new figure and axes is
+            created.
+
+        name : str, default=None
+            Name of calibration curve for labeling. If `None`, use the
+            name of the estimator.
+
+        ref_line : bool, default=True
+            If `True`, plots a reference line representing a perfectly
+            calibrated classifier.
+
+        **kwargs : dict
+            Keyword arguments to be passed to :func:`matplotlib.pyplot.plot`.
+
+        Returns
+        -------
+        display : :class:`~sklearn.calibration.CalibrationDisplay`
+            Object that stores computed values.
+        """
+        check_matplotlib_support("CalibrationDisplay.plot")
+        import matplotlib.pyplot as plt
+
+        if ax is None:
+            fig, ax = plt.subplots()
+
+        name = self.estimator_name if name is None else name
+
+        line_kwargs = {}
+        if name is not None:
+            line_kwargs["label"] = name
+        line_kwargs.update(**kwargs)
+
+        existing_ref_line = ('Perfectly calibrated' in
+                             ax.get_legend_handles_labels()[1])
+        if ref_line and not existing_ref_line:
+            ax.plot([0, 1], [0, 1], "k:", label="Perfectly calibrated")
+        self.line_ = ax.plot(self.prob_pred, self.prob_true, "s-",
+                             **line_kwargs)[0]
+
+        if "label" in line_kwargs:
+            ax.legend(loc="lower right")
+
+        ax.set(xlabel="Mean predicted probability",
+               ylabel="Fraction of positives")
+
+        self.ax_ = ax
+        self.figure_ = ax.figure
+        return self
+
+
+def plot_calibration_curve(estimator, X, y, *,
+                           n_bins=5, strategy='uniform',
+                           name=None, ref_line=True, ax=None, **kwargs):
+    """Plot calibration curve, also known as reliability diagrams, for binary
+    classifiers.
+
+    The average predicted probability for each bin is plotted on the x-axis
+    and the fraction of positive classes in each bin is plotted on the y-axis.
+
+    Extra keyword arguments will be passed to :func:`matplotlib.pyplot.plot`.
+
+    Read more in the :ref:`User Guide <calibration>`.
+
+    Parameters
+    ----------
+    estimator : estimator instance
+        Fitted classifier or a fitted :class:`~sklearn.pipeline.Pipeline` in
+        which the last estimator is a classifier. The classifier must
+        have a :term:`predict_proba` method.
+
+    X : {array-like, sparse matrix} of shape (n_samples, n_features)
+        Input values.
+
+    y : array-like of shape (n_samples,)
+        Binary target values.
+
+    n_bins : int, default=5
+        Number of bins to discretize the [0, 1] interval into when calculating
+        the calibration curve. A bigger number requires more data.
+
+    strategy : {'uniform', 'quantile'}, default='uniform'
+        Strategy used to define the widths of the bins.
+
+        - `'uniform'`: The bins have identical widths.
+        - `'quantile'`: The bins have the same number of samples and depend on
+          predicted probabilities.
+
+    name : str, default=None
+        Name for labeling curve. If `None`, the name of the estimator is used.
+
+    ref_line : bool, default=True
+        If `True`, plots a reference line representing a perfectly calibrated
+        classifier.
+
+    ax : matplotlib axes, default=None
+        Axes object to plot on. If `None`, a new figure and axes is created.
+
+    **kwargs : dict
+        Keyword arguments to be passed to :func:`matplotlib.pyplot.plot`.
+
+    Returns
+    -------
+    display : :class:`~sklearn.calibration.CalibrationDisplay`.
+        Object that stores computed values.
+
+    Examples
+    --------
+    >>> import matplotlib.pyplot as plt                     # doctest: +SKIP
+    >>> from sklearn import (datasets, calibration, model_selection,
+    ...                      linear_model)
+    >>> X, y = datasets.make_classification(random_state=0)
+    >>> X_train, X_test, y_train, y_test = model_selection.train_test_split(
+    ...     X, y, random_state=0)
+    >>> clf = linear_model.LogisticRegression(random_state=0)
+    >>> clf.fit(X_train, y_train)
+    LogisticRegression(random_state=0)
+    >>> metrics.plot_calibration_curve(clf, X_test, y_test)  # doctest: +SKIP
+    >>> plt.show()                                           # doctest: +SKIP
+    """
+    check_matplotlib_support("plot_calibration_curve")
+    binary_error = "Only binary classification is supported."
+
+    if not is_classifier(estimator):
+        raise ValueError("The 'estimator' parameter should be a fitted binary "
+                         "classifier")
+
+    try:
path: sklearn/calibration.py
text: @user1:
do we really need a try-except here? It seems that `_check_classifier_response_method` does it internally.

@author:
Ah yes, I did this so I can put in an error message that specifically says we need a `predict_proba`. `_check_classifier_response_method` message says 'decision_function or predict_proba' not defined.

@user1:
It seems that the error message is fine:

```py
    error_msg = "response method {} is not defined in {}"
    if response_method != "auto":
        prediction_method = getattr(estimator, response_method, None)
        if prediction_method is None:
            raise ValueError(error_msg.format(response_method,
                                              estimator.__class__.__name__))
```

@author:
Yes you are right, I have mis-read.

diff_hunk: @@ -5,130 +5,290 @@
 
 When performing classification one often wants to predict not only the class
 label, but also the associated probability. This probability gives some
-kind of confidence on the prediction. This example demonstrates how to display
-how well calibrated the predicted probabilities are and how to calibrate an
-uncalibrated classifier.
-
-The experiment is performed on an artificial dataset for binary classification
-with 100,000 samples (1,000 of them are used for model fitting) with 20
-features. Of the 20 features, only 2 are informative and 10 are redundant. The
-first figure shows the estimated probabilities obtained with logistic
-regression, Gaussian naive Bayes, and Gaussian naive Bayes with both isotonic
-calibration and sigmoid calibration. The calibration performance is evaluated
-with Brier score, reported in the legend (the smaller the better). One can
-observe here that logistic regression is well calibrated while raw Gaussian
-naive Bayes performs very badly. This is because of the redundant features
-which violate the assumption of feature-independence and result in an overly
-confident classifier, which is indicated by the typical transposed-sigmoid
-curve.
-
-Calibration of the probabilities of Gaussian naive Bayes with isotonic
-regression can fix this issue as can be seen from the nearly diagonal
-calibration curve. Sigmoid calibration also improves the brier score slightly,
-albeit not as strongly as the non-parametric isotonic regression. This can be
-attributed to the fact that we have plenty of calibration data such that the
-greater flexibility of the non-parametric model can be exploited.
-
-The second figure shows the calibration curve of a linear support-vector
-classifier (LinearSVC). LinearSVC shows the opposite behavior as Gaussian
-naive Bayes: the calibration curve has a sigmoid curve, which is typical for
-an under-confident classifier. In the case of LinearSVC, this is caused by the
-margin property of the hinge loss, which lets the model focus on hard samples
-that are close to the decision boundary (the support vectors).
-
-Both kinds of calibration can fix this issue and yield nearly identical
-results. This shows that sigmoid calibration can deal with situations where
-the calibration curve of the base classifier is sigmoid (e.g., for LinearSVC)
-but not where it is transposed-sigmoid (e.g., Gaussian naive Bayes).
+kind of confidence on the prediction. This example demonstrates how to
+visualize how well calibrated the predicted probabilities are using calibration
+curves, also known as reliability diagrams. Calibration of an uncalibrated
+classifier will also be demonstrated.
 """
-print(__doc__)
+
+# %%
 
 # Author: Alexandre Gramfort <alexandre.gramfort@telecom-paristech.fr>
 #         Jan Hendrik Metzen <jhm@informatik.uni-bremen.de>
 # License: BSD Style.
 
 import matplotlib.pyplot as plt
+from matplotlib.gridspec import GridSpec
+import pandas as pd
+import numpy as np
 
-from sklearn import datasets
+from sklearn.datasets import make_classification
+from sklearn.model_selection import train_test_split
 from sklearn.naive_bayes import GaussianNB
 from sklearn.svm import LinearSVC
 from sklearn.linear_model import LogisticRegression
-from sklearn.metrics import (brier_score_loss, precision_score, recall_score,
-                             f1_score)
-from sklearn.calibration import CalibratedClassifierCV, calibration_curve
-from sklearn.model_selection import train_test_split
+from sklearn.metrics import (precision_score, recall_score, f1_score,
+                             brier_score_loss, log_loss)
+from sklearn.calibration import CalibratedClassifierCV, plot_calibration_curve
 
+# %%
+# Dataset
+# -------
+#
+# We will use a synthetic binary classification dataset with 100,000 samples
+# and 20 features. Of the 20 features, only 2 are informative, 10 are
+# redundant (random combinations of the informative features) and the
+# remaining 8 are 'useless' (random numbers). Of the 100,000 samples, 1,000
+# will be used for model fitting and the rest for testing.
 
-# Create dataset of classification task with many redundant and few
-# informative features
-X, y = datasets.make_classification(n_samples=100000, n_features=20,
-                                    n_informative=2, n_redundant=10,
-                                    random_state=42)
+X, y = make_classification(n_samples=100000, n_features=20, n_informative=2,
+                           n_redundant=10, random_state=42)
 
 X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.99,
                                                     random_state=42)
 
+# %%
+# Calibration curves
+# ------------------
+#
+# First, we will compare:
+#
+# * :class:`~sklearn.linear_model.LogisticRegression` (used as baseline
+#   since very often, properly regularized logistic regression is well
+#   calibrated by default thanks to the use of log-loss)
+# * Raw :class:`~sklearn.naive_bayes.GaussianNB`
path: examples/calibration/plot_calibration_curve.py
text: @user1:
```suggestion
# * Uncalibrated :class:`~sklearn.naive_bayes.GaussianNB`
```

@author:
Fixed for all occurances

diff_hunk: @@ -4,119 +4,155 @@
 ========================================
 
 Well calibrated classifiers are probabilistic classifiers for which the output
-of the predict_proba method can be directly interpreted as a confidence level.
-For instance a well calibrated (binary) classifier should classify the samples
-such that among the samples to which it gave a predict_proba value close to
-0.8, approx. 80% actually belong to the positive class.
-
-LogisticRegression returns well calibrated predictions as it directly
-optimizes log-loss. In contrast, the other methods return biased probabilities,
-with different biases per method:
-
-* GaussianNaiveBayes tends to push probabilities to 0 or 1 (note the counts in
-  the histograms). This is mainly because it makes the assumption that features
-  are conditionally independent given the class, which is not the case in this
-  dataset which contains 2 redundant features.
-
-* RandomForestClassifier shows the opposite behavior: the histograms show
-  peaks at approx. 0.2 and 0.9 probability, while probabilities close to 0 or 1
-  are very rare. An explanation for this is given by Niculescu-Mizil and Caruana
-  [1]_: "Methods such as bagging and random forests that average predictions
-  from a base set of models can have difficulty making predictions near 0 and 1
-  because variance in the underlying base models will bias predictions that
-  should be near zero or one away from these values. Because predictions are
-  restricted to the interval [0,1], errors caused by variance tend to be one-
-  sided near zero and one. For example, if a model should predict p = 0 for a
-  case, the only way bagging can achieve this is if all bagged trees predict
-  zero. If we add noise to the trees that bagging is averaging over, this noise
-  will cause some trees to predict values larger than 0 for this case, thus
-  moving the average prediction of the bagged ensemble away from 0. We observe
-  this effect most strongly with random forests because the base-level trees
-  trained with random forests have relatively high variance due to feature
-  subsetting." As a result, the calibration curve shows a characteristic
-  sigmoid shape, indicating that the classifier could trust its "intuition"
-  more and return probabilities closer to 0 or 1 typically.
-
-* Support Vector Classification (SVC) shows an even more sigmoid curve as
-  the  RandomForestClassifier, which is typical for maximum-margin methods
-  (compare Niculescu-Mizil and Caruana [1]_), which focus on hard samples
-  that are close to the decision boundary (the support vectors).
-
-.. topic:: References:
-
-    .. [1] Predicting Good Probabilities with Supervised Learning,
-          A. Niculescu-Mizil & R. Caruana, ICML 2005
+of :term:`predict_proba` can be directly interpreted as a confidence level.
+For instance, a well calibrated (binary) classifier should classify the samples
+such that for the samples to which it gave a :term:`predict_proba` value close
+to 0.8, approximately 80% actually belong to the positive class.
 """
-print(__doc__)
+
+# %%
 
 # Author: Jan Hendrik Metzen <jhm@informatik.uni-bremen.de>
 # License: BSD Style.
 
-import numpy as np
-np.random.seed(0)
-
 import matplotlib.pyplot as plt
+from matplotlib.gridspec import GridSpec
+import numpy as np
 
-from sklearn import datasets
+from sklearn.datasets import make_classification
 from sklearn.naive_bayes import GaussianNB
 from sklearn.linear_model import LogisticRegression
 from sklearn.ensemble import RandomForestClassifier
 from sklearn.svm import LinearSVC
-from sklearn.calibration import calibration_curve
-
-X, y = datasets.make_classification(n_samples=100000, n_features=20,
-                                    n_informative=2, n_redundant=2)
+from sklearn.calibration import plot_calibration_curve
+
+# %%
+# Dataset
+# -------
+#
+# We will use a synthetic binary classification dataset with 100,000 samples
+# and 20 features. Of the 20 features, only 2 are informative, 2 are
+# redundant (random combinations of the informative features) and the
+# remaining 16 are 'useless' (random numbers). Of the 100,000 samples,
+# 100 will be used for model fitting and the remaining for testing.
+
+X, y = make_classification(
+  n_samples=100000, n_features=20, n_informative=2, n_redundant=2,
+  random_state=42
+)
 
 train_samples = 100  # Samples used for training the models
-
 X_train = X[:train_samples]
 X_test = X[train_samples:]
 y_train = y[:train_samples]
 y_test = y[train_samples:]
 
+# %%
+# Calibration curves
+# ------------------
+#
+# :class:`~sklearn.linear_model.LogisticRegression` returns well calibrated
+# predictions as it directly optimizes log-loss. In contrast, the other methods
+# return biased probabilities, with different biases per method:
+#
+# * :class:`~sklearn.naive_bayes.GaussianNB` tends to push
+#   probabilities to 0 or 1 (see histogram). This is mainly
+#   because it makes the assumption that features are conditionally independent
+#   given the class, which is not the case in this dataset which contains 2
+#   redundant features.
+#
+# * :class:`~sklearn.ensemble.RandomForestClassifier` shows the opposite
+#   behavior: the histograms show peaks at approx. 0.2 and 0.9 probability,
+#   while probabilities close to 0 or 1 are very rare. An explanation for this
+#   is given by Niculescu-Mizil and Caruana [1]_: "Methods such as bagging and
+#   random forests that average predictions from a base set of models can have
+#   difficulty making predictions near 0 and 1 because variance in the
+#   underlying base models will bias predictions that should be near zero or
+#   one away from these values. Because predictions are restricted to the
+#   interval [0,1], errors caused by variance tend to be one- sided near zero
+#   and one. For example, if a model should predict p = 0 for a case, the only
+#   way bagging can achieve this is if all bagged trees predict zero. If we add
+#   noise to the trees that bagging is averaging over, this noise will cause
+#   some trees to predict values larger than 0 for this case, thus moving the
+#   average prediction of the bagged ensemble away from 0. We observe this
+#   effect most strongly with random forests because the base-level trees
+#   trained with random forests have relatively high variance due to feature
+#   subsetting." As a result, the calibration curve shows a characteristic
+#   sigmoid shape, indicating that the classifier could trust its "intuition"
+#   more and return probabilities closer to 0 or 1 typically.
+#
+# * we use naively calibrated (min-max scaled to [0,1])
+#   :term:`decision_function` values to show the performance of
+#   :class:`~sklearn.svm.LinearSVC`. :class:`~sklearn.svm.LinearSVC` shows an
+#   even more sigmoid curve than the
+#   :class:`~sklearn.ensemble.RandomForestClassifier`, which is typical for
+#   maximum-margin methods [1]_ as they focus on difficult to classify samples
+#   that are close to the decision boundary (the support vectors).
+
+
+class NaivelyCalibratedLinearSVC(LinearSVC):
+    """LinearSVC with `predict_proba` method that naively scales
+    `decision_function` output."""
+
+    def fit(self, X, y):
+        super().fit(X, y)
+        df = self.decision_function(X)
+        self.df_min_ = df.min()
+        self.df_max_ = df.max()
+
+    def predict_proba(self, X):
+        """Min-max scale output of `decision_function` to [0,1]."""
+        df = self.decision_function(X)
+        calibrated_df = (df - self.df_min_) / (self.df_max_ - self.df_min_)
+        return np.clip(calibrated_df, 0, 1)
path: examples/calibration/plot_compare_calibration.py
text: @user1:
I think we should do the same as in the previous example and return a  2d array, which is our convention.

diff_hunk: @@ -5,130 +5,290 @@
 
 When performing classification one often wants to predict not only the class
 label, but also the associated probability. This probability gives some
-kind of confidence on the prediction. This example demonstrates how to display
-how well calibrated the predicted probabilities are and how to calibrate an
-uncalibrated classifier.
-
-The experiment is performed on an artificial dataset for binary classification
-with 100,000 samples (1,000 of them are used for model fitting) with 20
-features. Of the 20 features, only 2 are informative and 10 are redundant. The
-first figure shows the estimated probabilities obtained with logistic
-regression, Gaussian naive Bayes, and Gaussian naive Bayes with both isotonic
-calibration and sigmoid calibration. The calibration performance is evaluated
-with Brier score, reported in the legend (the smaller the better). One can
-observe here that logistic regression is well calibrated while raw Gaussian
-naive Bayes performs very badly. This is because of the redundant features
-which violate the assumption of feature-independence and result in an overly
-confident classifier, which is indicated by the typical transposed-sigmoid
-curve.
-
-Calibration of the probabilities of Gaussian naive Bayes with isotonic
-regression can fix this issue as can be seen from the nearly diagonal
-calibration curve. Sigmoid calibration also improves the brier score slightly,
-albeit not as strongly as the non-parametric isotonic regression. This can be
-attributed to the fact that we have plenty of calibration data such that the
-greater flexibility of the non-parametric model can be exploited.
-
-The second figure shows the calibration curve of a linear support-vector
-classifier (LinearSVC). LinearSVC shows the opposite behavior as Gaussian
-naive Bayes: the calibration curve has a sigmoid curve, which is typical for
-an under-confident classifier. In the case of LinearSVC, this is caused by the
-margin property of the hinge loss, which lets the model focus on hard samples
-that are close to the decision boundary (the support vectors).
-
-Both kinds of calibration can fix this issue and yield nearly identical
-results. This shows that sigmoid calibration can deal with situations where
-the calibration curve of the base classifier is sigmoid (e.g., for LinearSVC)
-but not where it is transposed-sigmoid (e.g., Gaussian naive Bayes).
+kind of confidence on the prediction. This example demonstrates how to
+visualize how well calibrated the predicted probabilities are using calibration
+curves, also known as reliability diagrams. Calibration of an uncalibrated
+classifier will also be demonstrated.
 """
-print(__doc__)
+
+# %%
 
 # Author: Alexandre Gramfort <alexandre.gramfort@telecom-paristech.fr>
 #         Jan Hendrik Metzen <jhm@informatik.uni-bremen.de>
 # License: BSD Style.
 
 import matplotlib.pyplot as plt
+from matplotlib.gridspec import GridSpec
+import pandas as pd
+import numpy as np
 
-from sklearn import datasets
+from sklearn.datasets import make_classification
+from sklearn.model_selection import train_test_split
 from sklearn.naive_bayes import GaussianNB
 from sklearn.svm import LinearSVC
 from sklearn.linear_model import LogisticRegression
-from sklearn.metrics import (brier_score_loss, precision_score, recall_score,
-                             f1_score)
-from sklearn.calibration import CalibratedClassifierCV, calibration_curve
-from sklearn.model_selection import train_test_split
+from sklearn.metrics import (precision_score, recall_score, f1_score,
+                             brier_score_loss, log_loss)
+from sklearn.calibration import CalibratedClassifierCV, plot_calibration_curve
 
+# %%
+# Dataset
+# -------
+#
+# We will use a synthetic binary classification dataset with 100,000 samples
+# and 20 features. Of the 20 features, only 2 are informative, 10 are
+# redundant (random combinations of the informative features) and the
+# remaining 8 are 'useless' (random numbers). Of the 100,000 samples, 1,000
+# will be used for model fitting and the rest for testing.
 
-# Create dataset of classification task with many redundant and few
-# informative features
-X, y = datasets.make_classification(n_samples=100000, n_features=20,
-                                    n_informative=2, n_redundant=10,
-                                    random_state=42)
+X, y = make_classification(n_samples=100000, n_features=20, n_informative=2,
+                           n_redundant=10, random_state=42)
 
 X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.99,
                                                     random_state=42)
 
+# %%
+# Calibration curves
+# ------------------
+#
+# First, we will compare:
+#
+# * :class:`~sklearn.linear_model.LogisticRegression` (used as baseline
+#   since very often, properly regularized logistic regression is well
+#   calibrated by default thanks to the use of log-loss)
+# * Raw :class:`~sklearn.naive_bayes.GaussianNB`
+# * :class:`~sklearn.naive_bayes.GaussianNB` with isotonic and sigmoid
+#   calibration (see :ref:`User Guide <calibration>`)
+#
+# Calibration curves for all 4 conditions are plotted below, with the average
+# predicted probability for each bin on the x-axis and the fraction of positive
+# classes in each bin on the y-axis.
+#
+
+lr = LogisticRegression(C=1.)
+gnb = GaussianNB()
+gnb_isotonic = CalibratedClassifierCV(gnb, cv=2, method='isotonic')
+gnb_sigmoid = CalibratedClassifierCV(gnb, cv=2, method='sigmoid')
+
+clf_list = [(lr, 'Logistic'),
+            (gnb, 'Naive Bayes'),
+            (gnb_isotonic, 'Naive Bayes + Isotonic'),
+            (gnb_sigmoid, 'Naive Bayes + Sigmoid')]
+
+fig = plt.figure(figsize=(10, 10))
+gs = GridSpec(4, 2)
+colors = plt.cm.get_cmap('Dark2')
+
+ax1 = fig.add_subplot(gs[:2, :2])
+viz_objects = {}
+for i, (clf, name) in enumerate(clf_list):
+    clf.fit(X_train, y_train)
+    viz = plot_calibration_curve(
+        clf, X_test, y_test, n_bins=10, name=name, ax=ax1, color=colors(i)
+    )
+    viz_objects[name] = viz
+
+ax1.grid()
+ax1.set_title('Calibration plots (Naive Bayes)')
+
+# Add histogram
+grid_positions = [(2, 0), (2, 1), (3, 0), (3, 1)]
+for i, (_, name) in enumerate(clf_list):
+    row, col = grid_positions[i]
+    ax = fig.add_subplot(gs[row, col])
+
+    ax.hist(
+        viz_objects[name].y_prob, range=(0, 1), bins=10, label=name,
+        color=colors(i)
+    )
+    ax.set(title=name, xlabel="Mean predicted probability", ylabel="Count")
+
+plt.tight_layout()
+plt.show()
+
+# %%
+# Raw :class:`~sklearn.naive_bayes.GaussianNB` performs very badly because of
path: examples/calibration/plot_calibration_curve.py
text: @user1:
raw -> uncalibrated

Also maybe "performs very badly" -> "is poorly calibrated", since "performance" may be ambiguous

diff_hunk: @@ -4,119 +4,155 @@
 ========================================
 
 Well calibrated classifiers are probabilistic classifiers for which the output
-of the predict_proba method can be directly interpreted as a confidence level.
-For instance a well calibrated (binary) classifier should classify the samples
-such that among the samples to which it gave a predict_proba value close to
-0.8, approx. 80% actually belong to the positive class.
-
-LogisticRegression returns well calibrated predictions as it directly
-optimizes log-loss. In contrast, the other methods return biased probabilities,
-with different biases per method:
-
-* GaussianNaiveBayes tends to push probabilities to 0 or 1 (note the counts in
-  the histograms). This is mainly because it makes the assumption that features
-  are conditionally independent given the class, which is not the case in this
-  dataset which contains 2 redundant features.
-
-* RandomForestClassifier shows the opposite behavior: the histograms show
-  peaks at approx. 0.2 and 0.9 probability, while probabilities close to 0 or 1
-  are very rare. An explanation for this is given by Niculescu-Mizil and Caruana
-  [1]_: "Methods such as bagging and random forests that average predictions
-  from a base set of models can have difficulty making predictions near 0 and 1
-  because variance in the underlying base models will bias predictions that
-  should be near zero or one away from these values. Because predictions are
-  restricted to the interval [0,1], errors caused by variance tend to be one-
-  sided near zero and one. For example, if a model should predict p = 0 for a
-  case, the only way bagging can achieve this is if all bagged trees predict
-  zero. If we add noise to the trees that bagging is averaging over, this noise
-  will cause some trees to predict values larger than 0 for this case, thus
-  moving the average prediction of the bagged ensemble away from 0. We observe
-  this effect most strongly with random forests because the base-level trees
-  trained with random forests have relatively high variance due to feature
-  subsetting." As a result, the calibration curve shows a characteristic
-  sigmoid shape, indicating that the classifier could trust its "intuition"
-  more and return probabilities closer to 0 or 1 typically.
-
-* Support Vector Classification (SVC) shows an even more sigmoid curve as
-  the  RandomForestClassifier, which is typical for maximum-margin methods
-  (compare Niculescu-Mizil and Caruana [1]_), which focus on hard samples
-  that are close to the decision boundary (the support vectors).
-
-.. topic:: References:
-
-    .. [1] Predicting Good Probabilities with Supervised Learning,
-          A. Niculescu-Mizil & R. Caruana, ICML 2005
+of :term:`predict_proba` can be directly interpreted as a confidence level.
+For instance, a well calibrated (binary) classifier should classify the samples
+such that for the samples to which it gave a :term:`predict_proba` value close
+to 0.8, approximately 80% actually belong to the positive class.
 """
-print(__doc__)
+
+# %%
 
 # Author: Jan Hendrik Metzen <jhm@informatik.uni-bremen.de>
 # License: BSD Style.
 
-import numpy as np
-np.random.seed(0)
-
 import matplotlib.pyplot as plt
+from matplotlib.gridspec import GridSpec
+import numpy as np
 
-from sklearn import datasets
+from sklearn.datasets import make_classification
 from sklearn.naive_bayes import GaussianNB
 from sklearn.linear_model import LogisticRegression
 from sklearn.ensemble import RandomForestClassifier
 from sklearn.svm import LinearSVC
-from sklearn.calibration import calibration_curve
-
-X, y = datasets.make_classification(n_samples=100000, n_features=20,
-                                    n_informative=2, n_redundant=2)
+from sklearn.calibration import plot_calibration_curve
+
+# %%
+# Dataset
+# -------
+#
+# We will use a synthetic binary classification dataset with 100,000 samples
+# and 20 features. Of the 20 features, only 2 are informative, 2 are
+# redundant (random combinations of the informative features) and the
+# remaining 16 are 'useless' (random numbers). Of the 100,000 samples,
+# 100 will be used for model fitting and the remaining for testing.
+
+X, y = make_classification(
+  n_samples=100000, n_features=20, n_informative=2, n_redundant=2,
+  random_state=42
+)
 
 train_samples = 100  # Samples used for training the models
-
 X_train = X[:train_samples]
 X_test = X[train_samples:]
 y_train = y[:train_samples]
 y_test = y[train_samples:]
 
+# %%
+# Calibration curves
+# ------------------
+#
+# :class:`~sklearn.linear_model.LogisticRegression` returns well calibrated
+# predictions as it directly optimizes log-loss. In contrast, the other methods
+# return biased probabilities, with different biases per method:
+#
+# * :class:`~sklearn.naive_bayes.GaussianNB` tends to push
+#   probabilities to 0 or 1 (see histogram). This is mainly
+#   because it makes the assumption that features are conditionally independent
+#   given the class, which is not the case in this dataset which contains 2
+#   redundant features.
+#
+# * :class:`~sklearn.ensemble.RandomForestClassifier` shows the opposite
+#   behavior: the histograms show peaks at approx. 0.2 and 0.9 probability,
+#   while probabilities close to 0 or 1 are very rare. An explanation for this
+#   is given by Niculescu-Mizil and Caruana [1]_: "Methods such as bagging and
+#   random forests that average predictions from a base set of models can have
+#   difficulty making predictions near 0 and 1 because variance in the
+#   underlying base models will bias predictions that should be near zero or
+#   one away from these values. Because predictions are restricted to the
+#   interval [0,1], errors caused by variance tend to be one- sided near zero
+#   and one. For example, if a model should predict p = 0 for a case, the only
+#   way bagging can achieve this is if all bagged trees predict zero. If we add
+#   noise to the trees that bagging is averaging over, this noise will cause
+#   some trees to predict values larger than 0 for this case, thus moving the
+#   average prediction of the bagged ensemble away from 0. We observe this
+#   effect most strongly with random forests because the base-level trees
+#   trained with random forests have relatively high variance due to feature
+#   subsetting." As a result, the calibration curve shows a characteristic
+#   sigmoid shape, indicating that the classifier could trust its "intuition"
+#   more and return probabilities closer to 0 or 1 typically.
path: examples/calibration/plot_compare_calibration.py
text: @user1:
maybe just this, as to be consistent with the previous example

```suggestion
#   subsetting." As a result, the calibration curve shows a characteristic
#   sigmoid shape, indicating that the classifier is under-confident
```

diff_hunk: @@ -753,3 +755,208 @@ def calibration_curve(y_true, y_prob, *, normalize=False, n_bins=5,
     prob_pred = bin_sums[nonzero] / bin_total[nonzero]
 
     return prob_true, prob_pred
+
+
+class CalibrationDisplay:
+    """Calibration visualization.
+
+    It is recommend to use :func:`~sklearn.calibration.plot_calibration_curve`
+    to create a visualizer. All parameters are stored as attributes.
+
+    Read more in the :ref:`User Guide <calibration>`.
+
+    Parameters
+    -----------
+    prob_true : ndarray
+        The proportion of samples whose class is the positive class (fraction
+        of positives), in each bin.
+
+    prob_pred : ndarray
+        The mean predicted probability in each bin.
+
+    y_prob : ndarray of shape (n_samples,)
+        Probability estimates for the positive class, for each sample.
+
+    estimator_name : str, default=None
+        Name of estimator. If `None`, then the estimator name is not shown.
+
+    Attributes
+    ----------
+    line_ : matplotlib Artist
+        Calibration curve.
+
+    ax_ : matplotlib Axes
+        Axes with calibration curve.
+
+    figure_ : matplotlib Figure
+        Figure containing the curve.
+    """
+    def __init__(self, prob_true, prob_pred, y_prob, *, estimator_name=None):
+        self.prob_true = prob_true
+        self.prob_pred = prob_pred
+        self.y_prob = y_prob
+        self.estimator_name = estimator_name
+
+    def plot(self, ax=None, *, name=None, ref_line=True, **kwargs):
+        """Plot visualization.
+
+        Extra keyword arguments will be passed to
+        :func:`matplotlib.pyplot.plot`.
+
+        Parameters
+        ----------
+        ax : Matplotlib Axes, default=None
+            Axes object to plot on. If `None`, a new figure and axes is
+            created.
+
+        name : str, default=None
+            Name of calibration curve for labeling. If `None`, use the
+            name of the estimator.
+
+        ref_line : bool, default=True
+            If `True`, plots a reference line representing a perfectly
+            calibrated classifier.
+
+        **kwargs : dict
+            Keyword arguments to be passed to :func:`matplotlib.pyplot.plot`.
+
+        Returns
+        -------
+        display : :class:`~sklearn.calibration.CalibrationDisplay`
+            Object that stores computed values.
+        """
+        check_matplotlib_support("CalibrationDisplay.plot")
+        import matplotlib.pyplot as plt
+
+        if ax is None:
+            fig, ax = plt.subplots()
+
+        name = self.estimator_name if name is None else name
+
+        line_kwargs = {}
+        if name is not None:
+            line_kwargs["label"] = name
+        line_kwargs.update(**kwargs)
+
+        existing_ref_line = ('Perfectly calibrated' in
+                             ax.get_legend_handles_labels()[1])
+        if ref_line and not existing_ref_line:
+            ax.plot([0, 1], [0, 1], "k:", label="Perfectly calibrated")
+        self.line_ = ax.plot(self.prob_pred, self.prob_true, "s-",
+                             **line_kwargs)[0]
+
+        if "label" in line_kwargs:
+            ax.legend(loc="lower right")
+
+        ax.set(xlabel="Mean predicted probability",
+               ylabel="Fraction of positives")
+
+        self.ax_ = ax
+        self.figure_ = ax.figure
+        return self
+
+
+def plot_calibration_curve(estimator, X, y, *,
+                           n_bins=5, strategy='uniform',
+                           name=None, ref_line=True, ax=None, **kwargs):
+    """Plot calibration curve, also known as reliability diagrams, for binary
+    classifiers.
+
+    The average predicted probability for each bin is plotted on the x-axis
+    and the fraction of positive classes in each bin is plotted on the y-axis.
+
+    Extra keyword arguments will be passed to :func:`matplotlib.pyplot.plot`.
+
+    Read more in the :ref:`User Guide <calibration>`.
+
+    Parameters
+    ----------
+    estimator : estimator instance
+        Fitted classifier or a fitted :class:`~sklearn.pipeline.Pipeline` in
+        which the last estimator is a classifier. The classifier must
+        have a :term:`predict_proba` method.
+
+    X : {array-like, sparse matrix} of shape (n_samples, n_features)
+        Input values.
+
+    y : array-like of shape (n_samples,)
+        Binary target values.
+
+    n_bins : int, default=5
+        Number of bins to discretize the [0, 1] interval into when calculating
+        the calibration curve. A bigger number requires more data.
+
+    strategy : {'uniform', 'quantile'}, default='uniform'
+        Strategy used to define the widths of the bins.
+
+        - `'uniform'`: The bins have identical widths.
+        - `'quantile'`: The bins have the same number of samples and depend on
+          predicted probabilities.
+
+    name : str, default=None
+        Name for labeling curve. If `None`, the name of the estimator is used.
+
+    ref_line : bool, default=True
+        If `True`, plots a reference line representing a perfectly calibrated
+        classifier.
+
+    ax : matplotlib axes, default=None
+        Axes object to plot on. If `None`, a new figure and axes is created.
+
+    **kwargs : dict
+        Keyword arguments to be passed to :func:`matplotlib.pyplot.plot`.
+
+    Returns
+    -------
+    display : :class:`~sklearn.calibration.CalibrationDisplay`.
+        Object that stores computed values.
+
+    Examples
+    --------
+    >>> import matplotlib.pyplot as plt                     # doctest: +SKIP
+    >>> from sklearn import (datasets, calibration, model_selection,
+    ...                      linear_model)
+    >>> X, y = datasets.make_classification(random_state=0)
+    >>> X_train, X_test, y_train, y_test = model_selection.train_test_split(
+    ...     X, y, random_state=0)
+    >>> clf = linear_model.LogisticRegression(random_state=0)
+    >>> clf.fit(X_train, y_train)
+    LogisticRegression(random_state=0)
+    >>> metrics.plot_calibration_curve(clf, X_test, y_test)  # doctest: +SKIP
+    >>> plt.show()                                           # doctest: +SKIP
+    """
+    check_matplotlib_support("plot_calibration_curve")
+    binary_error = "Only binary classification is supported."
+
+    if not is_classifier(estimator):
+        raise ValueError("The 'estimator' parameter should be a fitted binary "
+                         "classifier")
+
+    try:
+        prediction_method = _check_classifier_response_method(
+            estimator, response_method='predict_proba'
+        )
+    except ValueError:
+        raise ValueError("Response method 'predict_proba' not defined in "
+                         f"{estimator.__class__.__name__}")
+
+    y_prob = prediction_method(X)
+
+    if not len(estimator.classes_) == 2:
+        raise ValueError(binary_error)
+    if y_prob.ndim != 1:
path: sklearn/calibration.py
text: @user1:
In relation to my comment about the output of NaivelyCalibratedLinearSVC.predict_proba, I don't think we should allow 1d vectors here. I could be wrong but I don't think such estimators would be API-compliant.

diff_hunk: @@ -753,3 +755,208 @@ def calibration_curve(y_true, y_prob, *, normalize=False, n_bins=5,
     prob_pred = bin_sums[nonzero] / bin_total[nonzero]
 
     return prob_true, prob_pred
+
+
+class CalibrationDisplay:
+    """Calibration visualization.
+
+    It is recommend to use :func:`~sklearn.calibration.plot_calibration_curve`
+    to create a visualizer. All parameters are stored as attributes.
+
+    Read more in the :ref:`User Guide <calibration>`.
+
+    Parameters
+    -----------
+    prob_true : ndarray
+        The proportion of samples whose class is the positive class (fraction
+        of positives), in each bin.
+
+    prob_pred : ndarray
+        The mean predicted probability in each bin.
+
+    y_prob : ndarray of shape (n_samples,)
+        Probability estimates for the positive class, for each sample.
+
+    estimator_name : str, default=None
+        Name of estimator. If `None`, then the estimator name is not shown.
+
+    Attributes
+    ----------
+    line_ : matplotlib Artist
+        Calibration curve.
+
+    ax_ : matplotlib Axes
+        Axes with calibration curve.
+
+    figure_ : matplotlib Figure
+        Figure containing the curve.
+    """
+    def __init__(self, prob_true, prob_pred, y_prob, *, estimator_name=None):
+        self.prob_true = prob_true
+        self.prob_pred = prob_pred
+        self.y_prob = y_prob
+        self.estimator_name = estimator_name
+
+    def plot(self, ax=None, *, name=None, ref_line=True, **kwargs):
+        """Plot visualization.
+
+        Extra keyword arguments will be passed to
+        :func:`matplotlib.pyplot.plot`.
+
+        Parameters
+        ----------
+        ax : Matplotlib Axes, default=None
+            Axes object to plot on. If `None`, a new figure and axes is
+            created.
+
+        name : str, default=None
+            Name of calibration curve for labeling. If `None`, use the
+            name of the estimator.
+
+        ref_line : bool, default=True
+            If `True`, plots a reference line representing a perfectly
+            calibrated classifier.
+
+        **kwargs : dict
+            Keyword arguments to be passed to :func:`matplotlib.pyplot.plot`.
+
+        Returns
+        -------
+        display : :class:`~sklearn.calibration.CalibrationDisplay`
+            Object that stores computed values.
+        """
+        check_matplotlib_support("CalibrationDisplay.plot")
+        import matplotlib.pyplot as plt
+
+        if ax is None:
+            fig, ax = plt.subplots()
+
+        name = self.estimator_name if name is None else name
+
+        line_kwargs = {}
+        if name is not None:
+            line_kwargs["label"] = name
+        line_kwargs.update(**kwargs)
+
+        existing_ref_line = ('Perfectly calibrated' in
+                             ax.get_legend_handles_labels()[1])
+        if ref_line and not existing_ref_line:
+            ax.plot([0, 1], [0, 1], "k:", label="Perfectly calibrated")
+        self.line_ = ax.plot(self.prob_pred, self.prob_true, "s-",
+                             **line_kwargs)[0]
+
+        if "label" in line_kwargs:
+            ax.legend(loc="lower right")
+
+        ax.set(xlabel="Mean predicted probability",
+               ylabel="Fraction of positives")
+
+        self.ax_ = ax
+        self.figure_ = ax.figure
+        return self
+
+
+def plot_calibration_curve(estimator, X, y, *,
+                           n_bins=5, strategy='uniform',
+                           name=None, ref_line=True, ax=None, **kwargs):
+    """Plot calibration curve, also known as reliability diagrams, for binary
+    classifiers.
+
+    The average predicted probability for each bin is plotted on the x-axis
+    and the fraction of positive classes in each bin is plotted on the y-axis.
+
+    Extra keyword arguments will be passed to :func:`matplotlib.pyplot.plot`.
+
+    Read more in the :ref:`User Guide <calibration>`.
+
+    Parameters
+    ----------
+    estimator : estimator instance
+        Fitted classifier or a fitted :class:`~sklearn.pipeline.Pipeline` in
+        which the last estimator is a classifier. The classifier must
+        have a :term:`predict_proba` method.
+
+    X : {array-like, sparse matrix} of shape (n_samples, n_features)
+        Input values.
+
+    y : array-like of shape (n_samples,)
+        Binary target values.
+
+    n_bins : int, default=5
+        Number of bins to discretize the [0, 1] interval into when calculating
+        the calibration curve. A bigger number requires more data.
+
+    strategy : {'uniform', 'quantile'}, default='uniform'
+        Strategy used to define the widths of the bins.
+
+        - `'uniform'`: The bins have identical widths.
+        - `'quantile'`: The bins have the same number of samples and depend on
+          predicted probabilities.
+
+    name : str, default=None
+        Name for labeling curve. If `None`, the name of the estimator is used.
+
+    ref_line : bool, default=True
+        If `True`, plots a reference line representing a perfectly calibrated
+        classifier.
+
+    ax : matplotlib axes, default=None
+        Axes object to plot on. If `None`, a new figure and axes is created.
+
+    **kwargs : dict
+        Keyword arguments to be passed to :func:`matplotlib.pyplot.plot`.
+
+    Returns
+    -------
+    display : :class:`~sklearn.calibration.CalibrationDisplay`.
+        Object that stores computed values.
+
+    Examples
+    --------
+    >>> import matplotlib.pyplot as plt                     # doctest: +SKIP
+    >>> from sklearn import (datasets, calibration, model_selection,
+    ...                      linear_model)
+    >>> X, y = datasets.make_classification(random_state=0)
+    >>> X_train, X_test, y_train, y_test = model_selection.train_test_split(
+    ...     X, y, random_state=0)
+    >>> clf = linear_model.LogisticRegression(random_state=0)
+    >>> clf.fit(X_train, y_train)
+    LogisticRegression(random_state=0)
+    >>> metrics.plot_calibration_curve(clf, X_test, y_test)  # doctest: +SKIP
+    >>> plt.show()                                           # doctest: +SKIP
path: sklearn/calibration.py
text: @user1:
this fails: metrics isn't imported and `plot_calibration_curbe` isn't there anyway. We need to be careful about skipping doctests ;)