Repository: scikit-learn/scikit-learn
Pull request: ENH Add libsvm-like calibration procedure to CalibratedClassifierCV
<!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#pull-request-checklist
-->

#### Reference Issues/PRs
closes #16145
Follows from PR #16167
The first step was to refactor `CalibratedClassifierCV`, which was done in #17803. This is a branch off of #17803. (will close #17803)


#### What does this implement/fix? Explain your changes.
Add `ensemble` option to `CalibratedClassifierCV`.
`ensemble=True` would be the default and would be the current implementation.
`ensemble=False` would be the same as the [libsvm implementation](https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/svm/src/libsvm/svm.cpp#L2099) (used when `probability=True`):

When `cv` is not `'prefit'`:
- Concatenate predictions from each cv. Use all predictions to train calibrator.
- Classifiers fit after each cv would be [destroyed](https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/svm/src/libsvm/svm.cpp#L2169)
- Use classifier fit on all data at prediction time

Note `ensemble=False` is the procedure described in the original Platt paper (see https://github.com/scikit-learn/scikit-learn/issues/16145#issue-551351752)

#### Any other comments?


<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->


Problem statement:
CalibratedClassifierCV should return an estimator calibrated via CV, not an ensemble of calibrated estimators
The current cross validation procedure adopted in the `CalibratedClassifierCV` does not follow the cross validation procedure described in the original Platt paper:
[Platt99] Probabilistic Outputs for Support Vector Machines and Comparisons to Regularized Likelihood Methods, J. Platt, (1999)

I checked also the other papers cited in the references for the `CalibratedClassifierCV` class and none of them describes the cross validation process it implements.

`CalibratedClassifierCV` currently fits and calibrates an estimator for each fold (calibration is performed on the test part of the fold).
All the estimators fit at each fold are kept in a list.
At prediction time, every estimator makes a prediction and the average of the returned values is the final prediction.

The estimator produced by `CalibratedClassifierCV` is thus an ensemble and not a single estimator calibrated on the whole training set via CV.
When using cross validation the original `base_estimator` is not used to make the prediction.

Platt99, describes a cross validation procedure that fits an estimator on each fold and the predictions for the test fold are saved. 
Then the predictions from all the folds are concatenated in a single list, and calibration parameters for the `base_estimator` are determined using such list.

Cross validation should be only a mean to calibrate the `base_estimator` on the same data it has been fit, not to fit a different estimator.

The procedure described in Platt99 is what one would expect from a proper application of a cross validation procedure, as the cross validation only determines the parameters of the calibration and does not fit the estimator.
It is also more efficient, as is does not store the estimators for each fold and requires a single predict at prediction time.

Hints from issue discussion:
> Then the predictions from all the folds are concatenated in a single list, and calibration parameters for the base_estimator are determined using such list.

In other words, we should use `cross_val_predict` instead of a `cross_val_score`.

> The procedure described in Platt99 is what one would expect from a proper application of a cross validation procedure, as the cross validation only determines the parameters of the calibration and does not fit the estimator.
It is also more efficient, as is does not store the estimators for each fold and requires a single predict at prediction time.

IMO, using the ensemble will be more stable and powerful than using a single learner. I assume that it should attenuate changes linked to model parameters found for different splits.

ping @GaelVaroquaux @agramfort you should be better aware than me for commenting.
@aesuli I thing I understand what you asking. Before suggesting any change to the way we do things 2 questions:
- you suggest that CalibratedClassifierCV should not change the base_estimator . If so how do you end a pipeline with such an estimator? where do you get your pretrain model from? can I still do this with fit(X, y) from scratch?
- in libsvm https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/svm/src/libsvm/svm.cpp#L2099 I see a for loop over folds when I fit with probability=True which is what we tried to replicate offering more than platt scaling / sigmoid for calibration method. Are you suggesting that what we implement is not exactly what libsvm does? thanks
@glemaitre right! `cross_val_predict` can be the way to get CV predictions for the whole training set and then compute calibration from them.

@agramfort 
> * you suggest that CalibratedClassifierCV should not change the base_estimator . If so how do you end a pipeline with such an estimator? where do you get your pretrain model from? can I still do this with fit(X, y) from scratch?
I meant that the CalibratedClassifierCV should not alter the form of the base_estimator.
If the base estimator is a linear_svm it should fit a linear_svm on the whole training set, not fit an ensemble of linear_svm on folds.

The fit(X,y) of the CalibratedClassifierCV could be something like (rough pseudo code):

```
def fit(X,y):
  if not self.cv == 'prefit': #Platt99-style CV calibration
    self.base_estimator.fit(X,y)
    cv_predictions = cross_val_predict(self.base_estimator, X, y, cv=self.cv)
    self.calibration_params = self.estimate_calibration(cv_predictions)
  else: # prefit, assuming base_estimator is already fit on different data
    predictions = self.base_estimator.predict_proba(X)
    self.calibration_params = self.estimate_calibration(predictions)

def predict_proba(X):
    predictions = self.base_estimator.predict_proba(X)
    calibrated_predictions = self.apply_calibration_model(predictions, self.calibration_params)
    return calibrated_predictions
```

> * in libsvm https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/svm/src/libsvm/svm.cpp#L2099 I see a for loop over folds when I fit with probability=True which is what we tried to replicate offering more than platt scaling / sigmoid for calibration method. Are you suggesting that what we implement is not exactly what libsvm does? thanks

Yes the current CalibratedClassiferCV implementation differs from libsvm. 
svm.cpp performs CV only to get predictions that are all stored together in [`dec_values`](https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/svm/src/libsvm/svm.cpp#L2090)
`dev_values` is populated in various points of the code depending on the composition of the training set for a fold (see [here](https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/svm/src/libsvm/svm.cpp#L2139) and [here](https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/svm/src/libsvm/svm.cpp#L2167)).
At the end of each fold the model fitted for that fold is [destroyed](https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/svm/src/libsvm/svm.cpp#L2169).
The calibration parameters are computed on [all the predictions for the training set](https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/svm/src/libsvm/svm.cpp#L2176).
got it. We could expose this strategy with a parameter ensemble True or
False or something
I see also that it would not reduce fit time but increase it as you would
fit on the full data too.
The predict time would be faster though.

I would be curious to see if it improves performance too. I would imagine
that the cross_val_predict
strategy has more variance due to the absence of ensemble.

can you give it a try?

>

OK, I'll work on this in the next days.

Commit message: update docs

Patch to review:
diff --git a/doc/modules/calibration.rst b/doc/modules/calibration.rst
index cfc185c854edb..78edf290b6751 100644
--- a/doc/modules/calibration.rst
+++ b/doc/modules/calibration.rst
@@ -96,8 +96,8 @@ in [0, 1]. Denoting the output of the classifier for a given sample by :math:`f_
 the calibrator tries to predict :math:`p(y_i = 1 | f_i)`.
 
 The samples that are used to fit the calibrator should not be the same
-samples used to fit the classifier, as this would
-introduce bias. The classifier performance on its training data would be
+samples used to fit the classifier, as this would introduce bias.
+This is because performance of the classifier on its training data would be
 better than for novel data. Using the classifier output from training data
 to fit the calibrator would thus result in a biased calibrator that maps to
 probabilities closer to 0 and 1 than it should.
@@ -107,22 +107,30 @@ Usage
 
 The :class:`CalibratedClassifierCV` class is used to calibrate a classifier.
 
-:class:`CalibratedClassifierCV` uses a cross-validation approach to fit both
-the classifier and the regressor. The data is split into k
-`(train_set, test_set)` couples (as determined by `cv`). The classifier
-(`base_estimator`) is trained on the train set, and its predictions on the
-test set are used to fit a regressor. This ensures that the data used to fit
-the classifier is always disjoint from the data used to fit the calibrator.
-After fitting, we end up with k
-`(classifier, regressor)` couples where each regressor maps the output of
+:class:`CalibratedClassifierCV` uses a cross-validation approach to ensure
+unbiased data is always used to fit the calibrator. The data is split into k
+`(train_set, test_set)` couples (as determined by `cv`). When `ensemble=True`
+(default), the classifier (`base_estimator`) is trained on the train set, and
+its predictions on the test set are used to fit the calibrator (either a
+sigmoid or isotonic regressor). After fitting, we end up with an ensemble of
+k `(classifier, calibrator)` couples where each calibrator maps the output of
 its corresponding classifier into [0, 1]. Each couple is exposed in the
 `calibrated_classifiers_` attribute, where each entry is a calibrated
 classifier with a :term:`predict_proba` method that outputs calibrated
 probabilities. The output of :term:`predict_proba` for the main
 :class:`CalibratedClassifierCV` instance corresponds to the average of the
-predicted probabilities of the `k` estimators in the
-`calibrated_classifiers_` list. The output of :term:`predict` is the class
-that has the highest probability.
+predicted probabilities of the `k` estimators in the `calibrated_classifiers_`
+list. The output of :term:`predict` is the class that has the highest
+probability.
+
+When `ensemble=False`, cross-validation is used to obtain 'unbiased'
+predictions for all the data, by concatenating the testing subset predictions.
+These unbiased predictions are used to train the calibrator. The attribute
+`calibrated_classifiers_` consists of only one `(classifier, calibrator)`
+couple where the classifier is the `base_estimator` trained on all the data.
+In this case the output of :term:`predict_proba` for
+:class:`CalibratedClassifierCV` is the predicted probabilities obtained
+from the single `(classifier, calibrator)` couple.
 
 Alternatively an already fitted classifier can be calibrated by setting
 `cv="prefit"`. In this case, the data is not split and all of it is used to
diff --git a/sklearn/calibration.py b/sklearn/calibration.py
index abea91e7e53df..060882fb3c4c0 100644
--- a/sklearn/calibration.py
+++ b/sklearn/calibration.py
@@ -10,6 +10,7 @@
 import warnings
 from inspect import signature
 from contextlib import suppress
+from functools import partial
 
 from math import log
 import numpy as np
@@ -18,59 +19,41 @@
 from scipy.special import expit
 from scipy.special import xlogy
 from scipy.optimize import fmin_bfgs
-from .preprocessing import LabelEncoder
 
 from .base import (BaseEstimator, ClassifierMixin, RegressorMixin, clone,
                    MetaEstimatorMixin)
-from .preprocessing import label_binarize, LabelBinarizer
+from .preprocessing import label_binarize, LabelEncoder
 from .utils import check_array, indexable, column_or_1d
+from .utils.multiclass import check_classification_targets
 from .utils.validation import check_is_fitted, check_consistent_length
 from .utils.validation import _check_sample_weight
 from .pipeline import Pipeline
 from .isotonic import IsotonicRegression
 from .svm import LinearSVC
-from .model_selection import check_cv
+from .model_selection import check_cv, cross_val_predict
 from .utils.validation import _deprecate_positional_args
 
 
-def _fit_calibrated_classifer(estimator, X, y, train, test, supports_sw,
-                              method, classes, sample_weight=None):
-    """Fit calibrated classifier for a given dataset split.
-
-    Returns
-    -------
-    calibrated_classifier : estimator object
-        The calibrated estimator
-    """
-    if sample_weight is not None and supports_sw:
-        estimator.fit(X[train], y[train],
-                      sample_weight=sample_weight[train])
-    else:
-        estimator.fit(X[train], y[train])
-
-    calibrated_classifier = _CalibratedClassifier(estimator,
-                                                  method=method,
-                                                  classes=classes)
-    sw = None if sample_weight is None else sample_weight[test]
-    calibrated_classifier.fit(X[test], y[test], sample_weight=sw)
-    return calibrated_classifier
-
-
 class CalibratedClassifierCV(ClassifierMixin,
                              MetaEstimatorMixin,
                              BaseEstimator):
     """Probability calibration with isotonic regression or logistic regression.
 
     This class uses cross-validation to both estimate the parameters of a
-    classifier and subsequently calibrate a classifier. For each cv split it
-    fits a copy of the base estimator to the training folds, and calibrates it
-    using the testing fold. For prediction, predicted probabilities are
-    averaged across these individual calibrated classifiers.
-
-    Already fitted classifiers can be calibrated via the parameter cv="prefit".
-    In this case, no cross-validation is used and all provided data is used
-    for calibration. The user has to take care manually that data for model
-    fitting and calibration are disjoint.
+    classifier and subsequently calibrate a classifier. With default
+    `ensemble=True`, for each cv split it
+    fits a copy of the base estimator to the training subset, and calibrates it
+    using the testing subset. For prediction, predicted probabilities are
+    averaged across these individual calibrated classifiers. When
+    `ensemble=False`, cross-validation is used to obtain unbiased predictions,
+    from the testing subset, which are used for calibration. For prediction,
+    the base estimator, trained using all the data, is used. This is the method
+    implemented when `probabilities=True` for :mod:`sklearn.svm` estimators.
+
+    Already fitted classifiers can be calibrated via the parameter
+    `cv="prefit"`. In this case, no cross-validation is used and all provided
+    data is used for calibration. The user has to take care manually that data
+    for model fitting and calibration are disjoint.
 
     The calibration is based on the :term:`decision_function` method of the
     `base_estimator` if it exists, else on :term:`predict_proba`.
@@ -92,7 +75,7 @@ class CalibratedClassifierCV(ClassifierMixin,
         ``(<<1000)`` since it tends to overfit.
 
     cv : integer, cross-validation generator, iterable or "prefit", \
-            default=None
+         default=None
         Determines the cross-validation splitting strategy.
         Possible inputs for cv are:
 
@@ -121,7 +104,7 @@ class CalibratedClassifierCV(ClassifierMixin,
         ``-1`` means using all processors.
 
         Base estimator clones are fitted in parallel across cross-validation
-        iterations. Therefore parallelism happens only when cv != "prefit".
+        iterations. Therefore parallelism happens only when `cv` != "prefit".
 
         See :term:`Glossary <n_jobs>` for more details.
 
@@ -151,15 +134,43 @@ class CalibratedClassifierCV(ClassifierMixin,
 
         .. versionadded:: 0.24
 
+    ensemble : bool, default=True
+        Determines how the calibrator is fitted when `cv` is not `'prefit'`.
+        Ignored if `cv='prefit'`.
+
+        If `True`, the `base_estimator` is fitted using training data and
+        calibrated using testing data, for each `cv` fold. The final estimator
+        is an ensemble of `n_cv` fitted classifer and calibrator pairs, where
+        `n_cv` is the number of cross-validation folds. The output is the
+        average predicted probabilities of all pairs.
+
+        If `False`, `cv` is used to compute unbiased predictions (with the
+        test fold), which are concatenated and used to train the calibrator
+        (sigmoid or isotonic model). At prediction time, the classifier used
+        is the `base_estimator` trained on all the data.
+        Note this method is implemented when `probabilities=True` for
+        :mod:`sklearn.svm` estimators.
+
     Attributes
     ----------
     classes_ : array, shape (n_classes)
         The class labels.
 
-    calibrated_classifiers_ : list (len() equal to cv or 1 if cv == "prefit")
-        The list of calibrated classifiers, one for each cross-validation
-        split, which has been fitted on training folds and
-        calibrated on the testing fold.
+    calibrated_classifiers_ : list (len() equal to cv or 1 if `cv="prefit"` \
+                              or `ensemble=False`)
+        The list of classifier and calibrator pairs.
+
+        - When `cv="prefit"`, the fitted `base_estimator` and fitted
+          calibrator.
+        - When `cv` is not "prefit" and `ensemble=True`, `n_cv` fitted
+          `base_estimator` and calibrator pairs. `n_cv` is the number of
+          cross-validation folds.
+        - When `cv` is not "prefit" and `ensemble=False`, the `base_estimator`,
+          fitted on all the data, and fitted calibrator.
+
+    n_features_in_ : int
+        The number of features in `X`. If `cv='prefit'`, number of features
+        in the data used to fit `base_estimator`.
 
     Examples
     --------
@@ -218,16 +229,17 @@ class CalibratedClassifierCV(ClassifierMixin,
     @_deprecate_positional_args
     def __init__(self, base_estimator=None, *, method='sigmoid',
                  cv=None, n_jobs=None, pre_dispatch='2*n_jobs',
-                 verbose=0):
+                 verbose=0, ensemble=True):
         self.base_estimator = base_estimator
         self.method = method
         self.cv = cv
         self.n_jobs = n_jobs
         self.verbose = verbose
         self.pre_dispatch = pre_dispatch
+        self.ensemble = ensemble
 
     def fit(self, X, y, sample_weight=None):
-        """Fit the calibrated model
+        """Fit the calibrated model.
 
         Parameters
         ----------
@@ -245,9 +257,9 @@ def fit(self, X, y, sample_weight=None):
         self : object
             Returns an instance of self.
         """
+        check_classification_targets(y)
         X, y = indexable(X, y)
 
-        self.calibrated_classifiers_ = []
         if self.base_estimator is None:
             # we want all classifiers that don't expose a random_state
             # to be deterministic (and we don't want to expose this one).
@@ -255,8 +267,10 @@ def fit(self, X, y, sample_weight=None):
         else:
             base_estimator = self.base_estimator
 
+        self.calibrated_classifiers_ = []
         if self.cv == "prefit":
-            # Set `n_features_in_` attribute
+            # `classes_` and `n_features_in_` should be consistent with that
+            # of base_estimator
             if isinstance(self.base_estimator, Pipeline):
                 check_is_fitted(self.base_estimator[-1])
             else:
@@ -265,17 +279,35 @@ def fit(self, X, y, sample_weight=None):
                 self.n_features_in_ = base_estimator.n_features_in_
             self.classes_ = self.base_estimator.classes_
 
-            calibrated_classifier = _CalibratedClassifier(
-                base_estimator, method=self.method)
-            calibrated_classifier.fit(X, y, sample_weight)
+            pred_method = _get_prediction_method(base_estimator)
+            n_classes = len(self.classes_)
+            preds = _get_predictions(pred_method, X, n_classes)
+
+            calibrated_classifier = _fit_calibrator(
+                base_estimator, preds, y, self.classes_, self.method,
+                sample_weight
+            )
             self.calibrated_classifiers_.append(calibrated_classifier)
         else:
             X, y = self._validate_data(
                 X, y, accept_sparse=['csc', 'csr', 'coo'],
                 force_all_finite=False, allow_nd=True
             )
-            le = LabelBinarizer().fit(y)
-            self.classes_ = le.classes_
+            # Set `classes_` using all `y`
+            label_encoder_ = LabelEncoder().fit(y)
+            self.classes_ = label_encoder_.classes_
+            n_classes = len(self.classes_)
+
+            # sample_weight checks
+            fit_parameters = signature(base_estimator.fit).parameters
+            supports_sw = "sample_weight" in fit_parameters
+            if sample_weight is not None:
+                sample_weight = _check_sample_weight(sample_weight, X)
+                if not supports_sw:
+                    estimator_name = type(base_estimator).__name__
+                    warnings.warn("Since %s does not support sample_weights, "
+                                  "sample weights will only be used for the "
+                                  "calibration itself." % estimator_name)
 
             # Check that each cross-validation fold can have at least one
             # example per class
@@ -287,43 +319,47 @@ def fit(self, X, y, sample_weight=None):
                 n_folds = None
             if n_folds and np.any([np.sum(y == class_) < n_folds
                                    for class_ in self.classes_]):
-                raise ValueError(f"Requesting {n_folds}-fold cross-validation "
-                                 f"but provided less than {n_folds} examples "
-                                 "for at least one class.")
-
+                raise ValueError(f"Requesting {n_folds}-fold "
+                                 "cross-validation but provided less than "
+                                 f"{n_folds} examples for at least one class.")
             cv = check_cv(self.cv, y, classifier=True)
-            fit_parameters = signature(base_estimator.fit).parameters
-            supports_sw = "sample_weight" in fit_parameters
-
-            if sample_weight is not None:
-                sample_weight = _check_sample_weight(sample_weight, X)
-
-                if not supports_sw:
-                    estimator_name = type(base_estimator).__name__
-                    warnings.warn("Since %s does not support sample_weights, "
-                                  "sample weights will only be used for the "
-                                  "calibration itself." % estimator_name)
-
-            parallel = Parallel(n_jobs=self.n_jobs, verbose=self.verbose,
-                                pre_dispatch=self.pre_dispatch)
 
-            self.calibrated_classifiers_ = parallel(delayed(
-                _fit_calibrated_classifer)(clone(base_estimator),
-                                           X, y,
-                                           train=train, test=test,
-                                           method=self.method,
-                                           classes=self.classes_,
-                                           supports_sw=supports_sw,
-                                           sample_weight=sample_weight)
-                                                    for train, test
-                                                    in cv.split(X, y))
+            if self.ensemble:
+                parallel = Parallel(n_jobs=self.n_jobs, verbose=self.verbose,
+                                    pre_dispatch=self.pre_dispatch)
+
+                self.calibrated_classifiers_ = parallel(
+                    delayed(_get_pred_fit_calibrator)(
+                        clone(base_estimator), X, y, train=train, test=test,
+                        method=self.method, classes=self.classes_,
+                        supports_sw=supports_sw, sample_weight=sample_weight)
+                    for train, test in cv.split(X, y)
+                )
+            else:
+                this_estimator = clone(base_estimator)
+                method_name = _get_prediction_method(this_estimator).__name__
+                pred_method = partial(
+                    cross_val_predict, estimator=this_estimator, X=X, y=y,
+                    cv=cv, method=method_name
+                )
+                preds = _get_predictions(pred_method, X, n_classes)
+
+                if sample_weight is not None and supports_sw:
+                    this_estimator.fit(X, y, sample_weight)
+                else:
+                    this_estimator.fit(X, y)
+                calibrated_classifier = _fit_calibrator(
+                    this_estimator, preds, y, self.classes_, self.method,
+                    sample_weight
+                )
+                self.calibrated_classifiers_.append(calibrated_classifier)
 
         return self
 
     def predict_proba(self, X):
-        """Posterior probabilities of classification
+        """Calibrated probabilities of classification.
 
-        This function returns posterior probabilities of classification
+        This function returns calibrated probabilities of classification
         according to each class on an array of test vectors X.
 
         Parameters
@@ -377,145 +413,229 @@ def _more_tags(self):
         }
 
 
-class _CalibratedClassifier:
-    """Probability calibration with isotonic regression or sigmoid.
+def _get_pred_fit_calibrator(estimator, X, y, train, test, supports_sw,
+                             method, classes, sample_weight=None):
+    """Get predictions and fit calibrator for a given dataset split.
+
+    Parameters
+    ----------
+    estimator : estimator instance
+        Cloned base estimator.
+
+    X : array-like, shape (n_samples, n_features)
+        Sample data.
+
+    y : array-like, shape (n_samples,)
+        Targets.
 
-    It assumes that base_estimator has already been fit, and trains the
-    calibration on the input set of the fit function. Note that this class
-    should not be used as an estimator directly. Use CalibratedClassifierCV
-    with cv="prefit" instead.
+    train : ndarray, shape (n_train_indicies,)
+        Indices of the training subset.
+
+    test : ndarray, shape (n_test_indicies,)
+        Indices of the testing subset.
+
+    supports_sw : bool
+        Whether or not the `estimator` supports sample weights.
+
+    method : {'sigmoid', 'isotonic'}
+        Method to use for calibration.
+
+    classes : ndarray, shape (n_classes,)
+        The target classes.
+
+    sample_weight : array-like, default=None
+        Sample weights for `X`.
+
+    Returns
+    -------
+    calibrated_classifier : _CalibratedClassiferPipeline instance
+    """
+    if sample_weight is not None and supports_sw:
+        estimator.fit(X[train], y[train],
+                      sample_weight=sample_weight[train])
+    else:
+        estimator.fit(X[train], y[train])
+
+    n_classes = len(classes)
+    pred_method = _get_prediction_method(estimator)
+    preds = _get_predictions(pred_method, X[test], n_classes)
+
+    sw = None if sample_weight is None else sample_weight[test]
+    calibrated_classifier = _fit_calibrator(
+        estimator, preds, y[test], classes, method, sample_weight=sw
+    )
+    return calibrated_classifier
+
+
+def _get_prediction_method(clf):
+    """Return prediction method.
+
+    `decision_function` method of `clf` returned, if it
+    exists, otherwise `predict_proba` method returned.
 
     Parameters
     ----------
-    base_estimator : instance BaseEstimator
-        The classifier whose output decision function needs to be calibrated
-        to offer more accurate predict_proba outputs. No default value since
-        it has to be an already fitted estimator.
+    clf : Estimator instance
+        Fitted classifier to obtain the prediction method from.
 
-    method : 'sigmoid' | 'isotonic'
-        The method to use for calibration. Can be 'sigmoid' which
-        corresponds to Platt's method or 'isotonic' which is a
-        non-parametric approach based on isotonic regression.
+    Returns
+    -------
+    prediction_method : callable
+        The prediction method.
+    """
+    if hasattr(clf, 'decision_function'):
+        method = getattr(clf, 'decision_function')
+    elif hasattr(clf, 'predict_proba'):
+        method = getattr(clf, 'predict_proba')
+    else:
+        raise RuntimeError("'base_estimator' has no 'decision_function' or "
+                           "'predict_proba' method.")
+    return method
 
-    classes : array-like, shape (n_classes,), optional
-            Contains unique classes used to fit the base estimator.
-            if None, then classes is extracted from the given target values
-            in fit().
 
-    See also
-    --------
-    CalibratedClassifierCV
+def _get_predictions(pred_method, X, n_classes):
+    """Returns predictions for `X` and reshapes binary outputs to shape
+    (n_samples, 1).
 
-    References
+    Parameters
     ----------
-    .. [1] Obtaining calibrated probability estimates from decision trees
-           and naive Bayesian classifiers, B. Zadrozny & C. Elkan, ICML 2001
+    pred_method : callable
+        Prediction method.
 
-    .. [2] Transforming Classifier Scores into Accurate Multiclass
-           Probability Estimates, B. Zadrozny & C. Elkan, (KDD 2002)
+    X : array-like or None
+        Data used to obtain predictions.
 
-    .. [3] Probabilistic Outputs for Support Vector Machines and Comparisons to
-           Regularized Likelihood Methods, J. Platt, (1999)
+    n_classes : int
+        Number of classes present.
 
-    .. [4] Predicting Good Probabilities with Supervised Learning,
-           A. Niculescu-Mizil & R. Caruana, ICML 2005
+    Returns
+    -------
+    preds : array-like, shape (X.shape[0], len(clf.classes_))
+        The predictions. Note if there are 2 classes, array is of shape
+        (X.shape[0], 1).
     """
-    @_deprecate_positional_args
-    def __init__(self, base_estimator, *, method='sigmoid', classes=None):
-        self.base_estimator = base_estimator
-        self.method = method
-        self.classes = classes
+    preds = pred_method(X=X)
+    if hasattr(pred_method, '__name__'):
+        method = pred_method.__name__
+    else:
+        method = signature(pred_method).parameters['method'].default
 
-    def _preproc(self, X):
-        n_classes = len(self.classes_)
-        if hasattr(self.base_estimator, "decision_function"):
-            df = self.base_estimator.decision_function(X)
-            if df.ndim == 1:
-                df = df[:, np.newaxis]
-        elif hasattr(self.base_estimator, "predict_proba"):
-            df = self.base_estimator.predict_proba(X)
-            if n_classes == 2:
-                df = df[:, 1:]
-        else:
-            raise RuntimeError('classifier has no decision_function or '
-                               'predict_proba method.')
+    if method == 'decision_function':
+        if preds.ndim == 1:
+            preds = preds[:, np.newaxis]
+    elif method == 'predict_proba':
+        if n_classes == 2:
+            preds = preds[:, 1:]
+    else:
+        raise RuntimeError("'pred_method' needs to be one of "
+                           "'decision_function' or 'predict_proba'.")
+    return preds
 
-        idx_pos_class = self.label_encoder_.\
-            transform(self.base_estimator.classes_)
 
-        return df, idx_pos_class
+def _fit_calibrator(clf, preds, y, classes, method, sample_weight=None):
+    """Fit calibrator(s) and return a `_CalibratedClassiferPipeline`
+    instance.
 
-    def fit(self, X, y, sample_weight=None):
-        """Calibrate the fitted model
+    `n_classes` (i.e. `len(clf.classes_)`) calibrators are fitted.
+    However, if `n_classes` equals 2, one calibrator is fitted.
 
-        Parameters
-        ----------
-        X : array-like, shape (n_samples, n_features)
-            Training data.
+    Parameters
+    ----------
+    clf : estimator instance
+        Fitted classifier.
 
-        y : array-like, shape (n_samples,)
-            Target values.
+    preds : array-like, shape (n_samples, n_classes) or (n_samples, 1) when \
+            binary.
+        Predictions for calibrating the predictions.
 
-        sample_weight : array-like of shape (n_samples,), default=None
-            Sample weights. If None, then samples are equally weighted.
+    y : array-like, shape (n_samples,)
+        The targets.
 
-        Returns
-        -------
-        self : object
-            Returns an instance of self.
-        """
+    classes : ndarray, shape (n_classes,)
+        All the prediction classes.
 
-        self.label_encoder_ = LabelEncoder()
-        if self.classes is None:
-            self.label_encoder_.fit(y)
+    method : {'sigmoid', 'isotonic'}
+        The method to use for calibration.
+
+    sample_weight : ndarray, shape (n_samples,), default=None
+        Sample weights. If None, then samples are equally weighted.
+
+    Returns
+    -------
+    pipeline : _CalibratedClassiferPipeline instance
+    """
+    Y = label_binarize(y, classes=classes)
+    label_encoder = LabelEncoder().fit(classes)
+    pos_class_indices = label_encoder.transform(clf.classes_)
+    calibrators = []
+    for class_idx, this_pred in zip(pos_class_indices, preds.T):
+        if method == 'isotonic':
+            calibrator = IsotonicRegression(out_of_bounds='clip')
+        elif method == 'sigmoid':
+            calibrator = _SigmoidCalibration()
         else:
-            self.label_encoder_.fit(self.classes)
+            raise ValueError("'method' should be one of: 'sigmoid' or "
+                             f"'isotonic'. Got {method}.")
+        calibrator.fit(this_pred, Y[:, class_idx], sample_weight)
+        calibrators.append(calibrator)
 
-        self.classes_ = self.label_encoder_.classes_
-        Y = label_binarize(y, classes=self.classes_)
+    pipeline = _CalibratedClassiferPipeline(clf, calibrators, classes)
+    return pipeline
 
-        df, idx_pos_class = self._preproc(X)
-        self.calibrators_ = []
 
-        for k, this_df in zip(idx_pos_class, df.T):
-            if self.method == 'isotonic':
-                calibrator = IsotonicRegression(out_of_bounds='clip')
-            elif self.method == 'sigmoid':
-                calibrator = _SigmoidCalibration()
-            else:
-                raise ValueError('method should be "sigmoid" or '
-                                 '"isotonic". Got %s.' % self.method)
-            calibrator.fit(this_df, Y[:, k], sample_weight)
-            self.calibrators_.append(calibrator)
+class _CalibratedClassiferPipeline:
+    """Pipeline-like chaining a fitted classifier and its fitted calibrators.
 
-        return self
+    Parameters
+    ----------
+    clf : estimator instance
+        Fitted classifier.
+
+    calibrators : list of fitted estimator instances
+        List of fitted calibrators (either 'IsotonicRegression' or
+        '_SigmoidCalibration'). The number of calibrators equals the number of
+        classes. However, if there are 2 classes, the list contains only one
+        fitted calibrator.
+
+    classes : ndarray, shape (n_classes,)
+        All the prediction classes.
+    """
+    def __init__(self, clf, calibrators, classes):
+        self.clf = clf
+        self.calibrators = calibrators
+        self.classes = classes
 
     def predict_proba(self, X):
-        """Posterior probabilities of classification
+        """Calculate calibrated probabilities.
 
-        This function returns posterior probabilities of classification
-        according to each class on an array of test vectors X.
+        Calculates classification calibrated probabilities
+        for each class, in a one-vs-all manner, for `X`.
 
         Parameters
         ----------
-        X : array-like, shape (n_samples, n_features)
-            The samples.
+        X : ndarray, shape (n_samples, n_features)
+            The sample data.
 
         Returns
         -------
-        C : array, shape (n_samples, n_classes)
-            The predicted probas. Can be exact zeros.
+        proba : array, shape (n_samples, n_classes)
+            The predicted probabilities. Can be exact zeros.
         """
-        n_classes = len(self.classes_)
-        proba = np.zeros((X.shape[0], n_classes))
+        n_classes = len(self.classes)
+        pred_method = _get_prediction_method(self.clf)
+        preds = _get_predictions(pred_method, X, n_classes)
 
-        df, idx_pos_class = self._preproc(X)
+        label_encoder = LabelEncoder().fit(self.classes)
+        pos_class_indices = label_encoder.transform(self.clf.classes_)
 
-        for k, this_df, calibrator in \
-                zip(idx_pos_class, df.T, self.calibrators_):
+        proba = np.zeros((X.shape[0], n_classes))
+        for class_idx, this_pred, calibrator in \
+                zip(pos_class_indices, preds.T, self.calibrators):
             if n_classes == 2:
-                k += 1
-            proba[:, k] = calibrator.predict(this_df)
+                # When binary, `preds` consists only of predictions for
+                # clf.classes_[1] but `pos_class_indices` = 0
+                class_idx += 1
+            proba[:, class_idx] = calibrator.predict(this_pred)
 
         # Normalize the probabilities
         if n_classes == 2:
@@ -532,12 +652,12 @@ def predict_proba(self, X):
         return proba
 
 
-def _sigmoid_calibration(df, y, sample_weight=None):
+def _sigmoid_calibration(pred, y, sample_weight=None):
     """Probability Calibration with sigmoid method (Platt 2000)
 
     Parameters
     ----------
-    df : ndarray, shape (n_samples,)
+    pred : ndarray, shape (n_samples,)
         The decision function or predict proba for the samples.
 
     y : ndarray, shape (n_samples,)
@@ -558,10 +678,10 @@ def _sigmoid_calibration(df, y, sample_weight=None):
     ----------
     Platt, "Probabilistic Outputs for Support Vector Machines"
     """
-    df = column_or_1d(df)
+    pred = column_or_1d(pred)
     y = column_or_1d(y)
 
-    F = df  # F follows Platt's notations
+    F = pred  # F follows Platt's notations
 
     # Bayesian priors (see Platt end of section 2.2)
     prior0 = float(np.sum(y <= 0))
@@ -715,7 +835,7 @@ def calibration_curve(y_true, y_prob, *, normalize=False, n_bins=5,
     array([0. , 0.5, 1. ])
     >>> prob_pred
     array([0.2  , 0.525, 0.85 ])
-    """
+     """
     y_true = column_or_1d(y_true)
     y_prob = column_or_1d(y_prob)
     check_consistent_length(y_true, y_prob)
diff --git a/sklearn/tests/test_calibration.py b/sklearn/tests/test_calibration.py
index f4e6f6d62d938..475eca507e8f4 100644
--- a/sklearn/tests/test_calibration.py
+++ b/sklearn/tests/test_calibration.py
@@ -15,11 +15,12 @@
                                     assert_raises, ignore_warnings)
 from sklearn.exceptions import NotFittedError
 from sklearn.datasets import make_classification, make_blobs
-from sklearn.preprocessing import LabelBinarizer
-from sklearn.model_selection import KFold
+from sklearn.preprocessing import LabelEncoder
+from sklearn.model_selection import KFold, cross_val_predict
 from sklearn.naive_bayes import MultinomialNB
 from sklearn.ensemble import RandomForestClassifier, RandomForestRegressor
 from sklearn.svm import LinearSVC
+from sklearn.isotonic import IsotonicRegression
 from sklearn.feature_extraction import DictVectorizer
 from sklearn.pipeline import Pipeline
 from sklearn.impute import SimpleImputer
@@ -29,7 +30,8 @@
 from sklearn.calibration import calibration_curve
 
 
-def test_calibration():
+@pytest.mark.parametrize('ensemble', [True, False])
+def test_calibration(ensemble):
     """Test calibration objects with isotonic and sigmoid"""
     n_samples = 100
     X, y = make_classification(n_samples=2 * n_samples, n_features=6,
@@ -47,7 +49,7 @@ def test_calibration():
     clf = MultinomialNB().fit(X_train, y_train, sample_weight=sw_train)
     prob_pos_clf = clf.predict_proba(X_test)[:, 1]
 
-    pc_clf = CalibratedClassifierCV(clf, cv=y.size + 1)
+    pc_clf = CalibratedClassifierCV(clf, cv=y.size + 1, ensemble=ensemble)
     assert_raises(ValueError, pc_clf.fit, X, y)
 
     # Naive Bayes with calibration
@@ -55,7 +57,9 @@ def test_calibration():
                                       (sparse.csr_matrix(X_train),
                                        sparse.csr_matrix(X_test))]:
         for method in ['isotonic', 'sigmoid']:
-            pc_clf = CalibratedClassifierCV(clf, method=method, cv=2)
+            pc_clf = CalibratedClassifierCV(
+                clf, method=method, cv=2, ensemble=ensemble
+            )
             # Note that this fit overwrites the fit on the entire training
             # set
             pc_clf.fit(this_X_train, y_train, sample_weight=sw_train)
@@ -94,13 +98,17 @@ def test_calibration():
 
         # Check failure cases:
         # only "isotonic" and "sigmoid" should be accepted as methods
-        clf_invalid_method = CalibratedClassifierCV(clf, method="foo")
+        clf_invalid_method = CalibratedClassifierCV(
+            clf, method="foo", ensemble=ensemble
+        )
         assert_raises(ValueError, clf_invalid_method.fit, X_train, y_train)
 
         # base-estimators should provide either decision_function or
         # predict_proba (most regressors, for instance, should fail)
         clf_base_regressor = \
-            CalibratedClassifierCV(RandomForestRegressor(), method="sigmoid")
+            CalibratedClassifierCV(
+                RandomForestRegressor(), method="sigmoid", ensemble=ensemble
+            )
         assert_raises(RuntimeError, clf_base_regressor.fit, X_train, y_train)
 
 
@@ -110,25 +118,31 @@ def test_calibration_default_estimator():
     calib_clf = CalibratedClassifierCV(cv=2)
     calib_clf.fit(X, y)
 
-    base_est = calib_clf.calibrated_classifiers_[0].base_estimator
+    base_est = calib_clf.calibrated_classifiers_[0].clf
     assert isinstance(base_est, LinearSVC)
 
 
-def test_calibration_cv_splitter():
+@pytest.mark.parametrize('ensemble', [True, False])
+def test_calibration_cv_splitter(ensemble):
     # Check when `cv` is a CV splitter
     X, y = make_classification(n_samples=100, n_features=6, random_state=42)
 
     splits = 5
     kfold = KFold(n_splits=splits)
-    calib_clf = CalibratedClassifierCV(cv=kfold)
+    calib_clf = CalibratedClassifierCV(cv=kfold, ensemble=ensemble)
     assert isinstance(calib_clf.cv, KFold)
     assert calib_clf.cv.n_splits == splits
 
     calib_clf.fit(X, y)
-    assert len(calib_clf.calibrated_classifiers_) == splits
+    if ensemble:
+        assert len(calib_clf.calibrated_classifiers_) == splits
+    else:
+        assert len(calib_clf.calibrated_classifiers_) == 1
 
 
-def test_sample_weight():
+@pytest.mark.parametrize('method', ['sigmoid', 'isotonic'])
+@pytest.mark.parametrize('ensemble', [True, False])
+def test_sample_weight(method, ensemble):
     n_samples = 100
     X, y = make_classification(n_samples=2 * n_samples, n_features=6,
                                random_state=42)
@@ -138,48 +152,52 @@ def test_sample_weight():
         X[:n_samples], y[:n_samples], sample_weight[:n_samples]
     X_test = X[n_samples:]
 
-    for method in ['sigmoid', 'isotonic']:
-        base_estimator = LinearSVC(random_state=42)
-        calibrated_clf = CalibratedClassifierCV(base_estimator, method=method)
-        calibrated_clf.fit(X_train, y_train, sample_weight=sw_train)
-        probs_with_sw = calibrated_clf.predict_proba(X_test)
+    base_estimator = LinearSVC(random_state=42)
+    calibrated_clf = CalibratedClassifierCV(
+        base_estimator, method=method, ensemble=ensemble
+    )
+    calibrated_clf.fit(X_train, y_train, sample_weight=sw_train)
+    probs_with_sw = calibrated_clf.predict_proba(X_test)
 
-        # As the weights are used for the calibration, they should still yield
-        # a different predictions
-        calibrated_clf.fit(X_train, y_train)
-        probs_without_sw = calibrated_clf.predict_proba(X_test)
+    # As the weights are used for the calibration, they should still yield
+    # different predictions
+    calibrated_clf.fit(X_train, y_train)
+    probs_without_sw = calibrated_clf.predict_proba(X_test)
 
-        diff = np.linalg.norm(probs_with_sw - probs_without_sw)
-        assert diff > 0.1
+    diff = np.linalg.norm(probs_with_sw - probs_without_sw)
+    assert diff > 0.1
 
 
-@pytest.mark.parametrize("method", ['sigmoid', 'isotonic'])
-def test_parallel_execution(method):
+@pytest.mark.parametrize('method', ['sigmoid', 'isotonic'])
+@pytest.mark.parametrize('ensemble', [True, False])
+def test_parallel_execution(method, ensemble):
     """Test parallel calibration"""
     X, y = make_classification(random_state=42)
     X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)
 
     base_estimator = LinearSVC(random_state=42)
 
-    cal_clf_parallel = CalibratedClassifierCV(base_estimator,
-                                              method=method, n_jobs=2)
+    cal_clf_parallel = CalibratedClassifierCV(
+        base_estimator, method=method, n_jobs=2, ensemble=ensemble
+    )
     cal_clf_parallel.fit(X_train, y_train)
     probs_parallel = cal_clf_parallel.predict_proba(X_test)
 
-    cal_clf_sequential = CalibratedClassifierCV(base_estimator,
-                                                method=method,
-                                                n_jobs=1)
+    cal_clf_sequential = CalibratedClassifierCV(
+        base_estimator, method=method, n_jobs=1, ensemble=ensemble
+    )
     cal_clf_sequential.fit(X_train, y_train)
     probs_sequential = cal_clf_sequential.predict_proba(X_test)
 
     assert_allclose(probs_parallel, probs_sequential)
 
 
-def test_calibration_multiclass():
-    """Test calibration for multiclass """
-    # test multi-class setting with classifier that implements
-    # only decision function
-    clf = LinearSVC()
+@pytest.mark.parametrize('method', ['sigmoid', 'isotonic'])
+@pytest.mark.parametrize('ensemble', [True, False])
+def test_calibration_multiclass(method, ensemble):
+    """Test calibration for multiclass with classifier that implements
+    only decision function."""
+    clf = LinearSVC(random_state=7)
     X, y_idx = make_blobs(n_samples=100, n_features=2, random_state=42,
                           centers=3, cluster_std=3.0)
 
@@ -192,27 +210,30 @@ def test_calibration_multiclass():
     X_test, y_test = X[1::2], y[1::2]
 
     clf.fit(X_train, y_train)
-    for method in ['isotonic', 'sigmoid']:
-        cal_clf = CalibratedClassifierCV(clf, method=method, cv=2)
-        cal_clf.fit(X_train, y_train)
-        probas = cal_clf.predict_proba(X_test)
-        assert_array_almost_equal(np.sum(probas, axis=1), np.ones(len(X_test)))
-
-        # Check that log-loss of calibrated classifier is smaller than
-        # log-loss of naively turned OvR decision function to probabilities
-        # via softmax
-        def softmax(y_pred):
-            e = np.exp(-y_pred)
-            return e / e.sum(axis=1).reshape(-1, 1)
-
-        uncalibrated_log_loss = \
-            log_loss(y_test, softmax(clf.decision_function(X_test)))
-        calibrated_log_loss = log_loss(y_test, probas)
-        assert uncalibrated_log_loss >= calibrated_log_loss
+
+    cal_clf = CalibratedClassifierCV(
+        clf, method=method, cv=2, ensemble=ensemble
+    )
+    cal_clf.fit(X_train, y_train)
+    probas = cal_clf.predict_proba(X_test)
+    # Check probabilities sum to 1
+    assert_array_almost_equal(np.sum(probas, axis=1), np.ones(len(X_test)))
+
+    # Check that log-loss of calibrated classifier is smaller than
+    # log-loss obtained by naively turning OvR decision function to
+    # probabilities via softmax
+    def softmax(y_pred):
+        e = np.exp(-y_pred)
+        return e / e.sum(axis=1).reshape(-1, 1)
+
+    uncalibrated_log_loss = \
+        log_loss(y_test, softmax(clf.decision_function(X_test)))
+    calibrated_log_loss = log_loss(y_test, probas)
+    assert uncalibrated_log_loss >= calibrated_log_loss
 
     # Test that calibration of a multiclass classifier decreases log-loss
     # for RandomForestClassifier
-    X, y = make_blobs(n_samples=100, n_features=2, random_state=42,
+    X, y = make_blobs(n_samples=1500, n_features=2, random_state=42,
                       cluster_std=3.0)
     X_train, y_train = X[::2], y[::2]
     X_test, y_test = X[1::2], y[1::2]
@@ -222,12 +243,13 @@ def softmax(y_pred):
     clf_probs = clf.predict_proba(X_test)
     loss = log_loss(y_test, clf_probs)
 
-    for method in ['isotonic', 'sigmoid']:
-        cal_clf = CalibratedClassifierCV(clf, method=method, cv=3)
-        cal_clf.fit(X_train, y_train)
-        cal_clf_probs = cal_clf.predict_proba(X_test)
-        cal_loss = log_loss(y_test, cal_clf_probs)
-        assert loss > cal_loss
+    cal_clf = CalibratedClassifierCV(
+        clf, method=method, cv=3, ensemble=ensemble
+    )
+    cal_clf.fit(X_train, y_train)
+    cal_clf_probs = cal_clf.predict_proba(X_test)
+    cal_loss = log_loss(y_test, cal_clf_probs)
+    assert loss > cal_loss
 
 
 def test_calibration_prefit():
@@ -276,6 +298,33 @@ def test_calibration_prefit():
                         brier_score_loss(y_test, prob_pos_pc_clf))
 
 
+@pytest.mark.parametrize('method', ['sigmoid', 'isotonic'])
+def test_calibration_ensemble_false(method):
+    """Test that `ensemble=False` is the same as using predictions from
+    `cross_val_predict` to train calibrator."""
+    X, y = make_classification(n_samples=100, n_features=6, random_state=7)
+    clf = LinearSVC(random_state=7)
+
+    cal_clf = CalibratedClassifierCV(clf, method=method, cv=3, ensemble=False)
+    cal_clf.fit(X, y)
+    cal_probas = cal_clf.predict_proba(X[:10, :])
+
+    # Get probas manually
+    unbiased_preds = cross_val_predict(
+        clf, X, y, cv=3, method='decision_function'
+    )
+    if method == 'isotonic':
+        calibrator = IsotonicRegression(out_of_bounds='clip')
+    else:
+        calibrator = _SigmoidCalibration()
+    calibrator.fit(unbiased_preds, y)
+    # Use `clf` fit on all data
+    clf.fit(X, y)
+    clf_df = clf.decision_function(X[:10, :])
+    manual_probas = calibrator.predict(clf_df)
+    assert_allclose(cal_probas[:, 1], manual_probas)
+
+
 def test_sigmoid_calibration():
     """Test calibration values with Platt sigmoid model"""
     exF = np.array([5, -4, 1.0])
@@ -329,7 +378,8 @@ def test_calibration_curve():
                   strategy='percentile')
 
 
-def test_calibration_nan_imputer():
+@pytest.mark.parametrize('ensemble', [True, False])
+def test_calibration_nan_imputer(ensemble):
     """Test that calibration can accept nan"""
     X, y = make_classification(n_samples=10, n_features=2,
                                n_informative=2, n_redundant=0,
@@ -338,42 +388,56 @@ def test_calibration_nan_imputer():
     clf = Pipeline(
         [('imputer', SimpleImputer()),
          ('rf', RandomForestClassifier(n_estimators=1))])
-    clf_c = CalibratedClassifierCV(clf, cv=2, method='isotonic')
+    clf_c = CalibratedClassifierCV(
+        clf, cv=2, method='isotonic', ensemble=ensemble
+    )
     clf_c.fit(X, y)
     clf_c.predict(X)
 
 
-def test_calibration_prob_sum():
+@pytest.mark.parametrize('ensemble', [True, False])
+def test_calibration_prob_sum(ensemble):
     # Test that sum of probabilities is 1. A non-regression test for
     # issue #7796
     num_classes = 2
     X, y = make_classification(n_samples=10, n_features=5,
                                n_classes=num_classes)
-    clf = LinearSVC(C=1.0)
-    clf_prob = CalibratedClassifierCV(clf, method="sigmoid", cv=LeaveOneOut())
+    clf = LinearSVC(C=1.0, random_state=7)
+    clf_prob = CalibratedClassifierCV(
+        clf, method="sigmoid", cv=LeaveOneOut(), ensemble=ensemble
+    )
     clf_prob.fit(X, y)
 
     probs = clf_prob.predict_proba(X)
     assert_array_almost_equal(probs.sum(axis=1), np.ones(probs.shape[0]))
 
 
-def test_calibration_less_classes():
+@pytest.mark.parametrize('ensemble', [True, False])
+def test_calibration_less_classes(ensemble):
     # Test to check calibration works fine when train set in a test-train
     # split does not contain all classes
     # Since this test uses LOO, at each iteration train set will not contain a
     # class label
     X = np.random.randn(10, 5)
     y = np.arange(10)
-    clf = LinearSVC(C=1.0)
-    cal_clf = CalibratedClassifierCV(clf, method="sigmoid", cv=LeaveOneOut())
+    clf = LinearSVC(C=1.0, random_state=7)
+    cal_clf = CalibratedClassifierCV(
+        clf, method="sigmoid", cv=LeaveOneOut(), ensemble=ensemble
+    )
     cal_clf.fit(X, y)
 
     for i, calibrated_classifier in \
             enumerate(cal_clf.calibrated_classifiers_):
         proba = calibrated_classifier.predict_proba(X)
-        assert_array_equal(proba[:, i], np.zeros(len(y)))
-        assert np.all(np.hstack([proba[:, :i],
-                                 proba[:, i + 1:]]))
+        if ensemble:
+            # Check 'missing' class has proba=0
+            assert_array_equal(proba[:, i], np.zeros(len(y)))
+            # Check for all other classes proba!=0
+            assert np.all(np.hstack([proba[:, :i],
+                                     proba[:, i + 1:]]))
+        else:
+            # Check `proba` all 1/n_classes
+            assert np.allclose(proba, 1 / proba.shape[0])
 
 
 @ignore_warnings(category=FutureWarning)
@@ -452,6 +516,6 @@ def test_calibration_attributes(clf, cv):
         assert_array_equal(calib_clf.classes_, clf.classes_)
         assert calib_clf.n_features_in_ == clf.n_features_in_
     else:
-        classes = LabelBinarizer().fit(y).classes_
+        classes = LabelEncoder().fit(y).classes_
         assert_array_equal(calib_clf.classes_, classes)
         assert calib_clf.n_features_in_ == X.shape[1]


Code review comments:
diff_hunk: @@ -377,145 +413,229 @@ def _more_tags(self):
         }
 
 
-class _CalibratedClassifier:
-    """Probability calibration with isotonic regression or sigmoid.
+def _get_pred_fit_calibrator(estimator, X, y, train, test, supports_sw,
+                             method, classes, sample_weight=None):
+    """Get predictions and fit calibrator for a given dataset split.
+
+    Parameters
+    ----------
+    estimator : estimator instance
+        Cloned base estimator.
+
+    X : array-like, shape (n_samples, n_features)
+        Sample data.
+
+    y : array-like, shape (n_samples,)
+        Targets.
 
-    It assumes that base_estimator has already been fit, and trains the
-    calibration on the input set of the fit function. Note that this class
-    should not be used as an estimator directly. Use CalibratedClassifierCV
-    with cv="prefit" instead.
+    train : ndarray, shape (n_train_indicies,)
+        Indices of the training subset.
+
+    test : ndarray, shape (n_test_indicies,)
+        Indices of the testing subset.
+
+    supports_sw : bool
+        Whether or not the `estimator` supports sample weights.
+
+    method : {'sigmoid', 'isotonic'}
+        Method to use for calibration.
+
+    classes : ndarray, shape (n_classes,)
+        The target classes.
+
+    sample_weight : array-like, default=None
+        Sample weights for `X`.
+
+    Returns
+    -------
+    calibrated_classifier : _CalibratedClassiferPipeline instance
+    """
+    if sample_weight is not None and supports_sw:
+        estimator.fit(X[train], y[train],
+                      sample_weight=sample_weight[train])
+    else:
+        estimator.fit(X[train], y[train])
+
+    n_classes = len(classes)
+    pred_method = _get_prediction_method(estimator)
+    preds = _get_predictions(pred_method, X[test], n_classes)
+
+    sw = None if sample_weight is None else sample_weight[test]
+    calibrated_classifier = _fit_calibrator(
+        estimator, preds, y[test], classes, method, sample_weight=sw
+    )
+    return calibrated_classifier
+
+
+def _get_prediction_method(clf):
+    """Return prediction method.
+
+    `decision_function` method of `clf` returned, if it
+    exists, otherwise `predict_proba` method returned.
 
     Parameters
     ----------
-    base_estimator : instance BaseEstimator
-        The classifier whose output decision function needs to be calibrated
-        to offer more accurate predict_proba outputs. No default value since
-        it has to be an already fitted estimator.
+    clf : Estimator instance
+        Fitted classifier to obtain the prediction method from.
 
-    method : 'sigmoid' | 'isotonic'
-        The method to use for calibration. Can be 'sigmoid' which
-        corresponds to Platt's method or 'isotonic' which is a
-        non-parametric approach based on isotonic regression.
+    Returns
+    -------
+    prediction_method : callable
+        The prediction method.
+    """
+    if hasattr(clf, 'decision_function'):
+        method = getattr(clf, 'decision_function')
+    elif hasattr(clf, 'predict_proba'):
+        method = getattr(clf, 'predict_proba')
+    else:
+        raise RuntimeError("'base_estimator' has no 'decision_function' or "
+                           "'predict_proba' method.")
+    return method
 
-    classes : array-like, shape (n_classes,), optional
-            Contains unique classes used to fit the base estimator.
-            if None, then classes is extracted from the given target values
-            in fit().
 
-    See also
-    --------
-    CalibratedClassifierCV
+def _get_predictions(pred_method, X, n_classes):
+    """Returns predictions for `X` and reshapes binary outputs to shape
+    (n_samples, 1).
 
-    References
+    Parameters
     ----------
-    .. [1] Obtaining calibrated probability estimates from decision trees
-           and naive Bayesian classifiers, B. Zadrozny & C. Elkan, ICML 2001
+    pred_method : callable
+        Prediction method.
 
-    .. [2] Transforming Classifier Scores into Accurate Multiclass
-           Probability Estimates, B. Zadrozny & C. Elkan, (KDD 2002)
+    X : array-like or None
+        Data used to obtain predictions.
 
-    .. [3] Probabilistic Outputs for Support Vector Machines and Comparisons to
-           Regularized Likelihood Methods, J. Platt, (1999)
+    n_classes : int
+        Number of classes present.
 
-    .. [4] Predicting Good Probabilities with Supervised Learning,
-           A. Niculescu-Mizil & R. Caruana, ICML 2005
+    Returns
+    -------
+    preds : array-like, shape (X.shape[0], len(clf.classes_))
+        The predictions. Note if there are 2 classes, array is of shape
+        (X.shape[0], 1).
     """
-    @_deprecate_positional_args
-    def __init__(self, base_estimator, *, method='sigmoid', classes=None):
-        self.base_estimator = base_estimator
-        self.method = method
-        self.classes = classes
+    preds = pred_method(X=X)
+    if hasattr(pred_method, '__name__'):
+        method = pred_method.__name__
path: sklearn/calibration.py
text: @user1:
call this name or method_name?

diff_hunk: @@ -377,145 +413,229 @@ def _more_tags(self):
         }
 
 
-class _CalibratedClassifier:
-    """Probability calibration with isotonic regression or sigmoid.
+def _get_pred_fit_calibrator(estimator, X, y, train, test, supports_sw,
+                             method, classes, sample_weight=None):
+    """Get predictions and fit calibrator for a given dataset split.
+
+    Parameters
+    ----------
+    estimator : estimator instance
+        Cloned base estimator.
+
+    X : array-like, shape (n_samples, n_features)
+        Sample data.
+
+    y : array-like, shape (n_samples,)
+        Targets.
 
-    It assumes that base_estimator has already been fit, and trains the
-    calibration on the input set of the fit function. Note that this class
-    should not be used as an estimator directly. Use CalibratedClassifierCV
-    with cv="prefit" instead.
+    train : ndarray, shape (n_train_indicies,)
+        Indices of the training subset.
+
+    test : ndarray, shape (n_test_indicies,)
+        Indices of the testing subset.
+
+    supports_sw : bool
+        Whether or not the `estimator` supports sample weights.
+
+    method : {'sigmoid', 'isotonic'}
+        Method to use for calibration.
+
+    classes : ndarray, shape (n_classes,)
+        The target classes.
+
+    sample_weight : array-like, default=None
+        Sample weights for `X`.
+
+    Returns
+    -------
+    calibrated_classifier : _CalibratedClassiferPipeline instance
+    """
+    if sample_weight is not None and supports_sw:
+        estimator.fit(X[train], y[train],
+                      sample_weight=sample_weight[train])
+    else:
+        estimator.fit(X[train], y[train])
+
+    n_classes = len(classes)
+    pred_method = _get_prediction_method(estimator)
+    preds = _get_predictions(pred_method, X[test], n_classes)
+
+    sw = None if sample_weight is None else sample_weight[test]
+    calibrated_classifier = _fit_calibrator(
+        estimator, preds, y[test], classes, method, sample_weight=sw
+    )
+    return calibrated_classifier
+
+
+def _get_prediction_method(clf):
+    """Return prediction method.
+
+    `decision_function` method of `clf` returned, if it
+    exists, otherwise `predict_proba` method returned.
 
     Parameters
     ----------
-    base_estimator : instance BaseEstimator
-        The classifier whose output decision function needs to be calibrated
-        to offer more accurate predict_proba outputs. No default value since
-        it has to be an already fitted estimator.
+    clf : Estimator instance
+        Fitted classifier to obtain the prediction method from.
 
-    method : 'sigmoid' | 'isotonic'
-        The method to use for calibration. Can be 'sigmoid' which
-        corresponds to Platt's method or 'isotonic' which is a
-        non-parametric approach based on isotonic regression.
+    Returns
+    -------
+    prediction_method : callable
+        The prediction method.
+    """
+    if hasattr(clf, 'decision_function'):
+        method = getattr(clf, 'decision_function')
+    elif hasattr(clf, 'predict_proba'):
+        method = getattr(clf, 'predict_proba')
+    else:
+        raise RuntimeError("'base_estimator' has no 'decision_function' or "
+                           "'predict_proba' method.")
+    return method
 
-    classes : array-like, shape (n_classes,), optional
-            Contains unique classes used to fit the base estimator.
-            if None, then classes is extracted from the given target values
-            in fit().
 
-    See also
-    --------
-    CalibratedClassifierCV
+def _get_predictions(pred_method, X, n_classes):
+    """Returns predictions for `X` and reshapes binary outputs to shape
+    (n_samples, 1).
 
-    References
+    Parameters
     ----------
-    .. [1] Obtaining calibrated probability estimates from decision trees
-           and naive Bayesian classifiers, B. Zadrozny & C. Elkan, ICML 2001
+    pred_method : callable
+        Prediction method.
 
-    .. [2] Transforming Classifier Scores into Accurate Multiclass
-           Probability Estimates, B. Zadrozny & C. Elkan, (KDD 2002)
+    X : array-like or None
+        Data used to obtain predictions.
 
-    .. [3] Probabilistic Outputs for Support Vector Machines and Comparisons to
-           Regularized Likelihood Methods, J. Platt, (1999)
+    n_classes : int
+        Number of classes present.
 
-    .. [4] Predicting Good Probabilities with Supervised Learning,
-           A. Niculescu-Mizil & R. Caruana, ICML 2005
+    Returns
+    -------
+    preds : array-like, shape (X.shape[0], len(clf.classes_))
+        The predictions. Note if there are 2 classes, array is of shape
+        (X.shape[0], 1).
     """
-    @_deprecate_positional_args
-    def __init__(self, base_estimator, *, method='sigmoid', classes=None):
-        self.base_estimator = base_estimator
-        self.method = method
-        self.classes = classes
+    preds = pred_method(X=X)
+    if hasattr(pred_method, '__name__'):
+        method = pred_method.__name__
+    else:
+        method = signature(pred_method).parameters['method'].default
 
-    def _preproc(self, X):
-        n_classes = len(self.classes_)
-        if hasattr(self.base_estimator, "decision_function"):
-            df = self.base_estimator.decision_function(X)
-            if df.ndim == 1:
-                df = df[:, np.newaxis]
-        elif hasattr(self.base_estimator, "predict_proba"):
-            df = self.base_estimator.predict_proba(X)
-            if n_classes == 2:
-                df = df[:, 1:]
-        else:
-            raise RuntimeError('classifier has no decision_function or '
-                               'predict_proba method.')
+    if method == 'decision_function':
+        if preds.ndim == 1:
+            preds = preds[:, np.newaxis]
+    elif method == 'predict_proba':
+        if n_classes == 2:
+            preds = preds[:, 1:]
+    else:
+        raise RuntimeError("'pred_method' needs to be one of "
+                           "'decision_function' or 'predict_proba'.")
path: sklearn/calibration.py
text: @user1:
I guess this will never be hit by the tests since it's done in _get_prediction_method. Maybe just do

```py
        else:  # pragma: no cover
            # this branch should be unreachable.
            raise ValueError
```

diff_hunk: @@ -110,25 +118,31 @@ def test_calibration_default_estimator():
     calib_clf = CalibratedClassifierCV(cv=2)
     calib_clf.fit(X, y)
 
-    base_est = calib_clf.calibrated_classifiers_[0].base_estimator
+    base_est = calib_clf.calibrated_classifiers_[0].clf
path: sklearn/tests/test_calibration.py
text: @user1:
I prefer  `clf` as well but we should try to be backward compatible here

diff_hunk: @@ -287,43 +319,47 @@ def fit(self, X, y, sample_weight=None):
                 n_folds = None
             if n_folds and np.any([np.sum(y == class_) < n_folds
                                    for class_ in self.classes_]):
-                raise ValueError(f"Requesting {n_folds}-fold cross-validation "
-                                 f"but provided less than {n_folds} examples "
-                                 "for at least one class.")
-
+                raise ValueError(f"Requesting {n_folds}-fold "
+                                 "cross-validation but provided less than "
+                                 f"{n_folds} examples for at least one class.")
             cv = check_cv(self.cv, y, classifier=True)
-            fit_parameters = signature(base_estimator.fit).parameters
-            supports_sw = "sample_weight" in fit_parameters
-
-            if sample_weight is not None:
-                sample_weight = _check_sample_weight(sample_weight, X)
-
-                if not supports_sw:
-                    estimator_name = type(base_estimator).__name__
-                    warnings.warn("Since %s does not support sample_weights, "
-                                  "sample weights will only be used for the "
-                                  "calibration itself." % estimator_name)
-
-            parallel = Parallel(n_jobs=self.n_jobs, verbose=self.verbose,
-                                pre_dispatch=self.pre_dispatch)
 
-            self.calibrated_classifiers_ = parallel(delayed(
-                _fit_calibrated_classifer)(clone(base_estimator),
-                                           X, y,
-                                           train=train, test=test,
-                                           method=self.method,
-                                           classes=self.classes_,
-                                           supports_sw=supports_sw,
-                                           sample_weight=sample_weight)
-                                                    for train, test
-                                                    in cv.split(X, y))
+            if self.ensemble:
+                parallel = Parallel(n_jobs=self.n_jobs, verbose=self.verbose,
+                                    pre_dispatch=self.pre_dispatch)
+
+                self.calibrated_classifiers_ = parallel(
+                    delayed(_get_pred_fit_calibrator)(
+                        clone(base_estimator), X, y, train=train, test=test,
+                        method=self.method, classes=self.classes_,
+                        supports_sw=supports_sw, sample_weight=sample_weight)
+                    for train, test in cv.split(X, y)
+                )
+            else:
+                this_estimator = clone(base_estimator)
+                method_name = _get_prediction_method(this_estimator).__name__
+                pred_method = partial(
+                    cross_val_predict, estimator=this_estimator, X=X, y=y,
+                    cv=cv, method=method_name
path: sklearn/calibration.py
text: @user1:
looks like we should be passing `n_jobs`here now

@user1:
also predispatch and verbose, though I think we should remove them (https://github.com/scikit-learn/scikit-learn/pull/18030)

@author:
Will add all 3 here and have the removal dealt with in #18030 ?

diff_hunk: @@ -276,6 +298,33 @@ def test_calibration_prefit():
                         brier_score_loss(y_test, prob_pos_pc_clf))
 
 
+@pytest.mark.parametrize('method', ['sigmoid', 'isotonic'])
+def test_calibration_ensemble_false(method):
+    """Test that `ensemble=False` is the same as using predictions from
+    `cross_val_predict` to train calibrator."""
+    X, y = make_classification(n_samples=100, n_features=6, random_state=7)
+    clf = LinearSVC(random_state=7)
+
+    cal_clf = CalibratedClassifierCV(clf, method=method, cv=3, ensemble=False)
+    cal_clf.fit(X, y)
+    cal_probas = cal_clf.predict_proba(X[:10, :])
path: sklearn/tests/test_calibration.py
text: @user1:
why limit to 10?

@author:
Good point!

diff_hunk: @@ -377,145 +413,229 @@ def _more_tags(self):
         }
 
 
-class _CalibratedClassifier:
-    """Probability calibration with isotonic regression or sigmoid.
+def _get_pred_fit_calibrator(estimator, X, y, train, test, supports_sw,
+                             method, classes, sample_weight=None):
+    """Get predictions and fit calibrator for a given dataset split.
+
+    Parameters
+    ----------
+    estimator : estimator instance
+        Cloned base estimator.
+
+    X : array-like, shape (n_samples, n_features)
+        Sample data.
+
+    y : array-like, shape (n_samples,)
+        Targets.
 
-    It assumes that base_estimator has already been fit, and trains the
-    calibration on the input set of the fit function. Note that this class
-    should not be used as an estimator directly. Use CalibratedClassifierCV
-    with cv="prefit" instead.
+    train : ndarray, shape (n_train_indicies,)
+        Indices of the training subset.
+
+    test : ndarray, shape (n_test_indicies,)
+        Indices of the testing subset.
+
+    supports_sw : bool
+        Whether or not the `estimator` supports sample weights.
+
+    method : {'sigmoid', 'isotonic'}
+        Method to use for calibration.
+
+    classes : ndarray, shape (n_classes,)
+        The target classes.
+
+    sample_weight : array-like, default=None
+        Sample weights for `X`.
+
+    Returns
+    -------
+    calibrated_classifier : _CalibratedClassiferPipeline instance
+    """
+    if sample_weight is not None and supports_sw:
+        estimator.fit(X[train], y[train],
+                      sample_weight=sample_weight[train])
+    else:
+        estimator.fit(X[train], y[train])
+
+    n_classes = len(classes)
+    pred_method = _get_prediction_method(estimator)
+    preds = _get_predictions(pred_method, X[test], n_classes)
+
+    sw = None if sample_weight is None else sample_weight[test]
+    calibrated_classifier = _fit_calibrator(
+        estimator, preds, y[test], classes, method, sample_weight=sw
+    )
+    return calibrated_classifier
+
+
+def _get_prediction_method(clf):
+    """Return prediction method.
+
+    `decision_function` method of `clf` returned, if it
+    exists, otherwise `predict_proba` method returned.
 
     Parameters
     ----------
-    base_estimator : instance BaseEstimator
-        The classifier whose output decision function needs to be calibrated
-        to offer more accurate predict_proba outputs. No default value since
-        it has to be an already fitted estimator.
+    clf : Estimator instance
+        Fitted classifier to obtain the prediction method from.
 
-    method : 'sigmoid' | 'isotonic'
-        The method to use for calibration. Can be 'sigmoid' which
-        corresponds to Platt's method or 'isotonic' which is a
-        non-parametric approach based on isotonic regression.
+    Returns
+    -------
+    prediction_method : callable
+        The prediction method.
+    """
+    if hasattr(clf, 'decision_function'):
+        method = getattr(clf, 'decision_function')
+    elif hasattr(clf, 'predict_proba'):
+        method = getattr(clf, 'predict_proba')
+    else:
+        raise RuntimeError("'base_estimator' has no 'decision_function' or "
+                           "'predict_proba' method.")
+    return method
 
-    classes : array-like, shape (n_classes,), optional
-            Contains unique classes used to fit the base estimator.
-            if None, then classes is extracted from the given target values
-            in fit().
 
-    See also
-    --------
-    CalibratedClassifierCV
+def _get_predictions(pred_method, X, n_classes):
+    """Returns predictions for `X` and reshapes binary outputs to shape
path: sklearn/calibration.py
text: @user1:
as a super nit pep8 recommends using "return" and "reshape" instead of using third person writing

@author:
No problem, good to know!

diff_hunk: @@ -377,145 +413,229 @@ def _more_tags(self):
         }
 
 
-class _CalibratedClassifier:
-    """Probability calibration with isotonic regression or sigmoid.
+def _get_pred_fit_calibrator(estimator, X, y, train, test, supports_sw,
+                             method, classes, sample_weight=None):
+    """Get predictions and fit calibrator for a given dataset split.
path: sklearn/calibration.py
text: @user1:
```suggestion
    """Compute predictions and fit a calibrator for a given dataset split.
```