CalibratedClassifierCV fails silently with large confidence scores

https://github.com/scikit-learn/scikit-learn/blob/7f9bad99d6e0a3e8ddf92a7e5561245224dab102/sklearn/calibration.py#L892

With default `method='sigmoid'`, `CalibratedClassifierCV` may fail silently, giving an AUC score of 0.5, apparently when the classifier outputs large confidence scores. This happens in particular with `SGDClassifier`. You may question the quality of `SGDClassifier`; that's a separate question. But if you change `method` to `isotonic`, then AUC is correct, showing that despite suspiciously large confidence scores, `SGDClassifier` does rank the samples correctly. Code to reproduce:
```python
import numpy as np
from sklearn.calibration import CalibratedClassifierCV
from sklearn.linear_model import SGDClassifier
from sklearn.model_selection import cross_val_score
# Setting up a trivially easy classification problem
r = 0.67
N = 1000
y_train = np.array([1]*int(N*r)+[0]*(N-int(N*r)))
X_train = 1e5*y_train.reshape((-1,1)) + np.random.default_rng(42).normal(size=N)
model = CalibratedClassifierCV(SGDClassifier(loss='squared_hinge',random_state=42))
print('Logistic calibration: ', cross_val_score(model,X_train,y_train,scoring='roc_auc').mean())
model = CalibratedClassifierCV(SGDClassifier(loss='squared_hinge',random_state=42),method='isotonic')
print('Isotonic calibration: ', cross_val_score(model,X_train,y_train,scoring='roc_auc').mean())
```
Expected output:
```text
Logistic calibration:  0.5
Isotonic calibration:  1.0
```
The logistic calibration code appears to fail silently at this line https://github.com/scikit-learn/scikit-learn/blob/7f9bad99d6e0a3e8ddf92a7e5561245224dab102/sklearn/calibration.py#L892)
Because `disp=False`, the caller does not know that `fmin_bfgs` has failed. If you turn on the warning, then you can see that it fails without one iteration, so that logistic calibration outputs constant probability.
```python
from math import log
from scipy.special import expit, xlogy
from sklearn.utils import column_or_1d
from scipy.optimize import fmin_bfgs

def _sigmoid_calibration_debug(predictions, y, sample_weight=None):
    """Probability Calibration with sigmoid method (Platt 2000)
    Parameters
    ----------
    predictions : ndarray of shape (n_samples,)
        The decision function or predict proba for the samples.
    y : ndarray of shape (n_samples,)
        The targets.
    sample_weight : array-like of shape (n_samples,), default=None
        Sample weights. If None, then samples are equally weighted.
    Returns
    -------
    a : float
        The slope.
    b : float
        The intercept.
    References
    ----------
    Platt, "Probabilistic Outputs for Support Vector Machines"
    """
    predictions = column_or_1d(predictions)
    y = column_or_1d(y)

    F = predictions  # F follows Platt's notations

    # Bayesian priors (see Platt end of section 2.2):
    # It corresponds to the number of samples, taking into account the
    # `sample_weight`.
    mask_negative_samples = y <= 0
    if sample_weight is not None:
        prior0 = (sample_weight[mask_negative_samples]).sum()
        prior1 = (sample_weight[~mask_negative_samples]).sum()
    else:
        prior0 = float(np.sum(mask_negative_samples))
        prior1 = y.shape[0] - prior0
    T = np.zeros_like(y, dtype=np.float64)
    T[y > 0] = (prior1 + 1.0) / (prior1 + 2.0)
    T[y <= 0] = 1.0 / (prior0 + 2.0)
    T1 = 1.0 - T

    def objective(AB):
        # From Platt (beginning of Section 2.2)
        P = expit(-(AB[0] * F + AB[1]))
        loss = -(xlogy(T, P) + xlogy(T1, 1.0 - P))
        if sample_weight is not None:
            return (sample_weight * loss).sum()
        else:
            return loss.sum()

    def grad(AB):
        # gradient of the objective function
        P = expit(-(AB[0] * F + AB[1]))
        TEP_minus_T1P = T - P
        if sample_weight is not None:
            TEP_minus_T1P *= sample_weight
        dA = np.dot(TEP_minus_T1P, F)
        dB = np.sum(TEP_minus_T1P)
        return np.array([dA, dB])

    AB0 = np.array([0.0, log((prior0 + 1.0) / (prior1 + 1.0))])
    AB_ = fmin_bfgs(objective, AB0, fprime=grad, disp=True) # Turn on outputs
    return AB_[0], AB_[1]

import sklearn
sklearn.calibration._sigmoid_calibration = _sigmoid_calibration_debug

model = CalibratedClassifierCV(SGDClassifier(loss='squared_hinge',random_state=42))
print('Logistic calibration: ', cross_val_score(model,X_train,y_train,scoring='roc_auc').mean())
```
Expected output:
```text
Warning: Desired error not necessarily achieved due to precision loss.
         Current function value: 100.908259
         Iterations: 0
         Function evaluations: 15
         Gradient evaluations: 3
Warning: Desired error not necessarily achieved due to precision loss.
         Current function value: 101.623705
         Iterations: 0
         Function evaluations: 15
         Gradient evaluations: 3
Warning: Desired error not necessarily achieved due to precision loss.
         Current function value: 101.623705
         Iterations: 0
         Function evaluations: 15
         Gradient evaluations: 3
Warning: Desired error not necessarily achieved due to precision loss.
         Current function value: 101.623705
         Iterations: 0
         Function evaluations: 15
         Gradient evaluations: 3
Warning: Desired error not necessarily achieved due to precision loss.
         Current function value: 101.623705
         Iterations: 0
         Function evaluations: 15
         Gradient evaluations: 3
Warning: Desired error not necessarily achieved due to precision loss.
         Current function value: 100.908259
         Iterations: 0
         Function evaluations: 15
         Gradient evaluations: 3
Warning: Desired error not necessarily achieved due to precision loss.
         Current function value: 101.623705
         Iterations: 0
         Function evaluations: 15
         Gradient evaluations: 3
Warning: Desired error not necessarily achieved due to precision loss.
         Current function value: 101.623705
         Iterations: 0
         Function evaluations: 15
         Gradient evaluations: 3
Warning: Desired error not necessarily achieved due to precision loss.
         Current function value: 101.623705
         Iterations: 0
         Function evaluations: 15
         Gradient evaluations: 3
Warning: Desired error not necessarily achieved due to precision loss.
         Current function value: 101.623705
         Iterations: 0
         Function evaluations: 15
         Gradient evaluations: 3
Warning: Desired error not necessarily achieved due to precision loss.
         Current function value: 100.908259
         Iterations: 0
         Function evaluations: 15
         Gradient evaluations: 3
Warning: Desired error not necessarily achieved due to precision loss.
         Current function value: 101.623705
         Iterations: 0
         Function evaluations: 15
         Gradient evaluations: 3
Warning: Desired error not necessarily achieved due to precision loss.
         Current function value: 101.623705
         Iterations: 0
         Function evaluations: 15
         Gradient evaluations: 3
Warning: Desired error not necessarily achieved due to precision loss.
         Current function value: 101.623705
         Iterations: 0
         Function evaluations: 15
         Gradient evaluations: 3
Warning: Desired error not necessarily achieved due to precision loss.
         Current function value: 101.623705
         Iterations: 0
         Function evaluations: 15
         Gradient evaluations: 3
Warning: Desired error not necessarily achieved due to precision loss.
         Current function value: 100.908259
         Iterations: 0
         Function evaluations: 15
         Gradient evaluations: 3
Warning: Desired error not necessarily achieved due to precision loss.
         Current function value: 101.623705
         Iterations: 0
         Function evaluations: 15
         Gradient evaluations: 3
Warning: Desired error not necessarily achieved due to precision loss.
         Current function value: 101.623705
         Iterations: 0
         Function evaluations: 15
         Gradient evaluations: 3
Warning: Desired error not necessarily achieved due to precision loss.
         Current function value: 101.623705
         Iterations: 0
         Function evaluations: 15
         Gradient evaluations: 3
Warning: Desired error not necessarily achieved due to precision loss.
         Current function value: 101.623705
         Iterations: 0
         Function evaluations: 15
         Gradient evaluations: 3
Warning: Desired error not necessarily achieved due to precision loss.
         Current function value: 100.908259
         Iterations: 0
         Function evaluations: 15
         Gradient evaluations: 3
Warning: Desired error not necessarily achieved due to precision loss.
         Current function value: 101.623705
         Iterations: 0
         Function evaluations: 15
         Gradient evaluations: 3
Warning: Desired error not necessarily achieved due to precision loss.
         Current function value: 101.623705
         Iterations: 0
         Function evaluations: 15
         Gradient evaluations: 3
Warning: Desired error not necessarily achieved due to precision loss.
         Current function value: 101.623705
         Iterations: 0
         Function evaluations: 15
         Gradient evaluations: 3
Warning: Desired error not necessarily achieved due to precision loss.
         Current function value: 101.623705
         Iterations: 0
         Function evaluations: 15
         Gradient evaluations: 3
Logistic calibration:  0.5
```
Perhaps a more robust solution is to replace `fmin_bfgs` by a derivative-free minimization algorithm such as `scipy.optimize.minimize` with `method='Nelder-Mead'`.

COMMENT:
Thanks for the report and the analysis. Since there are only 2 parameters, we could probably materialize the Hessian and use exact Newton. When the Hessian is not (numerically) positive definite or when the line search fails we could probably detect it and fallback to Nelder-Mead or something similar, although I am not sure if the fallback solver would be guaranteed to no suffer from some other pathological edge cases.

Labels: Bug