CalibratedClassifierCV in 1.4 broke the compatibility with custom estimators that outputs float32.

### Describe the bug

Hi, this is an issue from xgboost forwarded here https://github.com/dmlc/xgboost/issues/10004 with copied code and backtrace.

XGBoost outputs float32 in its inference procedure, it seems the latest version of sklearn no longer works with it. May I ask do you want to accept a PR for converting inputs to float64 inside the calibrator, or the downstream estimators should simply output float64 unconditionally?

cc @jstammers

### Steps/Code to Reproduce

``` python
import xgboost as xgb
from sklearn.calibration import CalibratedClassifierCV
from sklearn.naive_bayes import GaussianNB
import pandas as pd
import numpy as np

df = pd.DataFrame(
    {"x": np.random.random(size=100), "y": np.random.choice([0, 1], size=100)}
)

model = xgb.XGBClassifier()

sk_model = GaussianNB()

sk_calibrator = CalibratedClassifierCV(sk_model)

sk_calibrator.fit(df[["x"]], df["y"]) # runs successfully

calibrator = CalibratedClassifierCV(model)

calibrator.fit(df[["x"]], df["y"])
```

### Expected Results

No exception.

### Actual Results

```
Traceback (most recent call last):
  File "~\incorrect_classifier.py", line 21, in <module>
    calibrator.fit(df[["x"]], df["y"])
  File "~\AppData\Local\pypoetry\Cache\virtualenvs\env-vqbIi1FM-py3.10\lib\site-packages\sklearn\base.py", line 1351, in wrapper
    return fit_method(estimator, *args, **kwargs)
  File "~\AppData\Local\pypoetry\Cache\virtualenvs\env-vqbIi1FM-py3.10\lib\site-packages\sklearn\calibration.py", line 403, in fit
    self.calibrated_classifiers_ = parallel(
  File "~\AppData\Local\pypoetry\Cache\virtualenvs\env-vqbIi1FM-py3.10\lib\site-packages\sklearn\utils\parallel.py", line 67, in __call__
    return super().__call__(iterable_with_config)
  File "~\AppData\Local\pypoetry\Cache\virtualenvs\env-vqbIi1FM-py3.10\lib\site-packages\joblib\parallel.py", line 1863, in __call__
    return output if self.return_generator else list(output)
  File "~\AppData\Local\pypoetry\Cache\virtualenvs\env-vqbIi1FM-py3.10\lib\site-packages\joblib\parallel.py", line 1792, in _get_sequential_output
    res = func(*args, **kwargs)
  File "~\AppData\Local\pypoetry\Cache\virtualenvs\env-vqbIi1FM-py3.10\lib\site-packages\sklearn\utils\parallel.py", line 129, in __call__
    return self.function(*args, **kwargs)
  File "~\AppData\Local\pypoetry\Cache\virtualenvs\env-vqbIi1FM-py3.10\lib\site-packages\sklearn\calibration.py", line 616, in _fit_classifier_calibrator_pair
    calibrated_classifier = _fit_calibrator(
  File "~\AppData\Local\pypoetry\Cache\virtualenvs\env-vqbIi1FM-py3.10\lib\site-packages\sklearn\calibration.py", line 663, in _fit_calibrator
    calibrator.fit(this_pred, Y[:, class_idx], sample_weight)
  File "~\AppData\Local\pypoetry\Cache\virtualenvs\env-vqbIi1FM-py3.10\lib\site-packages\sklearn\calibration.py", line 888, in fit
    self.a_, self.b_ = _sigmoid_calibration(X, y, sample_weight)
  File "~\AppData\Local\pypoetry\Cache\virtualenvs\env-vqbIi1FM-py3.10\lib\site-packages\sklearn\calibration.py", line 835, in _sigmoid_calibration
    opt_result = minimize(
  File "~\AppData\Local\pypoetry\Cache\virtualenvs\env-vqbIi1FM-py3.10\lib\site-packages\scipy\optimize\_minimize.py", line 710, in minimize
    res = _minimize_lbfgsb(fun, x0, args, jac, bounds,
  File "~\AppData\Local\pypoetry\Cache\virtualenvs\env-vqbIi1FM-py3.10\lib\site-packages\scipy\optimize\_lbfgsb_py.py", line 307, in _minimize_lbfgsb
    sf = _prepare_scalar_function(fun, x0, jac=jac, args=args, epsilon=eps,
  File "~\AppData\Local\pypoetry\Cache\virtualenvs\env-vqbIi1FM-py3.10\lib\site-packages\scipy\optimize\_optimize.py", line 383, in _prepare_scalar_function
    sf = ScalarFunction(fun, x0, args, grad, hess,
  File "~\AppData\Local\pypoetry\Cache\virtualenvs\env-vqbIi1FM-py3.10\lib\site-packages\scipy\optimize\_differentiable_functions.py", line 158, in __init__
    self._update_fun()
  File "~\AppData\Local\pypoetry\Cache\virtualenvs\env-vqbIi1FM-py3.10\lib\site-packages\scipy\optimize\_differentiable_functions.py", line 251, in _update_fun
    self._update_fun_impl()
  File "~\AppData\Local\pypoetry\Cache\virtualenvs\env-vqbIi1FM-py3.10\lib\site-packages\scipy\optimize\_differentiable_functions.py", line 155, in update_fun
    self.f = fun_wrapped(self.x)
  File "~\AppData\Local\pypoetry\Cache\virtualenvs\env-vqbIi1FM-py3.10\lib\site-packages\scipy\optimize\_differentiable_functions.py", line 137, in fun_wrapped
    fx = fun(np.copy(x), *args)
  File "~\AppData\Local\pypoetry\Cache\virtualenvs\env-vqbIi1FM-py3.10\lib\site-packages\scipy\optimize\_optimize.py", line 77, in __call__
    self._compute_if_needed(x, *args)
  File "~\AppData\Local\pypoetry\Cache\virtualenvs\env-vqbIi1FM-py3.10\lib\site-packages\scipy\optimize\_optimize.py", line 71, in _compute_if_needed
    fg = self.fun(x, *args)
  File "~\AppData\Local\pypoetry\Cache\virtualenvs\env-vqbIi1FM-py3.10\lib\site-packages\sklearn\calibration.py", line 824, in loss_grad
    l, g = bin_loss.loss_gradient(
  File "~\AppData\Local\pypoetry\Cache\virtualenvs\env-vqbIi1FM-py3.10\lib\site-packages\sklearn\_loss\loss.py", line 254, in loss_gradient
    self.closs.loss_gradient(
  File "sklearn\\_loss\\_loss.pyx", line 1784, in sklearn._loss._loss.CyHalfBinomialLoss.loss_gradient  
ValueError: Buffer dtype mismatch, expected 'const double' but got 'float'
```

### Versions

```shell
System:
    python: 3.10.13 | packaged by conda-forge | (main, Dec 23 2023, 15:36:39) [GCC 12.3.0]
executable: /home/runner/micromamba/envs/illuin_dev/bin/python3.10
   machine: Linux-6.2.0-1018-azure-x86_64-with-glibc2.35

Python dependencies:
      sklearn: 1.4.0
          pip: 23.3.2
   setuptools: 69.0.3
        numpy: 1.26.3
        scipy: 1.12.0
       Cython: None
       pandas: 2.2.0
   matplotlib: None
       joblib: 1.3.2
threadpoolctl: 3.2.0

Built with OpenMP: True

threadpoolctl info:
       user_api: openmp
   internal_api: openmp
    num_threads: 2
         prefix: libomp
       filepath: /home/runner/micromamba/envs/illuin_dev/lib/libomp.so
        version: None

       user_api: blas
   internal_api: openblas
    num_threads: 2
         prefix: libopenblas
       filepath: /home/runner/micromamba/envs/illuin_dev/lib/libopenblasp-r0.3.26.so
        version: 0.3.26
threading_layer: pthreads
   architecture: SkylakeX
```

COMMENT:
Please open a new issue and provide all the necessary information. You are not going to get any information on a closed issue.

Labels: Bug, module:calibration