Talip7/scikit-learn-issues-multilabel huggingface dataset:
Array API support for CalibratedClassifierCV

### Describe the workflow you want to enable

Towards #26024. 
Use `CalibratedClassifierCV` with pytorch or tensorflow models.
This has become even more interesting use case with #31068.

### Describe your proposed solution

In line with out Array API adoption path.

- [x] Temperature scaling
- [ ] Platt scaling
- [ ] Isotonic regression

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
#32246 has implemented array API support for the temperature scaling method which is arguably the lowest hanging fruit of the three methods.

Adding support for the other methods might be more challenging. I haven't checked the details but, I think that:

- Platt scaling uses LBFGS to tune its 2 continuous parameters (per class). I don't know if the scipy optimize API allows for non-NumPy array namespaces even if the gradient computing callable accepts it. Note that since the number of parameters to optimize for is very small, I think that it's ok to convert them back and forth if the expensive parts of the computation can happen on the original namespace, but I am not sure if that the case or not. Or we could just accept the array API namespace data to pass it to the underlying base estimator and convert the estimators predictions back to NumPy to fit for the calibration stage for each cv fold.
- isotonic regression is implemented in Cython so I don't think it's possible to support array API without conversion. But here again the cost of calibration might be small compared to the cost of fitting the underlying estimators, so maybe it's still worth accepting non-NumPy inputs to feed the fit method of the underlying estimator with and then convert the predictions to NumPy prior to passing them to isotonic regression Cython code.

Labels: New Feature, help wanted, Hard, module:calibration, Array API
Array API support for CalibratedClassifierCV

### Describe the workflow you want to enable

Towards #26024. 
Use `CalibratedClassifierCV` with pytorch or tensorflow models.
This has become even more interesting use case with #31068.

### Describe your proposed solution

In line with out Array API adoption path.

- [x] Temperature scaling
- [ ] Platt scaling
- [ ] Isotonic regression

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
> I don't know if the scipy optimize API allows for non-NumPy array namespaces even if the gradient computing callable accepts it.

I think the answer is not yet but it should be doable.

Labels: New Feature, help wanted, Hard, module:calibration, Array API
Array API support for CalibratedClassifierCV

### Describe the workflow you want to enable

Towards #26024. 
Use `CalibratedClassifierCV` with pytorch or tensorflow models.
This has become even more interesting use case with #31068.

### Describe your proposed solution

In line with out Array API adoption path.

- [x] Temperature scaling
- [ ] Platt scaling
- [ ] Isotonic regression

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
I'll start working on this!

Labels: New Feature, help wanted, Hard, module:calibration, Array API
Array API support for CalibratedClassifierCV

### Describe the workflow you want to enable

Towards #26024. 
Use `CalibratedClassifierCV` with pytorch or tensorflow models.
This has become even more interesting use case with #31068.

### Describe your proposed solution

In line with out Array API adoption path.

- [x] Temperature scaling
- [ ] Platt scaling
- [ ] Isotonic regression

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
Sorry I didn't see the ping before. Re: https://github.com/scikit-learn/scikit-learn/issues/31869#issuecomment-3252373524, do you mean like nonlinear constraints in multiple dimensions? (Because technically `find_minimum` is constrained minimization because it enforces bounds.)

> benchmarks may be a different matter

They are not intended to be fast for inexpensive user callables acting on size-1 arrays ; I didn't minimize overhead. But I've seen, say, 300x improvement on large arrays with CuPy on pretty practical examples. And actually, Albert implemented the same algorithm as `find_minimum` for inverting ufuncs in `special`, all in compiled code, and it was only twice as fast as using `find_root` on the ufuncs (with reasonably sized NumPy arrays), so it's not terrible.

Labels: New Feature, help wanted, Hard, module:calibration, Array API
Array API support for CalibratedClassifierCV

### Describe the workflow you want to enable

Towards #26024. 
Use `CalibratedClassifierCV` with pytorch or tensorflow models.
This has become even more interesting use case with #31068.

### Describe your proposed solution

In line with out Array API adoption path.

- [x] Temperature scaling
- [ ] Platt scaling
- [ ] Isotonic regression

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
> (Because technically `find_minimum` is constrained minimization because it enforces bounds.)

that's what I was referring too, yeah

Labels: New Feature, help wanted, Hard, module:calibration, Array API
Array API support for CalibratedClassifierCV

### Describe the workflow you want to enable

Towards #26024. 
Use `CalibratedClassifierCV` with pytorch or tensorflow models.
This has become even more interesting use case with #31068.

### Describe your proposed solution

In line with out Array API adoption path.

- [x] Temperature scaling
- [ ] Platt scaling
- [ ] Isotonic regression

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
Oh, ok, so you had already gotten your answer. Haha well sorry I'm two months late to the party.

Labels: New Feature, help wanted, Hard, module:calibration, Array API
Ordinal Encoder Type Hints State unknown_value should be float, but this produces an error.

### Describe the bug

Following the type hints of the OrdinalEncoder I set the unknown_value parameter to -1.0.

<img width="507" height="146" alt="Image" src="https://github.com/user-attachments/assets/b9c86ab1-7a23-47b3-ad89-9de091c8d81e" />

This produces an error when handle_unknown='use_encoded_value' as it needs and int. Should hopefully just be as easy as updating the type hints unless there is something I'm missing?

### Steps/Code to Reproduce

```python
ordinal_encoder = OrdinalEncoder(
                handle_unknown="use_encoded_value", unknown_value=-1
            )

ordinal_encoder.fit_transform(...)
```

### Expected Results

Expected result would be to not get an error when following type hints.

### Actual Results

An error is raised about the type of the unknown_value

### Versions

```shell
internal_api: openblas
    num_threads: 12
         prefix: libscipy_openblas
       filepath: /databricks/python3/lib/python3.12/site-packages/scipy.libs/libscipy_openblas-68440149.so
        version: 0.3.28
threading_layer: pthreads
   architecture: SkylakeX

       user_api: openmp
   internal_api: openmp
    num_threads: 6
         prefix: libgomp
       filepath: /databricks/python3/lib/python3.12/site-packages/torch/lib/libgomp-a34b3233.so.1
        version: None

       user_api: openmp
   internal_api: openmp
    num_threads: 12
         prefix: libgomp
       filepath: /databricks/python3/lib/python3.12/site-packages/scikit_learn.libs/libgomp-a34b3233.so.1.0.0
        version: None
```

COMMENT:
Thanks for your issue.

The scikit-learn codebase use very little typehints so the typehints you are seeing don't come from us. You should figure out where your typehints are coming from (likely https://github.com/microsoft/python-type-stubs?) and report the issues there.

Closing this one.

Labels: Bug, Needs Triage
Intercepts of Newton-Cholesky logistic regression get corrupted when warm starting

### Describe the bug

When using multinomial logistic regression with warm starts from a previous iteration, the final coefficients in the model are correct, but the intercepts somehow get filled with incorrect numbers somewhere.

As a result, predictions from a warm-started model differ from those of a cold-start model that has more iterations on the same data.

The issue appears to have been introduced recently as it works fine with version 1.5, but not with 1.6 or 1.7.

### Steps/Code to Reproduce

```python
import numpy as np
from sklearn.datasets import load_iris
from sklearn.linear_model import LogisticRegression
X, y = load_iris(return_X_y=True)

model1 = LogisticRegression(
    solver="newton-cholesky",
    max_iter=2
).fit(X, y)
model2 = LogisticRegression(
    solver="newton-cholesky",
    max_iter=1,
    warm_start=True
).fit(X, y).fit(X, y)

np.testing.assert_almost_equal(
    model1.coef_,
    model2.coef_
)

np.testing.assert_almost_equal(
    model1.predict_proba(X[:5]),
    model2.predict_proba(X[:5])
)
```

### Expected Results

Intercepts should be the same, up to shifting by a constant if needed.

### Actual Results

Intercepts are different, as are predicted probabilities

### Versions

```shell
System:
    python: 3.12.6 | packaged by conda-forge | (main, Sep 22 2024, 14:16:49) [GCC 13.3.0]
executable: /home/david/miniforge3/bin/python
   machine: Linux-6.12.33+deb12-amd64-x86_64-with-glibc2.36

Python dependencies:
      sklearn: 1.7.1
          pip: 24.2
   setuptools: 74.1.2
        numpy: 2.0.1
        scipy: 1.14.1
       Cython: 3.1.0
       pandas: 2.2.3
   matplotlib: 3.9.2
       joblib: 1.4.2
threadpoolctl: 3.5.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: openblas
    num_threads: 20
         prefix: libscipy_openblas
       filepath: /home/david/.local/lib/python3.12/site-packages/numpy.libs/libscipy_openblas64_-99b71e71.so
        version: 0.3.27
threading_layer: pthreads
   architecture: Haswell

       user_api: blas
   internal_api: mkl
    num_threads: 14
         prefix: libmkl_rt
       filepath: /home/david/miniforge3/lib/libmkl_rt.so.2
        version: 2023.2-Product
threading_layer: gnu

       user_api: openmp
   internal_api: openmp
    num_threads: 20
         prefix: libgomp
       filepath: /home/david/miniforge3/lib/libgomp.so.1.0.0
        version: None
```

COMMENT:
Why should it be the same? Model 1 uses 2 newton iterations, model 2 only 1. The warm start does not take effect as there is nothing to "warm start with".

Note that multinomial newton-cholesky is only available as of version 1.6.

Labels: Bug, module:linear_model
Intercepts of Newton-Cholesky logistic regression get corrupted when warm starting

### Describe the bug

When using multinomial logistic regression with warm starts from a previous iteration, the final coefficients in the model are correct, but the intercepts somehow get filled with incorrect numbers somewhere.

As a result, predictions from a warm-started model differ from those of a cold-start model that has more iterations on the same data.

The issue appears to have been introduced recently as it works fine with version 1.5, but not with 1.6 or 1.7.

### Steps/Code to Reproduce

```python
import numpy as np
from sklearn.datasets import load_iris
from sklearn.linear_model import LogisticRegression
X, y = load_iris(return_X_y=True)

model1 = LogisticRegression(
    solver="newton-cholesky",
    max_iter=2
).fit(X, y)
model2 = LogisticRegression(
    solver="newton-cholesky",
    max_iter=1,
    warm_start=True
).fit(X, y).fit(X, y)

np.testing.assert_almost_equal(
    model1.coef_,
    model2.coef_
)

np.testing.assert_almost_equal(
    model1.predict_proba(X[:5]),
    model2.predict_proba(X[:5])
)
```

### Expected Results

Intercepts should be the same, up to shifting by a constant if needed.

### Actual Results

Intercepts are different, as are predicted probabilities

### Versions

```shell
System:
    python: 3.12.6 | packaged by conda-forge | (main, Sep 22 2024, 14:16:49) [GCC 13.3.0]
executable: /home/david/miniforge3/bin/python
   machine: Linux-6.12.33+deb12-amd64-x86_64-with-glibc2.36

Python dependencies:
      sklearn: 1.7.1
          pip: 24.2
   setuptools: 74.1.2
        numpy: 2.0.1
        scipy: 1.14.1
       Cython: 3.1.0
       pandas: 2.2.3
   matplotlib: 3.9.2
       joblib: 1.4.2
threadpoolctl: 3.5.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: openblas
    num_threads: 20
         prefix: libscipy_openblas
       filepath: /home/david/.local/lib/python3.12/site-packages/numpy.libs/libscipy_openblas64_-99b71e71.so
        version: 0.3.27
threading_layer: pthreads
   architecture: Haswell

       user_api: blas
   internal_api: mkl
    num_threads: 14
         prefix: libmkl_rt
       filepath: /home/david/miniforge3/lib/libmkl_rt.so.2
        version: 2023.2-Product
threading_layer: gnu

       user_api: openmp
   internal_api: openmp
    num_threads: 20
         prefix: libgomp
       filepath: /home/david/miniforge3/lib/libgomp.so.1.0.0
        version: None
```

COMMENT:
@lorentzenchr `model2` with `warm_start=True` is fit twice though so it does 2 newton iterations right (just mentioning this in case you missed it)? I was going to ping you for insights but looks like you are already on it, thanks!

For reference: a `git bisect` points at #28840 that implemented multiclass for newton-cholesky solver and was released in 1.6. So in 1.6 newton-cholesky use multinomial solver and in 1.5 was using One vs Rest. Using `multi_class='ovr'` make the snippet pass for scikit-learn >= 1.6.

Labels: Bug, module:linear_model
Intercepts of Newton-Cholesky logistic regression get corrupted when warm starting

### Describe the bug

When using multinomial logistic regression with warm starts from a previous iteration, the final coefficients in the model are correct, but the intercepts somehow get filled with incorrect numbers somewhere.

As a result, predictions from a warm-started model differ from those of a cold-start model that has more iterations on the same data.

The issue appears to have been introduced recently as it works fine with version 1.5, but not with 1.6 or 1.7.

### Steps/Code to Reproduce

```python
import numpy as np
from sklearn.datasets import load_iris
from sklearn.linear_model import LogisticRegression
X, y = load_iris(return_X_y=True)

model1 = LogisticRegression(
    solver="newton-cholesky",
    max_iter=2
).fit(X, y)
model2 = LogisticRegression(
    solver="newton-cholesky",
    max_iter=1,
    warm_start=True
).fit(X, y).fit(X, y)

np.testing.assert_almost_equal(
    model1.coef_,
    model2.coef_
)

np.testing.assert_almost_equal(
    model1.predict_proba(X[:5]),
    model2.predict_proba(X[:5])
)
```

### Expected Results

Intercepts should be the same, up to shifting by a constant if needed.

### Actual Results

Intercepts are different, as are predicted probabilities

### Versions

```shell
System:
    python: 3.12.6 | packaged by conda-forge | (main, Sep 22 2024, 14:16:49) [GCC 13.3.0]
executable: /home/david/miniforge3/bin/python
   machine: Linux-6.12.33+deb12-amd64-x86_64-with-glibc2.36

Python dependencies:
      sklearn: 1.7.1
          pip: 24.2
   setuptools: 74.1.2
        numpy: 2.0.1
        scipy: 1.14.1
       Cython: 3.1.0
       pandas: 2.2.3
   matplotlib: 3.9.2
       joblib: 1.4.2
threadpoolctl: 3.5.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: openblas
    num_threads: 20
         prefix: libscipy_openblas
       filepath: /home/david/.local/lib/python3.12/site-packages/numpy.libs/libscipy_openblas64_-99b71e71.so
        version: 0.3.27
threading_layer: pthreads
   architecture: Haswell

       user_api: blas
   internal_api: mkl
    num_threads: 14
         prefix: libmkl_rt
       filepath: /home/david/miniforge3/lib/libmkl_rt.so.2
        version: 2023.2-Product
threading_layer: gnu

       user_api: openmp
   internal_api: openmp
    num_threads: 20
         prefix: libgomp
       filepath: /home/david/miniforge3/lib/libgomp.so.1.0.0
        version: None
```

COMMENT:
> Why should it be the same? Model 1 uses 2 newton iterations, model 2 only 1. The warm start does not take effect as there is nothing to "warm start with".
> 
> Note that multinomial newton-cholesky is only available as of version 1.6.

Also if they weren't equivalent, the coefficients wouldn't end up being exactly the same - which in this case they are, with only the intercepts differing.

Labels: Bug, module:linear_model
Intercepts of Newton-Cholesky logistic regression get corrupted when warm starting

### Describe the bug

When using multinomial logistic regression with warm starts from a previous iteration, the final coefficients in the model are correct, but the intercepts somehow get filled with incorrect numbers somewhere.

As a result, predictions from a warm-started model differ from those of a cold-start model that has more iterations on the same data.

The issue appears to have been introduced recently as it works fine with version 1.5, but not with 1.6 or 1.7.

### Steps/Code to Reproduce

```python
import numpy as np
from sklearn.datasets import load_iris
from sklearn.linear_model import LogisticRegression
X, y = load_iris(return_X_y=True)

model1 = LogisticRegression(
    solver="newton-cholesky",
    max_iter=2
).fit(X, y)
model2 = LogisticRegression(
    solver="newton-cholesky",
    max_iter=1,
    warm_start=True
).fit(X, y).fit(X, y)

np.testing.assert_almost_equal(
    model1.coef_,
    model2.coef_
)

np.testing.assert_almost_equal(
    model1.predict_proba(X[:5]),
    model2.predict_proba(X[:5])
)
```

### Expected Results

Intercepts should be the same, up to shifting by a constant if needed.

### Actual Results

Intercepts are different, as are predicted probabilities

### Versions

```shell
System:
    python: 3.12.6 | packaged by conda-forge | (main, Sep 22 2024, 14:16:49) [GCC 13.3.0]
executable: /home/david/miniforge3/bin/python
   machine: Linux-6.12.33+deb12-amd64-x86_64-with-glibc2.36

Python dependencies:
      sklearn: 1.7.1
          pip: 24.2
   setuptools: 74.1.2
        numpy: 2.0.1
        scipy: 1.14.1
       Cython: 3.1.0
       pandas: 2.2.3
   matplotlib: 3.9.2
       joblib: 1.4.2
threadpoolctl: 3.5.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: openblas
    num_threads: 20
         prefix: libscipy_openblas
       filepath: /home/david/.local/lib/python3.12/site-packages/numpy.libs/libscipy_openblas64_-99b71e71.so
        version: 0.3.27
threading_layer: pthreads
   architecture: Haswell

       user_api: blas
   internal_api: mkl
    num_threads: 14
         prefix: libmkl_rt
       filepath: /home/david/miniforge3/lib/libmkl_rt.so.2
        version: 2023.2-Product
threading_layer: gnu

       user_api: openmp
   internal_api: openmp
    num_threads: 20
         prefix: libgomp
       filepath: /home/david/miniforge3/lib/libgomp.so.1.0.0
        version: None
```

COMMENT:
@lesteve Thanks for point out the .fit.fit.

This is a tricky issue. Please note that model1 as well as model2 emit a ConvergenceWarning.
The reason why the intercept is different is that the `NewtonCholeskySolver` calls `finalize` at the end of its work, see https://github.com/scikit-learn/scikit-learn/blob/3d35e02d762e305fc3500b5ec59418340eeb90bf/sklearn/linear_model/_glm/_newton_solver.py#L606

There, the intercept is symmetrized to adhere to our convention that coefficients and intercept should sum to zero. Therefore, two separate calls to fit with one step and warm start are not the same as one call to fit with 2 steps.

Labels: Bug, module:linear_model
Intercepts of Newton-Cholesky logistic regression get corrupted when warm starting

### Describe the bug

When using multinomial logistic regression with warm starts from a previous iteration, the final coefficients in the model are correct, but the intercepts somehow get filled with incorrect numbers somewhere.

As a result, predictions from a warm-started model differ from those of a cold-start model that has more iterations on the same data.

The issue appears to have been introduced recently as it works fine with version 1.5, but not with 1.6 or 1.7.

### Steps/Code to Reproduce

```python
import numpy as np
from sklearn.datasets import load_iris
from sklearn.linear_model import LogisticRegression
X, y = load_iris(return_X_y=True)

model1 = LogisticRegression(
    solver="newton-cholesky",
    max_iter=2
).fit(X, y)
model2 = LogisticRegression(
    solver="newton-cholesky",
    max_iter=1,
    warm_start=True
).fit(X, y).fit(X, y)

np.testing.assert_almost_equal(
    model1.coef_,
    model2.coef_
)

np.testing.assert_almost_equal(
    model1.predict_proba(X[:5]),
    model2.predict_proba(X[:5])
)
```

### Expected Results

Intercepts should be the same, up to shifting by a constant if needed.

### Actual Results

Intercepts are different, as are predicted probabilities

### Versions

```shell
System:
    python: 3.12.6 | packaged by conda-forge | (main, Sep 22 2024, 14:16:49) [GCC 13.3.0]
executable: /home/david/miniforge3/bin/python
   machine: Linux-6.12.33+deb12-amd64-x86_64-with-glibc2.36

Python dependencies:
      sklearn: 1.7.1
          pip: 24.2
   setuptools: 74.1.2
        numpy: 2.0.1
        scipy: 1.14.1
       Cython: 3.1.0
       pandas: 2.2.3
   matplotlib: 3.9.2
       joblib: 1.4.2
threadpoolctl: 3.5.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: openblas
    num_threads: 20
         prefix: libscipy_openblas
       filepath: /home/david/.local/lib/python3.12/site-packages/numpy.libs/libscipy_openblas64_-99b71e71.so
        version: 0.3.27
threading_layer: pthreads
   architecture: Haswell

       user_api: blas
   internal_api: mkl
    num_threads: 14
         prefix: libmkl_rt
       filepath: /home/david/miniforge3/lib/libmkl_rt.so.2
        version: 2023.2-Product
threading_layer: gnu

       user_api: openmp
   internal_api: openmp
    num_threads: 20
         prefix: libgomp
       filepath: /home/david/miniforge3/lib/libgomp.so.1.0.0
        version: None
```

COMMENT:
After iteration 2, the predicted probabilities differ, same with your original code snippet.

The question is whether this requires a fix (removing finalize), or whether we can live with it (as users are not supposed to use the solver that way).

Up until now, I can live with it. Is there a good use case to come to another conclusion?

Labels: Bug, module:linear_model