Talip7/scikit-learn-issues-multilabel huggingface dataset:
BUG Problem when `CalibratedClassifierCV` train contains 2 classes but data contains more

### Describe the bug

In `CalibratedClassifierCV` when a train split contains 2 classes (binary) but the data contains more (>=3) classes, we assume the data is binary:

https://github.com/scikit-learn/scikit-learn/blob/d20e0b9abc4a4798d1fd839db50b19c01723094e/sklearn/calibration.py#L605-L607

and we only end up fitting one calibrator:

https://github.com/scikit-learn/scikit-learn/blob/d20e0b9abc4a4798d1fd839db50b19c01723094e/sklearn/calibration.py#L620-L621

Context: noticed when looking #29545 and trying to update [`test_calibration_less_classes`](https://github.com/scikit-learn/scikit-learn/blob/d20e0b9abc4a4798d1fd839db50b19c01723094e/sklearn/tests/test_calibration.py#L441)

### Steps/Code to Reproduce

```python
import numpy as np

from sklearn.model_selection import KFold
from sklearn.tree import DecisionTreeClassifier
from sklearn.calibration import CalibratedClassifierCV

X = np.random.randn(12, 5)
y = [0, 0, 0, 0] + [1, 1, 1, 1] + [2, 2, 2, 2]
clf = DecisionTreeClassifier(random_state=7)
cal_clf = CalibratedClassifierCV(
    clf, method="sigmoid", cv=KFold(3), ensemble=True
)
cal_clf.fit(X, y)
for i in range(3):
    print(f'Fold: {i}')
    proba = cal_clf.calibrated_classifiers_[i].predict_proba(X)
    print(proba)
```

### Expected Results

Expect proba to be 0 ONLY for the class not present in the train subset.

### Actual Results

```
Fold: 0  # train contains class 1 and 2, we take the first `pos_class_indices` (1) to be the positive class
[[0. 1. 0.]
 [0. 1. 0.]
 [0. 1. 0.]
 [0. 1. 0.]
 [0. 1. 0.]
 [0. 1. 0.]
 [0. 1. 0.]
 [0. 1. 0.]
 [0. 1. 0.]
 [0. 1. 0.]
 [0. 1. 0.]
 [0. 1. 0.]]
Fold: 1  # train contains class 0 and 2, 0 is the first `pos_class_indices`
[[1. 0. 0.]
 [1. 0. 0.]
 [1. 0. 0.]
 [1. 0. 0.]
 [1. 0. 0.]
 [1. 0. 0.]
 [1. 0. 0.]
 [1. 0. 0.]
 [1. 0. 0.]
 [1. 0. 0.]
 [1. 0. 0.]
 [1. 0. 0.]]
Fold: 2  # train contains class 0 and 1, `0` is the first `pos_class_indices`
[[1. 0. 0.]
 [1. 0. 0.]
 [1. 0. 0.]
 [1. 0. 0.]
 [1. 0. 0.]
 [1. 0. 0.]
 [1. 0. 0.]
 [1. 0. 0.]
 [1. 0. 0.]
 [1. 0. 0.]
 [1. 0. 0.]
 [1. 0. 0.]]
```

A reasonable fix is to check when `CalibratedClassifierCV.classes_` is greater than `estimator.classes_` and output both proba and 1 - proba (assuming we can know which class the estimator deemed to be the positive class).

It does raise the question of whether we should warn when this happens..?


### Versions

```shell
Used main
```

COMMENT:
Opening an issue for the CV should make sense and we probably discussed this problem in a drafting meeting or developer meeting.

Labels: Bug
BUG Problem when `CalibratedClassifierCV` train contains 2 classes but data contains more

### Describe the bug

In `CalibratedClassifierCV` when a train split contains 2 classes (binary) but the data contains more (>=3) classes, we assume the data is binary:

https://github.com/scikit-learn/scikit-learn/blob/d20e0b9abc4a4798d1fd839db50b19c01723094e/sklearn/calibration.py#L605-L607

and we only end up fitting one calibrator:

https://github.com/scikit-learn/scikit-learn/blob/d20e0b9abc4a4798d1fd839db50b19c01723094e/sklearn/calibration.py#L620-L621

Context: noticed when looking #29545 and trying to update [`test_calibration_less_classes`](https://github.com/scikit-learn/scikit-learn/blob/d20e0b9abc4a4798d1fd839db50b19c01723094e/sklearn/tests/test_calibration.py#L441)

### Steps/Code to Reproduce

```python
import numpy as np

from sklearn.model_selection import KFold
from sklearn.tree import DecisionTreeClassifier
from sklearn.calibration import CalibratedClassifierCV

X = np.random.randn(12, 5)
y = [0, 0, 0, 0] + [1, 1, 1, 1] + [2, 2, 2, 2]
clf = DecisionTreeClassifier(random_state=7)
cal_clf = CalibratedClassifierCV(
    clf, method="sigmoid", cv=KFold(3), ensemble=True
)
cal_clf.fit(X, y)
for i in range(3):
    print(f'Fold: {i}')
    proba = cal_clf.calibrated_classifiers_[i].predict_proba(X)
    print(proba)
```

### Expected Results

Expect proba to be 0 ONLY for the class not present in the train subset.

### Actual Results

```
Fold: 0  # train contains class 1 and 2, we take the first `pos_class_indices` (1) to be the positive class
[[0. 1. 0.]
 [0. 1. 0.]
 [0. 1. 0.]
 [0. 1. 0.]
 [0. 1. 0.]
 [0. 1. 0.]
 [0. 1. 0.]
 [0. 1. 0.]
 [0. 1. 0.]
 [0. 1. 0.]
 [0. 1. 0.]
 [0. 1. 0.]]
Fold: 1  # train contains class 0 and 2, 0 is the first `pos_class_indices`
[[1. 0. 0.]
 [1. 0. 0.]
 [1. 0. 0.]
 [1. 0. 0.]
 [1. 0. 0.]
 [1. 0. 0.]
 [1. 0. 0.]
 [1. 0. 0.]
 [1. 0. 0.]
 [1. 0. 0.]
 [1. 0. 0.]
 [1. 0. 0.]]
Fold: 2  # train contains class 0 and 1, `0` is the first `pos_class_indices`
[[1. 0. 0.]
 [1. 0. 0.]
 [1. 0. 0.]
 [1. 0. 0.]
 [1. 0. 0.]
 [1. 0. 0.]
 [1. 0. 0.]
 [1. 0. 0.]
 [1. 0. 0.]
 [1. 0. 0.]
 [1. 0. 0.]
 [1. 0. 0.]]
```

A reasonable fix is to check when `CalibratedClassifierCV.classes_` is greater than `estimator.classes_` and output both proba and 1 - proba (assuming we can know which class the estimator deemed to be the positive class).

It does raise the question of whether we should warn when this happens..?


### Versions

```shell
Used main
```

COMMENT:
I see you have opened an issue about this already: https://github.com/scikit-learn/scikit-learn/issues/29558, so just linking

Labels: Bug
BUG Problem when `CalibratedClassifierCV` train contains 2 classes but data contains more

### Describe the bug

In `CalibratedClassifierCV` when a train split contains 2 classes (binary) but the data contains more (>=3) classes, we assume the data is binary:

https://github.com/scikit-learn/scikit-learn/blob/d20e0b9abc4a4798d1fd839db50b19c01723094e/sklearn/calibration.py#L605-L607

and we only end up fitting one calibrator:

https://github.com/scikit-learn/scikit-learn/blob/d20e0b9abc4a4798d1fd839db50b19c01723094e/sklearn/calibration.py#L620-L621

Context: noticed when looking #29545 and trying to update [`test_calibration_less_classes`](https://github.com/scikit-learn/scikit-learn/blob/d20e0b9abc4a4798d1fd839db50b19c01723094e/sklearn/tests/test_calibration.py#L441)

### Steps/Code to Reproduce

```python
import numpy as np

from sklearn.model_selection import KFold
from sklearn.tree import DecisionTreeClassifier
from sklearn.calibration import CalibratedClassifierCV

X = np.random.randn(12, 5)
y = [0, 0, 0, 0] + [1, 1, 1, 1] + [2, 2, 2, 2]
clf = DecisionTreeClassifier(random_state=7)
cal_clf = CalibratedClassifierCV(
    clf, method="sigmoid", cv=KFold(3), ensemble=True
)
cal_clf.fit(X, y)
for i in range(3):
    print(f'Fold: {i}')
    proba = cal_clf.calibrated_classifiers_[i].predict_proba(X)
    print(proba)
```

### Expected Results

Expect proba to be 0 ONLY for the class not present in the train subset.

### Actual Results

```
Fold: 0  # train contains class 1 and 2, we take the first `pos_class_indices` (1) to be the positive class
[[0. 1. 0.]
 [0. 1. 0.]
 [0. 1. 0.]
 [0. 1. 0.]
 [0. 1. 0.]
 [0. 1. 0.]
 [0. 1. 0.]
 [0. 1. 0.]
 [0. 1. 0.]
 [0. 1. 0.]
 [0. 1. 0.]
 [0. 1. 0.]]
Fold: 1  # train contains class 0 and 2, 0 is the first `pos_class_indices`
[[1. 0. 0.]
 [1. 0. 0.]
 [1. 0. 0.]
 [1. 0. 0.]
 [1. 0. 0.]
 [1. 0. 0.]
 [1. 0. 0.]
 [1. 0. 0.]
 [1. 0. 0.]
 [1. 0. 0.]
 [1. 0. 0.]
 [1. 0. 0.]]
Fold: 2  # train contains class 0 and 1, `0` is the first `pos_class_indices`
[[1. 0. 0.]
 [1. 0. 0.]
 [1. 0. 0.]
 [1. 0. 0.]
 [1. 0. 0.]
 [1. 0. 0.]
 [1. 0. 0.]
 [1. 0. 0.]
 [1. 0. 0.]
 [1. 0. 0.]
 [1. 0. 0.]
 [1. 0. 0.]]
```

A reasonable fix is to check when `CalibratedClassifierCV.classes_` is greater than `estimator.classes_` and output both proba and 1 - proba (assuming we can know which class the estimator deemed to be the positive class).

It does raise the question of whether we should warn when this happens..?


### Versions

```shell
Used main
```

COMMENT:
Oh yes indeed. I did not get so much feedback. I should put it in a dev discussion.

Labels: Bug
Follow-up after mean_poisson_deviance array API PR

As a follow-up for #29227.

The following command fails locally:
```
pytest -vl sklearn/metrics/tests/test_common.py -k 'api_regression_metric and mean_poisson_deviance'
```
see full error in https://github.com/scikit-learn/scikit-learn/pull/29227#issuecomment-2244729661

but setting `SCIPY_ARRAY_API=1` works:
```
SCIPY_ARRAY_API=1 pytest -vl sklearn/metrics/tests/test_common.py -k 'api_regression_metric and mean_poisson_deviance'
```

Honestly I think this is fair to say that as a newcomer to the array API work, my feeling is that it can break way too easily in very mysterious ways. There are probably a few unrelated reasons for this but I guess since at the last bi-weekly meetings we agreed that we should try to pay attention to this kind of breakages and improve the situation, here are a few comments on this #29277:
- I think we should have some tests with scipy>=1.14 and `SCIPY_ARRAY_API` unset. This seems something that could easily happen in practice and we should have tests that make sure that nothing breaks. It's not clear to me why only array-api-strict breaks and not pytorch example, maybe as the name implies array-api-strict is even stricter than other namespaces.
- I don't think this is acceptable to skip the test for scipy<1.14 since you know a reasonable user may use `array_api_dispatch=True` with scipy<1.14. To be fair it is not clear to me what happens in this case. I guess scipy will transform the array into a numpy array, and maybe the code afterwards has issues because some arrays are not in the same namespace? Anyway, we should make sure that the user code does not break. If this is really too hard to achieve the alternative is to raise an exception saying something like "to enable array API dispatch with this particular function you need scipy 1.14 and `SCIPY_ARRAY_API=1`".
- the warning about `SCIPY_ARRAY_API` seems too broad (as soon as `set_config(array_api_dispatch=True)` is called?). Could we not have warnings only in a few (at least for now) places where we know we call scipy functions?

_Originally posted by @lesteve in https://github.com/scikit-learn/scikit-learn/issues/29227#issuecomment-2244803503_

COMMENT:
We should probably add a check to our array API tests that raises an error/exception with a good message if the environment variable is not already set.

Labels: Array API
Follow-up after mean_poisson_deviance array API PR

As a follow-up for #29227.

The following command fails locally:
```
pytest -vl sklearn/metrics/tests/test_common.py -k 'api_regression_metric and mean_poisson_deviance'
```
see full error in https://github.com/scikit-learn/scikit-learn/pull/29227#issuecomment-2244729661

but setting `SCIPY_ARRAY_API=1` works:
```
SCIPY_ARRAY_API=1 pytest -vl sklearn/metrics/tests/test_common.py -k 'api_regression_metric and mean_poisson_deviance'
```

Honestly I think this is fair to say that as a newcomer to the array API work, my feeling is that it can break way too easily in very mysterious ways. There are probably a few unrelated reasons for this but I guess since at the last bi-weekly meetings we agreed that we should try to pay attention to this kind of breakages and improve the situation, here are a few comments on this #29277:
- I think we should have some tests with scipy>=1.14 and `SCIPY_ARRAY_API` unset. This seems something that could easily happen in practice and we should have tests that make sure that nothing breaks. It's not clear to me why only array-api-strict breaks and not pytorch example, maybe as the name implies array-api-strict is even stricter than other namespaces.
- I don't think this is acceptable to skip the test for scipy<1.14 since you know a reasonable user may use `array_api_dispatch=True` with scipy<1.14. To be fair it is not clear to me what happens in this case. I guess scipy will transform the array into a numpy array, and maybe the code afterwards has issues because some arrays are not in the same namespace? Anyway, we should make sure that the user code does not break. If this is really too hard to achieve the alternative is to raise an exception saying something like "to enable array API dispatch with this particular function you need scipy 1.14 and `SCIPY_ARRAY_API=1`".
- the warning about `SCIPY_ARRAY_API` seems too broad (as soon as `set_config(array_api_dispatch=True)` is called?). Could we not have warnings only in a few (at least for now) places where we know we call scipy functions?

_Originally posted by @lesteve in https://github.com/scikit-learn/scikit-learn/issues/29227#issuecomment-2244803503_

COMMENT:
I agree that the warning is essentially invisible. I knew it was somewhere in your screenshot but it took me two or three scans to actually find it :-/

Should we make the warning an error for tests?

Labels: Array API
Follow-up after mean_poisson_deviance array API PR

As a follow-up for #29227.

The following command fails locally:
```
pytest -vl sklearn/metrics/tests/test_common.py -k 'api_regression_metric and mean_poisson_deviance'
```
see full error in https://github.com/scikit-learn/scikit-learn/pull/29227#issuecomment-2244729661

but setting `SCIPY_ARRAY_API=1` works:
```
SCIPY_ARRAY_API=1 pytest -vl sklearn/metrics/tests/test_common.py -k 'api_regression_metric and mean_poisson_deviance'
```

Honestly I think this is fair to say that as a newcomer to the array API work, my feeling is that it can break way too easily in very mysterious ways. There are probably a few unrelated reasons for this but I guess since at the last bi-weekly meetings we agreed that we should try to pay attention to this kind of breakages and improve the situation, here are a few comments on this #29277:
- I think we should have some tests with scipy>=1.14 and `SCIPY_ARRAY_API` unset. This seems something that could easily happen in practice and we should have tests that make sure that nothing breaks. It's not clear to me why only array-api-strict breaks and not pytorch example, maybe as the name implies array-api-strict is even stricter than other namespaces.
- I don't think this is acceptable to skip the test for scipy<1.14 since you know a reasonable user may use `array_api_dispatch=True` with scipy<1.14. To be fair it is not clear to me what happens in this case. I guess scipy will transform the array into a numpy array, and maybe the code afterwards has issues because some arrays are not in the same namespace? Anyway, we should make sure that the user code does not break. If this is really too hard to achieve the alternative is to raise an exception saying something like "to enable array API dispatch with this particular function you need scipy 1.14 and `SCIPY_ARRAY_API=1`".
- the warning about `SCIPY_ARRAY_API` seems too broad (as soon as `set_config(array_api_dispatch=True)` is called?). Could we not have warnings only in a few (at least for now) places where we know we call scipy functions?

_Originally posted by @lesteve in https://github.com/scikit-learn/scikit-learn/issues/29227#issuecomment-2244803503_

COMMENT:
> Maybe it is an argument for skipping the mean_poisson_deviance test if scipy array API support is not enabled.

I am fine with this. This would mean catching `RuntimeError` in addition to `ImportError` in `sklearn.utils._testing` in #29571.

Labels: Array API
Follow-up after mean_poisson_deviance array API PR

As a follow-up for #29227.

The following command fails locally:
```
pytest -vl sklearn/metrics/tests/test_common.py -k 'api_regression_metric and mean_poisson_deviance'
```
see full error in https://github.com/scikit-learn/scikit-learn/pull/29227#issuecomment-2244729661

but setting `SCIPY_ARRAY_API=1` works:
```
SCIPY_ARRAY_API=1 pytest -vl sklearn/metrics/tests/test_common.py -k 'api_regression_metric and mean_poisson_deviance'
```

Honestly I think this is fair to say that as a newcomer to the array API work, my feeling is that it can break way too easily in very mysterious ways. There are probably a few unrelated reasons for this but I guess since at the last bi-weekly meetings we agreed that we should try to pay attention to this kind of breakages and improve the situation, here are a few comments on this #29277:
- I think we should have some tests with scipy>=1.14 and `SCIPY_ARRAY_API` unset. This seems something that could easily happen in practice and we should have tests that make sure that nothing breaks. It's not clear to me why only array-api-strict breaks and not pytorch example, maybe as the name implies array-api-strict is even stricter than other namespaces.
- I don't think this is acceptable to skip the test for scipy<1.14 since you know a reasonable user may use `array_api_dispatch=True` with scipy<1.14. To be fair it is not clear to me what happens in this case. I guess scipy will transform the array into a numpy array, and maybe the code afterwards has issues because some arrays are not in the same namespace? Anyway, we should make sure that the user code does not break. If this is really too hard to achieve the alternative is to raise an exception saying something like "to enable array API dispatch with this particular function you need scipy 1.14 and `SCIPY_ARRAY_API=1`".
- the warning about `SCIPY_ARRAY_API` seems too broad (as soon as `set_config(array_api_dispatch=True)` is called?). Could we not have warnings only in a few (at least for now) places where we know we call scipy functions?

_Originally posted by @lesteve in https://github.com/scikit-learn/scikit-learn/issues/29227#issuecomment-2244803503_

COMMENT:
Having to know to set the environment variable is annoying indeed. Some might suggest that `spin test` could take care of this for us! But I think this is also exactly what I think is an anti-pattern for tooling like spin (we'd be using it as a crutch to deal with things which are not done well).

I wonder if there is some Python magic (but not too magic) we can sprinkle onto scipy to enable array API support after having imported it?

If we skip all tests that require array API support in scipy, won't we end up skipping a lot of tests in the future? Basically, I am wondering what is going to be more annoying: having to setup things like the env variable or learning that your work breaks lots of tests only after opening a PR?

Labels: Array API
Follow-up after mean_poisson_deviance array API PR

As a follow-up for #29227.

The following command fails locally:
```
pytest -vl sklearn/metrics/tests/test_common.py -k 'api_regression_metric and mean_poisson_deviance'
```
see full error in https://github.com/scikit-learn/scikit-learn/pull/29227#issuecomment-2244729661

but setting `SCIPY_ARRAY_API=1` works:
```
SCIPY_ARRAY_API=1 pytest -vl sklearn/metrics/tests/test_common.py -k 'api_regression_metric and mean_poisson_deviance'
```

Honestly I think this is fair to say that as a newcomer to the array API work, my feeling is that it can break way too easily in very mysterious ways. There are probably a few unrelated reasons for this but I guess since at the last bi-weekly meetings we agreed that we should try to pay attention to this kind of breakages and improve the situation, here are a few comments on this #29277:
- I think we should have some tests with scipy>=1.14 and `SCIPY_ARRAY_API` unset. This seems something that could easily happen in practice and we should have tests that make sure that nothing breaks. It's not clear to me why only array-api-strict breaks and not pytorch example, maybe as the name implies array-api-strict is even stricter than other namespaces.
- I don't think this is acceptable to skip the test for scipy<1.14 since you know a reasonable user may use `array_api_dispatch=True` with scipy<1.14. To be fair it is not clear to me what happens in this case. I guess scipy will transform the array into a numpy array, and maybe the code afterwards has issues because some arrays are not in the same namespace? Anyway, we should make sure that the user code does not break. If this is really too hard to achieve the alternative is to raise an exception saying something like "to enable array API dispatch with this particular function you need scipy 1.14 and `SCIPY_ARRAY_API=1`".
- the warning about `SCIPY_ARRAY_API` seems too broad (as soon as `set_config(array_api_dispatch=True)` is called?). Could we not have warnings only in a few (at least for now) places where we know we call scipy functions?

_Originally posted by @lesteve in https://github.com/scikit-learn/scikit-learn/issues/29227#issuecomment-2244803503_

COMMENT:
I opened an issue upstream in scipy to ask for a programmatic way to do it: https://github.com/scipy/scipy/issues/21529.

Labels: Array API
Follow-up after mean_poisson_deviance array API PR

As a follow-up for #29227.

The following command fails locally:
```
pytest -vl sklearn/metrics/tests/test_common.py -k 'api_regression_metric and mean_poisson_deviance'
```
see full error in https://github.com/scikit-learn/scikit-learn/pull/29227#issuecomment-2244729661

but setting `SCIPY_ARRAY_API=1` works:
```
SCIPY_ARRAY_API=1 pytest -vl sklearn/metrics/tests/test_common.py -k 'api_regression_metric and mean_poisson_deviance'
```

Honestly I think this is fair to say that as a newcomer to the array API work, my feeling is that it can break way too easily in very mysterious ways. There are probably a few unrelated reasons for this but I guess since at the last bi-weekly meetings we agreed that we should try to pay attention to this kind of breakages and improve the situation, here are a few comments on this #29277:
- I think we should have some tests with scipy>=1.14 and `SCIPY_ARRAY_API` unset. This seems something that could easily happen in practice and we should have tests that make sure that nothing breaks. It's not clear to me why only array-api-strict breaks and not pytorch example, maybe as the name implies array-api-strict is even stricter than other namespaces.
- I don't think this is acceptable to skip the test for scipy<1.14 since you know a reasonable user may use `array_api_dispatch=True` with scipy<1.14. To be fair it is not clear to me what happens in this case. I guess scipy will transform the array into a numpy array, and maybe the code afterwards has issues because some arrays are not in the same namespace? Anyway, we should make sure that the user code does not break. If this is really too hard to achieve the alternative is to raise an exception saying something like "to enable array API dispatch with this particular function you need scipy 1.14 and `SCIPY_ARRAY_API=1`".
- the warning about `SCIPY_ARRAY_API` seems too broad (as soon as `set_config(array_api_dispatch=True)` is called?). Could we not have warnings only in a few (at least for now) places where we know we call scipy functions?

_Originally posted by @lesteve in https://github.com/scikit-learn/scikit-learn/issues/29227#issuecomment-2244803503_

COMMENT:
Closing this because in #29814.

Labels: Array API
GridSearchCV support for 'precomputed' kernel not documented

### Describe the issue linked to the documentation

GridSearchCV seems to work even with a precomputed kernel but there is nothing about it in the documentation. Is there a reason for this or did it just go unnoticed?

### Suggest a potential alternative/fix

Add documentation to the fit function of the class that it accepts a precomputed square matrix of shape (n_samples, n_samples).

COMMENT:
Hi @stepan-srsen,

indeed, it works with an estimator accepting a precomputed distance matrix, kernel, ... We can add something in the description of X here https://github.com/scikit-learn/scikit-learn/blob/70fdc843a4b8182d97a3508c1a426acc5e87e980/sklearn/model_selection/_search.py#L896-L898

Would you be interested in making a PR ?

Labels: Documentation