Talip7/scikit-learn-issues-multilabel huggingface dataset:
DOC Investigate scipy-doctest for more convenient doctests

I learned about [scipy-doctest](https://github.com/scipy/scipy_doctest) recent release in the [Scientific Python Discourse announcement](https://discuss.scientific-python.org/t/ann-scipy-doctest-package/1181). Apparently, scipy-doctest has been used internally in numpy and scipy for doctests for some time. In particular it allows floating point comparisons.

After a bit of work from us setting everything up, it would allow to have a few sprint / first good issues.

There is quite a few places where we used the doctest ellipsis, the quick and dirty following regexp finds 595 lines:
```
git grep -P '\d+\.\.\.' | wc -l
```

If you are not sure what I am talking about, this is the `...` for doctest in rst for docstrings e.g. the last line of this snippet:
```py
>>> from sklearn import svm, datasets
>>> from sklearn.model_selection import cross_val_score
>>> X, y = datasets.load_iris(return_X_y=True)
>>> clf = svm.SVC(random_state=0)
>>> cross_val_score(clf, X, y, cv=5, scoring='recall_macro')
array([0.96..., 0.96..., 0.96..., 0.93..., 1.        ])
```

An example of a doctest with a spurious failure recently: https://github.com/scikit-learn/scikit-learn/pull/29140#issuecomment-2139904739

If you are wondering about the difference to [pytest-doctestplus](https://github.com/scientific-python/pytest-doctestplus) look at [this](https://github.com/scipy/scipy_doctest?tab=readme-ov-file#prior-art-and-related-work). This does seem a bit unfortunate to have `scipy/scipy_doctest` and `scientific-python/pytest-doctestplus` but oh well (full disclosure I did not have time to look into the history) ...

COMMENT:
> there is a tweakable default rtol (default is 1e-3 IIRC)

The default atol and rtol are global, but if there's a strong need to make it, for instance, deduce the number of sig figs for each doctest from its `want` (in the example above, deduce atol=5e-3), scipy-doctest may consider an enhancement.

Labels: Documentation, Enhancement, RFC
DOC Investigate scipy-doctest for more convenient doctests

I learned about [scipy-doctest](https://github.com/scipy/scipy_doctest) recent release in the [Scientific Python Discourse announcement](https://discuss.scientific-python.org/t/ann-scipy-doctest-package/1181). Apparently, scipy-doctest has been used internally in numpy and scipy for doctests for some time. In particular it allows floating point comparisons.

After a bit of work from us setting everything up, it would allow to have a few sprint / first good issues.

There is quite a few places where we used the doctest ellipsis, the quick and dirty following regexp finds 595 lines:
```
git grep -P '\d+\.\.\.' | wc -l
```

If you are not sure what I am talking about, this is the `...` for doctest in rst for docstrings e.g. the last line of this snippet:
```py
>>> from sklearn import svm, datasets
>>> from sklearn.model_selection import cross_val_score
>>> X, y = datasets.load_iris(return_X_y=True)
>>> clf = svm.SVC(random_state=0)
>>> cross_val_score(clf, X, y, cv=5, scoring='recall_macro')
array([0.96..., 0.96..., 0.96..., 0.93..., 1.        ])
```

An example of a doctest with a spurious failure recently: https://github.com/scikit-learn/scikit-learn/pull/29140#issuecomment-2139904739

If you are wondering about the difference to [pytest-doctestplus](https://github.com/scientific-python/pytest-doctestplus) look at [this](https://github.com/scipy/scipy_doctest?tab=readme-ov-file#prior-art-and-related-work). This does seem a bit unfortunate to have `scipy/scipy_doctest` and `scientific-python/pytest-doctestplus` but oh well (full disclosure I did not have time to look into the history) ...

COMMENT:
> The default atol and rtol are global, but if there's a strong need to make it, for instance, deduce the number of sig figs for each doctest from its want (in the example above, deduce atol=5e-3), scipy-doctest may consider an enhancement.

Thanks for your continuous inputs on this! I don't think there is a strong need right now but it's good to know you would be open to the feature.

My current guess is that either @ogrisel missed that using only a few digits was a builtin feature scipy-doctest feature [^1], or I misunderstood him, which you know is also a possibility.

[^1]: even if he seems like a infinite source of knowledge and accuracy, @ogrisel [^3] is human after all, it can happen [^2] that he misses something :wink:

[^2]: very rarely though

[^3]: I pinged @grisel instead of @ogrisel, sorry @grisel my bad, feel free to unsubscribe after having been summoned from the void

Complete side-comment: wow you can do nested footnotes in Github :exploding_head:, this is going to improve my productivity a lot :upside_down_face:.

Labels: Documentation, Enhancement, RFC
TunedThreasholdClassifierCV failing inside a SearchCV object

I changed the existing example slightly, to put the estimator inside the SearchCV instead of tuning after the search. Here's the reproducer:

```py
# %%
from sklearn.datasets import fetch_openml

# %%
credit_card = fetch_openml(data_id=1597, as_frame=True, parser="pandas")
credit_card.frame.info()

# %%
columns_to_drop = ["Class"]
data = credit_card.frame.drop(columns=columns_to_drop)
target = credit_card.frame["Class"].astype(int)

# %%
def business_metric(y_true, y_pred, amount):
    mask_true_positive = (y_true == 1) & (y_pred == 1)
    mask_true_negative = (y_true == 0) & (y_pred == 0)
    mask_false_positive = (y_true == 0) & (y_pred == 1)
    mask_false_negative = (y_true == 1) & (y_pred == 0)
    fraudulent_refuse = (mask_true_positive.sum() * 50) + amount[
        mask_true_positive
    ].sum()
    fraudulent_accept = -amount[mask_false_negative].sum()
    legitimate_refuse = mask_false_positive.sum() * -5
    legitimate_accept = (amount[mask_true_negative] * 0.02).sum()
    return fraudulent_refuse + fraudulent_accept + legitimate_refuse + legitimate_accept


# %%
import sklearn
from sklearn.metrics import make_scorer

sklearn.set_config(enable_metadata_routing=True)
business_scorer = make_scorer(business_metric).set_score_request(amount=True)

# %%
amount = credit_card.frame["Amount"].to_numpy()

# %%
from sklearn.model_selection import train_test_split

data_train, data_test, target_train, target_test, amount_train, amount_test = (
    train_test_split(
        data, target, amount, stratify=target, test_size=0.5, random_state=42
    )
)

# %%
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import GridSearchCV
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import TunedThresholdClassifierCV

logistic_regression = make_pipeline(StandardScaler(), LogisticRegression())

tuned_model = TunedThresholdClassifierCV(
    estimator=logistic_regression,
    scoring=business_scorer,
    thresholds=100,
    n_jobs=2,
)

param_grid = {"estimator__logisticregression__C": np.logspace(-6, 6, 13)}
model = GridSearchCV(tuned_model, param_grid, scoring=business_scorer).fit(
    data_train, target_train, amount=amount_train
)
```

It gives: 

```py
AttributeError: 'TunedThresholdClassifierCV' object has no attribute '_response_method'. Did you mean: 'response_method'?
```

This solves the issue temporarily:

```py
tuned_model._response_method = "predict_proba"
```

cc @glemaitre @jeremiedbb 

I'd argue this needs to be fixed for the release.

COMMENT:
I open #29021 where I replaced setting a private `_response_method` in `fit` by a private method `_get_response_method` that can be called before `fit` to handle the metadata routing.

Labels: Bug, Blocker
Using decision boundary display to plot the relationship between any 2 features if model is fitted to more than 2 features

### Describe the workflow you want to enable

Currently, it seems like it is not possible to pass in a model that has been fitted to more than 2 features to the DecisionBoundaryDisplay.from_estimator method. Is it possible to allow that while only passing in the 2 features you are interested in looking at the relationship for?

### Describe your proposed solution

Allowing users to use [decision boundary display ](https://scikit-learn.org/stable/modules/generated/sklearn.inspection.DecisionBoundaryDisplay.html#sklearn.inspection.DecisionBoundaryDisplay.from_estimator) to plot the relationship between any 2 features in a model that is fitted for more than 2 features.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
Is the 2D partial dependence not better suited for your analysis?

Labels: New Feature, Needs Decision
Using decision boundary display to plot the relationship between any 2 features if model is fitted to more than 2 features

### Describe the workflow you want to enable

Currently, it seems like it is not possible to pass in a model that has been fitted to more than 2 features to the DecisionBoundaryDisplay.from_estimator method. Is it possible to allow that while only passing in the 2 features you are interested in looking at the relationship for?

### Describe your proposed solution

Allowing users to use [decision boundary display ](https://scikit-learn.org/stable/modules/generated/sklearn.inspection.DecisionBoundaryDisplay.html#sklearn.inspection.DecisionBoundaryDisplay.from_estimator) to plot the relationship between any 2 features in a model that is fitted for more than 2 features.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
Is this PR still needed ? I would like to contribute !

Also, in practice, couldn't we get an array of indices from the user corresponding to which feature we want to visualize (max 2) and set its default value to [0, 1] and set all other features to 0 before sending it to the estimator ?

A second parameter `features_default_values` could also be added to specify the default value of a feature. This would allow the user to control which plane in the feature space is seen during visualization.

Labels: New Feature, Needs Decision
Using decision boundary display to plot the relationship between any 2 features if model is fitted to more than 2 features

### Describe the workflow you want to enable

Currently, it seems like it is not possible to pass in a model that has been fitted to more than 2 features to the DecisionBoundaryDisplay.from_estimator method. Is it possible to allow that while only passing in the 2 features you are interested in looking at the relationship for?

### Describe your proposed solution

Allowing users to use [decision boundary display ](https://scikit-learn.org/stable/modules/generated/sklearn.inspection.DecisionBoundaryDisplay.html#sklearn.inspection.DecisionBoundaryDisplay.from_estimator) to plot the relationship between any 2 features in a model that is fitted for more than 2 features.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
Not clear yet to me if this feature is needed or not.

Labels: New Feature, Needs Decision
Using decision boundary display to plot the relationship between any 2 features if model is fitted to more than 2 features

### Describe the workflow you want to enable

Currently, it seems like it is not possible to pass in a model that has been fitted to more than 2 features to the DecisionBoundaryDisplay.from_estimator method. Is it possible to allow that while only passing in the 2 features you are interested in looking at the relationship for?

### Describe your proposed solution

Allowing users to use [decision boundary display ](https://scikit-learn.org/stable/modules/generated/sklearn.inspection.DecisionBoundaryDisplay.html#sklearn.inspection.DecisionBoundaryDisplay.from_estimator) to plot the relationship between any 2 features in a model that is fitted for more than 2 features.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
Hi, having this feature could be useful in my opinion, what @GM07 suggested sounds like a possible solution.

Labels: New Feature, Needs Decision
Using decision boundary display to plot the relationship between any 2 features if model is fitted to more than 2 features

### Describe the workflow you want to enable

Currently, it seems like it is not possible to pass in a model that has been fitted to more than 2 features to the DecisionBoundaryDisplay.from_estimator method. Is it possible to allow that while only passing in the 2 features you are interested in looking at the relationship for?

### Describe your proposed solution

Allowing users to use [decision boundary display ](https://scikit-learn.org/stable/modules/generated/sklearn.inspection.DecisionBoundaryDisplay.html#sklearn.inspection.DecisionBoundaryDisplay.from_estimator) to plot the relationship between any 2 features in a model that is fitted for more than 2 features.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
@ETTAN93 As mentioned in my previous comment, would the partial dependence a better tools for the inspection that you try to have: https://scikit-learn.org/stable/modules/partial_dependence.html

Labels: New Feature, Needs Decision
MultiOutputClassifier does not rely on estimator to provide pairwise tag

### Describe the bug

I use the `MultiOutputClassifier` function to make `SVC` multilabel. 

Then, if I use the linear or rbf kernel the cross_validation function works perfectly fine.

However, when I use `SVC` with precomputed kernel is having an `ValueError: Precomputed matrix must be a square matrix`. 

### Steps/Code to Reproduce

```python
import numpy as np
from sklearn.svm import SVC
from sklearn.metrics import pairwise_distances
from sklearn.multioutput import MultiOutputClassifier
from sklearn.model_selection import cross_val_score, cross_validate

svm = SVC(kernel='precomputed', C=100, random_state=42)
multilabel_classifier = MultiOutputClassifier(svm, n_jobs=-1)

X = np.random.rand(1000, 1000)
y = np.random.randint(0, 2, size=(1000, 6))

kernel_eucl = pairwise_distances(X, metric='euclidean')

cross_validate(
    multilabel_classifier, kernel_eucl, y, cv=10, scoring='f1_weighted', n_jobs=-1
)
```

### Expected Results

An weighted f1-score.

### Actual Results

```pytb
ValueError: 
All the 10 fits failed.
It is very likely that your model is misconfigured.
You can try to debug the error by setting error_score='raise'.

Below are more details about the failures:
--------------------------------------------------------------------------------
10 fits failed with the following error:

 File "C:\Users\bscuser\anaconda3\Lib\site-packages\sklearn\svm\_base.py", line 217, in fit
    raise ValueError(
ValueError: Precomputed matrix must be a square matrix. Input is a 900x1000 matrix.
```

### Versions

```shell
System:
    python: 3.11.7 | packaged by Anaconda, Inc. | (main, Dec 15 2023, 18:05:47) [MSC v.1916 64 bit (AMD64)]
executable: C:\Users\bscuser\anaconda3\python.exe
   machine: Windows-10-10.0.19045-SP0

Python dependencies:
      sklearn: 1.2.2
          pip: 23.3.1
   setuptools: 68.2.2
        numpy: 1.26.4
        scipy: 1.11.4
       Cython: None
       pandas: 2.1.4
   matplotlib: 3.8.0
       joblib: 1.2.0
threadpoolctl: 2.2.0

Built with OpenMP: True

threadpoolctl info:
       filepath: C:\Users\bscuser\anaconda3\Library\bin\mkl_rt.2.dll
         prefix: mkl_rt
       user_api: blas
   internal_api: mkl
        version: 2023.1-Product
    num_threads: 4
threading_layer: intel

       filepath: C:\Users\bscuser\anaconda3\vcomp140.dll
         prefix: vcomp
       user_api: openmp
   internal_api: openmp
        version: None
    num_threads: 4

       filepath: C:\Users\bscuser\anaconda3\Library\bin\libiomp5md.dll
         prefix: libiomp
       user_api: openmp
   internal_api: openmp
        version: None
    num_threads: 4
```

COMMENT:
Actually, we should probably come with a common test to be sure that all meta-estimators are relying on the underlying estimator to set this tag.

Labels: Bug
MultiOutputClassifier does not rely on estimator to provide pairwise tag

### Describe the bug

I use the `MultiOutputClassifier` function to make `SVC` multilabel. 

Then, if I use the linear or rbf kernel the cross_validation function works perfectly fine.

However, when I use `SVC` with precomputed kernel is having an `ValueError: Precomputed matrix must be a square matrix`. 

### Steps/Code to Reproduce

```python
import numpy as np
from sklearn.svm import SVC
from sklearn.metrics import pairwise_distances
from sklearn.multioutput import MultiOutputClassifier
from sklearn.model_selection import cross_val_score, cross_validate

svm = SVC(kernel='precomputed', C=100, random_state=42)
multilabel_classifier = MultiOutputClassifier(svm, n_jobs=-1)

X = np.random.rand(1000, 1000)
y = np.random.randint(0, 2, size=(1000, 6))

kernel_eucl = pairwise_distances(X, metric='euclidean')

cross_validate(
    multilabel_classifier, kernel_eucl, y, cv=10, scoring='f1_weighted', n_jobs=-1
)
```

### Expected Results

An weighted f1-score.

### Actual Results

```pytb
ValueError: 
All the 10 fits failed.
It is very likely that your model is misconfigured.
You can try to debug the error by setting error_score='raise'.

Below are more details about the failures:
--------------------------------------------------------------------------------
10 fits failed with the following error:

 File "C:\Users\bscuser\anaconda3\Lib\site-packages\sklearn\svm\_base.py", line 217, in fit
    raise ValueError(
ValueError: Precomputed matrix must be a square matrix. Input is a 900x1000 matrix.
```

### Versions

```shell
System:
    python: 3.11.7 | packaged by Anaconda, Inc. | (main, Dec 15 2023, 18:05:47) [MSC v.1916 64 bit (AMD64)]
executable: C:\Users\bscuser\anaconda3\python.exe
   machine: Windows-10-10.0.19045-SP0

Python dependencies:
      sklearn: 1.2.2
          pip: 23.3.1
   setuptools: 68.2.2
        numpy: 1.26.4
        scipy: 1.11.4
       Cython: None
       pandas: 2.1.4
   matplotlib: 3.8.0
       joblib: 1.2.0
threadpoolctl: 2.2.0

Built with OpenMP: True

threadpoolctl info:
       filepath: C:\Users\bscuser\anaconda3\Library\bin\mkl_rt.2.dll
         prefix: mkl_rt
       user_api: blas
   internal_api: mkl
        version: 2023.1-Product
    num_threads: 4
threading_layer: intel

       filepath: C:\Users\bscuser\anaconda3\vcomp140.dll
         prefix: vcomp
       user_api: openmp
   internal_api: openmp
        version: None
    num_threads: 4

       filepath: C:\Users\bscuser\anaconda3\Library\bin\libiomp5md.dll
         prefix: libiomp
       user_api: openmp
   internal_api: openmp
        version: None
    num_threads: 4
```

COMMENT:
Please don't provide an LLM suggestion that does not answer to the question. Here we need to work a common test and not just a non-regression specifically for the estimator.

Labels: Bug
MultiOutputClassifier does not rely on estimator to provide pairwise tag

### Describe the bug

I use the `MultiOutputClassifier` function to make `SVC` multilabel. 

Then, if I use the linear or rbf kernel the cross_validation function works perfectly fine.

However, when I use `SVC` with precomputed kernel is having an `ValueError: Precomputed matrix must be a square matrix`. 

### Steps/Code to Reproduce

```python
import numpy as np
from sklearn.svm import SVC
from sklearn.metrics import pairwise_distances
from sklearn.multioutput import MultiOutputClassifier
from sklearn.model_selection import cross_val_score, cross_validate

svm = SVC(kernel='precomputed', C=100, random_state=42)
multilabel_classifier = MultiOutputClassifier(svm, n_jobs=-1)

X = np.random.rand(1000, 1000)
y = np.random.randint(0, 2, size=(1000, 6))

kernel_eucl = pairwise_distances(X, metric='euclidean')

cross_validate(
    multilabel_classifier, kernel_eucl, y, cv=10, scoring='f1_weighted', n_jobs=-1
)
```

### Expected Results

An weighted f1-score.

### Actual Results

```pytb
ValueError: 
All the 10 fits failed.
It is very likely that your model is misconfigured.
You can try to debug the error by setting error_score='raise'.

Below are more details about the failures:
--------------------------------------------------------------------------------
10 fits failed with the following error:

 File "C:\Users\bscuser\anaconda3\Lib\site-packages\sklearn\svm\_base.py", line 217, in fit
    raise ValueError(
ValueError: Precomputed matrix must be a square matrix. Input is a 900x1000 matrix.
```

### Versions

```shell
System:
    python: 3.11.7 | packaged by Anaconda, Inc. | (main, Dec 15 2023, 18:05:47) [MSC v.1916 64 bit (AMD64)]
executable: C:\Users\bscuser\anaconda3\python.exe
   machine: Windows-10-10.0.19045-SP0

Python dependencies:
      sklearn: 1.2.2
          pip: 23.3.1
   setuptools: 68.2.2
        numpy: 1.26.4
        scipy: 1.11.4
       Cython: None
       pandas: 2.1.4
   matplotlib: 3.8.0
       joblib: 1.2.0
threadpoolctl: 2.2.0

Built with OpenMP: True

threadpoolctl info:
       filepath: C:\Users\bscuser\anaconda3\Library\bin\mkl_rt.2.dll
         prefix: mkl_rt
       user_api: blas
   internal_api: mkl
        version: 2023.1-Product
    num_threads: 4
threading_layer: intel

       filepath: C:\Users\bscuser\anaconda3\vcomp140.dll
         prefix: vcomp
       user_api: openmp
   internal_api: openmp
        version: None
    num_threads: 4

       filepath: C:\Users\bscuser\anaconda3\Library\bin\libiomp5md.dll
         prefix: libiomp
       user_api: openmp
   internal_api: openmp
        version: None
    num_threads: 4
```

COMMENT:
> but on a first glance this is well contained problem am I correct @glemaitre ?

yes indeed.

Labels: Bug
Incorrect documented output shape for `predict` method of linear models when `n_targets` > 1

### Describe the issue linked to the documentation

For some classes under `sklearn.linear_model` such as `LinearRegression`, `Ridge`, `RidgeCV`, and a bunch of others, the documentation for the `predict` method states that it returns
```
C : array, shape (n_samples,)
            Returns predicted values.
```
However, this is incorrect when the model is fitted with `y` with shape `(n_samples, n_targets)`, i.e. `n_targets` > 1, in which case the returned array should have shape `(n_samples, n_targets)`.

### Suggest a potential alternative/fix

The documentation for the return value of the `predict` method should be
```
C : array, shape (n_samples,) or (n_samples, n_targets)
            Returns predicted values.
```

COMMENT:
Indeed, it should be fixed. Thanks for noticing @hchau630. Do you wish to make a pull-request to address this problem?

Labels: Documentation
Incorrect documented output shape for `predict` method of linear models when `n_targets` > 1

### Describe the issue linked to the documentation

For some classes under `sklearn.linear_model` such as `LinearRegression`, `Ridge`, `RidgeCV`, and a bunch of others, the documentation for the `predict` method states that it returns
```
C : array, shape (n_samples,)
            Returns predicted values.
```
However, this is incorrect when the model is fitted with `y` with shape `(n_samples, n_targets)`, i.e. `n_targets` > 1, in which case the returned array should have shape `(n_samples, n_targets)`.

### Suggest a potential alternative/fix

The documentation for the return value of the `predict` method should be
```
C : array, shape (n_samples,) or (n_samples, n_targets)
            Returns predicted values.
```

COMMENT:
Sure, should be a quick fix.

Labels: Documentation