Talip7/scikit-learn-issues-multilabel huggingface dataset:
Allow for multiple scoring metrics in `RFECV`

### Workflow

In its current state, `RFECV` only allows for a single scoring metric. In my opinion, calculating multiple scores on each model using *k <= K* features would be extremely valuable.

For example, if I wanted to study how the precision and recall metrics of a binary classifier evolve as I feed less and less features to a model, I would have to run `RFECV` twice: one with `scoring='precision'` and another with `scoring='recall'`.
This is inefficient, as it implies running RFECV twice instead of once.

The `cv_results_` attribute of `GridSearchCV` returns one rank per metric used to evaluate each combination of hyperparameters. Replicating this behavior in `RFECV` would be extremely helpful.

### Proposed solution

#### Notation
- *K* is the number of folds used for cross-validation.
- *P* is the total number of features available.
- *p* is the number of features tried at each step. That is, an integer such that *`min_features_to_select` <= p <= P*.
- *m* is one of *M* performance metrics passed by the user (e.g., 'precision').

#### Solution
User can pass a list of strings representing *M* [predefined scoring metrics](https://scikit-learn.org/stable/modules/model_evaluation.html#common-cases-predefined-values) and at each step, the algorithm stores the performance metric of the *k* models trained with *p <= P* features.

The `cv_results_` attribute of the resulting `RFECV` would now include the following keys for each metric *m* and fold *k*:
- `'split{k}_test_{m}'`
- `mean_test_{m}`
- `'std_test_{m}'`
- `'rank_test_{m}'`

#### Example
```python
rfecv = RFECV(
    estimator=clf,  # Some classifier instance
    step=1,
    min_features_to_select=1,
    cv=10,
    scoring=['precision', 'recall', 'f1', 'roc_auc', 'accuracy']
)
```

##### Considerations
It is likely that `rank_test_{m1}` will differ from `rank_test_{m2}` for any pair of performance metrics *m1* and *m2*. Hence, adding this feature will no longer allow RFECV to automatically pick the best number of features, as the rankings can differ from one metric to another. This part of the workflow would be up to the user.

### Describe alternatives you've considered, if relevant

Running `RFECV` as many times as there are metrics.

### Additional context

I asked [this question](https://stackoverflow.com/questions/78420559/performing-recursive-feature-elimination-with-multiple-performance-metrics) on StackOverflow and the community agrees that the most viable way to do this is to run one RFECV per performance metric I need to evaluate.

COMMENT:
Did we improve a bit the situation by merging: https://github.com/scikit-learn/scikit-learn/pull/28360

Labels: New Feature
Allow for multiple scoring metrics in `RFECV`

### Workflow

In its current state, `RFECV` only allows for a single scoring metric. In my opinion, calculating multiple scores on each model using *k <= K* features would be extremely valuable.

For example, if I wanted to study how the precision and recall metrics of a binary classifier evolve as I feed less and less features to a model, I would have to run `RFECV` twice: one with `scoring='precision'` and another with `scoring='recall'`.
This is inefficient, as it implies running RFECV twice instead of once.

The `cv_results_` attribute of `GridSearchCV` returns one rank per metric used to evaluate each combination of hyperparameters. Replicating this behavior in `RFECV` would be extremely helpful.

### Proposed solution

#### Notation
- *K* is the number of folds used for cross-validation.
- *P* is the total number of features available.
- *p* is the number of features tried at each step. That is, an integer such that *`min_features_to_select` <= p <= P*.
- *m* is one of *M* performance metrics passed by the user (e.g., 'precision').

#### Solution
User can pass a list of strings representing *M* [predefined scoring metrics](https://scikit-learn.org/stable/modules/model_evaluation.html#common-cases-predefined-values) and at each step, the algorithm stores the performance metric of the *k* models trained with *p <= P* features.

The `cv_results_` attribute of the resulting `RFECV` would now include the following keys for each metric *m* and fold *k*:
- `'split{k}_test_{m}'`
- `mean_test_{m}`
- `'std_test_{m}'`
- `'rank_test_{m}'`

#### Example
```python
rfecv = RFECV(
    estimator=clf,  # Some classifier instance
    step=1,
    min_features_to_select=1,
    cv=10,
    scoring=['precision', 'recall', 'f1', 'roc_auc', 'accuracy']
)
```

##### Considerations
It is likely that `rank_test_{m1}` will differ from `rank_test_{m2}` for any pair of performance metrics *m1* and *m2*. Hence, adding this feature will no longer allow RFECV to automatically pick the best number of features, as the rankings can differ from one metric to another. This part of the workflow would be up to the user.

### Describe alternatives you've considered, if relevant

Running `RFECV` as many times as there are metrics.

### Additional context

I asked [this question](https://stackoverflow.com/questions/78420559/performing-recursive-feature-elimination-with-multiple-performance-metrics) on StackOverflow and the community agrees that the most viable way to do this is to run one RFECV per performance metric I need to evaluate.

COMMENT:
With #28360 the situation is much improved, but we need to be able to understand multiple metrics the same way as `*SeachCV` here. For now we assume a single value as metric value.

This is related to #28944 for me too, since we're choosing sth based on certain metrics values while understanding multiple metrics.

Labels: New Feature
DOC D2_log_loss_score is in wrong section

``D2_log_loss_score`` was added in https://github.com/scikit-learn/scikit-learn/pull/28351, but the function is documented in regression metrics with other D2 scores, while this one is a classification metric.

Ping @OmarManzoor for a follow-up PR maybe ?

COMMENT:
Sure thank you for pointing it out @jeremiedbb.

Labels: Documentation
BUG internal indexing tools trigger error with pandas < 2.0.0

[#28375](https://github.com/scikit-learn/scikit-learn/pull/28375#issuecomment-2088926826) triggers errors for pandas < 2.0.0, despite just using scikit-learn internal functionalities.

As documented in https://scikit-learn.org/dev/install.html, we have pandas >= 1.1.3.

COMMENT:
I have the feeling that we should not close this straight away as discussed in https://github.com/scikit-learn/scikit-learn/pull/28375#discussion_r1589235200.

Labels: Bug, Pandas compatibility
BUG internal indexing tools trigger error with pandas < 2.0.0

[#28375](https://github.com/scikit-learn/scikit-learn/pull/28375#issuecomment-2088926826) triggers errors for pandas < 2.0.0, despite just using scikit-learn internal functionalities.

As documented in https://scikit-learn.org/dev/install.html, we have pandas >= 1.1.3.

COMMENT:
@ogrisel could you explain what kind of bug the non-regression test would test?

Labels: Bug, Pandas compatibility
Update FAQ about pandas

Our FAQ is not up to date when it comes to pandas,
> [Why does scikit-learn not directly work with, for example, ](https://scikit-learn.org/1.4/faq.html#id13)[pandas.DataFrame](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.html#pandas.DataFrame)?
>
>The homogeneous NumPy and SciPy data objects currently expected are most efficient to process for most operations. Extensive work would also be needed to support Pandas categorical types. Restricting input to homogeneous types therefore reduces maintenance cost and encourages usage of efficient data structures.
>
> Note however that [ColumnTransformer](https://scikit-learn.org/1.4/modules/generated/sklearn.compose.ColumnTransformer.html#sklearn.compose.ColumnTransformer) makes it convenient to handle heterogeneous pandas dataframes by mapping homogeneous subsets of dataframe columns selected by name or dtype to dedicated scikit-learn transformers. Therefore [ColumnTransformer](https://scikit-learn.org/1.4/modules/generated/sklearn.compose.ColumnTransformer.html#sklearn.compose.ColumnTransformer) are often used in the first step of scikit-learn pipelines when dealing with heterogeneous dataframes (see [Pipeline: chaining estimators](https://scikit-learn.org/1.4/modules/compose.html#pipeline) for more details).
>
> See also [Column Transformer with Mixed Types](https://scikit-learn.org/1.4/auto_examples/compose/plot_column_transformer_mixed_types.html#sphx-glr-auto-examples-compose-plot-column-transformer-mixed-types-py) for an example of working with heterogeneous (e.g. categorical and numeric) data.

As of version 1.2 we have pandas-in-pandas-out, see https://scikit-learn.org/1.4/auto_examples/release_highlights/plot_release_highlights_1_2_0.html#pandas-output-with-set-output-api according to [SLEP018](https://scikit-learn-enhancement-proposals.readthedocs.io/en/latest/slep018/proposal.html).

Also, https://scikit-learn.org/dev/install.html mentions pandas purpose:
> benchmark, docs, examples, tests

We should document this better.

Relates to #10453.

COMMENT:
+1 for updating this changelog entry to mention that it's possible to output dataframes with the `set_output` API with a link to https://scikit-learn.org/dev/auto_examples/miscellaneous/plot_set_output.html for more details.

Labels: Documentation, Moderate, help wanted, Pandas compatibility
Update FAQ about pandas

Our FAQ is not up to date when it comes to pandas,
> [Why does scikit-learn not directly work with, for example, ](https://scikit-learn.org/1.4/faq.html#id13)[pandas.DataFrame](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.html#pandas.DataFrame)?
>
>The homogeneous NumPy and SciPy data objects currently expected are most efficient to process for most operations. Extensive work would also be needed to support Pandas categorical types. Restricting input to homogeneous types therefore reduces maintenance cost and encourages usage of efficient data structures.
>
> Note however that [ColumnTransformer](https://scikit-learn.org/1.4/modules/generated/sklearn.compose.ColumnTransformer.html#sklearn.compose.ColumnTransformer) makes it convenient to handle heterogeneous pandas dataframes by mapping homogeneous subsets of dataframe columns selected by name or dtype to dedicated scikit-learn transformers. Therefore [ColumnTransformer](https://scikit-learn.org/1.4/modules/generated/sklearn.compose.ColumnTransformer.html#sklearn.compose.ColumnTransformer) are often used in the first step of scikit-learn pipelines when dealing with heterogeneous dataframes (see [Pipeline: chaining estimators](https://scikit-learn.org/1.4/modules/compose.html#pipeline) for more details).
>
> See also [Column Transformer with Mixed Types](https://scikit-learn.org/1.4/auto_examples/compose/plot_column_transformer_mixed_types.html#sphx-glr-auto-examples-compose-plot-column-transformer-mixed-types-py) for an example of working with heterogeneous (e.g. categorical and numeric) data.

As of version 1.2 we have pandas-in-pandas-out, see https://scikit-learn.org/1.4/auto_examples/release_highlights/plot_release_highlights_1_2_0.html#pandas-output-with-set-output-api according to [SLEP018](https://scikit-learn-enhancement-proposals.readthedocs.io/en/latest/slep018/proposal.html).

Also, https://scikit-learn.org/dev/install.html mentions pandas purpose:
> benchmark, docs, examples, tests

We should document this better.

Relates to #10453.

COMMENT:
Working on this

Labels: Documentation, Moderate, help wanted, Pandas compatibility
Performance Degradation in MeanShift When Data Has No Variance

### Describe the bug

When data provided to `MeanShift` consists of values with no variance (for example, two clusters of 0 and 1), the performance becomes extremely slow.

I am unsure whether this is a bug or an unavoidable aspect of the algorithm's design. Any clarification would be appreciated.


### Steps/Code to Reproduce

```python
import numpy as np
from sklearn.cluster import MeanShift

x = np.concatenate([np.ones(100), np.zeros(100)])
_ = MeanShift().fit_predict(x.reshape(-1, 1)) # Slow

rng = np.random.default_rng(1)
x = np.concatenate([rng.uniform(0.0, 0.001, 100), rng.uniform(0.999, 1.0, 100)])
_ = MeanShift().fit_predict(x.reshape(-1, 1)) # Fast
```

Link to Google Colab: https://colab.research.google.com/drive/1hlqhtaD8T40hwcleUKoI4uzrW1XtSRA4?usp=sharing#scrollTo=6g5qI45KUW_i

### Expected Results

When data provided to `MeanShift` consists of values with no variance, the performance becomes as fast as when handling data with variance.


### Actual Results

If `MeanShift` receives a 1D array with no variance, the computation is significantly slower.

```python
import numpy as np
from sklearn.cluster import MeanShift

# Example where input has no variance
x = np.concatenate([np.ones(100), np.zeros(100)])
%timeit _ = MeanShift().fit_predict(x.reshape(-1, 1))
# Output: 24.9 s ± 340 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
```

Below is a control example, where the input has some variance:

```python
import numpy as np
from sklearn.cluster import MeanShift

# Example with minimal variance
rng = np.random.default_rng(1)
x = np.concatenate([rng.uniform(0.0, 0.001, 100), rng.uniform(0.999, 1.0, 100)])
%timeit _ = MeanShift().fit_predict(x.reshape(-1, 1))
# Output: 665 ms ± 101 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

```



### Versions

```shell
1.2.2
```

COMMENT:
The first snippet lead to `n_iter_ == max_iter` that is 300 while the second is 10.
So I assume that we don't trigger the stopping criteria.

Labels: Bug
Performance Degradation in MeanShift When Data Has No Variance

### Describe the bug

When data provided to `MeanShift` consists of values with no variance (for example, two clusters of 0 and 1), the performance becomes extremely slow.

I am unsure whether this is a bug or an unavoidable aspect of the algorithm's design. Any clarification would be appreciated.


### Steps/Code to Reproduce

```python
import numpy as np
from sklearn.cluster import MeanShift

x = np.concatenate([np.ones(100), np.zeros(100)])
_ = MeanShift().fit_predict(x.reshape(-1, 1)) # Slow

rng = np.random.default_rng(1)
x = np.concatenate([rng.uniform(0.0, 0.001, 100), rng.uniform(0.999, 1.0, 100)])
_ = MeanShift().fit_predict(x.reshape(-1, 1)) # Fast
```

Link to Google Colab: https://colab.research.google.com/drive/1hlqhtaD8T40hwcleUKoI4uzrW1XtSRA4?usp=sharing#scrollTo=6g5qI45KUW_i

### Expected Results

When data provided to `MeanShift` consists of values with no variance, the performance becomes as fast as when handling data with variance.


### Actual Results

If `MeanShift` receives a 1D array with no variance, the computation is significantly slower.

```python
import numpy as np
from sklearn.cluster import MeanShift

# Example where input has no variance
x = np.concatenate([np.ones(100), np.zeros(100)])
%timeit _ = MeanShift().fit_predict(x.reshape(-1, 1))
# Output: 24.9 s ± 340 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
```

Below is a control example, where the input has some variance:

```python
import numpy as np
from sklearn.cluster import MeanShift

# Example with minimal variance
rng = np.random.default_rng(1)
x = np.concatenate([rng.uniform(0.0, 0.001, 100), rng.uniform(0.999, 1.0, 100)])
%timeit _ = MeanShift().fit_predict(x.reshape(-1, 1))
# Output: 665 ms ± 101 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

```



### Versions

```shell
1.2.2
```

COMMENT:
I think that the fact that:

- `MeanShift` can reach `n_iter_ == max_iter` without raising a `ConvergenceWarning` is a bug,
- `MeanShift` does not converge in on 1D constant data in 300 iterations is another bug.

The second bug is probably caused by the `dist < stop_thresh` condition that should be loosened to `dist <= stop_thresh` because both `dist` and `stop_thresh` are 0.0  when data is constant.

Please feel free to open two PRs (one for each problem, in either order), along with non-regression tests.

Labels: Bug
Performance Degradation in MeanShift When Data Has No Variance

### Describe the bug

When data provided to `MeanShift` consists of values with no variance (for example, two clusters of 0 and 1), the performance becomes extremely slow.

I am unsure whether this is a bug or an unavoidable aspect of the algorithm's design. Any clarification would be appreciated.


### Steps/Code to Reproduce

```python
import numpy as np
from sklearn.cluster import MeanShift

x = np.concatenate([np.ones(100), np.zeros(100)])
_ = MeanShift().fit_predict(x.reshape(-1, 1)) # Slow

rng = np.random.default_rng(1)
x = np.concatenate([rng.uniform(0.0, 0.001, 100), rng.uniform(0.999, 1.0, 100)])
_ = MeanShift().fit_predict(x.reshape(-1, 1)) # Fast
```

Link to Google Colab: https://colab.research.google.com/drive/1hlqhtaD8T40hwcleUKoI4uzrW1XtSRA4?usp=sharing#scrollTo=6g5qI45KUW_i

### Expected Results

When data provided to `MeanShift` consists of values with no variance, the performance becomes as fast as when handling data with variance.


### Actual Results

If `MeanShift` receives a 1D array with no variance, the computation is significantly slower.

```python
import numpy as np
from sklearn.cluster import MeanShift

# Example where input has no variance
x = np.concatenate([np.ones(100), np.zeros(100)])
%timeit _ = MeanShift().fit_predict(x.reshape(-1, 1))
# Output: 24.9 s ± 340 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
```

Below is a control example, where the input has some variance:

```python
import numpy as np
from sklearn.cluster import MeanShift

# Example with minimal variance
rng = np.random.default_rng(1)
x = np.concatenate([rng.uniform(0.0, 0.001, 100), rng.uniform(0.999, 1.0, 100)])
%timeit _ = MeanShift().fit_predict(x.reshape(-1, 1))
# Output: 665 ms ± 101 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

```



### Versions

```shell
1.2.2
```

COMMENT:
It has been fixed in #28951 so closing this issue. Thanks @akikuno

Labels: Bug
Performance Degradation in MeanShift When Data Has No Variance

### Describe the bug

When data provided to `MeanShift` consists of values with no variance (for example, two clusters of 0 and 1), the performance becomes extremely slow.

I am unsure whether this is a bug or an unavoidable aspect of the algorithm's design. Any clarification would be appreciated.


### Steps/Code to Reproduce

```python
import numpy as np
from sklearn.cluster import MeanShift

x = np.concatenate([np.ones(100), np.zeros(100)])
_ = MeanShift().fit_predict(x.reshape(-1, 1)) # Slow

rng = np.random.default_rng(1)
x = np.concatenate([rng.uniform(0.0, 0.001, 100), rng.uniform(0.999, 1.0, 100)])
_ = MeanShift().fit_predict(x.reshape(-1, 1)) # Fast
```

Link to Google Colab: https://colab.research.google.com/drive/1hlqhtaD8T40hwcleUKoI4uzrW1XtSRA4?usp=sharing#scrollTo=6g5qI45KUW_i

### Expected Results

When data provided to `MeanShift` consists of values with no variance, the performance becomes as fast as when handling data with variance.


### Actual Results

If `MeanShift` receives a 1D array with no variance, the computation is significantly slower.

```python
import numpy as np
from sklearn.cluster import MeanShift

# Example where input has no variance
x = np.concatenate([np.ones(100), np.zeros(100)])
%timeit _ = MeanShift().fit_predict(x.reshape(-1, 1))
# Output: 24.9 s ± 340 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
```

Below is a control example, where the input has some variance:

```python
import numpy as np
from sklearn.cluster import MeanShift

# Example with minimal variance
rng = np.random.default_rng(1)
x = np.concatenate([rng.uniform(0.0, 0.001, 100), rng.uniform(0.999, 1.0, 100)])
%timeit _ = MeanShift().fit_predict(x.reshape(-1, 1))
# Output: 665 ms ± 101 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

```



### Versions

```shell
1.2.2
```

COMMENT:
@glemaitre @ogrisel 
I am very happy to contribute to a project I am always grateful for. 
Thank you very much for your guidance!

Labels: Bug
Undocumented change in tree_.value example for DecisionTreeClassifier between versions 1.3.2 and 1.4.2

### Describe the issue linked to the documentation

In the the 1.4.2 docs the [Understanding the decision tree structure page](https://scikit-learn.org/1.3/auto_examples/tree/plot_unveil_tree_structure.html#understanding-the-decision-tree-structure) provides code and output in order to inspect `tree_.value`, but the tree diagram and output from the code snippet are inconsistent.

The diagram shows integer values that represent the number of records in that class at each node.
The new output from the code appears to be the percentage? of the total number of records that are in the respective class.

The [1.3.2 docs](https://scikit-learn.org/1.3/auto_examples/tree/plot_unveil_tree_structure.html#what-is-the-values-array-used-here) were consistent on this page between the code output and the diagram lower describing the values array, so I expect that something changed between the versions but wasn't documented, at least here in this example.

I can't find where this change to `tree_.value` is documented and it appears to be causing confusion (see for example on [stack overflow](https://stackoverflow.com/questions/47719001/what-does-scikit-learn-decisiontreeclassifier-tree-value-do#comment138123014_47719621)) 





### Suggest a potential alternative/fix

I would suggest updating the visual and documenting more clearly what to expect from `tree_.value` for `DecisionTreeClassifier` in 1.4.2 since it is evidently different compared to 1.3.2.
I am working with some code that inspects the trees and would appreciate insight to make sure that I make the necessary adjustments to get the same values that I did with 1.3.2.

COMMENT:
Indeed, it should be a side effect of the bug fix in https://github.com/scikit-learn/scikit-learn/pull/27639 where we started to store proportion instead of absolute counts.

We should therefore update the documentation accordingly.

Labels: Documentation, Moderate, help wanted, module:tree