Talip7/scikit-learn-issues-multilabel huggingface dataset:
About using feature_Selection many times

### Describe the workflow you want to enable

I will state my question first.

When using a pipeline that combines a variable selection method with multiple estimators.
If I evaluate them with cross_validation, is there a good way not to evaluate fit for variable selection every time? 

```
from sklearn.feature_selection import SelectFromModel
from sklearn.linear_model import Lasso, TheilSenRegressor
from sklearn.ensemble import RandomForestRegressor
from sklearn.pipeline import Pipeline
from sklearn.model_selection import cross_val_predict, cross_val_score
from sklearn.datasets import load_diabetes
import itertools

X,y = load_diabetes(return_X_y=True)


selector = SelectFromModel(Lasso(random_state=0))
selectors = [selector]

estimators = [RandomForestRegressor(), TheilSenRegressor()]

pipelines = []
for selector_, estimator_ in itertools.product(selectors , estimators ):
    pipelines.append(Pipeline([("selector", selector_), ("estimator", estimator_)]))

# pipelines : [SelectFromModel,  RandomForestRegressor], [SelectFromModel,  TheilSenRegressor]

for pipe_ in pipelines:
    cross_val_predict(pipe_, X, y)

```

In the code above, the same selector in another pipeline repeats the same calculation.

This repeat can become a big problem when the number of estimators to be evaluated increases and when the computational complexity of the selector increases.
Is there a good way to avoid this repeat?




### Describe your proposed solution

If the method to solve the above problem is not implemented in the current sklearn, I would like to know a good method.

### Describe alternatives you've considered, if relevant

-

### Additional context

_No response_

COMMENT:
You should set using the `memory` parameter that will cache the intermediate results and avoid recomputation if the same data and processing is requested.

Labels: New Feature, Needs Triage
ValueError: buffer source array is read-only when derializing a Tree from a readonly buffer.

As observed on our Circle CI and reproduced locally:

```python-traceback
/home/circleci/project/examples/release_highlights/plot_release_highlights_0_24_0.py failed leaving traceback:
Traceback (most recent call last):
  File "/home/circleci/mambaforge/envs/testenv/lib/python3.8/site-packages/sphinx_gallery/gen_gallery.py", line 159, in call_memory
    return 0., func()
  File "/home/circleci/mambaforge/envs/testenv/lib/python3.8/site-packages/sphinx_gallery/gen_rst.py", line 466, in __call__
    exec(self.code, self.fake_main.__dict__)
  File "/home/circleci/project/examples/release_highlights/plot_release_highlights_0_24_0.py", line 211, in <module>
    display = PartialDependenceDisplay.from_estimator(
  File "/home/circleci/project/sklearn/inspection/_plot/partial_dependence.py", line 704, in from_estimator
    pd_results = Parallel(n_jobs=n_jobs, verbose=verbose)(
  File "/home/circleci/project/sklearn/utils/parallel.py", line 63, in __call__
    return super().__call__(iterable_with_config)
  File "/home/circleci/mambaforge/envs/testenv/lib/python3.8/site-packages/joblib/parallel.py", line 1098, in __call__
    self.retrieve()
  File "/home/circleci/mambaforge/envs/testenv/lib/python3.8/site-packages/joblib/parallel.py", line 975, in retrieve
    self._output.extend(job.get(timeout=self.timeout))
  File "/home/circleci/mambaforge/envs/testenv/lib/python3.8/site-packages/joblib/_parallel_backends.py", line 567, in wrap_future_result
    return future.result(timeout=timeout)
  File "/home/circleci/mambaforge/envs/testenv/lib/python3.8/concurrent/futures/_base.py", line 439, in result
    return self.__get_result()
  File "/home/circleci/mambaforge/envs/testenv/lib/python3.8/concurrent/futures/_base.py", line 388, in __get_result
    raise self._exception
joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed to un-serialize. Please ensure that the arguments of the function are all picklable.
```

https://app.circleci.com/pipelines/github/scikit-learn/scikit-learn/43516/workflows/d8bd88b5-872e-4f50-88d4-e41886117849/jobs/225593

COMMENT:
@ogrisel I'll fix this.

Labels: Bug, cython
ValueError: buffer source array is read-only when derializing a Tree from a readonly buffer.

As observed on our Circle CI and reproduced locally:

```python-traceback
/home/circleci/project/examples/release_highlights/plot_release_highlights_0_24_0.py failed leaving traceback:
Traceback (most recent call last):
  File "/home/circleci/mambaforge/envs/testenv/lib/python3.8/site-packages/sphinx_gallery/gen_gallery.py", line 159, in call_memory
    return 0., func()
  File "/home/circleci/mambaforge/envs/testenv/lib/python3.8/site-packages/sphinx_gallery/gen_rst.py", line 466, in __call__
    exec(self.code, self.fake_main.__dict__)
  File "/home/circleci/project/examples/release_highlights/plot_release_highlights_0_24_0.py", line 211, in <module>
    display = PartialDependenceDisplay.from_estimator(
  File "/home/circleci/project/sklearn/inspection/_plot/partial_dependence.py", line 704, in from_estimator
    pd_results = Parallel(n_jobs=n_jobs, verbose=verbose)(
  File "/home/circleci/project/sklearn/utils/parallel.py", line 63, in __call__
    return super().__call__(iterable_with_config)
  File "/home/circleci/mambaforge/envs/testenv/lib/python3.8/site-packages/joblib/parallel.py", line 1098, in __call__
    self.retrieve()
  File "/home/circleci/mambaforge/envs/testenv/lib/python3.8/site-packages/joblib/parallel.py", line 975, in retrieve
    self._output.extend(job.get(timeout=self.timeout))
  File "/home/circleci/mambaforge/envs/testenv/lib/python3.8/site-packages/joblib/_parallel_backends.py", line 567, in wrap_future_result
    return future.result(timeout=timeout)
  File "/home/circleci/mambaforge/envs/testenv/lib/python3.8/concurrent/futures/_base.py", line 439, in result
    return self.__get_result()
  File "/home/circleci/mambaforge/envs/testenv/lib/python3.8/concurrent/futures/_base.py", line 388, in __get_result
    raise self._exception
joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed to un-serialize. Please ensure that the arguments of the function are all picklable.
```

https://app.circleci.com/pipelines/github/scikit-learn/scikit-learn/43516/workflows/d8bd88b5-872e-4f50-88d4-e41886117849/jobs/225593

COMMENT:
Thanks!

Labels: Bug, cython
RFC enable github's pull request merge queue?

https://github.blog/changelog/2023-02-08-pull-request-merge-queue-public-beta/

It seems like a nice usability improvement.

COMMENT:
Just curious, I quickly read the article and I am not to sure this would be a game changer, but maybe I missed something ...

> Before merge queue, developers were often required to update their pull request branches prior to merging to ensure their changes wouldn't break the main branch when merged.

This rarely happens in scikit-learn right? I feel PRs don't step on each other toes in a vast majority of cases but again maybe I am missing something.

Labels: RFC
RFC enable github's pull request merge queue?

https://github.blog/changelog/2023-02-08-pull-request-merge-queue-public-beta/

It seems like a nice usability improvement.

COMMENT:
As I understand it the problem that a merge queue solves is that PRs are "constantly" being merged into `main` and you have to rebase on `main`/merge `main` into before being able/allowed to merge your PR. In case the combination of the new changes on `main` and your branch lead to a CI failure.

However I don't think the merging of PRs is done like this* in scikit-learn. So maybe the benefit is small?

What does seem to happen regularly is that your "what's new" changes conflict with `main` if your PR doesn't get merged soon. But merge queues don't help with that.

Overall, I feel like there is something I am missing regarding this new feature. It seems like a solution to a problem not a lot of people have?? (Or that I misunderstand something)

---
* If the CI was run three days ago for a PR and is green, then people merge a PR. Even if there have been changes to `main` in the mean time.

Labels: RFC
RFC enable github's pull request merge queue?

https://github.blog/changelog/2023-02-08-pull-request-merge-queue-public-beta/

It seems like a nice usability improvement.

COMMENT:
From memory, the issue that merge queue solves for happens around ~2 times a year in the past 3 years. For example, when a new test was added to `main` that would have failed in a PR and the PR was not updated with `main` recently. If we merge the PR, then it will cause `main` to fail.

In any case, I think it's okay to turn merge queues on and see if we like it. We can always turn it off if it's too much of a hassle.

Labels: RFC
RFC enable github's pull request merge queue?

https://github.blog/changelog/2023-02-08-pull-request-merge-queue-public-beta/

It seems like a nice usability improvement.

COMMENT:
I had a problem when trying to use it today in #25585. Not sure what's going one because everything was green on this PR but apparently there was a 4 min timeout (see details in the linked PR).

Labels: RFC
RFC enable github's pull request merge queue?

https://github.blog/changelog/2023-02-08-pull-request-merge-queue-public-beta/

It seems like a nice usability improvement.

COMMENT:
I saw the same issue with the merge queue in https://github.com/scikit-learn/scikit-learn/pull/25633. I increase the wait time to 90 minutes because Azure Pipeline took > 60 minutes to complete. Then I noticed that the "Check Changelog" GitHub Action never actually runs on the merge queue and waits forever. I suspect this is a bug with the merge queue feature.

Since the merge queue never actually merges even when all the tests pass, I am going to turn the merge queue off for now.

Labels: RFC
Add option to scale Matthews correlation coefficient (MCC) output to the [0, 1] range

### Describe the workflow you want to enable

Matthew's correlation coefficient is known to have a different range of possible values from most other classification performance metrics. While it's usual for metrics to lie in the [0, 1] range, MCC lies in [-1, 1] for the binary case. Even worse, this isn't exactly the case for the multiclass case, with the range being [-1/(n_classes-1), 1] (an informal demonstration is present in the Additional context section).

This makes it hard to compare or explain MCC in relation to other more common metrics. It would be of use to have an option to automatically scale the output of `matthews_corrcoef` metric to either the [0, 1] or [-1, 1] ranges.

### Describe your proposed solution

A solution would be to add a parameter with 3 possible values, such as

> scale: {‘ratio’, ‘corrcoef'} or None, default=None

That way, if the scale parameter equals `ratio` (ranging from 0 to 1) the output would simply need to be scaled to 
> `mcc = (mcc * (n_classes - 1) + 1) / n_classes`. 

Similarly, in the case where scale equals `corrcoef` (ranging from -1 to 1) we scale instead to 
> `mcc = (2 * (mcc * (n_classes - 1) + 1) / n_classes) - 1`.



### Describe alternatives you've considered, if relevant

_No response_

### Additional context

### Demonstration that MCC lies in the range [-1/(n_classes-1), 1] for an arbitrary number `n_classes` of possible classes.

Following from [definition](https://en.wikipedia.org/wiki/Phi_coefficient#Multiclass_case), we have that for n classes

> MCC = (cs - **t**.**p**) / (sqrt(s^2 - **p**.**p**) sqrt(s^2 - **t**.**t**))

where c is the total number of samples correctly predicted, s is the total number of samples, **t** is an n-dimensional vector counting the number of times each class occurred, and **p** is an n-dimensional vector counting the number of times each class was predicted.

Naturally, MCC will be lowest when c=0. So, we have 

> -**t**.**p** / (sqrt(s^2 - **p**.**p**) sqrt(s^2 - **t**.**t**))

Since we have a minus in the numerator, to get the minimum value we need to maximize the numerator and minimize the denominator. For a fixed number of samples we will meet both conditions when both **t** and **p** are diagonal vectors of the form s/n.**1**, where **1** is the 1-vector of n dimensions. Thus, we have

> -n(s/n)^2 / sqrt(s^2 - n(s/n)^2) sqrt(s^2 - n(s/n)^2)
> -n(s/n)^2 / (s^2 - n(s/n)^2)
> -(s^2/n) / (s^2 - (s^2/n))

Canceling everything out we arrive at -1/(n-1) for the minimum value of MCC for n classes. As a sanity check, we can calculate the value for n=2:

> -1/(2 - 1) = -1/1 = -1

COMMENT:
I am against modifying the original definition of the metric. This is a correlation coefficient and it makes sense to be in [-1, 1]. One can always write its own "normalized" metric and use the the `make_scorer` if such an hand-crafted metric is required.

Labels: New Feature
Add option to scale Matthews correlation coefficient (MCC) output to the [0, 1] range

### Describe the workflow you want to enable

Matthew's correlation coefficient is known to have a different range of possible values from most other classification performance metrics. While it's usual for metrics to lie in the [0, 1] range, MCC lies in [-1, 1] for the binary case. Even worse, this isn't exactly the case for the multiclass case, with the range being [-1/(n_classes-1), 1] (an informal demonstration is present in the Additional context section).

This makes it hard to compare or explain MCC in relation to other more common metrics. It would be of use to have an option to automatically scale the output of `matthews_corrcoef` metric to either the [0, 1] or [-1, 1] ranges.

### Describe your proposed solution

A solution would be to add a parameter with 3 possible values, such as

> scale: {‘ratio’, ‘corrcoef'} or None, default=None

That way, if the scale parameter equals `ratio` (ranging from 0 to 1) the output would simply need to be scaled to 
> `mcc = (mcc * (n_classes - 1) + 1) / n_classes`. 

Similarly, in the case where scale equals `corrcoef` (ranging from -1 to 1) we scale instead to 
> `mcc = (2 * (mcc * (n_classes - 1) + 1) / n_classes) - 1`.



### Describe alternatives you've considered, if relevant

_No response_

### Additional context

### Demonstration that MCC lies in the range [-1/(n_classes-1), 1] for an arbitrary number `n_classes` of possible classes.

Following from [definition](https://en.wikipedia.org/wiki/Phi_coefficient#Multiclass_case), we have that for n classes

> MCC = (cs - **t**.**p**) / (sqrt(s^2 - **p**.**p**) sqrt(s^2 - **t**.**t**))

where c is the total number of samples correctly predicted, s is the total number of samples, **t** is an n-dimensional vector counting the number of times each class occurred, and **p** is an n-dimensional vector counting the number of times each class was predicted.

Naturally, MCC will be lowest when c=0. So, we have 

> -**t**.**p** / (sqrt(s^2 - **p**.**p**) sqrt(s^2 - **t**.**t**))

Since we have a minus in the numerator, to get the minimum value we need to maximize the numerator and minimize the denominator. For a fixed number of samples we will meet both conditions when both **t** and **p** are diagonal vectors of the form s/n.**1**, where **1** is the 1-vector of n dimensions. Thus, we have

> -n(s/n)^2 / sqrt(s^2 - n(s/n)^2) sqrt(s^2 - n(s/n)^2)
> -n(s/n)^2 / (s^2 - n(s/n)^2)
> -(s^2/n) / (s^2 - (s^2/n))

Canceling everything out we arrive at -1/(n-1) for the minimum value of MCC for n classes. As a sanity check, we can calculate the value for n=2:

> -1/(2 - 1) = -1/1 = -1

COMMENT:
That's the thing: in the multiclass case it isn't a proper correlation coefficient, since it isn't in the [-1, 1] range. Currently the range is [-1/(n_classes-1), 1], which is only [-1, 1] for n_classes=2. 

I agree that the optional scaling to [0, 1] isn't really necessary, though.

Labels: New Feature
Add option to scale Matthews correlation coefficient (MCC) output to the [0, 1] range

### Describe the workflow you want to enable

Matthew's correlation coefficient is known to have a different range of possible values from most other classification performance metrics. While it's usual for metrics to lie in the [0, 1] range, MCC lies in [-1, 1] for the binary case. Even worse, this isn't exactly the case for the multiclass case, with the range being [-1/(n_classes-1), 1] (an informal demonstration is present in the Additional context section).

This makes it hard to compare or explain MCC in relation to other more common metrics. It would be of use to have an option to automatically scale the output of `matthews_corrcoef` metric to either the [0, 1] or [-1, 1] ranges.

### Describe your proposed solution

A solution would be to add a parameter with 3 possible values, such as

> scale: {‘ratio’, ‘corrcoef'} or None, default=None

That way, if the scale parameter equals `ratio` (ranging from 0 to 1) the output would simply need to be scaled to 
> `mcc = (mcc * (n_classes - 1) + 1) / n_classes`. 

Similarly, in the case where scale equals `corrcoef` (ranging from -1 to 1) we scale instead to 
> `mcc = (2 * (mcc * (n_classes - 1) + 1) / n_classes) - 1`.



### Describe alternatives you've considered, if relevant

_No response_

### Additional context

### Demonstration that MCC lies in the range [-1/(n_classes-1), 1] for an arbitrary number `n_classes` of possible classes.

Following from [definition](https://en.wikipedia.org/wiki/Phi_coefficient#Multiclass_case), we have that for n classes

> MCC = (cs - **t**.**p**) / (sqrt(s^2 - **p**.**p**) sqrt(s^2 - **t**.**t**))

where c is the total number of samples correctly predicted, s is the total number of samples, **t** is an n-dimensional vector counting the number of times each class occurred, and **p** is an n-dimensional vector counting the number of times each class was predicted.

Naturally, MCC will be lowest when c=0. So, we have 

> -**t**.**p** / (sqrt(s^2 - **p**.**p**) sqrt(s^2 - **t**.**t**))

Since we have a minus in the numerator, to get the minimum value we need to maximize the numerator and minimize the denominator. For a fixed number of samples we will meet both conditions when both **t** and **p** are diagonal vectors of the form s/n.**1**, where **1** is the 1-vector of n dimensions. Thus, we have

> -n(s/n)^2 / sqrt(s^2 - n(s/n)^2) sqrt(s^2 - n(s/n)^2)
> -n(s/n)^2 / (s^2 - n(s/n)^2)
> -(s^2/n) / (s^2 - (s^2/n))

Canceling everything out we arrive at -1/(n-1) for the minimum value of MCC for n classes. As a sanity check, we can calculate the value for n=2:

> -1/(2 - 1) = -1/1 = -1

COMMENT:
I agree with @glemaitre. Additionally, an explanation for the range is also explicitly documented. If conformity in range is required for whatever reason, it makes sense to manually rescale separately rather than alter it natively. Therefore I think it is reasonable to leave it as-is and am closing this issue. Let me know if you have any questions or concerns :)

Labels: New Feature
Support pandas nullable dtypes for scoring metrics

### Describe the workflow you want to enable

I would like to be able to pass data with the nullable pandas dtypes (`Int64`, `Float64`, and `boolean`) into sklearn metrics such as `matthews_corrcoef`, `accuracy_score`, and `f1_score` (and more) even if the data does not contain any nans. Currently, they result in one of several errors:

- If `y_true` and `y_pred` are both nullable types: `ValueError: unknown is not supported`
-  it only one of `y_true` or `y_pred` is nullable and the other is non nullable : `ValueError: Classification metrics can't handle a mix of unknown and binary [or multiclass] targets`
- Some metrics such as `log_loss` result in a different error when `y_true` is nullable:  `ValueError: Unknown label type: (0    1`

Repro with sklearn 1.2.1 and pandas 1.5.3:

```python
    import pandas as pd
    import pytest
    from sklearn import metrics

    for dtype in ['Int64', 'Float64', 'boolean']:
        # Error if only target uses nullable types
        X = pd.DataFrame({"a": pd.Series([1, 2, 3, 4]), 
                          "b": pd.Series([9,8,7,6])})

        # Two nullable dtypes used 
        y_true = pd.Series([1, 0, 1, 0], dtype=dtype)
        y_predicted = pd.Series([1, 0, 1, 0], dtype=dtype)
        with pytest.raises(ValueError, match="unknown is not supported"):
            metrics.accuracy_score(
                    y_true,
                    y_predicted,
                )

        # Only one nullable dtype used 
        y_predicted = pd.Series([1, 0, 1, 0], dtype="float64")
        with pytest.raises(ValueError, match="Classification metrics can't handle a mix of unknown and binary targets"):
            metrics.accuracy_score(
                    y_true,
                    y_predicted,
                )
``` 



### Describe your proposed solution

Sklearn should recognize the pandas nullable dtypes as the correct type of target for their scoring metrics like it does with the non nullable dtypes.

### Describe alternatives you've considered, if relevant

As this data doesn't have null values, we can convert to the non nullable dtype prior to passing to sklearn, but doing that will make it cumbersome to build software that leverages both the latest pandas dtypes and sklearn.

### Additional context

_No response_

COMMENT:
Thank you for opening this issue! With https://github.com/scikit-learn/scikit-learn/pull/25638 merged, nullable dtypes are supported and the provided snippet does not raise anymore.

Labels: New Feature, Needs Triage
Support pandas nullable dtypes for scoring metrics

### Describe the workflow you want to enable

I would like to be able to pass data with the nullable pandas dtypes (`Int64`, `Float64`, and `boolean`) into sklearn metrics such as `matthews_corrcoef`, `accuracy_score`, and `f1_score` (and more) even if the data does not contain any nans. Currently, they result in one of several errors:

- If `y_true` and `y_pred` are both nullable types: `ValueError: unknown is not supported`
-  it only one of `y_true` or `y_pred` is nullable and the other is non nullable : `ValueError: Classification metrics can't handle a mix of unknown and binary [or multiclass] targets`
- Some metrics such as `log_loss` result in a different error when `y_true` is nullable:  `ValueError: Unknown label type: (0    1`

Repro with sklearn 1.2.1 and pandas 1.5.3:

```python
    import pandas as pd
    import pytest
    from sklearn import metrics

    for dtype in ['Int64', 'Float64', 'boolean']:
        # Error if only target uses nullable types
        X = pd.DataFrame({"a": pd.Series([1, 2, 3, 4]), 
                          "b": pd.Series([9,8,7,6])})

        # Two nullable dtypes used 
        y_true = pd.Series([1, 0, 1, 0], dtype=dtype)
        y_predicted = pd.Series([1, 0, 1, 0], dtype=dtype)
        with pytest.raises(ValueError, match="unknown is not supported"):
            metrics.accuracy_score(
                    y_true,
                    y_predicted,
                )

        # Only one nullable dtype used 
        y_predicted = pd.Series([1, 0, 1, 0], dtype="float64")
        with pytest.raises(ValueError, match="Classification metrics can't handle a mix of unknown and binary targets"):
            metrics.accuracy_score(
                    y_true,
                    y_predicted,
                )
``` 



### Describe your proposed solution

Sklearn should recognize the pandas nullable dtypes as the correct type of target for their scoring metrics like it does with the non nullable dtypes.

### Describe alternatives you've considered, if relevant

As this data doesn't have null values, we can convert to the non nullable dtype prior to passing to sklearn, but doing that will make it cumbersome to build software that leverages both the latest pandas dtypes and sklearn.

### Additional context

_No response_

COMMENT:
Thank you for the quick work! Do you know if this will go out in the next patch release?

Labels: New Feature, Needs Triage