Talip7/scikit-learn-issues-multilabel huggingface dataset:
RFC: Breaking Changes for Version 2

A while ago we talked about the possibility of a version 2 with breaking changes. Specifically, this came up in the context of SLEP006: metadata routing, but there are other things we have wanted to break which we can do all in a single release. In this issue we can talk about the possibility of such a release, and maybe a few things we'd like to include in it.

I'll try to summarize the challenges of keeping backward compatibility (BC) in the context of SLEP006:

- `MetaEstimator(Estimator()).fit(X, y, sample_weight)` will raise according to SLEP006 since `sample_weight` is not requested by any object. For backward compatibility, during the transition period, it'll only warn and assume `sample_weight` is requested by `Estimator()`.
- What should the behavior of `GridSearchCV(Estimator(), scorer=scorer).fit(X, y, sample_weight)` be during the transition period? If we keep backward compatibility, it should warn (whatever the warning be), route `sample_weight` to `Estimator`, but not to the `scorer`. And that's only the case because we're keeping BC. From the user's perspective, it's very weird.
- `Pipeline` ATM in certain methods like `fit`, `fit_transform`, `predict`, routes `*_params` to the `fit/etc` of the last estimator only. And `decision_function` and `transform`, `score_samples` have no routing at all. Keeping this BC, proves to be challenging, and a bit nasty. ref: #24270
- In quite a few meta-estimators, we check if the sub-estimator has `sample_weight` in the signature of `fit`, and we pass the given sample weight to it if that's the case, and not if that's not the case. With routing, we would have a much better idea of when to pass sample weight and when not, but if we're keeping BC, it's really challenging to see if we should forward sample weights or not. Both AdaBoost (#24026) and Bagging (#24250) have been very challenging, to the point that we might end up having to check if a sub-estimator is a meta-estimator itself or not, and check signature of sub-estimator's `fit`, and check the given routing info to keep BC.

I'm pretty sure there are other issues we haven't foreseen, and they'll come up as we work on this project. We thought it'd be easy once we get to this point, but BC has proven to make it very tricky and has stalled the progress.

All of the above is adding substantial complexity which is probably error prone, and buggy. We have had to spend quite a bit of time every time we go back to review those bits to understand why we've done what we've done. Even as a temporary solution they're not very maintainable.

My proposal here is to have a version 2 which is not BC with previous releases in certain ways, and target a few breaking changes for it. We can aim to have the release in October 2023 or April/May 2024 as the version 2.

cc @scikit-learn/core-devs

COMMENT:
@lorentzenchr I personally don't want to go through the whole meta-estimator codebase twice for this 😁 And I think doing so, would probably add a year or two to the whole timeline lol.

Labels: RFC, Breaking Change
DOC: roc_auc_score wrong description of `average` parameter with `None` value

### Describe the issue linked to the documentation

Documentation of `sklearn.metrics.roc_auc_score` for `average` parameter ([lines 442-443](https://github.com/scikit-learn/scikit-learn/blob/main/sklearn/metrics/_ranking.py#L442-L443) of https://github.com/scikit-learn/scikit-learn/blob/main/sklearn/metrics/_ranking.py) clearly says that:
> For multiclass targets, `average=None` is only implemented for `multi_class='ovo'`

This is false because function `_multiclass_roc_auc_score` always throws a `NotImplementedError` error `if average is None and multi_class == "ovo"` ([lines 681-684](https://github.com/scikit-learn/scikit-learn/blob/main/sklearn/metrics/_ranking.py#L681-L684)) but it works flawlessly for `multi_class="ovr"`.

### Suggest a potential alternative/fix

Therefore, the documentation of `roc_auc_score` should be changed to: 
> For multiclass targets, `average=None` is only implemented for `multi_class='ovr'`

COMMENT:
Thank you for opening the issue! I agree that the docstring is incorrect. Are you interested in opening a PR to fix it?

Labels: Documentation, module:metrics
maximal information coefficient for feature selection

### Describe the workflow you want to enable

maximal information coeffcient

### Describe your proposed solution

maximal information coeffcient

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
I want to look into doing this, but can you tell me where in the repo it should be built? I have not contributed to this repo before, and a stronger description would be helpful. Thank you!

Labels: New Feature
maximal information coefficient for feature selection

### Describe the workflow you want to enable

maximal information coeffcient

### Describe your proposed solution

maximal information coeffcient

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
@leeauk21 Do you have more details regarding why this metric is useful?
Looking around to find the definition, I ended up on the following literature:
https://www.pnas.org/doi/abs/10.1073/pnas.1309933111

It seems that using mutual information directly is enough and MIC does not provide anything more.

So if you have additional thoughts and background this would be useful to evaluate whether or not we should integrate this feature.

Labels: New Feature
maximal information coefficient for feature selection

### Describe the workflow you want to enable

maximal information coeffcient

### Describe your proposed solution

maximal information coeffcient

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
I think Mutual information values can vary depending how you bin continuous data. whereas theoretically MIC wont but if MI works better then. There is no reason in implementing it. I found the above reason in Machine Learning in Probabilistic Perspective

Labels: New Feature
maximal information coefficient for feature selection

### Describe the workflow you want to enable

maximal information coeffcient

### Describe your proposed solution

maximal information coeffcient

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
If someone can craft a use-case where MIC stably return good values while MI can catastrophically fail depending on the binning, then why not. Otherwise it sounds like a YAGNI.

Labels: New Feature
maximal information coefficient for feature selection

### Describe the workflow you want to enable

maximal information coeffcient

### Describe your proposed solution

maximal information coeffcient

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
@ogrisel Hi, would [this paper](https://www.jstor.org/stable/26542523) suffice to prove the that MIC is useful? The point here is, in most cases, MIC provides similar scores to equally noisy relationships of different types, but MI does not. Though MIC is computationally more complex, there have been effective approximations that can be more easily evaluated.

The paper also gave pratical suggestions as for when and how to use MIC (starting from page 22 in the paper). It says:

> In some cases, many methods will return a large number of relationships, and so the number of relationships detected is less important than their relative ranking. This does happen in practice, for example in the gene expression analysis of [Heller et al. (2016)](https://arxiv.org/abs/1410.6758), in which several methods identified over half of the thousands of relationships in the data set as significant, as well as in the analysis of the WHO data set (can be found in section 8 of the paper, starting from page 24). In such cases, increasing the proportion of variable pairs identified as significant seems less important for scientific inquiry than having a meaningful way to prioritize the detected relationships for follow-up.

> Thus, a promising strategy for exploratory data analysis is: first, to compute a statistic designed to identify a large number of significant relationships of all kinds, and then second, to compute an equitable statistic on all significant relationships, ensuring a ranking that is meaningful.

Therefore, I do think adding the MIC metric as an option would be useful in specific cases. That being said, it is up to you and other maintainers to decide whether it is a useful feature for `scikit-learn`.

Labels: New Feature
maximal information coefficient for feature selection

### Describe the workflow you want to enable

maximal information coeffcient

### Describe your proposed solution

maximal information coeffcient

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
Based on my research utilizing a dataset encompassing diverse nonlinear correlations, a comparative analysis was conducted among the Maximal Information Coefficient (MIC), mutual information and other nonlinear correlation measures.

(I used sklearn.feature_selection.mutual_info_regression)

The results exhibited discrepancies between the two measures. The ability of mutual information and Kendall coefficient is similar, while MIC obviously has stronger ability.

These findings underscore the necessity of implementing MIC within the scikit-learn library. Not only for feature selection, but also for sklearn.metrics, just similar to mutual_info_score, mutual_info_classif, mutual_info_regression.

I'm sorry that my professional knowledge is insufficient to provide a more detailed explanation from a theoretical perspective. I can only present some experimental results. I hope they will be helpful to you.

<img width="845" height="415" alt="Image" src="https://github.com/user-attachments/assets/98ddc18d-88ee-4986-8e13-0f448b5fafef" />

<img width="1572" height="608" alt="Image" src="https://github.com/user-attachments/assets/bc7c0598-64b5-4b1e-b39b-eb7d7c3f65a8" />

Labels: New Feature
Change default behavior of `GaussianProcessRegressor.sample_y`'s `random_state` variable.

### Describe the workflow you want to enable

I think it is misleading to have the `sample_y` method to default to reproduce the same sample every time it is called.
The default value for `random_state` should be `None`, and the documentation `sample_y` should state that `None` ensures random behavior.



### Describe your proposed solution

Please see above box

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
Thank you for opening this issue! I agree it is strange to fix the random state in `sample_y`. For us to change the default, we would need to go through a [deprecation cycle](https://scikit-learn.org/dev/developers/contributing.html#change-the-default-value-of-a-parameter). Are you interested in opening a PR with the change?

Labels: New Feature, API, module:gaussian_process
Change default behavior of `GaussianProcessRegressor.sample_y`'s `random_state` variable.

### Describe the workflow you want to enable

I think it is misleading to have the `sample_y` method to default to reproduce the same sample every time it is called.
The default value for `random_state` should be `None`, and the documentation `sample_y` should state that `None` ensures random behavior.



### Describe your proposed solution

Please see above box

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
Absolutely! I will get right on that!

Labels: New Feature, API, module:gaussian_process
BUG IsotonicRegression errors with set_config(transform_output="pandas")

### Describe the bug

`IsotonicRegression.predict` is broken by `set_config(transform_output="pandas")`. Mabye similar to #25365.

### Steps/Code to Reproduce

```python
import numpy as np
from sklearn.isotonic import IsotonicRegression
y = np.arange(4)
x = np.ones(4)

iso = IsotonicRegression().fit(x, y)
iso.predict([1])  # array([1.5])

iso = IsotonicRegression().set_output(transform="pandas").fit(x, y)
iso.predict([1])
```

### Expected Results

```
array([1.5])
```

Or at least no error!

### Actual Results

```
TypeError: 'builtin_function_or_method' object is not iterable
```

### Versions

```shell
1.2.1
```

COMMENT:
Side note: Depending on the use case of scikit-learn estimators, library authors may want to deliberately set `set_output(transform="default")` in order to avoid unintentional behaviour.

Labels: Bug, Needs Triage
BUG IsotonicRegression errors with set_config(transform_output="pandas")

### Describe the bug

`IsotonicRegression.predict` is broken by `set_config(transform_output="pandas")`. Mabye similar to #25365.

### Steps/Code to Reproduce

```python
import numpy as np
from sklearn.isotonic import IsotonicRegression
y = np.arange(4)
x = np.ones(4)

iso = IsotonicRegression().fit(x, y)
iso.predict([1])  # array([1.5])

iso = IsotonicRegression().set_output(transform="pandas").fit(x, y)
iso.predict([1])
```

### Expected Results

```
array([1.5])
```

Or at least no error!

### Actual Results

```
TypeError: 'builtin_function_or_method' object is not iterable
```

### Versions

```shell
1.2.1
```

COMMENT:
I think that I solved the issue in the merged PR: https://github.com/scikit-learn/scikit-learn/pull/25500

The above snippet works with `main`.

Labels: Bug, Needs Triage
BUG IsotonicRegression errors with set_config(transform_output="pandas")

### Describe the bug

`IsotonicRegression.predict` is broken by `set_config(transform_output="pandas")`. Mabye similar to #25365.

### Steps/Code to Reproduce

```python
import numpy as np
from sklearn.isotonic import IsotonicRegression
y = np.arange(4)
x = np.ones(4)

iso = IsotonicRegression().fit(x, y)
iso.predict([1])  # array([1.5])

iso = IsotonicRegression().set_output(transform="pandas").fit(x, y)
iso.predict([1])
```

### Expected Results

```
array([1.5])
```

Or at least no error!

### Actual Results

```
TypeError: 'builtin_function_or_method' object is not iterable
```

### Versions

```shell
1.2.1
```

COMMENT:
@glemaitre Thanks. That PR did not pop up in my searches, but I focused on issues not PRs.

Labels: Bug, Needs Triage
BayesSearchCV default cv is wrong?

### Describe the issue linked to the documentation

Hi,

Is there a bug in the Docs for BayesSearchCV where it says that the default value for cv is 3 but I see it using 5 as the default when I run it with no explicit value for cv?

https://scikit-optimize.github.io/dev/modules/generated/skopt.BayesSearchCV.html#skopt.BayesSearchCV

I had a quick look in the code as below, the default in the code that is used seems to be 5.

cheers,
Rob.

---
Versions:

	import sklearn
	import skopt
	print(sklearn.__version__)
	print(skopt.__version__)

	1.0.2
	0.9.0
---
Docs from: https://scikit-optimize.github.io/dev/modules/generated/skopt.BayesSearchCV.html#skopt.BayesSearchCV

	cv: int, cross-validation generator or an iterable, optional
	Determines the cross-validation splitting strategy. Possible inputs for cv are:

	None, to use the default 3-fold cross validation,
----
https://github.com/scikit-learn/scikit-learn/blob/f2f3b3cef02df0365ebebadde4a51f2f98f9154d/sklearn/model_selection/_split.py#L2404

	def check_cv(cv=5, y=None, *, classifier=False):
		"""Input checker utility for building a cross-validator.
---

### Suggest a potential alternative/fix

Change the doc to say 5 fold rather than 3 fold.

COMMENT:
`scikit-optimize` is another open-source project not maintain by the developers of scikit-learn (or not directly). You should open this issue in their issue tracker: https://github.com/scikit-optimize/scikit-optimize/issues

Labels: Documentation, Needs Triage
Hello All,

Hello All,

I am struggling to run on Windows 11 a package MicroLIA which is dependent on skikit-learn. Its not clear to me if the problem is on my side or other.
I unstalled and re-installed via the following

conda uninstall scikit-learn

conda install -c conda-forge scikit-learn

I get the following error:

from MicroLIA import models
Traceback (most recent call last):

  File "C:\Users\XXX\AppData\Local\Temp\ipykernel_17972\1682702994.py", line 1, in <module>
    from MicroLIA import models

  File "C:\Users\XXX\.conda\envs\MicroLIA\lib\site-packages\MicroLIA\models.py", line 25, in <module>
    from MicroLIA.optimization import hyper_opt, borutashap_opt, KNN_imputation, MissForest_imputation

  File "C:\Users\XXX\.conda\envs\MicroLIA\lib\site-packages\MicroLIA\optimization.py", line 29, in <module>
    from missingpy import MissForest

  File "C:\Users\XXX\.conda\envs\MicroLIA\lib\site-packages\missingpy\__init__.py", line 1, in <module>
    from .knnimpute import KNNImputer

  File "C:\Users\XXX\.conda\envs\MicroLIA\lib\site-packages\missingpy\knnimpute.py", line 13, in <module>
    from sklearn.neighbors.base import _check_weights

ImportError: cannot import name '_check_weights' from 'sklearn.neighbors._base' (C:\Users\XXX\.conda\envs\MicroLIA\lib\site-packages\sklearn\neighbors\_base.py)

Any suggestions?

_Originally posted by @eddiemorris135 in https://github.com/scikit-learn/scikit-learn/discussions/25745_

COMMENT:
The issue is probably linked with the version of scikit-learn required by `missingpy`:

https://github.com/epsilon-machine/missingpy/blob/master/requirements.txt

This version is too old. Be aware that scikit-learn is providing a `KNNImputer` and you don't need `missingpy` anymore. This package seems unmaintained.

Labels: Needs Triage
Hello All,

Hello All,

I am struggling to run on Windows 11 a package MicroLIA which is dependent on skikit-learn. Its not clear to me if the problem is on my side or other.
I unstalled and re-installed via the following

conda uninstall scikit-learn

conda install -c conda-forge scikit-learn

I get the following error:

from MicroLIA import models
Traceback (most recent call last):

  File "C:\Users\XXX\AppData\Local\Temp\ipykernel_17972\1682702994.py", line 1, in <module>
    from MicroLIA import models

  File "C:\Users\XXX\.conda\envs\MicroLIA\lib\site-packages\MicroLIA\models.py", line 25, in <module>
    from MicroLIA.optimization import hyper_opt, borutashap_opt, KNN_imputation, MissForest_imputation

  File "C:\Users\XXX\.conda\envs\MicroLIA\lib\site-packages\MicroLIA\optimization.py", line 29, in <module>
    from missingpy import MissForest

  File "C:\Users\XXX\.conda\envs\MicroLIA\lib\site-packages\missingpy\__init__.py", line 1, in <module>
    from .knnimpute import KNNImputer

  File "C:\Users\XXX\.conda\envs\MicroLIA\lib\site-packages\missingpy\knnimpute.py", line 13, in <module>
    from sklearn.neighbors.base import _check_weights

ImportError: cannot import name '_check_weights' from 'sklearn.neighbors._base' (C:\Users\XXX\.conda\envs\MicroLIA\lib\site-packages\sklearn\neighbors\_base.py)

Any suggestions?

_Originally posted by @eddiemorris135 in https://github.com/scikit-learn/scikit-learn/discussions/25745_

COMMENT:
thanks for the tip!!

Labels: Needs Triage
Allow `norm=None` in `preprocessing.Normalizer`

### Describe the workflow you want to enable

I would like to be able to set `norm=None` in `preprocessing.Normalizer`, so that I can eaily tune `norm` as part of a pipeline. Currently, `norm` can only be one of `{‘l1’, ‘l2’, ‘max’}`. 

### Describe your proposed solution

Change `Normalizer` and associated functions (eg, `normalize`) to allow `norm=None`.

This would not be a massive change in API, especially given `feature_extraction.text.TfidfVectorizer` also has a `norm` keyword argument that can be `None`.

### Describe alternatives you've considered, if relevant

The alternative would be to retain the existing behaviour and require that people use `"passthrough"` to fulfil the requirement I have described above. This is an OK solution, but being able to set `norm=None` feels more intuitive to me. In fact, I tried this at first, assuming it would work based on my experience with `feature_extraction.text.TfidfVectorizer`, and was then surprised when it didn't.

If there is a general principle that I'm not aware of that explains the decision not to allow `norm=None` then fair enough. (I am thinking, for example, that there might be a view that a `Transformer` shouldn't allow a `passthrough` through any combination of keyword arguments.)

### Additional context

If this change would be accepted then I am happy to make a PR for it.

COMMENT:
Calling `preprocessing.Normalizer` with ``norm=None`` wouldn't have any impact on input data, except whether to copy it or not. Hence, in this case, the preprocessor won't have a meaning. I wonder if this is a prevalent use case.

In the meantime, you can alter the behavior of the `Normalizer` to support `None` norm by inheriting from it and overriding its ``transform`` method:

```python
from sklearn.preprocessing._data import Normalizer, normalize
from sklearn.utils.validation import check_array, FLOAT_DTYPES

class CustomNormalizer(Normalizer):

    def transform(self, X, copy=None):
        # copy-past of `sklearn/preprocessing/_data.py L1986-L1987`
        copy = copy if copy is not None else self.copy
        X = self._validate_data(X, accept_sparse="csr", reset=False)

        # copy-past of `sklearn/preprocessing/_data.py L1817-L1823`
        if self.norm is None:
            X = check_array(
                X,
                accept_sparse="csr",
                copy=copy,
                estimator="the normalize function",
                dtype=FLOAT_DTYPES)
        else:
            X = normalize(X, norm=self.norm, axis=1, copy=copy)

        return X
```

Example of usage:
```python
>>> X = [[4, 1, 2, 2],
         [1, 3, 9, 3],
         [5, 7, 5, 1]]
>>> normalizer = CustomNormalizer(norm=None)
>>> normalizer.transform(X)
array([[4., 1., 2., 2.],
       [1., 3., 9., 3.],
       [5., 7., 5., 1.]])
```

Labels: New Feature
Allow `norm=None` in `preprocessing.Normalizer`

### Describe the workflow you want to enable

I would like to be able to set `norm=None` in `preprocessing.Normalizer`, so that I can eaily tune `norm` as part of a pipeline. Currently, `norm` can only be one of `{‘l1’, ‘l2’, ‘max’}`. 

### Describe your proposed solution

Change `Normalizer` and associated functions (eg, `normalize`) to allow `norm=None`.

This would not be a massive change in API, especially given `feature_extraction.text.TfidfVectorizer` also has a `norm` keyword argument that can be `None`.

### Describe alternatives you've considered, if relevant

The alternative would be to retain the existing behaviour and require that people use `"passthrough"` to fulfil the requirement I have described above. This is an OK solution, but being able to set `norm=None` feels more intuitive to me. In fact, I tried this at first, assuming it would work based on my experience with `feature_extraction.text.TfidfVectorizer`, and was then surprised when it didn't.

If there is a general principle that I'm not aware of that explains the decision not to allow `norm=None` then fair enough. (I am thinking, for example, that there might be a view that a `Transformer` shouldn't allow a `passthrough` through any combination of keyword arguments.)

### Additional context

If this change would be accepted then I am happy to make a PR for it.

COMMENT:
Thanks @Badr-MOUFAD, that was my solution too – although I also modified `_validate_params` because the `Normalizer` was part of a pipeline on which `fit`was called.

My issue is a suggested feature (which I am happy to implement) rather than a request for help. Would my proposed change be accepted?

Labels: New Feature