Talip7/scikit-learn-issues-multilabel huggingface dataset:
⚠️ CI failed on Wheel builder (last failure: Sep 29, 2024) ⚠️

**CI is still failing on [Wheel builder](https://github.com/scikit-learn/scikit-learn/actions/runs/11089440252)** (Sep 29, 2024)

COMMENT:
## CI is no longer failing! ✅

[Successful run](https://github.com/scikit-learn/scikit-learn/actions/runs/11099024913) on Sep 30, 2024

Labels: Needs Triage
RFC Expose `xfail_checks` with a more flexible API

xref: https://github.com/scikit-learn/scikit-learn/pull/29818#issuecomment-2378967067

Right now we have the `tags._xfail_checks` which seems private since it has the leading underscore.

We're refactoring tests and making them more modular and much nicer to deal with, but still there are going to be cases where an estimator developer might want to skip a few tests, and not a whole category.

So the proposal here is to rename `_xfail_checks` to `xfail_checks` (with a deprecation cycle of one release?), and also add the ability for the developers to set the whether the tests should fail, warn, or be skipped/xfailed.

There's also the question of granularity: do we want to set the `warn/xfail/warn` to be set on the estimator level, or for each test?

Some alternatives could be:

# Option 1
```py
class MyEstimator(BaseEstimator):
    def __sklearn_tag__(self):
        tags = super().__sklearn_tag__()
        tags.xfail_checks = {
            "check_estimators_dtypes": ("some-error", "warn"/"skip"/"raise"),
        }
        return tags
```

# Option 2
```py
class MyEstimator(BaseEstimator):
    def __sklearn_tag__(self):
        tags = super().__sklearn_tag__()
        tags.xfailed_checks = "warn"/"skip"/"raise"
        tags.xfail_checks = {
            "check_estimators_dtypes": "some-error",
        }
        return tags
```

# Option 3
```py
class MyEstimator(BaseEstimator):
    def __sklearn_tag__(self):
        tags = super().__sklearn_tag__()
        tags.xfailed_checks = {
            "check_estimators_dtypes": "warn"/"skip"/"raise",
        }
        tags.xfail_checks = {
            "check_estimators_dtypes": "some-error",
        }
        return tags
```

cc @scikit-learn/core-devs  since it's public /developer API RFC

COMMENT:
I like this idea in general, and option 1 in particular. The test-by-test granularity may be helpful especially for third-party estimators that are rapidly iterating and may have varying degrees of compliance and expectations that change across tests. It's simple from an API perspective yet clearly communicates expectations.

Labels: RFC, Developer API
RFC Expose `xfail_checks` with a more flexible API

xref: https://github.com/scikit-learn/scikit-learn/pull/29818#issuecomment-2378967067

Right now we have the `tags._xfail_checks` which seems private since it has the leading underscore.

We're refactoring tests and making them more modular and much nicer to deal with, but still there are going to be cases where an estimator developer might want to skip a few tests, and not a whole category.

So the proposal here is to rename `_xfail_checks` to `xfail_checks` (with a deprecation cycle of one release?), and also add the ability for the developers to set the whether the tests should fail, warn, or be skipped/xfailed.

There's also the question of granularity: do we want to set the `warn/xfail/warn` to be set on the estimator level, or for each test?

Some alternatives could be:

# Option 1
```py
class MyEstimator(BaseEstimator):
    def __sklearn_tag__(self):
        tags = super().__sklearn_tag__()
        tags.xfail_checks = {
            "check_estimators_dtypes": ("some-error", "warn"/"skip"/"raise"),
        }
        return tags
```

# Option 2
```py
class MyEstimator(BaseEstimator):
    def __sklearn_tag__(self):
        tags = super().__sklearn_tag__()
        tags.xfailed_checks = "warn"/"skip"/"raise"
        tags.xfail_checks = {
            "check_estimators_dtypes": "some-error",
        }
        return tags
```

# Option 3
```py
class MyEstimator(BaseEstimator):
    def __sklearn_tag__(self):
        tags = super().__sklearn_tag__()
        tags.xfailed_checks = {
            "check_estimators_dtypes": "warn"/"skip"/"raise",
        }
        tags.xfail_checks = {
            "check_estimators_dtypes": "some-error",
        }
        return tags
```

cc @scikit-learn/core-devs  since it's public /developer API RFC

COMMENT:
I prefer Option 1 over the rest of the options. The only nit is to have the behavior come first:

```python
        tags.xfail_checks = {
            "check_estimators_dtypes": ("warn"/"skip"/"raise", "my reason"),
        }
```

This ordering is consistent with `pytest`: `pytest.mark.skip(reason="my reason")`. The "skip" behavior comes first and then a reason.

Labels: RFC, Developer API
RFC Expose `xfail_checks` with a more flexible API

xref: https://github.com/scikit-learn/scikit-learn/pull/29818#issuecomment-2378967067

Right now we have the `tags._xfail_checks` which seems private since it has the leading underscore.

We're refactoring tests and making them more modular and much nicer to deal with, but still there are going to be cases where an estimator developer might want to skip a few tests, and not a whole category.

So the proposal here is to rename `_xfail_checks` to `xfail_checks` (with a deprecation cycle of one release?), and also add the ability for the developers to set the whether the tests should fail, warn, or be skipped/xfailed.

There's also the question of granularity: do we want to set the `warn/xfail/warn` to be set on the estimator level, or for each test?

Some alternatives could be:

# Option 1
```py
class MyEstimator(BaseEstimator):
    def __sklearn_tag__(self):
        tags = super().__sklearn_tag__()
        tags.xfail_checks = {
            "check_estimators_dtypes": ("some-error", "warn"/"skip"/"raise"),
        }
        return tags
```

# Option 2
```py
class MyEstimator(BaseEstimator):
    def __sklearn_tag__(self):
        tags = super().__sklearn_tag__()
        tags.xfailed_checks = "warn"/"skip"/"raise"
        tags.xfail_checks = {
            "check_estimators_dtypes": "some-error",
        }
        return tags
```

# Option 3
```py
class MyEstimator(BaseEstimator):
    def __sklearn_tag__(self):
        tags = super().__sklearn_tag__()
        tags.xfailed_checks = {
            "check_estimators_dtypes": "warn"/"skip"/"raise",
        }
        tags.xfail_checks = {
            "check_estimators_dtypes": "some-error",
        }
        return tags
```

cc @scikit-learn/core-devs  since it's public /developer API RFC

COMMENT:
Somehow related, side note: maybe `check_estimator` could be made to return a structured result object that is programmatically introspectable.

This would allow third-party libraries to build and publish a scikit-learn compliance table to be integrated as part of their documentation. In case of XFAILed checks, the reason could be displayed as part of the report.

Currently, users would have to dig into CI logs and grep the pytest output, assuming those projects use the `parametrize_with_checks` as part of a pytest test suite instead of just calling `check_estimator`.

Thinking about, we could even start by eating our own dog food: we have no simple summary of all the XFAILed/XPASSed cases for scikit-learn estimators at the moment.

Labels: RFC, Developer API
RFC Expose `xfail_checks` with a more flexible API

xref: https://github.com/scikit-learn/scikit-learn/pull/29818#issuecomment-2378967067

Right now we have the `tags._xfail_checks` which seems private since it has the leading underscore.

We're refactoring tests and making them more modular and much nicer to deal with, but still there are going to be cases where an estimator developer might want to skip a few tests, and not a whole category.

So the proposal here is to rename `_xfail_checks` to `xfail_checks` (with a deprecation cycle of one release?), and also add the ability for the developers to set the whether the tests should fail, warn, or be skipped/xfailed.

There's also the question of granularity: do we want to set the `warn/xfail/warn` to be set on the estimator level, or for each test?

Some alternatives could be:

# Option 1
```py
class MyEstimator(BaseEstimator):
    def __sklearn_tag__(self):
        tags = super().__sklearn_tag__()
        tags.xfail_checks = {
            "check_estimators_dtypes": ("some-error", "warn"/"skip"/"raise"),
        }
        return tags
```

# Option 2
```py
class MyEstimator(BaseEstimator):
    def __sklearn_tag__(self):
        tags = super().__sklearn_tag__()
        tags.xfailed_checks = "warn"/"skip"/"raise"
        tags.xfail_checks = {
            "check_estimators_dtypes": "some-error",
        }
        return tags
```

# Option 3
```py
class MyEstimator(BaseEstimator):
    def __sklearn_tag__(self):
        tags = super().__sklearn_tag__()
        tags.xfailed_checks = {
            "check_estimators_dtypes": "warn"/"skip"/"raise",
        }
        tags.xfail_checks = {
            "check_estimators_dtypes": "some-error",
        }
        return tags
```

cc @scikit-learn/core-devs  since it's public /developer API RFC

COMMENT:
So I like the suggestion in https://github.com/scikit-learn/scikit-learn/issues/29951#issuecomment-2379692391

But I think I'm leaning towards giving this info as an input to `parametrize_with_tests` and `check_estimator` rather than having it inside the tags. This would also go along with passing information about constructor args / data generator, etc, which I'll implement for those two methods / decorator.

Labels: RFC, Developer API
RFC Expose `xfail_checks` with a more flexible API

xref: https://github.com/scikit-learn/scikit-learn/pull/29818#issuecomment-2378967067

Right now we have the `tags._xfail_checks` which seems private since it has the leading underscore.

We're refactoring tests and making them more modular and much nicer to deal with, but still there are going to be cases where an estimator developer might want to skip a few tests, and not a whole category.

So the proposal here is to rename `_xfail_checks` to `xfail_checks` (with a deprecation cycle of one release?), and also add the ability for the developers to set the whether the tests should fail, warn, or be skipped/xfailed.

There's also the question of granularity: do we want to set the `warn/xfail/warn` to be set on the estimator level, or for each test?

Some alternatives could be:

# Option 1
```py
class MyEstimator(BaseEstimator):
    def __sklearn_tag__(self):
        tags = super().__sklearn_tag__()
        tags.xfail_checks = {
            "check_estimators_dtypes": ("some-error", "warn"/"skip"/"raise"),
        }
        return tags
```

# Option 2
```py
class MyEstimator(BaseEstimator):
    def __sklearn_tag__(self):
        tags = super().__sklearn_tag__()
        tags.xfailed_checks = "warn"/"skip"/"raise"
        tags.xfail_checks = {
            "check_estimators_dtypes": "some-error",
        }
        return tags
```

# Option 3
```py
class MyEstimator(BaseEstimator):
    def __sklearn_tag__(self):
        tags = super().__sklearn_tag__()
        tags.xfailed_checks = {
            "check_estimators_dtypes": "warn"/"skip"/"raise",
        }
        tags.xfail_checks = {
            "check_estimators_dtypes": "some-error",
        }
        return tags
```

cc @scikit-learn/core-devs  since it's public /developer API RFC

COMMENT:
@adrinjalali I am +1 on passing the `xfail` information directly into `parametrize_with_tests` and `check_estimator` in https://github.com/scikit-learn/scikit-learn/pull/30149.

Labels: RFC, Developer API
⚠️ CI failed on Linux_Nightly.pylatest_pip_scipy_dev (last failure: Sep 25, 2024) ⚠️

**CI failed on [Linux_Nightly.pylatest_pip_scipy_dev](https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=70481&view=logs&j=dfe99b15-50db-5d7b-b1e9-4105c42527cf)** (Sep 25, 2024)
Unable to find junit file. Please see link for details.

COMMENT:
## CI is no longer failing! ✅

[Successful run](https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=70495&view=logs&j=dfe99b15-50db-5d7b-b1e9-4105c42527cf) on Sep 27, 2024

Labels: Needs Triage
⚠️ CI failed on Linux_Nightly.pylatest_pip_scipy_dev (last failure: Sep 25, 2024) ⚠️

**CI failed on [Linux_Nightly.pylatest_pip_scipy_dev](https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=70481&view=logs&j=dfe99b15-50db-5d7b-b1e9-4105c42527cf)** (Sep 25, 2024)
Unable to find junit file. Please see link for details.

COMMENT:
404 for the scipy nightly wheel but it works now. I think we can safely close this one

Labels: Needs Triage
Remove sokalmichener from distance metrics

SciPy is planning to remove `sokalmichener`: https://github.com/scipy/scipy/pull/21572

We reimplement `SokalMichenerDistance` in the distance metric, and it's exactly the same as the implementation `RogersTanimotoDistance`. We can follow SciPy's lead and remove `sokalmichener` as well.

REF:

https://github.com/scikit-learn/scikit-learn/blob/74a33757c8a8df84d227f28bbc9ec7ae2fb51dea/sklearn/metrics/_dist_metrics.pyx.tp#L2308

https://github.com/scikit-learn/scikit-learn/blob/74a33757c8a8df84d227f28bbc9ec7ae2fb51dea/sklearn/metrics/_dist_metrics.pyx.tp#L2455

COMMENT:
Thanks for opening the issue. Let me share more background from SciPy’s side.

The details are explained in https://github.com/scipy/scipy/issues/2011. In short, the two problems with `sokalmichener` are that (1) its implementation is equivalent to  `rogerstanimoto`, and (2) the standard definition is equivalent to `matching`.

The reason for deprecating the function, rather than changing its behavior, is to force the end user to make an informed choice (for `rogerstanimoto` or for `matching`).

Labels: API, module:metrics
`**params` documentation for `GridSearchCV.fit` is ambiguous

[`GridSearchCV.fit`](https://scikit-learn.org/dev/modules/generated/sklearn.model_selection.GridSearchCV.html#sklearn.model_selection.GridSearchCV.fit)

### Describe the issue linked to the documentation

The documentation for the `**params` parameter to the `fit` method of `GridSearchCV` leads to confusion. Here is the current text:


> Parameters passed to the `fit` method of the estimator, the scorer, and the CV splitter.
> 
> If a fit parameter is an array-like whose length is equal to `num_samples` then it will be split across CV groups along with `X` and `y`. For example, the [sample_weight](https://scikit-learn.org/dev/glossary.html#term-sample_weight) parameter is split because `len(sample_weights) = len(X)`.

I was worried that this meant that `grid_search.fit(X, y, groups=g)` would split `g` up across the CV partitions, which is definitely not the right behavior. The correct behavior is to pass the `groups` parameter unchanged to the CV splitter, e.g. `cv.split(X, y, groups=groups)`. I read through the source code and it does appear that the `groups` parameter will get passed through unchanged to `split`, so it looks like the behavior is correct. But we could use something in the docstring that clarifies this behavior.

### Suggest a potential alternative/fix

_No response_

COMMENT:
Yep I guess it would make sense to mention that `groups` is special and is only passed to the splitter? PR more than welcome!

Labels: Easy, Documentation, help wanted
`**params` documentation for `GridSearchCV.fit` is ambiguous

[`GridSearchCV.fit`](https://scikit-learn.org/dev/modules/generated/sklearn.model_selection.GridSearchCV.html#sklearn.model_selection.GridSearchCV.fit)

### Describe the issue linked to the documentation

The documentation for the `**params` parameter to the `fit` method of `GridSearchCV` leads to confusion. Here is the current text:


> Parameters passed to the `fit` method of the estimator, the scorer, and the CV splitter.
> 
> If a fit parameter is an array-like whose length is equal to `num_samples` then it will be split across CV groups along with `X` and `y`. For example, the [sample_weight](https://scikit-learn.org/dev/glossary.html#term-sample_weight) parameter is split because `len(sample_weights) = len(X)`.

I was worried that this meant that `grid_search.fit(X, y, groups=g)` would split `g` up across the CV partitions, which is definitely not the right behavior. The correct behavior is to pass the `groups` parameter unchanged to the CV splitter, e.g. `cv.split(X, y, groups=groups)`. I read through the source code and it does appear that the `groups` parameter will get passed through unchanged to `split`, so it looks like the behavior is correct. But we could use something in the docstring that clarifies this behavior.

### Suggest a potential alternative/fix

_No response_

COMMENT:
Will be working on this.

Labels: Easy, Documentation, help wanted
Training final model with cross validation and using it to get unbiased probabilities

### Describe the workflow you want to enable

I want to use crossvalidation with let's say k=4 in order to get four models. That means that each sample in my dataset was used to train 3 of the four models. Thus, if I want to get a prediction for a given sample, I need to use the one model that was not trained with that particular sample. 

However, when I train my models and I get the four pickle files, I cannot pick the right model in an out-of-the-box way. I.e. there is no way to tell which model was not trained with that specific sample.

### Describe your proposed solution

The models could be loaded in some sort of model interface, whiich would take care of picking the right model for the prediction:

```python
df_feat = _get_features()
model_wrapper = load_models('/path/to/models/model*.pkl')
l_prob = mode_wrapper.predict_proba(df_feat)
```

This would mean that the model would have to somehow know that a given feature was used to train it, like:

```python
prob = None
for model in l_model:
    if model.used_feature(sr_feat):
        continue

    prob=model.predict_proba(sr_feat)
```

Which would mean that the model would get pretty large, unless each feature gets stored in a sort of short way within the model. E.g. storing some sort of hash for the whole training dataset as an attribute.

### Describe alternatives you've considered, if relevant

I can implement this myself using derived classes, a hash check. However it would be good to have it done upstream, or maybe it already exists?

### Additional context

The way I understand this is done in the real world is that cross validation is used to validate the model against overtraining. Once the model is validated, one trains the model again with the whole dataset, instead of e.g. 75% of it as in the example above. Thus one uses only one model instead of 4.

However in particle physics, people do not like to use a model trained with sample `S1` to make predictions for sample `S1`. That is regarded as introduced biases, and we need to reuse the datasets that were used for training also for analysis. Thus we need a more user-friendly way of doing this picking.

COMMENT:
Sorry, some individual parts make sense but I am not 100% sure what you are trying to do ...

Can you share some code showing what you are currently doing with scikit-learn and which limitation you are hitting? Are you already using something like `sklearn.model_selection.cross_val_score` / `sklearn.model_selection.cross_validate` or `sklearn.model_selection.GridSearchCV`?

Labels: New Feature, Needs Info
Training final model with cross validation and using it to get unbiased probabilities

### Describe the workflow you want to enable

I want to use crossvalidation with let's say k=4 in order to get four models. That means that each sample in my dataset was used to train 3 of the four models. Thus, if I want to get a prediction for a given sample, I need to use the one model that was not trained with that particular sample. 

However, when I train my models and I get the four pickle files, I cannot pick the right model in an out-of-the-box way. I.e. there is no way to tell which model was not trained with that specific sample.

### Describe your proposed solution

The models could be loaded in some sort of model interface, whiich would take care of picking the right model for the prediction:

```python
df_feat = _get_features()
model_wrapper = load_models('/path/to/models/model*.pkl')
l_prob = mode_wrapper.predict_proba(df_feat)
```

This would mean that the model would have to somehow know that a given feature was used to train it, like:

```python
prob = None
for model in l_model:
    if model.used_feature(sr_feat):
        continue

    prob=model.predict_proba(sr_feat)
```

Which would mean that the model would get pretty large, unless each feature gets stored in a sort of short way within the model. E.g. storing some sort of hash for the whole training dataset as an attribute.

### Describe alternatives you've considered, if relevant

I can implement this myself using derived classes, a hash check. However it would be good to have it done upstream, or maybe it already exists?

### Additional context

The way I understand this is done in the real world is that cross validation is used to validate the model against overtraining. Once the model is validated, one trains the model again with the whole dataset, instead of e.g. 75% of it as in the example above. Thus one uses only one model instead of 4.

However in particle physics, people do not like to use a model trained with sample `S1` to make predictions for sample `S1`. That is regarded as introduced biases, and we need to reuse the datasets that were used for training also for analysis. Thus we need a more user-friendly way of doing this picking.

COMMENT:
You can simply use [`cross_val_predict`](https://scikit-learn.org/dev/modules/generated/sklearn.model_selection.cross_val_predict.html#sklearn.model_selection.cross_val_predict), but if you want to manually run CV and store all those models, then you'd need to also have your own wrappers to know which fold / data was used for which model, to load the right one. This is outside the scope of scikit-learn.

I personally have had this exact usecase in the past in a medical use-case, and I'd store the information about which data is used in which model.

Labels: New Feature, Needs Info
ImportError: cannot import name 'InconsistentVersionWarning' in sklearn.exceptions

### Describe the bug

The error message "ImportError: cannot import name 'InconsistentVersionWarning'“ occurs when there is an attempt to import the sklearn

### Steps/Code to Reproduce

import sklearn

### Expected Results

successful import

### Actual Results

ImportError: cannot import name 'InconsistentVersionWarning' from 'sklearn.exceptions' (/path/to/sklearn/exceptions.py)

### Versions

```shell
1.5.2
```

COMMENT:
Can you provide the full traceback. Basically, we do more that just importing scikit-learn in our CIs so we would have catch this issue.

So having the traceback will allow to understand what part is trying to import it.

Labels: Bug, Needs Triage