Talip7/scikit-learn-issues-multilabel huggingface dataset:
RFE results are inconsistent between machines with ties in feature importances at threshold

### Describe the bug

RFE uses np.argsort on the feature_importances from the estimator, this is not repeatable across machines. This only matters when there are ties in the feature importances that overlap with the threshold. For example if the importances are [0, 2, 0, 1] and it needs to eliminate 1 feature, it may not choose the same feature to eliminate on all machines.

### Steps/Code to Reproduce

It is difficult to write code that will show this as it requires multiple machines to test; however, the following test could be added to the _rfe unit tests to check if the sort is stable, though the sort could be consistent but not stable, so this isn't enough to prove an issue.
```
def test_rfe_ties_around_threshold():
    X, y = make_classification(n_features=47, random_state=0)
    clf = MockClassifier() # mock classifier returns constant feature_importances
    rfe = RFE(estimator=clf, n_features_to_select=4, step=2)
    rfe.fit(X, y)

    assert_array_equal(rfe.support_ ,np.array([*[False]*43, *[True]*4]))
```

### Expected Results

Expected to get the same results on all machines, being stable is not necessary though it is an easy way to enforce testable consistency.

### Actual Results

Current sort does not result in a stable sort and fails the previously provided unit test.

### Versions

```shell
main branch
```


This issue is most likely to occur with zero importance features, especially at the initial steps of RFE.

COMMENT:
Another thought on this. I think getting consistent results in the different machines with the same versions is pretty important. But stable sort is not the only or best way to do this.

While in the GroupKFold case it may not make sense to add an option, for RFE what if there was an `on_ties` option. I could imagine at least 3 options.
1. Current solution, not guaranteed to be consistent between machines, good enough for many probably.
2. Stable sort, consistent, slower, and biased to keep either the first or last features depending on the direction implemented.
3. Random, take a random seed and shuffle the ties. Even slower, but repeatable and no bias. Could be implemented by adding a randomly shuffled `np.arange` as the second key to a `np.lexsort` instead of argsort.

Labels: Bug
RFE results are inconsistent between machines with ties in feature importances at threshold

### Describe the bug

RFE uses np.argsort on the feature_importances from the estimator, this is not repeatable across machines. This only matters when there are ties in the feature importances that overlap with the threshold. For example if the importances are [0, 2, 0, 1] and it needs to eliminate 1 feature, it may not choose the same feature to eliminate on all machines.

### Steps/Code to Reproduce

It is difficult to write code that will show this as it requires multiple machines to test; however, the following test could be added to the _rfe unit tests to check if the sort is stable, though the sort could be consistent but not stable, so this isn't enough to prove an issue.
```
def test_rfe_ties_around_threshold():
    X, y = make_classification(n_features=47, random_state=0)
    clf = MockClassifier() # mock classifier returns constant feature_importances
    rfe = RFE(estimator=clf, n_features_to_select=4, step=2)
    rfe.fit(X, y)

    assert_array_equal(rfe.support_ ,np.array([*[False]*43, *[True]*4]))
```

### Expected Results

Expected to get the same results on all machines, being stable is not necessary though it is an easy way to enforce testable consistency.

### Actual Results

Current sort does not result in a stable sort and fails the previously provided unit test.

### Versions

```shell
main branch
```


This issue is most likely to occur with zero importance features, especially at the initial steps of RFE.

COMMENT:
So in general, we do not guarantee the same results between machines. Different machines for a broad range of reasons will give different results.

So my question is, why is this particular one so important, given that we'd get a performance hit as a result? The same goes for the other issues you've opened.

Unless we have a very clear motivation, I think we can close them as "won'tfix" since we are already giving different results across machines anyway.

Labels: Bug
RFE results are inconsistent between machines with ties in feature importances at threshold

### Describe the bug

RFE uses np.argsort on the feature_importances from the estimator, this is not repeatable across machines. This only matters when there are ties in the feature importances that overlap with the threshold. For example if the importances are [0, 2, 0, 1] and it needs to eliminate 1 feature, it may not choose the same feature to eliminate on all machines.

### Steps/Code to Reproduce

It is difficult to write code that will show this as it requires multiple machines to test; however, the following test could be added to the _rfe unit tests to check if the sort is stable, though the sort could be consistent but not stable, so this isn't enough to prove an issue.
```
def test_rfe_ties_around_threshold():
    X, y = make_classification(n_features=47, random_state=0)
    clf = MockClassifier() # mock classifier returns constant feature_importances
    rfe = RFE(estimator=clf, n_features_to_select=4, step=2)
    rfe.fit(X, y)

    assert_array_equal(rfe.support_ ,np.array([*[False]*43, *[True]*4]))
```

### Expected Results

Expected to get the same results on all machines, being stable is not necessary though it is an easy way to enforce testable consistency.

### Actual Results

Current sort does not result in a stable sort and fails the previously provided unit test.

### Versions

```shell
main branch
```


This issue is most likely to occur with zero importance features, especially at the initial steps of RFE.

COMMENT:
> So in general, we do not guarantee the same results between machines. Different machines for a broad range of reasons will give different results.
> 
> So my question is, why is this particular one so important, given that we'd get a performance hit as a result? The same goes for the other issues you've opened.
> 
> Unless we have a very clear motivation, I think we can close them as "won'tfix" since we are already giving different results across machines anyway.

I see, I wasn't familiar with this lack of consistency between machines in general for scikit-learn.

For my model we were required to show reproducibility and we run on Kubeflow (so different machines each time) and both the RFE and GroupKFold sorts caused changes in the final selected features and hyperparameters.

I thought these may be issues others would run into, so I proposed the fix. If not relevant, then I will just extend the classes with the fix on my end.

Labels: Bug
RFE results are inconsistent between machines with ties in feature importances at threshold

### Describe the bug

RFE uses np.argsort on the feature_importances from the estimator, this is not repeatable across machines. This only matters when there are ties in the feature importances that overlap with the threshold. For example if the importances are [0, 2, 0, 1] and it needs to eliminate 1 feature, it may not choose the same feature to eliminate on all machines.

### Steps/Code to Reproduce

It is difficult to write code that will show this as it requires multiple machines to test; however, the following test could be added to the _rfe unit tests to check if the sort is stable, though the sort could be consistent but not stable, so this isn't enough to prove an issue.
```
def test_rfe_ties_around_threshold():
    X, y = make_classification(n_features=47, random_state=0)
    clf = MockClassifier() # mock classifier returns constant feature_importances
    rfe = RFE(estimator=clf, n_features_to_select=4, step=2)
    rfe.fit(X, y)

    assert_array_equal(rfe.support_ ,np.array([*[False]*43, *[True]*4]))
```

### Expected Results

Expected to get the same results on all machines, being stable is not necessary though it is an easy way to enforce testable consistency.

### Actual Results

Current sort does not result in a stable sort and fails the previously provided unit test.

### Versions

```shell
main branch
```


This issue is most likely to occur with zero importance features, especially at the initial steps of RFE.

COMMENT:
Thanks for the context.

> RFE and GroupKFold sorts caused changes in the final selected features and hyperparameters.

Wouldn't that mean you shouldn't rely too much on the fact that those particular features are selected? In practice, you could try running the same pipeline a few times, changing your random seed or shuffling the data, and try to look at the variance in your hyper parameters or see how often certain features are selected or if the ones being switched are correlated with one another and other features in some way.

I personally would probably go down that route instead of trying to make things 100% replicable.

cc @glemaitre @lesteve @adam2392 , I don't think we'd want to add this due to the performance cost, since I'm not sure of the gain here, and the same for the GroupKFold issue.

Labels: Bug
RFE results are inconsistent between machines with ties in feature importances at threshold

### Describe the bug

RFE uses np.argsort on the feature_importances from the estimator, this is not repeatable across machines. This only matters when there are ties in the feature importances that overlap with the threshold. For example if the importances are [0, 2, 0, 1] and it needs to eliminate 1 feature, it may not choose the same feature to eliminate on all machines.

### Steps/Code to Reproduce

It is difficult to write code that will show this as it requires multiple machines to test; however, the following test could be added to the _rfe unit tests to check if the sort is stable, though the sort could be consistent but not stable, so this isn't enough to prove an issue.
```
def test_rfe_ties_around_threshold():
    X, y = make_classification(n_features=47, random_state=0)
    clf = MockClassifier() # mock classifier returns constant feature_importances
    rfe = RFE(estimator=clf, n_features_to_select=4, step=2)
    rfe.fit(X, y)

    assert_array_equal(rfe.support_ ,np.array([*[False]*43, *[True]*4]))
```

### Expected Results

Expected to get the same results on all machines, being stable is not necessary though it is an easy way to enforce testable consistency.

### Actual Results

Current sort does not result in a stable sort and fails the previously provided unit test.

### Versions

```shell
main branch
```


This issue is most likely to occur with zero importance features, especially at the initial steps of RFE.

COMMENT:
Having reproducible pipelines (at least via an option) can be pragmatically helpful. This can be helpful for caching stuff for instance, but also to make sure that you can easily isolate sources of variability that are expected (tied feature importance values used for feature selection) from those that are not (e.g. bugs in custom feature engineering code).

From a statistical point of view however, I agree with Adrin that you should not draw conclusions that are significantly impacted by the way you break ties (e.g. based on the original feature order when using stable-sorting).

One way to proceed could be to add a `random_state` argument to `RFE` and (always?) break ties randomly by adding some machine precision level random jitter on the feature importance values such that ties are deterministically broken by a user-settable hyper-parameter. This would then make it easy to run a statistical analysis of the impact of tie-breaking.

We have documented other problems related to our lack of random tie breaking in:

- https://github.com/scikit-learn/scikit-learn/issues/26428 (in hist gradient boosting classifier/regressor)
- https://github.com/scikit-learn/scikit-learn/issues/26428#issuecomment-1562885651 (in coordinate descent solvers).

Labels: Bug
RFE results are inconsistent between machines with ties in feature importances at threshold

### Describe the bug

RFE uses np.argsort on the feature_importances from the estimator, this is not repeatable across machines. This only matters when there are ties in the feature importances that overlap with the threshold. For example if the importances are [0, 2, 0, 1] and it needs to eliminate 1 feature, it may not choose the same feature to eliminate on all machines.

### Steps/Code to Reproduce

It is difficult to write code that will show this as it requires multiple machines to test; however, the following test could be added to the _rfe unit tests to check if the sort is stable, though the sort could be consistent but not stable, so this isn't enough to prove an issue.
```
def test_rfe_ties_around_threshold():
    X, y = make_classification(n_features=47, random_state=0)
    clf = MockClassifier() # mock classifier returns constant feature_importances
    rfe = RFE(estimator=clf, n_features_to_select=4, step=2)
    rfe.fit(X, y)

    assert_array_equal(rfe.support_ ,np.array([*[False]*43, *[True]*4]))
```

### Expected Results

Expected to get the same results on all machines, being stable is not necessary though it is an easy way to enforce testable consistency.

### Actual Results

Current sort does not result in a stable sort and fails the previously provided unit test.

### Versions

```shell
main branch
```


This issue is most likely to occur with zero importance features, especially at the initial steps of RFE.

COMMENT:
I'd be happy with what @ogrisel suggests here. WDYT @blitchj ?

Labels: Bug
RFE results are inconsistent between machines with ties in feature importances at threshold

### Describe the bug

RFE uses np.argsort on the feature_importances from the estimator, this is not repeatable across machines. This only matters when there are ties in the feature importances that overlap with the threshold. For example if the importances are [0, 2, 0, 1] and it needs to eliminate 1 feature, it may not choose the same feature to eliminate on all machines.

### Steps/Code to Reproduce

It is difficult to write code that will show this as it requires multiple machines to test; however, the following test could be added to the _rfe unit tests to check if the sort is stable, though the sort could be consistent but not stable, so this isn't enough to prove an issue.
```
def test_rfe_ties_around_threshold():
    X, y = make_classification(n_features=47, random_state=0)
    clf = MockClassifier() # mock classifier returns constant feature_importances
    rfe = RFE(estimator=clf, n_features_to_select=4, step=2)
    rfe.fit(X, y)

    assert_array_equal(rfe.support_ ,np.array([*[False]*43, *[True]*4]))
```

### Expected Results

Expected to get the same results on all machines, being stable is not necessary though it is an easy way to enforce testable consistency.

### Actual Results

Current sort does not result in a stable sort and fails the previously provided unit test.

### Versions

```shell
main branch
```


This issue is most likely to occur with zero importance features, especially at the initial steps of RFE.

COMMENT:
> I'd be happy with what @ogrisel suggests here. WDYT @blitchj ?

Sure, I'll make a PR / update my current one with this.

Labels: Bug
RFE results are inconsistent between machines with ties in feature importances at threshold

### Describe the bug

RFE uses np.argsort on the feature_importances from the estimator, this is not repeatable across machines. This only matters when there are ties in the feature importances that overlap with the threshold. For example if the importances are [0, 2, 0, 1] and it needs to eliminate 1 feature, it may not choose the same feature to eliminate on all machines.

### Steps/Code to Reproduce

It is difficult to write code that will show this as it requires multiple machines to test; however, the following test could be added to the _rfe unit tests to check if the sort is stable, though the sort could be consistent but not stable, so this isn't enough to prove an issue.
```
def test_rfe_ties_around_threshold():
    X, y = make_classification(n_features=47, random_state=0)
    clf = MockClassifier() # mock classifier returns constant feature_importances
    rfe = RFE(estimator=clf, n_features_to_select=4, step=2)
    rfe.fit(X, y)

    assert_array_equal(rfe.support_ ,np.array([*[False]*43, *[True]*4]))
```

### Expected Results

Expected to get the same results on all machines, being stable is not necessary though it is an easy way to enforce testable consistency.

### Actual Results

Current sort does not result in a stable sort and fails the previously provided unit test.

### Versions

```shell
main branch
```


This issue is most likely to occur with zero importance features, especially at the initial steps of RFE.

COMMENT:
@ogrisel @adrinjalali For my first pass at implementing this, I'm trying to see what the maximum value for "machine precision level" uniform random should be, but depending on the size I choose I can hit limitations on how many floating points are representable for very large datasets this may collide. This would result in the same reproducibility problem as the current implementation, just much more rare, requiring both a tie in importance and a random number collision.

Another option would be to make an integer value arange as the second column to an `np.lexsort`. In this case there could be no collisions and only the tied feature importances would be shuffled, though I suspect it would be slower, given it is a stable sort on 2 columns now. I could see this as both desirable and not desirable, depending on the expected confidence in the raw feature importances. _If only I could sample from a distribution for each feature's importance..._

Do either of you have thoughts on the scale of the random number?

Labels: Bug
RFE results are inconsistent between machines with ties in feature importances at threshold

### Describe the bug

RFE uses np.argsort on the feature_importances from the estimator, this is not repeatable across machines. This only matters when there are ties in the feature importances that overlap with the threshold. For example if the importances are [0, 2, 0, 1] and it needs to eliminate 1 feature, it may not choose the same feature to eliminate on all machines.

### Steps/Code to Reproduce

It is difficult to write code that will show this as it requires multiple machines to test; however, the following test could be added to the _rfe unit tests to check if the sort is stable, though the sort could be consistent but not stable, so this isn't enough to prove an issue.
```
def test_rfe_ties_around_threshold():
    X, y = make_classification(n_features=47, random_state=0)
    clf = MockClassifier() # mock classifier returns constant feature_importances
    rfe = RFE(estimator=clf, n_features_to_select=4, step=2)
    rfe.fit(X, y)

    assert_array_equal(rfe.support_ ,np.array([*[False]*43, *[True]*4]))
```

### Expected Results

Expected to get the same results on all machines, being stable is not necessary though it is an easy way to enforce testable consistency.

### Actual Results

Current sort does not result in a stable sort and fails the previously provided unit test.

### Versions

```shell
main branch
```


This issue is most likely to occur with zero importance features, especially at the initial steps of RFE.

COMMENT:
Maybe this would be suitable? 

```py
np.random.rand(n_random) * np.finfo(float_dtype).eps
```

Labels: Bug
Community section: add link to GitHub discussions

### Describe the issue linked to the documentation

Can we add a link to GitHub discussions in the footer of the home page?
- home page: https://scikit-learn.org/stable/
- link to add: https://github.com/scikit-learn/scikit-learn/discussions

### Suggest a potential alternative/fix

_No response_

COMMENT:
I'm not sure if it's a good idea. We haven't really been seeing much community interaction on the discussions part and we're mostly the people who answer there, and I don't think we can handle a lot more load of questions there.

Labels: Documentation
Community section: add link to GitHub discussions

### Describe the issue linked to the documentation

Can we add a link to GitHub discussions in the footer of the home page?
- home page: https://scikit-learn.org/stable/
- link to add: https://github.com/scikit-learn/scikit-learn/discussions

### Suggest a potential alternative/fix

_No response_

COMMENT:
I am fine adding a link to Discussions personally, but yeah this is a slightly tricky decision.

Personally my take on Github Discussions:
- I rarely look at them
- I think as maintainers, we should feel completely free to not look at them and not feel responsible for answering them. Easier said than done I know.
- the line between when to open an issue and when to open an discussion is thin, and if the probability for feed-back is a lot smaller for discussions I guess the incentive lead towards  opening an issue ...

From time to time there is a user that gets motivated and answer a few questions but I agree it is quite rare, see one case when I remember it happening:
https://github.com/scikit-learn/scikit-learn/discussions?discussions_q=involves%3Ai-aki-y

BTW I learned today that there is a Github [Community Insights](https://github.com/scikit-learn/scikit-learn/graphs/community). Not sure what to make of it except that Discussions are not used too much currently:
![image](https://github.com/user-attachments/assets/8e716ae3-11ad-42ee-b2ee-e165533793a4)

Labels: Documentation
Community section: add link to GitHub discussions

### Describe the issue linked to the documentation

Can we add a link to GitHub discussions in the footer of the home page?
- home page: https://scikit-learn.org/stable/
- link to add: https://github.com/scikit-learn/scikit-learn/discussions

### Suggest a potential alternative/fix

_No response_

COMMENT:
+1 for adding a link.

Labels: Documentation