Talip7/scikit-learn-issues-multilabel huggingface dataset:
Bug in train_test_split

### Describe the bug

I am pretty confident I found a bug in train_test_split. When using train_test_split multiple times in a row it produces wrong indexes in X_train and y_train.

### Steps/Code to Reproduce

Using this code at the first time in Jupyter notebook produced the correct result with two identical indexes.

`X_train, X_test, y_train, y_test = train_test_split(df, target, test_size=0.2, random_state=seed)`

I confirmed this with:
`X_train.index.equals(y_train.index)`


After loading the dataset in the variable `df` again and getting the target with `target = np.log1p(df['SalePrice'])` I could confirm
that `df` and `target` had the right indeces in the right order before I used the function. After the function was done X_train
and y_train had changed indeces. So there seems to be a memory related problen, since the first usage worked just fine but all tries which are done after that produce the wrong result. Clearing all output and using a reset in Jupyter notebook solves the issue for one try again.


### Expected Results

Of course I would expect the right indeces.

### Actual Results

![image](https://user-images.githubusercontent.com/56065345/160486870-07c47d40-b957-4c91-8a48-963476b67494.png)
This is the actual result from X_train - if you compare it the picture below you can see that the first samples of the table have been altered but the last ones remain intact.
![image](https://user-images.githubusercontent.com/56065345/160486984-1c586281-d337-446d-94f2-e945c6fea1fd.png)




### Versions

```shell
0.24.2 scikit learn, python 3.9
```

COMMENT:
I found that when I sort the index and print it it seems to be the index from 0 - samplesize, with a few indexes missing ....

![image](https://user-images.githubusercontent.com/56065345/160488519-6cbb2ada-d78f-4129-a620-483271092fa2.png)

Labels: Bug, module:model_selection, Needs Reproducible Code
Bug in train_test_split

### Describe the bug

I am pretty confident I found a bug in train_test_split. When using train_test_split multiple times in a row it produces wrong indexes in X_train and y_train.

### Steps/Code to Reproduce

Using this code at the first time in Jupyter notebook produced the correct result with two identical indexes.

`X_train, X_test, y_train, y_test = train_test_split(df, target, test_size=0.2, random_state=seed)`

I confirmed this with:
`X_train.index.equals(y_train.index)`


After loading the dataset in the variable `df` again and getting the target with `target = np.log1p(df['SalePrice'])` I could confirm
that `df` and `target` had the right indeces in the right order before I used the function. After the function was done X_train
and y_train had changed indeces. So there seems to be a memory related problen, since the first usage worked just fine but all tries which are done after that produce the wrong result. Clearing all output and using a reset in Jupyter notebook solves the issue for one try again.


### Expected Results

Of course I would expect the right indeces.

### Actual Results

![image](https://user-images.githubusercontent.com/56065345/160486870-07c47d40-b957-4c91-8a48-963476b67494.png)
This is the actual result from X_train - if you compare it the picture below you can see that the first samples of the table have been altered but the last ones remain intact.
![image](https://user-images.githubusercontent.com/56065345/160486984-1c586281-d337-446d-94f2-e945c6fea1fd.png)




### Versions

```shell
0.24.2 scikit learn, python 3.9
```

COMMENT:
Thanks for the report @nilslacroix. Could you come up with a minimal reproducer that exhibits the issue you're facing (https://scikit-learn.org/dev/developers/minimal_reproducer.html#minimal-reproducer) ?

Labels: Bug, module:model_selection, Needs Reproducible Code
Bug in train_test_split

### Describe the bug

I am pretty confident I found a bug in train_test_split. When using train_test_split multiple times in a row it produces wrong indexes in X_train and y_train.

### Steps/Code to Reproduce

Using this code at the first time in Jupyter notebook produced the correct result with two identical indexes.

`X_train, X_test, y_train, y_test = train_test_split(df, target, test_size=0.2, random_state=seed)`

I confirmed this with:
`X_train.index.equals(y_train.index)`


After loading the dataset in the variable `df` again and getting the target with `target = np.log1p(df['SalePrice'])` I could confirm
that `df` and `target` had the right indeces in the right order before I used the function. After the function was done X_train
and y_train had changed indeces. So there seems to be a memory related problen, since the first usage worked just fine but all tries which are done after that produce the wrong result. Clearing all output and using a reset in Jupyter notebook solves the issue for one try again.


### Expected Results

Of course I would expect the right indeces.

### Actual Results

![image](https://user-images.githubusercontent.com/56065345/160486870-07c47d40-b957-4c91-8a48-963476b67494.png)
This is the actual result from X_train - if you compare it the picture below you can see that the first samples of the table have been altered but the last ones remain intact.
![image](https://user-images.githubusercontent.com/56065345/160486984-1c586281-d337-446d-94f2-e945c6fea1fd.png)




### Versions

```shell
0.24.2 scikit learn, python 3.9
```

COMMENT:
Hi I tried to reproduce the error with another dataset but it works just fine with toy examples- I am really baffled here.
As you can see in the screenshot I feed two pandas Dataframes with a Int64Index into the function. Before I do this the indeces
are correctly aligned.

But after train_test_split is called, y_train (and y_test as well) are reindexed beginning at 0 and randomly mixed, which
results in a False index comparison.

![image](https://user-images.githubusercontent.com/56065345/160825746-bf285d52-2b5e-436a-9072-495ef756d9e3.png)

For comparison reason I wrote this reproducer, but this yields correct indeces. This is even more strange since the only difference is the dataframe, which is fed into the function. Maybe you guys have a clue why this could happen?

```py
#load dataset
from sklearn.datasets import load_diabetes
import pandas as pd
X, y = load_diabetes(return_X_y=True)
#convert to pandas, since I use pandas dataframes
X = pd.DataFrame(X)
y = pd.DataFrame(y)
#simulate my randomized dataset which is put into train_test_split
#randomize both dataframes with the same seed, so the index is in the same order
X = X.sample(frac = 1 , random_state=seed)
y = y.sample(frac = 1 , random_state=seed)
#print if index is equal before train_test_split
print(X.index.equals(y.index))
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=seed)
print(X_train.index.equals(y_train.index))
```

Labels: Bug, module:model_selection, Needs Reproducible Code
Bug in train_test_split

### Describe the bug

I am pretty confident I found a bug in train_test_split. When using train_test_split multiple times in a row it produces wrong indexes in X_train and y_train.

### Steps/Code to Reproduce

Using this code at the first time in Jupyter notebook produced the correct result with two identical indexes.

`X_train, X_test, y_train, y_test = train_test_split(df, target, test_size=0.2, random_state=seed)`

I confirmed this with:
`X_train.index.equals(y_train.index)`


After loading the dataset in the variable `df` again and getting the target with `target = np.log1p(df['SalePrice'])` I could confirm
that `df` and `target` had the right indeces in the right order before I used the function. After the function was done X_train
and y_train had changed indeces. So there seems to be a memory related problen, since the first usage worked just fine but all tries which are done after that produce the wrong result. Clearing all output and using a reset in Jupyter notebook solves the issue for one try again.


### Expected Results

Of course I would expect the right indeces.

### Actual Results

![image](https://user-images.githubusercontent.com/56065345/160486870-07c47d40-b957-4c91-8a48-963476b67494.png)
This is the actual result from X_train - if you compare it the picture below you can see that the first samples of the table have been altered but the last ones remain intact.
![image](https://user-images.githubusercontent.com/56065345/160486984-1c586281-d337-446d-94f2-e945c6fea1fd.png)




### Versions

```shell
0.24.2 scikit learn, python 3.9
```

COMMENT:
This issue is hard to debug without a reproducer. If you can not share your dataset, can you write a reproducer that generates a Dataframe around the size of your dataset to reproduce your issue?

Labels: Bug, module:model_selection, Needs Reproducible Code
Bug in train_test_split

### Describe the bug

I am pretty confident I found a bug in train_test_split. When using train_test_split multiple times in a row it produces wrong indexes in X_train and y_train.

### Steps/Code to Reproduce

Using this code at the first time in Jupyter notebook produced the correct result with two identical indexes.

`X_train, X_test, y_train, y_test = train_test_split(df, target, test_size=0.2, random_state=seed)`

I confirmed this with:
`X_train.index.equals(y_train.index)`


After loading the dataset in the variable `df` again and getting the target with `target = np.log1p(df['SalePrice'])` I could confirm
that `df` and `target` had the right indeces in the right order before I used the function. After the function was done X_train
and y_train had changed indeces. So there seems to be a memory related problen, since the first usage worked just fine but all tries which are done after that produce the wrong result. Clearing all output and using a reset in Jupyter notebook solves the issue for one try again.


### Expected Results

Of course I would expect the right indeces.

### Actual Results

![image](https://user-images.githubusercontent.com/56065345/160486870-07c47d40-b957-4c91-8a48-963476b67494.png)
This is the actual result from X_train - if you compare it the picture below you can see that the first samples of the table have been altered but the last ones remain intact.
![image](https://user-images.githubusercontent.com/56065345/160486984-1c586281-d337-446d-94f2-e945c6fea1fd.png)




### Versions

```shell
0.24.2 scikit learn, python 3.9
```

COMMENT:
Sorry just realised I wrote the same thing again. I can not reproduce the error to be honest, not with some toy df. It is also size independent because the error occasionally happens when I sample only like 100 samples from the whole dataframe. I really think this is memory related or some strange jupyter notebook bug, since it works again as soon as I restart and clear the notebook.

I just do not see any other way since I compare the indeces right before and after calling the function

Labels: Bug, module:model_selection, Needs Reproducible Code
Bug in train_test_split

### Describe the bug

I am pretty confident I found a bug in train_test_split. When using train_test_split multiple times in a row it produces wrong indexes in X_train and y_train.

### Steps/Code to Reproduce

Using this code at the first time in Jupyter notebook produced the correct result with two identical indexes.

`X_train, X_test, y_train, y_test = train_test_split(df, target, test_size=0.2, random_state=seed)`

I confirmed this with:
`X_train.index.equals(y_train.index)`


After loading the dataset in the variable `df` again and getting the target with `target = np.log1p(df['SalePrice'])` I could confirm
that `df` and `target` had the right indeces in the right order before I used the function. After the function was done X_train
and y_train had changed indeces. So there seems to be a memory related problen, since the first usage worked just fine but all tries which are done after that produce the wrong result. Clearing all output and using a reset in Jupyter notebook solves the issue for one try again.


### Expected Results

Of course I would expect the right indeces.

### Actual Results

![image](https://user-images.githubusercontent.com/56065345/160486870-07c47d40-b957-4c91-8a48-963476b67494.png)
This is the actual result from X_train - if you compare it the picture below you can see that the first samples of the table have been altered but the last ones remain intact.
![image](https://user-images.githubusercontent.com/56065345/160486984-1c586281-d337-446d-94f2-e945c6fea1fd.png)




### Versions

```shell
0.24.2 scikit learn, python 3.9
```

COMMENT:
I found the culprit.  I used Intel® Extension for Scikit-learn to speed up the computation. Turns out that when I import it with 

```
from sklearnex import patch_sklearn
patch_sklearn()
```
this bug appears. And even others, for example I just saw that the scoring method of SVR with a poly kernel produced different scores a few seconds later with the same X, y sets , without changing the parameters.

Labels: Bug, module:model_selection, Needs Reproducible Code
Bug in train_test_split

### Describe the bug

I am pretty confident I found a bug in train_test_split. When using train_test_split multiple times in a row it produces wrong indexes in X_train and y_train.

### Steps/Code to Reproduce

Using this code at the first time in Jupyter notebook produced the correct result with two identical indexes.

`X_train, X_test, y_train, y_test = train_test_split(df, target, test_size=0.2, random_state=seed)`

I confirmed this with:
`X_train.index.equals(y_train.index)`


After loading the dataset in the variable `df` again and getting the target with `target = np.log1p(df['SalePrice'])` I could confirm
that `df` and `target` had the right indeces in the right order before I used the function. After the function was done X_train
and y_train had changed indeces. So there seems to be a memory related problen, since the first usage worked just fine but all tries which are done after that produce the wrong result. Clearing all output and using a reset in Jupyter notebook solves the issue for one try again.


### Expected Results

Of course I would expect the right indeces.

### Actual Results

![image](https://user-images.githubusercontent.com/56065345/160486870-07c47d40-b957-4c91-8a48-963476b67494.png)
This is the actual result from X_train - if you compare it the picture below you can see that the first samples of the table have been altered but the last ones remain intact.
![image](https://user-images.githubusercontent.com/56065345/160486984-1c586281-d337-446d-94f2-e945c6fea1fd.png)




### Versions

```shell
0.24.2 scikit learn, python 3.9
```

COMMENT:
I am closing this issue because it is a `sklearnex` specific issue. Thank you for looking into the issue more.

Labels: Bug, module:model_selection, Needs Reproducible Code
Bug in train_test_split

### Describe the bug

I am pretty confident I found a bug in train_test_split. When using train_test_split multiple times in a row it produces wrong indexes in X_train and y_train.

### Steps/Code to Reproduce

Using this code at the first time in Jupyter notebook produced the correct result with two identical indexes.

`X_train, X_test, y_train, y_test = train_test_split(df, target, test_size=0.2, random_state=seed)`

I confirmed this with:
`X_train.index.equals(y_train.index)`


After loading the dataset in the variable `df` again and getting the target with `target = np.log1p(df['SalePrice'])` I could confirm
that `df` and `target` had the right indeces in the right order before I used the function. After the function was done X_train
and y_train had changed indeces. So there seems to be a memory related problen, since the first usage worked just fine but all tries which are done after that produce the wrong result. Clearing all output and using a reset in Jupyter notebook solves the issue for one try again.


### Expected Results

Of course I would expect the right indeces.

### Actual Results

![image](https://user-images.githubusercontent.com/56065345/160486870-07c47d40-b957-4c91-8a48-963476b67494.png)
This is the actual result from X_train - if you compare it the picture below you can see that the first samples of the table have been altered but the last ones remain intact.
![image](https://user-images.githubusercontent.com/56065345/160486984-1c586281-d337-446d-94f2-e945c6fea1fd.png)




### Versions

```shell
0.24.2 scikit learn, python 3.9
```

COMMENT:
Should we have a tag for issues that are due to this package rather than sklearn?

Labels: Bug, module:model_selection, Needs Reproducible Code
Bug in train_test_split

### Describe the bug

I am pretty confident I found a bug in train_test_split. When using train_test_split multiple times in a row it produces wrong indexes in X_train and y_train.

### Steps/Code to Reproduce

Using this code at the first time in Jupyter notebook produced the correct result with two identical indexes.

`X_train, X_test, y_train, y_test = train_test_split(df, target, test_size=0.2, random_state=seed)`

I confirmed this with:
`X_train.index.equals(y_train.index)`


After loading the dataset in the variable `df` again and getting the target with `target = np.log1p(df['SalePrice'])` I could confirm
that `df` and `target` had the right indeces in the right order before I used the function. After the function was done X_train
and y_train had changed indeces. So there seems to be a memory related problen, since the first usage worked just fine but all tries which are done after that produce the wrong result. Clearing all output and using a reset in Jupyter notebook solves the issue for one try again.


### Expected Results

Of course I would expect the right indeces.

### Actual Results

![image](https://user-images.githubusercontent.com/56065345/160486870-07c47d40-b957-4c91-8a48-963476b67494.png)
This is the actual result from X_train - if you compare it the picture below you can see that the first samples of the table have been altered but the last ones remain intact.
![image](https://user-images.githubusercontent.com/56065345/160486984-1c586281-d337-446d-94f2-e945c6fea1fd.png)




### Versions

```shell
0.24.2 scikit learn, python 3.9
```

COMMENT:
> Should we have a tag for issues that are due to this package rather than sklearn?

In recent memory, I think this is the first issue related to `sklearnex`. To me, this means not to label it.

If we really want to label this issue, there is a more generic class of issues: "upstream library". Depending on the issue, it may or may not be an issue with scikit-learn itself.

Labels: Bug, module:model_selection, Needs Reproducible Code
Bug in train_test_split

### Describe the bug

I am pretty confident I found a bug in train_test_split. When using train_test_split multiple times in a row it produces wrong indexes in X_train and y_train.

### Steps/Code to Reproduce

Using this code at the first time in Jupyter notebook produced the correct result with two identical indexes.

`X_train, X_test, y_train, y_test = train_test_split(df, target, test_size=0.2, random_state=seed)`

I confirmed this with:
`X_train.index.equals(y_train.index)`


After loading the dataset in the variable `df` again and getting the target with `target = np.log1p(df['SalePrice'])` I could confirm
that `df` and `target` had the right indeces in the right order before I used the function. After the function was done X_train
and y_train had changed indeces. So there seems to be a memory related problen, since the first usage worked just fine but all tries which are done after that produce the wrong result. Clearing all output and using a reset in Jupyter notebook solves the issue for one try again.


### Expected Results

Of course I would expect the right indeces.

### Actual Results

![image](https://user-images.githubusercontent.com/56065345/160486870-07c47d40-b957-4c91-8a48-963476b67494.png)
This is the actual result from X_train - if you compare it the picture below you can see that the first samples of the table have been altered but the last ones remain intact.
![image](https://user-images.githubusercontent.com/56065345/160486984-1c586281-d337-446d-94f2-e945c6fea1fd.png)




### Versions

```shell
0.24.2 scikit learn, python 3.9
```

COMMENT:
As an update: This is related to sklearnex for sure. The problem is hardware related. When using AMD CPUs, the functions and class which are replaced by sklearnex (for example train_test_split and estimators) producte wrong indeces and inconsistent scores instead of giving a warning. I think thats the part that is problematic, and that it advertises such a boost in performance without telling it is a hardware dependent library. May lead to future cases of such problems.

Labels: Bug, module:model_selection, Needs Reproducible Code
SequentialFeatureSelector should have a feature order attribute

### Describe the workflow you want to enable

I want to be able to run Forward Sequential Feature Selector and know the order which it picked the features.
Right now you can only get the indices or a boolean mask of all the features you've selected. Of course, you can run it once and select 1 feature, then run it again and select 2 features, and continue this to pull out the order that features were selected. However, this would be unnecessarily computationally costly.

### Describe your proposed solution

Add an attribute `feature_order`. I should be able to do `SequentialFeatureSelector.feature_order` and get an array of the indices of the features as they are selected.
I edited the source code (sklearn/feature_selection/_sequential.py) on my laptop with just 3 lines of code and it works.
Under the function `def fit(...)` I added:

> feature_order = []                                      # before the for loop
> feature_order.append(new_feature_idx) # this is at the end of the for loop
> self.feature_order = feature_order           # at the end of the function

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
Thanks for the proposal @samlobe . Looks like this issue is a duplicate of https://github.com/scikit-learn/scikit-learn/issues/19583.

Adding a `feature_order` might seem simple at first glance, but it can actually be ambiguous: see in particular my comment https://github.com/scikit-learn/scikit-learn/issues/19583#issuecomment-788752741

Labels: New Feature, Needs Triage
SequentialFeatureSelector should have a feature order attribute

### Describe the workflow you want to enable

I want to be able to run Forward Sequential Feature Selector and know the order which it picked the features.
Right now you can only get the indices or a boolean mask of all the features you've selected. Of course, you can run it once and select 1 feature, then run it again and select 2 features, and continue this to pull out the order that features were selected. However, this would be unnecessarily computationally costly.

### Describe your proposed solution

Add an attribute `feature_order`. I should be able to do `SequentialFeatureSelector.feature_order` and get an array of the indices of the features as they are selected.
I edited the source code (sklearn/feature_selection/_sequential.py) on my laptop with just 3 lines of code and it works.
Under the function `def fit(...)` I added:

> feature_order = []                                      # before the for loop
> feature_order.append(new_feature_idx) # this is at the end of the for loop
> self.feature_order = feature_order           # at the end of the function

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
I am closing this issue since it is a duplicate of #19583. You are welcome to continue the discussion there.

Labels: New Feature, Needs Triage