Talip7/scikit-learn-issues-multilabel huggingface dataset:
Drop Duplicates from Data Before Fitting Estimator in RFE

### Describe the workflow you want to enable

Currently, RFE makes a slice of the data that only includes specific features. Then, RFE fits the provided estimator to that slice. Making a slice of the data reduces the number of possible variations that can be found in the dataset, potentially leading to many duplicates being found in the dataset. Duplicates become highly prevalent when RFE needs to select a small number of features. Using a dataset with many duplicates to fit an estimator can result in misleading feature importance, bias towards the duplicated data points, and increased training time.

My proposal: Drop duplicates from array slice before fitting the provided estimator in RFE so that the model doesn't overfit and provide misleading metrics.

### Describe your proposed solution

Previous:
```python
estimator.fit(X[:, features], y, **fit_params)
```
Proposed Solution (Using Pandas):
```python
# Convert to Pandas DataFrame and Drop Duplicates
df = pd.concat([pd.DataFrame(X[:, features]), pd.Series(y)], axis = 1)
df = df.drop_duplicates(ignore_index = True) 

# Fit Estimator
estimator.fit(df.iloc[:, :-1], df.iloc[:, -1], **fit_params)
```
Proposed Solution (Using Numpy):
```python
# Create Data and Drop Duplicates
data = np.concatenate((X[:, features], y.reshape(-1, 1)), axis=1)
unique_data, unique_indices = np.unique(data, axis=0, return_index=True)

# Sort the unique data by the original order (Optional)
sorted_indices = np.sort(unique_indices)
unique_data_sorted = data[sorted_indices]

# Fit Estimator
estimator.fit(unique_data_sorted[:, :-1], unique_data_sorted[:, -1], **fit_params)
```
### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
Hi @JayTicku, this is not the kind of preprocessing is usually done automatically in scikit-learn. I don't think it should always be done automatically. Removing duplicates changes the distribution of the data which may not give the expected results.

Instead, I think you should be able to achieve it with the current version by implementing a custom transformer that removes the duplicates and pass a pipeline with this transformer followed by your final estimator to RFE.

What do you think @glemaitre ?

Labels: New Feature, Needs Decision
Drop Duplicates from Data Before Fitting Estimator in RFE

### Describe the workflow you want to enable

Currently, RFE makes a slice of the data that only includes specific features. Then, RFE fits the provided estimator to that slice. Making a slice of the data reduces the number of possible variations that can be found in the dataset, potentially leading to many duplicates being found in the dataset. Duplicates become highly prevalent when RFE needs to select a small number of features. Using a dataset with many duplicates to fit an estimator can result in misleading feature importance, bias towards the duplicated data points, and increased training time.

My proposal: Drop duplicates from array slice before fitting the provided estimator in RFE so that the model doesn't overfit and provide misleading metrics.

### Describe your proposed solution

Previous:
```python
estimator.fit(X[:, features], y, **fit_params)
```
Proposed Solution (Using Pandas):
```python
# Convert to Pandas DataFrame and Drop Duplicates
df = pd.concat([pd.DataFrame(X[:, features]), pd.Series(y)], axis = 1)
df = df.drop_duplicates(ignore_index = True) 

# Fit Estimator
estimator.fit(df.iloc[:, :-1], df.iloc[:, -1], **fit_params)
```
Proposed Solution (Using Numpy):
```python
# Create Data and Drop Duplicates
data = np.concatenate((X[:, features], y.reshape(-1, 1)), axis=1)
unique_data, unique_indices = np.unique(data, axis=0, return_index=True)

# Sort the unique data by the original order (Optional)
sorted_indices = np.sort(unique_indices)
unique_data_sorted = data[sorted_indices]

# Fit Estimator
estimator.fit(unique_data_sorted[:, :-1], unique_data_sorted[:, -1], **fit_params)
```
### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
Hi @jeremiedbb,

As RFE selects smaller sets of features, duplicates can become more prevalent in the dataset. In datasets with mainly numerical columns, duplicates may be less common due to the large range of values. However, in datasets with many categorical columns, duplicates can become widespread due to fewer unique values per feature, potentially affecting feature importance. Although removing duplicates alters the distribution of the data, leaving them in might be more detrimental. Instead of automatically dropping duplicates after selecting a smaller set of features, RFE could drop duplicates depending on their prevalence and the type of data.

Labels: New Feature, Needs Decision
Drop Duplicates from Data Before Fitting Estimator in RFE

### Describe the workflow you want to enable

Currently, RFE makes a slice of the data that only includes specific features. Then, RFE fits the provided estimator to that slice. Making a slice of the data reduces the number of possible variations that can be found in the dataset, potentially leading to many duplicates being found in the dataset. Duplicates become highly prevalent when RFE needs to select a small number of features. Using a dataset with many duplicates to fit an estimator can result in misleading feature importance, bias towards the duplicated data points, and increased training time.

My proposal: Drop duplicates from array slice before fitting the provided estimator in RFE so that the model doesn't overfit and provide misleading metrics.

### Describe your proposed solution

Previous:
```python
estimator.fit(X[:, features], y, **fit_params)
```
Proposed Solution (Using Pandas):
```python
# Convert to Pandas DataFrame and Drop Duplicates
df = pd.concat([pd.DataFrame(X[:, features]), pd.Series(y)], axis = 1)
df = df.drop_duplicates(ignore_index = True) 

# Fit Estimator
estimator.fit(df.iloc[:, :-1], df.iloc[:, -1], **fit_params)
```
Proposed Solution (Using Numpy):
```python
# Create Data and Drop Duplicates
data = np.concatenate((X[:, features], y.reshape(-1, 1)), axis=1)
unique_data, unique_indices = np.unique(data, axis=0, return_index=True)

# Sort the unique data by the original order (Optional)
sorted_indices = np.sort(unique_indices)
unique_data_sorted = data[sorted_indices]

# Fit Estimator
estimator.fit(unique_data_sorted[:, :-1], unique_data_sorted[:, -1], **fit_params)
```
### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
>  Removing duplicates changes the distribution of the data which may not give the expected results.

I also back-up this claim.

> However, in datasets with many categorical columns, duplicates can become widespread due to fewer unique values per feature, potentially affecting feature importance.

It just mean that this feature alone will not be predictive enough. For me, we should not alter the distribution of the dataset on which you learn.

In terms of API, we don't allow to change the number of rows in a dataset and we did not implement any sampler. So I don't think that we can go forward with this feature.

Labels: New Feature, Needs Decision
`linear_sum_assignment` no longer uses Hungarian method

### Describe the issue linked to the documentation

`consensus_score` on the docs is describing that the best match is found using the Hungarian method, but from what I can read, the SciPy implemantation is using the Jonker-Volgenant algorithm since v1.9 https://github.com/scipy/scipy/pull/15464 .

### Suggest a potential alternative/fix

I think the docs should reflect that, but I'm not sure about neither algorithms nor if I'm reading everything right, so I'd like some verification first before opening a pull request.

COMMENT:
Thanks for the report @notPlancha. It even looks like it's using the Jonker-Volgenant algorithm since 1.4 https://github.com/scipy/scipy/pull/10296. I agree that we need to update the doc. Would you like to make a PR ?

Labels: Documentation
COLAB fetch_20newsgroups gets 403 from https://ndownloader.figshare.com/files/5975967 but OK with browser

### Describe the bug

COLAB 

```
fetch_20newsgroups (20news-bydate.tar.gz) gets 403 from https://ndownloader.figshare.com/files/5975967 [which is really https://s3-eu-west-1.amazonaws.com/pfigshare-u-files/5975967/20newsbydate.tar.gz].
```

It might be AWS are blocking Google Colab addresses.

```python
ARCHIVE = RemoteFileMetadata(
    filename="20news-bydate.tar.gz",
    url="https://ndownloader.figshare.com/files/5975967",
    checksum="8f1b2514ca22a5ade8fbb9cfa5727df95fa587f4c87b786e15c759fa66d95610",
)
XX = _fetch_remote(ARCHIVE)

import urllib
import requests
response = requests.get("https://ndownloader.figshare.com/files/5975967",headers=headers)
print (response.text)
```

### Steps/Code to Reproduce

```python
fetch_20newsgroups gets 403 from https://ndownloader.figshare.com/files/5975967

ARCHIVE = RemoteFileMetadata(
    filename="20news-bydate.tar.gz",
    url="https://ndownloader.figshare.com/files/5975967",
    checksum="8f1b2514ca22a5ade8fbb9cfa5727df95fa587f4c87b786e15c759fa66d95610",
)
XX = _fetch_remote(ARCHIVE)

import urllib
import requests
response = requests.get("https://ndownloader.figshare.com/files/5975967",headers=headers)
print (response.text)
```

### Expected Results

403 should not be presented

### Actual Results

<html>
<head><title>403 Forbidden</title></head>
<body>
<center><h1>403 Forbidden</h1></center>
</body>
</html>

### Versions

```shell
System:
    python: 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0]
executable: /usr/bin/python3
   machine: Linux-6.1.85+-x86_64-with-glibc2.35

Python dependencies:
      sklearn: 1.2.2
          pip: 23.1.2
   setuptools: 67.7.2
        numpy: 1.25.2
        scipy: 1.11.4
       Cython: 3.0.10
       pandas: 2.0.3
   matplotlib: 3.7.1
       joblib: 1.4.2
threadpoolctl: 3.5.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: openblas
    num_threads: 2
         prefix: libopenblas
       filepath: /usr/local/lib/python3.10/dist-packages/numpy.libs/libopenblas64_p-r0-5007b62f.3.23.dev.so
        version: 0.3.23.dev
threading_layer: pthreads
   architecture: Haswell

       user_api: openmp
   internal_api: openmp
    num_threads: 2
         prefix: libgomp
       filepath: /usr/local/lib/python3.10/dist-packages/scikit_learn.libs/libgomp-a34b3233.so.1.0.0
        version: None

       user_api: blas
   internal_api: openblas
    num_threads: 2
         prefix: libopenblas
       filepath: /usr/local/lib/python3.10/dist-packages/scipy.libs/libopenblasp-r0-23e5df77.3.21.dev.so
        version: 0.3.21.dev
threading_layer: pthreads
   architecture: Haswell
```

COMMENT:
Duplicate of https://github.com/scikit-learn/scikit-learn/issues/28297

I'll close to have a single issue to track the problem

Labels: Bug, Needs Triage
COLAB fetch_20newsgroups gets 403 from https://ndownloader.figshare.com/files/5975967 but OK with browser

### Describe the bug

COLAB 

```
fetch_20newsgroups (20news-bydate.tar.gz) gets 403 from https://ndownloader.figshare.com/files/5975967 [which is really https://s3-eu-west-1.amazonaws.com/pfigshare-u-files/5975967/20newsbydate.tar.gz].
```

It might be AWS are blocking Google Colab addresses.

```python
ARCHIVE = RemoteFileMetadata(
    filename="20news-bydate.tar.gz",
    url="https://ndownloader.figshare.com/files/5975967",
    checksum="8f1b2514ca22a5ade8fbb9cfa5727df95fa587f4c87b786e15c759fa66d95610",
)
XX = _fetch_remote(ARCHIVE)

import urllib
import requests
response = requests.get("https://ndownloader.figshare.com/files/5975967",headers=headers)
print (response.text)
```

### Steps/Code to Reproduce

```python
fetch_20newsgroups gets 403 from https://ndownloader.figshare.com/files/5975967

ARCHIVE = RemoteFileMetadata(
    filename="20news-bydate.tar.gz",
    url="https://ndownloader.figshare.com/files/5975967",
    checksum="8f1b2514ca22a5ade8fbb9cfa5727df95fa587f4c87b786e15c759fa66d95610",
)
XX = _fetch_remote(ARCHIVE)

import urllib
import requests
response = requests.get("https://ndownloader.figshare.com/files/5975967",headers=headers)
print (response.text)
```

### Expected Results

403 should not be presented

### Actual Results

<html>
<head><title>403 Forbidden</title></head>
<body>
<center><h1>403 Forbidden</h1></center>
</body>
</html>

### Versions

```shell
System:
    python: 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0]
executable: /usr/bin/python3
   machine: Linux-6.1.85+-x86_64-with-glibc2.35

Python dependencies:
      sklearn: 1.2.2
          pip: 23.1.2
   setuptools: 67.7.2
        numpy: 1.25.2
        scipy: 1.11.4
       Cython: 3.0.10
       pandas: 2.0.3
   matplotlib: 3.7.1
       joblib: 1.4.2
threadpoolctl: 3.5.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: openblas
    num_threads: 2
         prefix: libopenblas
       filepath: /usr/local/lib/python3.10/dist-packages/numpy.libs/libopenblas64_p-r0-5007b62f.3.23.dev.so
        version: 0.3.23.dev
threading_layer: pthreads
   architecture: Haswell

       user_api: openmp
   internal_api: openmp
    num_threads: 2
         prefix: libgomp
       filepath: /usr/local/lib/python3.10/dist-packages/scikit_learn.libs/libgomp-a34b3233.so.1.0.0
        version: None

       user_api: blas
   internal_api: openblas
    num_threads: 2
         prefix: libopenblas
       filepath: /usr/local/lib/python3.10/dist-packages/scipy.libs/libopenblasp-r0-23e5df77.3.21.dev.so
        version: 0.3.21.dev
threading_layer: pthreads
   architecture: Haswell
```

COMMENT:
You can look at the issue that I linked at you will see that the investigations shows that this is not reliable. We are sure if this is an issue of the server requesting or receiving the request but this is not working as expected.

Labels: Bug, Needs Triage
⚠️ CI failed on Linux_free_threaded.pylatest_pip_free_threaded (last failure: Jun 14, 2024) ⚠️

**CI failed on [Linux_free_threaded.pylatest_pip_free_threaded](https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=67539&view=logs&j=8bc43b48-889f-54b9-cd8b-781ee8447bf2)** (Jun 14, 2024)
- test_minibatch_sensible_reassign[34]

COMMENT:
This might be a seed sensitivity problem of this test. We need to see if we can reproduce with other Python versions / platforms.

The error is:

```python
    def test_minibatch_sensible_reassign(global_random_seed):
        # check that identical initial clusters are reassigned
        # also a regression test for when there are more desired reassignments than
        # samples.
        zeroed_X, true_labels = make_blobs(
            n_samples=100, centers=5, random_state=global_random_seed
        )
        zeroed_X[::2, :] = 0
    
        km = MiniBatchKMeans(
            n_clusters=20, batch_size=10, random_state=global_random_seed, init="random"
        ).fit(zeroed_X)
        # there should not be too many exact zero cluster centers
>       assert km.cluster_centers_.any(axis=1).sum() > 10
E       AssertionError
```

any idea why pytest's `AssertionError` no longer display the values of the two expressions on each side of an binary operator?

Labels: Build / CI
⚠️ CI failed on Linux_free_threaded.pylatest_pip_free_threaded (last failure: Jun 14, 2024) ⚠️

**CI failed on [Linux_free_threaded.pylatest_pip_free_threaded](https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=67539&view=logs&j=8bc43b48-889f-54b9-cd8b-781ee8447bf2)** (Jun 14, 2024)
- test_minibatch_sensible_reassign[34]

COMMENT:
The `test_minibatch_sensible_reassign` failure has been seen in other CI builds https://github.com/scikit-learn/scikit-learn/issues/27967#issuecomment-1863791083 or https://github.com/scikit-learn/scikit-learn/issues/26802#issuecomment-2126728286

Not sure about why pytest does not manage to give more information about the `AssertionError` ... same thing happens on vanilla Python in our CI e.g. this [build log](https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=66816&view=logs&j=dde5042c-7464-5d47-9507-31bdd2ee0a3a&t=4bd2dad8-62b3-5bf9-08a5-a9880c530c94)

Labels: Build / CI
⚠️ CI failed on Linux_free_threaded.pylatest_pip_free_threaded (last failure: Jun 14, 2024) ⚠️

**CI failed on [Linux_free_threaded.pylatest_pip_free_threaded](https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=67539&view=logs&j=8bc43b48-889f-54b9-cd8b-781ee8447bf2)** (Jun 14, 2024)
- test_minibatch_sensible_reassign[34]

COMMENT:
I tried to reproduce locally with either openblas or accelerate:

```
SKLEARN_TESTS_GLOBAL_RANDOM_SEED="all"  pytest -k test_minibatch_sensible_reassign sklearn/cluster/tests/test_k_means.py -v
```

but I cannot reproduce the assertion failure after several runs including when using `pytest-xdist` for different concurrency-induced non-deterministic execution patterns.

Labels: Build / CI
⚠️ CI failed on Linux_free_threaded.pylatest_pip_free_threaded (last failure: Jun 14, 2024) ⚠️

**CI failed on [Linux_free_threaded.pylatest_pip_free_threaded](https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=67539&view=logs&j=8bc43b48-889f-54b9-cd8b-781ee8447bf2)** (Jun 14, 2024)
- test_minibatch_sensible_reassign[34]

COMMENT:
## CI is no longer failing! ✅

[Successful run](https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=67758&view=logs&j=8bc43b48-889f-54b9-cd8b-781ee8447bf2) on Jun 20, 2024

Labels: Build / CI
⚠️ CI failed on Linux_free_threaded.pylatest_pip_free_threaded (last failure: Jun 14, 2024) ⚠️

**CI failed on [Linux_free_threaded.pylatest_pip_free_threaded](https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=67539&view=logs&j=8bc43b48-889f-54b9-cd8b-781ee8447bf2)** (Jun 14, 2024)
- test_minibatch_sensible_reassign[34]

COMMENT:
I had a look at the test and the threshold of 10 non-reassignments (over 20 clusters) does not seem particularly well justified.

However, by adding print statements on my local machines I consistently see values in the 14-19 range, so there might still be a platform-specific discrepancy that could explain reaching 10.

Still we might want to move the threshold to 8 instead of 10 to silence this failure (potentially at the cost of making this test a bit too weak).

Labels: Build / CI
⚠️ CI failed on Linux_free_threaded.pylatest_pip_free_threaded (last failure: Jun 14, 2024) ⚠️

**CI failed on [Linux_free_threaded.pylatest_pip_free_threaded](https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=67539&view=logs&j=8bc43b48-889f-54b9-cd8b-781ee8447bf2)** (Jun 14, 2024)
- test_minibatch_sensible_reassign[34]

COMMENT:
I can not reproduce either locally even using a lock-file to try to be as close as possible to the CI.

I do manage to reproduce in the CI, see https://github.com/scikit-learn/scikit-learn/pull/29278. ~I really don't get it, I changed the code to have more information but the error message with the assertion is not reflecting this :scream:.~ I got it it, rookie mistake on my part :sweat_smile:.

Labels: Build / CI
⚠️ CI failed on Linux_free_threaded.pylatest_pip_free_threaded (last failure: Jun 14, 2024) ⚠️

**CI failed on [Linux_free_threaded.pylatest_pip_free_threaded](https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=67539&view=logs&j=8bc43b48-889f-54b9-cd8b-781ee8447bf2)** (Jun 14, 2024)
- test_minibatch_sensible_reassign[34]

COMMENT:
And I think that the reason why there is not more info in the assertion message is because of an issue with Pytest assertion rewriting with editable install :face_with_head_bandage:, maybe  https://github.com/pytest-dev/pytest/issues/11783 but I don't think so ...

This may also be a quirk of meson editable installs, not sure ... 

What I have checked is that a Meson non-editable install works with Pytest assertion rewriting, a meson editable install does not.

Labels: Build / CI
⚠️ CI failed on Linux_free_threaded.pylatest_pip_free_threaded (last failure: Jun 14, 2024) ⚠️

**CI failed on [Linux_free_threaded.pylatest_pip_free_threaded](https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=67539&view=logs&j=8bc43b48-889f-54b9-cd8b-781ee8447bf2)** (Jun 14, 2024)
- test_minibatch_sensible_reassign[34]

COMMENT:
FYI I ended up opening an issue about Pytest assertion rewriting and meson ediable install in https://github.com/mesonbuild/meson-python/issues/646

Labels: Build / CI
[BUG] `roc_auc_score` is wrong (edge case)

### Describe the bug

Hi,

It seems that on the edge case when there are equal scores, [`roc_auc_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_auc_score.html#roc-auc-score) actually makes a wrong comuptation:
```pycon
>>> roc_auc_score([1, 0, 0, 1], [2, 5, 10, 10])
0.375
>>> # Expected 0.25 or 0.5, see below.
```
On would expect either `0.5` or `0.25` depending on whether or not the area is computed by interpolating to the convex hull (which makes sense with probabilistic mixtures for the ROC curve) or not.

I tried illustrating this example below (keep in mind that this behaviour is *speculated* by myself, I didn't go through any of `sklearn` code).

![roc_auc_bug](https://github.com/scikit-learn/scikit-learn/assets/114467748/4a902ca2-fc23-4341-b8b9-6234a9f40fd8)

Any feedback is welcome.
Cheers!

### Steps/Code to Reproduce

```pycon
from sklearn.metrics import roc_auc_score
roc_auc_score([1, 0, 0, 1], [2, 5, 10, 10])
```

### Expected Results

`0.25` or `0.5`

### Actual Results

`0.375`

### Versions

```shell
1.5.0
```

COMMENT:
Thanks for the report. Could you expand your report with the ROC curve plots using sklearn's `roc_curve` or `RocCurveDisplay` to better understand where the discrepancy comes from?

Labels: Bug