Talip7/scikit-learn-issues-multilabel huggingface dataset:
Add option to RocCurveDisplay to display the average of different length ROC curves

### Describe the workflow you want to enable

When using k-fold cross-validation the resulting ROC curves can vary in length if there are a different number of positive and/or negative samples in each fold. I would like to add an option to sklearn.metrics.RocCurveDisplay to display the average of different length ROC curves.

Section 8 of [An introduction to ROC analysis](https://ccrma.stanford.edu/workshops/mir2009/references/ROCintro.pdf) by Fawcett describes the method.

### Describe your proposed solution

```python
def average_roc_curves(roc_curves_list,):
    """
    This function takes the average of different length ROC curves returned from 
    sklearn.metrics.roc_curve. The function expects a list of fpr, tpr, and 
    thresholds. The function subsamples from these lists and returns an 
    average_fpr, average_tpr, and average_thresholds.

    Parameters
    ----------
    roc_curves_list : list
        A list of fpr, tpr, and thresholds
    Returns
    -------
    average_fpr : list
        A list of averaged false positive rates
    average_tpr : list
        A list of averaged true positive rates
    average_thresholds : list
        A list of averaged thresholds
    """
    kfolds = len(roc_curves_list)
    min_length = np.min(list(len(roc_curves_list[k][2]) for k in range(kfolds)))
    shortened_roc_curves_list = list()
    for k in range(kfolds):
        fpr, tpr, thresholds = roc_curves_list[k]
        indicies = list(i for i in range(len(thresholds)))
        selected_indicies = np.sort(np.random.choice(indicies, min_length, replace=False))
        shortened_roc_curves_list.append([fpr[selected_indicies], tpr[selected_indicies], thresholds[selected_indicies]])
    average_fpr, average_tpr, average_thresholds = np.mean(shortened_roc_curves_list, axis=0)
    return average_fpr, average_tpr, average_thresholds
```

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
[This paper](https://www.sciencedirect.com/science/article/abs/pii/S016786550500303X), published in 2006 in the Pattern Recognition Letters (with over 20,000 citations), discusses the method implemented in the code above. I would love to contribute by writing this feature. Please let me know if rewriting the function in vanilla python would be helpful.

Labels: New Feature, module:metrics, Needs Decision - Include Feature
Add option to RocCurveDisplay to display the average of different length ROC curves

### Describe the workflow you want to enable

When using k-fold cross-validation the resulting ROC curves can vary in length if there are a different number of positive and/or negative samples in each fold. I would like to add an option to sklearn.metrics.RocCurveDisplay to display the average of different length ROC curves.

Section 8 of [An introduction to ROC analysis](https://ccrma.stanford.edu/workshops/mir2009/references/ROCintro.pdf) by Fawcett describes the method.

### Describe your proposed solution

```python
def average_roc_curves(roc_curves_list,):
    """
    This function takes the average of different length ROC curves returned from 
    sklearn.metrics.roc_curve. The function expects a list of fpr, tpr, and 
    thresholds. The function subsamples from these lists and returns an 
    average_fpr, average_tpr, and average_thresholds.

    Parameters
    ----------
    roc_curves_list : list
        A list of fpr, tpr, and thresholds
    Returns
    -------
    average_fpr : list
        A list of averaged false positive rates
    average_tpr : list
        A list of averaged true positive rates
    average_thresholds : list
        A list of averaged thresholds
    """
    kfolds = len(roc_curves_list)
    min_length = np.min(list(len(roc_curves_list[k][2]) for k in range(kfolds)))
    shortened_roc_curves_list = list()
    for k in range(kfolds):
        fpr, tpr, thresholds = roc_curves_list[k]
        indicies = list(i for i in range(len(thresholds)))
        selected_indicies = np.sort(np.random.choice(indicies, min_length, replace=False))
        shortened_roc_curves_list.append([fpr[selected_indicies], tpr[selected_indicies], thresholds[selected_indicies]])
    average_fpr, average_tpr, average_thresholds = np.mean(shortened_roc_curves_list, axis=0)
    return average_fpr, average_tpr, average_thresholds
```

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
Or could `roc_curve` be designed to accept a list of `y_trues` and `y_scores` instead of a `cv` splitter (if an `average` parameter is passed)? This would reduce the amount inflation while still providing the same function.

Labels: New Feature, module:metrics, Needs Decision - Include Feature
Add option to RocCurveDisplay to display the average of different length ROC curves

### Describe the workflow you want to enable

When using k-fold cross-validation the resulting ROC curves can vary in length if there are a different number of positive and/or negative samples in each fold. I would like to add an option to sklearn.metrics.RocCurveDisplay to display the average of different length ROC curves.

Section 8 of [An introduction to ROC analysis](https://ccrma.stanford.edu/workshops/mir2009/references/ROCintro.pdf) by Fawcett describes the method.

### Describe your proposed solution

```python
def average_roc_curves(roc_curves_list,):
    """
    This function takes the average of different length ROC curves returned from 
    sklearn.metrics.roc_curve. The function expects a list of fpr, tpr, and 
    thresholds. The function subsamples from these lists and returns an 
    average_fpr, average_tpr, and average_thresholds.

    Parameters
    ----------
    roc_curves_list : list
        A list of fpr, tpr, and thresholds
    Returns
    -------
    average_fpr : list
        A list of averaged false positive rates
    average_tpr : list
        A list of averaged true positive rates
    average_thresholds : list
        A list of averaged thresholds
    """
    kfolds = len(roc_curves_list)
    min_length = np.min(list(len(roc_curves_list[k][2]) for k in range(kfolds)))
    shortened_roc_curves_list = list()
    for k in range(kfolds):
        fpr, tpr, thresholds = roc_curves_list[k]
        indicies = list(i for i in range(len(thresholds)))
        selected_indicies = np.sort(np.random.choice(indicies, min_length, replace=False))
        shortened_roc_curves_list.append([fpr[selected_indicies], tpr[selected_indicies], thresholds[selected_indicies]])
    average_fpr, average_tpr, average_thresholds = np.mean(shortened_roc_curves_list, axis=0)
    return average_fpr, average_tpr, average_thresholds
```

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
That sounds good. I created a branch on my GitHub for this feature ([see branch](https://github.com/ethanharvey98/scikit-learn/tree/roc-curve-cv)). Would that be the best way to move forward?

Labels: New Feature, module:metrics, Needs Decision - Include Feature
Add option to RocCurveDisplay to display the average of different length ROC curves

### Describe the workflow you want to enable

When using k-fold cross-validation the resulting ROC curves can vary in length if there are a different number of positive and/or negative samples in each fold. I would like to add an option to sklearn.metrics.RocCurveDisplay to display the average of different length ROC curves.

Section 8 of [An introduction to ROC analysis](https://ccrma.stanford.edu/workshops/mir2009/references/ROCintro.pdf) by Fawcett describes the method.

### Describe your proposed solution

```python
def average_roc_curves(roc_curves_list,):
    """
    This function takes the average of different length ROC curves returned from 
    sklearn.metrics.roc_curve. The function expects a list of fpr, tpr, and 
    thresholds. The function subsamples from these lists and returns an 
    average_fpr, average_tpr, and average_thresholds.

    Parameters
    ----------
    roc_curves_list : list
        A list of fpr, tpr, and thresholds
    Returns
    -------
    average_fpr : list
        A list of averaged false positive rates
    average_tpr : list
        A list of averaged true positive rates
    average_thresholds : list
        A list of averaged thresholds
    """
    kfolds = len(roc_curves_list)
    min_length = np.min(list(len(roc_curves_list[k][2]) for k in range(kfolds)))
    shortened_roc_curves_list = list()
    for k in range(kfolds):
        fpr, tpr, thresholds = roc_curves_list[k]
        indicies = list(i for i in range(len(thresholds)))
        selected_indicies = np.sort(np.random.choice(indicies, min_length, replace=False))
        shortened_roc_curves_list.append([fpr[selected_indicies], tpr[selected_indicies], thresholds[selected_indicies]])
    average_fpr, average_tpr, average_thresholds = np.mean(shortened_roc_curves_list, axis=0)
    return average_fpr, average_tpr, average_thresholds
```

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
Having an average ROC curve computed using cross validation would be fairly new API-wise to scikit-learn. To move forward we likely need:

1. Another maintainer that agrees this is a good way forward.
2. Pull request that includes updates to the user guide, the implementation, tests, and [an  example](https://scikit-learn.org/stable/auto_examples/index.html) on how to use the new feature. You can look at the [contributing guide](https://scikit-learn.org/dev/developers/contributing.html) for information on how to contribute.

@glemaitre @ogrisel Would you be interested in having this feature in scikit-learn?

Labels: New Feature, module:metrics, Needs Decision - Include Feature
Add option to RocCurveDisplay to display the average of different length ROC curves

### Describe the workflow you want to enable

When using k-fold cross-validation the resulting ROC curves can vary in length if there are a different number of positive and/or negative samples in each fold. I would like to add an option to sklearn.metrics.RocCurveDisplay to display the average of different length ROC curves.

Section 8 of [An introduction to ROC analysis](https://ccrma.stanford.edu/workshops/mir2009/references/ROCintro.pdf) by Fawcett describes the method.

### Describe your proposed solution

```python
def average_roc_curves(roc_curves_list,):
    """
    This function takes the average of different length ROC curves returned from 
    sklearn.metrics.roc_curve. The function expects a list of fpr, tpr, and 
    thresholds. The function subsamples from these lists and returns an 
    average_fpr, average_tpr, and average_thresholds.

    Parameters
    ----------
    roc_curves_list : list
        A list of fpr, tpr, and thresholds
    Returns
    -------
    average_fpr : list
        A list of averaged false positive rates
    average_tpr : list
        A list of averaged true positive rates
    average_thresholds : list
        A list of averaged thresholds
    """
    kfolds = len(roc_curves_list)
    min_length = np.min(list(len(roc_curves_list[k][2]) for k in range(kfolds)))
    shortened_roc_curves_list = list()
    for k in range(kfolds):
        fpr, tpr, thresholds = roc_curves_list[k]
        indicies = list(i for i in range(len(thresholds)))
        selected_indicies = np.sort(np.random.choice(indicies, min_length, replace=False))
        shortened_roc_curves_list.append([fpr[selected_indicies], tpr[selected_indicies], thresholds[selected_indicies]])
    average_fpr, average_tpr, average_thresholds = np.mean(shortened_roc_curves_list, axis=0)
    return average_fpr, average_tpr, average_thresholds
```

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
Plotting uncertainty or confidence intervals is not something straightforward and I think that there is a benefit to doing so for our users. 

I think that our displays should be able to be fed with the results of `cross_validate`. I recall experimenting with the following: https://github.com/scikit-learn/scikit-learn/pull/21211. I would think of something similar for the ROC display.

Labels: New Feature, module:metrics, Needs Decision - Include Feature
Add option to RocCurveDisplay to display the average of different length ROC curves

### Describe the workflow you want to enable

When using k-fold cross-validation the resulting ROC curves can vary in length if there are a different number of positive and/or negative samples in each fold. I would like to add an option to sklearn.metrics.RocCurveDisplay to display the average of different length ROC curves.

Section 8 of [An introduction to ROC analysis](https://ccrma.stanford.edu/workshops/mir2009/references/ROCintro.pdf) by Fawcett describes the method.

### Describe your proposed solution

```python
def average_roc_curves(roc_curves_list,):
    """
    This function takes the average of different length ROC curves returned from 
    sklearn.metrics.roc_curve. The function expects a list of fpr, tpr, and 
    thresholds. The function subsamples from these lists and returns an 
    average_fpr, average_tpr, and average_thresholds.

    Parameters
    ----------
    roc_curves_list : list
        A list of fpr, tpr, and thresholds
    Returns
    -------
    average_fpr : list
        A list of averaged false positive rates
    average_tpr : list
        A list of averaged true positive rates
    average_thresholds : list
        A list of averaged thresholds
    """
    kfolds = len(roc_curves_list)
    min_length = np.min(list(len(roc_curves_list[k][2]) for k in range(kfolds)))
    shortened_roc_curves_list = list()
    for k in range(kfolds):
        fpr, tpr, thresholds = roc_curves_list[k]
        indicies = list(i for i in range(len(thresholds)))
        selected_indicies = np.sort(np.random.choice(indicies, min_length, replace=False))
        shortened_roc_curves_list.append([fpr[selected_indicies], tpr[selected_indicies], thresholds[selected_indicies]])
    average_fpr, average_tpr, average_thresholds = np.mean(shortened_roc_curves_list, axis=0)
    return average_fpr, average_tpr, average_thresholds
```

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
That sounds great. Would the function take results from `cross_validate` instead of `y_true` and `y_score` (like in [#21211](https://github.com/scikit-learn/scikit-learn/pull/21211))?

```python
def roc_curve_cv(
    cv_results, X, y, *, average='vertical', pos_label=None, sample_weight=None, drop_intermediate=True
):
```

Should the function have a sample rate? In the paper (cited in [#23983 (comment)](https://github.com/scikit-learn/scikit-learn/issues/23983#issuecomment-1193040203)), the curves were sampled at FP rates from 0 through 1 by 0.1. Should the function use a similar sample rate, or should a sample rate only be introduced when the average ROC curve is graphed?

Labels: New Feature, module:metrics, Needs Decision - Include Feature
⚠️ CI failed on Linux_Nightly_PyPy.pypy3 ⚠️

**CI is still failing on [Linux_Nightly_PyPy.pypy3](https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=44954&view=logs&j=0b16f832-29d6-5b92-1c23-eb006f606a66)** (Jul 24, 2022)
Unable to find junit file. Please see link for details.

COMMENT:
## CI is no longer failing! ✅

[Successful run](https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=45076&view=logs&j=0b16f832-29d6-5b92-1c23-eb006f606a66) on Jul 27, 2022

Labels: Needs Triage
⚠️ CI failed on Linux_Nightly_PyPy.pypy3 ⚠️

**CI is still failing on [Linux_Nightly_PyPy.pypy3](https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=44954&view=logs&j=0b16f832-29d6-5b92-1c23-eb006f606a66)** (Jul 24, 2022)
Unable to find junit file. Please see link for details.

COMMENT:
Closing since the original failure seems random and the job has been passing for the last three days.

Labels: Needs Triage
Test Failed on CICD Pipeline with varied LinearRegression predictions

- Ref: https://stackoverflow.com/questions/73067554/test-failed-on-cicd-pipeline-with-varied-linearregression-predictions

We have a very simple use case, when we take some data `X`, `y` and fir it to `LinearRegression()` like below. and try to predict some more value.

We have set the seed to some non-zero number again mentioned below.

```py
  np.random.seed(42)
  regr = LinearRegression()
  regr.fit(x, y)
  
  pred = regr.predict(pred_set)[0]
```

The above prediction where as always same in one machine, changes when ran from different machine, specifically when we ran it in some CICD test pipeline on cloud environment.

As per my understanding, setting seed to a specific number should confirm the reproducibility, but is there anything I am missing here. Or If I need to set seed in some other way.

Kindly, let me know.

COMMENT:
There are many factors that can influence how an estimator behaves on different machines. For `LinearRegression`, can you provide more details such as:

1. Data you trained on
2. What the predictions were
3. How different were the predictions
4. Details on the machines you ran your script on.
5. What is the scikit-learn configuration: `python -c 'import sklearn; sklearn.show_versions()'`
6. Since `LinearRegression` uses `lstsq`, it could be different LAPACKs. To find out run: `python -c 'import scipy; scipy.__config__.show()'`

Labels: module:linear_model, Needs Investigation
Test Failed on CICD Pipeline with varied LinearRegression predictions

- Ref: https://stackoverflow.com/questions/73067554/test-failed-on-cicd-pipeline-with-varied-linearregression-predictions

We have a very simple use case, when we take some data `X`, `y` and fir it to `LinearRegression()` like below. and try to predict some more value.

We have set the seed to some non-zero number again mentioned below.

```py
  np.random.seed(42)
  regr = LinearRegression()
  regr.fit(x, y)
  
  pred = regr.predict(pred_set)[0]
```

The above prediction where as always same in one machine, changes when ran from different machine, specifically when we ran it in some CICD test pipeline on cloud environment.

As per my understanding, setting seed to a specific number should confirm the reproducibility, but is there anything I am missing here. Or If I need to set seed in some other way.

Kindly, let me know.

COMMENT:
1. Can you show the predictions locally and the docker machines as a NumPy array? Specifically, something like this:

```
# local machine predictions
[959.03, 842.213, 3213, ...]

# docker predictions
[949.03, 843.213, 32132, ...]
```

I want to see how different the predictions are. If there is only a slightly difference, than it is okay. It could be that different lapacks give results. (The SciPy config for `lapack_opt_info` is different for you locally vs on docker.)

2. What are the model's coefs on different machines? 

```python
regr = LinearRegression()
regr.fit(x, y)
print(regr.coef_)
```

Labels: module:linear_model, Needs Investigation
Test Failed on CICD Pipeline with varied LinearRegression predictions

- Ref: https://stackoverflow.com/questions/73067554/test-failed-on-cicd-pipeline-with-varied-linearregression-predictions

We have a very simple use case, when we take some data `X`, `y` and fir it to `LinearRegression()` like below. and try to predict some more value.

We have set the seed to some non-zero number again mentioned below.

```py
  np.random.seed(42)
  regr = LinearRegression()
  regr.fit(x, y)
  
  pred = regr.predict(pred_set)[0]
```

The above prediction where as always same in one machine, changes when ran from different machine, specifically when we ran it in some CICD test pipeline on cloud environment.

As per my understanding, setting seed to a specific number should confirm the reproducibility, but is there anything I am missing here. Or If I need to set seed in some other way.

Kindly, let me know.

COMMENT:
When I ask for the training data, I was referring to the NumPy array passed in. scikit-learn's repo has many issues to look through and anything you can do to make it easier for maintainers to understand your issue would be helpful. 

Specifically, I am trying to see if this a bug in scikit-learn, another library, user error, or an expected behavior.

Labels: module:linear_model, Needs Investigation
Unsupervised K-Means Clustering Algorithm

### Describe the workflow you want to enable

"the k-means algorithm and its extensions are always influenced by initializations with a necessary number of clusters a priori. That is, the k-means algorithm is not exactly an unsupervised clustering method. In this paper, we construct an unsupervised learning schema for the k-means algorithm so that it is free of initializations without parameter selection and can also simultaneously find an optimal number of clusters." from the abstract. 

### Describe your proposed solution

"To construct the k-means clustering algorithm with free of initializations and automatically find the number of clusters, we use the entropy concept. We borrow the idea from the EM algorithm by Yang et al." from section 3 (THE UNSUPERVISED K-MEANS CLUSTERING ALGORITHM).

### Describe alternatives you've considered, if relevant

R-EM, C-FS, X-Means, and RL-FCM

### Additional context

https://ieeexplore.ieee.org/document/9072123

309 citations (according to Google Scholar, not sure why IEEE shows a different number), released in 20 April 2020. I think there are up to 5 parameters "γ, β, c, a K, a K,"  might be wrong.

The total computational complexity for the U-k-means algorithm is O(ncd), where n is the number of data points, c is the number of clusters, and d is the dimension of data points.

Name of the authors: Kristina P. Sinaga, Miin-Shen Yang

COMMENT:
Would love to pick this up if no one is working on this.

Labels: New Feature, module:cluster, Needs Decision - Include Feature
Unsupervised K-Means Clustering Algorithm

### Describe the workflow you want to enable

"the k-means algorithm and its extensions are always influenced by initializations with a necessary number of clusters a priori. That is, the k-means algorithm is not exactly an unsupervised clustering method. In this paper, we construct an unsupervised learning schema for the k-means algorithm so that it is free of initializations without parameter selection and can also simultaneously find an optimal number of clusters." from the abstract. 

### Describe your proposed solution

"To construct the k-means clustering algorithm with free of initializations and automatically find the number of clusters, we use the entropy concept. We borrow the idea from the EM algorithm by Yang et al." from section 3 (THE UNSUPERVISED K-MEANS CLUSTERING ALGORITHM).

### Describe alternatives you've considered, if relevant

R-EM, C-FS, X-Means, and RL-FCM

### Additional context

https://ieeexplore.ieee.org/document/9072123

309 citations (according to Google Scholar, not sure why IEEE shows a different number), released in 20 April 2020. I think there are up to 5 parameters "γ, β, c, a K, a K,"  might be wrong.

The total computational complexity for the U-k-means algorithm is O(ncd), where n is the number of data points, c is the number of clusters, and d is the dimension of data points.

Name of the authors: Kristina P. Sinaga, Miin-Shen Yang

COMMENT:
@MrinalTyagi Do you think this would require a seperate estimator? Or could it be implemented in an existing one by specifying ```n_clusters='auto'```?

Labels: New Feature, module:cluster, Needs Decision - Include Feature