Talip7/scikit-learn-issues-multilabel huggingface dataset:
New algorithm: DP-Means

### Describe the workflow you want to enable

DP-Means is a nonparametric version of K-Means, which is rooted in the Dirichlet Process Mixture Model.
Proposed by[ Kulis and Jordan in 2011](https://arxiv.org/pdf/1111.0352.pdf), it is already a well cited and established algorithm.

Recently, we have published a version of it which scales very good, comparable to K-Means and is equivalent to the original algorithm (https://openreview.net/pdf?id=rnzVBD8jqlq).

Enabling it in scikit-learn would benefit the community, and would be great in my opinion.

### Describe your proposed solution

We have already utilized scikit-learn codebase in order to implement our algorithm efficiently, (https://github.com/BGU-CS-VIL/pdc-dp-means/tree/main/cluster), so I can easily fix this into a pull-request to scikit-learn code base (will need to add the relevant tests and docs as well).

As such, if this feature request is approved, I will work on the relevant pull request in order to add this to the code base.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
It's a bit more than that, the relation between the DPGMM and DP-Means is very similar to these of GMM and K-Means, e.g. if we use hard assignment and constant isotropic covariance's for the clusters.

pdc-dp-means is part of our lab codebase, so licensing shouldn't be a problem anyway.

I will start working on turning this into a pip installable package, with tests, docs and examples. The transition would be smoother in any case.

Thanks!

Labels: New Feature, module:cluster, Needs Decision - Include Feature
New algorithm: DP-Means

### Describe the workflow you want to enable

DP-Means is a nonparametric version of K-Means, which is rooted in the Dirichlet Process Mixture Model.
Proposed by[ Kulis and Jordan in 2011](https://arxiv.org/pdf/1111.0352.pdf), it is already a well cited and established algorithm.

Recently, we have published a version of it which scales very good, comparable to K-Means and is equivalent to the original algorithm (https://openreview.net/pdf?id=rnzVBD8jqlq).

Enabling it in scikit-learn would benefit the community, and would be great in my opinion.

### Describe your proposed solution

We have already utilized scikit-learn codebase in order to implement our algorithm efficiently, (https://github.com/BGU-CS-VIL/pdc-dp-means/tree/main/cluster), so I can easily fix this into a pull-request to scikit-learn code base (will need to add the relevant tests and docs as well).

As such, if this feature request is approved, I will work on the relevant pull request in order to add this to the code base.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
Took some time, but i "packaged" the code -

https://github.com/BGU-CS-VIL/pdc-dp-means

It still heavily relies on scikit-learn optimized functions, and got some issues when building for windows I was not yet able to resolve, but it is independent now with docs,tests and examples :)

Labels: New Feature, module:cluster, Needs Decision - Include Feature
Multi-class roc_auc_score raises error when y_true is not sampled with all label of classes

### Describe the bug

Sometimes we would like to train or validate a multi-class classification model without using large batch size or the term **n_sample** in scikit-learn but with too many number of classes **n_classes**. Let's say **n_sample**  < **n_classes**.  For the example below, `n_sample = 2` and `n_classes = 3`.  

In deep learning, huge model with high resolution input could lead to this situation easily. The [condition](https://github.com/scikit-learn/scikit-learn/blob/main/sklearn/metrics/_ranking.py#L682) might be too strong.

### Steps/Code to Reproduce

```
from sklearn.metrics import roc_auc_score
import numpy as np
from scipy.special import softmax
label = np.array([1, 2]) # ground truth label (n_sample,)
pred = softmax(np.random.random((2, 3)), axis=1) # model prediction (n_sample, n_class)
roc_auc_score(label, pred, average=None, multi_class="ovr")
```

### Expected Results

`array([auc1, auc2, 0])`

### Actual Results

```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/conda/lib/python3.8/site-packages/sklearn/metrics/_ranking.py", line 565, in roc_auc_score
    return _multiclass_roc_auc_score(
  File "/opt/conda/lib/python3.8/site-packages/sklearn/metrics/_ranking.py", line 674, in _multiclass_roc_auc_score
    raise ValueError(
ValueError: Number of classes in y_true not equal to the number of columns in 'y_score'
```

### Versions

```shell
System:
    python: 3.8.13 | packaged by conda-forge | (default, Mar 25 2022, 06:04:10)  [GCC 10.3.0]
executable: /opt/conda/bin/python
   machine: Linux-4.15.0-171-generic-x86_64-with-glibc2.10

Python dependencies:
      sklearn: 1.1.2
          pip: 22.2.2
   setuptools: 59.5.0
        numpy: 1.22.3
        scipy: 1.6.3
       Cython: 0.29.28
       pandas: 1.3.5
   matplotlib: 3.5.2
       joblib: 1.1.0
threadpoolctl: 3.1.0

Built with OpenMP: True

threadpoolctl info:
       user_api: openmp
   internal_api: openmp
         prefix: libgomp
       filepath: /opt/conda/lib/python3.8/site-packages/scikit_learn.libs/libgomp-a34b3233.so.1.0.0
        version: None
    num_threads: 48

       user_api: blas
   internal_api: openblas
         prefix: libopenblas
       filepath: /opt/conda/lib/libopenblasp-r0.3.20.so
        version: 0.3.20
threading_layer: pthreads
   architecture: SkylakeX
    num_threads: 48
```

COMMENT:
> roc_auc_score(label, pred, average=None, multi_class="ovr")

Does passing the `labels` argument help? Like so `roc_auc_score(label, pred, average=None, multi_class='ovr', labels=[0,1,2])`

Related: #11777

Labels: Bug, Needs Triage
Multi-class roc_auc_score raises error when y_true is not sampled with all label of classes

### Describe the bug

Sometimes we would like to train or validate a multi-class classification model without using large batch size or the term **n_sample** in scikit-learn but with too many number of classes **n_classes**. Let's say **n_sample**  < **n_classes**.  For the example below, `n_sample = 2` and `n_classes = 3`.  

In deep learning, huge model with high resolution input could lead to this situation easily. The [condition](https://github.com/scikit-learn/scikit-learn/blob/main/sklearn/metrics/_ranking.py#L682) might be too strong.

### Steps/Code to Reproduce

```
from sklearn.metrics import roc_auc_score
import numpy as np
from scipy.special import softmax
label = np.array([1, 2]) # ground truth label (n_sample,)
pred = softmax(np.random.random((2, 3)), axis=1) # model prediction (n_sample, n_class)
roc_auc_score(label, pred, average=None, multi_class="ovr")
```

### Expected Results

`array([auc1, auc2, 0])`

### Actual Results

```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/conda/lib/python3.8/site-packages/sklearn/metrics/_ranking.py", line 565, in roc_auc_score
    return _multiclass_roc_auc_score(
  File "/opt/conda/lib/python3.8/site-packages/sklearn/metrics/_ranking.py", line 674, in _multiclass_roc_auc_score
    raise ValueError(
ValueError: Number of classes in y_true not equal to the number of columns in 'y_score'
```

### Versions

```shell
System:
    python: 3.8.13 | packaged by conda-forge | (default, Mar 25 2022, 06:04:10)  [GCC 10.3.0]
executable: /opt/conda/bin/python
   machine: Linux-4.15.0-171-generic-x86_64-with-glibc2.10

Python dependencies:
      sklearn: 1.1.2
          pip: 22.2.2
   setuptools: 59.5.0
        numpy: 1.22.3
        scipy: 1.6.3
       Cython: 0.29.28
       pandas: 1.3.5
   matplotlib: 3.5.2
       joblib: 1.1.0
threadpoolctl: 3.1.0

Built with OpenMP: True

threadpoolctl info:
       user_api: openmp
   internal_api: openmp
         prefix: libgomp
       filepath: /opt/conda/lib/python3.8/site-packages/scikit_learn.libs/libgomp-a34b3233.so.1.0.0
        version: None
    num_threads: 48

       user_api: blas
   internal_api: openblas
         prefix: libopenblas
       filepath: /opt/conda/lib/libopenblasp-r0.3.20.so
        version: 0.3.20
threading_layer: pthreads
   architecture: SkylakeX
    num_threads: 48
```

COMMENT:
Hi @thatgeeman,
I think it won't help since `_unique(y_true)` is actually being used before all the calculation. see https://github.com/scikit-learn/scikit-learn/blob/main/sklearn/metrics/_ranking.py#L681

Labels: Bug, Needs Triage
Multi-class roc_auc_score raises error when y_true is not sampled with all label of classes

### Describe the bug

Sometimes we would like to train or validate a multi-class classification model without using large batch size or the term **n_sample** in scikit-learn but with too many number of classes **n_classes**. Let's say **n_sample**  < **n_classes**.  For the example below, `n_sample = 2` and `n_classes = 3`.  

In deep learning, huge model with high resolution input could lead to this situation easily. The [condition](https://github.com/scikit-learn/scikit-learn/blob/main/sklearn/metrics/_ranking.py#L682) might be too strong.

### Steps/Code to Reproduce

```
from sklearn.metrics import roc_auc_score
import numpy as np
from scipy.special import softmax
label = np.array([1, 2]) # ground truth label (n_sample,)
pred = softmax(np.random.random((2, 3)), axis=1) # model prediction (n_sample, n_class)
roc_auc_score(label, pred, average=None, multi_class="ovr")
```

### Expected Results

`array([auc1, auc2, 0])`

### Actual Results

```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/conda/lib/python3.8/site-packages/sklearn/metrics/_ranking.py", line 565, in roc_auc_score
    return _multiclass_roc_auc_score(
  File "/opt/conda/lib/python3.8/site-packages/sklearn/metrics/_ranking.py", line 674, in _multiclass_roc_auc_score
    raise ValueError(
ValueError: Number of classes in y_true not equal to the number of columns in 'y_score'
```

### Versions

```shell
System:
    python: 3.8.13 | packaged by conda-forge | (default, Mar 25 2022, 06:04:10)  [GCC 10.3.0]
executable: /opt/conda/bin/python
   machine: Linux-4.15.0-171-generic-x86_64-with-glibc2.10

Python dependencies:
      sklearn: 1.1.2
          pip: 22.2.2
   setuptools: 59.5.0
        numpy: 1.22.3
        scipy: 1.6.3
       Cython: 0.29.28
       pandas: 1.3.5
   matplotlib: 3.5.2
       joblib: 1.1.0
threadpoolctl: 3.1.0

Built with OpenMP: True

threadpoolctl info:
       user_api: openmp
   internal_api: openmp
         prefix: libgomp
       filepath: /opt/conda/lib/python3.8/site-packages/scikit_learn.libs/libgomp-a34b3233.so.1.0.0
        version: None
    num_threads: 48

       user_api: blas
   internal_api: openblas
         prefix: libopenblas
       filepath: /opt/conda/lib/libopenblasp-r0.3.20.so
        version: 0.3.20
threading_layer: pthreads
   architecture: SkylakeX
    num_threads: 48
```

COMMENT:
Why was this issue marked as completed? Is there a solution to it?

Labels: Bug, Needs Triage
Multi-class roc_auc_score raises error when y_true is not sampled with all label of classes

### Describe the bug

Sometimes we would like to train or validate a multi-class classification model without using large batch size or the term **n_sample** in scikit-learn but with too many number of classes **n_classes**. Let's say **n_sample**  < **n_classes**.  For the example below, `n_sample = 2` and `n_classes = 3`.  

In deep learning, huge model with high resolution input could lead to this situation easily. The [condition](https://github.com/scikit-learn/scikit-learn/blob/main/sklearn/metrics/_ranking.py#L682) might be too strong.

### Steps/Code to Reproduce

```
from sklearn.metrics import roc_auc_score
import numpy as np
from scipy.special import softmax
label = np.array([1, 2]) # ground truth label (n_sample,)
pred = softmax(np.random.random((2, 3)), axis=1) # model prediction (n_sample, n_class)
roc_auc_score(label, pred, average=None, multi_class="ovr")
```

### Expected Results

`array([auc1, auc2, 0])`

### Actual Results

```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/conda/lib/python3.8/site-packages/sklearn/metrics/_ranking.py", line 565, in roc_auc_score
    return _multiclass_roc_auc_score(
  File "/opt/conda/lib/python3.8/site-packages/sklearn/metrics/_ranking.py", line 674, in _multiclass_roc_auc_score
    raise ValueError(
ValueError: Number of classes in y_true not equal to the number of columns in 'y_score'
```

### Versions

```shell
System:
    python: 3.8.13 | packaged by conda-forge | (default, Mar 25 2022, 06:04:10)  [GCC 10.3.0]
executable: /opt/conda/bin/python
   machine: Linux-4.15.0-171-generic-x86_64-with-glibc2.10

Python dependencies:
      sklearn: 1.1.2
          pip: 22.2.2
   setuptools: 59.5.0
        numpy: 1.22.3
        scipy: 1.6.3
       Cython: 0.29.28
       pandas: 1.3.5
   matplotlib: 3.5.2
       joblib: 1.1.0
threadpoolctl: 3.1.0

Built with OpenMP: True

threadpoolctl info:
       user_api: openmp
   internal_api: openmp
         prefix: libgomp
       filepath: /opt/conda/lib/python3.8/site-packages/scikit_learn.libs/libgomp-a34b3233.so.1.0.0
        version: None
    num_threads: 48

       user_api: blas
   internal_api: openblas
         prefix: libopenblas
       filepath: /opt/conda/lib/libopenblasp-r0.3.20.so
        version: 0.3.20
threading_layer: pthreads
   architecture: SkylakeX
    num_threads: 48
```

COMMENT:
> Why was this issue marked as completed? Is there a solution to it?

Look at the last line of this comment:

https://github.com/scikit-learn/scikit-learn/issues/24636#issuecomment-1289287515

Labels: Bug, Needs Triage
HalvingRandomSearchCV error

### Describe the bug

https://github.com/scikit-learn/scikit-learn/blob/main/sklearn/model_selection/_search_successive_halving.py#L220
when resources is the hyperparameter of estimator, the best performance is not always at its maximum, right?. so in my opinion, in this case, we should select the best in whole iter instead of the best in last iter.

### Steps/Code to Reproduce

```py
from sklearn.datasets import load_iris
from sklearn.ensemble import RandomForestClassifier
from sklearn.experimental import enable_halving_search_cv
from sklearn.model_selection import  HalvingRandomSearchCV, KFold

X, y = load_iris(return_X_y=True)
clf = RandomForestClassifier(n_estimators=10, random_state=0)  

kf = KFold(shuffle=False)
param_distributions = {"max_depth": [2, 3, 4], "min_samples_split": list(range(2, 12))}
search = HalvingRandomSearchCV(clf, param_distributions, min_resources=15, max_resources=150,factor=3,
                            random_state=0, n_jobs=1,cv=kf, **resource='n_estimators'**
                            ).fit(X, y)

print(search.best_estimator_)
```

### Expected Results

```
RandomForestClassifier(max_depth=3, min_samples_split=5, n_estimators=15,
                       random_state=0)
```

### Actual Results

```
RandomForestClassifier(max_depth=4, min_samples_split=4, **n_estimators=135**,random_state=0)
```

### Versions

```shell
System:
    python: 3.8.13 | packaged by conda-forge | (default, Mar 25 2022, 05:59:45) [MSC v.1929 64 bit (AMD64)]
executable: C:\software\anaconda\python.exe
   machine: Windows-10-10.0.19043-SP0

Python dependencies:
          pip: 22.0.4
   setuptools: 62.1.0
      sklearn: 1.0.2
        numpy: 1.21.5
        scipy: 1.8.0
       Cython: 0.29.28
       pandas: 1.4.2
   matplotlib: 3.5.1
       joblib: 1.1.0
threadpoolctl: 3.1.0
```

COMMENT:
I agree that it is a bit unusual for the best set of parameters to not be the ones that achieved the highest test score. I wonder how often this happens with real world datasets.

Successive halving is like a tournament where at each round different parameters are compared with each other. And only the best from that round progress to the next. The winner of the tournament is the best one in the last round. With this formulation it makes sense that the winner isn't the overall best parameter combination, but the one from the last round. From a quick look at the various papers I can find that describe this algorithm, they all describe the algorithm as "play several rounds, after each round keep the top 50%, play a new round, continue until you have one winner".

So I'd file this under "edge case that happens with toy datasets".

Labels: Documentation, module:model_selection
HalvingRandomSearchCV error

### Describe the bug

https://github.com/scikit-learn/scikit-learn/blob/main/sklearn/model_selection/_search_successive_halving.py#L220
when resources is the hyperparameter of estimator, the best performance is not always at its maximum, right?. so in my opinion, in this case, we should select the best in whole iter instead of the best in last iter.

### Steps/Code to Reproduce

```py
from sklearn.datasets import load_iris
from sklearn.ensemble import RandomForestClassifier
from sklearn.experimental import enable_halving_search_cv
from sklearn.model_selection import  HalvingRandomSearchCV, KFold

X, y = load_iris(return_X_y=True)
clf = RandomForestClassifier(n_estimators=10, random_state=0)  

kf = KFold(shuffle=False)
param_distributions = {"max_depth": [2, 3, 4], "min_samples_split": list(range(2, 12))}
search = HalvingRandomSearchCV(clf, param_distributions, min_resources=15, max_resources=150,factor=3,
                            random_state=0, n_jobs=1,cv=kf, **resource='n_estimators'**
                            ).fit(X, y)

print(search.best_estimator_)
```

### Expected Results

```
RandomForestClassifier(max_depth=3, min_samples_split=5, n_estimators=15,
                       random_state=0)
```

### Actual Results

```
RandomForestClassifier(max_depth=4, min_samples_split=4, **n_estimators=135**,random_state=0)
```

### Versions

```shell
System:
    python: 3.8.13 | packaged by conda-forge | (default, Mar 25 2022, 05:59:45) [MSC v.1929 64 bit (AMD64)]
executable: C:\software\anaconda\python.exe
   machine: Windows-10-10.0.19043-SP0

Python dependencies:
          pip: 22.0.4
   setuptools: 62.1.0
      sklearn: 1.0.2
        numpy: 1.21.5
        scipy: 1.8.0
       Cython: 0.29.28
       pandas: 1.4.2
   matplotlib: 3.5.1
       joblib: 1.1.0
threadpoolctl: 3.1.0
```

COMMENT:
I assume that this is a limitation of the current implementation and it makes sense for `max_iter` or `n_estimators`. We can improve the documentation stating that the type of resource is expected to bring improvement when it increases.

@betatim WDYT?

Labels: Documentation, module:model_selection
HalvingRandomSearchCV error

### Describe the bug

https://github.com/scikit-learn/scikit-learn/blob/main/sklearn/model_selection/_search_successive_halving.py#L220
when resources is the hyperparameter of estimator, the best performance is not always at its maximum, right?. so in my opinion, in this case, we should select the best in whole iter instead of the best in last iter.

### Steps/Code to Reproduce

```py
from sklearn.datasets import load_iris
from sklearn.ensemble import RandomForestClassifier
from sklearn.experimental import enable_halving_search_cv
from sklearn.model_selection import  HalvingRandomSearchCV, KFold

X, y = load_iris(return_X_y=True)
clf = RandomForestClassifier(n_estimators=10, random_state=0)  

kf = KFold(shuffle=False)
param_distributions = {"max_depth": [2, 3, 4], "min_samples_split": list(range(2, 12))}
search = HalvingRandomSearchCV(clf, param_distributions, min_resources=15, max_resources=150,factor=3,
                            random_state=0, n_jobs=1,cv=kf, **resource='n_estimators'**
                            ).fit(X, y)

print(search.best_estimator_)
```

### Expected Results

```
RandomForestClassifier(max_depth=3, min_samples_split=5, n_estimators=15,
                       random_state=0)
```

### Actual Results

```
RandomForestClassifier(max_depth=4, min_samples_split=4, **n_estimators=135**,random_state=0)
```

### Versions

```shell
System:
    python: 3.8.13 | packaged by conda-forge | (default, Mar 25 2022, 05:59:45) [MSC v.1929 64 bit (AMD64)]
executable: C:\software\anaconda\python.exe
   machine: Windows-10-10.0.19043-SP0

Python dependencies:
          pip: 22.0.4
   setuptools: 62.1.0
      sklearn: 1.0.2
        numpy: 1.21.5
        scipy: 1.8.0
       Cython: 0.29.28
       pandas: 1.4.2
   matplotlib: 3.5.1
       joblib: 1.1.0
threadpoolctl: 3.1.0
```

COMMENT:
I think improving the docs would be good. Maybe in the user guide https://scikit-learn.org/stable/modules/grid_search.html#successive-halving-user-guide? Something like noting that you should choose your resource increase "large enough" so that you get a "large improvement in scores when taking into account statistical significance". I think increasing `n_estimators` from 10 to 15 probably doesn't make a big difference to the score, so you just increase your chances of seeing something "weird" like this.

I had a quick look at the scores and I think they are all quite close to another, which makes me wonder if the differences we see are statistically significant or not.

Labels: Documentation, module:model_selection
HalvingRandomSearchCV error

### Describe the bug

https://github.com/scikit-learn/scikit-learn/blob/main/sklearn/model_selection/_search_successive_halving.py#L220
when resources is the hyperparameter of estimator, the best performance is not always at its maximum, right?. so in my opinion, in this case, we should select the best in whole iter instead of the best in last iter.

### Steps/Code to Reproduce

```py
from sklearn.datasets import load_iris
from sklearn.ensemble import RandomForestClassifier
from sklearn.experimental import enable_halving_search_cv
from sklearn.model_selection import  HalvingRandomSearchCV, KFold

X, y = load_iris(return_X_y=True)
clf = RandomForestClassifier(n_estimators=10, random_state=0)  

kf = KFold(shuffle=False)
param_distributions = {"max_depth": [2, 3, 4], "min_samples_split": list(range(2, 12))}
search = HalvingRandomSearchCV(clf, param_distributions, min_resources=15, max_resources=150,factor=3,
                            random_state=0, n_jobs=1,cv=kf, **resource='n_estimators'**
                            ).fit(X, y)

print(search.best_estimator_)
```

### Expected Results

```
RandomForestClassifier(max_depth=3, min_samples_split=5, n_estimators=15,
                       random_state=0)
```

### Actual Results

```
RandomForestClassifier(max_depth=4, min_samples_split=4, **n_estimators=135**,random_state=0)
```

### Versions

```shell
System:
    python: 3.8.13 | packaged by conda-forge | (default, Mar 25 2022, 05:59:45) [MSC v.1929 64 bit (AMD64)]
executable: C:\software\anaconda\python.exe
   machine: Windows-10-10.0.19043-SP0

Python dependencies:
          pip: 22.0.4
   setuptools: 62.1.0
      sklearn: 1.0.2
        numpy: 1.21.5
        scipy: 1.8.0
       Cython: 0.29.28
       pandas: 1.4.2
   matplotlib: 3.5.1
       joblib: 1.1.0
threadpoolctl: 3.1.0
```

COMMENT:
> I had a quick look at the scores and I think they are all quite close to another, which makes me wonder if the differences we see are statistically significant or not.

I would not be surprised to get these results with random forest: a plateau will be reached while increasing `n_estimators`. One would probably want to pick the smallest forest to get a speed-up. However, we suffer the exact same issue with a `GridSearchCV`: we select based on the mean test score while we might be interested in other factors such as prediction speed.

Labels: Documentation, module:model_selection