Talip7/scikit-learn-issues-multilabel huggingface dataset:
StratifiedKFold and StratifiedGroupKFold for multilabel classification

### Describe the workflow you want to enable

Currently, these two functionalities only support binary/multiclass classification. I am looking for similar functionalities (maybe not even k fold, a train_test_split also suffices) in multilabel classification. 

### Describe your proposed solution

For the multilabel version of stratification, [this](https://github.com/trent-b/iterative-stratification) and [this](http://scikit.ml/api/skmultilearn.model_selection.iterative_stratification.html#module-skmultilearn.model_selection.iterative_stratification) libraries seem already provided a solution. Adding grouping to them should be similar to how `StratifiedGroupKFold` adds to `StratifiedKFold`.

### Describe alternatives you've considered, if relevant

See above for existing libraries developed based upon Sklearn.

### Additional context

_No response_

COMMENT:
Seems that this issue hasn't been updated for a long time. If nobody is working on this, I can make a PR starting from `MultilabelStratifiedKFold`.

Labels: New Feature, module:model_selection
StratifiedKFold and StratifiedGroupKFold for multilabel classification

### Describe the workflow you want to enable

Currently, these two functionalities only support binary/multiclass classification. I am looking for similar functionalities (maybe not even k fold, a train_test_split also suffices) in multilabel classification. 

### Describe your proposed solution

For the multilabel version of stratification, [this](https://github.com/trent-b/iterative-stratification) and [this](http://scikit.ml/api/skmultilearn.model_selection.iterative_stratification.html#module-skmultilearn.model_selection.iterative_stratification) libraries seem already provided a solution. Adding grouping to them should be similar to how `StratifiedGroupKFold` adds to `StratifiedKFold`.

### Describe alternatives you've considered, if relevant

See above for existing libraries developed based upon Sklearn.

### Additional context

_No response_

COMMENT:
Any progress now?
Is there a chance to use this feature in the next release?

Labels: New Feature, module:model_selection
StratifiedKFold and StratifiedGroupKFold for multilabel classification

### Describe the workflow you want to enable

Currently, these two functionalities only support binary/multiclass classification. I am looking for similar functionalities (maybe not even k fold, a train_test_split also suffices) in multilabel classification. 

### Describe your proposed solution

For the multilabel version of stratification, [this](https://github.com/trent-b/iterative-stratification) and [this](http://scikit.ml/api/skmultilearn.model_selection.iterative_stratification.html#module-skmultilearn.model_selection.iterative_stratification) libraries seem already provided a solution. Adding grouping to them should be similar to how `StratifiedGroupKFold` adds to `StratifiedKFold`.

### Describe alternatives you've considered, if relevant

See above for existing libraries developed based upon Sklearn.

### Additional context

_No response_

COMMENT:
Any progress or issues faced?

Labels: New Feature, module:model_selection
StratifiedKFold and StratifiedGroupKFold for multilabel classification

### Describe the workflow you want to enable

Currently, these two functionalities only support binary/multiclass classification. I am looking for similar functionalities (maybe not even k fold, a train_test_split also suffices) in multilabel classification. 

### Describe your proposed solution

For the multilabel version of stratification, [this](https://github.com/trent-b/iterative-stratification) and [this](http://scikit.ml/api/skmultilearn.model_selection.iterative_stratification.html#module-skmultilearn.model_selection.iterative_stratification) libraries seem already provided a solution. Adding grouping to them should be similar to how `StratifiedGroupKFold` adds to `StratifiedKFold`.

### Describe alternatives you've considered, if relevant

See above for existing libraries developed based upon Sklearn.

### Additional context

_No response_

COMMENT:
@gerardsimons I believe we are targeting v1.5 (#26423) and I will start working on it soon.

Labels: New Feature, module:model_selection
Early Stopping for GridSearchCV, RandomizedSearchCV

### Describe the workflow you want to enable

- I have a custom model implementing the BaseEstimator, for which I am using scikit-learn's hyperparameter searches.
- I am running an exhaustive grid search, all possible parameters for my model.
- If one parameter setting gives a sufficiently high accuracy (e.g. F1 = 1.0), I wish to stop the grid search and free the compute for other runs.

### Describe your proposed solution

GridSearchCV, RandomizedSearchCV, and others should have an early stopping criteria. I should be able to specify a threshold accuracy, such that when the value returned by the scoring function passes this threshold, other jobs are stopped.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

* As parameter search happens in parallel, it may require some rewrite of the parallelization code so that results can be checked as they return: https://github.com/scikit-learn/scikit-learn/blob/main/sklearn/model_selection/_search.py#L822 
See, for instance, https://github.com/joblib/joblib/issues/356#issuecomment-224508490 
* I am opening this feature request to start a discussion; I am happy to do the development and submit a pull request if we agree on what should be built.

COMMENT:
One additional comment: for my use case, I have  ~40 datasets, each of which I am training a model with ~72 runs in the grid, of which 0-20 of the runs might reach 1.0 F1 depending on the dataset. So an early stop would allow me to avoid unnecessary runs, reducing the amount of compute I need by roughly an order of magnitude.

Labels: New Feature, module:model_selection
Early Stopping for GridSearchCV, RandomizedSearchCV

### Describe the workflow you want to enable

- I have a custom model implementing the BaseEstimator, for which I am using scikit-learn's hyperparameter searches.
- I am running an exhaustive grid search, all possible parameters for my model.
- If one parameter setting gives a sufficiently high accuracy (e.g. F1 = 1.0), I wish to stop the grid search and free the compute for other runs.

### Describe your proposed solution

GridSearchCV, RandomizedSearchCV, and others should have an early stopping criteria. I should be able to specify a threshold accuracy, such that when the value returned by the scoring function passes this threshold, other jobs are stopped.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

* As parameter search happens in parallel, it may require some rewrite of the parallelization code so that results can be checked as they return: https://github.com/scikit-learn/scikit-learn/blob/main/sklearn/model_selection/_search.py#L822 
See, for instance, https://github.com/joblib/joblib/issues/356#issuecomment-224508490 
* I am opening this feature request to start a discussion; I am happy to do the development and submit a pull request if we agree on what should be built.

COMMENT:
And a final comment: this is a common enough request that it exists in other parameter search libraries:
https://docs.ray.io/en/latest/tune/api_docs/stoppers.html 
https://www.databricks.com/blog/2021/04/15/how-not-to-tune-your-model-with-hyperopt.html#:~:text=Use%20Early%20Stopping&text=It's%20possible%20that%20Hyperopt%20struggles,trials%20if%20progress%20has%20stopped.

Labels: New Feature, module:model_selection
Early Stopping for GridSearchCV, RandomizedSearchCV

### Describe the workflow you want to enable

- I have a custom model implementing the BaseEstimator, for which I am using scikit-learn's hyperparameter searches.
- I am running an exhaustive grid search, all possible parameters for my model.
- If one parameter setting gives a sufficiently high accuracy (e.g. F1 = 1.0), I wish to stop the grid search and free the compute for other runs.

### Describe your proposed solution

GridSearchCV, RandomizedSearchCV, and others should have an early stopping criteria. I should be able to specify a threshold accuracy, such that when the value returned by the scoring function passes this threshold, other jobs are stopped.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

* As parameter search happens in parallel, it may require some rewrite of the parallelization code so that results can be checked as they return: https://github.com/scikit-learn/scikit-learn/blob/main/sklearn/model_selection/_search.py#L822 
See, for instance, https://github.com/joblib/joblib/issues/356#issuecomment-224508490 
* I am opening this feature request to start a discussion; I am happy to do the development and submit a pull request if we agree on what should be built.

COMMENT:
This would be an interesting feature. To have a clean implementation, I think that we should `joblib` returning some generators to be able to intervene with the execution: https://github.com/joblib/joblib/pull/588

I assume that this generator feature could be useful for early-stopping in estimators like random forest or callbacks as well.

Labels: New Feature, module:model_selection
Early Stopping for GridSearchCV, RandomizedSearchCV

### Describe the workflow you want to enable

- I have a custom model implementing the BaseEstimator, for which I am using scikit-learn's hyperparameter searches.
- I am running an exhaustive grid search, all possible parameters for my model.
- If one parameter setting gives a sufficiently high accuracy (e.g. F1 = 1.0), I wish to stop the grid search and free the compute for other runs.

### Describe your proposed solution

GridSearchCV, RandomizedSearchCV, and others should have an early stopping criteria. I should be able to specify a threshold accuracy, such that when the value returned by the scoring function passes this threshold, other jobs are stopped.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

* As parameter search happens in parallel, it may require some rewrite of the parallelization code so that results can be checked as they return: https://github.com/scikit-learn/scikit-learn/blob/main/sklearn/model_selection/_search.py#L822 
See, for instance, https://github.com/joblib/joblib/issues/356#issuecomment-224508490 
* I am opening this feature request to start a discussion; I am happy to do the development and submit a pull request if we agree on what should be built.

COMMENT:
Can you expand on the kinds of changes (if any) that joblib would need to have? Does this issue need functionality from https://github.com/joblib/joblib/pull/588

Labels: New Feature, module:model_selection
Early Stopping for GridSearchCV, RandomizedSearchCV

### Describe the workflow you want to enable

- I have a custom model implementing the BaseEstimator, for which I am using scikit-learn's hyperparameter searches.
- I am running an exhaustive grid search, all possible parameters for my model.
- If one parameter setting gives a sufficiently high accuracy (e.g. F1 = 1.0), I wish to stop the grid search and free the compute for other runs.

### Describe your proposed solution

GridSearchCV, RandomizedSearchCV, and others should have an early stopping criteria. I should be able to specify a threshold accuracy, such that when the value returned by the scoring function passes this threshold, other jobs are stopped.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

* As parameter search happens in parallel, it may require some rewrite of the parallelization code so that results can be checked as they return: https://github.com/scikit-learn/scikit-learn/blob/main/sklearn/model_selection/_search.py#L822 
See, for instance, https://github.com/joblib/joblib/issues/356#issuecomment-224508490 
* I am opening this feature request to start a discussion; I am happy to do the development and submit a pull request if we agree on what should be built.

COMMENT:
Yes, because we can cleanly stop the parallel processing by stopping consuming the generator.

Labels: New Feature, module:model_selection
RandomForestClassifier outputs float instead of bool class-labels when fit on a pd.Series since 1.2.0

### Describe the bug

```
import numpy as np
import pandas as pd
import sklearn
from sklearn.ensemble import RandomForestClassifier

# 1.23.3 1.5.0 1.2.0
print(np.__version__, pd.__version__, sklearn.__version__)

x = np.random.random((20, 1))
y = x[:, 0] >= 0.3

x = pd.DataFrame(x)     # remove these and it works
y = pd.Series(y)    # remove these and it works

r = RandomForestClassifier()
r.fit(x, y)


yhat = r.predict(x)
assert np.issubdtype(yhat.dtype, np.bool_)  # fails since scikit 1.2.0
```


### Steps/Code to Reproduce

```
import numpy as np
import pandas as pd
import sklearn
from sklearn.ensemble import RandomForestClassifier

# 1.23.3 1.5.0 1.2.0
print(np.__version__, pd.__version__, sklearn.__version__)

x = np.random.random((20, 1))
y = x[:, 0] >= 0.3

x = pd.DataFrame(x)     # remove these and it works
y = pd.Series(y)    # remove these and it works

r = RandomForestClassifier()
r.fit(x, y)


yhat = r.predict(x)
print(yhat)
assert np.issubdtype(yhat.dtype, np.bool_)  # fails since scikit 1.2.0
```

### Expected Results

[True False False True ...]

### Actual Results

[1. 0. 0. 1. ...]
Traceback (most recent call last):
  File "/Users/XXX/PycharmProjects/flow/bla.py", line 21, in <module>
    assert np.issubdtype(yhat.dtype, np.bool_)
AssertionError

### Versions

```shell
System:
    python: 3.9.13 (main, Sep 21 2022, 15:17:17)  [Clang 13.0.0 (clang-1300.0.29.30)]
executable: /Users/niklas.fruehauf/.pyenv/versions/flow-venv-3.9/bin/python
   machine: macOS-12.6.1-x86_64-i386-64bit

Python dependencies:
      sklearn: 1.2.0
          pip: 22.3.1
   setuptools: 59.8.0
        numpy: 1.23.3
        scipy: 1.8.1
       Cython: 0.29.32
       pandas: 1.5.0
   matplotlib: 3.5.3
       joblib: 1.2.0
threadpoolctl: 3.1.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: openblas
         prefix: libopenblas
       filepath: /Users/niklas.fruehauf/.pyenv/versions/3.9.13/envs/flow-venv-3.9/lib/python3.9/site-packages/numpy/.dylibs/libopenblas64_.0.dylib
        version: 0.3.20
threading_layer: pthreads
   architecture: Haswell
    num_threads: 6

       user_api: openmp
   internal_api: openmp
         prefix: libomp
       filepath: /Users/niklas.fruehauf/.pyenv/versions/3.9.13/envs/flow-venv-3.9/lib/python3.9/site-packages/sklearn/.dylibs/libomp.dylib
        version: None
    num_threads: 12

       user_api: blas
   internal_api: openblas
         prefix: libopenblas
       filepath: /Users/niklas.fruehauf/.pyenv/versions/3.9.13/envs/flow-venv-3.9/lib/python3.9/site-packages/scipy/.dylibs/libopenblas.0.dylib
        version: 0.3.17
threading_layer: pthreads
   architecture: Haswell
    num_threads: 6
```

COMMENT:
I have tested that the snippet in the opening comment works on `main`, which was recently fixed in #25147. The fix will be released in 1.2.1.

@Zahlii Thank you for reporting the issue and providing the reproducible example!

Labels: Bug, Needs Triage
OneHotEncoder cuts predefined classes

### Describe the bug

When having predefined categories for the OneHotEncoder the categories get cut off. This lead to an error when trying to transform samples with the categories present....

### Steps/Code to Reproduce

```python
from sklearn.preprocessing import OneHotEncoder
import numpy as np
from sklearn.preprocessing import OneHotEncoder

# I know all my categories in advance and there will be no other
# Using only one sample to call fit since I need to call at some point

list_of_bounds = [["as","mmas","eas","ras","acs"], ["1", "2"]]

one_sample = ["as", "1"]

o_h_enc = OneHotEncoder(categories=list_of_bounds)
o_h_enc.fit(np.array(one_sample).reshape(1, -1) )


print(o_h_enc.categories_)
# [array(['as', 'mm', 'ea', 'ra', 'ac'], dtype='<U2'), array(['1', '2'], dtype='<U2')]
# The first category should be dtype object and contain the full names i.e.
# [array(["as","mmas","eas","ras","acs"], dtype='object'), array(['1', '2'], dtype='<U2')]

o_h_enc.transform(np.array(["mmas", "1"]).reshape((1, -1)))
```

### Expected Results

[array(["as","mmas","eas","ras","acs"], dtype='object'), array(['1', '2'], dtype='<U2')]


### Actual Results

```# [array(['as', 'mm', 'ea', 'ra', 'ac'], dtype='<U2'), array(['1', '2'], dtype='<U2')]
```

### Versions

```shell
Python dependencies:
      sklearn: 1.2.0
          pip: 21.1.2
   setuptools: 57.0.0
        numpy: 1.23.4
        scipy: 1.9.3
       Cython: None
       pandas: None
   matplotlib: 3.6.2
       joblib: 1.2.0
threadpoolctl: 3.1.0

Built with OpenMP: True
None
```

COMMENT:
I think this happens because we convert the category for each feature to the dtype of the input ([on this line](https://github.com/scikit-learn/scikit-learn/blob/754bd5245aa46b89a1d686a3326c2b853012ff3e/sklearn/preprocessing/_encoders.py#L100)).

In this case `<U2` is used because the input is a two character string. I'm not sure if it should be of type `object`, though. I would have guessed that the dtype should be `<U4`, a four character string as that is the dtype of the known category.

However, when you don't provide categories to the constructor, then the resulting dtype is `object`. So maybe that is the correct dtype.

Labels: Bug
OneHotEncoder cuts predefined classes

### Describe the bug

When having predefined categories for the OneHotEncoder the categories get cut off. This lead to an error when trying to transform samples with the categories present....

### Steps/Code to Reproduce

```python
from sklearn.preprocessing import OneHotEncoder
import numpy as np
from sklearn.preprocessing import OneHotEncoder

# I know all my categories in advance and there will be no other
# Using only one sample to call fit since I need to call at some point

list_of_bounds = [["as","mmas","eas","ras","acs"], ["1", "2"]]

one_sample = ["as", "1"]

o_h_enc = OneHotEncoder(categories=list_of_bounds)
o_h_enc.fit(np.array(one_sample).reshape(1, -1) )


print(o_h_enc.categories_)
# [array(['as', 'mm', 'ea', 'ra', 'ac'], dtype='<U2'), array(['1', '2'], dtype='<U2')]
# The first category should be dtype object and contain the full names i.e.
# [array(["as","mmas","eas","ras","acs"], dtype='object'), array(['1', '2'], dtype='<U2')]

o_h_enc.transform(np.array(["mmas", "1"]).reshape((1, -1)))
```

### Expected Results

[array(["as","mmas","eas","ras","acs"], dtype='object'), array(['1', '2'], dtype='<U2')]


### Actual Results

```# [array(['as', 'mm', 'ea', 'ra', 'ac'], dtype='<U2'), array(['1', '2'], dtype='<U2')]
```

### Versions

```shell
Python dependencies:
      sklearn: 1.2.0
          pip: 21.1.2
   setuptools: 57.0.0
        numpy: 1.23.4
        scipy: 1.9.3
       Cython: None
       pandas: None
   matplotlib: 3.6.2
       joblib: 1.2.0
threadpoolctl: 3.1.0

Built with OpenMP: True
None
```

COMMENT:
The PR #25174 linked above is an attempt to solve this. To help out you could take a look at the PR and leave feedback if you find something that should change.

Labels: Bug
DOC: type annotation for model returned from fit methods is "object"

### Describe the bug

I noticed that the type annotation for model returned from fit methods is "object". This makes IDE like pycharm unable to perform type hints:

![image](https://user-images.githubusercontent.com/48577571/206979644-9185bff3-e22f-47b0-b07d-d67e52776490.png)

![image](https://user-images.githubusercontent.com/48577571/206980081-21b877f7-b624-426d-9ce4-8ad39b08c6f1.png)


### Steps/Code to Reproduce

.

### Expected Results

The type of returned model should be the class itself.

### Actual Results

.

### Versions

```shell
System:
    python: 3.10.4 (main, Mar 31 2022, 08:41:55) [GCC 7.5.0]
executable: /home/auderson/miniconda3/envs/py3.10/bin/python
   machine: Linux-5.8.0-63-generic-x86_64-with-glibc2.32

Python dependencies:
      sklearn: 1.1.1
          pip: 22.3.1
   setuptools: 65.5.1
        numpy: 1.22.3
        scipy: 1.8.1
       Cython: 0.29.32
       pandas: 1.5.0
   matplotlib: 3.5.1
       joblib: 1.2.0
threadpoolctl: 3.1.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: openblas
         prefix: libopenblas
       filepath: /home/auderson/miniconda3/envs/py3.10/lib/libopenblasp-r0.3.21.so
        version: 0.3.21
threading_layer: pthreads
   architecture: Zen
    num_threads: 96

       user_api: openmp
   internal_api: openmp
         prefix: libomp
       filepath: /home/auderson/miniconda3/envs/py3.10/lib/libomp.so
        version: None
    num_threads: 1
```

COMMENT:
Thanks for reporting this @auderson 

Yes, we don't have specific classes for type annotating sklearn estimators. And since folks can inherit from our classes and change/add things, we can easily hard code class names there.

There were discussions about introducing `Protocol`s in our mixins, but I don't think it would change what we have in the docstring. What you're asking is not supported by us.

cc @glemaitre

Labels: Bug, Needs Triage
DOC: type annotation for model returned from fit methods is "object"

### Describe the bug

I noticed that the type annotation for model returned from fit methods is "object". This makes IDE like pycharm unable to perform type hints:

![image](https://user-images.githubusercontent.com/48577571/206979644-9185bff3-e22f-47b0-b07d-d67e52776490.png)

![image](https://user-images.githubusercontent.com/48577571/206980081-21b877f7-b624-426d-9ce4-8ad39b08c6f1.png)


### Steps/Code to Reproduce

.

### Expected Results

The type of returned model should be the class itself.

### Actual Results

.

### Versions

```shell
System:
    python: 3.10.4 (main, Mar 31 2022, 08:41:55) [GCC 7.5.0]
executable: /home/auderson/miniconda3/envs/py3.10/bin/python
   machine: Linux-5.8.0-63-generic-x86_64-with-glibc2.32

Python dependencies:
      sklearn: 1.1.1
          pip: 22.3.1
   setuptools: 65.5.1
        numpy: 1.22.3
        scipy: 1.8.1
       Cython: 0.29.32
       pandas: 1.5.0
   matplotlib: 3.5.1
       joblib: 1.2.0
threadpoolctl: 3.1.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: openblas
         prefix: libopenblas
       filepath: /home/auderson/miniconda3/envs/py3.10/lib/libopenblasp-r0.3.21.so
        version: 0.3.21
threading_layer: pthreads
   architecture: Zen
    num_threads: 96

       user_api: openmp
   internal_api: openmp
         prefix: libomp
       filepath: /home/auderson/miniconda3/envs/py3.10/lib/libomp.so
        version: None
    num_threads: 1
```

COMMENT:
I assume that the `Protocol` would not be enough. We should probably have to create our typing similar to Numpy:

- https://numpy.org/doc/stable/reference/typing.html#module-numpy.typing
- https://github.com/numpy/numpy/tree/main/numpy/_typing

Labels: Bug, Needs Triage