Talip7/scikit-learn-issues-multilabel huggingface dataset:
TST Add option to run tests on 32bit data

### Context

Currently most implementations are tested against 64bit datasets only. The re-factoring of some internals for computations on 32bit datasets brought the need to test user-facing interfaces on 32bit datasets (see https://github.com/scikit-learn/scikit-learn/pull/22590#issuecomment-1054546370). A few group of PRs (such as #22663) introduced a simple parametrisation on `dtype`:

```python
# ... 
DTYPES = (np.float32, np.float64)

# ...

@pytest.mark.parametrize("dtype", DTYPES)
def test(dtype):
    # ...
```

### Problem

Yet, parametrising tests on `dtype` might just make running the test suite (much) longer. Hence, we could add an option to be able to run such tests for 32bit data optionally. In this case, we also need to have a way to specify this option so that it can be triggered on the CI.

### Proposed solution

We can keep the tests parametrisation on `dtype` as proposed in PRs (such as #22663) but have the value of `DTYPES` be specified globally (e.g. `in sklearn/utils/_testing.py`) and adaptively on the value environement variable. Naively:

```python 
# sklearn/utils/_testing.py

# ...

DTYPES = [np.float64]

if os.environ.get("SKLEARN_RUN_32BIT_TESTS"):
    DTYPES.append(np.float32)
```


As for the CI, we could have a new `[test 32bit]` [commit message marker](https://scikit-learn.org/stable/developers/contributing.html#continuous-integration-ci) which would define this environement variable.

COMMENT:
> I think having the fixture on 1 job for the PR and all jobs for the nightly builds is a good compromise. Wdyt ?

It's a trade off between developer time and CI time. If there is a float32 issue in a PR that is not caught by the one PR job, then it requires more developer time to fix it after it is caught on nightly builds.

Let's enable the "1 job for the PR and all jobs for nightly" plan now. In the long term, I think it depends on how often does the above situation happens. If it's rare, then I'm okay with the compromise.

Labels: Build / CI, module:test-suite, workflow, float32
TST Add option to run tests on 32bit data

### Context

Currently most implementations are tested against 64bit datasets only. The re-factoring of some internals for computations on 32bit datasets brought the need to test user-facing interfaces on 32bit datasets (see https://github.com/scikit-learn/scikit-learn/pull/22590#issuecomment-1054546370). A few group of PRs (such as #22663) introduced a simple parametrisation on `dtype`:

```python
# ... 
DTYPES = (np.float32, np.float64)

# ...

@pytest.mark.parametrize("dtype", DTYPES)
def test(dtype):
    # ...
```

### Problem

Yet, parametrising tests on `dtype` might just make running the test suite (much) longer. Hence, we could add an option to be able to run such tests for 32bit data optionally. In this case, we also need to have a way to specify this option so that it can be triggered on the CI.

### Proposed solution

We can keep the tests parametrisation on `dtype` as proposed in PRs (such as #22663) but have the value of `DTYPES` be specified globally (e.g. `in sklearn/utils/_testing.py`) and adaptively on the value environement variable. Naively:

```python 
# sklearn/utils/_testing.py

# ...

DTYPES = [np.float64]

if os.environ.get("SKLEARN_RUN_32BIT_TESTS"):
    DTYPES.append(np.float32)
```


As for the CI, we could have a new `[test 32bit]` [commit message marker](https://scikit-learn.org/stable/developers/contributing.html#continuous-integration-ci) which would define this environement variable.

COMMENT:
Addressed by https://github.com/scikit-learn/scikit-learn/pull/22690.

Labels: Build / CI, module:test-suite, workflow, float32
GridSearchCV does not return trained estimator for each split vs cross_validate which does

### Describe the workflow you want to enable

GridSearchCV does not return trained estimator for each split vs cross_validate which does have trained estimators for each split. Instead GridSearchCV returns best_estimator_ which is usually being refit on X and thus can not be used on X to predict test data. cross_validate returns trained estimators which can be used for prediction on test data.

For me to effectively use result of GridseachCV for prediction on test data I need to retrain the estimators on my train/test splits using best params. This takes additional time and was done once already when GridsearchCV was fitted.

### Describe your proposed solution

Return trained estimators for each test split

### Describe alternatives you've considered, if relevant

rewrite GridSearchCV or retrain estimators again after GridsearchCV

### Additional context

_No response_

COMMENT:
`GridSearchCV` considers all of it's input `X` to be training data. It uses all the training data to find the best configuration with cross validation. Currently, the purpose of the models trained during cross validation is **only** for finding the best configuration. Once it finds the best configuration it refits it on all the training data to get `best_estimator_`. To have test data to evaluate `best_estimator_` with, one needs to split it off first before going through `GridSearchCV`. This is detailed in the [Cross-validation User guide](https://scikit-learn.org/stable/modules/cross_validation.html):

![Screen Shot 2022-03-03 at 12 12 47 PM](https://user-images.githubusercontent.com/5402633/156615870-b0244c46-2c16-432d-a3d7-9e7956828655.jpg)


To better understand your use case, lets say `cv=5`, so we end up with 5 estimators with the best configuration from grid search's internal cross validation procedure. Can you detail how you wish to compute predictions with those 5 estimators? i.e. What data are you going to use? How are you going to use those 5 models?

Labels: New Feature, module:model_selection
GridSearchCV does not return trained estimator for each split vs cross_validate which does

### Describe the workflow you want to enable

GridSearchCV does not return trained estimator for each split vs cross_validate which does have trained estimators for each split. Instead GridSearchCV returns best_estimator_ which is usually being refit on X and thus can not be used on X to predict test data. cross_validate returns trained estimators which can be used for prediction on test data.

For me to effectively use result of GridseachCV for prediction on test data I need to retrain the estimators on my train/test splits using best params. This takes additional time and was done once already when GridsearchCV was fitted.

### Describe your proposed solution

Return trained estimators for each test split

### Describe alternatives you've considered, if relevant

rewrite GridSearchCV or retrain estimators again after GridsearchCV

### Additional context

_No response_

COMMENT:
I could imagine an ensemble to make vote the 5 models instead to retrain on the full training set. Having access to these estimators could also be useful for inspection purposes. However, it should be an option to only store these models when requested.

Labels: New Feature, module:model_selection
GridSearchCV does not return trained estimator for each split vs cross_validate which does

### Describe the workflow you want to enable

GridSearchCV does not return trained estimator for each split vs cross_validate which does have trained estimators for each split. Instead GridSearchCV returns best_estimator_ which is usually being refit on X and thus can not be used on X to predict test data. cross_validate returns trained estimators which can be used for prediction on test data.

For me to effectively use result of GridseachCV for prediction on test data I need to retrain the estimators on my train/test splits using best params. This takes additional time and was done once already when GridsearchCV was fitted.

### Describe your proposed solution

Return trained estimators for each test split

### Describe alternatives you've considered, if relevant

rewrite GridSearchCV or retrain estimators again after GridsearchCV

### Additional context

_No response_

COMMENT:
Take, 
I will try to implement this feature by always keeping the current best models. I don't guarantee to produce qualified code though, I am mostly doing it for learning purpose.

Labels: New Feature, module:model_selection
GridSearchCV does not return trained estimator for each split vs cross_validate which does

### Describe the workflow you want to enable

GridSearchCV does not return trained estimator for each split vs cross_validate which does have trained estimators for each split. Instead GridSearchCV returns best_estimator_ which is usually being refit on X and thus can not be used on X to predict test data. cross_validate returns trained estimators which can be used for prediction on test data.

For me to effectively use result of GridseachCV for prediction on test data I need to retrain the estimators on my train/test splits using best params. This takes additional time and was done once already when GridsearchCV was fitted.

### Describe your proposed solution

Return trained estimators for each test split

### Describe alternatives you've considered, if relevant

rewrite GridSearchCV or retrain estimators again after GridsearchCV

### Additional context

_No response_

COMMENT:
@NartayXD  did you make some experiments checking performances between average / ensemble of predictions on test each N models for N folds in comparision with 1 model refit on entire data with best parameters? If yes, do you see it outperforms? Would be a good signature to see smth like that on couple of open datasets.

Labels: New Feature, module:model_selection
GridSearchCV does not return trained estimator for each split vs cross_validate which does

### Describe the workflow you want to enable

GridSearchCV does not return trained estimator for each split vs cross_validate which does have trained estimators for each split. Instead GridSearchCV returns best_estimator_ which is usually being refit on X and thus can not be used on X to predict test data. cross_validate returns trained estimators which can be used for prediction on test data.

For me to effectively use result of GridseachCV for prediction on test data I need to retrain the estimators on my train/test splits using best params. This takes additional time and was done once already when GridsearchCV was fitted.

### Describe your proposed solution

Return trained estimators for each test split

### Describe alternatives you've considered, if relevant

rewrite GridSearchCV or retrain estimators again after GridsearchCV

### Additional context

_No response_

COMMENT:
But the cross-validation used by GridsearchCV is only used to find the best hyperparameters or also to calculate the metrics of the model (such as the roc_auc) with cross-validation?
After GridsearchCV() do we need to use cross_validate() ? Or is it already included on it?

Labels: New Feature, module:model_selection
GridSearchCV does not return trained estimator for each split vs cross_validate which does

### Describe the workflow you want to enable

GridSearchCV does not return trained estimator for each split vs cross_validate which does have trained estimators for each split. Instead GridSearchCV returns best_estimator_ which is usually being refit on X and thus can not be used on X to predict test data. cross_validate returns trained estimators which can be used for prediction on test data.

For me to effectively use result of GridseachCV for prediction on test data I need to retrain the estimators on my train/test splits using best params. This takes additional time and was done once already when GridsearchCV was fitted.

### Describe your proposed solution

Return trained estimators for each test split

### Describe alternatives you've considered, if relevant

rewrite GridSearchCV or retrain estimators again after GridsearchCV

### Additional context

_No response_

COMMENT:
Nested cross validation is usually a good idea: https://scikit-learn.org/stable/auto_examples/model_selection/plot_nested_cross_validation_iris.html#sphx-glr-auto-examples-model-selection-plot-nested-cross-validation-iris-py

Labels: New Feature, module:model_selection
GridSearchCV does not return trained estimator for each split vs cross_validate which does

### Describe the workflow you want to enable

GridSearchCV does not return trained estimator for each split vs cross_validate which does have trained estimators for each split. Instead GridSearchCV returns best_estimator_ which is usually being refit on X and thus can not be used on X to predict test data. cross_validate returns trained estimators which can be used for prediction on test data.

For me to effectively use result of GridseachCV for prediction on test data I need to retrain the estimators on my train/test splits using best params. This takes additional time and was done once already when GridsearchCV was fitted.

### Describe your proposed solution

Return trained estimators for each test split

### Describe alternatives you've considered, if relevant

rewrite GridSearchCV or retrain estimators again after GridsearchCV

### Additional context

_No response_

COMMENT:
I would propose we close this issue. It is unclear what the original use-case was (no more details after Thomas' reply explaining how GridSearchCV works). WDYT @adrinjalali ?

Labels: New Feature, module:model_selection
GridSearchCV does not return trained estimator for each split vs cross_validate which does

### Describe the workflow you want to enable

GridSearchCV does not return trained estimator for each split vs cross_validate which does have trained estimators for each split. Instead GridSearchCV returns best_estimator_ which is usually being refit on X and thus can not be used on X to predict test data. cross_validate returns trained estimators which can be used for prediction on test data.

For me to effectively use result of GridseachCV for prediction on test data I need to retrain the estimators on my train/test splits using best params. This takes additional time and was done once already when GridsearchCV was fitted.

### Describe your proposed solution

Return trained estimators for each test split

### Describe alternatives you've considered, if relevant

rewrite GridSearchCV or retrain estimators again after GridsearchCV

### Additional context

_No response_

COMMENT:
I can imagine that this is sometimes interesting for inspection.

Labels: New Feature, module:model_selection
Add NaN handling in sklearn.preprocessing.KBinsDiscretizer

### Describe the workflow you want to enable

I Would like to enable `KBinsDiscretizer` to handle NaNs, in order to create bins representing missing values. This is a bit awkward for ordinal encoding, but makes a lot of sense for one hot encoding, where one can have a dummy variable for NaN values or simply set all dummis of a single feature to zero, meaning the feature value is missing.

### Describe your proposed solution

A few tweaks in `fit`, `transform` and `inverse_transform` are enough to ensure this behaviour, without changing the current behaviour of the transformer. The new estimator has two new parameters in the constructor:

`nan_handling`: {"handle","raise"}, defaults to "raise".
        defines the behaviour for hadling NaN. if "handle", the fit process will be performed excluding and the missing values of each column and the transform method will transform only not missing values and after that, a NaN bin is created (of index self.n_bins_ +1)

`create_nan_dummy`: bool
        Only applicable if encode == "onehot". Defines whether a new dummy column is created for NaNs or if the dummies of a missing value for a given feature are all set to zero


### Describe alternatives you've considered, if relevant

_No response_

### Additional context

I Have already implemented a solution for a project of mine. Just wanna know if this feature is intereseting for other people and if it makes sense to merge it into the main project.

COMMENT:
Thank you for opening this issue.

I am closing this issue because it is a duplicate of https://github.com/scikit-learn/scikit-learn/issues/9341. You are welcome to continue the discussion there.

Labels: New Feature, module:preprocessing, Needs Triage
SimpleImputer does not support pd.NA

### Describe the bug

Pandas' new NA type does not seem to be properly recognized as a none-type.  We currently see this when trying to use the SimpleImputer with a strategy of "most frequent".  Digging a little deeper, it seems to come to the _object_dtype_isnan() function in fixes.py being the culprit and returning a bool instead of a properly shaped mask of where the pd.NAs are.

```
def _object_dtype_isnan(X):
    return X != X
```

### Steps/Code to Reproduce

```
import pandas as pd
import numpy as np
from sklearn.impute import SimpleImputer

si = SimpleImputer(strategy="most_frequent")

x = np.array([[1,2,3,4], [2,3,4,pd.NA]])

si.fit(x)

res = sit.transform(x)
```

### Expected Results

No error is thrown and the pd.NA is reconized as a typical none-type.

### Actual Results

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/Caskroom/miniconda/base/envs/evalml_dev_39/lib/python3.9/site-packages/sklearn/impute/_base.py", line 358, in fit
    self.statistics_ = self._dense_fit(
  File "/usr/local/Caskroom/miniconda/base/envs/evalml_dev_39/lib/python3.9/site-packages/sklearn/impute/_base.py", line 434, in _dense_fit
    mask = missing_mask.transpose()
AttributeError: 'bool' object has no attribute 'transpose'

### Versions

```shell
>>> import sklearn; sklearn.show_versions()

System:
    python: 3.9.7 | packaged by conda-forge | (default, Sep 29 2021, 19:23:19)  [Clang 11.1.0 ]
executable: /usr/local/Caskroom/miniconda/base/envs/evalml_dev_39/bin/python
   machine: macOS-11.6.1-x86_64-i386-64bit

Python dependencies:
          pip: 22.0.2
   setuptools: 59.4.0
      sklearn: 1.0.1
        numpy: 1.21.4
        scipy: 1.7.3
       Cython: 0.29.17
       pandas: 1.4.0
   matplotlib: 3.5.0
       joblib: 1.1.0
threadpoolctl: 3.0.0

Built with OpenMP: True
```

COMMENT:
**Note this feature is on the `main` branch and will be released in 1.1** You can try it out by downloading the nightly build:

```bash
pip install --pre --extra-index https://pypi.anaconda.org/scipy-wheels-nightly/simple scikit-learn
```

To recognize `pd.NA`, set `missing_values` to `pd.NA`:

```python
import pandas as pd
import numpy as np
from sklearn.impute import SimpleImputer

x = np.array([[1,2,3,4],
              [2,3,4,pd.NA]])

si = SimpleImputer(strategy="most_frequent", missing_values=pd.NA)
si.fit(x)

print(si.transform(x))
# [[1 2 3 4]
#  [2 3 4 4]]
```

If you want numerical output, use a numerical pandas extension dtype (which is the use case for `pd.NA`):

```python
import pandas as pd
import numpy as np
from sklearn.impute import SimpleImputer

x = pd.DataFrame({"a": [1, 2, 2], "b": pd.Series([1, 1, pd.NA], dtype="Int64")})

si = SimpleImputer(strategy="most_frequent", missing_values=pd.NA)
si.fit(x)

print(si.transform(x))
# [[1. 1.]
# [2. 1.]
# [2. 1.]]
```

Labels: Bug, Needs Triage
multiple-output regression is inaccurate for many targets

### Describe the bug

I have tested multi output regression for num_target > 1000 , and have noticed that there is a substrantial difference in the predicted values if I do the regressions serially in a for loop, vs MultiOutputRegressor(reg) . I have tried varying the number of jobs -1, 1, 2, 10. the results only agrees when the njobs=1 otherwise there is a substantial different. 

### Steps/Code to Reproduce
```shell
reg=SkModel.LinearRegression(fit_intercept=fit_intersept)
regr=MultiOutputRegressor(reg,n_jobs=1).fit(X_train_split,Y_train_split)
Y_pred=regr.predict(X_test_split)

for col in tqdm(np.transpose(Y_train_split.values)):
            reg=self.lr.fit(X_train_split.values,col)
           all_cols.append(reg.coef_)
           Y_pred_col.append(self.lr.predict(X_test_split))
pred_2=np.stack(Y_pred_col)


diff=pred_2-Y_pred.transpose()
```
### Expected Results

diff should be all zero 

### Actual Results

diff is only 0 when njobs=1 

### Versions

```shell
System:
    python: 3.9.10 | packaged by conda-forge | (main, Feb  1 2022, 21:24:11)  [GCC 9.4.0]
executable: /om/user/ehoseini/miniconda3/envs/bplm_gpu/bin/python
   machine: Linux-3.10.0-1062.el7.x86_64-x86_64-with-glibc2.17
Python dependencies:
          pip: 22.0.3
   setuptools: 59.8.0
      sklearn: 1.0.2
        numpy: 1.21.0
        scipy: 1.8.0
       Cython: None
       pandas: 1.3.5
   matplotlib: 3.5.1
       joblib: 1.1.0
threadpoolctl: 3.1.0
Built with OpenMP: True
```

COMMENT:
Thank you for opening the issue. For us to help with the issue, please provide a [minimal reproducer](https://scikit-learn.org/dev/developers/minimal_reproducer.html).

Labels: Bug, Needs Reproducible Code, module:multioutput
multiple-output regression is inaccurate for many targets

### Describe the bug

I have tested multi output regression for num_target > 1000 , and have noticed that there is a substrantial difference in the predicted values if I do the regressions serially in a for loop, vs MultiOutputRegressor(reg) . I have tried varying the number of jobs -1, 1, 2, 10. the results only agrees when the njobs=1 otherwise there is a substantial different. 

### Steps/Code to Reproduce
```shell
reg=SkModel.LinearRegression(fit_intercept=fit_intersept)
regr=MultiOutputRegressor(reg,n_jobs=1).fit(X_train_split,Y_train_split)
Y_pred=regr.predict(X_test_split)

for col in tqdm(np.transpose(Y_train_split.values)):
            reg=self.lr.fit(X_train_split.values,col)
           all_cols.append(reg.coef_)
           Y_pred_col.append(self.lr.predict(X_test_split))
pred_2=np.stack(Y_pred_col)


diff=pred_2-Y_pred.transpose()
```
### Expected Results

diff should be all zero 

### Actual Results

diff is only 0 when njobs=1 

### Versions

```shell
System:
    python: 3.9.10 | packaged by conda-forge | (main, Feb  1 2022, 21:24:11)  [GCC 9.4.0]
executable: /om/user/ehoseini/miniconda3/envs/bplm_gpu/bin/python
   machine: Linux-3.10.0-1062.el7.x86_64-x86_64-with-glibc2.17
Python dependencies:
          pip: 22.0.3
   setuptools: 59.8.0
      sklearn: 1.0.2
        numpy: 1.21.0
        scipy: 1.8.0
       Cython: None
       pandas: 1.3.5
   matplotlib: 3.5.1
       joblib: 1.1.0
threadpoolctl: 3.1.0
Built with OpenMP: True
```

COMMENT:
Closing as no reproducer provided. Happy to reopen if one is provided.

Labels: Bug, Needs Reproducible Code, module:multioutput