Talip7/scikit-learn-issues-multilabel huggingface dataset:
passing cross validation data to custom score function in GridSearchCV

### Describe the workflow you want to enable

Custom score function some time need the feature values also but there is no way we could do this .We can not the use same dataset which we have used for gridsearchcv ,reason is the data we sent will be splited into training and testing data and passed to score function separately. 

### Describe your proposed solution

Just while passing y_true and y_pred , pass validation dataset also .

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
![image](https://user-images.githubusercontent.com/74662593/173105891-da2b2747-da09-4e57-b593-54efc3b3191c.png)

This is how cross-validation works right . At each iteration one fold becomes test dataset and remaining becomes training dataset . That means that training dataset for iteration of cross-validation changes. 

Example :
Lets say we have 3 features : 'ID' , 'feat1' , 'feat2' and a target variable called badge
And this is the sample dataset 

<google-sheets-html-origin>

ID | count | instant | badge
-- | -- | -- | --
1 | 0 | 3 | 1
1 | 1 | 4 | 1
1 | 2 | 5 | 1
2 | 1 | 1 | 0
3 | 0 | 1 | 0
3 | 1 | 3 | 1
3 | 2 | 5 | 1

**Explaination of the sample data :**
**badge is labeled for id's .**
That means we are having data at multiple instances of ids .
For id=1 ,at instant 3 (means at time=3 (lets say epochs)) ,count is 0 . Similarly at instant 4 ,count increased to 1 .

Lets say we are you XGBClassifier for training the model and now we have the following information .

<google-sheets-html-origin>

ID | count | instance | badge | y_pred
-- | -- | -- | -- | --
1 | 0 | 3 | 1 | 0
1 | 1 | 4 | 1 | 1
1 | 1 | 5 | 1 | 1
2 | 1 | 1 | 0 | 0
3 | 0 | 1 | 0 | 0
3 | 1 | 3 | 1 | 0
3 | 2 | 5 | 1 | 1

For some reason , I want to calculate rms error by grouping ID and count columns .
So to achieve this I have written a custom score function should take X as input right ? how can we do that ?  I thought of declaring X globally and using it in the function , it's okay if we don't want to use cross-validation . But the thing is since it is a XGBClassifier , we need to do hyper parameter tuning and we need to use gridsearchcv or randomsearchcv .Since it is a cross-validation, data using for training dataset changes at iteration (as discussed at the starting) so we cannot declare globally . We need to pass it to the function as a paramter.

Sorry for wasting your time if this feature is already available, If it is there please let me know .

Labels: New Feature, module:model_selection, Needs Reproducible Code, Metadata Routing
passing cross validation data to custom score function in GridSearchCV

### Describe the workflow you want to enable

Custom score function some time need the feature values also but there is no way we could do this .We can not the use same dataset which we have used for gridsearchcv ,reason is the data we sent will be splited into training and testing data and passed to score function separately. 

### Describe your proposed solution

Just while passing y_true and y_pred , pass validation dataset also .

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
What you describe seems to be linked to group cross-validation. We have an extension for almost all cross-validation strategies where you can group some data points together. For instance, https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GroupKFold.html is a modified version of the `KFold` that you show earlier.

However, nested cross-validation will not work because the group is not passed to the cross-validation of the inner CV of the CV search. This is where the proposal of sample-props comes into place: https://scikit-learn-enhancement-proposals.readthedocs.io/en/latest/slep006/proposal.html#nested-grouped-cross-validation-with-searchcv. We are currently implementing this feature.

I would think that if you want to have a more flexible way to compute the metric, it might actually be difficult to do that in scikit-learn such that it works with all the components (pipeline, search cv, cross-validation, etc.)

Labels: New Feature, module:model_selection, Needs Reproducible Code, Metadata Routing
passing cross validation data to custom score function in GridSearchCV

### Describe the workflow you want to enable

Custom score function some time need the feature values also but there is no way we could do this .We can not the use same dataset which we have used for gridsearchcv ,reason is the data we sent will be splited into training and testing data and passed to score function separately. 

### Describe your proposed solution

Just while passing y_true and y_pred , pass validation dataset also .

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
From the description here it's not clear what exactly you need. Please provide a reproducer, which can be copy/pasted in its entirety to show what the issue is, and what the desired behavior should be. Closing for now till reproducer is provided. I'm not sure if the issue is only that the groups are not passed around as @glemaitre suggests.

Labels: New Feature, module:model_selection, Needs Reproducible Code, Metadata Routing
Tree Based model max_features when setting float values is not from rounded number.

### Describe the bug

In the documentation, for the tree based model such as ExtraTrees, it says for the float max_features: 

- If float, then max_features is a fraction and round(max_features * n_features) features are considered at each split.

but it looks its not  a rounded number of max_features * n_features but int conversion of whatever that float is. 


### Steps/Code to Reproduce

for the same dataset, this two same model produce different results:

```
total_n_features = 10.
rounded of 10*0.66 = 7

from sklearn.datasets import load_diabetes
from sklearn.model_selection import train_test_split
from sklearn.ensemble import ExtraTreesRegressor
X, y = load_diabetes(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, random_state=0)

model1
reg1 = ExtraTreesRegressor(n_estimators=100, random_state=0, max_features=0.66)
reg1.fit(X_train, y_train)
print(reg1.score(X_test, y_test))

model2
reg2 = ExtraTreesRegressor(n_estimators=100, random_state=0, max_features=7)
reg2.fit(X_train, y_train)
print(reg2.score(X_test, y_test))
```

### Expected Results

Those above two should produce same results. 

### Actual Results

those two produce different results. 

Looking at the code, the problem seems to be int conversion even though the document says that the float is rounded number. 

The code that I found in the conversion was in the _classes.py in line number 342.

### Versions

```shell
1.1.1
```

COMMENT:
For now, I think it's best to update the documentation to be consistent with the implementation. The alternative is to change the implementation which is backward incompatible and changes the behavior for all tree based models.

@pradipneupane Are you interested in submitting a PR to update the docs?

Labels: Documentation, module:ensemble, module:tree
Tree Based model max_features when setting float values is not from rounded number.

### Describe the bug

In the documentation, for the tree based model such as ExtraTrees, it says for the float max_features: 

- If float, then max_features is a fraction and round(max_features * n_features) features are considered at each split.

but it looks its not  a rounded number of max_features * n_features but int conversion of whatever that float is. 


### Steps/Code to Reproduce

for the same dataset, this two same model produce different results:

```
total_n_features = 10.
rounded of 10*0.66 = 7

from sklearn.datasets import load_diabetes
from sklearn.model_selection import train_test_split
from sklearn.ensemble import ExtraTreesRegressor
X, y = load_diabetes(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, random_state=0)

model1
reg1 = ExtraTreesRegressor(n_estimators=100, random_state=0, max_features=0.66)
reg1.fit(X_train, y_train)
print(reg1.score(X_test, y_test))

model2
reg2 = ExtraTreesRegressor(n_estimators=100, random_state=0, max_features=7)
reg2.fit(X_train, y_train)
print(reg2.score(X_test, y_test))
```

### Expected Results

Those above two should produce same results. 

### Actual Results

those two produce different results. 

Looking at the code, the problem seems to be int conversion even though the document says that the float is rounded number. 

The code that I found in the conversion was in the _classes.py in line number 342.

### Versions

```shell
1.1.1
```

COMMENT:
I can create the PR to update the docs. It would be my first time contributing here. Do I need to update the doc-string in every place that uses this parameter or how is the process goes ? @thomasjpfan

Labels: Documentation, module:ensemble, module:tree
Tree Based model max_features when setting float values is not from rounded number.

### Describe the bug

In the documentation, for the tree based model such as ExtraTrees, it says for the float max_features: 

- If float, then max_features is a fraction and round(max_features * n_features) features are considered at each split.

but it looks its not  a rounded number of max_features * n_features but int conversion of whatever that float is. 


### Steps/Code to Reproduce

for the same dataset, this two same model produce different results:

```
total_n_features = 10.
rounded of 10*0.66 = 7

from sklearn.datasets import load_diabetes
from sklearn.model_selection import train_test_split
from sklearn.ensemble import ExtraTreesRegressor
X, y = load_diabetes(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, random_state=0)

model1
reg1 = ExtraTreesRegressor(n_estimators=100, random_state=0, max_features=0.66)
reg1.fit(X_train, y_train)
print(reg1.score(X_test, y_test))

model2
reg2 = ExtraTreesRegressor(n_estimators=100, random_state=0, max_features=7)
reg2.fit(X_train, y_train)
print(reg2.score(X_test, y_test))
```

### Expected Results

Those above two should produce same results. 

### Actual Results

those two produce different results. 

Looking at the code, the problem seems to be int conversion even though the document says that the float is rounded number. 

The code that I found in the conversion was in the _classes.py in line number 342.

### Versions

```shell
1.1.1
```

COMMENT:
> Do I need to update the doc-string in every place that uses this parameter or how is the process goes ?

I am okay with updating the docstring for `max_features` in every tree-based models in one PR.

Labels: Documentation, module:ensemble, module:tree
SelectFromModel threshold for ElasticNet is wrong

### Describe the bug

As an estimator with an L1 penalty, ElasticNet should default to 1e-5 as the selection threshold. SelectFromModel code does not account for this, since ElasticNet has no penalty parameter, and is not called *LASSO*.

https://github.com/scikit-learn/scikit-learn/blob/80598905e517759b4696c74ecc35c6e2eb508cff/sklearn/feature_selection/_from_model.py#L22-L31

### Steps/Code to Reproduce

from sklearn.feature_selection import SelectFromModel
from sklearn.linear_model import ElasticNet
from sklearn.datasets import make_regression

en = ElasticNet()
fs = SelectFromModel(estimator=en)

X, y = make_regression(n_samples=100, n_features=10, n_informative=3, random_state=42)

fs_fit = fs.fit(X=X, y=y)

print(fs_fit.threshold_)

### Expected Results

1e-5

### Actual Results

6.53140663228231

### Versions

```shell
System:
    python: 3.9.10 | packaged by conda-forge | (main, Feb  1 2022, 21:24:11)  [GCC 9.4.0]
executable: /opt/conda/bin/python
   machine: XXXXXXXX

Python dependencies:
      sklearn: 1.1.1
          pip: 22.0.3
   setuptools: 60.9.3
        numpy: 1.21.6
        scipy: 1.8.0
       Cython: 0.29.28
       pandas: 1.4.2
   matplotlib: 3.5.1
       joblib: 1.1.0
threadpoolctl: 3.1.0

Built with OpenMP: True

threadpoolctl info:
       user_api: openmp
   internal_api: openmp
         prefix: libgomp
       filepath: /opt/conda/lib/python3.9/site-packages/scikit_learn.libs/libgomp-a34b3233.so.1.0.0
        version: None
    num_threads: 16

       user_api: blas
   internal_api: openblas
         prefix: libopenblas
       filepath: /opt/conda/lib/libopenblasp-r0.3.20.so
        version: 0.3.20
threading_layer: pthreads
   architecture: XXXXXX
    num_threads: XXXXXX
```

COMMENT:
`ElasticNet` is a mix of L1 and L2. I think setting the threshold to `1e-5` only makes sense when `l1_ratio == 1`.

I am usually not happy with adding these special cases into an meta-estimator. Since there is already one for `Lasso`, I am okay with adding one here.

**Edit**: I meant `l1_ratio == 1`.

Labels: Bug, module:feature_extraction
SelectFromModel threshold for ElasticNet is wrong

### Describe the bug

As an estimator with an L1 penalty, ElasticNet should default to 1e-5 as the selection threshold. SelectFromModel code does not account for this, since ElasticNet has no penalty parameter, and is not called *LASSO*.

https://github.com/scikit-learn/scikit-learn/blob/80598905e517759b4696c74ecc35c6e2eb508cff/sklearn/feature_selection/_from_model.py#L22-L31

### Steps/Code to Reproduce

from sklearn.feature_selection import SelectFromModel
from sklearn.linear_model import ElasticNet
from sklearn.datasets import make_regression

en = ElasticNet()
fs = SelectFromModel(estimator=en)

X, y = make_regression(n_samples=100, n_features=10, n_informative=3, random_state=42)

fs_fit = fs.fit(X=X, y=y)

print(fs_fit.threshold_)

### Expected Results

1e-5

### Actual Results

6.53140663228231

### Versions

```shell
System:
    python: 3.9.10 | packaged by conda-forge | (main, Feb  1 2022, 21:24:11)  [GCC 9.4.0]
executable: /opt/conda/bin/python
   machine: XXXXXXXX

Python dependencies:
      sklearn: 1.1.1
          pip: 22.0.3
   setuptools: 60.9.3
        numpy: 1.21.6
        scipy: 1.8.0
       Cython: 0.29.28
       pandas: 1.4.2
   matplotlib: 3.5.1
       joblib: 1.1.0
threadpoolctl: 3.1.0

Built with OpenMP: True

threadpoolctl info:
       user_api: openmp
   internal_api: openmp
         prefix: libgomp
       filepath: /opt/conda/lib/python3.9/site-packages/scikit_learn.libs/libgomp-a34b3233.so.1.0.0
        version: None
    num_threads: 16

       user_api: blas
   internal_api: openblas
         prefix: libopenblas
       filepath: /opt/conda/lib/libopenblasp-r0.3.20.so
        version: 0.3.20
threading_layer: pthreads
   architecture: XXXXXX
    num_threads: XXXXXX
```

COMMENT:
Do you mean when l1_ratio != 0? Which I believe is the majority of elastic net cases, especially when the scikit ElasticNet module doesn't support l1_ratio = 0 very well. (**From the doc**: Currently, l1_ratio <= 0.01 is not reliable, unless you supply your own sequence of alpha.)

Labels: Bug, module:feature_extraction
SelectFromModel threshold for ElasticNet is wrong

### Describe the bug

As an estimator with an L1 penalty, ElasticNet should default to 1e-5 as the selection threshold. SelectFromModel code does not account for this, since ElasticNet has no penalty parameter, and is not called *LASSO*.

https://github.com/scikit-learn/scikit-learn/blob/80598905e517759b4696c74ecc35c6e2eb508cff/sklearn/feature_selection/_from_model.py#L22-L31

### Steps/Code to Reproduce

from sklearn.feature_selection import SelectFromModel
from sklearn.linear_model import ElasticNet
from sklearn.datasets import make_regression

en = ElasticNet()
fs = SelectFromModel(estimator=en)

X, y = make_regression(n_samples=100, n_features=10, n_informative=3, random_state=42)

fs_fit = fs.fit(X=X, y=y)

print(fs_fit.threshold_)

### Expected Results

1e-5

### Actual Results

6.53140663228231

### Versions

```shell
System:
    python: 3.9.10 | packaged by conda-forge | (main, Feb  1 2022, 21:24:11)  [GCC 9.4.0]
executable: /opt/conda/bin/python
   machine: XXXXXXXX

Python dependencies:
      sklearn: 1.1.1
          pip: 22.0.3
   setuptools: 60.9.3
        numpy: 1.21.6
        scipy: 1.8.0
       Cython: 0.29.28
       pandas: 1.4.2
   matplotlib: 3.5.1
       joblib: 1.1.0
threadpoolctl: 3.1.0

Built with OpenMP: True

threadpoolctl info:
       user_api: openmp
   internal_api: openmp
         prefix: libgomp
       filepath: /opt/conda/lib/python3.9/site-packages/scikit_learn.libs/libgomp-a34b3233.so.1.0.0
        version: None
    num_threads: 16

       user_api: blas
   internal_api: openblas
         prefix: libopenblas
       filepath: /opt/conda/lib/libopenblasp-r0.3.20.so
        version: 0.3.20
threading_layer: pthreads
   architecture: XXXXXX
    num_threads: XXXXXX
```

COMMENT:
Yea, I meant `l1_ratio == 1`.

Labels: Bug, module:feature_extraction
SelectFromModel threshold for ElasticNet is wrong

### Describe the bug

As an estimator with an L1 penalty, ElasticNet should default to 1e-5 as the selection threshold. SelectFromModel code does not account for this, since ElasticNet has no penalty parameter, and is not called *LASSO*.

https://github.com/scikit-learn/scikit-learn/blob/80598905e517759b4696c74ecc35c6e2eb508cff/sklearn/feature_selection/_from_model.py#L22-L31

### Steps/Code to Reproduce

from sklearn.feature_selection import SelectFromModel
from sklearn.linear_model import ElasticNet
from sklearn.datasets import make_regression

en = ElasticNet()
fs = SelectFromModel(estimator=en)

X, y = make_regression(n_samples=100, n_features=10, n_informative=3, random_state=42)

fs_fit = fs.fit(X=X, y=y)

print(fs_fit.threshold_)

### Expected Results

1e-5

### Actual Results

6.53140663228231

### Versions

```shell
System:
    python: 3.9.10 | packaged by conda-forge | (main, Feb  1 2022, 21:24:11)  [GCC 9.4.0]
executable: /opt/conda/bin/python
   machine: XXXXXXXX

Python dependencies:
      sklearn: 1.1.1
          pip: 22.0.3
   setuptools: 60.9.3
        numpy: 1.21.6
        scipy: 1.8.0
       Cython: 0.29.28
       pandas: 1.4.2
   matplotlib: 3.5.1
       joblib: 1.1.0
threadpoolctl: 3.1.0

Built with OpenMP: True

threadpoolctl info:
       user_api: openmp
   internal_api: openmp
         prefix: libgomp
       filepath: /opt/conda/lib/python3.9/site-packages/scikit_learn.libs/libgomp-a34b3233.so.1.0.0
        version: None
    num_threads: 16

       user_api: blas
   internal_api: openblas
         prefix: libopenblas
       filepath: /opt/conda/lib/libopenblasp-r0.3.20.so
        version: 0.3.20
threading_layer: pthreads
   architecture: XXXXXX
    num_threads: XXXXXX
```

COMMENT:
Hi, in the hope to facilitate the discussion, I opened a PR to add the special cases for `ElasticNet`.
By the way, I am curious where can I find the documentation says:

> Currently, l1_ratio <= 0.01 is not reliable, unless you supply your own sequence of alpha

Thanks!

Labels: Bug, module:feature_extraction
SelectFromModel threshold for ElasticNet is wrong

### Describe the bug

As an estimator with an L1 penalty, ElasticNet should default to 1e-5 as the selection threshold. SelectFromModel code does not account for this, since ElasticNet has no penalty parameter, and is not called *LASSO*.

https://github.com/scikit-learn/scikit-learn/blob/80598905e517759b4696c74ecc35c6e2eb508cff/sklearn/feature_selection/_from_model.py#L22-L31

### Steps/Code to Reproduce

from sklearn.feature_selection import SelectFromModel
from sklearn.linear_model import ElasticNet
from sklearn.datasets import make_regression

en = ElasticNet()
fs = SelectFromModel(estimator=en)

X, y = make_regression(n_samples=100, n_features=10, n_informative=3, random_state=42)

fs_fit = fs.fit(X=X, y=y)

print(fs_fit.threshold_)

### Expected Results

1e-5

### Actual Results

6.53140663228231

### Versions

```shell
System:
    python: 3.9.10 | packaged by conda-forge | (main, Feb  1 2022, 21:24:11)  [GCC 9.4.0]
executable: /opt/conda/bin/python
   machine: XXXXXXXX

Python dependencies:
      sklearn: 1.1.1
          pip: 22.0.3
   setuptools: 60.9.3
        numpy: 1.21.6
        scipy: 1.8.0
       Cython: 0.29.28
       pandas: 1.4.2
   matplotlib: 3.5.1
       joblib: 1.1.0
threadpoolctl: 3.1.0

Built with OpenMP: True

threadpoolctl info:
       user_api: openmp
   internal_api: openmp
         prefix: libgomp
       filepath: /opt/conda/lib/python3.9/site-packages/scikit_learn.libs/libgomp-a34b3233.so.1.0.0
        version: None
    num_threads: 16

       user_api: blas
   internal_api: openblas
         prefix: libopenblas
       filepath: /opt/conda/lib/libopenblasp-r0.3.20.so
        version: 0.3.20
threading_layer: pthreads
   architecture: XXXXXX
    num_threads: XXXXXX
```

COMMENT:
@haochunchang it is here

https://github.com/scikit-learn/scikit-learn/blob/80598905e517759b4696c74ecc35c6e2eb508cff/sklearn/linear_model/_coordinate_descent.py#L698-L701

Labels: Bug, module:feature_extraction
⚠️ CI failed on Linux_Nightly.pylatest_pip_scipy_dev ⚠️

**CI is still failing on [Linux_Nightly.pylatest_pip_scipy_dev](https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=42983&view=logs&j=dfe99b15-50db-5d7b-b1e9-4105c42527cf)**
- test_missing_value_handling[est0-maxabs_scale-True-False-omit_kwargs0]
- test_missing_value_handling[est3-scale-True-False-omit_kwargs3]
- test_missing_value_handling[est6-quantile_transform-True-False-omit_kwargs6]
- test_missing_value_handling[est8-robust_scale-True-False-omit_kwargs8]

COMMENT:
Apparently, we don't have the equivalence dense-sparse anymore.

Labels: Needs Triage
⚠️ CI failed on Linux_Nightly.pylatest_pip_scipy_dev ⚠️

**CI is still failing on [Linux_Nightly.pylatest_pip_scipy_dev](https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=42983&view=logs&j=dfe99b15-50db-5d7b-b1e9-4105c42527cf)**
- test_missing_value_handling[est0-maxabs_scale-True-False-omit_kwargs0]
- test_missing_value_handling[est3-scale-True-False-omit_kwargs3]
- test_missing_value_handling[est6-quantile_transform-True-False-omit_kwargs6]
- test_missing_value_handling[est8-robust_scale-True-False-omit_kwargs8]

COMMENT:
I deleted the redundant comments. They should no longer appear now that #23539 was merged: the description was edited by the bot 4 hours ago, so it seems to work as intended.

Labels: Needs Triage
⚠️ CI failed on Linux_Nightly.pylatest_pip_scipy_dev ⚠️

**CI is still failing on [Linux_Nightly.pylatest_pip_scipy_dev](https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=42983&view=logs&j=dfe99b15-50db-5d7b-b1e9-4105c42527cf)**
- test_missing_value_handling[est0-maxabs_scale-True-False-omit_kwargs0]
- test_missing_value_handling[est3-scale-True-False-omit_kwargs3]
- test_missing_value_handling[est6-quantile_transform-True-False-omit_kwargs6]
- test_missing_value_handling[est8-robust_scale-True-False-omit_kwargs8]

COMMENT:
All failures appear with the DIAgonal sparse matrix format. The same assertions with CSR / CSC do seem to pass.

I will try to craft a minimal reproducer and report upstream.

Labels: Needs Triage