Talip7/scikit-learn-issues-multilabel huggingface dataset:
CV integration for OOB-scoring

### Describe the workflow you want to enable

Out-of-Bag (OOB) scoring provides an estimate of the model generalizability for `RandomForest` without needing to refit the model several times as is demanded by k-fold cross validation (CV). Although `sklearn` provides a mechanism to obtain this estimate, it does *not* provide a mechanism to integrate it into the existing cross validation workflows. For example, we might have a `GridSearchCV` where we want to optimise hyperparameters for the forest, but only fitting *once* per parameter set. This in theory could be implemented using OOB error. 

As far as I can, the two parameters of interest here are `cv` and `scoring`, both inputs to all the CV-related classes, and ultimately to `cross_val_score()`. `scoring` can be implemented easily enough using a [custom scorer](https://scikit-learn.org/stable/modules/model_evaluation.html#implementing-your-own-scoring-object), since this has access to the final estimator and therefore the OOB error. What is problematic here is the `cv` argument, which *requires* that we split the dataset, and offers no alternative.

### Describe your proposed solution

* We add `sklearn.metrics.oob`, a scoring function that just returns the oob error on the trained classifier
* We add `sklearn.model_selection.IntegratedCV`, which is a cross validator that **does not split the data at all**. ie `IntegratedCV().split(X)` will return `X` unchanged

With the combination of these two entities, users will be able to perform OOB-based cross-validation

### Describe alternatives you've considered, if relevant

It is possible to apply general cross validation metrics to a `RandomForest`, such as k-folds. This is an alternative that already exists in sklearn today. However we are neglecting the significant (`k` times) speedup that could be obtained using OOB error.

### Additional context

This question is notably discussed in these threads:
* https://datascience.stackexchange.com/a/66238/83633
* https://datascience.stackexchange.com/questions/37393/scikitlearn-grid-search-random-forest-using-oob-as-metric
* https://datascience.stackexchange.com/questions/76304/gridsearchcv-with-random-forest-classifier

COMMENT:
A quick example of doing this without adding a new splitter or function is:

```python
from sklearn.model_selection import GridSearchCV
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import make_classification
import numpy as np

X, y = make_classification(random_state=42)
cv = GridSearchCV(
    RandomForestClassifier(oob_score=True, random_state=0),
    {"max_features": ["sqrt", "log2"]},
    cv=[(np.arange(X.shape[0]), np.empty(0, dtype=int))],
    scoring=lambda est, X, y: est.oob_score_,
)
results = cv.fit(X, y)
```

There is a maintenance cost to adding a new scorer and a new splitter that should only be used with estimators that have a `oob_score_` . If using the oob score is a good practice, then I prefer adding an example like the one above to show users how to do it.

Labels: New Feature, module:ensemble, Needs Decision - Include Feature
CV integration for OOB-scoring

### Describe the workflow you want to enable

Out-of-Bag (OOB) scoring provides an estimate of the model generalizability for `RandomForest` without needing to refit the model several times as is demanded by k-fold cross validation (CV). Although `sklearn` provides a mechanism to obtain this estimate, it does *not* provide a mechanism to integrate it into the existing cross validation workflows. For example, we might have a `GridSearchCV` where we want to optimise hyperparameters for the forest, but only fitting *once* per parameter set. This in theory could be implemented using OOB error. 

As far as I can, the two parameters of interest here are `cv` and `scoring`, both inputs to all the CV-related classes, and ultimately to `cross_val_score()`. `scoring` can be implemented easily enough using a [custom scorer](https://scikit-learn.org/stable/modules/model_evaluation.html#implementing-your-own-scoring-object), since this has access to the final estimator and therefore the OOB error. What is problematic here is the `cv` argument, which *requires* that we split the dataset, and offers no alternative.

### Describe your proposed solution

* We add `sklearn.metrics.oob`, a scoring function that just returns the oob error on the trained classifier
* We add `sklearn.model_selection.IntegratedCV`, which is a cross validator that **does not split the data at all**. ie `IntegratedCV().split(X)` will return `X` unchanged

With the combination of these two entities, users will be able to perform OOB-based cross-validation

### Describe alternatives you've considered, if relevant

It is possible to apply general cross validation metrics to a `RandomForest`, such as k-folds. This is an alternative that already exists in sklearn today. However we are neglecting the significant (`k` times) speedup that could be obtained using OOB error.

### Additional context

This question is notably discussed in these threads:
* https://datascience.stackexchange.com/a/66238/83633
* https://datascience.stackexchange.com/questions/37393/scikitlearn-grid-search-random-forest-using-oob-as-metric
* https://datascience.stackexchange.com/questions/76304/gridsearchcv-with-random-forest-classifier

COMMENT:
Okay then it seems the consensus is that this should just be an example. I will continue to work on the example proposed in #23391. I might keep it as a separate example unless there's a compelling reason to edit the existing one.

Both of you have mentioned that you want to compare the OOB scores from OOB versus cross validation so I'll focus on that. I think actually the OOB is pessimistic, but I'm not sure that really matters because you wouldn't use OOB for the final test set evaluation, only for model selection.

Labels: New Feature, module:ensemble, Needs Decision - Include Feature
CV integration for OOB-scoring

### Describe the workflow you want to enable

Out-of-Bag (OOB) scoring provides an estimate of the model generalizability for `RandomForest` without needing to refit the model several times as is demanded by k-fold cross validation (CV). Although `sklearn` provides a mechanism to obtain this estimate, it does *not* provide a mechanism to integrate it into the existing cross validation workflows. For example, we might have a `GridSearchCV` where we want to optimise hyperparameters for the forest, but only fitting *once* per parameter set. This in theory could be implemented using OOB error. 

As far as I can, the two parameters of interest here are `cv` and `scoring`, both inputs to all the CV-related classes, and ultimately to `cross_val_score()`. `scoring` can be implemented easily enough using a [custom scorer](https://scikit-learn.org/stable/modules/model_evaluation.html#implementing-your-own-scoring-object), since this has access to the final estimator and therefore the OOB error. What is problematic here is the `cv` argument, which *requires* that we split the dataset, and offers no alternative.

### Describe your proposed solution

* We add `sklearn.metrics.oob`, a scoring function that just returns the oob error on the trained classifier
* We add `sklearn.model_selection.IntegratedCV`, which is a cross validator that **does not split the data at all**. ie `IntegratedCV().split(X)` will return `X` unchanged

With the combination of these two entities, users will be able to perform OOB-based cross-validation

### Describe alternatives you've considered, if relevant

It is possible to apply general cross validation metrics to a `RandomForest`, such as k-folds. This is an alternative that already exists in sklearn today. However we are neglecting the significant (`k` times) speedup that could be obtained using OOB error.

### Additional context

This question is notably discussed in these threads:
* https://datascience.stackexchange.com/a/66238/83633
* https://datascience.stackexchange.com/questions/37393/scikitlearn-grid-search-random-forest-using-oob-as-metric
* https://datascience.stackexchange.com/questions/76304/gridsearchcv-with-random-forest-classifier

COMMENT:
> I'm not sure that really matters because you wouldn't use OOB for the final test set evaluation, only for model selection

That's why I think this is interesting to show the difference and mention it as a potential gotcha.

Labels: New Feature, module:ensemble, Needs Decision - Include Feature
CV integration for OOB-scoring

### Describe the workflow you want to enable

Out-of-Bag (OOB) scoring provides an estimate of the model generalizability for `RandomForest` without needing to refit the model several times as is demanded by k-fold cross validation (CV). Although `sklearn` provides a mechanism to obtain this estimate, it does *not* provide a mechanism to integrate it into the existing cross validation workflows. For example, we might have a `GridSearchCV` where we want to optimise hyperparameters for the forest, but only fitting *once* per parameter set. This in theory could be implemented using OOB error. 

As far as I can, the two parameters of interest here are `cv` and `scoring`, both inputs to all the CV-related classes, and ultimately to `cross_val_score()`. `scoring` can be implemented easily enough using a [custom scorer](https://scikit-learn.org/stable/modules/model_evaluation.html#implementing-your-own-scoring-object), since this has access to the final estimator and therefore the OOB error. What is problematic here is the `cv` argument, which *requires* that we split the dataset, and offers no alternative.

### Describe your proposed solution

* We add `sklearn.metrics.oob`, a scoring function that just returns the oob error on the trained classifier
* We add `sklearn.model_selection.IntegratedCV`, which is a cross validator that **does not split the data at all**. ie `IntegratedCV().split(X)` will return `X` unchanged

With the combination of these two entities, users will be able to perform OOB-based cross-validation

### Describe alternatives you've considered, if relevant

It is possible to apply general cross validation metrics to a `RandomForest`, such as k-folds. This is an alternative that already exists in sklearn today. However we are neglecting the significant (`k` times) speedup that could be obtained using OOB error.

### Additional context

This question is notably discussed in these threads:
* https://datascience.stackexchange.com/a/66238/83633
* https://datascience.stackexchange.com/questions/37393/scikitlearn-grid-search-random-forest-using-oob-as-metric
* https://datascience.stackexchange.com/questions/76304/gridsearchcv-with-random-forest-classifier

COMMENT:
Great discussion! And really helpful code of @thomasjpfan 

My typical workflow with random forests is quite as you describe:

1. Tune 1-2 hyper-parameters (mtry, max_depth) of the random forest via OOB performance.
2. Evaluate the performance of the final random forest on a separate evaluation data.
3. (Optional:) Compare the evaluation performance with with OOB performance of the selected model to get a feeling about reliability of the OOB tuning.

Labels: New Feature, module:ensemble, Needs Decision - Include Feature
fetch_openml difference between pandas and liac-arff parser

Seen in a [scipy-dev build](https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=42132&view=logs&j=dfe99b15-50db-5d7b-b1e9-4105c42527cf&t=a5a438e1-a911-5517-158f-26a140e5cbbf).

There are additional quotes in the pandas parser case.

cc @glemaitre

```py
import numpy as np
from sklearn.datasets import fetch_openml

mice_pandas = fetch_openml(name='miceprotein', version=4, parser="pandas")
mice_liac_arff = fetch_openml(name='miceprotein', version=4, parser="liac-arff")

assert np.testing.assert_array_equal(mice_pandas.target, mice_liac_arff.target)
```

Output:
```
AssertionError: 
Arrays are not equal

Mismatched elements: 1080 / 1080 (100%)
 x: array(["'c-CS-m'", "'c-CS-m'", "'c-CS-m'", ..., "'t-SC-s'", "'t-SC-s'",
       "'t-SC-s'"], dtype=object)
 y: array(['c-CS-m', 'c-CS-m', 'c-CS-m', ..., 't-SC-s', 't-SC-s', 't-SC-s'],
      dtype=object)
```

COMMENT:
Good catch. I assume that we need to set one of the following parameters in the `read_csv` call: `quotechar`, `quoting`, and `doublequote`.

I need to understand a bit what those are doing and we would need to check that this is working as well for different versions of the dataset.

Labels: Bug, module:datasets
fetch_openml difference between pandas and liac-arff parser

Seen in a [scipy-dev build](https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=42132&view=logs&j=dfe99b15-50db-5d7b-b1e9-4105c42527cf&t=a5a438e1-a911-5517-158f-26a140e5cbbf).

There are additional quotes in the pandas parser case.

cc @glemaitre

```py
import numpy as np
from sklearn.datasets import fetch_openml

mice_pandas = fetch_openml(name='miceprotein', version=4, parser="pandas")
mice_liac_arff = fetch_openml(name='miceprotein', version=4, parser="liac-arff")

assert np.testing.assert_array_equal(mice_pandas.target, mice_liac_arff.target)
```

Output:
```
AssertionError: 
Arrays are not equal

Mismatched elements: 1080 / 1080 (100%)
 x: array(["'c-CS-m'", "'c-CS-m'", "'c-CS-m'", ..., "'t-SC-s'", "'t-SC-s'",
       "'t-SC-s'"], dtype=object)
 y: array(['c-CS-m', 'c-CS-m', 'c-CS-m', ..., 't-SC-s', 't-SC-s', 't-SC-s'],
      dtype=object)
```

COMMENT:
Looks like both single quotes and double quotes are valid in ARFF, e.g. https://github.com/renatopp/liac-arff/issues/15#issuecomment-203694803.

Both single quotes and double quotes are used for example:
- Titanic uses double quotes: https://www.openml.org/search?type=data&sort=runs&id=40945&status=active
- Mice uses single quotes (hence the problemT here): https://www.openml.org/search?type=data&sort=runs&id=40966&status=active

I don't think there is a way to use two different `quotechar` in `pandas` (this is a stdlib `csv` thing) and I don't think that we can handle this via `quotechar`, `quoting` or `doublequote`.

There are probably some work-arounds like changing the delimiter to allow one quote characters on each side of the delimiter or post-process the string columns to remove quotes at the beginning and end.

Better suggestions more than welcome!

Labels: Bug, module:datasets
Scorer in sklearn.linear_model.RidgeCV

### Describe the bug

When changing scoring methods in sklearn.linear_model.RidgeCV the output remains the same 
![image](https://user-images.githubusercontent.com/32062516/168557712-85020880-cc1c-4f2f-b010-a625f4bfc216.png)
Is there a way to have this work properly?



### Steps/Code to Reproduce

![image](https://user-images.githubusercontent.com/32062516/168557843-67da398e-7bc7-49f6-87ad-7d56d83c1aa8.png)

### Expected Results

Different cv_values_

### Actual Results

Same cv_values_

### Versions

```shell
python 3.11
```

COMMENT:
The output is inline with the documentation:

```
After fit() has been called, this attribute will contain the mean 
squared errors if scoring is None otherwise it will contain standardized 
per point prediction values.
```

When providing a `scoring` parameter, you get the standardized per point prediction values and not the score.

However, with another bug regarding the scale of the predictions: xref https://github.com/scikit-learn/scikit-learn/issues/13998

Labels: Bug, Needs Triage
Scorer in sklearn.linear_model.RidgeCV

### Describe the bug

When changing scoring methods in sklearn.linear_model.RidgeCV the output remains the same 
![image](https://user-images.githubusercontent.com/32062516/168557712-85020880-cc1c-4f2f-b010-a625f4bfc216.png)
Is there a way to have this work properly?



### Steps/Code to Reproduce

![image](https://user-images.githubusercontent.com/32062516/168557843-67da398e-7bc7-49f6-87ad-7d56d83c1aa8.png)

### Expected Results

Different cv_values_

### Actual Results

Same cv_values_

### Versions

```shell
python 3.11
```

COMMENT:
We should then fix #13998

Labels: Bug, Needs Triage
The data type of input data for LinearRegression class will affect the results

### Describe the bug

Our team just used the class sklearn.linear_model.LinearRegression to do multi-linear regression. And, we found out that the same data, which means the values are identical for each element, with different data formats will return different coefficients. The array we use are all integer values but saved as float32 and float64, separately. However, the array in float64 as input get us the expected result, but float32 doesn't.

The code below is a simple reproducible example.

### Steps/Code to Reproduce

```python
In [1]: from sklearn.linear_model import LinearRegression
    ...: import numpy as np
    ...: rng = np.random.default_rng(0)
    ...: x_1_array = rng.integers(low=1000, high=10000, size=10_000_000)
    ...: x_2_array = rng.integers(low=1000, high=10000, size=10_000_000)
    ...: x_3_array = rng.integers(low=1000, high=10000, size=10_000_000)
    ...: y_array = 3 * x_1_array + x_2_array - 5 * x_3_array
    ...: intercept_data = rng.integers(low=-3, high=3)
    ...: print(f'True intercept {intercept_data}')
    ...: y_array += intercept_data
    ...: X_train = np.stack([x_1_array,x_2_array,x_3_array],axis=-1)
    ...: y_train = y_array
    ...: linreg = LinearRegression()
    ...: model = linreg.fit(X_train.astype(np.float64), y_train)
    ...: print(f'Float 64 Intercept: {linreg.intercept_}')
    ...: print(f'Float 64 Coefficient: {linreg.coef_}')
    ...: model = linreg.fit(X_train.astype(np.float32), y_train)
    ...: print(f'Float 32 Intercept: {linreg.intercept_}')
    ...: print(f'Float 32 Coefficient: {linreg.coef_}')
```

### Expected Results

![image](https://user-images.githubusercontent.com/51532833/168492124-d98f9663-4899-47ee-83b2-4952462c37a6.png)

### Actual Results

![image](https://user-images.githubusercontent.com/51532833/168492136-4e0f4100-9e48-4f58-aa2a-97e826f06c61.png)


### Versions

```shell
System:
    python: 3.7.9 (v3.7.9:13c94747c7, Aug 15 2020, 01:31:08)  [Clang 6.0 (clang-600.0.57)]
executable: /Users/lazy/PycharmProjects/projects/bin/python
   machine: Darwin-21.4.0-x86_64-i386-64bit

Python dependencies:
          pip: 21.3
   setuptools: 58.2.0
      sklearn: 1.0.2
        numpy: 1.21.4
        scipy: 1.7.2
       Cython: None
       pandas: 1.3.4
   matplotlib: 3.4.3
       joblib: 1.1.0
threadpoolctl: 3.1.0

Built with OpenMP: True
```

COMMENT:
I updated the snippet of code such that we set the seed.
I forsee that the issue is in the intercept values indeed. I could also reproduce the same behaviour with `Ridge` and avery small alpha value.

Labels: Bug, module:linear_model, float32
The data type of input data for LinearRegression class will affect the results

### Describe the bug

Our team just used the class sklearn.linear_model.LinearRegression to do multi-linear regression. And, we found out that the same data, which means the values are identical for each element, with different data formats will return different coefficients. The array we use are all integer values but saved as float32 and float64, separately. However, the array in float64 as input get us the expected result, but float32 doesn't.

The code below is a simple reproducible example.

### Steps/Code to Reproduce

```python
In [1]: from sklearn.linear_model import LinearRegression
    ...: import numpy as np
    ...: rng = np.random.default_rng(0)
    ...: x_1_array = rng.integers(low=1000, high=10000, size=10_000_000)
    ...: x_2_array = rng.integers(low=1000, high=10000, size=10_000_000)
    ...: x_3_array = rng.integers(low=1000, high=10000, size=10_000_000)
    ...: y_array = 3 * x_1_array + x_2_array - 5 * x_3_array
    ...: intercept_data = rng.integers(low=-3, high=3)
    ...: print(f'True intercept {intercept_data}')
    ...: y_array += intercept_data
    ...: X_train = np.stack([x_1_array,x_2_array,x_3_array],axis=-1)
    ...: y_train = y_array
    ...: linreg = LinearRegression()
    ...: model = linreg.fit(X_train.astype(np.float64), y_train)
    ...: print(f'Float 64 Intercept: {linreg.intercept_}')
    ...: print(f'Float 64 Coefficient: {linreg.coef_}')
    ...: model = linreg.fit(X_train.astype(np.float32), y_train)
    ...: print(f'Float 32 Intercept: {linreg.intercept_}')
    ...: print(f'Float 32 Coefficient: {linreg.coef_}')
```

### Expected Results

![image](https://user-images.githubusercontent.com/51532833/168492124-d98f9663-4899-47ee-83b2-4952462c37a6.png)

### Actual Results

![image](https://user-images.githubusercontent.com/51532833/168492136-4e0f4100-9e48-4f58-aa2a-97e826f06c61.png)


### Versions

```shell
System:
    python: 3.7.9 (v3.7.9:13c94747c7, Aug 15 2020, 01:31:08)  [Clang 6.0 (clang-600.0.57)]
executable: /Users/lazy/PycharmProjects/projects/bin/python
   machine: Darwin-21.4.0-x86_64-i386-64bit

Python dependencies:
          pip: 21.3
   setuptools: 58.2.0
      sklearn: 1.0.2
        numpy: 1.21.4
        scipy: 1.7.2
       Cython: None
       pandas: 1.3.4
   matplotlib: 3.4.3
       joblib: 1.1.0
threadpoolctl: 3.1.0

Built with OpenMP: True
```

COMMENT:
It should be noted that the problem is getting worse when the number of data points increases.

Labels: Bug, module:linear_model, float32