Talip7/scikit-learn-issues-multilabel huggingface dataset:
Unexpected output from Random Forest Classifer

### Describe the bug

I attempted to use Random Forest Classifier on a data with binarized labels. And I realized the predictions given out always had one class missing. I tried on my data and also tried on one of the scikit-learn datasets and the same observation was made. I want to plot an ROC curve at the day using the code specified on the [scikit learn page](https://scikit-learn.org/stable/auto_examples/model_selection/plot_roc.html)

### Steps/Code to Reproduce
Here is the link to the code [https://gist.github.com/vappiah/8e2190611f600854489b6c7539e7b8d0](https://gist.github.com/vappiah/8e2190611f600854489b6c7539e7b8d0)
### Expected Results
[0 1 2]

### Actual Results
[0 2]

### Versions
```shell
System:
    python: 3.8.10 (default, Nov 26 2021, 20:14:08)  [GCC 9.3.0]
executable: /usr/bin/python3
   machine: Linux-5.13.0-30-generic-x86_64-with-glibc2.29

Python dependencies:
          pip: 20.0.2
   setuptools: 45.2.0
      sklearn: 0.24.2
        numpy: 1.17.4
        scipy: 1.7.1
       Cython: None
       pandas: 1.3.1
   matplotlib: 3.4.2
       joblib: 1.0.1
threadpoolctl: 2.2.0

Built with OpenMP: True
```

COMMENT:
In this case, the model is performing poorly because of all the noisy features. One can help with adjusting the hyperparameters for the random forest. In the following example, I set `max_features=0.2`:

```python
import numpy as np
from sklearn import datasets
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import label_binarize
from sklearn.ensemble import RandomForestClassifier

iris = datasets.load_iris()
X, y = iris.data, iris.target
y = label_binarize(y, classes=[0, 1, 2])

random_state = np.random.RandomState(0)
n_samples, n_features = X.shape
X = np.c_[X, random_state.randn(n_samples, 200 * n_features)]

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.5, random_state=0)
classifier = RandomForestClassifier(random_state=0, max_features=0.2)
y_score = classifier.fit(X_train, y_train).predict(X_test)
classes = np.unique(y_score.argmax(axis=1))
print(classes)
# [0, 1, 2]
```

Note I used `RandomForestClassifier` here because it directly supports binarized targets. Also note that `RandomForestClassifier` can handle multiclass directly without needing to binarize:

```python
import numpy as np
from sklearn import datasets
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier

iris = datasets.load_iris()
X, y = iris.data, iris.target

random_state = np.random.RandomState(0)
n_samples, n_features = X.shape
X = np.c_[X, random_state.randn(n_samples, 200 * n_features)]

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.5, random_state=0, stratify=y
)
classifier = RandomForestClassifier(random_state=0, max_features=0.2)
y_score = classifier.fit(X_train, y_train).predict(X_test)
print(np.unique(y_score))
# [0, 1, 2]
```

Here I stratified based on `y` to preserve the percentage of samples for each class in the train and test set.

Labels: Question, module:ensemble
Unexpected output from Random Forest Classifer

### Describe the bug

I attempted to use Random Forest Classifier on a data with binarized labels. And I realized the predictions given out always had one class missing. I tried on my data and also tried on one of the scikit-learn datasets and the same observation was made. I want to plot an ROC curve at the day using the code specified on the [scikit learn page](https://scikit-learn.org/stable/auto_examples/model_selection/plot_roc.html)

### Steps/Code to Reproduce
Here is the link to the code [https://gist.github.com/vappiah/8e2190611f600854489b6c7539e7b8d0](https://gist.github.com/vappiah/8e2190611f600854489b6c7539e7b8d0)
### Expected Results
[0 1 2]

### Actual Results
[0 2]

### Versions
```shell
System:
    python: 3.8.10 (default, Nov 26 2021, 20:14:08)  [GCC 9.3.0]
executable: /usr/bin/python3
   machine: Linux-5.13.0-30-generic-x86_64-with-glibc2.29

Python dependencies:
          pip: 20.0.2
   setuptools: 45.2.0
      sklearn: 0.24.2
        numpy: 1.17.4
        scipy: 1.7.1
       Cython: None
       pandas: 1.3.1
   matplotlib: 3.4.2
       joblib: 1.0.1
threadpoolctl: 2.2.0

Built with OpenMP: True
```

COMMENT:
Thanks everyone, @thomasjpfan . It seems the link to my code was broken so I fixed it
One more question. I would like to plot an ROC Curve just like the one on the [scikit page](https://scikit-learn.org/stable/auto_examples/model_selection/plot_roc.html)
Is it necessary for me to include the OnevsRest classifier  ? eg. _OneVsRestClassifier(RandomForestClassifier(random_state=0, max_features=0.2))_

Labels: Question, module:ensemble
Unexpected output from Random Forest Classifer

### Describe the bug

I attempted to use Random Forest Classifier on a data with binarized labels. And I realized the predictions given out always had one class missing. I tried on my data and also tried on one of the scikit-learn datasets and the same observation was made. I want to plot an ROC curve at the day using the code specified on the [scikit learn page](https://scikit-learn.org/stable/auto_examples/model_selection/plot_roc.html)

### Steps/Code to Reproduce
Here is the link to the code [https://gist.github.com/vappiah/8e2190611f600854489b6c7539e7b8d0](https://gist.github.com/vappiah/8e2190611f600854489b6c7539e7b8d0)
### Expected Results
[0 1 2]

### Actual Results
[0 2]

### Versions
```shell
System:
    python: 3.8.10 (default, Nov 26 2021, 20:14:08)  [GCC 9.3.0]
executable: /usr/bin/python3
   machine: Linux-5.13.0-30-generic-x86_64-with-glibc2.29

Python dependencies:
          pip: 20.0.2
   setuptools: 45.2.0
      sklearn: 0.24.2
        numpy: 1.17.4
        scipy: 1.7.1
       Cython: None
       pandas: 1.3.1
   matplotlib: 3.4.2
       joblib: 1.0.1
threadpoolctl: 2.2.0

Built with OpenMP: True
```

COMMENT:
The smallest change to get it to work is:

```python
# Learn to predict each class against the other
classifier = OneVsRestClassifier(RandomForestClassifier(max_features=0.2))
y_score = classifier.fit(X_train, y_train).predict_proba(X_test)
```

Note that `predict_proba` is called to get `y_score` because [roc_curve](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_curve.html#sklearn.metrics.roc_curve) requires scores: 

> Target scores, can either be probability estimates of the positive class, confidence values, or non-thresholded measure of decisions (as returned by “decision_function” on some classifiers).

The random forest has `predict_proba`, but not a `decision_function`.

I am closing this issue as the original issue was resolved. Thanks again for opening the issue.

Labels: Question, module:ensemble
Unexpected output from Random Forest Classifer

### Describe the bug

I attempted to use Random Forest Classifier on a data with binarized labels. And I realized the predictions given out always had one class missing. I tried on my data and also tried on one of the scikit-learn datasets and the same observation was made. I want to plot an ROC curve at the day using the code specified on the [scikit learn page](https://scikit-learn.org/stable/auto_examples/model_selection/plot_roc.html)

### Steps/Code to Reproduce
Here is the link to the code [https://gist.github.com/vappiah/8e2190611f600854489b6c7539e7b8d0](https://gist.github.com/vappiah/8e2190611f600854489b6c7539e7b8d0)
### Expected Results
[0 1 2]

### Actual Results
[0 2]

### Versions
```shell
System:
    python: 3.8.10 (default, Nov 26 2021, 20:14:08)  [GCC 9.3.0]
executable: /usr/bin/python3
   machine: Linux-5.13.0-30-generic-x86_64-with-glibc2.29

Python dependencies:
          pip: 20.0.2
   setuptools: 45.2.0
      sklearn: 0.24.2
        numpy: 1.17.4
        scipy: 1.7.1
       Cython: None
       pandas: 1.3.1
   matplotlib: 3.4.2
       joblib: 1.0.1
threadpoolctl: 2.2.0

Built with OpenMP: True
```

COMMENT:
Thanks very much

Labels: Question, module:ensemble
Include entire range in `check_scalar` error message

Currently docstrings description for scalar ranges uses the interval syntax:

https://github.com/scikit-learn/scikit-learn/blob/42cc05c5ddac0e0c4392871a6825c53ac88ace36/sklearn/linear_model/_glm/glm.py#L462

While the error message uses a different notation:

```python
from sklearn.utils.validation import check_scalar

check_scalar(-1.0, "alpha", target_type=float, min_val=0.0)
# ValueError: alpha == -1.0, must be >= 0.0.
```

Should we update this notation to be consistent with the docstring? i.e:

```python
# ValueError: alpha == -1.0, must be in the range [0.0, inf)
```

(I am assuming that `inf` is **always** an open boundary)

CC @glemaitre

COMMENT:
I'm a fan of this. The consistency is very nice.

Labels: Enhancement, module:utils
Include entire range in `check_scalar` error message

Currently docstrings description for scalar ranges uses the interval syntax:

https://github.com/scikit-learn/scikit-learn/blob/42cc05c5ddac0e0c4392871a6825c53ac88ace36/sklearn/linear_model/_glm/glm.py#L462

While the error message uses a different notation:

```python
from sklearn.utils.validation import check_scalar

check_scalar(-1.0, "alpha", target_type=float, min_val=0.0)
# ValueError: alpha == -1.0, must be >= 0.0.
```

Should we update this notation to be consistent with the docstring? i.e:

```python
# ValueError: alpha == -1.0, must be in the range [0.0, inf)
```

(I am assuming that `inf` is **always** an open boundary)

CC @glemaitre

COMMENT:
I would be fine with this change.

Labels: Enhancement, module:utils
Include entire range in `check_scalar` error message

Currently docstrings description for scalar ranges uses the interval syntax:

https://github.com/scikit-learn/scikit-learn/blob/42cc05c5ddac0e0c4392871a6825c53ac88ace36/sklearn/linear_model/_glm/glm.py#L462

While the error message uses a different notation:

```python
from sklearn.utils.validation import check_scalar

check_scalar(-1.0, "alpha", target_type=float, min_val=0.0)
# ValueError: alpha == -1.0, must be >= 0.0.
```

Should we update this notation to be consistent with the docstring? i.e:

```python
# ValueError: alpha == -1.0, must be in the range [0.0, inf)
```

(I am assuming that `inf` is **always** an open boundary)

CC @glemaitre

COMMENT:
I opened #22722 that proposes a common design for parameter validation, to go beyond check_scalar.

Labels: Enhancement, module:utils
Estimator check for dtype preservation for regressors

### Describe the workflow you want to enable

As discussed in https://github.com/scikit-learn/scikit-learn/pull/22663#issuecomment-1058368882, we should have a common test that checks that the `predict` method of regressors preserves the dtype, similarly to `check_transformer_preserve_dtypes` for transformers.

More specifically I would expect the following to hold most of the time:

```python
>>> dtype = np.float32
>>> reg = Regressor().fit(X_train.astype(dtype), y_train.astype(dtype))
>>> assert reg.predict(X_test.astype(dtype)).dtype == dtype
```

If `X_train`, `y_train`, and `X_test` do not all share the same dtype, I would be in favor of leaving the behavior undefined (and unchecked).

This could be coupled with the existing `preserves_dtype` estimator tag.



### Describe your proposed solution

- Implement a new `check_regressor_preserve_dtypes` function, next to `check_transformer_preserve_dtypes`.
- Make sure it does not nothing if the regressor does not define the `preserves_dtype` estimator tag.
- Identify a few regressors where this property holds, add the tag for `float64` and `float32` and check that the common test pass with:

```
pytest -vk "check_regressor_preserve_dtypes and RegressorClassName"
```

Once done, create a meta-issue to track all regressors that should be labeled to preserve dtypes, similarly to what the following issue does for transformers:

- #11000

COMMENT:
/take

Labels: New Feature, module:test-suite, float32
Estimator check for dtype preservation for regressors

### Describe the workflow you want to enable

As discussed in https://github.com/scikit-learn/scikit-learn/pull/22663#issuecomment-1058368882, we should have a common test that checks that the `predict` method of regressors preserves the dtype, similarly to `check_transformer_preserve_dtypes` for transformers.

More specifically I would expect the following to hold most of the time:

```python
>>> dtype = np.float32
>>> reg = Regressor().fit(X_train.astype(dtype), y_train.astype(dtype))
>>> assert reg.predict(X_test.astype(dtype)).dtype == dtype
```

If `X_train`, `y_train`, and `X_test` do not all share the same dtype, I would be in favor of leaving the behavior undefined (and unchecked).

This could be coupled with the existing `preserves_dtype` estimator tag.



### Describe your proposed solution

- Implement a new `check_regressor_preserve_dtypes` function, next to `check_transformer_preserve_dtypes`.
- Make sure it does not nothing if the regressor does not define the `preserves_dtype` estimator tag.
- Identify a few regressors where this property holds, add the tag for `float64` and `float32` and check that the common test pass with:

```
pytest -vk "check_regressor_preserve_dtypes and RegressorClassName"
```

Once done, create a meta-issue to track all regressors that should be labeled to preserve dtypes, similarly to what the following issue does for transformers:

- #11000

COMMENT:
@ogrisel how to add tag for float64 and float32 to regressors. 

Should I add function like

>`def _more_tags(self):`
>>`        return {"preserves_dtype": [np.float32, np.float64]}`

to the estimator class  or something else, there are many inheritance so got a little confused.

Edit: resolved from issues #11000

Labels: New Feature, module:test-suite, float32
Estimator check for dtype preservation for regressors

### Describe the workflow you want to enable

As discussed in https://github.com/scikit-learn/scikit-learn/pull/22663#issuecomment-1058368882, we should have a common test that checks that the `predict` method of regressors preserves the dtype, similarly to `check_transformer_preserve_dtypes` for transformers.

More specifically I would expect the following to hold most of the time:

```python
>>> dtype = np.float32
>>> reg = Regressor().fit(X_train.astype(dtype), y_train.astype(dtype))
>>> assert reg.predict(X_test.astype(dtype)).dtype == dtype
```

If `X_train`, `y_train`, and `X_test` do not all share the same dtype, I would be in favor of leaving the behavior undefined (and unchecked).

This could be coupled with the existing `preserves_dtype` estimator tag.



### Describe your proposed solution

- Implement a new `check_regressor_preserve_dtypes` function, next to `check_transformer_preserve_dtypes`.
- Make sure it does not nothing if the regressor does not define the `preserves_dtype` estimator tag.
- Identify a few regressors where this property holds, add the tag for `float64` and `float32` and check that the common test pass with:

```
pytest -vk "check_regressor_preserve_dtypes and RegressorClassName"
```

Once done, create a meta-issue to track all regressors that should be labeled to preserve dtypes, similarly to what the following issue does for transformers:

- #11000

COMMENT:
Hi ! Can i work on this issue ? Is this issue still available ? It would be my first time contributing to the project.

Labels: New Feature, module:test-suite, float32
TST Add option to run tests on 32bit data

### Context

Currently most implementations are tested against 64bit datasets only. The re-factoring of some internals for computations on 32bit datasets brought the need to test user-facing interfaces on 32bit datasets (see https://github.com/scikit-learn/scikit-learn/pull/22590#issuecomment-1054546370). A few group of PRs (such as #22663) introduced a simple parametrisation on `dtype`:

```python
# ... 
DTYPES = (np.float32, np.float64)

# ...

@pytest.mark.parametrize("dtype", DTYPES)
def test(dtype):
    # ...
```

### Problem

Yet, parametrising tests on `dtype` might just make running the test suite (much) longer. Hence, we could add an option to be able to run such tests for 32bit data optionally. In this case, we also need to have a way to specify this option so that it can be triggered on the CI.

### Proposed solution

We can keep the tests parametrisation on `dtype` as proposed in PRs (such as #22663) but have the value of `DTYPES` be specified globally (e.g. `in sklearn/utils/_testing.py`) and adaptively on the value environement variable. Naively:

```python 
# sklearn/utils/_testing.py

# ...

DTYPES = [np.float64]

if os.environ.get("SKLEARN_RUN_32BIT_TESTS"):
    DTYPES.append(np.float32)
```


As for the CI, we could have a new `[test 32bit]` [commit message marker](https://scikit-learn.org/stable/developers/contributing.html#continuous-integration-ci) which would define this environement variable.

COMMENT:
I'm not sure there's a need to run all or almost all tests against float32.
Obviously estimators that convert data to float64 (those for which self._validate_data has dtype=np.float64) should not be tested.
For estimators that are supposed to support float32 without conversion, we need 3 tests:
  - for transformers we test that the transformed array has the same dtype as the input. This already is a common test. For regressors we can check the prediction as discussed in https://github.com/scikit-learn/scikit-learn/issues/22682.
  - A test that check that some specific attributes have the dtype as the data (i.e. cluster_centers_ for KMeans). Which attribute to check is estimator specific and should be decided on a case by case basis.
  - A test that check that the results, i.e the transformed data or some attributes, are the same between float32 and float64. For that we need a bit of tolerance because machine precision of float32 is 1.2e-7 which is bigger than the default rtol of assert_allclose. Besides, rounding errors can accumulate meaning that we should use a rtol around 1e-5 or even 1e-4.

I think having these tests should be enough, in the sense that if we check that the result is the same between float32 and float64 then by transitivity, testing some property of an estimator on float64 is guaranteed to apply to float32 as well.

Labels: Build / CI, module:test-suite, workflow, float32
TST Add option to run tests on 32bit data

### Context

Currently most implementations are tested against 64bit datasets only. The re-factoring of some internals for computations on 32bit datasets brought the need to test user-facing interfaces on 32bit datasets (see https://github.com/scikit-learn/scikit-learn/pull/22590#issuecomment-1054546370). A few group of PRs (such as #22663) introduced a simple parametrisation on `dtype`:

```python
# ... 
DTYPES = (np.float32, np.float64)

# ...

@pytest.mark.parametrize("dtype", DTYPES)
def test(dtype):
    # ...
```

### Problem

Yet, parametrising tests on `dtype` might just make running the test suite (much) longer. Hence, we could add an option to be able to run such tests for 32bit data optionally. In this case, we also need to have a way to specify this option so that it can be triggered on the CI.

### Proposed solution

We can keep the tests parametrisation on `dtype` as proposed in PRs (such as #22663) but have the value of `DTYPES` be specified globally (e.g. `in sklearn/utils/_testing.py`) and adaptively on the value environement variable. Naively:

```python 
# sklearn/utils/_testing.py

# ...

DTYPES = [np.float64]

if os.environ.get("SKLEARN_RUN_32BIT_TESTS"):
    DTYPES.append(np.float32)
```


As for the CI, we could have a new `[test 32bit]` [commit message marker](https://scikit-learn.org/stable/developers/contributing.html#continuous-integration-ci) which would define this environement variable.

COMMENT:
Is the plan to convert all tests that parameterize over `float32` and `float64` to use the fixture? 

Lets say that a contributor is adjusting an algorithm to fix a bug that would fail a test for `float32`, we would not know until the PR is merged and the nightly build runs (assuming https://github.com/scikit-learn/scikit-learn/pull/23110 is merged). With a commit trigger like https://github.com/scikit-learn/scikit-learn/pull/23026#discussion_r847519228, there will be more responsibility on the reviewer to remember to run tests for `float32` to catch `float32` specific bugs before merging.

Labels: Build / CI, module:test-suite, workflow, float32
TST Add option to run tests on 32bit data

### Context

Currently most implementations are tested against 64bit datasets only. The re-factoring of some internals for computations on 32bit datasets brought the need to test user-facing interfaces on 32bit datasets (see https://github.com/scikit-learn/scikit-learn/pull/22590#issuecomment-1054546370). A few group of PRs (such as #22663) introduced a simple parametrisation on `dtype`:

```python
# ... 
DTYPES = (np.float32, np.float64)

# ...

@pytest.mark.parametrize("dtype", DTYPES)
def test(dtype):
    # ...
```

### Problem

Yet, parametrising tests on `dtype` might just make running the test suite (much) longer. Hence, we could add an option to be able to run such tests for 32bit data optionally. In this case, we also need to have a way to specify this option so that it can be triggered on the CI.

### Proposed solution

We can keep the tests parametrisation on `dtype` as proposed in PRs (such as #22663) but have the value of `DTYPES` be specified globally (e.g. `in sklearn/utils/_testing.py`) and adaptively on the value environement variable. Naively:

```python 
# sklearn/utils/_testing.py

# ...

DTYPES = [np.float64]

if os.environ.get("SKLEARN_RUN_32BIT_TESTS"):
    DTYPES.append(np.float32)
```


As for the CI, we could have a new `[test 32bit]` [commit message marker](https://scikit-learn.org/stable/developers/contributing.html#continuous-integration-ci) which would define this environement variable.

COMMENT:
> we would not know until the PR is merged and the nightly build runs

The fixture is already enabled on 1 CI job so we should know in the PR (except in rare situations where there's a numerical instability only visible on another platform).

 > Is the plan to convert all tests that parameterize over float32 and float64 to use the fixture?

At first I didn't think so, but then I thought why not. Transformers that have received the preserves_dtype flag recently have been granted new tests with the fixture. I don't see a reason to treat the other transformers differently. Maybe we shouldn't do that for tests added for a bug fix though (maybe it's already the case :/ ).

Ideally I would always run tests for both precisions but parametrizing more and more tests over float64/float32 makes the CI take more and more time, and it's already too long (without mentioning energy consumption). I think having the fixture on 1 job for the PR and all jobs for the nightly builds is a good compromise. Wdyt ?

Labels: Build / CI, module:test-suite, workflow, float32