Repository: scikit-learn/scikit-learn
Pull request: FIX RandomForestRegressor OOB fails with integer-values multiple targets
Fixes #27814

Allow to compute the OOB score with `RandomForestRegressor` and an integral multi-target regression.

Problem statement:
RandomForestRegressor having problem with integer-values targets: The type of target cannot be used to compute OOB estimates
### Describe the bug

When having:
- RandomForestRegressor
- Multiple targets
- integer values only (e.g., 1.0, 2.0, 3.0, ...) in the targets
- oob_score=True

The check in `BaseForest` will raise the error:

```
ValueError: The type of target cannot be used to compute OOB estimates. Got multiclass-multioutput while only the following are supported: continuous, continuous-multioutput, binary, multiclass, multilabel-indicator.
```

because `type_of_target` misclassifies the target as multiclass instead of continuous when integer values are reported.
This is a bug because (1) I explicitly requested for a Regressor and (2) the classes are clearly to many to be a classification problem.

I could solve the problem by perturbing just a bit one value for each target, e.g.,:
```python 
for target in targets:
   df[target].iloc[0] *= 1.0001
```
but I would like to work out a more definitive fix in the program.


### Steps/Code to Reproduce

```python
import pandas as pd
from sklearn.ensemble import RandomForestRegressor

df = pd.DataFrame({
    "feat1": [1 ,2 ,3 ,4 ,5 ,6 ,7 ,8 ,9 ,10],
    "feat2": [2, 6, 8, 1, 3, 5, 7, 9, 4, 10],
    "target1": [4.0, 6.0, 7.0, 3.0, 5.0, 4.0, 6.0 ,7.0 ,8.0 ,9.0],
    "target2": [5.0, 5.0, 6.0, 7.0, 3.0, 4.0, 10.0,6.0,6.0,7.0],
})

rf = RandomForestRegressor(oob_score=True, random_state=42)
rf.fit(df[["feat1", "feat2"]], df[["target1", "target2"]])
```

### Expected Results

RandomForestRegressor(oob_score=True, random_state=42)

### Actual Results

```
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[5], line 12
      4 df = pd.DataFrame({
      5     "feat1": [1 ,2 ,3 ,4 ,5 ,6 ,7 ,8 ,9 ,10],
      6     "feat2": [2, 6, 8, 1, 3, 5, 7, 9, 4, 10],
      7     "target1": [4.0, 6.0, 7.0, 3.0, 5.0, 4.0, 6.0 ,7.0 ,8.0 ,9.0],
      8     "target2": [5.0, 5.0, 6.0, 7.0, 3.0, 4.0, 10.0,6.0,6.0,7.0],
      9 })
     11 rf = RandomForestRegressor(oob_score=True, random_state=42)
---> 12 rf.fit(df[["feat1", "feat2"]], df[["target1", "target2"]])

File c:\ProgramData\miniconda3\envs\py310\lib\site-packages\sklearn\ensemble\_forest.py:503, in BaseForest.fit(self, X, y, sample_weight)
    497     y_type = type_of_target(y)
    498     if y_type in ("multiclass-multioutput", "unknown"):
    499         # FIXME: we could consider to support multiclass-multioutput if
    500         # we introduce or reuse a constructor parameter (e.g.
    501         # oob_score) allowing our user to pass a callable defining the
    502         # scoring strategy on OOB sample.
--> 503         raise ValueError(
    504             "The type of target cannot be used to compute OOB "
    505             f"estimates. Got {y_type} while only the following are "
    506             "supported: continuous, continuous-multioutput, binary, "
    507             "multiclass, multilabel-indicator."
    508         )
    509     self._set_oob_score_and_attributes(X, y)
    511 # Decapsulate classes_ attributes

ValueError: The type of target cannot be used to compute OOB estimates. Got multiclass-multioutput while only the following are supported: continuous, continuous-multioutput, binary, multiclass, multilabel-indicator.
````






















### Versions

```shell
1.3.2
```

Hints from issue discussion:
Indeed, we should be able to compute the OOB score. We did not anticipate that `type_of_target` will return the invalid type. You PR as the right fix since that we can leverage already the `_estimator_type`.

Commit message: add tests for all combinations of conditions

Patch to review:
diff --git a/doc/whats_new/v1.4.rst b/doc/whats_new/v1.4.rst
index f3f40fd26e27a..c4f525d3d3040 100644
--- a/doc/whats_new/v1.4.rst
+++ b/doc/whats_new/v1.4.rst
@@ -331,6 +331,7 @@ Changelog
 - |API| In :class:`ensemble.AdaBoostClassifier`, the `algorithm` argument `SAMME.R` was
   deprecated and will be removed in 1.6. :pr:`26830` by :user:`Stefanie Senger
   <StefanieSenger>`.
+
 - |Enhancement| A fitted property, ``estimators_samples_``, was added to all Forest methods,
   including
   :class:`ensemble.RandomForestClassifier`, :class:`ensemble.RandomForestRegressor`,
@@ -338,6 +339,11 @@ Changelog
   which allows to retrieve the training sample indices used for each tree estimator.
   :pr:`26736` by :user:`Adam Li <adam2392>`.
 
+- |Fix| ValueError in :class:`ensemble.RandomForestRegressor` and
+  :class:`ensemble.ExtraTreesRegressor` when requesting OOB score with multioutput model
+  for the targets being all rounded to integer. It was recognized as a multiclass problem.
+  :pr:`27817` by :user:`Daniele Ongari <danieleongari>`
+
 :mod:`sklearn.feature_selection`
 ................................
 
diff --git a/sklearn/ensemble/_forest.py b/sklearn/ensemble/_forest.py
index f2d60cde0a0dc..b9727032d21a0 100644
--- a/sklearn/ensemble/_forest.py
+++ b/sklearn/ensemble/_forest.py
@@ -516,7 +516,10 @@ def fit(self, X, y, sample_weight=None):
             n_more_estimators > 0 or not hasattr(self, "oob_score_")
         ):
             y_type = type_of_target(y)
-            if y_type in ("multiclass-multioutput", "unknown"):
+            if y_type == "unknown" or (
+                self._estimator_type == "classifier"
+                and y_type == "multiclass-multioutput"
+            ):
                 # FIXME: we could consider to support multiclass-multioutput if
                 # we introduce or reuse a constructor parameter (e.g.
                 # oob_score) allowing our user to pass a callable defining the
diff --git a/sklearn/ensemble/tests/test_forest.py b/sklearn/ensemble/tests/test_forest.py
index b9c72f9d79d12..9c6fffa0c85c7 100644
--- a/sklearn/ensemble/tests/test_forest.py
+++ b/sklearn/ensemble/tests/test_forest.py
@@ -598,29 +598,36 @@ def test_forest_oob_warning(ForestEstimator):
 
 
 @pytest.mark.parametrize("ForestEstimator", FOREST_CLASSIFIERS_REGRESSORS.values())
-@pytest.mark.parametrize(
-    "X, y, params, err_msg",
-    [
-        (
-            iris.data,
-            iris.target,
-            {"oob_score": True, "bootstrap": False},
-            "Out of bag estimation only available if bootstrap=True",
-        ),
-        (
-            iris.data,
-            rng.randint(low=0, high=5, size=(iris.data.shape[0], 2)),
-            {"oob_score": True, "bootstrap": True},
-            "The type of target cannot be used to compute OOB estimates",
-        ),
-    ],
-)
-def test_forest_oob_error(ForestEstimator, X, y, params, err_msg):
+def test_forest_boostrap_false_oob_error(ForestEstimator):
+    X = iris.data
+    y = iris.target
+    params = {"oob_score": True, "bootstrap": False}
+    err_msg = "Out of bag estimation only available if bootstrap=True"
     estimator = ForestEstimator(**params)
     with pytest.raises(ValueError, match=err_msg):
         estimator.fit(X, y)
 
 
+@pytest.mark.parametrize("ForestClassifier", FOREST_CLASSIFIERS.values())
+def test_forest_moutput_int_oob_error(ForestClassifier):
+    X = iris.data
+    y = rng.randint(low=0, high=5, size=(iris.data.shape[0], 2))
+    params = {"oob_score": True, "bootstrap": True}
+    err_msg = "The type of target cannot be used to compute OOB estimates"
+    estimator = ForestClassifier(**params)
+    with pytest.raises(ValueError, match=err_msg):
+        estimator.fit(X, y)
+
+
+@pytest.mark.parametrize("ForestRegressor", FOREST_REGRESSORS.values())
+def test_forest_moutput_int_oob(ForestRegressor):
+    X = iris.data
+    y = rng.randint(low=0, high=5, size=(iris.data.shape[0], 2))
+    params = {"oob_score": True, "bootstrap": True}
+    estimator = ForestRegressor(**params)
+    estimator.fit(X, y)
+
+
 @pytest.mark.parametrize("oob_score", [True, False])
 def test_random_trees_embedding_raise_error_oob(oob_score):
     with pytest.raises(TypeError, match="got an unexpected keyword argument"):


Code review comments:
diff_hunk: @@ -598,29 +598,36 @@ def test_forest_oob_warning(ForestEstimator):
 
 
 @pytest.mark.parametrize("ForestEstimator", FOREST_CLASSIFIERS_REGRESSORS.values())
-@pytest.mark.parametrize(
-    "X, y, params, err_msg",
-    [
-        (
-            iris.data,
-            iris.target,
-            {"oob_score": True, "bootstrap": False},
-            "Out of bag estimation only available if bootstrap=True",
-        ),
-        (
-            iris.data,
-            rng.randint(low=0, high=5, size=(iris.data.shape[0], 2)),
-            {"oob_score": True, "bootstrap": True},
-            "The type of target cannot be used to compute OOB estimates",
-        ),
-    ],
-)
-def test_forest_oob_error(ForestEstimator, X, y, params, err_msg):
+def test_forest_boostrap_false_oob_error(ForestEstimator):
+    X = iris.data
+    y = iris.target
+    params = {"oob_score": True, "bootstrap": False}
+    err_msg = "Out of bag estimation only available if bootstrap=True"
     estimator = ForestEstimator(**params)
     with pytest.raises(ValueError, match=err_msg):
path: sklearn/ensemble/tests/test_forest.py
text: @user1:

```suggestion
    estimator = ForestEstimator(oob_score=True, bootstrap=False)
    err_msg = "Out of bag estimation only available if bootstrap=True"
    with pytest.raises(ValueError, match=err_msg):
```

diff_hunk: @@ -598,29 +598,36 @@ def test_forest_oob_warning(ForestEstimator):
 
 
 @pytest.mark.parametrize("ForestEstimator", FOREST_CLASSIFIERS_REGRESSORS.values())
-@pytest.mark.parametrize(
-    "X, y, params, err_msg",
-    [
-        (
-            iris.data,
-            iris.target,
-            {"oob_score": True, "bootstrap": False},
-            "Out of bag estimation only available if bootstrap=True",
-        ),
-        (
-            iris.data,
-            rng.randint(low=0, high=5, size=(iris.data.shape[0], 2)),
-            {"oob_score": True, "bootstrap": True},
-            "The type of target cannot be used to compute OOB estimates",
-        ),
-    ],
-)
-def test_forest_oob_error(ForestEstimator, X, y, params, err_msg):
+def test_forest_boostrap_false_oob_error(ForestEstimator):
+    X = iris.data
+    y = iris.target
+    params = {"oob_score": True, "bootstrap": False}
+    err_msg = "Out of bag estimation only available if bootstrap=True"
     estimator = ForestEstimator(**params)
     with pytest.raises(ValueError, match=err_msg):
         estimator.fit(X, y)
 
 
+@pytest.mark.parametrize("ForestClassifier", FOREST_CLASSIFIERS.values())
+def test_forest_moutput_int_oob_error(ForestClassifier):
+    X = iris.data
+    y = rng.randint(low=0, high=5, size=(iris.data.shape[0], 2))
+    params = {"oob_score": True, "bootstrap": True}
+    err_msg = "The type of target cannot be used to compute OOB estimates"
+    estimator = ForestClassifier(**params)
path: sklearn/ensemble/tests/test_forest.py
text: @user1:

```suggestion
    estimator = ForestClassifier(oob_score=True, bootstrap=True)
    err_msg = "The type of target cannot be used to compute OOB estimates"
```

diff_hunk: @@ -598,29 +598,36 @@ def test_forest_oob_warning(ForestEstimator):
 
 
 @pytest.mark.parametrize("ForestEstimator", FOREST_CLASSIFIERS_REGRESSORS.values())
-@pytest.mark.parametrize(
-    "X, y, params, err_msg",
-    [
-        (
-            iris.data,
-            iris.target,
-            {"oob_score": True, "bootstrap": False},
-            "Out of bag estimation only available if bootstrap=True",
-        ),
-        (
-            iris.data,
-            rng.randint(low=0, high=5, size=(iris.data.shape[0], 2)),
-            {"oob_score": True, "bootstrap": True},
-            "The type of target cannot be used to compute OOB estimates",
-        ),
-    ],
-)
-def test_forest_oob_error(ForestEstimator, X, y, params, err_msg):
+def test_forest_boostrap_false_oob_error(ForestEstimator):
+    X = iris.data
+    y = iris.target
+    params = {"oob_score": True, "bootstrap": False}
+    err_msg = "Out of bag estimation only available if bootstrap=True"
     estimator = ForestEstimator(**params)
     with pytest.raises(ValueError, match=err_msg):
         estimator.fit(X, y)
 
 
+@pytest.mark.parametrize("ForestClassifier", FOREST_CLASSIFIERS.values())
+def test_forest_moutput_int_oob_error(ForestClassifier):
+    X = iris.data
+    y = rng.randint(low=0, high=5, size=(iris.data.shape[0], 2))
+    params = {"oob_score": True, "bootstrap": True}
+    err_msg = "The type of target cannot be used to compute OOB estimates"
+    estimator = ForestClassifier(**params)
+    with pytest.raises(ValueError, match=err_msg):
+        estimator.fit(X, y)
+
+
+@pytest.mark.parametrize("ForestRegressor", FOREST_REGRESSORS.values())
+def test_forest_moutput_int_oob(ForestRegressor):
path: sklearn/ensemble/tests/test_forest.py
text: @user1:

```suggestion
def test_forest_multioutput_integral_regression_target(ForestRegressor):
    """Check that multioutput regression with integral values is not interpreted
    as a multiclass-multioutput target and OOB score can be computed.
    """
```

diff_hunk: @@ -598,29 +598,36 @@ def test_forest_oob_warning(ForestEstimator):
 
 
 @pytest.mark.parametrize("ForestEstimator", FOREST_CLASSIFIERS_REGRESSORS.values())
-@pytest.mark.parametrize(
-    "X, y, params, err_msg",
-    [
-        (
-            iris.data,
-            iris.target,
-            {"oob_score": True, "bootstrap": False},
-            "Out of bag estimation only available if bootstrap=True",
-        ),
-        (
-            iris.data,
-            rng.randint(low=0, high=5, size=(iris.data.shape[0], 2)),
-            {"oob_score": True, "bootstrap": True},
-            "The type of target cannot be used to compute OOB estimates",
-        ),
-    ],
-)
-def test_forest_oob_error(ForestEstimator, X, y, params, err_msg):
+def test_forest_boostrap_false_oob_error(ForestEstimator):
+    X = iris.data
+    y = iris.target
+    params = {"oob_score": True, "bootstrap": False}
+    err_msg = "Out of bag estimation only available if bootstrap=True"
     estimator = ForestEstimator(**params)
     with pytest.raises(ValueError, match=err_msg):
         estimator.fit(X, y)
 
 
+@pytest.mark.parametrize("ForestClassifier", FOREST_CLASSIFIERS.values())
+def test_forest_moutput_int_oob_error(ForestClassifier):
+    X = iris.data
+    y = rng.randint(low=0, high=5, size=(iris.data.shape[0], 2))
+    params = {"oob_score": True, "bootstrap": True}
+    err_msg = "The type of target cannot be used to compute OOB estimates"
+    estimator = ForestClassifier(**params)
+    with pytest.raises(ValueError, match=err_msg):
+        estimator.fit(X, y)
+
+
+@pytest.mark.parametrize("ForestRegressor", FOREST_REGRESSORS.values())
+def test_forest_moutput_int_oob(ForestRegressor):
+    X = iris.data
+    y = rng.randint(low=0, high=5, size=(iris.data.shape[0], 2))
+    params = {"oob_score": True, "bootstrap": True}
path: sklearn/ensemble/tests/test_forest.py
text: @user1:

```suggestion
```

diff_hunk: @@ -598,29 +598,36 @@ def test_forest_oob_warning(ForestEstimator):
 
 
 @pytest.mark.parametrize("ForestEstimator", FOREST_CLASSIFIERS_REGRESSORS.values())
-@pytest.mark.parametrize(
-    "X, y, params, err_msg",
-    [
-        (
-            iris.data,
-            iris.target,
-            {"oob_score": True, "bootstrap": False},
-            "Out of bag estimation only available if bootstrap=True",
-        ),
-        (
-            iris.data,
-            rng.randint(low=0, high=5, size=(iris.data.shape[0], 2)),
-            {"oob_score": True, "bootstrap": True},
-            "The type of target cannot be used to compute OOB estimates",
-        ),
-    ],
-)
-def test_forest_oob_error(ForestEstimator, X, y, params, err_msg):
+def test_forest_boostrap_false_oob_error(ForestEstimator):
+    X = iris.data
+    y = iris.target
+    params = {"oob_score": True, "bootstrap": False}
+    err_msg = "Out of bag estimation only available if bootstrap=True"
     estimator = ForestEstimator(**params)
     with pytest.raises(ValueError, match=err_msg):
         estimator.fit(X, y)
 
 
+@pytest.mark.parametrize("ForestClassifier", FOREST_CLASSIFIERS.values())
+def test_forest_moutput_int_oob_error(ForestClassifier):
+    X = iris.data
+    y = rng.randint(low=0, high=5, size=(iris.data.shape[0], 2))
+    params = {"oob_score": True, "bootstrap": True}
+    err_msg = "The type of target cannot be used to compute OOB estimates"
+    estimator = ForestClassifier(**params)
+    with pytest.raises(ValueError, match=err_msg):
+        estimator.fit(X, y)
+
+
+@pytest.mark.parametrize("ForestRegressor", FOREST_REGRESSORS.values())
+def test_forest_moutput_int_oob(ForestRegressor):
+    X = iris.data
+    y = rng.randint(low=0, high=5, size=(iris.data.shape[0], 2))
+    params = {"oob_score": True, "bootstrap": True}
+    estimator = ForestRegressor(**params)
+    estimator.fit(X, y)
path: sklearn/ensemble/tests/test_forest.py
text: @user1:
Here, we need to assert the output. The best would be to check that the OOB score corresponds really to R2 of the samples not used (out-of-bag) samples. Or we can reuse something that we check in the OOB test for regressor regarding the score.

diff_hunk: @@ -598,29 +598,36 @@ def test_forest_oob_warning(ForestEstimator):
 
 
 @pytest.mark.parametrize("ForestEstimator", FOREST_CLASSIFIERS_REGRESSORS.values())
-@pytest.mark.parametrize(
-    "X, y, params, err_msg",
-    [
-        (
-            iris.data,
-            iris.target,
-            {"oob_score": True, "bootstrap": False},
-            "Out of bag estimation only available if bootstrap=True",
-        ),
-        (
-            iris.data,
-            rng.randint(low=0, high=5, size=(iris.data.shape[0], 2)),
-            {"oob_score": True, "bootstrap": True},
-            "The type of target cannot be used to compute OOB estimates",
-        ),
-    ],
-)
-def test_forest_oob_error(ForestEstimator, X, y, params, err_msg):
+def test_forest_boostrap_false_oob_error(ForestEstimator):
+    X = iris.data
+    y = iris.target
+    params = {"oob_score": True, "bootstrap": False}
+    err_msg = "Out of bag estimation only available if bootstrap=True"
     estimator = ForestEstimator(**params)
     with pytest.raises(ValueError, match=err_msg):
         estimator.fit(X, y)
 
 
+@pytest.mark.parametrize("ForestClassifier", FOREST_CLASSIFIERS.values())
+def test_forest_moutput_int_oob_error(ForestClassifier):
+    X = iris.data
+    y = rng.randint(low=0, high=5, size=(iris.data.shape[0], 2))
+    params = {"oob_score": True, "bootstrap": True}
+    err_msg = "The type of target cannot be used to compute OOB estimates"
+    estimator = ForestClassifier(**params)
+    with pytest.raises(ValueError, match=err_msg):
+        estimator.fit(X, y)
+
+
+@pytest.mark.parametrize("ForestRegressor", FOREST_REGRESSORS.values())
+def test_forest_moutput_int_oob(ForestRegressor):
+    X = iris.data
+    y = rng.randint(low=0, high=5, size=(iris.data.shape[0], 2))
+    params = {"oob_score": True, "bootstrap": True}
+    estimator = ForestRegressor(**params)
path: sklearn/ensemble/tests/test_forest.py
text: @user1:

```suggestion
    estimator = ForestRegressor(oob_score=True, bootstrap=True)
```

diff_hunk: @@ -598,29 +598,36 @@ def test_forest_oob_warning(ForestEstimator):
 
 
 @pytest.mark.parametrize("ForestEstimator", FOREST_CLASSIFIERS_REGRESSORS.values())
-@pytest.mark.parametrize(
-    "X, y, params, err_msg",
-    [
-        (
-            iris.data,
-            iris.target,
-            {"oob_score": True, "bootstrap": False},
-            "Out of bag estimation only available if bootstrap=True",
-        ),
-        (
-            iris.data,
-            rng.randint(low=0, high=5, size=(iris.data.shape[0], 2)),
-            {"oob_score": True, "bootstrap": True},
-            "The type of target cannot be used to compute OOB estimates",
-        ),
-    ],
-)
-def test_forest_oob_error(ForestEstimator, X, y, params, err_msg):
+def test_forest_boostrap_false_oob_error(ForestEstimator):
+    X = iris.data
+    y = iris.target
+    params = {"oob_score": True, "bootstrap": False}
+    err_msg = "Out of bag estimation only available if bootstrap=True"
     estimator = ForestEstimator(**params)
     with pytest.raises(ValueError, match=err_msg):
         estimator.fit(X, y)
 
 
+@pytest.mark.parametrize("ForestClassifier", FOREST_CLASSIFIERS.values())
+def test_forest_moutput_int_oob_error(ForestClassifier):
path: sklearn/ensemble/tests/test_forest.py
text: @user1:

```suggestion
def test_classifier_error_oob_score_multiclass_multioutput(ForestClassifier):
    """Check that we raise an error with when requesting OOB score with
    multiclass-multioutput classification target.
    """
```

diff_hunk: @@ -598,29 +598,36 @@ def test_forest_oob_warning(ForestEstimator):
 
 
 @pytest.mark.parametrize("ForestEstimator", FOREST_CLASSIFIERS_REGRESSORS.values())
-@pytest.mark.parametrize(
-    "X, y, params, err_msg",
-    [
-        (
-            iris.data,
-            iris.target,
-            {"oob_score": True, "bootstrap": False},
-            "Out of bag estimation only available if bootstrap=True",
-        ),
-        (
-            iris.data,
-            rng.randint(low=0, high=5, size=(iris.data.shape[0], 2)),
-            {"oob_score": True, "bootstrap": True},
-            "The type of target cannot be used to compute OOB estimates",
-        ),
-    ],
-)
-def test_forest_oob_error(ForestEstimator, X, y, params, err_msg):
+def test_forest_boostrap_false_oob_error(ForestEstimator):
path: sklearn/ensemble/tests/test_forest.py
text: @user1:

```suggestion
def test_forest_oob_score_requires_bootstrap(ForestEstimator):
    """Check that we raise an error if OOB score is requested without
    activating bootstrapping.
    """
```