Repository: scikit-learn/scikit-learn
Pull request: FIX MultiOutputRegressor correctly ducktypes fitted estimators
<!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
Fixes https://github.com/scikit-learn/scikit-learn/issues/16549


#### What does this implement/fix? Explain your changes.
Checks the fitted `multioutput.estimators_` for `predict`.


<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->


Problem statement:
Issue with predict function of MultiOutputRegressor when used with StackingRegressor
<!--
Before submitting a bug, please make sure the issue hasn't been already
addressed by searching through the past issues.
-->

#### Describe the bug
An error is raised when I was trying to use MultiOutputRegressor along with StackingRegressor.

```pytb
ValueError                                Traceback (most recent call last)
<ipython-input-43-e4eb1ec64c28> in <module>
----> 1 reg.predict(X_train_norm)

~\Anaconda3\lib\site-packages\sklearn\multioutput.py in predict(self, X)
    188         check_is_fitted(self)
    189         if not hasattr(self.estimator, "predict"):
--> 190             raise ValueError("The base estimator should implement"
    191                              " a predict method")
    192 

ValueError: The base estimator should implement a predict method

-->
```
#### Steps/Code to Reproduce


Example:
```python
reg=MultiOutputRegressor(StackingRegressor
            (estimators=estimators,final_estimator=ExtraTreesRegressor(n_estimators=100)))
reg.fit(X_train_norm, y_train)
reg.predict(X_train_norm)
```

#### Expected Results
The expected result should be an array of predicted values.

#### Actual Results

```pytb
raise ValueError("The base estimator should implement"
    191                              " a predict method")
```

#### Versions
For scikit-learn >= 0.22.1
<!--
Please run the following snippet and paste the output below.
For scikit-learn >= 0.20:
import sklearn; sklearn.show_versions()
For scikit-learn < 0.20:
import platform; print(platform.platform())
import sys; print("Python", sys.version)
import numpy; print("NumPy", numpy.__version__)
import scipy; print("SciPy", scipy.__version__)
import sklearn; print("Scikit-Learn", sklearn.__version__)
import imblearn; print("Imbalanced-Learn", imblearn.__version__)
-->


<!-- Thanks for contributing! -->

Hints from issue discussion:
Please provide a complete, runnable code snippet that reproduces the error.
A reproducible example:

```python
from sklearn.datasets import load_linnerud
from sklearn.ensemble import ExtraTreesRegressor
from sklearn.ensemble import StackingRegressor
from sklearn.multioutput import MultiOutputRegressor

X, y = load_linnerud(return_X_y=True)
stacker = StackingRegressor(
    estimators=[("xx", ExtraTreesRegressor()), ("yy", ExtraTreesRegressor())],
    final_estimator=ExtraTreesRegressor()
)
reg = MultiOutputRegressor(
    estimator=stacker
)
reg.fit(X, y)
reg.predict(X)
```
The issue is because we delegate `predict` to `final_estimator_` which is not created before calling `fit` on the stacking. And the multioutput regressor call `hasattr` on `estimator` (which is always unfitted). Several solutions:

* remove `if_delegate_has_method` and `predict` will always exist;
* ducktype one of the `self.estimators_` instead of `self.estimator` in multioutput regresor.

I don't know what is best.
> ducktype one of the self.estimators_ instead of self.estimator in multioutput regresor.

This seems the right thing to do given that the MultiOutputRegressor must have been fitted when its predict is called.
Hi everyone, I have the same issue here with the following estimators : 

```
models = [
            ("u_dt", DecisionTreeRegressor()),
            ("t_lsvr", SVR(kernel="linear",
                    C= 0.008,
                    epsilon=0.25, 
                    shrinking=False,
                    tol= 5,
                    )),
            ("u_ridge", Ridge()),
            ("u_elastic", ElasticNet()),
            ("t_rbfsvr", SVR(kernel="rbf",
                    C= 3.7,
                    epsilon=0.06, 
                    gamma="scale", 
                    shrinking=False, 
                    tol= 0.6,
                    )),
        ]

        model = MultiOutputRegressor(
            StackingRegressor(
            estimators=models, 
            cv=5,
            n_jobs=-1,
            passthrough=True,
            verbose=0,
            ), n_jobs=None)

model.fit(X_train, y_train)
```

Do someone have a clue of how to fix it?
Thanks in advance.

EDIT : typo
EDIT 2 : I found this StackOverflow link : https://stackoverflow.com/questions/60378363/multioutput-stacking-regressor that proposed a workaround. By adding : 

```
model.estimator.estimators_ = model.estimator.estimators
model.estimator.final_estimator_ = model.estimator.final_estimator
model.estimator.stack_method_ = model.estimator.stack_method
```

right before the `fit` call, I ended up with another error that I couldn't solve : 

```
'tuple' object has no attribute 'p'
```

The only tuples I can think of are the tuples `("name_estimator", estimator)` in my `estimators` list. I'll continue investigating and I let you know.
Set **stack_method='predict'** in model initialization like so (dont know why the default 'auto' doesnt work):
```python
model = StackingClassifier(estimators=level0, final_estimator=level1, stack_method='predict', cv=5)
```

Commit message: Merge remote-tracking branch 'upstream/main' into multiouputregressor_predict

Patch to review:
diff --git a/doc/whats_new/v0.24.rst b/doc/whats_new/v0.24.rst
index f549b31f51aa7..20ccbef056edc 100644
--- a/doc/whats_new/v0.24.rst
+++ b/doc/whats_new/v0.24.rst
@@ -20,6 +20,13 @@ Changelog
   `'use_encoded_value'` strategies.
   :pr:`19234` by `Guillaume Lemaitre <glemaitre>`.
 
+:mod:`sklearn.multioutput`
+..........................
+
+- |Fix| :class:`multioutput.MultiOutputRegressor` now works with estimators
+  that dynamically define `predict` during fitted, such as
+  :class:`ensemble.StackingRegressor`. :pr:`19308` by `Thomas Fan`_.
+
 .. _changes_0_24_1:
 
 Version 0.24.1
diff --git a/sklearn/multioutput.py b/sklearn/multioutput.py
index 9987c01b13187..4cb01c524d59d 100644
--- a/sklearn/multioutput.py
+++ b/sklearn/multioutput.py
@@ -198,7 +198,7 @@ def predict(self, X):
             Note: Separate models are generated for each predictor.
         """
         check_is_fitted(self)
-        if not hasattr(self.estimator, "predict"):
+        if not hasattr(self.estimators_[0], "predict"):
             raise ValueError("The base estimator should implement"
                              " a predict method")
 
diff --git a/sklearn/tests/test_multioutput.py b/sklearn/tests/test_multioutput.py
index edfcdef1bf89c..294cc8c7a8ee8 100644
--- a/sklearn/tests/test_multioutput.py
+++ b/sklearn/tests/test_multioutput.py
@@ -13,6 +13,7 @@
 from sklearn import datasets
 from sklearn.base import clone
 from sklearn.datasets import make_classification
+from sklearn.datasets import load_linnerud
 from sklearn.ensemble import GradientBoostingRegressor, RandomForestClassifier
 from sklearn.exceptions import NotFittedError
 from sklearn.linear_model import Lasso
@@ -33,6 +34,7 @@
 from sklearn.dummy import DummyRegressor, DummyClassifier
 from sklearn.pipeline import make_pipeline
 from sklearn.impute import SimpleImputer
+from sklearn.ensemble import StackingRegressor
 
 
 def test_multi_target_regression():
@@ -648,3 +650,19 @@ def test_classifier_chain_tuple_invalid_order():
 
     with pytest.raises(ValueError, match='invalid order'):
         chain.fit(X, y)
+
+
+def test_multioutputregressor_ducktypes_fitted_estimator():
+    """Test that MultiOutputRegressor checks the fitted estimator for
+    predict. Non-regression test for #16549."""
+    X, y = load_linnerud(return_X_y=True)
+    stacker = StackingRegressor(
+        estimators=[("sgd", SGDRegressor(random_state=1))],
+        final_estimator=Ridge(),
+        cv=2
+    )
+
+    reg = MultiOutputRegressor(estimator=stacker).fit(X, y)
+
+    # Does not raise
+    reg.predict(X)


Code review comments:
diff_hunk: @@ -20,6 +20,13 @@ Changelog
   `'use_encoded_value'` strategies.
   :pr:`19234` by `Guillaume Lemaitre <glemaitre>`.
 
+:mod:`sklearn.multioutput`
+..........................
+
+- |Fix| :class:`multioutput.MultiOutputRegressor` now works with estimators
+  that dynamically define `predict` during fitted, such as
path: doc/whats_new/v0.24.rst
text: @user1:
```suggestion
  that dynamically define `predict` during fitting, such as
```