Repository: sktime/sktime
Pull request: Add inv_var method into AutoEnsembleForecaster
<!--
Thanks for contributing a pull request! Please ensure you have taken a look
at our contribution guide: https://github.com/alan-turing-institute/sktime/blob/main/CONTRIBUTING.md
-->


#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.

Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
See #1404

#### What does this implement/fix? Explain your changes.
<!--
A clear and concise description of what you have implemented.
-->

Implement `inv_var` method in `AutoEnsembleForecaster` that uses the inverse of variance of in-sample-fit errors as weights.  


#### What should a reviewer concentrate their feedback on?

There are some points that need to be discussed further:

* if `regressor` and `method` both `None`, which is the default. Now the default is `GradientBoostingRegressor`. 
* If they are provided both, raise a `ValueError`

#### Any other comments?
<!--
Please be aware that we are a loose team of volunteers so patience is necessary; assistance handling other issues is very welcome. We value all user contributions, no matter how minor they are. If we are slow to review, either the pull request needs some benchmarking, tinkering, convincing, etc. or more likely the reviewers are simply busy. In either case, we ask for your understanding during the review process.
-->

#### PR checklist
<!--
Please go through the checklist below. Please feel free to remove points if they are not applicable.
-->

##### For all contributions
- [ ] I've added myself to the [list of contributors](https://github.com/alan-turing-institute/sktime/blob/main/.all-contributorsrc).
- [ ] Optionally, I've updated sktime's [CODEOWNERS](https://github.com/alan-turing-institute/sktime/blob/main/CODEOWNERS) to receive notifications about future changes to these files.
- [ ] I've added unit tests and made sure they pass locally.
- [ ] 

<!--
Thanks for contributing!
-->


Problem statement:
[ENH] AutoEnsembleForeacaster should have a method to use inverse error as weights
**Describe the solution you'd like**
`AutoEnsembleForeacaster` should be able to automatically calculate internally the error of the forecasters and use the inverse error as weights. Credits to @RNKuhns who came up with this idea :)

There should be a `method="inverse-error"` added.

Hints from issue discussion:
@aiwalter are you working on this? If not, I can help. While I suggest  to add this `inverse-error` method into `EnsembleForecaster` instead of `AutoEnsembleForecaster`, such as `EnsembleForecaster(forecasters, weights='inverse-error')`.  If added into `AutoEnsembleForecaster`, `method` and `regressor` may be confused.
Hi @AngelPone we missed you :) especially for the HTS topic, but glad that you are back again :) I would put it into `AutoEnsembleForecaster`, we might have more such weight selection algos implemented there. Then we would have the regressor method and the inverse-error one. The `EnsembleForecaster` is for me more a vehicle for users to give custom weights. Also `weights='inverse-error'` would not fit the the type, currently `weights` is a list
@aiwalter agree. I will work on this.

Commit message: refine docstring

Patch to review:
diff --git a/sktime/forecasting/compose/_ensemble.py b/sktime/forecasting/compose/_ensemble.py
index d5ee0b2509d..a80b9dba9c3 100644
--- a/sktime/forecasting/compose/_ensemble.py
+++ b/sktime/forecasting/compose/_ensemble.py
@@ -39,15 +39,23 @@
 class AutoEnsembleForecaster(_HeterogenousEnsembleForecaster):
     """Automatically find best weights for the ensembled forecasters.
 
-    The AutoEnsembleForecaster uses a meta-model (regressor) to calculate the optimal
-    weights for ensemble aggregation with mean. The regressor has to be sklearn-like
-    and needs to have either an attribute `feature_importances_` or `coef_`, as this
-    is used as weights. Regressor can also be a sklearn.Pipeline.
+    The AutoEnsembleForecaster finds optimal weights for the ensembled forecasters
+    using given method or a meta-model (regressor) .
+    The regressor has to be sklearn-like and needs to have either an attribute
+    `feature_importances_` or `coef_`, as this is used as weights.
+    Regressor can also be a sklearn.Pipeline.
 
     Parameters
     ----------
     forecasters : list of (str, estimator) tuples
         Estimators to apply to the input series.
+    method : str, optional, default="feature-importance"
+        Strategy used to compute weights. Available choices: `feature-importance`,
+        `inverse-variance`.
+        `feature-importance`: use the `feature_importances_` or `coef_` as optimal
+        weights.
+        `inverse-variance`: use the inverse of forecasting error variance to compute
+        optimal weights, the passed regressor will be omitted.
     regressor : sklearn-like regressor, optional, default=None.
         Used to infer optimal weights from coefficients (linear models) or from
         feature importance scores (decision tree-based models). If None, then
@@ -105,6 +113,7 @@ class AutoEnsembleForecaster(_HeterogenousEnsembleForecaster):
     def __init__(
         self,
         forecasters,
+        method="feature-importance",
         regressor=None,
         test_size=None,
         random_state=None,
@@ -114,6 +123,7 @@ def __init__(
             forecasters=forecasters,
             n_jobs=n_jobs,
         )
+        self.method = method
         self.regressor = regressor
         self.test_size = test_size
         self.random_state = random_state
@@ -135,9 +145,6 @@ def _fit(self, y, X=None, fh=None):
         self : returns an instance of self.
         """
         _, forecasters = self._check_forecasters()
-        self.regressor_ = check_regressor(
-            regressor=self.regressor, random_state=self.random_state
-        )
 
         # get training data for meta-model
         if X is not None:
@@ -149,24 +156,46 @@ def _fit(self, y, X=None, fh=None):
             X_train, X_test = None, None
 
         # fit ensemble models
-        fh_regressor = ForecastingHorizon(y_test.index, is_relative=False)
-        self._fit_forecasters(forecasters, y_train, X_train, fh_regressor)
-        X_meta = pd.concat(self._predict_forecasters(fh_regressor, X_test), axis=1)
-
-        # fit meta-model (regressor) on predictions of ensemble models
-        # with y_test as endog/target
-        self.regressor_.fit(X=X_meta, y=y_test)
-
-        # check if regressor is a sklearn.Pipeline
-        if isinstance(self.regressor_, Pipeline):
-            # extract regressor from pipeline to access its attributes
-            self.weights_ = _get_weights(self.regressor_.steps[-1][1])
+        fh_test = ForecastingHorizon(y_test.index, is_relative=False)
+        self._fit_forecasters(forecasters, y_train, X_train, fh_test)
+
+        if self.method == "feature-importance":
+
+            self.regressor_ = check_regressor(
+                regressor=self.regressor, random_state=self.random_state
+            )
+            X_meta = pd.concat(self._predict_forecasters(fh_test, X_test), axis=1)
+
+            # fit meta-model (regressor) on predictions of ensemble models
+            # with y_test as endog/target
+            self.regressor_.fit(X=X_meta, y=y_test)
+
+            # check if regressor is a sklearn.Pipeline
+            if isinstance(self.regressor_, Pipeline):
+                # extract regressor from pipeline to access its attributes
+                self.weights_ = _get_weights(self.regressor_.steps[-1][1])
+            else:
+                self.weights_ = _get_weights(self.regressor_)
+
+        elif self.method == "inverse-variance":
+            # get in-sample forecasts
+            if self.regressor is not None:
+                Warning(f"regressor will not be used because ${self.method} is set.")
+            inv_var = np.array(
+                [
+                    1 / np.var(y_test - y_pred_test)
+                    for y_pred_test in self._predict_forecasters(fh_test, X)
+                ]
+            )
+            # standardize the inverse variance
+            self.weights_ = list(inv_var / np.sum(inv_var))
         else:
-            self.weights_ = _get_weights(self.regressor_)
+            raise NotImplementedError(
+                f"Given method {self.method} does not exist, "
+                f"please provide valid method parameter."
+            )
 
-        # fit forecasters with all data
         self._fit_forecasters(forecasters, y, X, fh)
-
         return self
 
     def _predict(self, fh, X=None, return_pred_int=False, alpha=DEFAULT_ALPHA):


Code review comments:
diff_hunk: @@ -39,15 +39,23 @@
 class AutoEnsembleForecaster(_HeterogenousEnsembleForecaster):
     """Automatically find best weights for the ensembled forecasters.
 
-    The AutoEnsembleForecaster uses a meta-model (regressor) to calculate the optimal
-    weights for ensemble aggregation with mean. The regressor has to be sklearn-like
-    and needs to have either an attribute `feature_importances_` or `coef_`, as this
-    is used as weights. Regressor can also be a sklearn.Pipeline.
+    The AutoEnsembleForecaster finds optimal weights for the ensembled forecasters
+    using given method or a meta-model (regressor) .
+    The regressor has to be sklearn-like and needs to have either an attribute
+    `feature_importances_` or `coef_`, as this is used as weights.
+    Regressor can also be a sklearn.Pipeline.
 
     Parameters
     ----------
     forecasters : list of (str, estimator) tuples
         Estimators to apply to the input series.
+    method : str, optional, default="feature-importance"
+        Strategy used to compute weights. Available choices: `feature-importance`,
+        `inverse-variance`.
+        `feature-importance`: use the `feature_importances_` or `coef_` as optimal
+        weights.
+        `inverse-variance`: use the inverse of forecasting error variance to compute
+        optimal weights, the passed regressor will be omitted.
path: sktime/forecasting/compose/_ensemble.py
text: @user1:
```suggestion
    method : str, optional, default="feature-importance"
        Strategy used to compute weights. Available choices:
        - ``feature-importance``: use the ``feature_importances_`` or ``coef_`` from given ``regressor`` as optimal
        weights.
        - ``inverse-variance``: use the inverse variance of the forecasting error (based on the internal train-test-split)
        to compute optimal weights, a given ``regressor`` will be omitted.
```

@user1:
just pushed this as formatting here might mess up