Repository: alan-turing-institute/sktime (python)

Issue:
[BUG] Error in `inverse-variance` method of `AutoEnsembleForecaster`
**Describe the bug**
When fitting the `AutoEnsembleForecaster` forecaster with exogenous data using the `inverse-variance` method, an error is raised. This is because `X` is passed to the internal forecasters instead of `X_test` when a prediction is made (see [here](https://github.com/alan-turing-institute/sktime/blob/4164639e1c521b112711c045d0f7e63013c1e4eb/sktime/forecasting/compose/_ensemble.py#L190)).

**To Reproduce**

```python
from sktime.datasets import load_longley
from sktime.forecasting.arima import ARIMA
from sktime.forecasting.compose import AutoEnsembleForecaster

forecasters = [
    ("forecaster_1", ARIMA()),
    ("forecaster_2", ARIMA()),
]

y, X = load_longley()


ensemble_forecaster = AutoEnsembleForecaster(
    forecasters=forecasters, method="inverse-variance"
)
ensemble_forecaster.fit(y, X)
```

This code snippet has the following output:

```
Traceback (most recent call last):
  File "test.py", line 16, in <module>
    ensemble_forecaster.fit(y, X)
  File "/Users/anhoang/Projekte/2022/Daimler/Chronos/chronos/.venv/lib/python3.8/site-packages/sktime/forecasting/base/_base.py", line 270, in fit
    self._fit(y=y_inner, X=X_inner, fh=fh)
  File "/Users/anhoang/Projekte/2022/Daimler/Chronos/chronos/.venv/lib/python3.8/site-packages/sktime/forecasting/compose/_ensemble.py", line 190, in _fit
    for y_pred_test in self._predict_forecasters(fh_test, X)
  File "/Users/anhoang/Projekte/2022/Daimler/Chronos/chronos/.venv/lib/python3.8/site-packages/sktime/forecasting/base/_meta.py", line 72, in _predict_forecasters
    return [forecaster.predict(fh=fh, X=X) for forecaster in self.forecasters_]
  File "/Users/anhoang/Projekte/2022/Daimler/Chronos/chronos/.venv/lib/python3.8/site-packages/sktime/forecasting/base/_meta.py", line 72, in <listcomp>
    return [forecaster.predict(fh=fh, X=X) for forecaster in self.forecasters_]
  File "/Users/anhoang/Projekte/2022/Daimler/Chronos/chronos/.venv/lib/python3.8/site-packages/sktime/forecasting/base/_base.py", line 325, in predict
    y_pred = self._predict(fh=fh, X=X_inner)
  File "/Users/anhoang/Projekte/2022/Daimler/Chronos/chronos/.venv/lib/python3.8/site-packages/sktime/forecasting/base/adapters/_pmdarima.py", line 76, in _predict
    return self._predict_fixed_cutoff(fh_oos, X=X)
  File "/Users/anhoang/Projekte/2022/Daimler/Chronos/chronos/.venv/lib/python3.8/site-packages/sktime/forecasting/base/adapters/_pmdarima.py", line 178, in _predict_fixed_cutoff
    result = self._forecaster.predict(
  File "/Users/anhoang/Projekte/2022/Daimler/Chronos/chronos/.venv/lib/python3.8/site-packages/pmdarima/arima/arima.py", line 785, in predict
    raise ValueError('X array dims (n_rows) != n_periods')
ValueError: X array dims (n_rows) != n_periods
```

**Expected behavior**

The enseble forecaster should run without error and summarize the forecasters based on the inverse error variance of their predictions.

**Additional context**
<!--
Add any other context about the problem here.
-->

**Versions**
<details>

System:
    python: 3.8.9 (default, Oct 16 2021, 12:00:06)  [Clang 12.0.0 (clang-1200.0.32.29)]
executable: /Users/anhoang/Projekte/2022/Daimler/Chronos/chronos/.venv/bin/python
   machine: macOS-11.6-x86_64-i386-64bit

Python dependencies:
          pip: 22.1.2
   setuptools: 63.2.0
      sklearn: 1.1.1
       sktime: 0.13.0
  statsmodels: 0.13.2
        numpy: 1.22.4
        scipy: 1.8.1
       pandas: 1.4.3
   matplotlib: 3.5.2
       joblib: 1.1.0
        numba: 0.56.0
     pmdarima: 1.8.5
      tsfresh: 0.19.0
</details>

<!-- Thanks for contributing! -->


Pull request:
 [BUG] Fix `AutoEnsembleForecaster` inverse variance bug
<!--
Thanks for contributing a pull request! Please ensure you have taken a look
at our contribution guide: https://github.com/alan-turing-institute/sktime/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
Fixes #3199
<!--
Example: Fixes #1234. See also #3456.

Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->


#### What does this implement/fix? Explain your changes.
Fix the error described in #3199 that occurred in `AutoEnsembleForecaster` when `method="inverse-variance"` was used. This was caused by passing `X` instead of `X_test` to the internal forecasters when making predictions in `fit` method. This has been fixed. In addition, tests for the `AutoEnsembleForecaster` have been added.
<!--
A clear and concise description of what you have implemented.
-->

#### Does your contribution introduce a new dependency? If yes, which one?
No

<!--
If your contribution does add a new hard dependency, we may suggest to initially develop your contribution in a separate companion package in https://github.com/sktime/ to keep external dependencies of the core sktime package to a minimum.
-->

#### What should a reviewer concentrate their feedback on?
<!-- This section is particularly useful if you have a pull request that is still in development. You can guide the reviews to focus on the parts that are ready for their comments. We suggest using bullets (indicated by * or -) and filled checkboxes [x] here -->

#### Any other comments?
No
<!--
Please be aware that we are a loose team of volunteers so patience is necessary; assistance handling other issues is very welcome. We value all user contributions, no matter how minor they are. If we are slow to review, either the pull request needs some benchmarking, tinkering, convincing, etc. or more likely the reviewers are simply busy. In either case, we ask for your understanding during the review process.
-->

#### PR checklist
<!--
Please go through the checklist below. Please feel free to remove points if they are not applicable.
-->

##### For all contributions
- [x] I've added myself to the [list of contributors](https://github.com/alan-turing-institute/sktime/blob/main/.all-contributorsrc).
- [x] Optionally, I've updated sktime's [CODEOWNERS](https://github.com/alan-turing-institute/sktime/blob/main/CODEOWNERS) to receive notifications about future changes to these files.
- [x] I've added unit tests and made sure they pass locally.
- [x] The PR title starts with either [ENH], [MNT], [DOC], or [BUG] indicating whether the PR topic is related to enhancement, maintenance, documentation, or bug.

##### For new estimators
- [x] I've added the estimator to the online documentation.
- [x] I've updated the existing example notebooks or provided a new one to showcase how my estimator works.


<!--
Thanks for contributing!
-->


Patch:
diff --git a/.all-contributorsrc b/.all-contributorsrc
index fd30624a6..8aca5c8e5 100644
--- a/.all-contributorsrc
+++ b/.all-contributorsrc
@@ -1739,6 +1739,14 @@
       "contributions": [
         "code",
       ]
+    },
+    {
+      "login": "AnH0ang",
+      "name": "An Hoang",
+      "profile": "https://github.com/AnH0ang",
+      "contributions": [
+        "bug"
+      ]
     }
   ],
   "projectName": "sktime",
diff --git a/sktime/forecasting/compose/_ensemble.py b/sktime/forecasting/compose/_ensemble.py
index 58ef6d045..2fe9c39ce 100644
--- a/sktime/forecasting/compose/_ensemble.py
+++ b/sktime/forecasting/compose/_ensemble.py
@@ -187,7 +187,7 @@ class AutoEnsembleForecaster(_HeterogenousEnsembleForecaster):
             inv_var = np.array(
                 [
                     1 / np.var(y_test - y_pred_test)
-                    for y_pred_test in self._predict_forecasters(fh_test, X)
+                    for y_pred_test in self._predict_forecasters(fh_test, X_test)
                 ]
             )
             # standardize the inverse variance


Test patch:
diff --git a/sktime/forecasting/compose/tests/test_autoensemble.py b/sktime/forecasting/compose/tests/test_autoensemble.py
new file mode 100644
index 000000000..65f39ef90
--- /dev/null
+++ b/sktime/forecasting/compose/tests/test_autoensemble.py
@@ -0,0 +1,65 @@
+#!/usr/bin/env python3 -u
+# -*- coding: utf-8 -*-
+# copyright: sktime developers, BSD-3-Clause License (see LICENSE file).
+"""Unit tests of AutoEnsembleForecaster functionality."""
+
+__author__ = ["mloning", "GuzalBulatova", "aiwalter", "RNKuhns", "AnH0ang"]
+
+import pandas as pd
+import pytest
+from sklearn.linear_model import LinearRegression
+from sklearn.tree import DecisionTreeRegressor
+
+from sktime.datasets import load_longley
+from sktime.forecasting.base import ForecastingHorizon
+from sktime.forecasting.compose import (
+    AutoEnsembleForecaster,
+    RecursiveTabularRegressionForecaster,
+)
+from sktime.forecasting.model_selection import temporal_train_test_split
+
+
+@pytest.mark.parametrize(
+    "forecasters",
+    [
+        [
+            (
+                "dt",
+                RecursiveTabularRegressionForecaster(
+                    DecisionTreeRegressor(random_state=42), window_length=3
+                ),
+            ),
+            (
+                "lr",
+                RecursiveTabularRegressionForecaster(
+                    LinearRegression(), window_length=3
+                ),
+            ),
+        ],
+    ],
+)
+@pytest.mark.parametrize(
+    "method",
+    ["inverse-variance", "feature-importance"],
+)
+def test_autoensembler(forecasters, method):
+    """Check that the prediction is a weighted mean of the individual predictions."""
+    y, X = load_longley()
+    y_train, y_test, X_train, X_test = temporal_train_test_split(y, X)
+
+    fh_test = ForecastingHorizon(y_test.index, is_relative=False)
+
+    ensemble_forecaster = AutoEnsembleForecaster(forecasters=forecasters, method=method)
+    ensemble_forecaster.fit(y_train, X_train)
+    y_pred = ensemble_forecaster.predict(fh=fh_test, X=X_test)
+
+    predictions = []
+    for _, forecaster in forecasters:
+        f = forecaster
+        f.fit(y_train, X_train)
+        f_pred = f.predict(fh=fh_test, X=X_test)
+        predictions.append(f_pred)
+    predictions = pd.DataFrame(predictions).T
+
+    assert (predictions.min(axis=1) <= y_pred).all()
+    assert (predictions.max(axis=1) >= y_pred).all()