Repository: sktime/sktime

Issue:
[BUG] `model_evaluation.evaluate` fails for `statsmodels` forecasters with exogenous features
**Describe the bug**

model_evaluation.evaluate creates X_test containing both train and test samples.  For statsmodels models with exogenous features X_test length must equal the forecast horizon and therefore this currently throws the error:

`ValueError: Provided exogenous values are not of the appropriate shape.`

This bug occurs for UnobservedComponents and SARIMAX.

The root cause of the bug is in the `_split` function:
https://github.com/sktime/sktime/blob/a5f824a5fff1c3fc1babd3f34b94c5055ec040b0/sktime/forecasting/model_evaluation/_functions.py#L60

This could be modified to 'X.iloc[test].sort_index()' for statsmodels models with exogenous features or alternatively by filtering X here:
https://github.com/sktime/sktime/blob/a5f824a5fff1c3fc1babd3f34b94c5055ec040b0/sktime/forecasting/base/adapters/_statsmodels.py#L104

**To Reproduce**

```python
from sktime.datasets import load_longley
from sktime.forecasting.structural import UnobservedComponents
from sktime.forecasting.model_evaluation import evaluate
from sktime.forecasting.model_selection import SlidingWindowSplitter

y, X = load_longley()

forecaster = UnobservedComponents(level='local linear trend')

cv = SlidingWindowSplitter(fh=[1,2,3],
                           window_length = 4,
                           step_length = 1)

results = evaluate(
        forecaster=forecaster, y=y, X = X,
        cv=cv, strategy='refit', return_data=True,
        error_score = 'raise'
        )
```

**Versions**

System:
python: 3.8.12 (default, Nov 20 2022, 10:50:06)  [Clang 14.0.0 (clang-1400.0.29.202)]
machine: macOS-13.0.1-arm64-arm-64bit

Python dependencies:
          pip: 21.2
   setuptools: 63.1.0
      sklearn: 1.1.3
       sktime: 0.14.0
  statsmodels: 0.13.5
        numpy: 1.22.0
        scipy: 1.9.3
       pandas: 1.4.1
   matplotlib: 3.6.2
       joblib: 1.2.0
        numba: 0.56.4
     pmdarima: None
      tsfresh: None


Discussion:
Thank you for raising this issue!  

I think the original reason that we have  `X.iloc[test_plus_train].sort_index()` is due to certain transformers which effectively trim the input for example `Differencer`. It is however not the most elegant solution.

In the short term I am leaning towards your suggestion solution of filtering X within the `statsmodel` wrapper class (as this won't cause the other issue again, but should fix yours.  

However in the long term I think that this is connected to another issue/conversation on how to solve this underlying issue #2880  

Would you be interested in making the PR to solve this issue?  I think you did a good job identifying the likely cause.   If not we can work together to resolve it.
Thanks for your response.  

I will happily make the PR to solve the issue by modifying the `statsmodels` wrapper class.  Would you mind if I add you as a reviewer on the PR?
@adoherty21  - certainly!  I would be happy to take a look at it! 

Resolving patch:
diff --git a/sktime/forecasting/base/adapters/_statsmodels.py b/sktime/forecasting/base/adapters/_statsmodels.py
--- a/sktime/forecasting/base/adapters/_statsmodels.py
+++ b/sktime/forecasting/base/adapters/_statsmodels.py
@@ -99,6 +99,16 @@ def _predict(self, fh, X=None):
         # statsmodels requires zero-based indexing starting at the
         # beginning of the training series when passing integers
         start, end = fh.to_absolute_int(self._y.index[0], self.cutoff)[[0, -1]]
+        fh_abs = fh.to_absolute(self.cutoff).to_pandas()
+
+        # bug fix for evaluate function as test_plus_train indices are passed
+        # statsmodels exog must contain test indices only.
+        # For discussion see https://github.com/sktime/sktime/issues/3830
+        if X is not None:
+            ind_drop = self._X.index
+            X = X.loc[~X.index.isin(ind_drop)]
+            # Entire range of the forecast horizon is required
+            X = X[: fh_abs[-1]]
 
         if "exog" in inspect.signature(self._forecaster.__init__).parameters.keys():
             y_pred = self._fitted_forecaster.predict(start=start, end=end, exog=X)
@@ -107,7 +117,7 @@ def _predict(self, fh, X=None):
 
         # statsmodels forecasts all periods from start to end of forecasting
         # horizon, but only return given time points in forecasting horizon
-        y_pred = y_pred.loc[fh.to_absolute(self.cutoff).to_pandas()]
+        y_pred = y_pred.loc[fh_abs]
         # ensure that name is not added nor removed
         # otherwise this may upset conversion to pd.DataFrame
         y_pred.name = self._y.name


Test patch:
diff --git a/sktime/forecasting/tests/test_structural.py b/sktime/forecasting/tests/test_structural.py
--- a/sktime/forecasting/tests/test_structural.py
+++ b/sktime/forecasting/tests/test_structural.py
@@ -7,7 +7,9 @@
 import pytest
 from pandas.testing import assert_series_equal
 
-from sktime.datasets import load_airline
+from sktime.datasets import load_airline, load_longley
+from sktime.forecasting.model_evaluation import evaluate
+from sktime.forecasting.model_selection import SlidingWindowSplitter
 from sktime.forecasting.structural import UnobservedComponents
 from sktime.utils.validation._dependencies import _check_soft_dependencies
 
@@ -304,3 +306,20 @@ def test_prediction_intervals_exog(alpha, coverage, level_sample_data_split):
         intervals_np = intervals_df.to_numpy().flatten()
         assert intervals_df.shape == (fh_length, 2)
         assert intervals_np[0] < intervals_np[1]
+
+
+def test_evaluate_exog():
+    """Test evaluate works when exogenous regressors are present."""
+    y, X = load_longley()
+    forecaster = UnobservedComponents(level="local linear trend")
+    cv = SlidingWindowSplitter(fh=[1, 2, 3], window_length=4, step_length=1)
+    results = evaluate(
+        forecaster=forecaster,
+        y=y,
+        X=X,
+        cv=cv,
+        strategy="refit",
+        return_data=True,
+        error_score="raise",
+    )
+    assert results.shape == (10, 8)