Repository: sktime/sktime

Issue:
[BUG] Forecasts are off when using `Differencer` within `FhPlexForecaster` 
**Describe the bug**
Forecasts are off when using `Differencer` in a pipeline nested within `FhPlexForecaster`. It seems as if the `inverse_transform` is not applied correctly.

**To Reproduce**
Include any `Differencer` transformation in a forecaster nested within `FhPlexForecaster`. The code below showcases the example (just comment the line applying the `Differencer` in the `TransformedTargetForecaster` pipe). It occurs independent of the how the `na_handling` parameter is chosen.

```python
import numpy as np
from sklearn.ensemble import RandomForestRegressor
from sktime.datasets import load_longley
from sktime.forecasting.base import ForecastingHorizon
from sktime.forecasting.compose import make_reduction, FhPlexForecaster, ForecastingPipeline, TransformedTargetForecaster
from sktime.forecasting.model_selection import ExpandingWindowSplitter
from sktime.forecasting.model_selection import ForecastingRandomizedSearchCV
from sktime.performance_metrics.forecasting import MeanSquaredError
from sklearn.preprocessing import MinMaxScaler
from sktime.transformations.series.adapt import TabularToSeriesAdaptor
from sktime.transformations.series.boxcox import LogTransformer
from sktime.transformations.series.difference import Differencer
from sktime.transformations.series.subset import IndexSubset


y, X = load_longley()
horizon = ForecastingHorizon(np.arange(1, 4), is_relative=True)

random_forest = RandomForestRegressor()
reduction_forecaster = make_reduction(random_forest, window_length=2, strategy="direct", windows_identical=False)

# Inner pipeline (transformers for X, forecaster)
forecasting_pipeline = ForecastingPipeline(
    [
        ("index_subset", IndexSubset(index_treatment="remove")),
        ("min_max_scaler", TabularToSeriesAdaptor(MinMaxScaler())),
        ("reduction_forecaster", reduction_forecaster),
    ]
)

# Outer pipeline (transformers for y, inner forecasting pipe)
transformed_target_forecaster = TransformedTargetForecaster(
    [
        ("log_transformer", LogTransformer()),
        ("first_differencer", Differencer(lags=1, na_handling="drop_na")),
        ("min_max_scaler", TabularToSeriesAdaptor(MinMaxScaler())),
        ("forecasting_pipeline", forecasting_pipeline),
    ]
)

transformed_target_forecaster.get_params()
grid = {"forecasting_pipeline__reduction_forecaster__estimator__max_features": np.linspace(0.1, 0.9, num=20)}

expanding_window_cv = ExpandingWindowSplitter(fh=horizon, step_length=1)

cv_model = ForecastingRandomizedSearchCV(
    forecaster=transformed_target_forecaster,
    # forecaster=reduction_forecaster,
    cv=expanding_window_cv,
    param_distributions=grid,
    n_iter=5,
    scoring=MeanSquaredError(),
    strategy="refit",
    refit=True,
    return_n_best_forecasters=1,
    update_behaviour="full_refit",
    error_score="raise",
    verbose=1,
)

horizon_specific_forecaster = FhPlexForecaster(forecaster=cv_model)

horizon_specific_forecaster.fit(y, X, fh=horizon)
horizon_specific_forecaster.predict(fh=horizon, X=X)
```

**Expected behavior**
A forecast with the inverse transformations applied correctly.

**Versions**
<details>
System:
    python: 3.8.10 (tags/v3.8.10:3d8993a, May  3 2021, 11:48:03) [MSC v.1928 64 bit (AMD64)]
executable: C:\Program Files\python-3.8.10\python.exe
   machine: Windows-10-10.0.19044-SP0
Python dependencies:
          pip: 21.1.1
       sktime: 0.20.1
      sklearn: 1.2.2
       skbase: 0.4.6
        numpy: 1.23.5
        scipy: 1.10.1
       pandas: 1.5.3
   matplotlib: 3.7.2
       joblib: 1.3.1
  statsmodels: 0.14.0
        numba: 0.57.1
     pmdarima: 2.0.3
      tsfresh: None
   tensorflow: None
tensorflow_probability: None
C:\Program Files\JetBrains\PyCharm 2022.1\plugins\python\helpers\pydev\_pydevd_bundle\pydevd_utils.py:606: FutureWarning: iteritems is deprecated and will be removed in a future version. Use .items instead.
  for item in s.iteritems():
</details>



Discussion:
My suspicion is, this happens whenever the `fh` is non-contiguous with the data seen, and the model/trasformer is one that assumes regular data. E.g., `fh = [2]`. Because then, the differencer will not have a value for `fh = [1]` - it needs this value to invert the differencing.

Interesting API designn question, where do we get it from? A naive way would e to "ask the forecaster", ut composition should be non-contextual, i.e., the differencer should not interact with the forecaster eyond its pulic interfaces.
I can see how this is tricky (and I guess it only makes sense if the user supplies a contiguous forecast horizon in the first place). 
> and I guess it only makes sense if the user supplies a contiguous forecast horizon in the first place

Yes, but that's not a workaround in the `FhPlexForecaster`! Because it always passes non-contiguous `fh` to the inner estimators.
I think a solution would be a parameter that controls whether internally we pass `[1]`, `[2]`, `[3]` - or `[1]`, `[1, 2]`, `[1, 2, 3]` etc and then subset the output to `[1]`, `[2]`, `[3]`.
@hliebert, since you mention, you are migrating a forecaster in your internal code base - how is this inversion step done there? What value is used for the index 1, to obtain the inverse difference (cumsum) for index 2?

I could imagine multiple things:

* the actual ground truth value (that would be leakage and make the forecaster break on unseen data)
* the forecast of the forecaster for index 1
* the forecaster of the forecaster for index 2, asked to produce a forecast for index 1 too

Another question is the "should", here.
You get option 2 (forecaster for index 1) if you have the differencer outside of the `FhPlexTransformer`.

For option 3, we´d need the extra parameter.

Resolving patch:
diff --git a/sktime/forecasting/compose/_fhplex.py b/sktime/forecasting/compose/_fhplex.py
--- a/sktime/forecasting/compose/_fhplex.py
+++ b/sktime/forecasting/compose/_fhplex.py
@@ -8,6 +8,8 @@
 
 import pandas as pd
 
+from sktime.datatypes._utilities import get_slice
+
 PANDAS_TS_MTYPES = ["pd.DataFrame", "pd-multiindex", "pd_multiindex_hier"]
 
 
@@ -16,7 +18,7 @@ class FhPlexForecaster(BaseForecaster):
 
     When provided with forecasting horizon [f1, f2, ..., fn],
     will fit forecaster with fh=f1 and parameters fh_params[f1] to forecast f1,
-    forecaster with fh=f2 and parameters fh_params[f1] to forecast f2, etc.
+    forecaster with fh=f2 and parameters fh_params[f2] to forecast f2, etc.
 
     To use different estimators per horizon, combine ``FhPlexForecaster`` with
     one of ``MultiplexForecaster`` and ``MultiplexTransformer``.
@@ -36,6 +38,14 @@ class FhPlexForecaster(BaseForecaster):
         if "relative", fh will be coerced to relative ForecastingHorizon
         if "absolute", fh will be coerced to absolute ForecastingHorizon
         if "as-is", fh will be coerced to ForecastingHorizon (but not relative/absolute)
+    fh_contiguous : bool, default=False
+        whether fh in inner loops are enforced to be contiguous
+        False: forecaster with fh_params[fN] is asked to forecast fN and only fN
+        True: forecaster with fh_params[fN] is asked to forecast 1, 2, ..., fN
+            and the output is then subset to the forecast of fN
+            this is required if the forecaster can only forecast contiguous horizons
+        CAUTION: if using grid search inside, then ``True`` will cause the
+        tuning metric to be evaluated on horizons 1, 2, ..., fN, not just fN
 
     Attributes
     ----------
@@ -66,10 +76,13 @@ class FhPlexForecaster(BaseForecaster):
         "capability:pred_int": True,
     }
 
-    def __init__(self, forecaster, fh_params=None, fh_lookup="relative"):
+    def __init__(
+        self, forecaster, fh_params=None, fh_lookup="relative", fh_contiguous=False
+    ):
         self.forecaster = forecaster
         self.fh_params = fh_params
         self.fh_lookup = fh_lookup
+        self.fh_contiguous = fh_contiguous
 
         super().__init__()
 
@@ -142,33 +155,53 @@ def _fit(self, y, X, fh):
         -------
         self : reference to self
         """
-        fh = self._get_fh_keys(fh)
-
         fh_params = self.fh_params
+        fh_contiguous = self.fh_contiguous
+
+        if fh_contiguous:
+            fh_rel = fh.to_relative(self.cutoff)
+
+        fh_keys = self._get_fh_keys(fh)
 
         self.forecasters_ = {}
 
-        for i, fh_key in enumerate(fh):
+        for i, fh_key in enumerate(fh_keys):
             if isinstance(fh_params, list):
                 ix = i
             else:
                 ix = fh_key
 
+            if not fh_contiguous or fh_rel[i] <= 0:
+                fh_i = [fh_key]
+            else:
+                fh_rel = fh.to_relative(self.cutoff)
+                fh_i = range(1, fh_rel[i] + 1)
+
             params_ix = self._plexfun(ix)
             f_ix = self.forecaster.clone().set_params(**params_ix)
-            self.forecasters_[fh_key] = f_ix.fit(y=y, X=X, fh=[fh[i]])
+            self.forecasters_[fh_key] = f_ix.fit(y=y, X=X, fh=fh_i)
 
         return self
 
     def _get_preds(self, fh_keys, method="predict", **kwargs):
         """Get prediction DataFrame for method."""
+        fh_contiguous = self.fh_contiguous
+
+        if fh_contiguous:
+            fh_abs = fh_keys.to_absolute(self.cutoff)
+
         fh_keys = self._get_fh_keys(fh_keys)
 
         y_preds = []
 
-        for fh_key in fh_keys:
+        for i, fh_key in enumerate(fh_keys):
             fh_method = getattr(self.forecasters_[fh_key], method)
-            y_preds += [fh_method(**kwargs)]
+            if not fh_contiguous:
+                y_preds += [fh_method(**kwargs)]
+            else:
+                y_pred = fh_method(**kwargs)
+                y_pred_slice = get_slice(y_pred, fh_abs[i])
+                y_preds += [y_pred_slice]
 
         y_pred = pd.concat(y_preds, axis=0)
         return y_pred.sort_index()
@@ -449,7 +482,12 @@ def get_test_params(cls, parameter_set="default"):
         f = NaiveForecaster()
 
         params1 = {"forecaster": f, "fh_params": naive_list}
-        params2 = {"forecaster": f, "fh_params": naive_dict, "fh_lookup": "relative"}
+        params2 = {
+            "forecaster": f,
+            "fh_params": naive_dict,
+            "fh_lookup": "relative",
+            "fh_contiguous": True,
+        }
         params3 = {"forecaster": f, "fh_params": naive_str, "fh_lookup": "relative"}
 
         return [params1, params2, params3]


Test patch: