Repository: sktime/sktime

Issue:
[BUG] Graphical pipeline requires `X` in `predict`
**Describe the bug**
The graphical pipeline requires `X` in the prediction step if a transformation on `y` is present in the pipeline. This doesn't make too much sense because `X` has been computed in the `fit` step and can't be provided in the `fit` by the user. 

**To Reproduce**
<!--
Add a Minimal, Complete, and Verifiable example (for more details, see e.g. https://stackoverflow.com/help/mcve

If the code is too long, feel free to put it in a public gist and link it in the issue: https://gist.github.com
-->

```python
from sktime.forecasting.compose import make_reduction
from sklearn.linear_model import Ridge
from sktime.transformations.series.difference import Differencer
from sktime.transformations.series.lag import Lag
from sktime.transformations.series.impute import Imputer
from sktime.datasets import load_airline
from sktime.split import temporal_train_test_split
from sktime.pipeline import Pipeline

y = load_airline()
y_train, y_test = temporal_train_test_split(y)

forecaster = make_reduction(Ridge(), window_length=12, strategy="recursive")

pipe = Pipeline()
pipe = pipe.add_step(Differencer(lags=[1, 3]), "differencer", edges={"X": "y"})
pipe = pipe.add_step(
    forecaster, name="forecaster", edges={"X": "differencer", "y": "y"}
)
# can't pass X=y_train, at predict-time this would be X=X_pred=y_pred
pipe.fit(y=y_train)
y_pred = pipe.predict(fh=[1, 2, 3])
```
Output:
```
ValueError: `X` must be passed to `predict` if `X` is given in `fit`.
```
**Expected behavior**

The expected behavior is that `X` is also computed in the predict step similar to the fit.  

**Additional context**

I'm working with the sktime-dev installation including the changes from (https://github.com/sktime/sktime/pull/5656)

**Versions**
<details>

<!--
Please run the following code snippet and paste the output here:

from sktime import show_versions; show_versions()
-->
System:
    python: 3.9.13 (main, Aug 25 2022, 23:51:50) [MSC v.1916 64 bit (AMD64)]
executable: [c:\Users\user\anaconda3\python.exe](file:///C:/Users/user/anaconda3/python.exe)
   machine: Windows-10-10.0.22621-SP0

Python dependencies:
          pip: 22.3.1
       sktime: 0.25.0
      sklearn: 1.0.2
       skbase: 0.6.1
        numpy: 1.23.5
        scipy: 1.9.3
       pandas: 1.4.4
   matplotlib: 3.6.2
       joblib: 1.1.1
        numba: 0.56.4
  statsmodels: 0.14.1
     pmdarima: 2.0.4
statsforecast: 1.7.1
      tsfresh: None
      tslearn: None
        torch: 1.13.1+cpu
   tensorflow: None
tensorflow_probability: None

</details>

<!-- Thanks for contributing! -->



Discussion:
Is this a duplicate of https://github.com/sktime/sktime/issues/5653?

#5653 was fixed by a PR, and the fix will be released with 0.25.1 any minute now.
Oh, you are claiming that #5356 does *not* fix the problem in your above case.

So we need to check whether it is the same problem or not, and if yes why the fix did not work.

FYI @benHeid 
What I'm saying is that https://github.com/sktime/sktime/issues/5653 fixed the problem in `fit` but now there is a new bug in `predict`.

Resolving patch:
diff --git a/sktime/pipeline/pipeline.py b/sktime/pipeline/pipeline.py
--- a/sktime/pipeline/pipeline.py
+++ b/sktime/pipeline/pipeline.py
@@ -598,10 +598,18 @@ def predict_residuals(self, X, y=None, **kwargs):
     def _initiate_call(self, X, y, kwargs):
         if not self._assembled:
             self._assemble_steps()
-        for step in self.assembled_steps.values():
-            step.reset()
-        self.assembled_steps["X"].buffer = X
-        self.assembled_steps["y"].buffer = y
+        for key, step in self.assembled_steps.items():
+            # Empty the buffer of all steps except for the dummy
+            # steps X and y (input steps)
+            if key in ["X", "y"]:
+                step.reset(reset_buffer=False)
+            else:
+                step.reset()
+        # Overwrite the buffer of X and y if data are provided
+        if X is not None:
+            self.assembled_steps["X"].buffer = X
+        if y is not None:
+            self.assembled_steps["y"].buffer = y
         self.kwargs.update(kwargs)
 
     def _method_allowed(self, method):
diff --git a/sktime/pipeline/step.py b/sktime/pipeline/step.py
--- a/sktime/pipeline/step.py
+++ b/sktime/pipeline/step.py
@@ -73,9 +73,10 @@ def __init__(
         self.input_edges = input_edges
         self.params = params
 
-    def reset(self):
+    def reset(self, reset_buffer=True):
         """Reset the step."""
-        self.buffer = None
+        if reset_buffer:
+            self.buffer = None
         self.mode = ""
 
     def get_allowed_method(self):


Test patch:
diff --git a/sktime/pipeline/tests/regression_tests/test_pipeline_regression.py b/sktime/pipeline/tests/regression_tests/test_pipeline_regression.py
--- a/sktime/pipeline/tests/regression_tests/test_pipeline_regression.py
+++ b/sktime/pipeline/tests/regression_tests/test_pipeline_regression.py
@@ -2,10 +2,11 @@
 import pandas as pd
 import pytest
 from skbase.utils.dependencies import _check_soft_dependencies
+from sklearn.linear_model import Ridge
 
 from sktime.classification.dummy import DummyClassifier
-from sktime.datasets import load_arrow_head, load_longley
-from sktime.forecasting.compose import ForecastX
+from sktime.datasets import load_airline, load_arrow_head, load_longley
+from sktime.forecasting.compose import ForecastX, make_reduction
 from sktime.forecasting.naive import NaiveForecaster
 from sktime.forecasting.sarimax import SARIMAX
 from sktime.pipeline.pipeline import Pipeline
@@ -317,3 +318,20 @@ def test_forecasterX_regression():
     general_pipeline.fit(y=y, X=X, fh=[1, 2, 3])
     result_general = general_pipeline.predict(None, None)
     pd.testing.assert_series_equal(result, result_general)
+
+
+def test_lagged_y_prediction():
+    # regression test for issue 5830
+    y = load_airline()
+    y_train, y_test = temporal_train_test_split(y)
+
+    forecaster = make_reduction(Ridge(), window_length=12, strategy="recursive")
+
+    pipe = Pipeline()
+    pipe = pipe.add_step(Differencer(lags=[1, 3]), "differencer", edges={"X": "y"})
+    pipe = pipe.add_step(
+        forecaster, name="forecaster", edges={"X": "differencer", "y": "y"}
+    )
+    pipe.fit(y=y_train)
+    y_pred = pipe.predict(fh=y_test.index)
+    assert y_pred.shape == y_test.shape