Repository: sktime/sktime (Python)

Issue:
[BUG] Graphical pipeline requires `X` in `predict`
**Describe the bug**
The graphical pipeline requires `X` in the prediction step if a transformation on `y` is present in the pipeline. This doesn't make too much sense because `X` has been computed in the `fit` step and can't be provided in the `fit` by the user. 

**To Reproduce**
<!--
Add a Minimal, Complete, and Verifiable example (for more details, see e.g. https://stackoverflow.com/help/mcve

If the code is too long, feel free to put it in a public gist and link it in the issue: https://gist.github.com
-->

```python
from sktime.forecasting.compose import make_reduction
from sklearn.linear_model import Ridge
from sktime.transformations.series.difference import Differencer
from sktime.transformations.series.lag import Lag
from sktime.transformations.series.impute import Imputer
from sktime.datasets import load_airline
from sktime.split import temporal_train_test_split
from sktime.pipeline import Pipeline

y = load_airline()
y_train, y_test = temporal_train_test_split(y)

forecaster = make_reduction(Ridge(), window_length=12, strategy="recursive")

pipe = Pipeline()
pipe = pipe.add_step(Differencer(lags=[1, 3]), "differencer", edges={"X": "y"})
pipe = pipe.add_step(
    forecaster, name="forecaster", edges={"X": "differencer", "y": "y"}
)
# can't pass X=y_train, at predict-time this would be X=X_pred=y_pred
pipe.fit(y=y_train)
y_pred = pipe.predict(fh=[1, 2, 3])
```
Output:
```
ValueError: `X` must be passed to `predict` if `X` is given in `fit`.
```
**Expected behavior**

The expected behavior is that `X` is also computed in the predict step similar to the fit.  

**Additional context**

I'm working with the sktime-dev installation including the changes from (https://github.com/sktime/sktime/pull/5656)

**Versions**
<details>

<!--
Please run the following code snippet and paste the output here:

from sktime import show_versions; show_versions()
-->
System:
    python: 3.9.13 (main, Aug 25 2022, 23:51:50) [MSC v.1916 64 bit (AMD64)]
executable: [c:\Users\user\anaconda3\python.exe](file:///C:/Users/user/anaconda3/python.exe)
   machine: Windows-10-10.0.22621-SP0

Python dependencies:
          pip: 22.3.1
       sktime: 0.25.0
      sklearn: 1.0.2
       skbase: 0.6.1
        numpy: 1.23.5
        scipy: 1.9.3
       pandas: 1.4.4
   matplotlib: 3.6.2
       joblib: 1.1.1
        numba: 0.56.4
  statsmodels: 0.14.1
     pmdarima: 2.0.4
statsforecast: 1.7.1
      tsfresh: None
      tslearn: None
        torch: 1.13.1+cpu
   tensorflow: None
tensorflow_probability: None

</details>

<!-- Thanks for contributing! -->



Discussion:
Is this a duplicate of https://github.com/sktime/sktime/issues/5653?

#5653 was fixed by a PR, and the fix will be released with 0.25.1 any minute now.
Oh, you are claiming that #5356 does *not* fix the problem in your above case.

So we need to check whether it is the same problem or not, and if yes why the fix did not work.

FYI @benHeid 
What I'm saying is that https://github.com/sktime/sktime/issues/5653 fixed the problem in `fit` but now there is a new bug in `predict`.
It is not the same bug as the one for `fit`. The problem is a bit different. The cause of this bug is that the differencer is not called during `predict` (or with no data, I am unsure), since the buffers are emptied, because of the new data. Consequently, the result of the `differencer` step is None. Which is then passed as `X` to the forecaster. I fixed this issue by not emptying the buffers if no new data are provided... 

Sidenote: I would not like to concatenate the data to the buffers. Since this would cause head ache if we want to enable global forecasting.... Since this would require to check how to concatenate, and probably also would cause that predict would then be applied to all columns.. [It is late perhaps the sidenote makes no sense...]



Patch:
diff --git a/sktime/pipeline/pipeline.py b/sktime/pipeline/pipeline.py
index 207e6bcb102..8f23b6cb009 100644
--- a/sktime/pipeline/pipeline.py
+++ b/sktime/pipeline/pipeline.py
@@ -598,10 +598,18 @@ def predict_residuals(self, X, y=None, **kwargs):
     def _initiate_call(self, X, y, kwargs):
         if not self._assembled:
             self._assemble_steps()
-        for step in self.assembled_steps.values():
-            step.reset()
-        self.assembled_steps["X"].buffer = X
-        self.assembled_steps["y"].buffer = y
+        for key, step in self.assembled_steps.items():
+            # Empty the buffer of all steps except for the dummy
+            # steps X and y (input steps)
+            if key in ["X", "y"]:
+                step.reset(reset_buffer=False)
+            else:
+                step.reset()
+        # Overwrite the buffer of X and y if data are provided
+        if X is not None:
+            self.assembled_steps["X"].buffer = X
+        if y is not None:
+            self.assembled_steps["y"].buffer = y
         self.kwargs.update(kwargs)
 
     def _method_allowed(self, method):
diff --git a/sktime/pipeline/step.py b/sktime/pipeline/step.py
index e2c58286007..01d2274f075 100644
--- a/sktime/pipeline/step.py
+++ b/sktime/pipeline/step.py
@@ -73,9 +73,10 @@ def __init__(
         self.input_edges = input_edges
         self.params = params
 
-    def reset(self):
+    def reset(self, reset_buffer=True):
         """Reset the step."""
-        self.buffer = None
+        if reset_buffer:
+            self.buffer = None
         self.mode = ""
 
     def get_allowed_method(self):


Test patch:
diff --git a/sktime/pipeline/tests/regression_tests/test_pipeline_regression.py b/sktime/pipeline/tests/regression_tests/test_pipeline_regression.py
index e32b398d1ce..c555139edd8 100644
--- a/sktime/pipeline/tests/regression_tests/test_pipeline_regression.py
+++ b/sktime/pipeline/tests/regression_tests/test_pipeline_regression.py
@@ -2,10 +2,11 @@
 import pandas as pd
 import pytest
 from skbase.utils.dependencies import _check_soft_dependencies
+from sklearn.linear_model import Ridge
 
 from sktime.classification.dummy import DummyClassifier
-from sktime.datasets import load_arrow_head, load_longley
-from sktime.forecasting.compose import ForecastX
+from sktime.datasets import load_airline, load_arrow_head, load_longley
+from sktime.forecasting.compose import ForecastX, make_reduction
 from sktime.forecasting.naive import NaiveForecaster
 from sktime.forecasting.sarimax import SARIMAX
 from sktime.pipeline.pipeline import Pipeline
@@ -317,3 +318,20 @@ def test_forecasterX_regression():
     general_pipeline.fit(y=y, X=X, fh=[1, 2, 3])
     result_general = general_pipeline.predict(None, None)
     pd.testing.assert_series_equal(result, result_general)
+
+
+def test_lagged_y_prediction():
+    # regression test for issue 5830
+    y = load_airline()
+    y_train, y_test = temporal_train_test_split(y)
+
+    forecaster = make_reduction(Ridge(), window_length=12, strategy="recursive")
+
+    pipe = Pipeline()
+    pipe = pipe.add_step(Differencer(lags=[1, 3]), "differencer", edges={"X": "y"})
+    pipe = pipe.add_step(
+        forecaster, name="forecaster", edges={"X": "differencer", "y": "y"}
+    )
+    pipe.fit(y=y_train)
+    y_pred = pipe.predict(fh=y_test.index)
+    assert y_pred.shape == y_test.shape