Repository: sktime/sktime

Issue:
[BUG] WindowSummarizer unexpected NaN
**Describe the bug**
<!--
A clear and concise description of what the bug is.
-->
With Sktime 0.25.0 the WindowSummarizer returns unexpected missing values.

**To Reproduce**
<!--
Add a Minimal, Complete, and Verifiable example (for more details, see e.g. https://stackoverflow.com/help/mcve

If the code is too long, feel free to put it in a public gist and link it in the issue: https://gist.github.com
-->

```python
import numpy as np
import pandas as pd
from sktime.transformations.series.summarize import WindowSummarizer

df = pd.DataFrame(np.arange(10))
transformer = WindowSummarizer(lag_feature={"mean": [[-1, 1], [-1, 2]]}, n_jobs=1)
df_transformed = transformer.fit_transform(df)

print(df_transformed)
```

**Expected behavior**
<!--
A clear and concise description of what you expected to happen.
-->
value 0.5 in 1st row 2nd column rather than NaN

**Additional context**
<!--
Add any other context about the problem here.
-->

**Versions**
<details>

<!--
Please run the following code snippet and paste the output here:

from sktime import show_versions; show_versions()
-->

System:
    python: 3.10.12 | packaged by Anaconda, Inc. | (main, Jul  5 2023, 19:01:18) [MSC v.1916 64 bit (AMD64)]
executable: [c:\Users\ramon.bussing\AppData\Local\anaconda3\envs\enerfore-dev\python.exe](file:///C:/Users/Ram0nB/AppData/Local/anaconda3/envs/enerfore-dev/python.exe)
   machine: Windows-10-10.0.19044-SP0

Python dependencies:
          pip: 23.2.1
       sktime: 0.25.0
      sklearn: 1.3.0
       skbase: 0.6.1
        numpy: 1.22.4
        scipy: 1.8.1
       pandas: 2.1.1
   matplotlib: None
       joblib: 1.3.2
        numba: None
  statsmodels: 0.14.0
     pmdarima: None
statsforecast: None
      tsfresh: None
      tslearn: None
        torch: None
   tensorflow: None
tensorflow_probability: None

</details>

<!-- Thanks for contributing! -->



Discussion:
I can reproduce this behavior. The reason for this behavior is that in the function `_window_feature` in summarize.py first shift is called on the DataFrame and then the rolling feature is calculated. 
More specifically, this issue arises since shift is not extending the index. Thus, I assume it would be better to first calculate the rolling feature and do the shifting afterwards. 
I'd like to start working on this issue. Is just switching the order of `shift` and `rolling` in `_window_feature` in summarize.py going to solve the issue of returning NaN?

> I'd like to start working on this issue. Is just switching the order of `shift` and `rolling` in `_window_feature` in summarize.py going to solve the issue of returning NaN?
> 
> 
Thank you that you are working on that. 

I assume that this fixes the issue. But this needs to be tested with the code that is producing the error. 

Btw. I assigned you to the issue. Just ping me if you need help. 
I tried switching the `shift` and `rolling` in `_window_feature` in summarize.py . It now works for the code snippet `1` but does not work for `2`.
1)
df = pd.DataFrame(np.arange(10))
transformer = WindowSummarizer(lag_feature={"mean": [[-1, 1], [-1, 2]]}, n_jobs=1)
df_transformed = transformer.fit_transform(df)

print(df_transformed)

2)

kwargs = {
    "lag_feature": {
        "lag": [1],
        "mean": [[1, 3], [3, 6]],
        "std": [[1, 4]],
    }
}

transformer = WindowSummarizer(**kwargs)
y_transformed = transformer.fit_transform(y_train)

y_transformed.head(10)

But when i try to run  like in `./sktime/examples/01c_forecasting_hierarchical_global.ipynb` it runs into an Error.

"./sktime/sktime/transformations/series/summarize.py", line 460, in _window_feature
    raise ValueError("Input Z must be either a DataFrame or a Series.")
`ValueError: Input Z must be either a DataFrame or a Series.`

Even though y_train is a DataFrame. Could you please help me out.

![Screenshot from 2024-02-28 22-13-00](https://github.com/sktime/sktime/assets/120820143/f0fc16c2-bb95-41eb-b0b1-51d806e078fb)
![Screenshot from 2024-02-28 22-12-33](https://github.com/sktime/sktime/assets/120820143/150c880d-2d64-4712-aff2-384bb20473fe)
I will look at your PR today. Could you ensure that your branch is uptodate. 
@benHeid, would you mind if I give it a try?
Hello, @fkiraly and @benHeid

I think, I have narrowed down what is causing the hierarchal error, Can you tell me what needs to be done if we receive a hierarchal data frame as an input?
 
@sahusiddharth, an easy way to generate hierarchical data for testing is 

```python
from sktime.utils._testing.hierarchical import _make_hierarchical
y = _make_hierarchical()
```

Resolving patch:
diff --git a/sktime/transformations/series/summarize.py b/sktime/transformations/series/summarize.py
--- a/sktime/transformations/series/summarize.py
+++ b/sktime/transformations/series/summarize.py
@@ -473,38 +473,37 @@ class description for in-depth explanation.
             if bfill is False:
                 feat = Z.apply(
                     lambda x: getattr(
-                        x.shift(lag).rolling(
-                            window=window_length, min_periods=window_length
-                        ),
+                        x.rolling(window=window_length, min_periods=window_length),
                         summarizer,
-                    )()
+                    )().shift(lag)
                 )
             else:
                 feat = Z.apply(
                     lambda x: getattr(
-                        x.shift(lag)
-                        .bfill()
-                        .rolling(window=window_length, min_periods=window_length),
+                        x.rolling(window=window_length, min_periods=window_length),
                         summarizer,
                     )()
+                    .shift(lag)
+                    .bfill()
                 )
     else:
-        if bfill is False:
-            feat = Z.shift(lag)
-        else:
-            feat = Z.shift(lag).bfill()
         if isinstance(Z, pd.core.groupby.generic.SeriesGroupBy) and callable(
             summarizer
         ):
-            feat = feat.rolling(window_length).apply(summarizer, raw=True)
+            feat = Z.rolling(window_length).apply(summarizer, raw=True)
         elif not isinstance(Z, pd.core.groupby.generic.SeriesGroupBy) and callable(
             summarizer
         ):
-            feat = feat.apply(
+            feat = Z.apply(
                 lambda x: x.rolling(
                     window=window_length, min_periods=window_length
                 ).apply(summarizer, raw=True)
             )
+        if bfill is False:
+            feat = Z.shift(lag)
+        else:
+            feat = Z.shift(lag).bfill()
+
         feat = pd.DataFrame(feat)
     if bfill is True:
         feat = feat.bfill()


Test patch:
diff --git a/sktime/transformations/series/tests/test_window_summarizer.py b/sktime/transformations/series/tests/test_window_summarizer.py
--- a/sktime/transformations/series/tests/test_window_summarizer.py
+++ b/sktime/transformations/series/tests/test_window_summarizer.py
@@ -76,6 +76,13 @@ def count_gt100(x):
         count_gt100: [[3, 2]],
     }
 }
+
+kwargs_negative_lags = {
+    "lag_feature": {
+        "mean": [[-1, 1], [-1, 2]],
+    }
+}
+
 # Generate named and unnamed y
 y_train.name = None
 y_train_named = y_train.copy()
@@ -94,6 +101,20 @@ def count_gt100(x):
 @pytest.mark.parametrize(
     "kwargs, column_names, y, target_cols, truncate",
     [
+        (
+            kwargs_negative_lags,
+            ["a_mean_-1_1", "a_mean_-1_2"],
+            y_pd,
+            None,
+            None,
+        ),
+        (
+            kwargs_negative_lags,
+            ["y_mean_-1_1", "y_mean_-1_2"],
+            y_hierarchical,
+            None,
+            None,
+        ),
         (
             kwargs,
             ["y_lag_1", "y_mean_1_3", "y_mean_1_12", "y_std_1_4"],