Repository: sktime/sktime

Issue:
[BUG] Imputer imputes too many pd-multiindex missing values
**Describe the bug**
<!--
A clear and concise description of what the bug is.
-->
The Imputer supports the "ffill" and "bfill" methods. The documentation states that only one of these methods will be applied when choosing one, but this is not the case.

Moreover, when imputing pd-multiindex data, the "ffill" after the "bfill" (and vice versa) will be done on global level rather than instance level.

**To Reproduce**
<!--
Add a Minimal, Complete, and Verifiable example (for more details, see e.g. https://stackoverflow.com/help/mcve

If the code is too long, feel free to put it in a public gist and link it in the issue: https://gist.github.com
-->

```python
import numpy as np
from sktime.datatypes import get_examples
from sktime.transformations.series.impute import Imputer

df = get_examples(mtype="pd-multiindex")[0]
df.iloc[3:6,:] = np.nan

df_imp = Imputer(method="ffill").fit_transform(df)

print(df)
print(df_imp)
```

Which outputs:

```
                      var_0  var_1
instances timepoints              
0         0             1.0    4.0
          1             2.0    5.0
          2             3.0    6.0
1         0             NaN    NaN
          1             NaN    NaN
          2             NaN    NaN
2         0             1.0   42.0
          1             2.0    5.0
          2             3.0    6.0
                      var_0  var_1
instances timepoints              
0         0             1.0    4.0
          1             2.0    5.0
          2             3.0    6.0
1         0             1.0   42.0
          1             1.0   42.0
          2             1.0   42.0
2         0             1.0   42.0
          1             2.0    5.0
          2             3.0    6.0
```

**Expected behavior**
<!--
A clear and concise description of what you expected to happen.
-->
I expect the missing values not to be imputed since one instance is entirely missing.

**Additional context**
<!--
Add any other context about the problem here.
-->
When looking at the following lines we can see the issue
https://github.com/sktime/sktime/blob/e0e66f1e847d1bed06fbea0a511b666eae8f895d/sktime/transformations/series/impute.py#L226-L272
- Only in the case of a multiindex, after "ffill" is applied to a grouped pandas dataframe on instance level, "bfill" will be applied to the entire dataframe.
- Only in the case of a multiindex, after "bfill" is applied to a grouped pandas dataframe on instance level, "ffill" will be applied to the entire dataframe.


Suggested fix:
- To keep the imputation behavior only per instance level, line 239 and 243 should therefore also impute on the grouped dataframe.
- The documentation should state that ffill and bfill will be applied after every method.

**Versions**
<details>

<!--
Please run the following code snippet and paste the output here:

from sktime import show_versions; show_versions()
-->
System:
    python: 3.10.14 | packaged by Anaconda, Inc. | (main, Mar 21 2024, 16:20:14) [MSC v.1916 64 bit (AMD64)]
executable: [c:\Users\ramon.bussing\AppData\Local\anaconda3\envs\enerfore\python.exe](file:///C:/Users/Ram0nB/AppData/Local/anaconda3/envs/enerfore/python.exe)
   machine: Windows-10-10.0.22621-SP0

Python dependencies:
          pip: 23.3.1
       sktime: 0.27.1
      sklearn: 1.4.1.post1
       skbase: 0.7.5
        numpy: 1.26.4
        scipy: 1.12.0
       pandas: 2.1.4
   matplotlib: None
       joblib: 1.3.2
        numba: None
  statsmodels: 0.14.1
     pmdarima: 2.0.4
statsforecast: None
      tsfresh: None
      tslearn: None
        torch: None
   tensorflow: None
tensorflow_probability: None
</details>

<!-- Thanks for contributing! -->



Discussion:
Ah, indeed! So you'd need to group it again before the `bfill`, right?

Would you like to open a PR with a fix? Seems like the correct diagnosis already!

Your example would also make a good test.
> Ah, indeed! So you'd need to group it again before the `bfill`, right?
> 
> Would you like to open a PR with a fix? Seems like the correct diagnosis already!

Exactly! I'm happy to do so, I'll pick it up tomorrow.
Excellent.

Regarding the doc/logic mismatch, the standard way to address this is to change the doc, not the logic - to avoid to break currently working code.

Changing the logic can be done later, but should be accompanied by warnings etc - if we decide to do that.
EDIT: some profiling shows that the current solution is the fastest, even for a small dataframe. `df.index.levels > 4` is approximately 1/3 slower than `isinstance(df, MultiIndex)`:
```python
>>> df = pd.DataFrame(dict(x=[1,2,3], y=[4,5,6]))
>>> timeit(lambda: isinstance(df, pd.MultiIndex), number=100000000)
6.388806922000015
>>> timeit(lambda: df.index.nlevels > 1, number=100000000)
9.203695821999986
```
Olde comment:
As a side-note, I
'm not
 wasn't sure whether this is the best way to check for presence of a MultiIndex.
https://github.com/sktime/sktime/blob/e0e66f1e847d1bed06fbea0a511b666eae8f895d/sktime/transformations/series/impute.py#L180
https://github.com/sktime/sktime/blob/e0e66f1e847d1bed06fbea0a511b666eae8f895d/sktime/transformations/series/impute.py#L233

Resolving patch:
diff --git a/sktime/transformations/series/impute.py b/sktime/transformations/series/impute.py
--- a/sktime/transformations/series/impute.py
+++ b/sktime/transformations/series/impute.py
@@ -23,7 +23,9 @@ class Imputer(BaseTransformer):
     Parameters
     ----------
     method : str, default="drift"
-        Method to fill the missing values.
+        Method to fill the missing values. Not all methods can extrapolate, so after
+        ``method`` is applied the remaining missing values are filled with ``ffill``
+        then ``bfill``.
 
         * "drift" : drift/trend values by sktime.PolynomialTrendForecaster(degree=1)
           first, X in transform() is filled with ffill then bfill
@@ -231,22 +233,26 @@ def _transform(self, X, y=None):
         elif self.method == "constant":
             return X.fillna(value=self.value)
         elif isinstance(index, pd.MultiIndex):
-            X_grouped = X.groupby(level=list(range(index.nlevels - 1)))
+            X_group_levels = list(range(index.nlevels - 1))
 
             if self.method in ["backfill", "bfill"]:
-                X = X_grouped.bfill()
-                # fill trailing NAs of panel instances with reverse method
-                return X.ffill()
+                X = X.groupby(level=X_group_levels).bfill()
             elif self.method in ["pad", "ffill"]:
-                X = X_grouped.ffill()
-                # fill leading NAs of panel instances with reverse method
-                return X.bfill()
+                X = X.groupby(level=X_group_levels).ffill()
             elif self.method == "mean":
-                return X_grouped.fillna(value=self._mean)
+                X = X.groupby(level=X_group_levels).fillna(value=self._mean)
             elif self.method == "median":
-                return X_grouped.fillna(value=self._median)
+                X = X.groupby(level=X_group_levels).fillna(value=self._median)
             else:
                 raise AssertionError("Code should not be reached")
+
+            # fill first/last elements of series,
+            # as some methods can't impute those
+            X = X.groupby(level=X_group_levels).ffill()
+            X = X.groupby(level=X_group_levels).bfill()
+
+            return X
+
         else:
             if self.method in ["backfill", "bfill"]:
                 X = X.bfill()


Test patch:
diff --git a/sktime/transformations/series/tests/test_imputer.py b/sktime/transformations/series/tests/test_imputer.py
--- a/sktime/transformations/series/tests/test_imputer.py
+++ b/sktime/transformations/series/tests/test_imputer.py
@@ -8,7 +8,9 @@
 import numpy as np
 import pytest
 
+from sktime.datatypes import get_examples
 from sktime.forecasting.naive import NaiveForecaster
+from sktime.transformations.compose import TransformByLevel
 from sktime.transformations.series.impute import Imputer
 from sktime.utils._testing.forecasting import make_forecasting_problem
 from sktime.utils._testing.hierarchical import _make_hierarchical
@@ -58,6 +60,44 @@ def test_imputer(method, Z, value, forecaster):
     assert not y_hat.isnull().to_numpy().any()
 
 
+@pytest.mark.parametrize(
+    "method",
+    [
+        "linear",
+        "nearest",
+        "mean",
+        "median",
+        "backfill",
+        "pad",
+    ],
+)
+def test_impute_multiindex(method):
+    """Test for data leakage in case of pd-multiindex data.
+
+    Failure case in bug #6224
+    """
+
+    df = get_examples(mtype="pd-multiindex")[0].copy()
+    df.iloc[:3, :] = np.nan  # instance 0 entirely missing
+    df.iloc[3:4, :] = np.nan  # instance 1 first timepoint missing
+    df.iloc[8:, :] = np.nan  # instance 2 last timepoint missing
+
+    imp = Imputer(method=method)
+    df_imp = imp.fit_transform(df)
+
+    # instance 0 entirely missing, so it should remain missing
+    assert np.array_equal(df.iloc[:3, :], df_imp.iloc[:3, :], equal_nan=True)
+
+    # instance 1 and 2 should not have any missing values
+    assert not df_imp.iloc[3:, :].isna().any().any()
+
+    # test consistency between applying the imputer to every instance separately,
+    # vs applying them to the panel
+    imp_tbl = TransformByLevel(Imputer(method=method))
+    df_imp_tbl = imp_tbl.fit_transform(df)
+    assert np.array_equal(df_imp, df_imp_tbl, equal_nan=True)
+
+
 def test_imputer_forecaster_y():
     """Test that forecaster imputer works with y.