Repository: sktime/sktime
Pull request: [BUG] fix BaseWindowSplitter get_n_split method for hierarchical data
Fixes #4971

#### What does this implement/fix? Explain your changes.
<!--
A clear and concise description of what you have implemented.
-->

The logic of the `BaseWindowSplitter.get_n_split` method does not work for hierarchical cases. This is mainly because it counts cutoffs on y, assuming that it is either a **single** or a **multivariate** time series. Therefore, when hierarchical data is passed, the logic breaks because it counts cutoff as if the "pd_multiindex_hier" is a single series when it is not.

Solution
Using the existence implementation, for when pd_multiindex_hier datatype is passed, the cutoff point is counted based on the first instance of the lowest level time series hierarchy. 

Other comments
I also found out that other splitters, like `CutoffSplitter`, may return the wrong number of splits because they use the same logic as `BaseWindowSplitter` to count the splits. Also, It seems that the original implementation for hierarchical splitter #2599 may not consider how `get_n_splits` method should return in the case of hierarchical.

Problem statement:
[BUG] get_n_splits of window splitter does not reflect the number of splits for hierarchical data
**Describe the bug**
<!--
A clear and concise description of what the bug is.
-->

`get_n_splits` method returns the wrong number of splits when hierarchical data is passed into window based splitter. 

**To Reproduce**

```python
from sktime.utils._testing.hierarchical import _make_hierarchical
from sktime.forecasting.model_selection import ExpandingWindowSplitter, SlidingWindowSplitter

y_hierarchical = _make_hierarchical(hierarchy_levels=(2, 3), min_timepoints=12, max_timepoints=12, random_state=0)
fh = [1]
step_length = 1
window_length = 10

# each unique series (lowest hierachical level) should have two splits
cv = ExpandingWindowSplitter(initial_window=window_length, fh=fh, step_length=step_length)
sliding = SlidingWindowSplitter(fh=fh, window_length=window_length, step_length=step_length, start_with_window=True)

n_splits_cv = cv.get_n_splits(y_hierarchical)
n_splits_sliding = sliding.get_n_splits(y_hierarchical)

print(f"Expanding Window splits : {n_splits_cv}")
print(f"Sliding Window Number of splits : {n_splits_sliding}")
```

Output
```
Expanding Window Number of splits : 62
Sliding Window Number of splits : 62
```
**Expected behavior**
<!--
A clear and concise description of what you expected to happen.
-->
 
Shouldnt the number of split is based on either of these two?:
1. Return number of splits based on datatype train/test splits which is two in this case
```python
print(f"Expanding Window splits: {len(list(cv.split(y_hierarchical)))}")
print(f"Sliding Window splits: {len(list(sliding.split(y_hierarchical)))}")
```
2. Return number of splits based on splits of the lowest hierachical level series. In this case it will be 6 * 2 = 12

**Versions**
<details>

System:
python: 3.9.17 (main, Jul  5 2023, 20:47:11) [MSC v.1916 64 bit (AMD64)]
machine: Windows-10-10.0.19045-SP0

Python dependencies:
pip: 23.2.1
sktime: 0.21.0
sklearn: 1.3.0
skbase: 0.4.6
numpy: 1.24.3
scipy: 1.11.1
pandas: 2.0.3
matplotlib: 3.7.2
joblib: 1.3.1
statsmodels: 0.14.0
numba: 0.57.1
pmdarima: 2.0.3
tsfresh: 0.20.1
tensorflow: 2.13.0
tensorflow_probability: None

</details>

Hints from issue discussion:
@fkiraly, do you have opinion on which of the two should be the expected behaviour of `get_n_splits` method for hierarchical data?
I think it should always be equal to what `split` would return?

Commit message: change to check_is_scitype

Patch to review:
diff --git a/sktime/forecasting/model_selection/_split.py b/sktime/forecasting/model_selection/_split.py
index 5fb23b18e8f..bca136ad2c6 100644
--- a/sktime/forecasting/model_selection/_split.py
+++ b/sktime/forecasting/model_selection/_split.py
@@ -996,12 +996,26 @@ def get_n_splits(self, y: Optional[ACCEPTED_Y_TYPES] = None) -> int:
         n_splits : int
             The number of splits.
         """
+        from sktime.datatypes import check_is_scitype
+
         if y is None:
             raise ValueError(
                 f"{self.__class__.__name__} requires `y` to compute the "
                 f"number of splits."
             )
-        return len(self.get_cutoffs(y))
+
+        # n_splits based on the first instance of the lowest level series cutoffs
+        if check_is_scitype(y, scitype="Hierarchical"):
+            index = self._coerce_to_index(y)
+            for _, values in y.groupby(index.droplevel(-1)):
+                # convert to a single ts
+                instance_series = values.reset_index().iloc[:, -2:]
+                instance_series.set_index(instance_series.columns[0], inplace=True)
+                n_splits = len(self.get_cutoffs(instance_series))
+                break
+        else:
+            n_splits = len(self.get_cutoffs(y))
+        return n_splits
 
     def get_cutoffs(self, y: Optional[ACCEPTED_Y_TYPES] = None) -> np.ndarray:
         """Return the cutoff points in .iloc[] context.
diff --git a/sktime/forecasting/model_selection/tests/test_split.py b/sktime/forecasting/model_selection/tests/test_split.py
index 887551003bc..91b3cb45a7c 100644
--- a/sktime/forecasting/model_selection/tests/test_split.py
+++ b/sktime/forecasting/model_selection/tests/test_split.py
@@ -711,3 +711,20 @@ def test_same_loc_splitter_hierarchical():
     for (t1, tt1), (t2, tt2) in zip(split_template_loc, split_templated_loc):
         assert np.all(t1 == t2)
         assert np.all(tt1 == tt2)
+
+
+@pytest.mark.parametrize("CV", [SlidingWindowSplitter, ExpandingWindowSplitter])
+@pytest.mark.parametrize("fh", [*TEST_FHS, *TEST_FHS_TIMEDELTA])
+@pytest.mark.parametrize("window_length", TEST_WINDOW_LENGTHS)
+@pytest.mark.parametrize("step_length", TEST_STEP_LENGTHS)
+def test_windowbase_splitter_get_n_split_hierarchical(
+    CV, fh, window_length, step_length
+):
+    """Test that WindowBaseSplitter.get_n_splits works for hierarchical data."""
+    # see bugs 4971
+    y = _make_hierarchical((2, 3), 15, 15)
+    if _inputs_are_supported([fh, window_length, step_length]):
+        cv = CV(fh, window_length, step_length)
+        assert cv.get_n_splits(y) == len(
+            list(cv.split(y))
+        ), "get_n_splits does not equal the number of splits in the output."


Code review comments:
diff_hunk: @@ -996,12 +996,26 @@ def get_n_splits(self, y: Optional[ACCEPTED_Y_TYPES] = None) -> int:
         n_splits : int
             The number of splits.
         """
+        from sktime.datatypes import check_is_scitype
+
         if y is None:
             raise ValueError(
                 f"{self.__class__.__name__} requires `y` to compute the "
                 f"number of splits."
             )
-        return len(self.get_cutoffs(y))
+
+        # n_splits based on the first instance of the lowest level series cutoffs
+        if check_is_scitype(y, scitype="Hierarchical"):
path: sktime/forecasting/model_selection/_split.py
text: @user1:
this should also cover `"Panel"`, which is separate (e.g., 2-level `MultiIndex`)

@author:
same as below. Kindle review them again to check if it's ok