Repository: sktime/sktime

Issue:
[ENH] Simplified expanding window splitter anchored at time series end
**Is your feature request related to a problem? Please describe.**
This is not so much a feature request but me sharing a thing I did that works nicely for me, in case others find it useful. Feel free to close.

The idea is that when defining cross validation, most of the time I just care about two things
1. the number of folds (which drives run time and variance of results) and
2. the horizon/number of test items 
 
By default, the splittier should use as much data as possible.

So this is another splitter that takes those two args:
```py
ExpandingGreedySplitter(horizon=7, folds=5)
```

This will create 5 folds, each with 7 days in the test set, arranged such that the test sets in total cover the last 35 days with no overlaps, and each train set is all data before the test set.

(I'm aware that having `horizon` as an int is not the sktime way but this is what works for me and how I think about the size of the validation set in cross validation, so I'm just sharing as is.)

It also takes a `step_length`, which defaults to the same as horizon for simplicity since that's what I want most of the time.

My work flow tends to be to set a low number for folds and horizon in the early stages of grid search to get a ballpark idea of what values work, then increase the folds and horizon to get more refined results. I found with the existing options this was a bit fiddly.

**Describe the solution you'd like**
This is the class. A bit hacky in places, certainly not accounting for all use cases, but again, gets the job done for my uses.
```py
class ExpandingGreedySplitter(BaseSplitter):
    def __init__(self, horizon, folds=5, step_length=None):
        super().__init__()
        self.folds = folds
        self.horizon = horizon
        self.fh = np.arange(horizon) + 1
        self.step_length = step_length or horizon

    def split(self, y):
        y = self._coerce_to_index(y)
        if isinstance(y, pd.MultiIndex):
            dates = y.get_level_values(y.names[-1])
        else:
            dates = y

        if isinstance(dates, pd.PeriodIndex):
            freq = dates.freq
            dates = dates.to_timestamp()
        else:
            freq_str = pd.infer_freq(dates)
            freq = pd.tseries.frequencies.to_offset(freq_str)

        if freq is None:
            print("Couldn't work out frequency, assuming 'Day'")
            freq = pd.tseries.offsets.Day()

        trn_ends = pd.date_range(
            end=dates.max() - freq * self.horizon,
            periods=self.folds,
            freq=freq * self.step_length,
        )

        for trn_end in trn_ends:
            tst_end = trn_end + freq * self.horizon
            trn_indices = np.flatnonzero(dates <= trn_end)
            tst_indices = np.flatnonzero((dates > trn_end) & (dates <= tst_end))
            yield trn_indices, tst_indices
```

This assumes all instances _end_ at the same date but can have any start date (again, works for my case, not sure how often this assumption holds for others).

**Describe alternatives you've considered**
I originally wrote a `make_expanding_window_cv` function that used `ExpandingWindowSplitter` but this didn't work when my instances had different lengths.




Discussion:
Hm, very interesting!

I think your interface design is great - it seems more convenient and intuitive in a common use case that users want (i.e., chipping off regular pieces at the end)

Would you like to contribute this in a PR?
(I could also do that, but there are these GREAT BADGES you could earn 😁 )

I'll comment on some interface requirements below, but I'd consider these secondary to the fact that this is a great idea and nice design imo.
Regarding interface requirements - @davidgilbertson, I think your ideas are great, but you seem to have a habit to jerry rig your class interfaces so they fit together with the framework. Probably you don't care too much since it works for your use case, but there is a maintainability risk there.

Picture for illustration:
![image](https://github.com/sktime/sktime/assets/7985502/8d064788-8ab0-488d-9221-e70ad32a27df)

Don't get me wrong, your content points are pertinent, your use cases are informative, and your designs are useful.

All I'm saying, if you go the effort of doing the "boring" part of the work too, after having built your new innovative invention, and weld a default power chord connector onto it, its less likely to burn down the house later 😄, and other people might use it too and improve it!

Aware that the interface is not obvious for splitters (there's no extension template, so maybe we should write one).
In your code, there's two bits that should be improved:

* the `sklearn` interface design (which `sktime` also follows for compatibility) assumes that `__init__` params are written to self and never changed. That's not the case for `step_length` - if it is `None`, then `self._step_length` will be sth else. The usual way to deal with it is `self._step_length = step_length or horizon`, and then use `self._step_length` later.
(note, this interface design is debatable, just like the funny third pin on the Swiss power plug, but it's `sklearn`'s so that's what we have to follow so it works in all cases)
* `sktime` uses the [template pattern](https://github.com/kamranahmedse/design-patterns-for-humans#-template-method) for class methods, so `split` itself should not be overwritten, but `_split` is the implementer's abstract method (similar to `fit`/`_fit`, etc). If you rename your `split` to `_split`, it should work, and you get for free that you don't need to assume that all instances end at the same date anymore.

Some minor comments:

* splitters are assumed to work for integer index as well, but that case should be easier than the date-time case.
* another potential design would be to add an `anchor` parameter to `ExpandingWindowSplitter`, and it can be `"start"` or `"end"`. Worth considering, although yours might be simpler to grasp so better for the user.

> you seem to have a habit to jerry rig your class interfaces so they fit together with the framework

Yes this is true, although to be fair, what you're seeing is the experimental code that I'm pasting into GitHub issues. My _real_ code is a work of art :)

I've fixed the `step_length` issues (FYI it might be good in the templates to reference the [Developing scikit-learn estimators](https://scikit-learn.org/stable/developers/develop.html) docs, this clarified a few things for me that I didn't see in the sktime docs).

Now about overriding `_split` instead of `split`. That makes sense, but the reason I overrode `split` was to bypass the vectorization logic; because doing one big batch split is _much_ faster. On my dataset (3 million rows, 1,700 instances), a vectorized split is 6 seconds vs about 1 for the big batch.

To comply with the template pattern I could define `_split_vectorized` to call `self._split()`, but the reason doing it in one big batch works is because of the date index (and assumption about matching end dates). If it should also work with an arbitrary integer index then that case would need the vectorized split. The speed isn't a huge issue since it tends to be insignificant compared to the fit time, but still, fast is better than slow.

Thoughts?
> Yes this is true, although to be fair, what you're seeing is the experimental code that I'm pasting into GitHub issues. My real code is a work of art :)

Please invite me to the vernissage😁

> I've fixed the step_length issues (FYI it might be good in the templates to reference the [Developing scikit-learn estimators](https://scikit-learn.org/stable/developers/develop.html) docs, this clarified a few things for me that I didn't see in the sktime docs).

Good point - although there are some minor clashes between `sktime` an `sklearn` due to some legacy architecture in `sklearn` (e.g., how tags are handled, `get_fitted_params` etc).
What exactly would you like to see or were you missing?

FYI, this is the full `sktime` extension guide: https://www.sktime.net/en/latest/developer_guide/add_estimators.html

>  That makes sense, but the reason I overrode split was to bypass the vectorization logic; becausre doing one big batch split is much faster. On my dataset (3 million rows, 1,700 instances), a vectorized split is 6 seconds vs about 1 for the big batch.

Ah, I see! Yes, you are right that this is a design flaw if you consider the extension contract - that vectorization is always enforced.

Originally, this was a crutch to allow quick-and-dirty generalization to hierarchical data, but now we're relying on it too much.

For similar reasons, I've introduced a tag `"split_hierarchical"` that, if `True`, prevents vectorization from happening - but then you have to deal with `pd.DataFrame` with arbitrary number of multiindex levels in `_split`.
see here for the PR https://github.com/sktime/sktime/pull/4851

So, using this tag should now work if you branch off 0.20.1 or later?

PS: I also added a note of in general doing what you've done here when/if we refactor and consolidate the splitters in this issue: https://github.com/sktime/sktime/issues/4889 - replace vectorization by faster native functionality on hierarchical data.
> What exactly would you like to see or were you missing?

I didn't get that it needed `__init__` to be special ("There should be no logic, not even input validation, and the parameters should not be changed."). Maybe the sktime docs did tell me this and I just needed to be told twice. Also, linking to those docs is nice for sktime's image because you can say "look, all these painful rules aren't ours, they come from scikit-learn!".

Having thought about this a bit more, the following is a way to do splitting of hierarchical data quite quickly (runs in ~23% the time), that doesn't rely on dates or alignment of indexes between instances, or iterating over groups and concatenating the results.

It first creates an `instance_idx` series, the same length as the dataset, which counts down to 0 for each instance individually (the start value is different for different sized instances). Once you have this you don't need to actually loop over the groups, just pick which positions (relative to the end) you want in each split.

```py
class ExpandingGreedySplitter(BaseSplitter):
    _tags = {"split_hierarchical": True}

    def __init__(self, horizon: int, folds: int = 5, step_length: int = None):
        super().__init__()
        self.folds = folds
        self.horizon = horizon
        self.step_length = step_length
        self.fh = np.arange(horizon) + 1

    def _split(self, y: pd.Index):
        y_index = self._coerce_to_index(y)

        groups = pd.Series(index=y_index).groupby(y_index.names[:-1])
        instance_idx = groups.transform("size") - groups.cumcount() - 1

        step_length = self.step_length or self.horizon

        for i in reversed(range(self.folds)):
            tst_end = i * step_length
            trn_end = tst_end + step_length
            trn_indices = np.flatnonzero(instance_idx >= trn_end)
            tst_indices = np.flatnonzero(
                (instance_idx < trn_end) & (instance_idx >= tst_end)
            )
            yield trn_indices, tst_indices
```

`split_hierarchical` is exactly what this needed - good stuff.

I'll do a PR with this in it and some tests, using https://github.com/sktime/sktime/pull/4851 as a guide.
> I didn't get that it needed `__init__` to be special ("There should be no logic, not even input validation, and the parameters should not be changed."). Maybe the sktime docs did tell me this and I just needed to be told twice. Also, linking to those docs is nice for sktime's image because you can say "look, all these painful rules aren't ours, they come from scikit-learn!".

We actually deviate from this in `sktime`, this is one of the major changes from `sklearn`!

That is, `__init__` may contain input validation logic and tag setting, but only after the `super.__init__` call. It also must set all parameters to `self` and never change them again.

> `split_hierarchical` is exactly what this needed - good stuff.

Indeed - that became clear only through the discussions with the splitter-related bug reports though!
(that's why it's great to have these 😄 )

> I'll do a PR with this in it and some tests, using https://github.com/sktime/sktime/pull/4851 as a guide.

🎆 🚀 🎆 

Resolving patch:
diff --git a/sktime/forecasting/model_selection/__init__.py b/sktime/forecasting/model_selection/__init__.py
--- a/sktime/forecasting/model_selection/__init__.py
+++ b/sktime/forecasting/model_selection/__init__.py
@@ -9,6 +9,7 @@
     "SingleWindowSplitter",
     "SlidingWindowSplitter",
     "temporal_train_test_split",
+    "ExpandingGreedySplitter",
     "ExpandingWindowSplitter",
     "TestPlusTrainSplitter",
     "ForecastingGridSearchCV",
@@ -18,6 +19,7 @@
 
 from sktime.forecasting.model_selection._split import (
     CutoffSplitter,
+    ExpandingGreedySplitter,
     ExpandingWindowSplitter,
     SameLocSplitter,
     SingleWindowSplitter,
diff --git a/sktime/forecasting/model_selection/_split.py b/sktime/forecasting/model_selection/_split.py
--- a/sktime/forecasting/model_selection/_split.py
+++ b/sktime/forecasting/model_selection/_split.py
@@ -3,6 +3,7 @@
 """Implement dataset splitting for model evaluation and selection."""
 
 __all__ = [
+    "ExpandingGreedySplitter",
     "ExpandingWindowSplitter",
     "SlidingWindowSplitter",
     "CutoffSplitter",
@@ -11,7 +12,14 @@
     "temporal_train_test_split",
     "TestPlusTrainSplitter",
 ]
-__author__ = ["mloning", "kkoralturk", "khrapovs", "chillerobscuro", "fkiraly"]
+__author__ = [
+    "mloning",
+    "kkoralturk",
+    "khrapovs",
+    "chillerobscuro",
+    "fkiraly",
+    "davidgilbertson",
+]
 
 from typing import Iterator, Optional, Tuple, Union
 
@@ -1185,6 +1193,93 @@ def _split_windows(self, **kwargs) -> SPLIT_GENERATOR_TYPE:
         return self._split_windows_generic(expanding=True, **kwargs)
 
 
+class ExpandingGreedySplitter(BaseSplitter):
+    """Splitter that uses all available data.
+
+    Takes an integer `test_size` that defines the number of steps included in the
+    test set of each fold. The train set of each fold will contain all data before
+    the test set. If the data contains multiple instances, `test_size` is
+    _per instance_.
+
+    If no `step_length` is defined, the test sets (one for each fold) will be
+    adjacent, taken from the end of the dataset.
+
+    For example, with `test_size=7` and `folds=5`, the test sets in total will cover
+    the last 35 steps of the data with no overlap.
+
+    Parameters
+    ----------
+    test_size : int
+        The number of steps included in the test set of each fold.
+    folds : int, default = 5
+        The number of folds.
+    step_length : int, optional
+        The number of steps advanced for each fold. Defaults to `test_size`.
+
+    Examples
+    --------
+    >>> import numpy as np
+    >>> from sktime.forecasting.model_selection import ExpandingGreedySplitter
+
+    >>> ts = np.arange(10)
+    >>> splitter = ExpandingGreedySplitter(test_size=3, folds=2)
+    >>> list(splitter.split(ts))  # doctest: +SKIP
+    [
+        (array([0, 1, 2, 3]), array([4, 5, 6])),
+        (array([0, 1, 2, 3, 4, 5, 6]), array([7, 8, 9]))
+    ]
+    """
+
+    _tags = {"split_hierarchical": True}
+
+    def __init__(self, test_size: int, folds: int = 5, step_length: int = None):
+        super().__init__()
+        self.test_size = test_size
+        self.folds = folds
+        self.step_length = step_length
+        self.fh = np.arange(test_size) + 1
+
+    def _split(self, y: pd.Index) -> SPLIT_GENERATOR_TYPE:
+        if isinstance(y, pd.MultiIndex):
+            groups = pd.Series(index=y).groupby(y.names[:-1])
+            reverse_idx = groups.transform("size") - groups.cumcount() - 1
+        else:
+            reverse_idx = np.arange(len(y))[::-1]
+
+        step_length = self.step_length or self.test_size
+
+        for i in reversed(range(self.folds)):
+            tst_end = i * step_length
+            trn_end = tst_end + self.test_size
+            trn_indices = np.flatnonzero(reverse_idx >= trn_end)
+            tst_indices = np.flatnonzero(
+                (reverse_idx < trn_end) & (reverse_idx >= tst_end)
+            )
+            yield trn_indices, tst_indices
+
+    @classmethod
+    def get_test_params(cls, parameter_set="default"):
+        """Return testing parameter settings for the splitter.
+
+        Parameters
+        ----------
+        parameter_set : str, default="default"
+            Name of the set of test parameters to return, for use in tests. If no
+            special parameters are defined for a value, will return `"default"` set.
+
+        Returns
+        -------
+        params : dict or list of dict, default = {}
+            Parameters to create testing instances of the class
+            Each dict are parameters to construct an "interesting" test instance, i.e.,
+            `MyClass(**params)` or `MyClass(**params[i])` creates a valid test instance.
+            `create_test_instance` uses the first (or only) dictionary in `params`
+        """
+        params1 = {"test_size": 1}
+        params2 = {"test_size": 3, "folds": 2, "step_length": 2}
+        return [params1, params2]
+
+
 class SingleWindowSplitter(BaseSplitter):
     r"""Single window splitter.
 


Test patch:
diff --git a/sktime/forecasting/model_selection/tests/test_split.py b/sktime/forecasting/model_selection/tests/test_split.py
--- a/sktime/forecasting/model_selection/tests/test_split.py
+++ b/sktime/forecasting/model_selection/tests/test_split.py
@@ -11,6 +11,7 @@
 from sktime.forecasting.base import ForecastingHorizon
 from sktime.forecasting.model_selection import (
     CutoffSplitter,
+    ExpandingGreedySplitter,
     ExpandingWindowSplitter,
     SameLocSplitter,
     SingleWindowSplitter,
@@ -325,6 +326,92 @@ def test_sliding_window_splitter_start_with_empty_window(
             )
 
 
+def test_expanding_greedy_splitter_lengths():
+    """Test that ExpandingGreedySplitter returns the correct lengths."""
+    y = np.arange(10)
+    cv = ExpandingGreedySplitter(test_size=2, folds=3)
+
+    lengths = [(len(trn), len(tst)) for trn, tst in cv.split_series(y)]
+    assert lengths == [(4, 2), (6, 2), (8, 2)]
+
+
+def test_expanding_greedy_splitter_dates():
+    """Test that ExpandingGreedySplitter splits on dates correctly."""
+    ts = _make_series(index_type="period")
+    first_date = ts.index[0]
+    last_date = ts.index[-1]
+    cv = ExpandingGreedySplitter(test_size=2, folds=3)
+
+    train_starts = []
+    train_ends = []
+    test_starts = []
+    test_ends = []
+
+    for trn, tst in cv.split_series(ts):
+        train_starts.append(trn.index[0])
+        train_ends.append(trn.index[-1])
+        test_starts.append(tst.index[0])
+        test_ends.append(tst.index[-1])
+
+    assert train_starts == [first_date, first_date, first_date]
+    assert train_ends == [last_date - 6, last_date - 4, last_date - 2]
+    assert test_starts == [last_date - 5, last_date - 3, last_date - 1]
+    assert test_ends == [last_date - 4, last_date - 2, last_date]
+
+
+def test_expanding_greedy_splitter_hierarchy():
+    """Test that ExpandingGreedySplitter handles uneven hierarchical data."""
+    y_panel = _make_hierarchical(
+        hierarchy_levels=(2, 4),
+        max_timepoints=20,
+        min_timepoints=10,
+        same_cutoff=False,
+    )
+
+    cv = ExpandingGreedySplitter(test_size=2, folds=3)
+
+    # Since same_cutoff=False above, we will have a potentially different end date for
+    # each instance in the hierarchy. Below we test that the final fold ends at the
+    # correct end date for each instance
+    last_test_fold = list(cv.split_series(y_panel))[-1][1]
+
+    last_test_dates_actual = (
+        last_test_fold.reset_index(-1)
+        .groupby(last_test_fold.index.names[:-1])
+        .tail(1)["time"]
+    )
+    last_test_dates_expected = (
+        y_panel.reset_index(-1).groupby(y_panel.index.names[:-1]).tail(1)["time"]
+    )
+
+    assert last_test_dates_actual.eq(last_test_dates_expected).all()
+
+
+def test_expanding_greedy_splitter_consecutive():
+    """Test that ExpandingGreedySplitter results in consecutive test periods."""
+    y_panel = _make_hierarchical(
+        hierarchy_levels=(2, 4),
+        max_timepoints=20,
+        min_timepoints=10,
+        same_cutoff=False,
+    )
+
+    cv = ExpandingGreedySplitter(test_size=2, folds=3)
+
+    test_dfs = [tst for _, tst in cv.split_series(y_panel)]
+
+    combined_test_df = pd.concat(test_dfs).sort_index()
+
+    # We check the .diff(). Equal diffs = uniformly spaced timestamps
+    has_consecutive_index = (
+        combined_test_df.reset_index(-1)
+        .groupby(combined_test_df.index.names[:-1])["time"]
+        .agg(lambda s: len(s.diff().value_counts()) == 1)
+    )
+
+    assert has_consecutive_index.all()
+
+
 @pytest.mark.parametrize("y", TEST_YS)
 @pytest.mark.parametrize("fh", [*TEST_FHS, *TEST_FHS_TIMEDELTA])
 @pytest.mark.parametrize("step_length", TEST_STEP_LENGTHS)