Repository: sktime/sktime
Pull request #6360 (2024-04-28 23:48:52)

[ENH] `ExpandingCutoffSplitter` - splitter by moving cutoff
<!--
Welcome to sktime, and thanks for contributing!
Please have a look at our contribution guide:
https://www.sktime.net/en/latest/get_involved/contributing.html
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.

Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests.
If no issue exists, you can open one here: https://github.com/sktime/sktime/issues
-->
https://github.com/sktime/sktime/discussions/6327
https://github.com/sktime/sktime/issues/6359

#### What does this implement/fix? Explain your changes.
<!--
A clear and concise description of what you have implemented.
-->
This PR introduces a new splitter, `ExpandingCutoffSplitter`, to the time series forecasting module of sktime. This splitter starts with an initial training window based on a specified cutoff point and expands incrementally in each split. This feature addresses the need for a dynamic adjustment of the training window in time series forecasting based on a cutoff date, especially in cases where many different time series will be tested on common cutoff dates but otherwise have different starting points.

#### Does your contribution introduce a new dependency? If yes, which one?
No new dependencies are introduced with this PR.

<!--
Only relevant if you changed pyproject.toml.
We try to minimize dependencies in the core dependency set. There
are also further specific instructions to follow for soft dependencies.
See here for handling dependencies in sktime: https://www.sktime.net/en/latest/developer_guide/dependencies.html
-->

#### What should a reviewer concentrate their feedback on?
- Implementation of the `ExpandingCutoffSplitter` logic.
- Integration with the existing splitter architecture.
- Applicability and ease of use in various forecasting scenarios.
<!-- This section is particularly useful if you have a pull request that is still in development. You can guide the reviews to focus on the parts that are ready for their comments. We suggest using bullets (indicated by * or -) and filled checkboxes [x] here -->

#### Did you add any tests for the change?
Yes, tests have been added to ensure the `ExpandingCutoffSplitter` works as intended and interacts correctly with other splitters and forecasting tools in sktime.

<!-- This section is useful if you have added a test in addition to the existing ones. This will ensure that further changes to these files won't introduce the same kind of bug. It is considered good practice to add tests with newly added code to enforce the fact that the code actually works. This will reduce the chance of introducing logical bugs.
-->

#### Any other comments?
<!--
We value all user contributions, no matter how small or complex they are. If you have any questions, feel free to post
in the dev-chat channel on the sktime discord https://discord.com/invite/54ACzaFsn7. If we are slow to review (>3 working days), likewise feel free to ping us on discord. Thank you for your understanding during the review process.
-->

#### PR checklist
<!--
Please go through the checklist below. Please feel free to remove points if they are not applicable.
-->

##### For all contributions
- [ x] I've added myself to the [list of contributors](https://github.com/sktime/sktime/blob/main/CONTRIBUTORS.md) with any new badges I've earned :-)
  How to: add yourself to the [all-contributors file](https://github.com/sktime/sktime/blob/main/.all-contributorsrc) in the `sktime` root directory (not the `CONTRIBUTORS.md`). Common badges: `code` - fixing a bug, or adding code logic. `doc` - writing or improving documentation or docstrings. `bug` - reporting or diagnosing a bug (get this plus `code` if you also fixed the bug in the PR).`maintenance` - CI, test framework, release.
  See here for [full badge reference](https://allcontributors.org/docs/en/emoji-key)
- [ ] Optionally, for added estimators: I've added myself and possibly to the `maintainers` tag - do this if you want to become the owner or maintainer of an estimator you added.
  See here for further details on the [algorithm maintainer role](https://www.sktime.net/en/latest/get_involved/governance.html#algorithm-maintainers).
- [ x] The PR title starts with either [ENH], [MNT], [DOC], or [BUG]. [BUG] - bugfix, [MNT] - CI, test framework, [ENH] - adding or improving code, [DOC] - writing or improving documentation or docstrings.

##### For new estimators
- [ ] I've added the estimator to the API reference - in `docs/source/api_reference/taskname.rst`, follow the pattern.
- [ ] I've added one or more illustrative usage examples to the docstring, in a pydocstyle compliant `Examples` section.
- [ ] If the estimator relies on a soft dependency, I've set the `python_dependencies` tag and ensured
  dependency isolation, see the [estimator dependencies guide](https://www.sktime.net/en/latest/developer_guide/dependencies.html#adding-a-soft-dependency).

<!--
Thanks for contributing!
-->


ninedigits: @fkiraly I think there are some issues with other commits getting included here, but I think the core logic of `ExpandingCutoffSplitter` is still worth reviewing.
fkiraly: > @fkiraly I think there are some issues with other commits getting included here, but I think the core logic of `ExpandingCutoffSplitter` is still worth reviewing.

I don't see any other commits? All looks fine.

Further, could you kindly ensure your code passes formatting, then the automated tests will run.
Guide: https://www.sktime.net/en/stable/developer_guide/coding_standards.html
fkiraly: restarted tests
ninedigits: > Looks good!
> 
> Minor points:
> 
> * I think the `Parameters` docstring section has one indent too much
> * do you want to say a few words how the different index/offset combinations are interpreted, in the docstring preamble?
> * could you add at least one more test parameter set?

Done. 
fkiraly: seems like the docstring example is failing - looks like a minor thing
ninedigits: > seems like the docstring example is failing - looks like a minor thing

Ah, I think I fixed it now. 
fkiraly: The failure is unrelated, it is this sporadic problem: https://github.com/sktime/sktime/issues/6406
ninedigits: > The failure is unrelated, it is this sporadic problem: #6406

Oh, got it. Anything left to do on my end? 

Patch:
diff --git a/docs/source/api_reference/split.rst b/docs/source/api_reference/split.rst
index ee1b27fb7..742ce6c0b 100644
--- a/docs/source/api_reference/split.rst
+++ b/docs/source/api_reference/split.rst
@@ -47,6 +47,7 @@ They have tag ``"split_type"="temporal"``.
     SingleWindowSplitter
     SlidingWindowSplitter
     ExpandingWindowSplitter
+    ExpandingCutoffSplitter
     ExpandingGreedySplitter
     TemporalTrainTestSplitter
 
diff --git a/sktime/split/__init__.py b/sktime/split/__init__.py
index 012fc952a..6c1c2e630 100644
--- a/sktime/split/__init__.py
+++ b/sktime/split/__init__.py
@@ -3,6 +3,7 @@
 __all__ = [
     "CutoffSplitter",
     "CutoffFhSplitter",
+    "ExpandingCutoffSplitter",
     "ExpandingGreedySplitter",
     "ExpandingWindowSplitter",
     "InstanceSplitter",
@@ -15,6 +16,7 @@ __all__ = [
 ]
 
 from sktime.split.cutoff import CutoffFhSplitter, CutoffSplitter
+from sktime.split.expandingcutoff import ExpandingCutoffSplitter
 from sktime.split.expandinggreedy import ExpandingGreedySplitter
 from sktime.split.expandingwindow import ExpandingWindowSplitter
 from sktime.split.instance import InstanceSplitter
diff --git a/sktime/split/expandingcutoff.py b/sktime/split/expandingcutoff.py
new file mode 100644
index 000000000..c4ca15d9a
--- /dev/null
+++ b/sktime/split/expandingcutoff.py
@@ -0,0 +1,213 @@
+#!/usr/bin/env python3 -u
+# copyright: sktime developers, BSD-3-Clause License (see LICENSE file)
+"""Splitter that successively expands the training window."""
+
+__author__ = ["ninedigits"]
+
+__all__ = [
+    "ExpandingCutoffSplitter",
+]
+
+from typing import Optional
+
+import numpy as np
+import pandas as pd
+
+from sktime.split.base import BaseSplitter
+from sktime.split.base._common import ACCEPTED_Y_TYPES, _check_fh
+from sktime.utils.validation.forecasting import check_step_length
+
+
+def _is_datetimelike(x):
+    return isinstance(x, (pd.Timestamp, np.datetime64, pd.Period))
+
+
+def _is_int(arg):
+    return isinstance(arg, (int, np.int_))
+
+
+def _validate_cutoff(cutoff):
+    if not (_is_datetimelike(cutoff) or _is_int(cutoff)):
+        raise TypeError(
+            "Cutoff value must be datetime-like or integer but instead "
+            f"found type(cutoff) = {type(cutoff)}"
+        )
+    return cutoff
+
+
+class ExpandingCutoffSplitter(BaseSplitter):
+    """
+    Expanding cutoff splitter for time series data.
+
+    This splitter combines elements of `ExpandingWindowSplitter` and `CutoffSplitter`
+    to create training and testing sets. Unlike `ExpandingWindowSplitter` which begins
+    with a fixed initial window, this splitter uses a specific cutoff point as the
+    starting window for the training set. The training set then expands incrementally
+    in each split until it reaches the end of the series.
+
+    The test set is defined by a forecast horizon relative to the last point in the
+    training set, containing as many subsequent indices as specified by the `fh`
+    parameter.
+
+    The valid types of y-index and cutoff pairings are datelike-datelike, datelike-int,
+    and int-int. When a datelike index is combined with an int cutoff, the cutoff
+    functions as an iloc indexer. When an int index is paired with a positive int
+    cutoff, the cutoff serves as a loc indexer. If the int cutoff is negative, it
+    functions as an iloc indexer.
+                          c
+                          |
+    |---------------------|----fh----|------|
+    | * * * * * * * * * * x x x x x x - - - |
+    | * * * * * * * * * * * x x x x x x - - |
+    | * * * * * * * * * * * * x x x x x x - |
+    | * * * * * * * * * * * * * x x x x x x |
+
+    ``c`` = cutoff date or index.
+
+    ``*`` = training fold.
+
+    ``x`` = test fold.
+
+    Parameters
+    ----------
+    cutoff (int or pd.Timestamp):
+        The initial cutoff point in the series, which marks the beginning of the
+        first test set.
+    fh (int, list, or np.array):
+        Forecasting horizon, determining the size and  indices of the test sets.
+        It can be an integer, a list, or an array.
+    step_length (int):
+        The step length to expand the training set size in each split.
+
+    Examples
+    --------
+    >>> import pandas as pd
+    >>> from sktime.split import ExpandingCutoffSplitter
+    >>> date_range = pd.date_range(start='2020-Q1', end='2021-Q3', freq='QS')
+    >>> y = pd.DataFrame(index=pd.PeriodIndex(date_range, freq='Q'))
+    >>> cutoff = pd.Period('2021-Q1')
+    >>> cv = ExpandingCutoffSplitter(cutoff=cutoff, fh=[1, 2], step_length=1)
+    >>> list(cv.split(y)) # doctest: +SKIP
+    [(array([0, 1, 2, 3]), array([4, 5])), (array([0, 1, 2, 3, 4]), array([5, 6]))]
+    """
+
+    def __init__(self, cutoff, fh, step_length):
+        super().__init__()
+        self.cutoff = _validate_cutoff(cutoff)
+        self.fh = fh
+        self._fh = _check_fh(fh)
+        self.step_length = step_length
+        return
+
+    def _split(self, y, fh=None):
+        """
+        Generate indices to split data into training and testing sets.
+
+        Parameters
+        ----------
+        y : array-like, shape = [n_samples]
+            Time series data.
+        fh : int, default=None
+            Forecast horizon, if None, uses self.fh
+
+        Yields
+        ------
+        train : ndarray
+            The training set indices for that split.
+        test : ndarray
+            The testing set indices for that split.
+        """
+        if fh is None:
+            fh = self._fh
+        offset = fh.to_numpy().max() + 1
+        for cutoff in self.get_cutoffs(y):
+            train_window = np.arange(0, cutoff + 1, step=1)
+            test_window = np.arange(cutoff + 1, cutoff + offset, step=1)
+            yield train_window, test_window
+
+    def _validate_y(self, y):
+        y = self._coerce_to_index(y)
+        cutoff = self.cutoff
+        y0 = y[0]
+        cond1 = _is_datetimelike(y0) and (_is_datetimelike(cutoff) or _is_int(cutoff))
+        cond2 = _is_int(y0) and _is_int(cutoff)
+        if not (cond1 or cond2):
+            raise TypeError(
+                "Valid combinations for y and cutoff types are "
+                "datelike-datelike, datelike-int, or int-int, but instead "
+                f"found {type(y0)}-{type(cutoff)}"
+            )
+        return y
+
+    def _get_first_cutoff_index(self, y_index):
+        y0 = y_index[0]
+        if _is_int(self.cutoff) and self.cutoff < 0:
+            index = len(y_index) + self.cutoff - 1
+            if index < 0:
+                raise IndexError(
+                    f"Cutoff of value {self.cutoff} is out of bounds "
+                    f"for y of length {len(y_index)}"
+                )
+        elif _is_datetimelike(y0) and _is_int(self.cutoff) and (self.cutoff > 0):
+            index = self.cutoff - 1
+        else:
+            index = np.argmax(y_index == self.cutoff) - 1
+            if index == -1:
+                raise TypeError(
+                    "Could not find matching index, make sure that "
+                    f"type(y) {type(y0)} is compatible with type(cutoff) "
+                    f"{type(self.cutoff)}"
+                )
+        return index
+
+    def get_cutoffs(self, y: Optional[ACCEPTED_Y_TYPES] = None) -> np.ndarray:
+        """Return the cutoff points in .iloc[] context.
+
+        Parameters
+        ----------
+        y : pd.Series or pd.Index, optional (default=None)
+            Time series to split
+
+        Returns
+        -------
+        cutoffs : 1D np.ndarray of int
+            iloc location indices, in reference to y, of cutoff indices
+        """
+        if y is None:
+            raise ValueError(
+                f"{self.__class__.__name__} requires `y` to compute the cutoffs."
+            )
+        y = self._validate_y(y)
+        fh = self._fh
+        step_length = check_step_length(self.step_length)
+        cutoff_index = self._get_first_cutoff_index(y)
+        cutoffs = np.array([cutoff_index])
+        offset = fh.to_numpy().max()
+        while cutoff_index + offset < len(y) - 1:
+            cutoff_index += step_length
+            cutoffs = np.append(cutoffs, cutoff_index)
+        return cutoffs
+
+    @classmethod
+    def get_test_params(cls, parameter_set="default"):
+        """Return testing parameter settings for the splitter.
+
+        Parameters
+        ----------
+        parameter_set : str, default="default"
+            Name of the set of test parameters to return, for use in tests. If no
+            special parameters are defined for a value, will return ``"default"`` set.
+
+        Returns
+        -------
+        params : dict or list of dict, default = {}
+            Parameters to create testing instances of the class
+            Each dict are parameters to construct an "interesting" test instance, i.e.,
+            ``MyClass(**params)`` or ``MyClass(**params[i])`` creates a valid test
+            instance.
+            ``create_test_instance`` uses the first (or only) dictionary in ``params``
+        """
+        return [
+            {"fh": [2, 4], "cutoff": 3, "step_length": 1},
+            {"fh": [1, 2, 3], "cutoff": 3, "step_length": 2},
+        ]


Test patch:
diff --git a/sktime/split/tests/test_expandingcutoff.py b/sktime/split/tests/test_expandingcutoff.py
new file mode 100644
index 000000000..7c7133120
--- /dev/null
+++ b/sktime/split/tests/test_expandingcutoff.py
@@ -0,0 +1,213 @@
+# copyright: sktime developers, BSD-3-Clause License (see LICENSE file)
+"""Tests for expanding cutoff splitter."""
+
+import numpy as np
+import pandas as pd
+import pytest
+
+from sktime.forecasting.base import ForecastingHorizon
+from sktime.split import ExpandingCutoffSplitter
+from sktime.split.tests.test_split import _check_cv
+from sktime.utils._testing.hierarchical import _make_hierarchical
+from sktime.utils._testing.series import _make_series
+
+
+def test_expandingcutoff_datelike_index_001():
+    """Test datetime index with _check_cv"""
+    y = _make_series(n_timepoints=10, random_state=42)
+    cutoff = y.index[3]
+    fh = ForecastingHorizon([1, 2, 3], freq=y.index.freq)
+    cv = ExpandingCutoffSplitter(cutoff=cutoff, fh=fh, step_length=1)
+    train_windows, test_windows, cutoffs, n_splits = _check_cv(cv, y)
+    np.testing.assert_array_equal(cutoffs, cv.get_cutoffs(y))
+
+
+def test_expandingcutoff_int_index_002():
+    """Test int index with _check_cv"""
+    y = _make_series(n_timepoints=10, index_type="int", random_state=42)
+    cutoff = y.index[3]
+    fh = ForecastingHorizon([1, 2, 3])
+    cv = ExpandingCutoffSplitter(cutoff=cutoff, fh=fh, step_length=1)
+    train_windows, test_windows, cutoffs, n_splits = _check_cv(cv, y)
+    np.testing.assert_array_equal(cutoffs, cv.get_cutoffs(y))
+
+
+def test_expandingcutoff_ytype_cutofftype_combos_003a():
+    """Test invalid param combo"""
+    # Datetime cutoff
+    i = 3
+    y1 = _make_series(n_timepoints=10, index_type="datetime", random_state=42)
+    fh = ForecastingHorizon([1, 2, 3])
+    cutoffs = [-7, y1.index[i], 5]
+    step_lengths = [1, 2, 3]
+    for cutoff in cutoffs:
+        for step_length in step_lengths:
+            cv1 = ExpandingCutoffSplitter(cutoff=cutoff, fh=fh, step_length=step_length)
+            _check_cv(cv1, y1)
+
+
+def test_expandingcutoff_ytype_cutofftype_combos_003b():
+    """Test invalid param combo"""
+    # Datetime cutoff
+    y1 = _make_series(n_timepoints=10, index_type="int", random_state=42)
+    fh = ForecastingHorizon([1, 2, 3])
+    cutoffs = [-7, y1.index[3]]
+    step_lengths = [1, 2, 3]
+    for cutoff in cutoffs:
+        for step_length in step_lengths:
+            cv1 = ExpandingCutoffSplitter(cutoff=cutoff, fh=fh, step_length=step_length)
+            _check_cv(cv1, y1)
+
+
+def test_expandingcutoff_ytype_cutofftype_combos_003c():
+    """Test invalid param combo"""
+    # Datetime cutoff
+    y1 = _make_series(n_timepoints=10, index_type="int", random_state=42)
+    fh = ForecastingHorizon([1, 2, 3])
+    # Y-index is int but cutoff is datetime
+    with pytest.raises(TypeError):
+        cv1 = ExpandingCutoffSplitter(
+            cutoff=pd.Timestamp("2000-01-01"), fh=fh, step_length=1
+        )
+        _check_cv(cv1, y1)
+
+
+def test_expandingcutoff_splitloc_004():
+    """Test split loc"""
+    y = _make_series(n_timepoints=10, random_state=42)
+    cutoff = y.index[3]
+    fh = ForecastingHorizon([1, 2, 3], freq=y.index.freq)
+    cv = ExpandingCutoffSplitter(cutoff=cutoff, fh=fh, step_length=1)
+    train_test_values = list(cv.split_loc(y))
+    last_train, last_test = train_test_values[-1]
+    expected_last_train = y.index[0:7]
+    expected_last_test = y.index[7:]
+    test_cases = [(expected_last_train, last_train), (expected_last_test, last_test)]
+    for expected, actual in test_cases:
+        np.testing.assert_array_equal(expected.values, actual.values)
+
+
+def test_expandingcutoff_hiearchical_splitloc_005():
+    """Test hiearchical splitloc with datetime"""
+    y = _make_hierarchical(
+        min_timepoints=6,
+        max_timepoints=10,
+        same_cutoff=True,
+        hierarchy_levels=(1, 2),
+        index_type="datetime",
+        random_state=42,
+    )
+    y_index = y.index.get_level_values(-1)
+    cutoff_index = -4
+    cutoff = y_index[cutoff_index]
+    fh = ForecastingHorizon([1, 2, 3])
+    cv = ExpandingCutoffSplitter(cutoff=cutoff, fh=fh, step_length=1)
+
+    train_test = list(cv.split_loc(y))
+
+    y_train_expected = (
+        y.groupby(level=-2)
+        .apply(lambda row: row[0:cutoff_index])
+        .index.get_level_values(-1)
+    )
+    y_test_expected = (
+        y.groupby(level=-2)
+        .apply(lambda row: row[cutoff_index : cutoff_index + fh[-1]])
+        .index.get_level_values(-1)
+    )
+    pd.testing.assert_index_equal(
+        y_train_expected, train_test[0][0].get_level_values(-1)
+    )
+    pd.testing.assert_index_equal(
+        y_test_expected, train_test[0][1].get_level_values(-1)
+    )
+
+
+def test_expandingcutoff_hiearchical_forecastbylevel_006():
+    """Test hiearchical with forecast by level"""
+    from sktime.forecasting.compose import ForecastByLevel
+    from sktime.forecasting.model_selection import ForecastingGridSearchCV
+    from sktime.forecasting.naive import NaiveForecaster
+    from sktime.utils._testing.hierarchical import _make_hierarchical
+
+    y = _make_hierarchical(
+        min_timepoints=6,
+        max_timepoints=10,
+        same_cutoff=True,
+        hierarchy_levels=(1, 2),
+        index_type="datetime",
+        random_state=42,
+    )
+    y_index = y.index.get_level_values(-1)
+    last_date = y_index.max()
+    cutoff_index = -4
+    cutoff = y_index[cutoff_index]
+    fh = ForecastingHorizon([1, 2, 3])
+    cv = ExpandingCutoffSplitter(cutoff=cutoff, fh=fh, step_length=1)
+    forecaster = NaiveForecaster()
+    param_grid = {"strategy": ["last", "mean", "drift"]}
+    gscv = ForecastByLevel(
+        ForecastingGridSearchCV(forecaster=forecaster, param_grid=param_grid, cv=cv)
+    )
+    gscv.fit(y, fh=fh)
+    y_pred = gscv.predict()
+    expected_last_forecast_date = last_date + pd.Timedelta(days=fh[-1])
+    actual_last_forecast_date = y_pred.index.max()
+    assert expected_last_forecast_date, actual_last_forecast_date
+
+
+def test_expandingcutoff_fh_list_007():
+    """Test fh as list with _check_cv"""
+    y = _make_series(n_timepoints=10, random_state=42)
+    cutoff = y.index[3]
+    fh = [1, 2, 3]
+    cv = ExpandingCutoffSplitter(cutoff=cutoff, fh=fh, step_length=1)
+    train_windows, test_windows, cutoffs, n_splits = _check_cv(cv, y)
+    np.testing.assert_array_equal(cutoffs, cv.get_cutoffs(y))
+
+
+def test_expanding_cutoff_period_008():
+    date_range = pd.date_range(
+        start=pd.Timestamp("2020-Q1"), end=pd.Timestamp("2021-Q4"), freq="QS"
+    )
+    y = pd.DataFrame({"date": date_range})
+    y = y.reset_index()
+    y = y.set_index("date")
+    y.index = pd.PeriodIndex(y.index, freq="Q")
+
+    cutoff = pd.Period("2021-Q1")
+    fh = ForecastingHorizon([1, 2, 3])
+    cv = ExpandingCutoffSplitter(cutoff=cutoff, fh=fh, step_length=2)
+    _check_cv(cv, y)
+
+    # pandas periods and timestamps are both datelike but don't work directly
+    cutoff = pd.Timestamp("2021-Q1")
+    with pytest.raises(TypeError):
+        cv = ExpandingCutoffSplitter(cutoff=cutoff, fh=fh, step_length=1)
+        _check_cv(cv, y)
+
+
+def _make_splits_listlike(splits):
+    splits_new = []
+    for train, test in splits:
+        splits_new.append([train.tolist(), test.tolist()])
+    return splits_new
+
+
+def test_expanding_cutoff_docstring_examples():
+    date_range = pd.date_range(start="2020-Q1", end="2021-Q3", freq="QS")
+    y = pd.DataFrame(index=pd.PeriodIndex(date_range, freq="Q"))
+    cv1 = ExpandingCutoffSplitter(cutoff=pd.Period("2021-Q1"), fh=[1, 2], step_length=1)
+    splits1 = list(cv1.split(y))
+
+    cv2 = ExpandingCutoffSplitter(cutoff=4, fh=[1, 2], step_length=1)
+    splits2 = list(cv2.split(y))
+
+    cv3 = ExpandingCutoffSplitter(cutoff=-3, fh=[1, 2], step_length=1)
+    splits3 = list(cv3.split(y))
+
+    assert (
+        _make_splits_listlike(splits1)
+        == _make_splits_listlike(splits2)
+        == _make_splits_listlike(splits3)
+    )