Repository: sktime/sktime
Pull request #5542 (2023-11-07 13:01:29)

[BUG] in reduced regressor, copy or truncate `X` if it does not fit the forecasting horizon.
<!--
Welcome to sktime, and thanks for contributing!
Please have a look at our contribution guide:
https://www.sktime.net/en/latest/get_involved/contributing.html
-->

#### Reference Issues/PRs
fixes partly #5127 


#### What does this implement/fix? Explain your changes.
If `X` is too small or to large, it copies the last value of `X` or truncates `X`.

#### What should a reviewer concentrate their feedback on?
Does it really match the interface to copy the last values of `X` or is this confusing?

#### Did you add any tests for the change?

I added tests to check if non-matching `X` still lead to forecasts with the correct shape

##### For all contributions
- [X] I've added myself to the [list of contributors](https://github.com/sktime/sktime/blob/main/CONTRIBUTORS.md) with any new badges I've earned :-)
  How to: add yourself to the [all-contributors file](https://github.com/sktime/sktime/blob/main/.all-contributorsrc) in the `sktime` root directory (not the `CONTRIBUTORS.md`). Common badges: `code` - fixing a bug, or adding code logic. `doc` - writing or improving documentation or docstrings. `bug` - reporting or diagnosing a bug (get this plus `code` if you also fixed the bug in the PR).`maintenance` - CI, test framework, release.
  See here for [full badge reference](https://allcontributors.org/docs/en/emoji-key)
- [ ] Optionally, I've added myself and possibly others to the [CODEOWNERS](https://github.com/sktime/sktime/blob/main/CODEOWNERS) file - do this if you want to become the owner or maintainer of an estimator you added.
  See here for further details on the [algorithm maintainer role](https://www.sktime.net/en/latest/get_involved/governance.html#algorithm-maintainers).
- [X] The PR title starts with either [ENH], [MNT], [DOC], or [BUG]. [BUG] - bugfix, [MNT] - CI, test framework, [ENH] - adding or improving code, [DOC] - writing or improving documentation or docstrings.



Patch:
diff --git a/.github/workflows/validate_datasets.yml b/.github/workflows/validate_datasets.yml
index ece655708..d7eabb9b3 100644
--- a/.github/workflows/validate_datasets.yml
+++ b/.github/workflows/validate_datasets.yml
@@ -1,12 +1,8 @@
 name: datasets validation
 on:
-  pull_request:
-    branches:
-      - main
-    paths:
-      - sktime/datasets/**
   schedule:
     - cron: 0 0 1 * *
+  workflow_call:
   workflow_dispatch:
 jobs:
   test_internal_data:
diff --git a/.github/workflows/validate_extras.yml b/.github/workflows/validate_extras.yml
index 1f0f82259..69d205af7 100644
--- a/.github/workflows/validate_extras.yml
+++ b/.github/workflows/validate_extras.yml
@@ -1,12 +1,8 @@
 name: extras validation
 on:
-  pull_request:
-    branches:
-      - main
-    paths:
-      - pyproject.toml
   schedule:
     - cron: 0 0 1 * *
+  workflow_call:
   workflow_dispatch:
 jobs:
   test_installations:
diff --git a/sktime/forecasting/compose/_reduce.py b/sktime/forecasting/compose/_reduce.py
index cde9fe614..1fa634987 100644
--- a/sktime/forecasting/compose/_reduce.py
+++ b/sktime/forecasting/compose/_reduce.py
@@ -901,15 +901,27 @@ class _RecursiveReducer(_Reducer):
             fh_max = fh.to_relative(self.cutoff)[-1]
 
             y_pred = np.zeros(fh_max)
+
+            # Array with input data for prediction.
             last = np.zeros((1, n_columns, window_length + fh_max))
 
             # Fill pre-allocated arrays with available data.
             last[:, 0, :window_length] = y_last
             if X is not None:
-                last[:, 1:, :window_length] = X_last.T
-                last[:, 1:, window_length:] = X.iloc[
-                    -(last.shape[2] - window_length) :, :
-                ].T
+                X_to_use = np.concatenate(
+                    [X_last.T, X.iloc[-(last.shape[2] - window_length) :, :].T], axis=1
+                )
+                if X_to_use.shape[1] < window_length + fh_max:
+                    X_to_use = np.pad(
+                        X_to_use,
+                        ((0, 0), (0, window_length + fh_max - X_to_use.shape[1])),
+                        "edge",
+                    )
+                elif X_to_use.shape[1] > window_length + fh_max:
+                    X_to_use = X_to_use[:, : window_length + fh_max]
+                # else X_to_use.shape[1] == window_length + fh_max
+                # and there are no additional steps to take
+                last[:, 1:] = X_to_use
 
             # Recursively generate predictions by iterating over forecasting horizon.
             for i in range(fh_max):


Test patch:
diff --git a/.github/workflows/test_all.yml b/.github/workflows/test_all.yml
index 2afe9d65a..e00cd1265 100644
--- a/.github/workflows/test_all.yml
+++ b/.github/workflows/test_all.yml
@@ -47,7 +47,7 @@ jobs:
           python-version-identifier: ${{ matrix.python-version }}
           sub-sample-estimators: "False"
           test-affected-estimators: "False"
-  test_components:
+  test_components_with_extras:
     needs: code_quality
     name: test module
     strategy:
@@ -84,3 +84,39 @@ jobs:
           python-version-identifier: ${{ matrix.python-version }}
           sub-sample-estimators: "False"
           test-affected-estimators: "False"
+  test_components_without_extras:
+    needs: code_quality
+    name: test all components without extras
+    strategy:
+      fail-fast: false
+      matrix:
+        python-version:
+          - "3.8"
+          - "3.9"
+          - "3.10"
+          - "3.11"
+          - "3.12"
+        operating-system:
+          - macos-latest
+          - ubuntu-latest
+          - windows-latest
+    runs-on: ${{ matrix.operating-system }}
+    steps:
+      - name: checkout pull request branch
+        uses: actions/checkout@v4
+      - name: update local git tracking reference
+        run: git remote set-branches origin main
+      - name: update local shallow clone
+        run: git fetch --depth 1
+      - name: create python virtual environment
+        uses: actions/setup-python@v4
+        with:
+          python-version: ${{ matrix.python-version }}
+      - name: install core, test and all soft dependencies
+        run: python3 -m pip install .[all_extras_pandas2,tests]
+      - name: run unit tests
+        run: python3 -m pytest sktime --ignore sktime/alignment --ignore sktime/annotation --ignore sktime/classification --ignore sktime/clustering --ignore sktime/datasets --ignore sktime/forecasting --ignore sktime/networks --ignore sktime/param_est --ignore sktime/regression --ignore sktime/transformations --matrixdesign False --only_changed_modules False
+      - name: test coverage
+        uses: codecov/codecov-action@v3
+        with:
+          flags: ${{ matrix.python-version }},components_without_extras
diff --git a/.github/workflows/test_code_quality.yml b/.github/workflows/test_code_quality.yml
index 20b2eb76b..8e2b60ee1 100644
--- a/.github/workflows/test_code_quality.yml
+++ b/.github/workflows/test_code_quality.yml
@@ -24,6 +24,28 @@ jobs:
       - name: check missing __init__ files
         run: build_tools/fail_on_missing_init_files.sh
         shell: bash
+  detect:
+    name: detect changes
+    runs-on: ubuntu-latest
+    permissions:
+      pull-requests: read
+    outputs:
+      components: ${{ steps.filter.outputs.changes }}
+    steps:
+      - uses: dorny/paths-filter@v2
+        id: filter
+        with:
+          filters: |
+            datasets:
+              - sktime/base/**
+              - sktime/datasets/**
+            pyproject:
+              - pyproject.toml
+  test_install_extras:
+    needs: detect
+    name: validate extra installs
+    if: ${{ needs.detect.outputs.pyproject == 'true' }}
+    uses: ./.github/workflows/validate_extras.yml
   unit_test_base:
     needs: code_quality
     name: run unit tests for base framework
@@ -32,3 +54,8 @@ jobs:
     needs: code_quality
     name: run unit tests for individual components
     uses: ./.github/workflows/test_components.yml
+  unit_test_datasets:
+    needs: detect
+    name: run unit tests for datasets
+    if: ${{ needs.detect.outputs.datasets == 'true' }}
+    uses: ./.github/workflows/validate_datasets.yml
diff --git a/.github/workflows/test_components.yml b/.github/workflows/test_components.yml
index ea91af5b9..417c34f03 100644
--- a/.github/workflows/test_components.yml
+++ b/.github/workflows/test_components.yml
@@ -15,22 +15,31 @@ jobs:
         with:
           filters: |
             alignment:
+              - sktime/base/**
               - sktime/alignment/**
             annotation:
+              - sktime/base/**
               - sktime/annotation/**
             classification:
+              - sktime/base/**
               - sktime/classification/**
             clustering:
+              - sktime/base/**
               - sktime/clustering/**
             forecasting:
+              - sktime/base/**
               - sktime/forecasting/**
             networks:
+              - sktime/base/**
               - sktime/networks/**
             param_est:
+              - sktime/base/**
               - sktime/param_est/**
             regression:
+              - sktime/base/**
               - sktime/regression/**
             transformations:
+              - sktime/base/**
               - sktime/transformations/**
   test:
     needs: detect
@@ -48,7 +57,7 @@ jobs:
           - macos-latest
           - ubuntu-latest
           - windows-latest
-        sktime-component: ${{ fromJSON(needs.detect.outputs.components) }}
+        sktime-component: ${{ fromJSON(needs.detect_components_with_extras.outputs.components) }}
     runs-on: ${{ matrix.operating-system }}
     steps:
       - name: checkout pull request branch
@@ -60,3 +69,67 @@ jobs:
           python-version-identifier: ${{ matrix.python-version }}
           sub-sample-estimators: "True"
           test-affected-estimators: "True"
+  detect_components_without_extras:
+    name: check for changes in individual components without extras
+    runs-on: ubuntu-latest
+    permissions:
+      pull-requests: read
+    outputs:
+      components_without_extras: ${{ steps.filter.outputs.components_without_extras }}
+    steps:
+      - uses: dorny/paths-filter@v2
+        id: filter
+        with:
+          filters: |
+            components_without_extras:
+              - sktime/base/**
+              - sktime/_contrib/**
+              - sktime/benchmarking/**
+              - sktime/datatypes/**
+              - sktime/distances/**
+              - sktime/dists_kernels/**
+              - sktime/performance_metrics/**
+              - sktime/pipeline/**
+              - sktime/proba/**
+              - sktime/registry/**
+              - sktime/series_as_features/**
+              - sktime/split/**
+              - sktime/tests/**
+              - sktime/utils/**
+  test_components_without_extras:
+    needs: detect_components_without_extras
+    name: test all components without extras
+    if: ${{ needs.detect.outputs.components_without_extras == 'true' }}
+    strategy:
+      fail-fast: false
+      matrix:
+        python-version:
+          - "3.8"
+          - "3.9"
+          - "3.10"
+          - "3.11"
+          - "3.12"
+        operating-system:
+          - macos-latest
+          - ubuntu-latest
+          - windows-latest
+    runs-on: ${{ matrix.operating-system }}
+    steps:
+      - name: checkout pull request branch
+        uses: actions/checkout@v4
+      - name: update local git tracking reference
+        run: git remote set-branches origin main
+      - name: update local shallow clone
+        run: git fetch --depth 1
+      - name: create python virtual environment
+        uses: actions/setup-python@v4
+        with:
+          python-version: ${{ matrix.python-version }}
+      - name: install core, test and all soft dependencies
+        run: python3 -m pip install .[all_extras_pandas2,tests]
+      - name: run unit tests
+        run: python3 -m pytest sktime --ignore sktime/alignment --ignore sktime/annotation --ignore sktime/classification --ignore sktime/clustering --ignore sktime/datasets --ignore sktime/forecasting --ignore sktime/networks --ignore sktime/param_est --ignore sktime/regression --ignore sktime/transformations --matrixdesign True --only_changed_modules True
+      - name: test coverage
+        uses: codecov/codecov-action@v3
+        with:
+          flags: ${{ matrix.python-version }},components_without_extras
diff --git a/sktime/forecasting/compose/tests/test_reduce.py b/sktime/forecasting/compose/tests/test_reduce.py
index 13758a2e7..26b980e4f 100644
--- a/sktime/forecasting/compose/tests/test_reduce.py
+++ b/sktime/forecasting/compose/tests/test_reduce.py
@@ -588,3 +588,28 @@ def test_direct_vs_recursive():
     assert pred_dir_max.head(1).equals(pred_rec_max.head(1))
     assert pred_dir_max.head(1).equals(pred_rec_spec.head(1))
     assert not pred_dir_max.head(1).equals(pred_dir_spec.head(1))
+
+
+def test_recursive_reducer_X_not_fit_to_fh():
+    """Test recursive reducer with X that do not fit the fh.
+
+    I.e., either X is longer or smaller than max_fh
+    """
+    y = load_airline()
+    y_train, y_test = temporal_train_test_split(y)
+    X_train = y_train
+    X_test = y_test
+
+    forecaster = make_reduction(
+        LinearRegression(), window_length=2, strategy="recursive"
+    )
+    forecaster.fit(y_train, X_train)
+
+    pred1 = forecaster.predict(X=X_test[:1], fh=[1, 2, 3])
+    assert pred1.shape == (3,)
+    pred2 = forecaster.predict(X=X_test[:2], fh=[1, 2, 3])
+    assert pred2.shape == (3,)
+    pred3 = forecaster.predict(X=X_test[:3], fh=[1, 2, 3])
+    assert pred3.shape == (3,)
+    pred4 = forecaster.predict(X=X_test, fh=[1])
+    assert pred4.shape == (1,)