Repository: sktime/sktime

Issue:
[BUG] `ForecastingGridSearchCV` does not save `best_forecaster_`, `cv_results_` and other properties for multivariate time series
**Describe the bug**
<!--
A clear and concise description of what the bug is.
-->
`ForecastingGridSearchCV` does not save `best_forecaster_`, `cv_results_` and other properties for multivariate time series.
**To Reproduce**
<!--
Add a Minimal, Complete, and Verifiable example (for more details, see e.g. https://stackoverflow.com/help/mcve

If the code is too long, feel free to put it in a public gist and link it in the issue: https://gist.github.com
-->
This works:
```python
y = pd.DataFrame(index=pd.date_range("2021-01-01", "2021-01-10"),
                 data={"a": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10], 
                       "b": [-1, -2, -3, -4, -5, -6, -7, -8, -9, -10]})
forecasters = NaiveForecaster(window_length=5)

cv = ExpandingWindowSplitter(fh=2, initial_window=5, step_length=1)
forecaster_param_grid = {"strategy": ["mean", "drift"]}

pipeline = ForecastingGridSearchCV(
    forecasters, cv=cv,
    param_grid=forecaster_param_grid,
    error_score='raise'
)
display(pipeline.fit(y[["a"]]).predict(fh=np.arange(1, 5)))
display(pipeline.best_params_)
display(pipeline.cv_results_)
```
<img width="1438" alt="image" src="https://github.com/sktime/sktime/assets/121803583/dff16974-dc07-497f-89a5-b18927780097">

This does not:
```python
y = pd.DataFrame(index=pd.date_range("2021-01-01", "2021-01-10"),
                 data={"a": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10], 
                       "b": [-1, -2, -3, -4, -5, -6, -7, -8, -9, -10]})
forecasters = NaiveForecaster(window_length=5)

cv = ExpandingWindowSplitter(fh=2, initial_window=5, step_length=1)
forecaster_param_grid = {"strategy": ["mean", "drift"]}

pipeline = ForecastingGridSearchCV(
    forecasters, cv=cv,
    param_grid=forecaster_param_grid,
    error_score='raise'
)
display(pipeline.fit(y).predict(fh=np.arange(1, 5))). # change here
display(pipeline.best_params_)
display(pipeline.cv_results_)
```
Fit is performed well, but the properties are not available
<img width="678" alt="image" src="https://github.com/sktime/sktime/assets/121803583/73229209-6a2b-4831-9d85-b3f3c555ebff">


**Expected behavior**
<!--
A clear and concise description of what you expected to happen.
-->
`best_params_`. `cv_results_` etc are available for multivariate series too.

**Versions**
<details>

<!--
Please run the following code snippet and paste the output here:

from sktime import show_versions; show_versions()
-->
System:
    python: 3.8.13 (default, Oct 19 2022, 17:54:22)  [Clang 12.0.0 ]
executable: /Users/grigorii/opt/miniconda3/envs/hcem_test/bin/python
   machine: macOS-10.16-x86_64-i386-64bit

Python dependencies:
          pip: 22.3
       sktime: 0.22.0
      sklearn: 1.0.2
       skbase: 0.4.6
        numpy: 1.24.4
        scipy: 1.9.3
       pandas: 1.4.4
   matplotlib: 3.5.3
       joblib: 1.2.0
        numba: 0.57.1
  statsmodels: 0.13.4
     pmdarima: 1.8.5
statsforecast: 1.5.0
      tsfresh: None
      tslearn: None
        torch: None
   tensorflow: None
tensorflow_probability: None
</details>

<!-- Thanks for contributing! -->
Thanks a lot for such an awesome package!


Discussion:
Hm, this is an interesting one - I am not sure whether this is a bug.

Currently, the tuning forecasters vectorize over columns (variables) if and only if the wrapped forecaster does.

That is, the tuning happens per variable, if the forecaster inside is also applied per variable.

The best parameter is per column (variable), and can be found in the attribute `forecasters_`, which is a `pd.DataFrame` with the tuned forecasters being entries - these have `best_params_` etc.

Now, the question is - is this expected or counterintuitive? That is, should a direct application of grid search always select the same parameter for all columns, and have a single `best_params_` attribute?

(if it's counterintuitive, it would be a bug, or at least bug-like)
@fkiraly thanks a lot for a quick answer!

I do AutoML with `MultiplexForecaster`. My expectation is that when I do so, the same model will be chosen for all the components of the multivariate time series. For example, if I choose between `NaiveForecaster(window_length=5, strategy="mean")` and `Prophet()` I want to see both components behaving either very simple or rather complex. Also, I expected the same interface for univariate and multivariate series. But now I also see your reasoning, and it might indeed be beneficial to find various hyperparameter combinations for various components in some scenarios.

My suggestion would be to have a separate input (`global_param_grid`?) for hyperparameters to be selected globally, and leave `param_grid` to be selected per-series as it is happening now.

Alternatively, add a field to CV classes which would set one or another behavior, with the current one being the default.

I see, @SmirnGregHM, that's a pretty strong argument - interface consistency, or, more precisely, the current setup violates it. This leads to user surprisal or user frustration even, as a strong user expectation is interface consistency (e.g., the assumption that `best_params_` always exists should be safe to make).

I would agree with your argumentation in favour of making the interface uniform, and to ensure that columns behave like rows - same parameter everywhere.

I also want to ping @yarnabrina who is also working on autoML use cases.

Now, regarding the solution, I think adding more parameters to the tuners is a solution, but not the best solution. Instead, users who want to tune columns independently would stick the grid search estimator into `ColumnEnsembleForecaster` - no additional arguments needed to any existing estimator.
for the bugfixing board: I now agree that this is a bug.

Resolving patch:
diff --git a/sktime/forecasting/model_selection/_tune.py b/sktime/forecasting/model_selection/_tune.py
--- a/sktime/forecasting/model_selection/_tune.py
+++ b/sktime/forecasting/model_selection/_tune.py
@@ -2,7 +2,7 @@
 # copyright: sktime developers, BSD-3-Clause License (see LICENSE file)
 """Implements grid search functionality to tune forecasters."""
 
-__author__ = ["mloning"]
+__author__ = ["mloning", "fkiraly", "aiwalter"]
 __all__ = [
     "ForecastingGridSearchCV",
     "ForecastingRandomizedSearchCV",
@@ -11,6 +11,7 @@
 
 from collections.abc import Sequence
 from typing import Dict, List, Optional, Union
+from warnings import warn
 
 import numpy as np
 import pandas as pd
@@ -36,6 +37,8 @@ class BaseGridSearch(_DelegatedForecaster):
         "capability:pred_int:insample": True,
     }
 
+    # todo 0.24.0: replace all tune_by_variable defaults in this file with False
+    # remove deprecation message in BaseGridSearch.__init__
     def __init__(
         self,
         forecaster,
@@ -50,6 +53,8 @@ def __init__(
         return_n_best_forecasters=1,
         update_behaviour="full_refit",
         error_score=np.nan,
+        tune_by_instance=False,
+        tune_by_variable=None,
     ):
         self.forecaster = forecaster
         self.cv = cv
@@ -63,13 +68,30 @@ def __init__(
         self.return_n_best_forecasters = return_n_best_forecasters
         self.update_behaviour = update_behaviour
         self.error_score = error_score
+        self.tune_by_instance = tune_by_instance
+        self.tune_by_variable = tune_by_variable
+
         super().__init__()
+
+        # todo 0.24.0: remove this
+        if tune_by_variable is None:
+            warn(
+                f"in {self.__class__.__name__}, the default for tune_by_variable "
+                "will change from True to False in 0.24.0. "
+                "This will tune one parameter setting for all variables, while "
+                "currently it tunes one parameter per variable. "
+                "In order to maintain the current behaviour, ensure to set "
+                "the parameter tune_by_variable to True explicitly before upgrading "
+                "to version 0.24.0.",
+                DeprecationWarning,
+            )
+            tune_by_variable = True
+
         tags_to_clone = [
             "requires-fh-in-fit",
             "capability:pred_int",
             "capability:pred_int:insample",
             "capability:insample",
-            "scitype:y",
             "ignores-exogeneous-X",
             "handles-missing-data",
             "y_inner_mtype",
@@ -81,6 +103,11 @@ def __init__(
         self._extend_to_all_scitypes("y_inner_mtype")
         self._extend_to_all_scitypes("X_inner_mtype")
 
+        # this ensures univariate broadcasting over variables
+        # if tune_by_variable is True
+        if tune_by_variable:
+            self.set_tags(**{"scitype:y": "univariate"})
+
     # attribute for _DelegatedForecaster, which then delegates
     #     all non-overridden methods are same as of getattr(self, _delegate_name)
     #     see further details in _DelegatedForecaster docstring
@@ -91,6 +118,11 @@ def _extend_to_all_scitypes(self, tagname):
 
         Mutates self tag with name `tagname`.
         If no mtypes are present of a time series scitype, adds a pandas based one.
+        If only univariate pandas scitype is present for Series ("pd.Series"),
+        also adds the multivariate one ("pd.DataFrame").
+
+        If tune_by_instance is True, only Series mtypes are added,
+        and potentially present Panel or Hierarchical mtypes are removed.
 
         Parameters
         ----------
@@ -104,12 +136,22 @@ def _extend_to_all_scitypes(self, tagname):
         if not isinstance(tagval, list):
             tagval = [tagval]
         scitypes = mtype_to_scitype(tagval, return_unique=True)
+        # if no Series mtypes are present, add pd.DataFrame
         if "Series" not in scitypes:
             tagval = tagval + ["pd.DataFrame"]
+        # ensure we have a Series mtype capable of multivariate
+        elif "pd.Series" in tagval and "pd.DataFrame" not in tagval:
+            tagval = ["pd.DataFrame"] + tagval
+        # if no Panel mtypes are present, add pd.DataFrame based one
         if "Panel" not in scitypes:
             tagval = tagval + ["pd-multiindex"]
+        # if no Hierarchical mtypes are present, add pd.DataFrame based one
         if "Hierarchical" not in scitypes:
             tagval = tagval + ["pd_multiindex_hier"]
+
+        if self.tune_by_instance:
+            tagval = [x for x in tagval if mtype_to_scitype(x) == "Series"]
+
         self.set_tags(**{tagname: tagval})
 
     def _get_fitted_params(self):
@@ -418,6 +460,22 @@ class ForecastingGridSearchCV(BaseGridSearch):
         Value to assign to the score if an exception occurs in estimator fitting. If set
         to "raise", the exception is raised. If a numeric value is given,
         FitFailedWarning is raised.
+    tune_by_instance : bool, optional (default=False)
+        Whether to tune parameter by each time series instance separately,
+        in case of Panel or Hierarchical data passed to the tuning estimator.
+        Only applies if time series passed are Panel or Hierarchical.
+        If True, clones of the forecaster will be fit to each instance separately,
+        and are available in fields of the forecasters_ attribute.
+        Has the same effect as applying ForecastByLevel wrapper to self.
+        If False, the same best parameter is selected for all instances.
+    tune_by_variable : bool, optional (default=True)
+        Whether to tune parameter by each time series variable separately,
+        in case of multivariate data passed to the tuning estimator.
+        Only applies if time series passed are strictly multivariate.
+        If True, clones of the forecaster will be fit to each variable separately,
+        and are available in fields of the forecasters_ attribute.
+        Has the same effect as applying ColumnEnsembleForecaster wrapper to self.
+        If False, the same best parameter is selected for all variables.
 
     Attributes
     ----------
@@ -441,6 +499,12 @@ class ForecastingGridSearchCV(BaseGridSearch):
     n_best_scores_: list of float
         The scores of n_best_forecasters_ sorted from best to worst
         score of forecasters
+    forecasters_ : pd.DataFramee
+        DataFrame with all fitted forecasters and their parameters.
+        Only present if tune_by_instance=True or tune_by_variable=True,
+        and at least one of the two is applicable.
+        In this case, the other attributes are not present in self,
+        only in the fields of forecasters_.
 
     Examples
     --------
@@ -520,6 +584,8 @@ def __init__(
         backend="loky",
         update_behaviour="full_refit",
         error_score=np.nan,
+        tune_by_instance=False,
+        tune_by_variable=None,
     ):
         super().__init__(
             forecaster=forecaster,
@@ -534,6 +600,8 @@ def __init__(
             backend=backend,
             update_behaviour=update_behaviour,
             error_score=error_score,
+            tune_by_instance=tune_by_instance,
+            tune_by_variable=tune_by_variable,
         )
         self.param_grid = param_grid
 
@@ -701,6 +769,22 @@ class ForecastingRandomizedSearchCV(BaseGridSearch):
         Value to assign to the score if an exception occurs in estimator fitting. If set
         to "raise", the exception is raised. If a numeric value is given,
         FitFailedWarning is raised.
+    tune_by_instance : bool, optional (default=False)
+        Whether to tune parameter by each time series instance separately,
+        in case of Panel or Hierarchical data passed to the tuning estimator.
+        Only applies if time series passed are Panel or Hierarchical.
+        If True, clones of the forecaster will be fit to each instance separately,
+        and are available in fields of the forecasters_ attribute.
+        Has the same effect as applying ForecastByLevel wrapper to self.
+        If False, the same best parameter is selected for all instances.
+    tune_by_variable : bool, optional (default=True)
+        Whether to tune parameter by each time series variable separately,
+        in case of multivariate data passed to the tuning estimator.
+        Only applies if time series passed are strictly multivariate.
+        If True, clones of the forecaster will be fit to each variable separately,
+        and are available in fields of the forecasters_ attribute.
+        Has the same effect as applying ColumnEnsembleForecaster wrapper to self.
+        If False, the same best parameter is selected for all variables.
 
     Attributes
     ----------
@@ -718,6 +802,12 @@ class ForecastingRandomizedSearchCV(BaseGridSearch):
     n_best_scores_: list of float
         The scores of n_best_forecasters_ sorted from best to worst
         score of forecasters
+    forecasters_ : pd.DataFramee
+        DataFrame with all fitted forecasters and their parameters.
+        Only present if tune_by_instance=True or tune_by_variable=True,
+        and at least one of the two is applicable.
+        In this case, the other attributes are not present in self,
+        only in the fields of forecasters_.
     """
 
     def __init__(
@@ -737,6 +827,8 @@ def __init__(
         backend="loky",
         update_behaviour="full_refit",
         error_score=np.nan,
+        tune_by_instance=False,
+        tune_by_variable=None,
     ):
         super().__init__(
             forecaster=forecaster,
@@ -751,6 +843,8 @@ def __init__(
             backend=backend,
             update_behaviour=update_behaviour,
             error_score=error_score,
+            tune_by_instance=tune_by_instance,
+            tune_by_variable=tune_by_variable,
         )
         self.param_distributions = param_distributions
         self.n_iter = n_iter
@@ -906,6 +1000,22 @@ class ForecastingSkoptSearchCV(BaseGridSearch):
     backend : str, optional (default="loky")
         Specify the parallelisation backend implementation in joblib, where
         "loky" is used by default.
+    tune_by_instance : bool, optional (default=False)
+        Whether to tune parameter by each time series instance separately,
+        in case of Panel or Hierarchical data passed to the tuning estimator.
+        Only applies if time series passed are Panel or Hierarchical.
+        If True, clones of the forecaster will be fit to each instance separately,
+        and are available in fields of the forecasters_ attribute.
+        Has the same effect as applying ForecastByLevel wrapper to self.
+        If False, the same best parameter is selected for all instances.
+    tune_by_variable : bool, optional (default=True)
+        Whether to tune parameter by each time series variable separately,
+        in case of multivariate data passed to the tuning estimator.
+        Only applies if time series passed are strictly multivariate.
+        If True, clones of the forecaster will be fit to each variable separately,
+        and are available in fields of the forecasters_ attribute.
+        Has the same effect as applying ColumnEnsembleForecaster wrapper to self.
+        If False, the same best parameter is selected for all variables.
 
     Attributes
     ----------
@@ -923,6 +1033,12 @@ class ForecastingSkoptSearchCV(BaseGridSearch):
     n_best_scores_: list of float
         The scores of n_best_forecasters_ sorted from best to worst
         score of forecasters
+    forecasters_ : pd.DataFramee
+        DataFrame with all fitted forecasters and their parameters.
+        Only present if tune_by_instance=True or tune_by_variable=True,
+        and at least one of the two is applicable.
+        In this case, the other attributes are not present in self,
+        only in the fields of forecasters_.
 
     Examples
     --------
@@ -982,6 +1098,8 @@ def __init__(
         backend: str = "loky",
         update_behaviour: str = "full_refit",
         error_score=np.nan,
+        tune_by_instance=False,
+        tune_by_variable=None,
     ):
         self.param_distributions = param_distributions
         self.n_iter = n_iter
@@ -1001,6 +1119,8 @@ def __init__(
             backend=backend,
             update_behaviour=update_behaviour,
             error_score=error_score,
+            tune_by_instance=tune_by_instance,
+            tune_by_variable=tune_by_variable,
         )
 
     def _fit(self, y, X=None, fh=None):


Test patch:
diff --git a/sktime/forecasting/model_selection/tests/test_tune.py b/sktime/forecasting/model_selection/tests/test_tune.py
--- a/sktime/forecasting/model_selection/tests/test_tune.py
+++ b/sktime/forecasting/model_selection/tests/test_tune.py
@@ -1,9 +1,8 @@
 #!/usr/bin/env python3 -u
 # copyright: sktime developers, BSD-3-Clause License (see LICENSE file)
-"""Test grid search CV."""
+"""Test forecasting tuners."""
 
 __author__ = ["mloning", "fkiraly"]
-__all__ = ["test_gscv", "test_rscv"]
 
 import numpy as np
 import pytest
@@ -81,12 +80,13 @@ def _check_cv(forecaster, tuner, cv, param_grid, y, X, scoring):
     assert param_grid[best_idx].items() <= fitted_params.items()
 
 
-def _create_hierarchical_data():
+def _create_hierarchical_data(n_columns=1):
     y = _make_hierarchical(
         random_state=TEST_RANDOM_SEEDS[0],
         hierarchy_levels=(2, 2),
         min_timepoints=15,
         max_timepoints=15,
+        n_columns=n_columns,
     )
     X = _make_hierarchical(
         random_state=TEST_RANDOM_SEEDS[1],
@@ -97,7 +97,10 @@ def _create_hierarchical_data():
     return y, X
 
 
-NAIVE = NaiveForecaster(strategy="mean")
+# estimator fixtures used for tuning
+# set_tags in NaiveForecaster ensures that it is univariate and broadcasts
+# this is currently the case, but a future improved NaiveForecaster may reduce coverage
+NAIVE = NaiveForecaster(strategy="mean").set_tags(**{"scitype:y": "univariate"})
 NAIVE_GRID = {"window_length": TEST_WINDOW_LENGTHS_INT}
 PIPE = TransformedTargetForecaster(
     [
@@ -126,15 +129,24 @@ def _create_hierarchical_data():
 @pytest.mark.parametrize("scoring", TEST_METRICS)
 @pytest.mark.parametrize("cv", CVs)
 @pytest.mark.parametrize("error_score", ERROR_SCORES)
-def test_gscv(forecaster, param_grid, cv, scoring, error_score):
+@pytest.mark.parametrize("multivariate", [True, False])
+def test_gscv(forecaster, param_grid, cv, scoring, error_score, multivariate):
     """Test ForecastingGridSearchCV."""
-    y, X = load_longley()
+    if multivariate:
+        X, y = load_longley()
+    else:
+        y, X = load_longley()
+
     gscv = ForecastingGridSearchCV(
         forecaster,
         param_grid=param_grid,
         cv=cv,
         scoring=scoring,
         error_score=error_score,
+        # todo 0.24.0: remove this
+        # and/or add a test for tune_by_variable=True
+        # in this case, the forecaster is expeceted to vectorize over columns
+        tune_by_variable=False,
     )
     gscv.fit(y, X)
 
@@ -177,6 +189,7 @@ def test_rscv(forecaster, param_grid, cv, scoring, error_score, n_iter, random_s
         ParameterSampler(param_grid, n_iter, random_state=random_state)
     )
     _check_cv(forecaster, rscv, cv, param_distributions, y, X, scoring)
+    _check_fitted_params_keys(rscv.get_fitted_params())
 
 
 @pytest.mark.skipif(
@@ -189,20 +202,26 @@ def test_rscv(forecaster, param_grid, cv, scoring, error_score, n_iter, random_s
 @pytest.mark.parametrize("scoring", TEST_METRICS)
 @pytest.mark.parametrize("cv", CVs)
 @pytest.mark.parametrize("error_score", ERROR_SCORES)
-def test_gscv_hierarchical(forecaster, param_grid, cv, scoring, error_score):
+@pytest.mark.parametrize("n_cols", [1, 2])
+def test_gscv_hierarchical(forecaster, param_grid, cv, scoring, error_score, n_cols):
     """Test ForecastingGridSearchCV."""
-    y, X = _create_hierarchical_data()
+    y, X = _create_hierarchical_data(n_columns=n_cols)
     gscv = ForecastingGridSearchCV(
         forecaster,
         param_grid=param_grid,
         cv=cv,
         scoring=scoring,
         error_score=error_score,
+        # todo 0.24.0: remove this
+        # and/or add a test for tune_by_variable=True
+        # in this case, the forecaster is expeceted to vectorize over columns
+        tune_by_variable=False,
     )
     gscv.fit(y, X)
 
     param_grid = ParameterGrid(param_grid)
     _check_cv(forecaster, gscv, cv, param_grid, y, X, scoring)
+    _check_fitted_params_keys(gscv.get_fitted_params())
 
 
 @pytest.mark.skipif(