Repository: sktime/sktime

Issue:
[ENH] `GridSearchCV` for time series classification
There are some good reasons to write an `sktime` native grid search for time series classification (and regression):

* discussion in https://github.com/sktime/sktime/issues/4128 which highlights that classifiers not compatible with `pickle` persistence cannot be combined with `sklearn` grid search (but the `sktime` serialization interface could replace this)
* input/output checks would wokr properly, and more mtypes such as `pd-multiindex` would become compatible with the grid search
* this would provide a basis for scaling and distributed computing, e.g., via `dask`, see https://github.com/sktime/sktime/issues/4013

Opening this for design discussion.

Of course, `sklearn` `GridSearchCV` would remain compatible with `sktime` when used with `sklearn` compatible mtypes and estimators that are persistence compatible with `sklearn` contracts.


Discussion:
It seems like a good feature to add and I've been meaning to take on a medium-sized issue. This seems like a task for me. I'll add an informal design proposal in the upcoming days followed by a STEP after our initial discussion concludes.

Resolving patch:
diff --git a/sktime/classification/model_selection/__init__.py b/sktime/classification/model_selection/__init__.py
new file mode 100644
--- /dev/null
+++ b/sktime/classification/model_selection/__init__.py
@@ -0,0 +1,6 @@
+"""Tuning of time series classifiers."""
+# copyright: sktime developers, BSD-3-Clause License (see LICENSE file)
+
+from sktime.classification.model_selection._tune import TSCGridSearchCV
+
+__all__ = ["TSCGridSearchCV"]
diff --git a/sktime/classification/model_selection/_tune.py b/sktime/classification/model_selection/_tune.py
new file mode 100644
--- /dev/null
+++ b/sktime/classification/model_selection/_tune.py
@@ -0,0 +1,442 @@
+"""Tuning for time series classifiers."""
+
+__author__ = ["fkiraly", "achieveordie"]
+
+import numpy as np
+from sklearn.model_selection import GridSearchCV
+
+from sktime.classification._delegate import _DelegatedClassifier
+
+
+class TSCGridSearchCV(_DelegatedClassifier):
+    """Exhaustive search over specified parameter values for an estimator.
+
+    Adapts sklearn GridSearchCV for sktime time series classifiers
+
+    Optimizes hyper-parameters of `estimators` by exhaustive grid search.
+
+    Parameters
+    ----------
+    estimator : estimator object
+        This is assumed to implement the scikit-learn estimator interface.
+        Either estimator needs to provide a ``score`` function,
+        or ``scoring`` must be passed.
+
+    param_grid : dict or list of dictionaries
+        Dictionary with parameters names (`str`) as keys and lists of
+        parameter settings to try as values, or a list of such
+        dictionaries, in which case the grids spanned by each dictionary
+        in the list are explored. This enables searching over any sequence
+        of parameter settings.
+
+    scoring : str, callable, list, tuple or dict, default=None
+        Strategy to evaluate the performance of the cross-validated model on
+        the test set.
+
+        If `scoring` represents a single score, one can use:
+
+        - a single string (see :ref:`scoring_parameter`);
+        - a callable (see :ref:`scoring`) that returns a single value.
+
+        If `scoring` represents multiple scores, one can use:
+
+        - a list or tuple of unique strings;
+        - a callable returning a dictionary where the keys are the metric
+          names and the values are the metric scores;
+        - a dictionary with metric names as keys and callables a values.
+
+    n_jobs : int, default=None
+        Number of jobs to run in parallel.
+        ``None`` means 1 unless in a :obj:`joblib.parallel_backend` context.
+        ``-1`` means using all processors. See :term:`Glossary <n_jobs>`
+        for more details.
+
+    refit : bool, str, or callable, default=True
+        Refit an estimator using the best found parameters on the whole
+        dataset. If ``False``, the ``predict`` and ``predict_probab`` will not work.
+
+        For multiple metric evaluation, this needs to be a `str` denoting the
+        scorer that would be used to find the best parameters for refitting
+        the estimator at the end.
+
+        Where there are considerations other than maximum score in
+        choosing a best estimator, ``refit`` can be set to a function which
+        returns the selected ``best_index_`` given ``cv_results_``. In that
+        case, the ``best_estimator_`` and ``best_params_`` will be set
+        according to the returned ``best_index_`` while the ``best_score_``
+        attribute will not be available.
+
+        The refitted estimator is made available at the ``best_estimator_``
+        attribute and permits using ``predict`` directly on this
+        ``GridSearchCV`` instance.
+
+        Also for multiple metric evaluation, the attributes ``best_index_``,
+        ``best_score_`` and ``best_params_`` will only be available if
+        ``refit`` is set and all of them will be determined w.r.t this specific
+        scorer.
+
+        See ``scoring`` parameter to know more about multiple metric
+        evaluation.
+
+    cv : int, cross-validation generator or an iterable, default=None
+        Determines the cross-validation splitting strategy.
+        Possible inputs for cv are:
+
+        - None, to use the default 5-fold cross validation,
+        - integer, to specify the number of folds in a `(Stratified)KFold`,
+        - :term:`CV splitter`,
+        - An iterable yielding (train, test) splits as arrays of indices.
+
+        For integer/None inputs, if the estimator is a classifier and ``y`` is
+        either binary or multiclass, :class:`StratifiedKFold` is used. In all
+        other cases, :class:`KFold` is used. These splitters are instantiated
+        with `shuffle=False` so the splits will be the same across calls.
+
+        Refer :ref:`User Guide <cross_validation>` for the various
+        cross-validation strategies that can be used here.
+
+    verbose : int
+        Controls the verbosity: the higher, the more messages.
+
+        - >1 : the computation time for each fold and parameter candidate is
+          displayed;
+        - >2 : the score is also displayed;
+        - >3 : the fold and candidate parameter indexes are also displayed
+          together with the starting time of the computation.
+
+    pre_dispatch : int, or str, default='2*n_jobs'
+        Controls the number of jobs that get dispatched during parallel
+        execution. Reducing this number can be useful to avoid an
+        explosion of memory consumption when more jobs get dispatched
+        than CPUs can process. This parameter can be:
+
+            - None, in which case all the jobs are immediately
+              created and spawned. Use this for lightweight and
+              fast-running jobs, to avoid delays due to on-demand
+              spawning of the jobs
+
+            - An int, giving the exact number of total jobs that are
+              spawned
+
+            - A str, giving an expression as a function of n_jobs,
+              as in '2*n_jobs'
+
+    error_score : 'raise' or numeric, default=np.nan
+        Value to assign to the score if an error occurs in estimator fitting.
+        If set to 'raise', the error is raised. If a numeric value is given,
+        FitFailedWarning is raised. This parameter does not affect the refit
+        step, which will always raise the error.
+
+    return_train_score : bool, default=False
+        If ``False``, the ``cv_results_`` attribute will not include training
+        scores.
+        Computing training scores is used to get insights on how different
+        parameter settings impact the overfitting/underfitting trade-off.
+        However computing the scores on the training set can be computationally
+        expensive and is not strictly required to select the parameters that
+        yield the best generalization performance.
+
+    tune_by_variable : bool, optional (default=False)
+        Whether to tune parameter by each time series variable separately,
+        in case of multivariate data passed to the tuning estimator.
+        Only applies if time series passed are strictly multivariate.
+        If True, clones of the estimator will be fit to each variable separately,
+        and are available in fields of the classifiers_ attribute.
+        Has the same effect as applying ColumnEnsembleClassifier wrapper to self.
+        If False, the same best parameter is selected for all variables.
+
+    Attributes
+    ----------
+    cv_results_ : dict of numpy (masked) ndarrays
+        A dict with keys as column headers and values as columns, that can be
+        imported into a pandas ``DataFrame``.
+
+        For multi-metric evaluation, the scores for all the scorers are
+        available in the ``cv_results_`` dict at the keys ending with that
+        scorer's name (``'_<scorer_name>'``) instead of ``'_score'`` shown
+        above. ('split0_test_precision', 'mean_train_precision' etc.)
+
+    best_estimator_ : estimator
+        Estimator that was chosen by the search, i.e. estimator
+        which gave highest score (or smallest loss if specified)
+        on the left out data. Not available if ``refit=False``.
+
+        See ``refit`` parameter for more information on allowed values.
+
+    best_score_ : float
+        Mean cross-validated score of the best_estimator
+
+        For multi-metric evaluation, this is present only if ``refit`` is
+        specified.
+
+        This attribute is not available if ``refit`` is a function.
+
+    best_params_ : dict
+        Parameter setting that gave the best results on the hold out data.
+
+        For multi-metric evaluation, this is present only if ``refit`` is
+        specified.
+
+    best_index_ : int
+        The index (of the ``cv_results_`` arrays) which corresponds to the best
+        candidate parameter setting.
+
+        The dict at ``search.cv_results_['params'][search.best_index_]`` gives
+        the parameter setting for the best model, that gives the highest
+        mean score (``search.best_score_``).
+
+        For multi-metric evaluation, this is present only if ``refit`` is
+        specified.
+
+    scorer_ : function or a dict
+        Scorer function used on the held out data to choose the best
+        parameters for the model.
+
+        For multi-metric evaluation, this attribute holds the validated
+        ``scoring`` dict which maps the scorer key to the scorer callable.
+
+    n_splits_ : int
+        The number of cross-validation splits (folds/iterations).
+
+    refit_time_ : float
+        Seconds used for refitting the best model on the whole dataset.
+
+        This is present only if ``refit`` is not False.
+
+    multimetric_ : bool
+        Whether or not the scorers compute several metrics.
+
+    classes_ : ndarray of shape (n_classes,)
+        The classes labels. This is present only if ``refit`` is specified and
+        the underlying estimator is a classifier.
+
+    n_features_in_ : int
+        Number of features seen during :term:`fit`. Only defined if
+        `best_estimator_` is defined (see the documentation for the `refit`
+        parameter for more details) and that `best_estimator_` exposes
+        `n_features_in_` when fit.
+
+    feature_names_in_ : ndarray of shape (`n_features_in_`,)
+        Names of features seen during :term:`fit`. Only defined if
+        `best_estimator_` is defined (see the documentation for the `refit`
+        parameter for more details) and that `best_estimator_` exposes
+        `feature_names_in_` when fit.
+
+    See Also
+    --------
+    ParameterGrid : Generates all the combinations of a hyperparameter grid.
+    train_test_split : Utility function to split the data into a development
+        set usable for fitting a GridSearchCV instance and an evaluation set
+        for its final evaluation.
+    sklearn.metrics.make_scorer : Make a scorer from a performance metric or
+        loss function.
+    """
+
+    _tags = {
+        "X_inner_mtype": ["nested_univ", "numpy3D"],
+        "y_inner_mtype": ["numpy2D"],
+        "capability:multivariate": True,
+        "capability:multioutput": True,
+        "capability:unequal_length": True,
+        "capability:missing_values": True,
+        "capability:multithreading": True,
+        "capability:predict_proba": True,
+    }
+
+    def __init__(
+        self,
+        estimator,
+        param_grid,
+        scoring=None,
+        n_jobs=None,
+        refit=True,
+        cv=None,
+        verbose=0,
+        pre_dispatch="2*n_jobs",
+        error_score=np.nan,
+        return_train_score=False,
+        tune_by_variable=False,
+    ):
+        self.estimator = estimator
+        self.param_grid = param_grid
+        self.scoring = scoring
+        self.n_jobs = n_jobs
+        self.refit = refit
+        self.cv = cv
+        self.verbose = verbose
+        self.pre_dispatch = pre_dispatch
+        self.error_score = error_score
+        self.return_train_score = return_train_score
+        self.tune_by_variable = tune_by_variable
+
+        super().__init__()
+
+        DELEGATED_PARAMS = [
+            "estimator",
+            "param_grid",
+            "scoring",
+            "n_jobs",
+            "refit",
+            "cv",
+            "verbose",
+            "pre_dispatch",
+            "error_score",
+            "return_train_score",
+        ]
+
+        gcsvargs = {k: getattr(self, k) for k in DELEGATED_PARAMS}
+
+        self.estimator_ = GridSearchCV(**gcsvargs)
+
+        if self.tune_by_variable:
+            self.set_tags(**{"capability:multioutput": False})
+
+    def _fit(self, X, y):
+        """Fit time series classifier to training data.
+
+        private _fit containing the core logic, called from fit
+
+        Writes to self:
+            Sets fitted model attributes ending in "_".
+
+        Parameters
+        ----------
+        X : guaranteed to be of a type in self.get_tag("X_inner_mtype")
+            if self.get_tag("X_inner_mtype") = "numpy3D":
+            3D np.ndarray of shape = [n_instances, n_dimensions, series_length]
+            if self.get_tag("X_inner_mtype") = "pd-multiindex:":
+            pd.DataFrame with columns = variables,
+            index = pd.MultiIndex with first level = instance indices,
+            second level = time indices
+            for list of other mtypes, see datatypes.SCITYPE_REGISTER
+            for specifications, see examples/AA_datatypes_and_datasets.ipynb
+        y : guaranteed to be of a type in self.get_tag("y_inner_mtype")
+            1D iterable, of shape [n_instances]
+            or 2D iterable, of shape [n_instances, n_dimensions]
+            class labels for fitting
+            if self.get_tag("capaility:multioutput") = False, guaranteed to be 1D
+            if self.get_tag("capaility:multioutput") = True, guaranteed to be 2D
+
+        Returns
+        -------
+        self : Reference to self.
+        """
+        if y.shape[1] == 1:
+            y = y.flatten()
+
+        estimator = self._get_delegate()
+        estimator.fit(X=X, y=y)
+
+        fitted_param_names = [
+            "cv_results_",
+            "best_estimator_",
+            "best_score_",
+            "best_params_",
+            "best_index_",
+            "scorer_",
+            "n_splits_",
+            "refit_time_",
+            "multimetric_",
+            "classes_",
+        ]
+
+        for p in fitted_param_names:
+            if hasattr(estimator, p):
+                val = getattr(estimator, p)
+                setattr(self, p, val)
+
+        return self
+
+    def _predict(self, X):
+        """Predict labels for sequences in X.
+
+        private _predict containing the core logic, called from predict
+
+        State required:
+            Requires state to be "fitted".
+
+        Accesses in self:
+            Fitted model attributes ending in "_"
+
+        Parameters
+        ----------
+        X : guaranteed to be of a type in self.get_tag("X_inner_mtype")
+            if self.get_tag("X_inner_mtype") = "numpy3D":
+                3D np.ndarray of shape = [n_instances, n_dimensions, series_length]
+            if self.get_tag("X_inner_mtype") = "nested_univ":
+                pd.DataFrame with each column a dimension, each cell a pd.Series
+            for list of other mtypes, see datatypes.SCITYPE_REGISTER
+            for specifications, see examples/AA_datatypes_and_datasets.ipynb
+
+        Returns
+        -------
+        y : 1D np.array of int, of shape [n_instances] - predicted class labels
+            indices correspond to instance indices in X
+        """
+        estimator = self._get_delegate()
+        y_pred = estimator.predict(X=X)
+        if y_pred.ndim == 1:
+            y_pred = y_pred.reshape(-1, 1)
+        return y_pred
+
+    # the delegate is an sklearn estimator and it does not have get_fitted_params
+    # therefore we have to override _get_fitted_params from the delegator,
+    # which would otherwise call it
+    def _get_fitted_params(self):
+        """Get fitted parameters.
+
+        private _get_fitted_params, called from get_fitted_params
+
+        State required:
+            Requires state to be "fitted".
+
+        Returns
+        -------
+        fitted_params : dict with str keys
+            fitted parameters, keyed by names of fitted parameter
+        """
+        return {}
+
+    @classmethod
+    def get_test_params(cls, parameter_set="default"):
+        """Return testing parameter settings for the estimator.
+
+        Parameters
+        ----------
+        parameter_set : str, default="default"
+            Name of the set of test parameters to return, for use in tests. If no
+            special parameters are defined for a value, will return `"default"` set.
+            For classifiers, a "default" set of parameters should be provided for
+            general testing, and a "results_comparison" set for comparing against
+            previously recorded results if the general set does not produce suitable
+            probabilities to compare against.
+
+        Returns
+        -------
+        params : dict or list of dict, default={}
+            Parameters to create testing instances of the class.
+            Each dict are parameters to construct an "interesting" test instance, i.e.,
+            `MyClass(**params)` or `MyClass(**params[i])` creates a valid test instance.
+            `create_test_instance` uses the first (or only) dictionary in `params`.
+        """
+        from sklearn.gaussian_process.kernels import RBF, DotProduct
+        from sklearn.metrics import accuracy_score
+
+        from sktime.classification.kernel_based import TimeSeriesSVC
+        from sktime.dists_kernels import AggrDist
+
+        mean_eucl_tskernel = AggrDist(DotProduct())
+        mean_rbf_tskernel = AggrDist(RBF())
+
+        param1 = {
+            "estimator": TimeSeriesSVC(kernel=mean_rbf_tskernel, probability=True),
+            "param_grid": {"C": [0.1, 1]},
+        }
+
+        param2 = {
+            "estimator": TimeSeriesSVC(kernel=mean_eucl_tskernel, probability=True),
+            "param_grid": {"kernel__transformer": [DotProduct(), RBF()]},
+            "scoring": accuracy_score,
+        }
+
+        return [param1, param2]


Test patch: