Repository: sktime/sktime
Pull request: [ENH] permutation pipeline prototype, delegator design
Alternative design to https://github.com/sktime/sktime/pull/3443 which does not add the functionality to the existing pipelines, but makes it extrinsic as a delegator.

More modular design-wise, and more flexible as it can be applied to any heterogeneous ensemble (as long as it is a forecaster).

Problem statement:
[ENH] Tuning order of transformers in pipelines as a hyper-parameter
**Is your feature request related to a problem? Please describe.**
We have currently three sequential pipeline classes `TransformedTragetForecaster`, `ForecastingPipeline` and `TransformerPipeline`. Having several transformers chained raises the question which is the best order (permutation) of the given transformers.
For example in the pipeline below, what is the best order of transformers?
```python
    >>> pipe = TransformedTargetForecaster(
        steps=[
    ...     ("outlier", HampelFilter()),
    ...     ("impute", Imputer()),
    ...     ("detrend", Detrender()),
    ...     ("deseason", Deseasonalizer()),
    ...     ("minmax", TabularToSeriesAdaptor(MinMaxScaler())),
    ...     ("difference", Differencer()),
    ...     ("forecast", NaiveForecaster(strategy="drift")),
    ... ])
```

**Describe the solution you'd like**

To tune the permutations of transformers, we would need another parameter that allow to give several sortings of the steps.
A solution could look as follows:
```python
    >>> pipe = TransformedTargetForecaster(
        steps=[
    ...     ("outlier", HampelFilter()),
    ...     ("impute", Imputer()),
    ...     ("detrend", Detrender()),
    ...     ("deseason", Deseasonalizer()),
    ...     ("minmax", TabularToSeriesAdaptor(MinMaxScaler())),
    ...     ("difference", Differencer()),
    ...     ("forecast", NaiveForecaster(strategy="drift")),
    ... ],
    ... permutation=["difference", "outlier", "deseason", "detrend", "impute", "minmax", "forecast"],
    ... )
```
This could be implemented as a `Mixin` class so all sequential pipeline classes could inherit and implement it. To be discussed if `"forecast"` step should be part of `permutation` list or if it should just stay at position where it is given. The param `permutation` could also accept `permutation="all"` to try all `n!` combinations with `n` as number of given transformers.

**Describe alternatives you've considered**
I thought first of having a separate transformer composition for this, but now I think its actually best to integrate this into the existing pipeline classes.

**Additional context**
Grid search, tuning

@miraep8  would be interested about your feedback as you wrote the `MultiplexTransformer` which solves the 1:n selection of transformers.
Also pinging @benHeid and @fkiraly

Hints from issue discussion:
Very interesting idea! This one is not implemented yet 😁 

On first sight, I would say, this should *not* be attached to the current pipeline classes, that's attaching "too much responsibility".

Options I can think of:

* descendants with the added functionality
* a `PermutationMultiplexer` or `PipelinePermuter` or `InferParams`, which takes as args: `permute_arg` = name of the argument that is list of estimators (default=`steps`), and a `permutation` (default=identity). It behaves like the wrapped estimator, but with `permute_arg` permuted by `permutation`

What would be nice in the second idea if it were polymorphic.
> that's attaching "too much responsibility"

I think that should not be an argument, we should always design from a user perspective and not how convenient it is for us to code it. Thats why I think the solution I propose is the most intuitive and user-friendly solution. Having another composition for this would look quite weird especially given that user already needs to nest `TransformedTargetForecaster` and `ForecastingPipeline` is transforming `y` and `X`. So I wouldnt want to request from user another pipeline nesting inside this.

> descendants with the added functionality

Can you explain this a bit more please?

Commit message: fix typo in docstring

Patch to review:
diff --git a/sktime/base/_meta.py b/sktime/base/_meta.py
index a139f4a6f16..6c69aa1d27e 100644
--- a/sktime/base/_meta.py
+++ b/sktime/base/_meta.py
@@ -17,14 +17,6 @@ class _HeterogenousMetaEstimator(BaseEstimator):
     Partly adapted from sklearn utils.metaestimator.py.
     """
 
-    def get_params(self, deep=True):
-        """Return estimator parameters."""
-        raise NotImplementedError("abstract method")
-
-    def set_params(self, **params):
-        """Set estimator parameters."""
-        raise NotImplementedError("abstract method")
-
     def is_composite(self):
         """Check if the object is composite.
 
diff --git a/sktime/forecasting/compose/__init__.py b/sktime/forecasting/compose/__init__.py
index 460fdc680a9..45dfd4b7b49 100644
--- a/sktime/forecasting/compose/__init__.py
+++ b/sktime/forecasting/compose/__init__.py
@@ -25,6 +25,7 @@
     "make_reduction",
     "BaggingForecaster",
     "ForecastByLevel",
+    "Permute",
 ]
 
 from sktime.forecasting.compose._bagging import BaggingForecaster
@@ -38,6 +39,7 @@
 from sktime.forecasting.compose._pipeline import (
     ForecastingPipeline,
     ForecastX,
+    Permute,
     TransformedTargetForecaster,
 )
 from sktime.forecasting.compose._reduce import (
diff --git a/sktime/forecasting/compose/_pipeline.py b/sktime/forecasting/compose/_pipeline.py
index 4714f5606ee..14beabf8d69 100644
--- a/sktime/forecasting/compose/_pipeline.py
+++ b/sktime/forecasting/compose/_pipeline.py
@@ -8,7 +8,9 @@
 import pandas as pd
 
 from sktime.base import _HeterogenousMetaEstimator
+from sktime.datatypes import ALL_TIME_SERIES_MTYPES
 from sktime.forecasting.base._base import BaseForecaster
+from sktime.forecasting.base._delegate import _DelegatedForecaster
 from sktime.registry import scitype
 from sktime.utils.validation.series import check_series
 
@@ -1448,3 +1450,192 @@ def get_test_params(cls, parameter_set="default"):
         params = {"forecaster_X": fx, "forecaster_y": fy}
 
         return params
+
+
+class Permute(_DelegatedForecaster, _HeterogenousMetaEstimator):
+    """Permutation compositor for permuting pipeline steps.
+
+    The compositor can be used to permute the sequence of any meta-estimator,
+    including ForecastingPipeline and TransformedTargetForecaster.
+
+    The `steps_arg` parameter needs to be pointed to the "steps"-like parameter
+    of the wrapped forecaster and `permutation` switches the sequence of steps.
+
+    Not very useful on its own, but
+    useful in combination with tuning or auto-ML wrappers on top of this.
+
+    Parameters
+    ----------
+    estimator : sktime forecaster, inheriting from BaseForecaster
+        must have parameter with name `steps_arg`
+        estimator whose steps are being permuted
+    permutation : list of str, or None, optional, default = None
+        if not None, must be equal length as getattr(estimator, steps_arg)
+        and elements must be equal to names of estimator.steps_arg estimators
+        names are unique names as created by _get_estimator_tuples (if unnamed list),
+        or first string element of tuples, of estimator.steps_arg
+        list is interpreted as range of permutation of names
+        if None, is interpreted as the identity permutation
+    steps_arg : string, optional, default="steps"
+        name of the steps parameter. getattr(estimator, steps_arg) must be
+        list of estimators, or list of (str, estimator) pairs
+
+    Examples
+    --------
+    >>> from sktime.datasets import load_airline
+    >>> from sktime.forecasting.base import ForecastingHorizon
+    >>> from sktime.forecasting.compose import ForecastingPipeline, Permute
+    >>> from sktime.forecasting.naive import NaiveForecaster
+    >>> from sktime.transformations.series.boxcox import BoxCoxTransformer
+    >>> from sktime.transformations.series.exponent import ExponentTransformer
+
+    Simple example: permute sequence of estimator in forecasting pipeline
+    >>> y = load_airline()
+    >>> fh = ForecastingHorizon([1, 2, 3])
+    >>> pipe = ForecastingPipeline(
+    ...     [
+    ...         ("boxcox", BoxCoxTransformer()),
+    ...         ("exp", ExponentTransformer(3)),
+    ...         ("naive", NaiveForecaster()),
+    ...     ]
+    ... )
+    >>> # this results in the pipeline with sequence "exp", "boxcox", "naive"
+    >>> permuted = Permute(pipe, ["exp", "boxcox", "naive"])
+    >>> pipe = pipe.fit(y, fh=fh)
+    >>> y_pred = pipe.predict()
+
+    The permuter is useful in combination with grid search (toy example):
+    >>> from sktime.datasets import load_shampoo_sales
+    >>> from sktime.forecasting.model_selection import (
+    ...     ExpandingWindowSplitter,
+    ...     ForecastingGridSearchCV,
+    ... )
+    >>> fh = [1,2,3]
+    >>> cv = ExpandingWindowSplitter(
+    ...     start_with_window=True,
+    ...     fh=fh)
+    >>> forecaster = NaiveForecaster()
+    >>> # check which of the two sequences of transformers is better
+    >>> param_grid = {
+    ...     "permutation" : [["boxcox", "exp", "naive"], ["exp", "boxcox", "naive"]]
+    ... }
+    >>> gscv = ForecastingGridSearchCV(
+    ...     forecaster=forecaster,
+    ...     param_grid=param_grid,
+    ...     cv=cv)
+    """
+
+    _tags = {
+        "scitype:y": "both",
+        "y_inner_mtype": ALL_TIME_SERIES_MTYPES,
+        "X_inner_mtype": ALL_TIME_SERIES_MTYPES,
+        "ignores-exogeneous-X": False,
+        "requires-fh-in-fit": False,
+        "handles-missing-data": True,
+        "capability:pred_int": True,
+        "X-y-must-have-same-index": False,
+    }
+
+    _delegate_name = "estimator_"
+
+    def __init__(self, estimator, permutation, steps_arg="steps"):
+        self.estimator = estimator
+        self.permutation = permutation
+        self.steps_arg = steps_arg
+
+        super(Permute, self).__init__()
+        tags_to_clone = [
+            "ignores-exogeneous-X",  # does estimator ignore the exogeneous X?
+            "capability:pred_int",  # can the estimator produce prediction intervals?
+            "requires-fh-in-fit",  # is forecasting horizon already required in fit?
+            "enforce_index_type",  # index type that needs to be enforced in X/y
+            "fit_is_empty",
+        ]
+
+        self.clone_tags(self.estimator, tags_to_clone)
+
+        self._set_permuted_estimator()
+
+    def _set_permuted_estimator(self):
+        """Set self.estimator_ based on permutation arg."""
+        estimator = self.estimator
+        permutation = self.permutation
+        steps_arg = self.steps_arg
+
+        self.estimator_ = estimator.clone()
+
+        if permutation is not None:
+
+            inner_estimators = getattr(estimator, steps_arg)
+            estimator_tuples = self._get_estimator_tuples(inner_estimators)
+
+            estimator_dict = {x[0]: x[1] for x in estimator_tuples}
+
+            # check that all permutation are str values
+            if not all(isinstance(item, str) for item in permutation):
+                raise ValueError("permutation must be None or a list of strings")
+
+            # check that permutation contains same step names as given in steps
+            if not set(estimator_dict.keys()) == set(permutation):
+                raise ValueError(
+                    f"""Permutations must contain exactly the same step names as
+                    the names of steps in getattr(estimator, steps_arg),
+                    found tuple names {set(estimator_dict.keys())} but got
+                    permutation {set(permutation)}."""
+                )
+
+            estimator_tuples_permuted = [(k, estimator_dict[k]) for k in permutation]
+
+            self.estimator_ = estimator.clone()
+            self.estimator_.set_params(**{steps_arg: estimator_tuples_permuted})
+
+    @classmethod
+    def get_test_params(cls, parameter_set="default"):
+        """Return testing parameter settings for the estimator.
+
+        Parameters
+        ----------
+        parameter_set : str, default="default"
+            Name of the set of test parameters to return, for use in tests. If no
+            special parameters are defined for a value, will return `"default"` set.
+            There are currently no reserved values for forecasters.
+
+        Returns
+        -------
+        params : dict or list of dict, default = {}
+            Parameters to create testing instances of the class
+            Each dict are parameters to construct an "interesting" test instance, i.e.,
+            `MyClass(**params)` or `MyClass(**params[i])` creates a valid test instance.
+            `create_test_instance` uses the first (or only) dictionary in `params`
+        """
+        from sktime.forecasting.naive import NaiveForecaster
+        from sktime.transformations.series.boxcox import BoxCoxTransformer
+        from sktime.transformations.series.exponent import ExponentTransformer
+
+        # transformers mixed with-without fit, ForecastingPipeline
+        # steps are (str, estimator)
+        params1 = {
+            "estimator": ForecastingPipeline(
+                [
+                    ("foo", BoxCoxTransformer()),
+                    ("bar", ExponentTransformer(3)),
+                    ("foobar", NaiveForecaster()),
+                ]
+            ),
+            "permutation": ["bar", "foo", "foobar"],
+        }
+
+        # transformers have no fit, TransformedTargetForecaster
+        # steps are only estimator
+        params2 = {
+            "estimator": TransformedTargetForecaster(
+                [ExponentTransformer(0.5), NaiveForecaster(), ExponentTransformer(3)]
+            ),
+            "permutation": [
+                "NaiveForecaster",
+                "ExponentTransformer_1",
+                "ExponentTransformer_2",
+            ],
+        }
+
+        return [params1, params2]


Code review comments:
diff_hunk: @@ -1448,3 +1450,192 @@ def get_test_params(cls, parameter_set="default"):
         params = {"forecaster_X": fx, "forecaster_y": fy}
 
         return params
+
+
+class Permute(_DelegatedForecaster, _HeterogenousMetaEstimator):
+    """Permutation compositor for permuting pipeline steps.
+
+    The compositor can be used to permute the sequence of any meta-estimator,
+    including ForecastingPipeline and TransformedTargetForecaster.
+
+    The `steps_arg` parameter needs to be pointed to the "steps"-like parameter
+    of the wrapped forecaster and `permutation` switches the sequence of steps.
+
+    Not very useful on its own, but
+    useful in combination with tuning or auto-ML wrappers on top of this.
+
+    Parameters
+    ----------
+    estimator : sktime forecaster, inheriting from BaseForecaster
+        must have parameter with name `steps_arg`
+        estimator whose steps are being permuted
+    permutation : list of str, or None, optional, default = None
+        if not None, must be equal length as getattr(estimator, steps_arg)
+        and elements must be equal to names of estimator.steps_arg estimators
+        names are unique names as created by _get_estimator_tuples (if unnamed list),
+        or first string element of tuples, of estimator.steps_arg
+        list is interpreted as range of permutation of names
+        if None, is interpreted as the identity permutation
+    steps_arg : string, optional, default="steps"
+        name of the steps parameter. getattr(estimator, steps_arg) must be
+        list of estimators, or list of (str, estimator) pairs
+
+    Examples
+    --------
+    >>> from sktime.datasets import load_airline
+    >>> from sktime.forecasting.base import ForecastingHorizon
+    >>> from sktime.forecasting.compose import ForecastingPipeline, Permute
+    >>> from sktime.forecasting.naive import NaiveForecaster
+    >>> from sktime.transformations.series.boxcox import BoxCoxTransformer
+    >>> from sktime.transformations.series.exponent import ExponentTransformer
+
+    Simple example: permute sequence of estimator in forecasting pipeline
+    >>> y = load_airline()
+    >>> fh = ForecastingHorizon([1, 2, 3])
+    >>> pipe = ForecastingPipeline(
+    ...     [
+    ...         ("boxcox", BoxCoxTransformer()),
+    ...         ("exp", ExponentTransformer(3)),
+    ...         ("naive", NaiveForecaster()),
+    ...     ]
+    ... )
+    >>> # this results in the pipeline with sequence "exp", "boxcox", "naive"
+    >>> permuted = Permute(pipe, ["exp", "boxcox", "naive"])
+    >>> pipe = pipe.fit(y, fh=fh)
+    >>> y_pred = pipe.predict()
+
+    The permuter is useful in combination with grid search (toy example):
+    >>> from sktime.datasets import load_shampoo_sales
+    >>> from sktime.forecasting.model_selection import (
+    ...     ExpandingWindowSplitter,
+    ...     ForecastingGridSearchCV,
+    ... )
+    >>> fh = [1,2,3]
+    >>> cv = ExpandingWindowSplitter(
+    ...     start_with_window=True,
+    ...     fh=fh)
+    >>> forecaster = NaiveForecaster()
+    >>> # check which of the two sequences of transformers is better
+    >>> param_grid = {
+    ...     "permutation" : [["boxcox", "exp", "naive"], ["exp", "boxcox", "naive"]]
+    ... }
+    >>> gscv = ForecastingGridSearchCV(
+    ...     forecaster=forecaster,
path: sktime/forecasting/compose/_pipeline.py
text: @user1:
I am somewhat confused by this example.  Shouldn't the "forecaster" here be `permuted` from above? Not NaiveForecaster()? 

@author:
it should be `permuted` indeed. Why does this not raise an error?

@author:
fixed