Repository: sktime/sktime

Issue:
[ENH] TemporianTransformer
`TemporianTransformer` proposal. General objective and design validated in today's community meetup with @fkiraly and @benHeid. 

# Context

## Temporian

- [Temporian](https://github.com/google/temporian) is a library for safe, simple and efficient preprocessing and feature engineering of temporal data in Python.
- An [EventSet](https://temporian.readthedocs.io/en/stable/reference/temporian/EventSet/) is the structure used by Temporian to store temporal data. It is to Temporian what a DataFrame is to Pandas.
- A Temporian operator is a function taking as input zero, one or multiple EventSets, zero, one, or multiple parameters, and outputting one or multiple EventSets.
- A Temporian [Graph](https://temporian.readthedocs.io/en/stable/user_guide/#eager-mode-vs-graph-mode) is a set of Temporian operators organized into a computation graph where the output of one operator can become the input of another operator.
- A Temporian function is a Python function decorated with @tp.compile. A Temporian function is both a regular Python function and a Temporian graph.
- EventSet can be converted back and from a dictionary of NumPy arrays and pandas DataFrames ([from_pandas()](https://temporian.readthedocs.io/en/stable/reference/temporian/io/from_pandas/) and [to_pandas()](https://temporian.readthedocs.io/en/stable/reference/temporian/io/to_pandas/)).

## Sktime

- A Transformer is a class that derives from the [BaseTransformer](https://github.com/sktime/sktime/blob/main/sktime/transformations/base.py#L100) class and contains time series pre-processing logic.
- We are especially interested in sktime’s [series-to-series](https://www.sktime.net/en/latest/api_reference/transformations.html#series-to-series-transformers) transformers, which are similar to Temporian functions, in that they receive and return a time series.

# Proposal
Creating a `TemporianTransformer` series-to-series transformer in sktime, which allows users to embed a custom (user-defined) Temporian function into their sktime pipeline.

# Design Overview

- We create a TemporianTransformer class that inherits from the [BaseTransformer](https://github.com/sktime/sktime/blob/main/sktime/transformations/base.py#L100).
- This TemporianTransformer class takes a Temporian function as a constructor argument.
- This  Temporian function is restricted to take one input EventSet and output one EventSet.
- The `transformer_supersimple` template should be enough.
- The `TemporianTransformer` specifies the following via tags:
    - Input types: `pd.DataFrame` (flat index), `pd-multiindex` (single-level index) and `pd_multiindex_hier` (several levels)
    - `fit_is_empty: True`
    - `python_dependencies` to add temporian dependency
    - `python_version` if Temporian doesn't support all currently supported py versions
- The data flow during the execution of a TemporianTransformer is as follows:
    - The TemporianTransformer receives one of the defined input types (all are pandas types).
    - The pandas Dataframe is imported into a Temporian EventSet.
    - The user’s Temporian function is executed on the EventSet.
    - The output EventSet is converted into a pandas DataFrame.

# API Definition

Proposed API (pseudocode-ish, doesn’t follow sktime’s templates/conventions).
For reference: sktime’s [Lag](https://www.sktime.net/en/latest/api_reference/auto_generated/sktime.transformations.series.lag.Lag.html#sktime.transformations.series.lag.Lag) transformer docs.

```python
class TemporianTransformer(tp_function: Callable[[tp.EventSet], tp.EventSet]):
"""
TemporianTransformer. Applies a Temporian function to a time series.

Conversion from and to Temporian's EventSet is handled internally by sktime.

Parameters:
tp_function: a Temporian function to apply on the time series. The function must receive and return a single Temporian EventSet. The function can be compiled with @tp.compile for its execution graph to be optimized by Temporian.
"""
...
```

# API Usage Example

We are working with a dataset of sales of several stores in the same chain.
We want to calculate the percentage of the total each store’s sales represent, as a weekly moving average.

```python
import temporian as tp
from sktime.transformations.series.temporian import TemporianTransformer
from sktime.forecasting.compose import ForecastingPipeline
from sktime.forecasting.naive import NaiveForecaster

@tp.compile  # Optional (see Open Questions below)
def my_tp_function(evset: tp.EventSet) -> tp.EventSet:
 	# "evset" is indexed by "store" and "state"
	sales = evset['sales']  # Selects the "sales" column / feature
	one_week = tp.duration.weeks(1)  # A duration of 1 week
	weekly_store_sales = sales.simple_moving_average(one_week)  # 1 week simple moving average over "sales" for each store
	weekly_total_sales = sales.drop_index('store', keep=False).simple_moving_average(one_week).propagate(weekly_store_sales)
	return weekly_store_sales / weekly_total_sales  # Ratio of sales of a store to all the other stores in the same state.

pipe = ForecastingPipeline(
	steps=[
                ("rate_per_store", TemporianTransformer(my_tp_function)),
                ("forecaster", NaiveForecaster(strategy="drift")),
        ]
)
...
```

Notes:
- In the snippet above we assume sktime will pass Temporian an EventSet indexed by "store" and "state". In sktime, those are represented by [panel time series](https://www.sktime.net/en/latest/get_started.html#data-types) (i.e. for multiple instances/entities).
- The user can also pass Multivariate time series and Singlevariate time series which get converted into flat (no-index) EventSets.

# Open Questions

- Do we want to automatically compile the received Temporian function if it wasn’t compiled by the user? Yes, unless we find a reason not to.


Discussion:
Feel free to assign to me @fkiraly, and eager to get feedback from other sktime devs. Cheers!
Nice! As discussed in today's communiy meetup, it might be nice to add `EventSet` in a second or third step, as a supported data container to `sktime`. This can be done based on the first PR by:

* isolating the conversion and back-conversion functionality into function snippets
* moving these to the `datatypes` module, adding multiple `EventSet` based mtypes (for `Series`, `Panel`, `Hierarchical`)
* changing the `X_inner_mtype` for `TemporianTransformer` to the list of the three mtypes

The overall logic when passing anything to the transformer would not change, this would be a pure refactor from the perspective of running code - and would enable use of `EventSet` in any `sktime` estimator in the future, not just transformers.

(this would be similar to the addtion of the `dask` or `polars` mtypes)

Minor edit: the name of the mtype is actually `pd_multiindex_hier`, not `pd-multiindex-hier` - this is unfortunately a historical inconsistency, but it has been hard to change due to in a central location.

Resolving patch:
diff --git a/sktime/transformations/series/temporian.py b/sktime/transformations/series/temporian.py
new file mode 100644
--- /dev/null
+++ b/sktime/transformations/series/temporian.py
@@ -0,0 +1,154 @@
+# copyright: sktime developers, BSD-3-Clause License (see LICENSE file)
+
+"""Implementation of TemporianTransformer. Contributed by the Temporian team."""
+
+__author__ = ["ianspektor", "javiber"]
+
+from sktime.transformations.base import BaseTransformer
+
+
+class TemporianTransformer(BaseTransformer):
+    """Applies a Temporian function to the input time series.
+
+    This transformer applies a Temporian [1]_ function to the input time series.
+
+    The conversion from sktime's internal representation to Temporian's EventSet [2]_
+    and back is handled automatically by the transformer.
+
+    Parameters
+    ----------
+    function : Callable[[temporian.EventSet], temporian.EventSet]
+        Temporian function to apply to the input time series. The function must receive
+        and return a single Temporian EventSet, and can apply an arbitrary number of
+        Temporian operators to its input.
+    compile : bool, default=False
+        If True, the function will be compiled using Temporian's @tp.compile [3]_
+        decorator, which can lead to significant speedups by optimizing the graph of
+        operations.
+
+    Examples
+    --------
+    >>> from sktime.datasets import load_airline
+    >>> from sktime.transformations.series.temporian import TemporianTransformer
+    >>> import temporian as tp  # doctest: +SKIP
+    >>>
+    >>> def function(evset):  # doctest: +SKIP
+    ...     return evset.simple_moving_average(tp.duration.days(3 * 365))  \
+        # doctest: +SKIP
+    >>> transformer = TemporianTransformer(function=function)  # doctest: +SKIP
+    >>> X = load_airline()  # doctest: +SKIP
+    >>> X_averaged = transformer.fit_transform(X)  # doctest: +SKIP
+
+    References
+    ----------
+    .. [1] https://temporian.readthedocs.io/en/stable/
+    .. [2] https://temporian.readthedocs.io/en/stable/reference/temporian/EventSet/
+    .. [3] https://temporian.readthedocs.io/en/stable/reference/temporian/compile/
+    """
+
+    _tags = {
+        # packaging info
+        # --------------
+        "authors": ["ianspektor", "javiber"],
+        "maintainers": ["ianspektor", "javiber"],
+        "python_dependencies": ["temporian"],
+        "python_version": ">=3.8",
+        # estimator tags
+        # --------------
+        "univariate-only": False,
+        "scitype:transform-input": "Series",
+        "scitype:transform-output": "Series",
+        "scitype:instancewise": True,
+        "scitype:transform-labels": "None",
+        "X_inner_mtype": "pd.DataFrame",
+        "fit_is_empty": True,
+        "capability:inverse_transform": False,
+        "capability:unequal_length": True,
+        "handles-missing-data": False,
+    }
+
+    def __init__(self, function, compile=False):
+        self.function = function
+        self.compile = compile
+
+        super().__init__()
+
+    def _transform(self, X, y=None):
+        """Transform X and return a transformed version.
+
+        private _transform containing core logic, called from transform
+
+        Parameters
+        ----------
+        X : pd.DataFrame
+            if self.get_tag("univariate-only")==True:
+                guaranteed to have a single column
+            if self.get_tag("univariate-only")==False: no restrictions apply
+        y : None, present only for interface compatibility
+
+        Returns
+        -------
+        transformed version of X
+        """
+        import temporian as tp
+
+        X_noindex = X.reset_index(drop=False)
+        X_noindex.columns = X_noindex.columns.astype(str)
+        timestamps_col = X_noindex.columns[0]
+        evset = tp.from_pandas(X_noindex, timestamps=timestamps_col)
+
+        function = tp.compile(self.function) if self.compile else self.function
+
+        res = function(evset)
+        if not isinstance(res, tp.EventSet):
+            raise TypeError(
+                f"Expected return type to be an EventSet but received a {type(res)}"
+            )
+
+        # Test that the sampling was not modified, otherwise the conversion back to
+        # pandas and the set_index would not work as expected
+        try:
+            res.check_same_sampling(evset)
+        except ValueError as exc:
+            raise ValueError(
+                "The resulting EventSet must have the same sampling as the input. "
+                "Visit our docs for more info "
+                "https://temporian.readthedocs.io/en/stable/user_guide/#sampling"
+            ) from exc
+
+        res = tp.to_pandas(res, timestamps=False)
+        res = res.set_index(X.index)
+
+        return res
+
+    @classmethod
+    def get_test_params(cls, parameter_set="default"):
+        """Return testing parameter settings for the estimator.
+
+        Parameters
+        ----------
+        parameter_set : str, default="default"
+            Name of the set of test parameters to return, for use in tests. If no
+            special parameters are defined for a value, will return `"default"` set.
+            There are currently no reserved values for transformers.
+
+        Returns
+        -------
+        params : dict or list of dict, default = {}
+            Parameters to create testing instances of the class
+            Each dict are parameters to construct an "interesting" test instance, i.e.,
+            `MyClass(**params)` or `MyClass(**params[i])` creates a valid test instance.
+            `create_test_instance` uses the first (or only) dictionary in `params`
+        """
+        return [
+            {"function": _test_function_single_op, "compile": False},
+            {"function": _test_function_many_ops, "compile": True},
+        ]
+
+
+def _test_function_single_op(evset):
+    return evset + 1
+
+
+def _test_function_many_ops(evset):
+    return evset.simple_moving_average(10).lag(10).resample(evset)


Test patch:
diff --git a/sktime/tests/test_softdeps.py b/sktime/tests/test_softdeps.py
--- a/sktime/tests/test_softdeps.py
+++ b/sktime/tests/test_softdeps.py
@@ -53,6 +53,7 @@
 EXCEPTED_FROM_GET_PARAMS_CHECK = [
     "PyODAnnotator",  # adapters always require soft dep. Here: pyod
     "HCrystalBallAdapter",  # adapters always require soft dep. Here: hcrystalball
+    "TemporianTransformer",  # adapters always require soft dep. Here: temporian
 ]
 
 
diff --git a/sktime/transformations/series/tests/test_temporian.py b/sktime/transformations/series/tests/test_temporian.py
new file mode 100644
--- /dev/null
+++ b/sktime/transformations/series/tests/test_temporian.py
@@ -0,0 +1,225 @@
+"""Tests for TemporianTransformer."""
+
+# copyright: sktime developers, BSD-3-Clause License (see LICENSE file).
+
+__author__ = ["ianspektor", "javiber"]
+
+from datetime import datetime
+from unittest.mock import patch
+
+import numpy as np
+import pandas as pd
+import pytest
+
+from sktime.datasets import load_solar
+from sktime.datatypes import get_examples
+from sktime.tests.test_switch import run_test_for_class
+from sktime.transformations.series.temporian import TemporianTransformer
+
+
+@pytest.mark.skipif(
+    not run_test_for_class(TemporianTransformer),
+    reason="run test only if softdeps are present and incrementally (if requested)",
+)
+def test_flat_univariate():
+    """Tests basic function works on flat (non-indexed) univariate time series."""
+    X = load_solar()[0:32]
+
+    def function(evset):
+        return evset["solar_gen"] + 1
+
+    transformer = TemporianTransformer(function=function)
+    X_transformed = transformer.fit_transform(X=X)
+
+    pd.testing.assert_series_equal(X_transformed, X + 1)
+
+
+@pytest.mark.skipif(
+    not run_test_for_class(TemporianTransformer),
+    reason="run test only if softdeps are present and incrementally (if requested)",
+)
+def test_panel():
+    """Tests basic function works on Panels."""
+
+    def function(evset):
+        return evset + 1
+
+    examples = get_examples("pd-multiindex", as_scitype="Panel").values()
+    assert len(examples) > 0
+
+    for X in examples:
+        transformer = TemporianTransformer(function=function)
+        X_transformed = transformer.fit_transform(X=X)
+        pd.testing.assert_frame_equal(X_transformed, X + 1)
+
+
+@pytest.mark.skipif(
+    not run_test_for_class(TemporianTransformer),
+    reason="run test only if softdeps are present and incrementally (if requested)",
+)
+def test_hier():
+    """Tests basic function works on Hierarchical."""
+
+    def function(evset):
+        return evset + 1
+
+    examples = get_examples("pd_multiindex_hier", as_scitype="Hierarchical").values()
+    assert len(examples) > 0
+
+    for X in examples:
+        transformer = TemporianTransformer(function=function)
+        X_transformed = transformer.fit_transform(X=X)
+        pd.testing.assert_frame_equal(X_transformed, X + 1)
+
+
+@pytest.mark.skipif(
+    not run_test_for_class(TemporianTransformer),
+    reason="run test only if softdeps are present and incrementally (if requested)",
+)
+def test_dtypes():
+    """Tests basic function works on all dtypes."""
+
+    def function(evset):
+        return evset + 1
+
+    X = pd.DataFrame(
+        {
+            str(dtype): np.zeros(10).astype(dtype)
+            for dtype in [
+                np.int32,
+                np.int64,
+                np.float32,
+                np.float64,
+            ]
+        }
+    )
+    transformer = TemporianTransformer(function=function)
+    X_transformed = transformer.fit_transform(X=X)
+    pd.testing.assert_frame_equal(X_transformed, X + 1)
+
+
+@pytest.mark.skipif(
+    not run_test_for_class(TemporianTransformer),
+    reason="run test only if softdeps are present and incrementally (if requested)",
+)
+def test_compiled():
+    """Tests compiling the function."""
+    import temporian as tp
+
+    def function(evset):
+        return evset + 1
+
+    X = get_examples("pd.Series", as_scitype="Series")[0]
+
+    true_compile = tp.compile
+
+    with patch.object(tp, "compile") as compile_patch:
+        compile_patch.side_effect = true_compile
+        transformer = TemporianTransformer(function=function, compile=True)
+        X_transformed = transformer.fit_transform(X=X)
+        pd.testing.assert_series_equal(X_transformed, X + 1)
+        compile_patch.assert_called_once_with(function)
+
+
+@pytest.mark.skipif(
+    not run_test_for_class(TemporianTransformer),
+    reason="run test only if softdeps are present and incrementally (if requested)",
+)
+def test_compile_already_compiled():
+    """Tests compiling doesn't fail if the function was compiled already."""
+    import temporian as tp
+
+    @tp.compile
+    def function(evset):
+        return evset + 1
+
+    X = get_examples("pd.Series", as_scitype="Series")[0]
+
+    true_compile = tp.compile
+
+    with patch.object(tp, "compile") as compile_patch:
+        compile_patch.side_effect = true_compile
+        transformer = TemporianTransformer(function, compile=True)
+        X_transformed = transformer.fit_transform(X=X)
+        pd.testing.assert_series_equal(X_transformed, X + 1)
+        compile_patch.assert_called_once_with(function)
+
+
+@pytest.mark.skipif(
+    not run_test_for_class(TemporianTransformer),
+    reason="run test only if softdeps are present and incrementally (if requested)",
+)
+def test_multiple_output():
+    """Tests errors when functions return incorrect types."""
+
+    def function(evset):
+        split = datetime(1950, 7, 16)
+        return evset.before(split), evset.after(split)
+
+    X = load_solar()
+
+    transformer = TemporianTransformer(function=function)
+    with pytest.raises(
+        TypeError, match="Expected return type to be an EventSet but received"
+    ):
+        _ = transformer.fit_transform(X=X)
+
+
+@pytest.mark.skipif(
+    not run_test_for_class(TemporianTransformer),
+    reason="run test only if softdeps are present and incrementally (if requested)",
+)
+def test_multiple_input():
+    """Tests errors on incorrectly defined functions."""
+
+    def function(evset, incorrect_param):
+        return evset + 1
+
+    X = load_solar()
+
+    transformer = TemporianTransformer(function=function)
+    with pytest.raises(TypeError, match="missing 1 required positional"):
+        _ = transformer.fit_transform(X=X)
+
+
+@pytest.mark.skipif(
+    not run_test_for_class(TemporianTransformer),
+    reason="run test only if softdeps are present and incrementally (if requested)",
+)
+def test_change_sampling():
+    """Tests error when changing the sampling."""
+    import temporian as tp
+
+    # in this dataset each row is a period of 1/2 hour
+    # lagging for one hour would remove the first two rows thus changing the sampling
+    X = load_solar()
+
+    def function(evset):
+        return evset.lag(tp.duration.hours(1))
+
+    transformer = TemporianTransformer(function=function, compile=True)
+    with pytest.raises(
+        ValueError,
+        match="The resulting EventSet must have the same sampling as the input",
+    ):
+        _ = transformer.fit_transform(X=X)
+
+
+@pytest.mark.skipif(
+    not run_test_for_class(TemporianTransformer),
+    reason="run test only if softdeps are present and incrementally (if requested)",
+)
+def test_complex_function():
+    import temporian as tp
+
+    # in this dataset each row is a period of 1/2 hour
+
+    def function(evset):
+        return evset.simple_moving_average(tp.duration.days(1)).resample(evset)
+
+    X = load_solar()
+
+    transformer = TemporianTransformer(function=function, compile=True)
+    result = transformer.fit_transform(X=X)
+    assert result.iloc[0] == X.iloc[0]
+    assert np.allclose(result.iloc[-1], X.iloc[-48:].mean())