Repository: sktime/sktime

Issue:
[ENH] F-Stat as an optional quality measure for ShapeletTransformClassifier
**Is your feature request related to a problem? Please describe.**
Hills et al 2014 claim F-stat measure is both faster and more discriminative  than information gain. It would be nice to have it as an option for the classifier and the transforms in general.


**Describe the solution you'd like**
Add quality_measure as a parameter to the shapelet-related classes, add f-stat as an option next to information_gain (and other quality measures ideally) and replace uses and calculations of the information gain with the quality measure given from the parameter. Now the classes seem to all be hard coded to use information gain.






Discussion:
Hm, the `sktime` shapelet transform, `ShapeletTransform`, does indeed not support anova f-stat. It is also quite old code which is no longer maintained by its authors.

I see two options here, not mutually exclusive:

* the [`pyts` shapelet transform](https://pyts.readthedocs.io/en/stable/generated/pyts.transformation.ShapeletTransform.html#pyts.transformation.ShapeletTransform) has f-stat as an option next to entropy (it's the option `criterion="anova"`). We could simply interface `pyts`, it has a number of nice transformers and other estimators that could be interesting. The extension template `transformer_supersimple` should provide a quick starting point for this.
* as the `ShapeletTransform` is abandoned by its authors, someone could pick it up and add f-stat as a criterion. I suppose this would require adding one level of abstraction with regards to the criterion, at any point where `binary_entropy` and related utilities are called.

Would you like to pick up either of these?
adding the good first issue for the solution pathway "interfacing `pyts`".

Extending the `sktime` `ShapeletTransform` seems fiddlier, not a good first issue imo.
I'd like to work on the first option of interfacing `pyts`. Any pointers and other examples I could look to for a reference would be really helpful.
Sure!

The class `RocketPyts` is an example of interfacing a transformer from `pyts`. I would expect the pattern to be easily adaptable - of course there could be complications, as `pyts` interfaces are relatively new, but wouldn't expect any.

Some non-obvious things:

* when we interface multiple variants of the same algorithm, the pattern followed so far is changing a modul to a folder with private submodules and common export
* we should not forget to add new estimators to the API reference in the docs
Ok so I make a new folder called `shapelet_transform` in transformations/panel and put `shapelet_transform.py` in it along with the new `shapelet_transform_pyts.py` that I will be making. Is this correct or any naming conventions I should follow?
Yes, that's how I would do it - plus an `__init__` which exports the classes from both, to make sure we don't breka any existing imports. I would also make the new `py` files, private, i.e., starting with an underscore, e.g., `_shapelet_transform.py`.

Resolving patch:
diff --git a/sktime/transformations/panel/shapelet_transform/__init__.py b/sktime/transformations/panel/shapelet_transform/__init__.py
new file mode 100644
--- /dev/null
+++ b/sktime/transformations/panel/shapelet_transform/__init__.py
@@ -0,0 +1,14 @@
+"""Shapelet Transformers."""
+__all__ = [
+    "ShapeletTransform",
+    "RandomShapeletTransform",
+    "ShapeletTransformPyts",
+]
+
+from sktime.transformations.panel.shapelet_transform._shapelet_transform import (
+    RandomShapeletTransform,
+    ShapeletTransform,
+)
+from sktime.transformations.panel.shapelet_transform._shapelet_transform_pyts import (
+    ShapeletTransformPyts,
+)
diff --git a/sktime/transformations/panel/shapelet_transform.py b/sktime/transformations/panel/shapelet_transform/_shapelet_transform.py
similarity index 100%
rename from sktime/transformations/panel/shapelet_transform.py
rename to sktime/transformations/panel/shapelet_transform/_shapelet_transform.py
diff --git a/sktime/transformations/panel/shapelet_transform/_shapelet_transform_pyts.py b/sktime/transformations/panel/shapelet_transform/_shapelet_transform_pyts.py
new file mode 100644
--- /dev/null
+++ b/sktime/transformations/panel/shapelet_transform/_shapelet_transform_pyts.py
@@ -0,0 +1,196 @@
+"""Shapelet transformer, from pyts."""
+# copyright: sktime developers, BSD-3-Clause License (see LICENSE file)
+
+__author__ = ["Abhay-Lejith"]
+__all__ = ["ShapeletTransformPyts"]
+
+from sktime.base.adapters._pyts import _PytsAdapter
+from sktime.transformations.base import BaseTransformer
+
+
+class ShapeletTransformPyts(_PytsAdapter, BaseTransformer):
+    """Shapelet Transform, from ``pyts``.
+
+    Direct interface to ``pyts.transformation.ShapeletTransform``.
+
+    The Shapelet Transform algorithm extracts shapelets from a data set of time series
+    and returns the distances between the shapelets and the time series. A shapelet is
+    defined as a subset of a time series, that is a set of values from consecutive time
+    points. The distance between a shapelet and a time series is defined as the minimum
+    of the distances between this shapelet and all the shapelets of same length
+    extracted from this time series. The most discriminative shapelets are selected.
+    Two criteria are made available: mutual information and F-scores.
+
+    Parameters
+    ----------
+    n_shapelets : int or 'auto' (default = 'auto')
+        The number of shapelets to keep. If 'auto', `n_timestamps // 2`
+        shapelets are considered, where `n_timestamps` is the number of
+        time points in the dataset. Note that there might be a smaller
+        number of shapelets if fewer than ``n_shapelets`` shapelets have been
+        extracted during the search.
+
+    criterion : 'mutual_info' or 'anova' (default = 'mutual_info')
+        Criterion to perform the selection of the shapelets.
+        'mutual_info' uses the mutual information, while 'anova' use
+        the ANOVA F-value.
+
+    window_sizes : array-like or 'auto '(default = 'auto')
+        Size of the sliding windows. If 'auto', the range for the
+        window sizes is determined automatically.
+        Otherwise, all the elements must be either integers or floats.
+        In the latter case, each element represents the percentage
+        of the size of each time series and must be between 0 and 1; the size
+        of the sliding windows will be computed as
+        ``np.ceil(window_sizes * n_timestamps)``.
+
+    window_steps : None or array-like (default = None)
+        Step of the sliding windows. If None, each ``window_step`` is equal
+        to 1. Otherwise, all the elements must be either integers or floats.
+        In the latter case, each element represents the percentage of the size
+        of each time series and must be between 0 and 1; the step of the
+        sliding windows will be computed as
+        ``np.ceil(window_steps * n_timestamps)``.
+        Must be None if ``window_sizes='auto'``.
+
+    remove_similar : bool (default = True)
+        If True, self-similar shapelets are removed, keeping only
+        the non-self-similar shapelets with the highest scores. Two
+        shapelets are considered to be self-similar if they are
+        taken from the the same time series and have at least one
+        overlapping index.
+
+    sort : bool (default = False)
+        If True, shapelets are sorted in descending order according to
+        their associated scores. If False, the order is undefined.
+
+    verbose : int (default = 0)
+        Verbosity level when fitting: if non zero, progress messages are
+        printed. Above 50, the output is sent to stdout.
+        The frequency of the messages increases with the verbosity level.
+
+    random_state : int, RandomState instance or None (default = None)
+        If int, random_state is the seed used by the random number generator;
+        If RandomState instance, random_state is the random number generator;
+        If None, the random number generator is the RandomState instance used
+        by ``np.random``. Only used if ``window_sizes='auto'`` in order to
+        subsample the dataset to find the best range or if
+        ``criterion=='mutual_info'`` to add small noise to the data.
+
+    n_jobs : None or int (default = None)
+        The number of jobs to run in parallel for ``fit``.
+        If -1, then the number of jobs is set to the number of cores.
+
+    Attributes
+    ----------
+    shapelets_ : array, shape = (n_shapelets,)
+        The array with the selected shapelets.
+
+    indices_ : array, shape = (n_shapelets, 3)
+        The indices for the corresponding shapelets in the training set.
+        The first column consists of the indices of the samples.
+        The second column consists of the starting indices (included)
+        of the shapelets. The third column consists of the ending indices
+        (excluded) of the shapelets.
+
+    scores_ : array, shape = (n_shapelets,)
+        The scores associated to the shapelets. The higher, the more
+        discriminant.
+        If ``criterion='mutual_info'``, mutual information scores are reported.
+        If ``criterion='anova'``, F-scores are reported.
+
+    window_range_ : None or tuple
+        Range of the window sizes if ``window_sizes='auto'``. None otherwise.
+
+    References
+    ----------
+    .. [1] J. Lines, L. M. Davis, J. Hills and A. Bagnall, "A Shapelet
+           Transform for Time Series Classification". Data Mining and Knowledge
+           Discovery, 289-297 (2012).
+
+    Examples
+    --------
+    >>> from sktime.transformations.panel.shapelet_transform import (
+    ...     ShapeletTransformPyts
+    ... )
+    >>> from sktime.datasets import load_unit_test
+    >>> X_train, y_train = load_unit_test(split="train") # doctest: +SKIP
+    >>> stp = ShapeletTransformPyts() # doctest: +SKIP
+    >>> stp.fit(X_train,y_train) # doctest: +SKIP
+    >>> stp.transform(X_train) # doctest: +SKIP
+    """
+
+    _tags = {
+        # packaging info
+        # --------------
+        "authors": "Abhay-Lejith",
+        "python_dependencies": "pyts",
+        # univariate-only controls whether internal X can be univariate/multivariate
+        # if True (only univariate), always applies vectorization over variables
+        "univariate-only": True,
+        "scitype:transform-input": "Series",
+        "scitype:transform-output": "Primitives",
+        "scitype:instancewise": False,
+        "fit_is_empty": False,
+        "y_inner_mtype": "numpy1D",
+        "requires_y": True,
+    }
+
+    _estimator_attr = "_pyts_shapelet_transform"
+
+    def _get_pyts_class(self):
+        """Get pyts class.
+
+        should import and return pyts class
+        """
+        from pyts.transformation.shapelet_transform import ShapeletTransform
+
+        return ShapeletTransform
+
+    def __init__(
+        self,
+        n_shapelets="auto",
+        criterion="mutual_info",
+        window_sizes="auto",
+        window_steps=None,
+        remove_similar=True,
+        sort=False,
+        verbose=0,
+        random_state=None,
+        n_jobs=None,
+    ):
+        self.n_shapelets = n_shapelets
+        self.criterion = criterion
+        self.window_sizes = window_sizes
+        self.window_steps = window_steps
+        self.remove_similar = remove_similar
+        self.sort = sort
+        self.verbose = verbose
+        self.random_state = random_state
+        self.n_jobs = n_jobs
+
+        super().__init__()
+
+    @classmethod
+    def get_test_params(cls, parameter_set="default"):
+        """
+        Return testing parameter settings for the estimator.
+
+        Parameters
+        ----------
+        parameter_set : str, default="default"
+            Name of the set of test parameters to return, for use in tests. If no
+            special parameters are defined for a value, will return `"default"` set.
+
+
+        Returns
+        -------
+        params : dict or list of dict, default = {}
+            Parameters to create testing instances of the class
+            Each dict are parameters to construct an "interesting" test instance, i.e.,
+            `MyClass(**params)` or `MyClass(**params[i])` creates a valid test instance.
+            `create_test_instance` uses the first (or only) dictionary in `params`
+        """
+        params1 = {"criterion": "anova", "n_shapelets": 10}
+        params2 = {"window_sizes": [3]}
+        return [params1, params2]


Test patch:
diff --git a/sktime/transformations/panel/shapelet_transform/tests/__init__.py b/sktime/transformations/panel/shapelet_transform/tests/__init__.py
new file mode 100644
--- /dev/null
+++ b/sktime/transformations/panel/shapelet_transform/tests/__init__.py
@@ -0,0 +1 @@
+"""Shapelet transform unit tests."""
diff --git a/sktime/transformations/panel/tests/test_shapelet_transform.py b/sktime/transformations/panel/shapelet_transform/tests/test_shapelet_transform.py
similarity index 100%
rename from sktime/transformations/panel/tests/test_shapelet_transform.py
rename to sktime/transformations/panel/shapelet_transform/tests/test_shapelet_transform.py
diff --git a/sktime/transformations/panel/shapelet_transform/tests/test_shapelet_transform_pyts.py b/sktime/transformations/panel/shapelet_transform/tests/test_shapelet_transform_pyts.py
new file mode 100644
--- /dev/null
+++ b/sktime/transformations/panel/shapelet_transform/tests/test_shapelet_transform_pyts.py
@@ -0,0 +1,40 @@
+"""ShapeletTransformPyts test code."""
+import numpy as np
+import pytest
+from numpy import testing
+
+from sktime.datasets import load_unit_test
+from sktime.tests.test_switch import run_test_for_class
+from sktime.transformations.panel.shapelet_transform import ShapeletTransformPyts
+
+
+@pytest.mark.skipif(
+    not run_test_for_class(ShapeletTransformPyts),
+    reason="run test only if softdeps are present and incrementally (if requested)",
+)
+def test_stp_on_unit_test():
+    """Test of ShapeletTransformPyts on unit test data."""
+    # load unit test data
+    X_train, y_train = load_unit_test(split="train")
+    indices = np.random.RandomState(0).choice(len(y_train), 5, replace=False)
+
+    # fit the shapelet transform
+    stp = ShapeletTransformPyts(n_shapelets=5, random_state=0)
+    stp.fit(X_train.iloc[indices], y_train[indices])
+
+    # assert transformed data is the same
+    data = stp.transform(X_train.iloc[indices])
+    testing.assert_array_almost_equal(
+        data, shapelet_transform_unit_test_data, decimal=4
+    )
+
+
+shapelet_transform_unit_test_data = np.array(
+    [
+        [176.0, 14.866069, 17.262677, 0.000000, 7.0],
+        [0.0, 51.908573, 82.197932, 65.517173, 15.0],
+        [223.0, 30.413813, 0.000000, 17.262677, 7.0],
+        [1.0, 61.159627, 71.515733, 59.502101, 19.0],
+        [32.0, 0.000000, 30.413813, 14.866069, 0.0],
+    ]
+)