Repository: aeon-toolkit/aeon
Pull request #1965 (2024-08-13 12:47:34)

[ENH] Adds the option of using callable distances for kNN regressor
#### Reference Issues/PRs

- Part of: 
    - https://github.com/aeon-toolkit/aeon/pull/1838
    - https://github.com/aeon-toolkit/aeon/issues/1654.
- Needed for KDD distance's notebook:
    -  https://github.com/aeon-toolkit/aeon-tutorials/pull/11

#### What does this implement/fix? Explain your changes.

kNN regressor can't use callable distances. I've made the appropriate changes for allowing its use. 

Also, makes minor minor changes in doc for kNN classifier. 

#### Does your contribution introduce a new dependency? If yes, which one?

None. 

#### Any other comments?

Nope.

### PR checklist


##### For all contributions
- [X] I've added myself to the [list of contributors](https://github.com/aeon-toolkit/aeon/blob/main/.all-contributorsrc). Alternatively, you can use the [@all-contributors](https://allcontributors.org/docs/en/bot/usage) bot to do this for you.
- [X] The PR title starts with either [ENH], [MNT], [DOC], [BUG], [REF], [DEP] or [GOV] indicating whether the PR topic is related to enhancement, maintenance, documentation, bugs, refactoring, deprecation or governance.

##### For new estimators and functions
- [ ] I've added the estimator to the online [API documentation](https://www.aeon-toolkit.org/en/latest/api_reference.html).
- [ ] (OPTIONAL) I've added myself as a `__maintainer__` at the top of relevant files and want to be contacted regarding its maintenance. Unmaintained files may be removed. This is for the full file, and you should not add yourself if you are just making minor changes or do not want to help maintain its contents.

##### For developers with write access
- [ ] (OPTIONAL) I've updated aeon's [CODEOWNERS](https://github.com/aeon-toolkit/aeon/blob/main/CODEOWNERS) to receive notifications about future changes to these files.

<!--
Thanks for contributing!
-->


aeon-actions-bot[bot]: 
## Thank you for contributing to `aeon`

I have added the following labels to this PR based on the title: **[ $\\color{#FEF1BE}{\\textsf{enhancement}}$ ]**.
I have added the following labels to this PR based on the changes made: **[ $\\color{#BCAE15}{\\textsf{classification}}$, $\\color{#7E0206}{\\textsf{regression}}$ ]**. Feel free to change these if they do not properly represent the PR.

The [Checks](https://github.com/aeon-toolkit/aeon/pull/1965/checks) tab will show the status of our automated tests. You can click on individual test runs in the tab or "Details" in the panel below to see more information if there is a failure.

If our `pre-commit` code quality check fails, any trivial fixes will automatically be pushed to your PR unless it is a draft.

Don't hesitate to ask questions on the `aeon` [Slack](https://join.slack.com/t/aeon-toolkit/shared_invite/zt-22vwvut29-HDpCu~7VBUozyfL_8j3dLA) channel if you have any.

### PR CI actions

These checkboxes will add labels to enable/disable CI functionality for this PR. This may not take effect immediately, and a new commit may be required to run the new configuration.

- [ ] Run `pre-commit` checks for all files
- [ ] Run all `pytest` tests and configurations
- [ ] Run all notebook example tests
- [ ] Run numba-disabled `codecov` tests
- [ ] Stop automatic `pre-commit` fixes (always disabled for drafts)
- [ ] Push an empty commit to re-run CI checks
    
MatthewMiddlehurst: Is this the same functionality as the classification version? good to keep them as similar as possible.

Results seem to have changed a little, probably due to the testing distance change?
dguijo: > Is this the same functionality as the classification version? good to keep them as similar as possible.
> 
> Results seem to have changed a little, probably due to the testing distance change?

Yes, it is the same functionality. It was a little outdated: it didn't allow the use of callable distances. 

Re results, you're right, it should be due to the fact of changing the default distance for tests.

Everything should be fine now. Thank you @MatthewMiddlehurst 😄.

Patch:
diff --git a/aeon/classification/distance_based/_time_series_neighbors.py b/aeon/classification/distance_based/_time_series_neighbors.py
index 87da5f6c..92dc2e7e 100644
--- a/aeon/classification/distance_based/_time_series_neighbors.py
+++ b/aeon/classification/distance_based/_time_series_neighbors.py
@@ -29,7 +29,7 @@ class KNeighborsTimeSeriesClassifier(BaseClassifier):
     Parameters
     ----------
     n_neighbors : int, default = 1
-        k for knn.
+        Set k for knn.
     weights : str or callable, default = 'uniform'
         Mechanism for weighting a vote one of: 'uniform', 'distance', or a callable
         function.
@@ -96,7 +96,8 @@ class KNeighborsTimeSeriesClassifier(BaseClassifier):
         super().__init__()
 
     def _fit(self, X, y):
-        """Fit the model using X as training data and y as target values.
+        """
+        Fit the model using X as training data and y as target values.
 
         Parameters
         ----------
@@ -113,7 +114,8 @@ class KNeighborsTimeSeriesClassifier(BaseClassifier):
         return self
 
     def _predict_proba(self, X):
-        """Return probability estimates for the provided data.
+        """
+        Return probability estimates for the provided data.
 
         Parameters
         ----------
@@ -143,15 +145,15 @@ class KNeighborsTimeSeriesClassifier(BaseClassifier):
         return preds
 
     def _predict(self, X):
-        """Predict the class labels for the provided data.
+        """
+        Predict the class labels for the provided data.
 
         Parameters
         ----------
         X : 3D np.ndarray of shape = (n_cases, n_channels, n_timepoints) or list of
         shape[n_cases] of 2D arrays shape (n_channels,n_timepoints_i)
-                If the series are all equal length, a numpy3D will be passed. If
-                unequal, a list of 2D numpy arrays is passed, which may have
-                different lengths.
+        If the series are all equal length, a numpy3D will be passed. If unequal, a list
+        of 2D numpy arrays is passed, which may have different lengths.
 
         Returns
         -------
@@ -173,7 +175,8 @@ class KNeighborsTimeSeriesClassifier(BaseClassifier):
         return preds
 
     def _kneighbors(self, X):
-        """Find the K-neighbors of a point.
+        """
+        Find the K-neighbors of a point.
 
         Returns indices and weights of each point.
 
diff --git a/aeon/regression/distance_based/_time_series_neighbors.py b/aeon/regression/distance_based/_time_series_neighbors.py
index a0de7398..eebba6f8 100644
--- a/aeon/regression/distance_based/_time_series_neighbors.py
+++ b/aeon/regression/distance_based/_time_series_neighbors.py
@@ -8,6 +8,8 @@ It can also be used with callables, or aeon (pairwise transformer) estimators.
 __maintainer__ = []
 __all__ = ["KNeighborsTimeSeriesRegressor"]
 
+from typing import Callable, List, Union
+
 import numpy as np
 
 from aeon.distances import get_distance_function
@@ -20,32 +22,33 @@ class KNeighborsTimeSeriesRegressor(BaseRegressor):
     """
     K-Nearest Neighbour Time Series Regressor.
 
-    An adapted K-Neighbors Regressor for time series data.
-
-    This class is a KNN regressor which supports time series distance measures.
-    It has hardcoded string references to numba based distances in aeon.distances,
-    and can also be used with callables, or aeon (pairwise transformer) estimators.
+    A KNN classifier which supports time series distance measures.
+    It determines distance function through string references to numba
+    based distances in aeon.distances, and can also be used with callables.
 
     Parameters
     ----------
     n_neighbors : int, default = 1
         Set k for knn.
-    weights : str or callable function, default = 'uniform'
-        Mechanism for weighting a vote.
-        one of: 'uniform', 'distance', or a callable function.
+    weights : str or callable, default = 'uniform'
+        Mechanism for weighting a vote one of: 'uniform', 'distance', or a callable
+        function.
     distance : str or callable, default ='dtw'
-        Distance measure between time series
-        if str, must be one of the following strings:
-            'euclidean', 'squared', 'dtw', 'ddtw', 'wdtw', 'wddtw',
-            'lcss', 'edr', 'erp', 'msm', 'twe'
-        this will substitute a hard-coded distance metric from aeon.distances
-        When mpdist is used, the subsequence length (parameter m) must be set
-            Example: knn_mpdist = KNeighborsTimeSeriesClassifier(
-                                metric='mpdist', metric_params={'m':30})
-        if callable, must be of signature (X: np.ndarray, X2: np.ndarray) -> np.ndarray
-            output must be mxn array if X is array of m Series, X2 of n Series.
+        Distance measure between time series.
+        Distance metric to compute similarity between time series. A list of valid
+        strings for metrics can be found in the documentation for
+        :func:`aeon.distances.get_distance_function` or through calling
+        :func:`aeon.distances.get_distance_function_names`. If a
+        callable is passed it must be
+        a function that takes two 2d numpy arrays of shape ``(n_channels,
+        n_timepoints)`` as input and returns a float.
     distance_params : dict, default = None
-        Dictionary for metric parameters , in case that distance is a str.
+        Dictionary for metric parameters for the case that distance is a str.
+    n_jobs : int, default = None
+        The number of parallel jobs to run for neighbors search.
+        ``None`` means 1 unless in a :obj:`joblib.parallel_backend` context.
+        ``-1`` means using all processors. See :term:`Glossary <n_jobs>`
+        for more details. Parameter for compatibility purposes, still unimplemented.
 
     Examples
     --------
@@ -68,14 +71,20 @@ class KNeighborsTimeSeriesRegressor(BaseRegressor):
 
     def __init__(
         self,
-        distance="dtw",
-        distance_params=None,
-        n_neighbors=1,
-        weights="uniform",
-    ):
+        distance: Union[str, Callable] = "dtw",
+        distance_params: dict = None,
+        n_neighbors: int = 1,
+        weights: Union[str, Callable] = "uniform",
+        n_jobs: int = 1,
+    ) -> None:
         self.distance = distance
         self.distance_params = distance_params
         self.n_neighbors = n_neighbors
+        self.n_jobs = n_jobs
+
+        self._distance_params = distance_params
+        if self._distance_params is None:
+            self._distance_params = {}
 
         if weights not in WEIGHTS_SUPPORTED:
             raise ValueError(
@@ -84,47 +93,41 @@ class KNeighborsTimeSeriesRegressor(BaseRegressor):
             )
         self.weights = weights
 
-        self._distance_params = distance_params
-        if self._distance_params is None:
-            self._distance_params = {}
-
         super().__init__()
 
     def _fit(self, X, y):
-        """Fit the model using X as training data and y as target values.
+        """
+        Fit the model using X as training data and y as target values.
 
         Parameters
         ----------
         X : 3D np.ndarray of shape = (n_cases, n_channels, n_timepoints) or list of
         shape[n_cases] of 2D arrays shape (n_channels,n_timepoints_i)
-                If the series are all equal length, a numpy3D will be passed. If
-                unequal, a list of 2D numpy arrays is passed, which may have
-                different lengths.
+        If the series are all equal length, a numpy3D will be passed. If unequal, a list
+        of 2D numpy arrays is passed, which may have different lengths.
         y : array-like, shape = (n_cases)
-            The class labels.
+            The output value.
         """
-        if isinstance(self.distance, str):
-            self.metric_ = get_distance_function(metric=self.distance)
-
+        self.metric_ = get_distance_function(metric=self.distance)
         self.X_ = X
         self.y_ = y
         return self
 
     def _predict(self, X):
-        """Predict the target values for the provided data.
+        """
+        Predict the target values for the provided data.
 
         Parameters
         ----------
         X : 3D np.ndarray of shape = (n_cases, n_channels, n_timepoints) or list of
         shape[n_cases] of 2D arrays shape (n_channels,n_timepoints_i)
-                If the series are all equal length, a numpy3D will be passed. If
-                unequal, a list of 2D numpy arrays is passed, which may have
-                different lengths.
+        If the series are all equal length, a numpy3D will be passed. If unequal, a list
+        of 2D numpy arrays is passed, which may have different lengths.
 
         Returns
         -------
         y : array of shape (n_cases)
-            Class labels for each data sample.
+            Output values for each data sample.
         """
         self.check_is_fitted()
 
@@ -136,17 +139,15 @@ class KNeighborsTimeSeriesRegressor(BaseRegressor):
         return preds
 
     def _kneighbors(self, X):
-        """Find the K-neighbors of a point.
+        """
+        Find the K-neighbors of a point.
 
         Returns indices and weights of each point.
 
         Parameters
         ----------
-        X : 3D np.ndarray of shape = (n_cases, n_channels, n_timepoints) or list of
-        shape[n_cases] of 2D arrays shape (n_channels,n_timepoints_i)
-                If the series are all equal length, a numpy3D will be passed. If
-                unequal, a list of 2D numpy arrays is passed, which may have
-                different lengths.
+        X : np.ndarray
+            A single time series instance if shape = (n_channels, n_timepoints)
 
         Returns
         -------
@@ -172,7 +173,6 @@ class KNeighborsTimeSeriesRegressor(BaseRegressor):
 
         if self.weights == "distance":
             ws = distances[closest_idx]
-
             # Using epsilon ~= 0 to avoid division by zero
             ws = 1 / (ws + np.finfo(float).eps)
         elif self.weights == "uniform":
@@ -181,3 +181,26 @@ class KNeighborsTimeSeriesRegressor(BaseRegressor):
             raise Exception(f"Invalid kNN weights: {self.weights}")
 
         return closest_idx, ws
+
+    @classmethod
+    def get_test_params(cls, parameter_set: str = "default") -> Union[dict, List[dict]]:
+        """Return testing parameter settings for the estimator.
+
+        Parameters
+        ----------
+        parameter_set : str, default="default"
+            Name of the set of test parameters to return, for use in tests. If no
+            special parameters are defined for a value, will return `"default"` set.
+
+        Returns
+        -------
+        params : dict or list of dict, default={}
+            Parameters to create testing instances of the class.
+            Each dict are parameters to construct an "interesting" test instance, i.e.,
+            `MyClass(**params)` or `MyClass(**params[i])` creates a valid test instance.
+            `create_test_instance` uses the first (or only) dictionary in `params`.
+        """
+        # non-default distance and algorithm
+        params1 = {"distance": "euclidean"}
+
+        return [params1]


Test patch:
diff --git a/aeon/classification/distance_based/tests/test_time_series_neighbors.py b/aeon/classification/distance_based/tests/test_time_series_neighbors.py
index 0d674b4a..7746439d 100644
--- a/aeon/classification/distance_based/tests/test_time_series_neighbors.py
+++ b/aeon/classification/distance_based/tests/test_time_series_neighbors.py
@@ -2,10 +2,9 @@
 
 import pytest
 
-from aeon.classification.distance_based._time_series_neighbors import (
-    KNeighborsTimeSeriesClassifier,
-)
+from aeon.classification.distance_based import KNeighborsTimeSeriesClassifier
 from aeon.datasets import load_unit_test
+from aeon.distances import get_distance_function
 
 distance_functions = [
     "euclidean",
@@ -65,8 +64,11 @@ def test_knn_bounding_matrix(distance_key):
         return
     X_train, y_train = load_unit_test(split="train")
     X_test, y_test = load_unit_test(split="test")
+
+    distance_callable = get_distance_function(distance_key)
+
     knn = KNeighborsTimeSeriesClassifier(
-        distance=distance_key, distance_params={"window": 0.5}
+        distance=distance_callable, distance_params={"window": 0.5}
     )
     knn.fit(X_train, y_train)
     pred = knn.predict(X_test)
diff --git a/aeon/regression/distance_based/tests/test_time_series_neighbors.py b/aeon/regression/distance_based/tests/test_time_series_neighbors.py
index 46499ec6..cb814755 100644
--- a/aeon/regression/distance_based/tests/test_time_series_neighbors.py
+++ b/aeon/regression/distance_based/tests/test_time_series_neighbors.py
@@ -1,25 +1,73 @@
 """Tests for KNeighborsTimeSeriesRegressor."""
 
-import numpy as np
+import pytest
+from numpy.testing import assert_almost_equal
+from sklearn.metrics import mean_squared_error
 
-from aeon.regression.distance_based._time_series_neighbors import (
-    KNeighborsTimeSeriesRegressor,
-)
+from aeon.datasets import load_covid_3month
+from aeon.distances import get_distance_function
+from aeon.regression.distance_based import KNeighborsTimeSeriesRegressor
 
+distance_functions = ["euclidean", "dtw", "wdtw", "msm", "erp", "adtw", "twe"]
 
-def test_knn_neighbors():
-    """Tests kneighbors method."""
-    from sklearn.datasets import make_regression
-    from sklearn.model_selection import train_test_split
+# expected mse on test set using default parameters.
+expected_mse = {
+    "euclidean": 0.002815386587822589,
+    "dtw": 0.002921957478363366,
+    "wdtw": 0.0025029139202436303,
+    "msm": 0.002427566155284863,
+    "erp": 0.002247674986547397,
+    "adtw": 0.00265555172857104,
+    "twe": 0.0028423024613138774,
+}
 
-    X, y = make_regression(n_samples=9, n_features=3, random_state=42)
-    X = X.reshape((X.shape[0], 1, X.shape[1]))
-    X_train, X_test, y_train, _ = train_test_split(X, y, test_size=3, random_state=42)
+# expected mse on test set using window params.
+expected_mse_window = {
+    "dtw": 0.0027199984296669712,
+    "wdtw": 0.0026043512829531305,
+    "msm": 0.002413148537646331,
+    "erp": 0.0021331320891357546,
+    "adtw": 0.0027602314681382163,
+    "twe": 0.0030244991099088346,
+}
 
-    model = KNeighborsTimeSeriesRegressor(n_neighbors=5, weights="distance")
-    model.fit(X_train, y_train)
 
+@pytest.mark.parametrize("distance_key", distance_functions)
+def test_knn_neighbors(distance_key):
+    """Tests kneighbors method."""
+    X_train, y_train = load_covid_3month(split="train")
+    X_test, y_test = load_covid_3month(split="test")
+
+    model = KNeighborsTimeSeriesRegressor(
+        n_neighbors=1, weights="distance", distance=distance_key
+    )
+    model.fit(X_train, y_train)
     y_pred = model.predict(X_test)
-    y_pred_expected = np.array([-144.410377008, -25.55876587, -229.9764678])
 
-    assert np.abs(y_pred - y_pred_expected).max() < 1e-4
+    mse = mean_squared_error(y_test, y_pred)
+    assert_almost_equal(mse, expected_mse[distance_key])
+
+
+@pytest.mark.parametrize("distance_key", distance_functions)
+def test_knn_bounding_matrix(distance_key):
+    """Test knn with custom bounding parameters, and using callables."""
+    if distance_key == "euclidean" or distance_key == "squared":
+        return
+    X_train, y_train = load_covid_3month(split="train")
+    X_test, y_test = load_covid_3month(split="test")
+    distance_callable = get_distance_function(distance_key)
+
+    knn = KNeighborsTimeSeriesRegressor(
+        distance=distance_callable, distance_params={"window": 0.1}
+    )
+    knn.fit(X_train, y_train)
+    y_pred = knn.predict(X_test)
+
+    mse = mean_squared_error(y_test, y_pred)
+    assert_almost_equal(mse, expected_mse_window[distance_key])
+
+
+if __name__ == "__main__":
+    for distance_key in distance_functions:
+        test_knn_neighbors(distance_key)
+        test_knn_bounding_matrix(distance_key)
diff --git a/aeon/testing/expected_results/expected_regressor_outputs.py b/aeon/testing/expected_results/expected_regressor_outputs.py
index ecc4a1ef..cd89dc2c 100644
--- a/aeon/testing/expected_results/expected_regressor_outputs.py
+++ b/aeon/testing/expected_results/expected_regressor_outputs.py
@@ -108,11 +108,11 @@ covid_3month_preds["KNeighborsTimeSeriesRegressor"] = np.array(
     [
         0.0081,
         0.1111,
-        0.0408,
-        0.0212,
+        0.0,
+        0.0,
         0.0557,
         0.0408,
-        0.0818,
+        0.0212,
         0.1111,
         0.1111,
         0.0,
@@ -294,16 +294,16 @@ cardano_sentiment_preds["RocketRegressor"] = np.array(
 
 cardano_sentiment_preds["KNeighborsTimeSeriesRegressor"] = np.array(
     [
+        0.178,
+        0.0,
         0.3503,
+        0.0,
         -0.101,
-        0.3847,
         0.0,
-        0.3847,
         0.0,
-        0.178,
         0.0,
         0.0,
-        0.3847,
+        0.25,
     ]
 )