Repository: sktime/sktime

Issue:
[BUG] Cross validation has no effect on KNeighborsTimeSeriesClassifier distance measure parameters
**Describe the bug**

Using scikit-learn cross validation functions on sktime's KNN classifier to set the distance measure hyperparameters always produces the same score.


**To Reproduce**
<!--
Add a Minimal, Complete, and Verifiable example (for more details, see e.g. https://stackoverflow.com/help/mcve

If the code is too long, feel free to put it in a public gist and link it in the issue: https://gist.github.com
-->

```python
from sktime.classification.distance_based import KNeighborsTimeSeriesClassifier
from sktime.datasets import load_basic_motions

from sklearn.model_selection import train_test_split
from sklearn.model_selection import GridSearchCV

import pandas as pd

X, y = load_basic_motions(return_X_y = True)
x_train, x_test, y_train, y_test = train_test_split(X.iloc[:, [0]], y)

param_matrix = {
                "distance_params": [{"w": x / 100} for x in range(0, 100)]
            }
grid = GridSearchCV(
                estimator=KNeighborsTimeSeriesClassifier(
                    distance="dtw", n_neighbors=1
                ),
                param_grid=param_matrix,
                cv=10,
                scoring="accuracy",
            )

grid.fit(x_train, y_train)

print(pd.DataFrame(grid.cv_results_))

    mean_fit_time  std_fit_time  mean_score_time  std_score_time param_distance_params  ... split8_test_score  split9_test_score  mean_test_score  std_test_score  rank_test_score
0        0.003862      0.000397         0.033888        0.001333            {'w': 0.0}  ...               1.0                1.0              1.0             0.0                1
1        0.003609      0.000302         0.033037        0.000653           {'w': 0.01}  ...               1.0                1.0              1.0             0.0                1
2        0.003663      0.000329         0.033945        0.002118           {'w': 0.02}  ...               1.0                1.0              1.0             0.0                1
3        0.003961      0.000472         0.034648        0.001769           {'w': 0.03}  ...               1.0                1.0              1.0             0.0                1
4        0.003810      0.000332         0.033339        0.000514           {'w': 0.04}  ...               1.0                1.0              1.0             0.0                1
..            ...           ...              ...             ...                   ...  ...               ...                ...              ...             ...              ...
95       0.003410      0.000301         0.033238        0.001309           {'w': 0.95}  ...               1.0                1.0              1.0             0.0                1
96       0.003910      0.000375         0.033840        0.001541           {'w': 0.96}  ...               1.0                1.0              1.0             0.0                1
97       0.003459      0.000270         0.032236        0.000451           {'w': 0.97}  ...               1.0                1.0              1.0             0.0                1
98       0.003660      0.000230         0.032135        0.000351           {'w': 0.98}  ...               1.0                1.0              1.0             0.0                1
99       0.003659      0.000230         0.032436        0.000898           {'w': 0.99}  ...               1.0                1.0              1.0             0.0                1

[100 rows x 19 columns]
```

**Expected behavior**

Using different parameters should produce different scores.


**Additional context**

I've tested this bug using multiple different datasets and distance measures, and all produced the same issue. I believe this issue is caused by the use of both "distance_params" and "metric_params" to store the distance measure parameters. Only the value in metric_params is actually passed to the distance measures, but when using cross validation, this is never set as the constuctor is not called, but rather the inital classifier is cloned, so the default value is always used. 

This could be fixed by removing "distance_params" entirely and replacing with "metric_params", which does lead to different scores for different hyperparameters, as shown in the results below.
```python
    mean_fit_time  std_fit_time  mean_score_time  std_score_time param_metric_params  ... split8_test_score  split9_test_score  mean_test_score  std_test_score  rank_test_score
0        0.003861      0.000395         0.017496        0.000875          {'w': 0.0}  ...          0.333333           0.833333         0.600000        0.213437              100
1        0.004018      0.000502         0.018389        0.000853         {'w': 0.01}  ...          1.000000           0.833333         0.933333        0.110554               99
2        0.003804      0.000402         0.018915        0.000394         {'w': 0.02}  ...          1.000000           0.833333         0.950000        0.076376               98
3        0.004018      0.000325         0.021227        0.003138         {'w': 0.03}  ...          1.000000           0.833333         0.966667        0.066667               97
4        0.004812      0.001080         0.022917        0.002269         {'w': 0.04}  ...          1.000000           0.833333         0.983333        0.050000               94
..            ...           ...              ...             ...                 ...  ...               ...                ...              ...             ...              ...
95       0.003760      0.000336         0.033539        0.000612         {'w': 0.95}  ...          1.000000           1.000000         1.000000        0.000000                1
96       0.003957      0.000815         0.035548        0.002654         {'w': 0.96}  ...          1.000000           1.000000         1.000000        0.000000                1
97       0.003910      0.000301         0.033188        0.000201         {'w': 0.97}  ...          1.000000           1.000000         1.000000        0.000000                1
98       0.003609      0.000201         0.033138        0.000351         {'w': 0.98}  ...          1.000000           1.000000         1.000000        0.000000                1
99       0.003762      0.000248         0.033348        0.000860         {'w': 0.99}  ...          1.000000           1.000000         1.000000        0.000000                1

[100 rows x 19 columns]
```

I am raising this as an issue as there may be better solutions whilst keeping the variable names clear.
**Versions**
<details>

<!--
Please run the following code snippet and paste the output here:
 
from sktime import show_versions; show_versions()
-->

System:
    python: 3.7.9 (tags/v3.7.9:13c94747c7, Aug 17 2020, 18:58:18) [MSC v.1900 64 bit (AMD64)]
executable: C:\Users\XXX\AppData\Local\Programs\Python\Python37\python.exe
   machine: Windows-10-10.0.16299-SP0

Python dependencies:
          pip: 20.2.3
   setuptools: 56.2.0
      sklearn: 0.24.2
       sktime: 0.6.1
  statsmodels: 0.12.2
        numpy: 1.20.3
        scipy: 1.5.3
       Cython: 0.29.14
       pandas: 1.1.3
   matplotlib: 3.3.2
       joblib: 0.17.0
        numba: 0.52.0rc3
     pmdarima: 1.7.1
      tsfresh: 0.17.0

<!-- Thanks for contributing! -->



Discussion:
thanks for that aiden, I will create a branch for this and we can look together if you have time?
also, using the default cross validation is probably really inefficient, since it will involve repeated distance calculations. I think Jason wrote an internal mechanism for distance measure tuning. Nevertheless, the classifiers should work with scikit learn croiss validation, so we will fix this bug 
https://github.com/alan-turing-institute/sktime/tree/knn-cross-validation-bug

One solution that would keep the same variable names might be to add a set_params function to the knn classifier, which sets metric_params to equal distance_params. Its a bit of a hacky solution and I'm not sure if it would cause issues elsewhere, but I think it would fix this issue.
Roll this into this issue here 
https://github.com/alan-turing-institute/sktime/issues/596


Resolving patch:
diff --git a/sktime/classification/distance_based/_time_series_neighbors.py b/sktime/classification/distance_based/_time_series_neighbors.py
--- a/sktime/classification/distance_based/_time_series_neighbors.py
+++ b/sktime/classification/distance_based/_time_series_neighbors.py
@@ -102,9 +102,9 @@ def __init__(
         distance_params=None,
         **kwargs
     ):
-        self._distance_params = distance_params
-        if distance_params is None:
-            self._distance_params = {}
+        # self._distance_params = distance_params
+        # if distance_params is None:
+        #    self._distance_params = {}
         self.distance = distance
         self.distance_params = distance_params
 
@@ -141,11 +141,13 @@ def _fit(self, X, y):
         """
         # Transpose to work correctly with distance functions
         X = X.transpose((0, 2, 1))
-
         if isinstance(self.distance, str):
-            self.metric = distance_factory(
-                X[0], X[0], metric=self.distance, **self._distance_params
-            )
+            if self.distance_params is None:
+                self.metric = distance_factory(X[0], X[0], metric=self.distance)
+            else:
+                self.metric = distance_factory(
+                    X[0], X[0], metric=self.distance, **self.distance_params
+                )
 
         y = np.asarray(y)
         check_classification_targets(y)
diff --git a/sktime/distances/lower_bounding.py b/sktime/distances/lower_bounding.py
--- a/sktime/distances/lower_bounding.py
+++ b/sktime/distances/lower_bounding.py
@@ -388,7 +388,11 @@ def create_bounding_matrix(
         _y = self._check_input_timeseries(y)
         if self.int_val == 2:
             if not isinstance(sakoe_chiba_window_radius, float):
-                raise ValueError("The sakoe chiba window must be a float.")
+                raise ValueError(
+                    f"The sakoe chiba window must be a float, passed "
+                    f"{sakoe_chiba_window_radius} of type"
+                    f" {type(sakoe_chiba_window_radius)}."
+                )
             bounding_matrix = sakoe_chiba(_x, _y, sakoe_chiba_window_radius)
         elif self.int_val == 3:
             if not isinstance(itakura_max_slope, float):


Test patch: