Repository: aeon-toolkit/aeon
Pull request #1993 (2024-08-20 12:48:48)

[ENH] Added Regressor for Channel Scorer
#1951 

what at does this implement/fix? Explain your changes.
I have added a channel_scorer for a regressor which evaluates through MSE.




aeon-actions-bot[bot]: 
## Thank you for contributing to `aeon`

I did not find any labels to add based on the title. Please add the [ENH], [MNT], [BUG], [DOC], [REF], [DEP] and/or [GOV] tags to your pull requests titles. For now you can add the labels manually.
I have added the following labels to this PR based on the changes made: **[ $\\color{#41A8F6}{\\textsf{transformations}}$ ]**. Feel free to change these if they do not properly represent the PR.

The [Checks](https://github.com/aeon-toolkit/aeon/pull/1993/checks) tab will show the status of our automated tests. You can click on individual test runs in the tab or "Details" in the panel below to see more information if there is a failure.

If our `pre-commit` code quality check fails, any trivial fixes will automatically be pushed to your PR unless it is a draft.

Don't hesitate to ask questions on the `aeon` [Slack](https://join.slack.com/t/aeon-toolkit/shared_invite/zt-22vwvut29-HDpCu~7VBUozyfL_8j3dLA) channel if you have any.

### PR CI actions

These checkboxes will add labels to enable/disable CI functionality for this PR. This may not take effect immediately, and a new commit may be required to run the new configuration.

- [ ] Run `pre-commit` checks for all files
- [ ] Run all `pytest` tests and configurations
- [ ] Run all notebook example tests
- [ ] Run numba-disabled `codecov` tests
- [ ] Stop automatic `pre-commit` fixes (always disabled for drafts)
- [ ] Push an empty commit to re-run CI checks
    
aryanpola: @TonyBagnall 
aryanpola: Is there any way to run these tests locally? @MatthewMiddlehurst 
MatthewMiddlehurst: Yes, it all uses the `pytest` framework. You should be able to run all tests or individual ones by configuring the command

https://docs.pytest.org/en/stable/how-to/index.html
aryanpola: @MatthewMiddlehurst Is the code fine? please let me know if there are any changes to be made.
aryanpola: @baraline I changed the code and ran it, there is an error occurring. Can you help me with it.
aryanpola: @MatthewMiddlehurst Please let me know if there any more changes to be made.
baraline: One option would be to have the estimator as a mandatory parameter, moving the burden of choosing the correct one depending on the task to the user. Regarding the scoring function, having it as a parameter (which can default to accuracy/mse based on the estimator type) can be really useful for cases where these two default metrics are not ideal to use. 
TonyBagnall: thanks for this, just catching up, will take a look and discuss and get back to you
TonyBagnall: so I think it is currently over complex, but I'm not actually sure of my preferred solution. I think it best to require the estimator be passed then just fail if y is non numeric, but Im not actually sure. I'll discuss and get back to you
MatthewMiddlehurst: I like @baraline's suggestion, the issue is this is a published method which uses ROCKET by default I believe @TonyBagnall? 

We discussed this a bit at ECML, I think the best way is to introduce a "scoring" parameter which takes a string or function defaulting to accuracy, and keep the default estimator to ROCKET classifier.

There's not really a good way to automatically to select between classification and regression, and we typically try to stay away from those types of ambiguous inferences.
baraline: If this is indeed a published method Matthew would be best 
TonyBagnall: > I like @baraline's suggestion, the issue is this is a published method which uses ROCKET by default I believe @TonyBagnall?
> 
> We discussed this a bit at ECML, I think the best way is to introduce a "scoring" parameter which takes a string or function defaulting to accuracy, and keep the default estimator to ROCKET classifier.
> 
> There's not really a good way to automatically to select between classification and regression, and we typically try to stay away from those types of ambiguous inferences.

so its published but I dont think that should block things, I rthink @baraline's solution is probably the best on reflection, in the example I can explain the relationship to the paper later

This is assuming we are talking about this: 
"One option would be to have the estimator as a mandatory parameter, moving the burden of choosing the correct one depending on the task to the user. Regarding the scoring function, having it as a parameter (which can default to accuracy/mse based on the estimator type) can be really useful for cases where these two default metrics are not ideal to use."

so make both the estimator and the scorer and option. Sorry it became so complex @aryanpola 
aryanpola: 


> > I like @baraline's suggestion, the issue is this is a published method which uses ROCKET by default I believe @TonyBagnall?
> > We discussed this a bit at ECML, I think the best way is to introduce a "scoring" parameter which takes a string or function defaulting to accuracy, and keep the default estimator to ROCKET classifier.
> > There's not really a good way to automatically to select between classification and regression, and we typically try to stay away from those types of ambiguous inferences.
> 
> so its published but I dont think that should block things, I rthink @baraline's solution is probably the best on reflection, in the example I can explain the relationship to the paper later
> 
> This is assuming we are talking about this: "One option would be to have the estimator as a mandatory parameter, moving the burden of choosing the correct one depending on the task to the user. Regarding the scoring function, having it as a parameter (which can default to accuracy/mse based on the estimator type) can be really useful for cases where these two default metrics are not ideal to use."
> 
> so make both the estimator and the scorer and option. Sorry it became so complex @aryanpola

@baraline @MatthewMiddlehurst @TonyBagnall 
1. So now, if the user doesn't select any estimator it defaults to data type, depending on the data type it defaults to either a classifier or a regressor where the user's scoring function is used, else the default mse and accuracy scores.
2. If the user selects a BaseClassifier as estimator, the user's scoring function is used else default accuracy score.
3. If the user selects a BaseRegressor as estimator, the user's scoring function is used else default mse scoring.

aryanpola: @MatthewMiddlehurst all the cases are passing locally. Adding the estimator in aeon/transformations/collection/tests/test_all_collection_transformers.py causing more errors.
aryanpola: > Couple more comments, but I think this should be good after. Thanks for making the suggested changes, was a bit complex 🙂

Please let me know if there's anything more to do. Really liking the work, the community is constructive and responsive 😊.
MatthewMiddlehurst: I am struggling to figure out why the results for RDST/RIST have changed. This is not happening in any other PR. I thought maybe it was due to this being a fork, but other PRs on the same fork seem to be fine.
aryanpola: @MatthewMiddlehurst Maybe set random state in `ExtraTreesClassifier' which could provide a deterministic result?
baraline: > I am struggling to figure out why the deterministic results for RDST/RIST have changed. This is not happening in any other PR. I thought maybe it was due to this being a fork, but other PRs on the same fork seem to be fine.

Sooo, from what I understand, this is an issue with the legacy numpy generator when using numba : 

> Numba supports top-level functions from the numpy.random module, but does not allow you to create individual RandomState instances. The same algorithms are used as for the standard random module, but with an independent internal state: seeding or drawing numbers from one generator won’t affect the other.

Looking at the docs of the random module:

> Numba supports top-level functions from the [random](https://docs.python.org/3/library/random.html#module-random) module, but does not allow you to create individual Random instances. A Mersenne-Twister generator is used, with a dedicated internal state. It is initialized at startup with entropy drawn from the operating system.

That last part is I think what might be getting us some trouble with numba random using legacy numpy generator as **it is initialized at startup with entropy drawn from the operating system.**

The solution would be to switch to the new random API :

> Numba supports numpy.random.Generator() objects. As of version 0.56, users can pass individual NumPy Generator objects into Numba functions and use their methods inside the functions. The same algorithms are used as NumPy for random number generation hence maintaining parity between the random number generated using NumPy and Numba under identical arguments (also the same documentation notes as NumPy Generator methods apply). The current Numba support for Generator is not thread-safe, hence we do not recommend using Generator methods in methods with parallel execution logic.

But then we'll need to check the last part, as **Generator is not thread-safe**. I don't know what this would imply though...

MatthewMiddlehurst: My issue is more why it is only impacting this PR though.
baraline: I remember some bug with RDST that only appeared sporadically on macOS, maybe this is kind of the same thing? Don't have a certain answer, only that the changes in this PR have no reason to impact the failing methods ...
aryanpola: ohh! What made the test cases pass this time?
MatthewMiddlehurst: Fixed in a another PR, still not sure why it only appeared here though.

Patch:
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
index 265e81c2..4a647e75 100644
--- a/.pre-commit-config.yaml
+++ b/.pre-commit-config.yaml
@@ -1,6 +1,6 @@
 repos:
   - repo: https://github.com/pre-commit/pre-commit-hooks
-    rev: v4.6.0
+    rev: v5.0.0
     hooks:
       - id: check-ast
       - id: check-added-large-files
@@ -29,7 +29,7 @@ repos:
         args: [ "--create", "--python-folders", "aeon" ]
 
   - repo: https://github.com/astral-sh/ruff-pre-commit
-    rev: v0.6.8
+    rev: v0.6.9
     hooks:
       - id: ruff
         args: [ "--fix"]
diff --git a/aeon/transformations/collection/channel_selection/_channel_scorer.py b/aeon/transformations/collection/channel_selection/_channel_scorer.py
index 32c211dc..1559d7c5 100644
--- a/aeon/transformations/collection/channel_selection/_channel_scorer.py
+++ b/aeon/transformations/collection/channel_selection/_channel_scorer.py
@@ -1,30 +1,51 @@
+"""Selects channels based on an estimate of performance."""
+
+__maintainer__ = ["TonyBagnall"]
+__all__ = ["ChannelScorer"]
+
 import math
-from typing import Optional
+from typing import Callable
+from typing import Dict as TypingDict
+from typing import List as TypingList
+from typing import Optional, Union
 
 import numpy as np
-from sklearn.metrics import accuracy_score
+from sklearn.metrics import accuracy_score, mean_squared_error
 
+from aeon.base import BaseEstimator
 from aeon.classification.base import BaseClassifier
-from aeon.classification.convolution_based._rocket_classifier import RocketClassifier
+from aeon.regression.base import BaseRegressor
 from aeon.transformations.collection.channel_selection.base import BaseChannelSelector
 
-__maintainer__ = ["TonyBagnall"]
-__all__ = ["ChannelScorer"]
-
 
 class ChannelScorer(BaseChannelSelector):
-    """Channel scorer performs channel selection using a single channel classifier.
+    """Performs channel selection using a single channel classifier or regressor.
 
-    ChannelScorer uses a time series classifier to score each channel using an
-    estimate of accuracy on the training data, then selects a proportion of the top
+    ChannelScorer uses a time series classifier or a regressor to score each channel
+    using an estimate of accuracy on the training data fro classifier or mean
+    squared error for regressor, then selects a proportion of the top
     channels to keep. Can be configured through the constructor to use any time
-    series classifier and could easily be adapted to use forward selection or elbow
+    series estimator and could easily be adapted to use forward selection or elbow
     class methods. Approximately as described in [1]_.
 
     Parameters
     ----------
-    classifier, BaseClassifier, default = MiniROCKET
-    proportion : float, default = 0.2
+    estimator : BaseEstimator
+        The time series estimator used to score each channel.
+
+    scoring_function : Callable, optional (default=None)
+        Scoring function used to evaluate the performance of each channel.
+        Defaults to:
+        - `accuracy_score` if using a classifier.
+        - `mean_squared_error` if using a regressor.
+
+    score_sign : float, optional (default=None)
+        The sign used to determine whether higher or lower scores are better.
+        Defaults to:
+        - 1.0 for accuracy_score (higher scores are better).
+        - -1.0 for mean_squared_error (lower scores are better).
+
+    proportion : float, default = 0.4
         Proportion of channels to keep, rounded up to nearest integer.
 
     References
@@ -41,16 +62,20 @@ class ChannelScorer(BaseChannelSelector):
 
     def __init__(
         self,
-        classifier: Optional[BaseClassifier] = None,
+        estimator: Optional[BaseEstimator],
+        scoring_function: Callable = None,
+        score_sign: float = None,
         proportion: float = 0.4,
     ):
         self.proportion = proportion
-        self.classifier = classifier
+        self.estimator = estimator
+        self.scoring_function = scoring_function
+        self.score_sign = score_sign
         super().__init__()
 
-    def _fit(self, X, y):
+    def _fit(self, X: np.ndarray, y: Union[np.ndarray, TypingList]):
         """
-        Fit ECP to a specified X and y.
+        Fit to a specified X and y.
 
         Parameters
         ----------
@@ -66,23 +91,37 @@ class ChannelScorer(BaseChannelSelector):
         if self.proportion <= 0 or self.proportion > 1:
             raise ValueError("proportion must be in the range 0-1")
 
-        if self.classifier is None:
-            self.classifier_ = RocketClassifier(
-                rocket_transform="minirocket", num_kernels=5000
+        if isinstance(self.estimator, BaseClassifier):
+            self.estimator_ = self.estimator.clone()
+            scoring_function = (
+                accuracy_score
+                if self.scoring_function is None
+                else self.scoring_function
             )
-        elif not isinstance(self.classifier, BaseClassifier):
-            raise ValueError(
-                "parameter classifier must be None or an instance of a  "
-                "BaseClassifier."
+            score_sign = 1.0 if self.score_sign is None else self.score_sign
+        elif isinstance(self.estimator, BaseRegressor):
+            self.estimator_ = self.estimator.clone()
+            scoring_function = (
+                mean_squared_error
+                if self.scoring_function is None
+                else self.scoring_function
             )
+            score_sign = -1.0 if self.score_sign is None else self.score_sign
         else:
-            self.classifier_ = self.classifier.clone()
+            raise ValueError(
+                "parameter estimator must be an instance of BaseClassifier, "
+                "BaseRegressor."
+            )
+
         n_channels = X.shape[1]
         scores = np.zeros(n_channels)
         # Evaluate each channel with the classifier
         for i in range(n_channels):
-            preds = self.classifier_.fit_predict(X[:, i, :], y)
-            scores[i] = accuracy_score(y, preds)
+            preds = self.estimator_.fit_predict(X[:, i, :], y)
+            scores[i] = score_sign * scoring_function(
+                y, preds
+            )  # Applying the proper scoring function
+
         # Select the top n_keep channels
         sorted_indices = np.argsort(-scores)
         n_keep = math.ceil(n_channels * self.proportion)
@@ -90,7 +129,7 @@ class ChannelScorer(BaseChannelSelector):
         return self
 
     @classmethod
-    def get_test_params(cls, parameter_set="default"):
+    def get_test_params(cls, parameter_set: str = "default") -> TypingDict[str, any]:
         """Return testing parameter settings for the estimator.
 
         Parameters
@@ -106,4 +145,4 @@ class ChannelScorer(BaseChannelSelector):
         """
         from aeon.classification import DummyClassifier
 
-        return {"classifier": DummyClassifier(), "proportion": 0.4}
+        return {"estimator": DummyClassifier(), "proportion": 0.4}


Test patch:
diff --git a/aeon/testing/estimator_checking/_yield_collection_transformation_checks.py b/aeon/testing/estimator_checking/_yield_collection_transformation_checks.py
index 20f859d5..ee7636d5 100644
--- a/aeon/testing/estimator_checking/_yield_collection_transformation_checks.py
+++ b/aeon/testing/estimator_checking/_yield_collection_transformation_checks.py
@@ -35,7 +35,7 @@ def check_channel_selectors(estimator_class):
     """
     X, _ = make_example_3d_numpy(n_cases=20, n_channels=6, n_timepoints=30)
     y = np.array([0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1])
-    cs = estimator_class()
+    cs = estimator_class.create_test_instance(return_first=True)
     assert not cs.get_tag("fit_is_empty")
     cs.fit(X, y)
     assert cs.channels_selected_ is not None
diff --git a/aeon/transformations/collection/channel_selection/tests/test_channel_scorer.py b/aeon/transformations/collection/channel_selection/tests/test_channel_scorer.py
index 5607be3a..a57ac558 100644
--- a/aeon/transformations/collection/channel_selection/tests/test_channel_scorer.py
+++ b/aeon/transformations/collection/channel_selection/tests/test_channel_scorer.py
@@ -3,32 +3,79 @@
 import numpy as np
 import pytest
 
+from aeon.regression import DummyRegressor
 from aeon.testing.data_generation import make_example_3d_numpy
 from aeon.testing.mock_estimators import MockClassifierFullTags
 from aeon.transformations.collection.channel_selection import ChannelScorer
 
 
-def test_channel_scorer():
-    """Test the channel scorer."""
+def test_channel_scorer_with_classifier():
+    """Test the channel scorer for classifier."""
     # Test selects the correct number of channels
-    cs = ChannelScorer(classifier=MockClassifierFullTags(), proportion=0.5)
+    cs = ChannelScorer(estimator=MockClassifierFullTags(), proportion=0.5)
     X, _ = make_example_3d_numpy(n_cases=20, n_channels=10)
     y = np.array([0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1])
     cs.fit(X, y)
     assert len(cs.channels_selected_) == 5
     X, _ = make_example_3d_numpy(n_cases=20, n_channels=9)
     assert len(cs.channels_selected_) == 5
+
+    # Test for proportion out of range
     with pytest.raises(ValueError, match="proportion must be in the range 0-1"):
-        cs = ChannelScorer(classifier=MockClassifierFullTags(), proportion=1.1)
+        cs = ChannelScorer(estimator=MockClassifierFullTags(), proportion=1.1)
         cs.fit(X, y)
+
     with pytest.raises(ValueError, match="proportion must be in the range 0-1"):
-        ChannelScorer(classifier=MockClassifierFullTags(), proportion=-1)
+        ChannelScorer(estimator=MockClassifierFullTags(), proportion=-1)
         cs.fit(X, y)
-    cs = ChannelScorer(classifier=MockClassifierFullTags(), proportion=1.0)
+
+    # Test for selecting all channels when proportion=1.0
+    cs = ChannelScorer(estimator=MockClassifierFullTags(), proportion=1.0)
     cs.fit(X, y)
     assert len(cs.channels_selected_) == 9
+
+    # Test for invalid estimator
+    with pytest.raises(
+        ValueError,
+        match="parameter estimator must be an instance of BaseClassifier, "
+        "BaseRegressor",
+    ):
+        cs = ChannelScorer(estimator="FOOBAR", proportion=0.5)
+        cs.fit(X, y)
+
+
+def test_channel_scorer_with_regressor():
+    """Test the channel scorer for regressor."""
+    # Test selects the correct number of channels
+    cs = ChannelScorer(estimator=DummyRegressor(), proportion=0.5)
+    X, _ = make_example_3d_numpy(n_cases=20, n_channels=10)
+    y = np.random.rand(20)
+    cs.fit(X, y)
+    assert len(cs.channels_selected_) == 5
+
+    X, _ = make_example_3d_numpy(n_cases=20, n_channels=9)
+    cs.fit(X, y)
+    assert len(cs.channels_selected_) == 5
+
+    # Test for proportion out of range
+    with pytest.raises(ValueError, match="proportion must be in the range 0-1"):
+        cs = ChannelScorer(estimator=DummyRegressor(), proportion=1.1)
+        cs.fit(X, y)
+
+    with pytest.raises(ValueError, match="proportion must be in the range 0-1"):
+        cs = ChannelScorer(estimator=DummyRegressor(), proportion=-0.1)
+        cs.fit(X, y)
+
+    # Test for selecting all channels when proportion=1.0
+    cs = ChannelScorer(estimator=DummyRegressor(), proportion=1.0)
+    cs.fit(X, y)
+    assert len(cs.channels_selected_) == 9
+
+    # Test for invalid estimator
     with pytest.raises(
-        ValueError, match="parameter classifier must be None or an instance of"
+        ValueError,
+        match="parameter estimator must be an instance of BaseClassifier, "
+        "BaseRegressor",
     ):
-        cs = ChannelScorer(classifier="FOOBAR", proportion=0.5)
+        cs = ChannelScorer(estimator="FOOBAR", proportion=0.5)
         cs.fit(X, y)