Repository: sktime/sktime
Pull request: [ENH] `iisignature` backend option for `SignatureTransformer`
<!--
Thanks for contributing a pull request! Please ensure you have taken a look
at our contribution guide: https://github.com/sktime/sktime/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
Fixes #1591. 

This PR also adds an option to choose no augmentation, currently the code returns an error if augmentations is None -- the user will be forced to choose an augmentation.

#### What does this implement/fix? Explain your changes.
<!--
A clear and concise description of what you have implemented.
-->
Fixes a bug where a runtime error is returned if each point of the time-series has dimension more than 25. 
An enhanced flexibility to allow to user to choose no stream augmentation.

#### Does your contribution introduce a new dependency? If yes, which one?
Soft dependency of the package iisignature is needed for the fix of #1591. 

<!--
If your contribution does add a new hard dependency, we may suggest to initially develop your contribution in a separate companion package in https://github.com/sktime/ to keep external dependencies of the core sktime package to a minimum.
-->

#### What should a reviewer concentrate their feedback on?

<!-- This section is particularly useful if you have a pull request that is still in development. You can guide the reviews to focus on the parts that are ready for their comments. We suggest using bullets (indicated by * or -) and filled checkboxes [x] here -->

#### Did you add any tests for the change?
No

<!-- This section is useful if you have added a test in addition to the existing ones. This will ensure that further changes to these files won't introduce the same kind of bug. It is considered good practice to add tests with newly added code to enforce the fact that the code actually works. This will reduce the chance of introducing logical bugs.
-->

#### Any other comments?
<!--
Please be aware that we are a loose team of volunteers so patience is necessary; assistance handling other issues is very welcome. We value all user contributions, no matter how minor they are. If we are slow to review, either the pull request needs some benchmarking, tinkering, convincing, etc. or more likely the reviewers are simply busy. In either case, we ask for your understanding during the review process.
-->

#### PR checklist
<!--
Please go through the checklist below. Please feel free to remove points if they are not applicable.
-->

##### For all contributions
- [x ] I've added myself to the [list of contributors](https://github.com/sktime/sktime/blob/main/CONTRIBUTORS.md) with any new badges I've earned :-)
  How to: add yourself to the [all-contributors file](https://github.com/sktime/sktime/blob/main/.all-contributorsrc) in the `sktime` root directory (not the `CONTRIBUTORS.md`). Common badges: `code` - fixing a bug, or adding code logic. `doc` - writing or improving documentation or docstrings. `bug` - reporting or diagnosing a bug (get this plus `code` if you also fixed the bug in the PR).`maintenance` - CI, test framework, release.
  See here for [full badge reference](https://allcontributors.org/docs/en/emoji-key)
- [ ] Optionally, I've added myself and possibly others to the [CODEOWNERS](https://github.com/sktime/sktime/blob/main/CODEOWNERS) file - do this if you want to become the owner or maintainer of an estimator you added.
  See here for further details on the [algorithm maintainer role](https://www.sktime.net/en/latest/get_involved/governance.html#algorithm-maintainers).
- [ ] The PR title starts with either [ENH], [MNT], [DOC], or [BUG]. [BUG] - bugfix, [MNT] - CI, test framework, [ENH] - adding or improving code, [DOC] - writing or improving documentation or docstrings.

##### For new estimators
- [ ] I've added the estimator to the API reference - in `docs/source/api_reference/taskname.rst`, follow the pattern.
- [ ] I've added one or more illustrative usage examples to the docstring, in a pydocstyle compliant `Examples` section.
- [ ] If the estimator relies on a soft dependency, I've set the `python_dependencies` tag and ensured
  dependency isolation, see the [estimator dependencies guide](https://www.sktime.net/en/latest/developer_guide/dependencies.html#adding-a-soft-dependency).


<!--
Thanks for contributing!
-->



Problem statement:
[BUG] `RuntimeError` when `SignatureTransformer` is run on large datasets
SignatureTransformer throws a RuntimeError when calling fit on the DuckDuckGeese dataset. I assume this is an issue with the number of dimensions?

```
File "E:\_ProgramFiles\Anaconda3\envs\sktime-workshop-boss\lib\site-packages\esig\backends.py", line 118, in compute_signature
    return tosig.stream2sig(stream, depth)
RuntimeError: Legitimate width 2 <-> 256 exceeded
```

Dataset | Train | Test | Class | Length | Dimension 
-- | -- | -- | -- | -- | -- 
DuckDuckGeese | 50 | 50 | 5 | 270 | 1345

Hints from issue discussion:
For some more context, we are running signatures on the whole UEA multivariate archive. I would like to include the method in our future evaluations.

Pinging @jambo6 as they may know more.
This is actually an error thrown by `esig` to help prevent memory errors down the line. 

Signature feature dimension increases exponentially with the input dimension of the data. The input dimension here is `1345`, a signature of `depth=2` would then have on the order of `1345 ** 2` features (nearly 2 million). You can see how this quickly becomes a problem. 

Without additional modifications (e.g. using random projections) signatures do not scale well to large input dimensions. My advice would be to skip any examples where dimension exceeds 256 and chalk signatures down as DNF. 

That's fine, I think these limitations should be noted in the docs, however. Could maybe be handled a bit better than a RuntimeError in the esig package as well, but that's less important. 

I stopped after the first dataset error, but quite a few more also seem to have problems. Some examples:


Dataset | Train | Test | Class | Length | Dimension | Error
-- | -- | -- | -- | -- | -- | --
FingerMovements | 316 | 100 | 2 | 50 | 28 | `RuntimeError: Legitimate depth of 2<->3 for records with width 29 exceeds limit`
Heartbeat | 204 | 205 | 2 | 405 | 61 | `RuntimeError: Legitimate depth of 2<->2 for records with width 62 exceeds limit`
FaceDetection | 5890 | 3524 | 2 | 62 | 144 | `RuntimeError: Legitimate width 2 <-> 256 exceeded`

I assume this is still dimension related, the limit seems to be ~20 ish.




I encountered the same problem in my work. What I did was to replace the backend of the esig by iisignature in _compute(see https://github.com/datasig-ac-uk/esig), which can handle paths with more dimensions. Note that this requires iisignature, in addition to esig, to be installed on your machine. Happy to write a patch for this issue if that's useful. 
> Happy to write a patch for this issue if that's useful.

Sure, that would be appreciated!

Do you have any insight on the conditions for the runtime error to surface?
> > Happy to write a patch for this issue if that's useful.
> 
> Sure, that would be appreciated!
> 
> Do you have any insight on the conditions for the runtime error to surface?

Okay, I can start working on that. 

Yes, from my experiment the error seems to surface whenever the dimension of the original time series is larger than 24/25, as @MatthewMiddlehurst has noted.

Commit message: responde to requested change, allow the user to specify additional iisignature backend parameters and check for soft-dependency

Patch to review:
diff --git a/.all-contributorsrc b/.all-contributorsrc
index 093a48a5fd0..878c4050b39 100644
--- a/.all-contributorsrc
+++ b/.all-contributorsrc
@@ -2365,6 +2365,15 @@
         "bug",
         "code"
       ]
+    },
+    {
+      "login": "sz85512678",
+      "name": "Zhen Shao",
+      "avatar_url": "https://avatars.githubusercontent.com/sz85512678",
+      "profile": "https://github.com/sz85512678",
+      "contributions": [
+        "code"
+      ]
     }
   ]
 }
diff --git a/pyproject.toml b/pyproject.toml
index d3015541474..3c6fd9f6c98 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -95,6 +95,7 @@ regression = [
 ]
 transformations = [
     "esig>=0.9.7,<0.10",
+    "iisignature>=0.24",
     "filterpy>=1.4.5,<1.5",
     "holidays>=0.29,<0.33",
     "mne>=1.5,<1.6",
@@ -112,6 +113,7 @@ all_extras = [
     "dask",
     "dtw-python",
     "esig==0.9.7; python_version < '3.10'",
+    "iisignature>=0.24",
     "filterpy>=1.4.5; python_version < '3.11'",
     "gluonts>=0.9.0",
     "h5py",
diff --git a/sktime/transformations/panel/signature_based/_augmentations.py b/sktime/transformations/panel/signature_based/_augmentations.py
index 0a3d2305ea4..2cf8e853059 100644
--- a/sktime/transformations/panel/signature_based/_augmentations.py
+++ b/sktime/transformations/panel/signature_based/_augmentations.py
@@ -43,11 +43,11 @@ def _make_augmentation_pipeline(augmentation_list):
     if augmentation_list is not None:
         if isinstance(augmentation_list, str):
             augmentation_list = (augmentation_list,)
-    assert all(
-        [x in list(AUGMENTATIONS.keys()) for x in augmentation_list]
-    ), "augmentation_list must only contain string elements from {}. Given: {}.".format(
-        list(AUGMENTATIONS.keys()), augmentation_list
-    )
+        assert all(
+            [x in list(AUGMENTATIONS.keys()) for x in augmentation_list]
+        ), "augmentation_list must only contain string elements from {}. Given: {}.".format(
+            list(AUGMENTATIONS.keys()), augmentation_list
+        )
 
     # Setup pipeline
     if augmentation_list is not None:
diff --git a/sktime/transformations/panel/signature_based/_compute.py b/sktime/transformations/panel/signature_based/_compute.py
index c75269b8e3d..76613bde1f0 100644
--- a/sktime/transformations/panel/signature_based/_compute.py
+++ b/sktime/transformations/panel/signature_based/_compute.py
@@ -45,6 +45,7 @@ def __init__(
         sig_tfm=None,
         sig_depth=None,
         rescaling=None,
+        backend=None,
     ):
         super().__init__()
         self.window_name = window_name
@@ -54,6 +55,7 @@ def __init__(
         self.sig_tfm = sig_tfm
         self.sig_depth = sig_depth
         self.rescaling = rescaling
+        self.backend = backend
 
         self.window = _window_getter(
             self.window_name, self.window_depth, self.window_length, self.window_step
@@ -61,6 +63,8 @@ def __init__(
 
     def _transform(self, X, y=None):
         import esig
+        if self.backend == "iisignature":
+            esig.set_backend("iisignature")
 
         depth = self.sig_depth
         data = np.swapaxes(X, 1, 2)
diff --git a/sktime/transformations/panel/signature_based/_signature_method.py b/sktime/transformations/panel/signature_based/_signature_method.py
index 0ef27595f2c..3642e845c80 100644
--- a/sktime/transformations/panel/signature_based/_signature_method.py
+++ b/sktime/transformations/panel/signature_based/_signature_method.py
@@ -10,6 +10,8 @@
     _WindowSignatureTransform,
 )
 
+from sktime.utils.validation._dependencies import _check_soft_dependencies
+
 
 class SignatureTransformer(BaseTransformer):
     """Transformation class from the signature method.
@@ -32,6 +34,7 @@ class SignatureTransformer(BaseTransformer):
     sig_tfm: str, String to specify the type of signature transform. One of:
         ['signature', 'logsignature']).
     depth: int, Signature truncation depth.
+    backend: str, The backend to use for signature computation. One of: 'esig', or 'iisignature'.
 
     Attributes
     ----------
@@ -62,7 +65,15 @@ def __init__(
         rescaling=None,
         sig_tfm="signature",
         depth=4,
+        backend="esig",
     ):
+        if backend == "esig":
+            _check_soft_dependencies("esig")
+        elif backend == "iisignature":
+            _check_soft_dependencies("iisignature")
+        else:
+            raise ValueError("backend must be one of 'esig' or 'iisignature'")
+
         self.augmentation_list = augmentation_list
         self.window_name = window_name
         self.window_depth = window_depth
@@ -71,6 +82,7 @@ def __init__(
         self.rescaling = rescaling
         self.sig_tfm = sig_tfm
         self.depth = depth
+        self.backend = backend
 
         super().__init__()
         self.setup_feature_pipeline()
@@ -86,6 +98,7 @@ def setup_feature_pipeline(self):
             sig_tfm=self.sig_tfm,
             sig_depth=self.depth,
             rescaling=self.rescaling,
+            backend=self.backend,
         )
 
         # The so-called 'signature method' as defined in the reference paper


Code review comments:
diff_hunk: @@ -95,6 +95,7 @@ regression = [
 ]
 transformations = [
     "esig>=0.9.7,<0.10",
+    "iisignature>=0.24",
path: pyproject.toml
text: @user2:
I think these should have an upper bound as well, that should be current rounded up to 0.1, if I'm not mistaken, @user1?

@user1:
Yes, it should. Even though it has not been released for over 4 years, we can safely use ">=0.24,<0.25" etc.

However, I've a separate concern. As per https://github.com/bottler/iisignature/issues/20, installing this library needs `numpy` pre-installed. I don't think that's necessarily to be the case for users. Even if I assume it is, as they can manually check the error and address it, how can we use it in our CI?

(Assuming that it's not being tested currently in CI, don't see a new test being added.)

@user2:
what does "pre-installed" mean, here? Do we not have `numpy` as a core dependency?

@user1:
(**DISCLAIMER**: The following is just my understanding, and I'm not at all confident.)

`pip install sktime[all_extras]` download all packages, builds all that's necessary and then installs all of them together. If any of the package installation fails, all gets cancelled. So, even if `numpy` is a core dependency and `iisignature` is soft, they will be installed simultaneously.

Based on the issue, it looks like `numpy` needs to be installed already even before installation of this package starts. Installing together is not an option, as during building step it (`setup.py`) needs `numpy` which will not be present if `pip install sktime[all_extras]` is run on a new blank environment.

@user2:
> Based on the issue, it looks like numpy needs to be installed already even before installation of this package starts. Installing together is not an option

Ouch, really? Then we can't make it a soft dependency in `all_extras`.

We can still add it to the estimator, of course, although that begs the question how we test.

@author:
hmmm this seems like a real issue, @user2 how do you think it's best to proceed? I think it's not acceptable that pip install sktime[all_extras] will return an error. 

@author:
P.S. have added the version upper bound. 

@user1:
@author can you please check my suspicion once, i.e. what happens if you create a new empty environment and tries to install sktime along with all_extras? If it works, we're all good.

If not, then at least specifying in extras seem to be out of scope, and we can test with a dedicated job only.

@author:
@user1 I think that is a good idea to test it. I am not sure how should I build sktime and upload it to pip though, as I understand your suspicion will surface when the user runs pip install. Would you be able to help? Is there a way to check this from the github repository directly without building it? Thanks! 

@user1:
Yes, there is. You can use pip to install from a remote or local repo with or without extra dependencies.

You can also build and install sktime wheel locally using `build` and `pip`, it's not needed at all to upload to PyPI. But it'll be much simpler if you install from git repository itself, with or without editable option.

Please refer to [development version part of this documentation](https://www.sktime.net/en/stable/installation.html#id6).

@author:
Thank you for the help! I can confirm that your suspicion is correct, I have the following error when I install it in a new virtual environment: 

Collecting iisignature>=0.24
  Using cached iisignature-0.24.tar.gz (1.3 MB)
  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error
  
  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [6 lines of output]
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/private/var/folders/zs/r35ht9w90bzfpdzytk4tfh9m0000gn/T/pip-install-oke9sm8v/iisignature_d008469a531c4a188be319198b5ceca5/setup.py", line 2, in <module>
          import numpy, os, sys
      ModuleNotFoundError: No module named 'numpy'
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

@user2 @user1 What might be the best way to moving forward?

@user2:
not sure - one option would be to leave it up to the user to ensure the requirements for the backend.

I.e., we can still offer this as an option in the estimator, and not add `iisignature` to the dependency sets but the drawback would be that we can't test the backend.

This might be better than the alternative to not offer it, though. Perhaps doing this with a warning for users that the backend support is experimental.

Not sure, what do you think, @user1?

@user2:
Also related:

https://github.com/sktime/sktime/pull/5474, and the python 3.10 version bound.

FYI @author, `SignatureTransformer` currently has a quite restrictive version bound of python <3.10 on it.

Are any of your changes relaxing that requirement? I think it was coming from `esig`. Have you tried to remove the tag and test it on higher python versions?

@author:
Unfortunately it does still have the python <3.10 version bound, as esig is still used as the front-end. 
A more long term solution could be to replace the signature computation by signax, which is much more recent: https://github.com/anh-tong/signax. It uses JAX, not sure how compatible it is with the sktime eco-system? 

@user2:
> It uses JAX, not sure how compatible it is with the sktime eco-system?

`sktime` core is an interface and library layer, so if the dependencies are properly isolated, anything python is compatible. Maybe a nice next project?

@author:
I see, indeed maybe a nice next project!

diff_hunk: @@ -62,7 +65,15 @@ def __init__(
         rescaling=None,
         sig_tfm="signature",
         depth=4,
+        backend="esig",
     ):
+        if backend == "esig":
+            _check_soft_dependencies("esig")
+        elif backend == "iisignature":
+            _check_soft_dependencies("iisignature")
+        else:
+            raise ValueError("backend must be one of 'esig' or 'iisignature'")
path: sktime/transformations/panel/signature_based/_signature_method.py
text: @user1:
not blocking - to improve this error message, you should refer to `SignatureTransformer`, as the error could be raised in a pipeline, and users would not know which component raised it.

@author:
Good point, thanks. Done.