Repository: sktime/sktime

Issue:
random_state not working for MiniRocketMultivariate
Hi, 

It seems like the random_state argument is not working for MiniRocketMultivariate().

Under _fit_multi(), num_channels_per_combination is randomly generated but the seed is not set to ensure reproducibility.


Discussion:
@dyangny Can you give examples of code that illustrate the problem?
OK, I just looked at the code (at `transformations/panel/rocket/_minirocket_multivariate.py`), and it might be as simple as adding the following to the top of the definition of `_fit_multi`:

```python
if seed is not None:
        np.random.seed(seed)
```
> OK, I just looked at the code (at `transformations/panel/rocket/_minirocket_multivariate.py`), and it might be as simple as adding the following to the top of the definition of `_fit_multi`:
> 
> ```python
> if seed is not None:
>         np.random.seed(seed)
> ```

Yes. I think that should work.
Note there are other issues with the code, e.g. it is very hard to read and there are no tests for it (as far as I can see. `test_Rocket.py` only tests main rocket)
For seeding, we usually use: 
```
rng = check_random_state(random_state)
random_int = rng.randint()
```
where `check_random_state` comes from scikit-learn and `rng` is a random state instance from `numpy`. 

@angus924 any thoughts on adding this?
> num_channels_per_combination is randomly generated but the seed is not set

This is my oversight, sorry.  The seed should be set (or, should be able to be set) here.

> it is very hard to read

Again, my bad.  If I get time at some point, I'll try to clean it up.  If there's a particular section (e.g., channel assignment) which is a problem, I'm happy to try to explain it.  (There's a bigger picture here, which is that `MiniRocketMultivariate` is really only intended to provide basic multivariate functionality for MiniRocket.  However, I don't have a time frame for when we might be able to add better multivariate support.)

[edit: I should add, there's another issue which I haven't had the chance to address yet, in that MiniRocketMultivariate is approximately 10x slower than 'it should be', that's another thing on the to do list.] 

> there are no tests for it

I'm pretty sure there are tests in `.../transformations/panel/tests/test_MiniRocketMultivariate.py`, although maybe I'm misunderstanding what's meant here.

> ```python
> if seed is not None:
>         np.random.seed(seed)
> ```

This is probably the best fix for the moment (see next comment).

> ```python
> rng = check_random_state(random_state)
> random_int = rng.randint()
> ```

This would work here, as the channel assignment is done outside Numba.  However, in the background we have the same old problem that Numba has it's own way of handling random, and you can't pass a random state into/out of Numba code (which is why MiniRocket/MiniRocketMultivariate are limited to using integer seeds).  As such, if we went with this approach for channel assignment, we'd have one way of handling the seed/random state for one part of the code (sampling bias values) and another way of handling the seed/random state for another part of the code (channel assignment).  Accordingly, I'd be inclined to go with the `if seed is not None: ...` approach above, for simplicity/consistency.  Having said that, I don't really have a strong view on it, I'd be happy either way.
> If there's a particular section (e.g., channel assignment) which is a problem, I'm happy to try to explain it.

I only spent ~10 minutes reading the code (mainly to understand the seeding issue) and I had a skim of the accompanying paper, and it might just be a case of the algorithm itself is not easy to understand if you first do not know rocket (note I do not know anything about rocket).

An example of something hard to understand without prior knowledge is what the different variables mean, e.g. what do `A`, `G` or `C_alpha` mean.  Another example of something hard to understand is why certain numbers have been chosen, e.g. `for gamma_index in range(9 // 2)`. 9 looks like it is a hyperparameter here.

> I'm pretty sure there are tests in `.../transformations/panel/tests/test_MiniRocketMultivariate.py`, although maybe I'm misunderstanding what's meant here.

My mistake. I made bad assumption of assuming tests would be in the test_Rocket.py file.
@Lovkush-A, it should be obvious that 9 has been chosen as a random number, also see
https://dilbert.com/strip/2001-10-25

Jokes aside, I agree that some documentation would be useful.
> An example of something hard to understand without prior knowledge is what the different variables mean, e.g. what do `A`, `G` or `C_alpha` mean. Another example of something hard to understand is why certain numbers have been chosen, e.g. `for gamma_index in range(9 // 2)`. 9 looks like it is a hyperparameter here.

That's a good point, these will mean nothing in isolation.  I named them like that to match the corresponding quantities in the paper, for better or worse, but I can see that it means nothing outside that context.

> it should be obvious that 9 has been chosen as a random number

Unfortunately, it's not that exciting.  It is because kernel length is (always) 9, so I tried to keep that reference in there rather than writing `range(4)` (we are filling the rows of a matrix/array with k=9 rows in two parts, for convoluted reasons, first 4 rows, then last 4 rows).

Anyway, you can see that I have well thought-out, bad reasons for everything.
@angus924 An idea came up in recent discussion that researchers who want to code up their state-of-the-art algorithms but do not have time to document/refactor for accessibility can be paired up with somebody from sktime community who does have time to do so.

I figured I might as well get this initiative started by offering to help refactor this minirocket algorithm (and other algorithms, if appropriate)! What do you think, Angus?
Sorry for the slow reply.

Thanks very much @Lovkush-A.

The one thing I would say about refactoring is that as one of the main attributes (or claims, at least) in relation to MiniRocket is in relation to speed/scalability (i.e., that it is fast), it would be important to make sure that the refactored code is at least as fast as the original code.  (The complicating factor here is that the code doing all of the real work is Numba-compiled code, which may make this tricky.)  People, understandably, are likely to treat the sktime implementation as official/matching the published version.

Having said that:
- even if only the non-Numba/plain python code was revised, this might be worthwhile; and
- as I mentioned somewhere above, the multivariate version of MiniRocket is actually quite a bit slower than 'it should be' (like-for-like, compared to the univariate version), so there may be room to make a big improvement, but it's not really important.
Quick addition.

For scientific reproducibility, I think it's always fine to have multiple versions of algorithms:

* a "reference" version that corresponds to usage and results in earlier publications, e.g., that reproduces runtimes, bugs, etc
* a "latest" version that is the algorithm with tweaks and bugfixes. This would be by default the one `sktime` loads.

I would go so far to not just have the "historical" version in earlier commits, but designated with shorthand aliases and loadable if requested, e.g., `"MiniRocket2018AngusEtAl"` so it has a unique and referenceable name as a code artefact.

(if we get too many of these we can consider moving them to a "reproducibility" themed repo, but "too much" is currently not a problem)
@angus924 Alright! Sounds like you like the idea. We should arrange to have a quick chat, maybe 15-30 mins. Things worth discussing:
* quick intros
* how the algorithms in sktime match up with your papers, e.g. which algorithms are for which papers, where experiments were carried out, etc.
* quick intuition of the algorithms, to speed up understanding of them.  Or any other advice to help speed things up, e.g. recommendation of which papers I should read and in which order.

What is best way to contact you to arrange time for a meeting?
@fkiraly 

> I would go so far to not just have the "historical" version in earlier commits, but designated with shorthand aliases and loadable if requested, e.g., `"MiniRocket2018AngusEtAl"`

How would this designation be done, explicitly? Also, this would only give access to the algorithm, but not the precise experiments that were run to get the results in the paper.

Separate to that suggestion, do you know what The Turing Way recommends in terms of making experiments/algorithms reproducible?
>How would this designation be done, explicitly? Also, this would only give access to the algorithm, but not the precise experiments that were run to get the results in the paper.

Well, the experiment, given a clean benchmarking module, would look sth like this

`results = benchmarking(config, algorithms = {"nameA": algoA, "nameB": algoB, "nameC:" algoC etc`

where `algoA = MiniRocket2018AngusEtAl(someparams)`

>Separate to that suggestion, do you know what The Turing Way recommends in terms of making experiments/algorithms reproducible?

Turing way considers these questions out of (the de-facto) scope to my knowledge, at least as long as it corresponds to methodological content - see also https://github.com/alan-turing-institute/the-turing-way/issues/315 and additions since which didn't touch these matters. 
E.g., as far as I'm aware, Turing Way considers out of scope content items such as what is a "good" experiment (e.g., use baselines, compare to state-of-art, quantify and report effect size and evidence quantifier etc), or what are good coding patterns for a good benchmarking experiment (e.g., estimator objects, estimator references, unified interfaces, or such).

If you look carefully at @kirstiejane's message, it's not actually out of scope in-principle - but out of the de-facto scope in the sense that there is no critical mass working on these topics (as far as I'm aware). @kirstiejane can perhaps say more about this in case she reads this.
> Well, the experiment, given a clean benchmarking module, would look sth like this

And how would this (ideal) experiment be made accessible to future people? Suppose in three years time somebody wants to re-run the experiment, how would that happen?

The only thing I can think of is to create a fresh github repo, with a python script/notebook with the code to run the experiment and a requirements file stating exactly which versions of all packages were used to run the experiment.

(To clarify, my question is more about the logistics of making things reproducible, rather than DS methodology.)
Hi @Lovkush-A, thanks, I've sent you an email re setting up a meeting.

@fkiraly, I think if the refactored code is at least roughly similar to the original in terms of doing whatever it's meant to be doing (or better), it's not a big deal.

It's tricky to try to use other people's code at the best of times, and it's difficult to kind of "enforce" (not quite the right word...) the use of a particular version.

What I have seen with Rocket, for example, is that some papers, even if using the original implementation or something equivalent to it, use it in a different way than was intended/in a way that does not actually match the way it is used in our paper (e.g., always using the ridge classifier, even for large datasets).

The other thing to bear in mind is that the practical relevance of minirocket, if any, is likely to be very short lived, so I doubt there's too much value in strictly archiving different versions.
>What I have seen with Rocket, for example, is that some papers, even if using the original implementation or something equivalent to it, use it in a different way than was intended/in a way that does not actually match the way it is used in our paper (e.g., always using the ridge classifier, even for large datasets).

Precisely the problem I'm referring to - if there would be `MiniRocket2018AngusEtAl` or similar you could point to, it would at least partially resolve this in that any reviewer can point to "why did you not use the reference implementation in your comparison but a weird strawman of the same name"?

>The other thing to bear in mind is that the practical relevance of minirocket, if any, is likely to be very short lived, so I doubt there's too much value in strictly archiving different versions.

I get that, but it would make earlier studies less reproducible, since you can't fully exclude that changes in the implementation also change its "scientific behaviour".

In any case, we don't need to start with that since it may be out of scope for mini-rocket - but @kirstiejane and @evelinag at the Turing could start building a library of algorithms like this for their reproducibility/standardization endeavours (you're welcome to the idea, @kirstiejane and @evelinag - if you already had it, now you have two😄)

Resolving patch:
diff --git a/sktime/transformations/panel/rocket/_minirocket_multivariate.py b/sktime/transformations/panel/rocket/_minirocket_multivariate.py
--- a/sktime/transformations/panel/rocket/_minirocket_multivariate.py
+++ b/sktime/transformations/panel/rocket/_minirocket_multivariate.py
@@ -493,6 +493,8 @@ def _quantiles(n):
 
 
 def _fit_multi(X, num_features=10_000, max_dilations_per_kernel=32, seed=None):
+    if seed is not None:
+        np.random.seed(seed)
 
     _, n_columns, n_timepoints = X.shape
 


Test patch: