Venue: ICLR 2022 Conference

Reviews:

final_score: rating: null
review_content: For W2, it’ is expected to show the proposed advantages clearly, both in theoretical analysis and in experiment evaluation.

For W3, as the authors claimed, the randomized signature is better than ROCKET because it is theoretically guaranteed. However, what the associated theory can benefit the TSC problem is not clarified (Maybe it contributes to the interpretability but we cannot see in the paper).  Straightforward evidence is expected.

For W1 and W4, I believe the contribution between theory and showcase is a tradeoff when evaluating this paper. As other reviewers have also pointed out, the theoretical contribution of this paper is minimal. The authors have also said in your reply that your contribution focuses on the implementation of the original ideas and the efforts are paid on engineering issues. I believe these efforts are not academic contributions. So I prefer the contributions of this paper in the interesting showcases. That is why I propose W4. And I expect the authors to showcase more feature-based tasks for time-series, such as time-series prediction and outlier detection.  
review_title: Feedback to the discussions

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The manuscript proposes that by combining the signature transform with ideas from reservoir computing one may reduce computational cost and spatial overhead. This is demonstrated on some toy problems as well as some real world datasets

main_review: The manuscript builds on previous work that has showed that the signature is a reservoir by demonstrating the performance of the randomized signature. This adds to the literature on reducing overhead in signature methods, which has grown a lot in the last few years but is still a very active area of research. As it stands however, the theoretical contribution of the manuscript is minimal as far as I can tell, and it is mainly a showcase of what can be done with the randomized signature.

While the breadth of experiments is impressive, there is a lack of comparison with other signature based baselines such as neural CDE or RDE methods or low-rank projections. Ideally there should also be some discussion about what the benefits are of using the randomized signature over the other methods and when it is more or less suitable.

Minor points:

 - Text is, in my opinion, overly verbose and would probably improve if cut down a bit.

 - Many published papers are cited as arXiv preprints.

 - Section 3.2 paragraph 1: "problem, in particular kernelization techniques, see, e.g., Kidger and Lyons (2020)" This paper does not propose kernelization techniques.

Overall, the paper is decent and might be very useful in certain situations, but it lacks discussion, and the empirical contributions are rather weak considering the lack of technical innovation.  

summary_of_the_review: This is a good showcase of the power of reservoir computing in signature methods, but there is room for improvement.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper showcases the effectiveness of a recently introduced reservoir of random features, i.e., randomized signatures.
main_review: S1：This paper gives significant insights into the randomized signatures by showcasing its performance on non-parametric system identification and time series classification problems.

S2: The background and theory related to randomized signatures are well described.

S3: The experimental results provide evidence of the effectiveness of the presented strategy.

W1: The original contribution of this paper is limited. The methods of reservoir computing and random signature are mature components. The paper just uses these strategies in a trivial manner.

W2: The claimed advantages of the proposed randomized signature layer over the existing solution, such as inferring mapping from data, are not evident. It is expected that the results are provided to show the success of the randomized signature layer on resolving the problems such as the pointed over-parametrization, data hungriness, expensive training cost.

W3: Although the signature is well theoretically guaranteed, it is not clarified how these guarantees can contribute to the problem of time series classification in comparison with the original Rocket where the randomized convolution kernels are used.

W4: A minor issue. As the main contribution of this paper is to showcase the effectiveness of the randomized signature layer, it’s expected that its performance on more representative tasks are evaluated and reported.   

summary_of_the_review: In summary, I believe this paper is under the acceptance threshold because of its limited contributions as well as some other issues, as I have pointed from W1 to W4.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a Signature-based method for learning feature representation of time series, denoted as $d$-dimensional path $X$. By definition, Signature is a sequence of $M$ tensors where each tensor is a sum of integral terms defined on dimensions of  $X$. Theoretically Signature is guaranteed to linearly explain the solution of any differential equation driven by $X$ which implies that it encodes well $X$. Computing Signature however has time complexity $O(d^M)$ rendering it impractical for learning purposes. To alleviate the issue, the paper introduces Randomized Signature, which can be computed in $O(k^2 d)$ only where $k$ is the number of Randomized Signatures. Each of such Signature has $O(kd)$ dimensions. As stated by Theorem 2 in the paper, Randomized Signature approximates well the original Signature. Experiments on differential equations show Randomized Signature finds solutions close to ground-truth solutions. Experiments on time series classification also show its potential.
main_review: Strengths:
S1: Following Theorem 1 theoretically justifies that Signature of $X$ is suited to solving differential equations driven by $X$.

S2: Following Theorem 2, Randomized Signature approximates Signature while being much cheaper to compute. The approximation power of Randomized Signature to the original Signature is empirically verified in Section 4.1 (in particular Figure 3).

S3: In Section 4.3, Randomized Signature can find solutions of differential equations that are close to ground-truth solutions.

S4: For time series classification, Randomized Signature when combined with Rocket (Dempster et al., 2020) gives better performance than existing related deep learning methods. It however does not perform as well as Rocket.

Weaknesses:
W1: The paper provides informal versions of Theorems 1 and 2 only. I cannot find the formal versions as well as their proofs in the appendix. I would recommend the authors to provide more details. This will help to make points such as "comparable approximation power as the Signature" in Theorem 2 more rigorous.

W2: How to choose activation $\sigma$? There seems also no explanation about this in Section 4.

W3: From time complexity analysis in Section 3.2, the runtime speed up seems significant. However, the experiment in Section 4.1 reveals using Randomized Signature yields less than an order of magnitude speed up (3 times in particular). This can be because $k$ is quite large compared to $M$. The authors should provide more detailed runtime comparisons between the two types of signature in the experiments in Section 4.3.

W4: Most experiments in Section 4.3 are on data with $d \leq 5$. How about data with higher number of dimensions such as more than 10? It would be interesting to see scalability to $d$ in both runtime and quality of Randomized Signature. Could the authors provide a summary of the 128 UCI data sets for time series classification? This will help to better understand impact of $d$ on the performance.

W5: Randomized Signature + Rocket performs worse than Rocket, due to limitations of the update scheme in Algorithm 1. The authors already provided some justifications in the last paragraph of Section 4.4.
summary_of_the_review: Using Signature theory for time series analysis is a promising direction with some theoretical guarantees. The paper however is not self-contained and some details are lacking. Please help to address the above comments.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors utilise random projections before computation of the signature resulting in "Randomised signatures" from which similar information can be extracted (compared with the standardly applied signature transform) with reduced computational burden and lower overall feature dimension. 

The authors demonstrate the utility of this randomised methodology on a variety of tasks including learning rough differential equations and standard time series classification benchmarks. 
main_review: Overall this paper is very well written and tackles an interesting problem - dealing with the blowup in the dimensionality of the signature.

A lot of recent papers in the ML literature that utilise the signature pay little notice of the severe feature blowup associated with increasing signature feature dimension. The result of this is a number of methods that do not scale well to tasks with large numbers of feature channels. This paper outlines an interesting method of dealing with this by way of random projections applied to the path 

## Strengths
- Well written with a good introductory section to signatures and their usefulness. 
- Randomised signature layers are an interesting way to reduce signature dimension - a current bottleneck for a lot of signature methodologies. 

## Weaknesses
- **Method is not entirely novel** - Lyons et al. "A feature set for streams and an application to high-frequency financial tick data" introduce this idea of randomised signatures and it is considered in Morrill et al. "A Generalised Signature Method for Multivariate Time Series Feature Extraction". Neither of these papers are mentioned/referenced by the authors. 
- **Performance on time series classification tasks seems poor** -  Whilst the "rocket_ours" classifier comes second of all classifiers considered, "rocket" was already the top performer. I'm not sure how I should be considering this other than that the randomised signature features make the rocket classifier worse -- why would I use randomised signature features then? 
- **Informal theorem is too informal** - The randomised signature theorem with the claim "has comparable approximation power as the Signature" needs more explanation. Whilst I am a fan of informal theorem statements in such papers in general, I need some indication of what such a statement actually means. This can be easily remedied by a high level explanation of what is going on, or a mathematical statement as to what this comparable approximation power actually means. It also needs to reference a full, formal theorem somewhere. 
summary_of_the_review: This paper would represent the best reference to the idea of "randomised signature" features if accepted. However, it is currently lacking in that it does not adequately reference prior work, and it has theorems that are slightly too vague. If these things can be rectified, I would consider raising my score.
review_title: null

Meta-review: This paper empirically evaluates the performance (in time and accuracy) of randomized signatures for time series, an idea that was developed theoretically in a series of recent paper. While reviewers acknowledge that implementing and testing this idea is relevant, they also consider that the lack of methodological and theoretical novelty, combined with the fact that the experimental results do not convincingly show that randomized signatures outperform existing methods on a variety of tasks, puts the paper below the acceptance bar.
Decision: Reject