Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper provides an intensive mathematical characterization of the S4 model introduced at the last ICLR conference. The paper contains a detailed explanation of the HiPPO initialization schema used in the S4 model and as the main result provides a proof that the seemingly ad-hoc approach of the original S4 is a time-invariant orthogonal SSM. Additionally, the paper provides interpretation of the timescale parameter of S4 and a guidelines on how to set it for a given task.

strength_and_weaknesses: ## Strengths
- The paper is a well written exposition of the mathematical details behind the S4 model, although see my comments on the poor structure below.
- Some additional results clarifying previously unjustified parameters of the S4 model.
- Some experimental evidence that the corrections and modifications of S4 guided by the presented theoretical results may lead to improvements in performance.
## Weaknesses
- The paper is poorly structured and at times feels like a part of a chapter on results related to S4 simply pasted into the ICLR format:
  - Discussion and conclusions are missing
  - Figure 1 and Figure 5 are virtually the same and their purpose is not too clear
  - Figure caption of Figure 1 reads like an abstract rather than figure description.
  - Most figures and tables are difficult to interpret - lacking axis labels, subfigures are not addressed in a common way (e.g. Figure 1a ) but with a descriptive textual statements of their locations.
  - As written, the paper is not ready for publication and needs substantial editing and modifications before it can be ready.
- It is unclear whether the results provide enough increment over the S4 paper to warrant a separate publication
  - Understandably, the manuscript consist a lot of background material on S4 and other prior and relevant work. S4 is a relatively recent model and without this background the paper would have been impossible to follow. However, this leads to my doubt of the format these result should be published in. Possibly a note to the original paper would be better.
- Comparisons in Table 1 are lacking 1D CNN and LSTM models
  - I understand that the claim is that these models do not work for long sequences, but a demonstration would be helpful, especially as the S4 model has inspired some boosts in CNN's performance: https://arxiv.org/abs/2206.03398 https://arxiv.org/abs/2210.09298
clarity,_quality,_novelty_and_reproducibility: The paper is poorly structured and is not ready for publication in the current form, however the pieces separately are well written (minus the figures). Potentially this is a quality work, but needs work. The work may be of incremental value, but it is difficult to judge.

summary_of_the_review: The paper contains an important explanations of why the S4 model works and how to properly tune it, together with a modification that boosts performance of S4 a little. However, it is not written as a complete self-contained paper and needs improvements to be publishable.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper makes three key contributions: (1) theoretical understanding of the role of the state matrix in modeling long-term dependencies. (2) derivation of new HiPPO matrices and showing that it captures many of the existing variants. (3) explaining the role of time-scale $\Delta$ in capturing the long-term dependencies.



strength_and_weaknesses: The theoretical contributions of the paper seem interesting. The generalized HiPPO leads to better understanding of the role of initialization in the S4 models. It seems that the experiments can be improved with more baselines.
clarity,_quality,_novelty_and_reproducibility: The paper reads easily. I cannot judge the quality and novelty properly.
summary_of_the_review: The paper makes three key contributions: (1) theoretical understanding of the role of the state matrix in modeling long-term dependencies. (2) derivation of new HiPPO matrices and showing that it captures many of the existing variants. (3) explaining the role of time-scale $\Delta$ in capturing the long-term dependencies.

This paper is out of my area of expertise and I cannot judiciously assess the quality of the paper. Unfortunately, I read the paper late and I could not withdraw from reviewing it.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors consider theoretical properties of the Structured State Space model (S4) (i.e., a deep learning state space modeling framework) and propose a characterization in terms of exponentially-warped Legendre polynomials demonstrating its theoretical properties and generalizations including to existing S4 procedures thereby in particular enhancing the characterization of long range dependencies. The developed approach is found to be superior on six benchmark from Long Range Arena and naturally extends the work of Gu et al., 2020; 2021; 2022a to address limitations within the existing S4 frameworks.
strength_and_weaknesses: Strengths:
The paper is in general well written and provides strong theoretical insights into initialization procedures and generalizations of S4.
The approach provides enhanced inference highlighting the utility using exponentially-warped Legendre polynomials.
Insights in regards to the setting of the time-scale parameter \Delta.

Weaknesses:
The paper is an extension of prior work on S4 of Gu et al., 2020; 2021; 2022a and although useful the novelty of the proposed approach appears somewhat limited. 

The paper is very dense reverting much information to appendices whereas the paper lacks a discussion and conclusion section and ends somewhat abruptly with the experimental results. I understand the space limitations requires strong compromises but I would encourage the authors to discuss implications at the end to make the paper self-contained within the 9 pages.

clarity,_quality,_novelty_and_reproducibility: The paper is well written but very dense and much information referred to the supplementary. The novelty lies in the application of legendre polynomials and theoretical properties derived including interpretation and initialization of the time-scale parameter \Delta. These derivations are non-trivial and can warrant publication. 
summary_of_the_review: An extension on the work of S4 including improved initializations and extensions to account for long range dependencies using legendre polynomials. The paper extends existing recent work of Gu et al., 2020; 2021; 2022a addressing limitations of the S4 framework. These results improve upon the existing S4 but the novelty of the framework is somewhat limited, however, the derivations non-trivial and the innovations useful which can warrant publication.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors study the problem of learning continuous-time dynamical systems using NNs. They focus on the Structured State Space Sequence model (S4) and on a particular matrix for initializing the SSM called a HiPPO matrix. They used Legender polynomials to explain the ability of the HiPPO scheme to capture long dependencies. Their analysis helps them derive new guidelines and variants of the method that lead to performance improvement on Long Range Arena and Path X.
strength_and_weaknesses: Strengths: The paper is mostly well-written, and the English level is satisfactory. The paper provides a contribution to an important problem by analyzing a leading method in this field. The analysis also leads to practical improvements in several challenging tasks that require modeling long dependencies with NNs.

Weakness: The paper is very busy with information, without much space for intuitive explanations of some of the results. I would move some plots to the appendix, and add a conclusion and expand on some of the explanations to add intuition to the reader
clarity,_quality,_novelty_and_reproducibility: The paper is mostly clear, and the quality of the research is high. The authors provide some novelty with the new analysis and the new variations proposed to the HiPPO framework. The experiments are well explained, making the results reproducible. 

Minor:
The word sequence is missing in the first sentence of the introduction
 In 2.1, you mention a projection to N-dimensional space, so it may be worthwhile to specify the dimension of the matrices in Equations 2 and 3 for clarity.

P4 an extra bracket appears after the inner product <p,q>

P4 “HiPPO called this” - this sentence seems broken.

Why are there no bold methods in the first free columns in table 1? Isn’t bold representing the leading method?

It would be better to add a conclusion and move one of the figures to the appendix.

summary_of_the_review: Overall the paper is well-written and provides a nice contribution to the community. The HiPPO framework seems very promising, and the authors provide new analysis and new variations of the methods with Legender polynomials or Fourier Basis functions. This enables them to improve SOTA in several tasks.
For these reasons, I recommend accepting the paper.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper investigates the problem of learning continuous-time dynamical systems using neural networks, and considers theoretical properties of the Structured State Space model (S4). It provides a characterization in terms of exponentially-warped Legendre polynomials demonstrating its theoretical properties and generalizations. The paper is in general well written and provides strong theoretical insights into initialization procedures and generalizations of S4. The updated version improves much over the initial submission, and it successfully addressed many concerns raised by the reviewers. I found that there are still quite some small English errors.
justification_for_why_not_higher_score:  The updated version improves much over the initial submission, and it successfully addressed many concerns raised by the reviewers. I found that there are still quite some small English errors.
justification_for_why_not_lower_score: The paper is in general well written and provides strong theoretical insights into initialization procedures and generalizations of S4.
Decision: Accept: poster