Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper addresses the problem of long-range sequence modeling. The authors propose the s5 layer, which is built atop and modified version of structured state-space-sequence, s4. Utilizing parallel scans, s5 improves over s4 in terms of performance and complexity. They show improved results on LRA benchmark without incurring additional complexity comparing with S4. 


strength_and_weaknesses: The proposed layer, s5, replaces the frequency-domain approach used by S4 with a recurrent, time-domain approach bypassing non-trivial steps needed for S4. The method achieves high performance with comparable complexity of s4. 

The source code link provided in the paper isn't working, The authors should publish anonymized code for the review purposes with the correct link. 
clarity,_quality,_novelty_and_reproducibility: the paper is well-motivated and well-written. 
summary_of_the_review: This is a well-written, well-motivated paper.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper introduces the new state space layer S5 which combines multiple single-input single-output state space models to a single multiple-input multiple-output state space model. The proposed state space layer can be efficiently implemented by a parallel scan. S5 achieves state-of-the-art results on multiple benchmarks, including the difficult Path-X task.
strength_and_weaknesses:  - [+] The paper is very well written with a concise and clearly structured background section.
 - [+] All experiments are repeated multiple times with different seeds, leading to more stable and thus scientifically valuable results.
 - [+] Detailed and comprehensive comparison with S4 and its components with helpful illustrations and descriptions.
 - [+] The reviewer highly acknowledges the inclusion of concurrent work in the comparison.
 - [-] Although all experiments are repeated multiple times, no error bars are reported. Especially in cases where multiple methods perform similarly, a corresponding deviation measure proves to be really 
       helpful for judging.
 - [-] It is not clear what the reported scores in Table 1 are. Assuming the scores to be accuracy scores, the average accuracy provides only limited insight under the assumption that the tasks are of
       varying difficulty (i.e. different number of classes with different relative class sizes, etc.). The reviewer suggests reporting the average rank for each method instead.
 - [-] Regarding the results reported in Table 3: are the results marked with an asterisk (*) computed on 5 seeds, and all others (i.e. „CRU (our run)“ and „S5“) on 20 seeds? If yes, this might skew the
       result. The reviewer suggests using the same test protocol for all methods in this case. If not, the reviewer suggests clarifying the use of different seeds.


clarity,_quality,_novelty_and_reproducibility: The paper is clearly written and the quality of the presented content meets the conference standards.
The suggested implementation is novel in a way that it improves the S4 layer by a more efficient computation method. It would have been nice, though, if the authors would have provided the corresponding implementations already via OpenReview.
summary_of_the_review: The paper is very well written and easy to follow. All necessary background information is concisely introduced. The authors provide extensive analyses w.r.t. ablations, runtime and memory footprints. Some ambiguities and questions still remain, whereas the reviewer hopes they will be clarified during the rebuttal period.
*The reviewer will likely increase the score if the weaknesses and questions are clarified in sufficient detail.*

### Questions

 - What might be the main reason or intuition why S5 has better results than S4 w.r.t. scores, besides computational efficiency/runtime?
 - Why wasn’t S4 used as a baseline in Table 3?
 - What is the hyperparameter search space of the baseline methods?

---
### Update

The response of the authors addressed our concerns. Therefore, the score has been raised (as promised).
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper presents an extension of structured state space sequence (S4) layers [Gu et al. 2021], which the authors refer to as S5 layers.
While S4 layers use a bank of (deterministic) linear continuous state-space models with subsequent mixing layers, S5 uses a single multi-input multi-output SSM that does not require mixing layers. Furthermore, S4 parallelises the (offline) computation for the sequence of linear operations using a 1D convolution identity and an implementation in frequency domain (FFT). This has been done for computational reasons (parallelisation). However, this paper shows that S5 layers can match the computational and memory complexity (also shown empirically), by using the scan operation. 
The authors show connections between S4 and S5, which they use for building on the initialization of S4. The authors also show via extensive ablation studies (in the supplementary material), that the initialisation as well as the continuous-time formulation with learned discretisation timescale parameters are the main drivers for these models.

strength_and_weaknesses: This paper is very well written, precise, and easy to follow. I found the background on S4 layers incl. the diagram very useful and more accessible than the original paper, and I also think the authors made a good choice in explaining their extension S5 by contrasting them to S4. It is also great to see the easy-to-implement jax code in the appendix.

The experimental evaluation is extensive, with many and the most important baselines, evaluated on commonly used long-range-dependency benchmarks. It is especially nice to see ablation studies in Appendix E, evaluating different model sizes (making it more or less similar to S4) as well as initialisation and continuous-time (vs. direct discrete-time implementation), showing that these are the most important features. 

I did not find major problems or concerns. 
But I would be interested if the relation between S4 and S5 (App. D) could be made with less restrictions. Especially assumption 2 seems a bit strict? 
I would also have liked to see how the model deals with uncertainty/unpredictability, which is often the case in forecasting (e.g. the stock market would be an extreme case). The sequential image classification tasks are a bit unrealistic/artificial. But I do realise this is not the goal of the paper, and previous work from this domain also does not evaluate in such settings. 
clarity,_quality,_novelty_and_reproducibility: The paper is very clear and high quality. It is easy to be reproduced with code given in the appendix, and linked (although I was not able to access it). One of the advantages of the approach is also that it is much easier to implement and use than the S4 layers as it does not require both the sequential and convolution & FFT mode implementation (only sequential).
While the model itself "just" extends previous work, there are several technical contributions that are novel in this context.

summary_of_the_review: Overall I enjoyed reading the paper, it is high-quality work, in terms of method, experimentation, and the presentation.
I did not find any major problems or concerns.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: For handling long sequential inputs as required in many modern applications of neural nets, this paper proposes an "S5" layer, "Simplified State Space Layers for Sequential Modeling", which is more expressive than the previous "S4" layer from ICLR 2021, while retaining computational efficiency. The two key changes: (1) single-input single-output banks + mixing layers for dealing with multi-dimensional inputs are changed to multi-input multi-output processing. (2) time-domain processing rather than frequency domain. 

Strengths:
- The technical improvements over S4 are important, lifting restrictions of the earlier, already quite successful model. 
- The results on various long-range sequence modeling tasks show comparable performance with the very best transformer attention based models, without incorporating any of that machinery.
- Implementation appears straightforward, suggesting ease of use by others in the community.
- Evaluated against the very latest work possible, including even work released during the week of the ICLR deadline.
- Well-presented. 

Weaknesses:
- Evaluations are on somewhat synthetic tasks, although this is in keeping with the current practice in long-range sequence modeling.
- It would be good to demonstrate some qualitative examples, such as visualizing how information from various stages of historical input are used in producing current outputs.
justification_for_why_not_lower_score: A promising advance in long-range sequence modeling with deep neural nets coming from a very different perspective than self-attention and transformers, but which brings it on par with those much more widely researched methods. In my opinion, this would be of widespread interest at ICLR and would make a good choice for oral presentation.
Decision: Accept: notable-top-5%