Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper builds upon the recently proposed S4 architecture, which was shown to be effective at modeling long-range dependencies. The authors propose simplifications to the S4 model by modifying the kernel such that, instead of the originally proposed diagonal-plus-low-rank kernel, they can remove the low rank correction. The theoretical contribution includes proving a proposition which says that under some mild assumptions, this approximation is equivalent to the original kernel formulation. Empirically, the authors show that it is competitive with the original S4 model on the Long Range Arena (LRA) benchmark.
strengths_and_weaknesses: Strengths:
1. The S4 model itself is quite new and interesting, and as such, further studies to analyze and simplify it are useful for the community at large.
2. The 2 methods (DSS-exp and DSS-softmax) are theoretically motivated and demonstrate good empirical performance, where they achieve results comparable to the original S4 on most tasks.
3. The paper is clear and well-written, and the background described in Section 2 is sufficient for new readers (not familiar with S4) to understand the paper. 

Weaknesses:
1. My primary concern is that “simplicity” is subjective. The authors motivate DSS by saying that it simplifies computation of the SSM kernel. It would be a stronger result if they can show that it also provides quantitative gains, perhaps in terms of training time/flops.
2. Proposition 1 is based on the assumption that the state matrix $A$ is diagonalizable, and the authors say that this is “a mild technical assumption.” Can they justify this assumption, for instance, by showing that it holds for some toy example where explicit computation of the state matrices is feasible?
3. In Table 1, the large gap between S4 and DSS on the “TEXT” task is attributed to learning rate tuning. Could the authors also try to tune the baseline learning rate for S4, since these numbers can significantly change the “AVG” result?
4. In Section 4.1, the authors try to analyze the effect of restricting DSS to local interactions. As is clear from Table 3, this would be very task-dependent for any model (not just S4), and it is not clear what is the insight gained from this experiment.
5. In Section 4.2, the discussion in the last 3 paragraphs refer to figures in the Appendix. I think the main paper should be self-contained. Perhaps the authors could move the figures to the main paper by reducing some of the “Related work” content in Section 5.

**Update (Aug 8):** After discussions and new time benchmarking results from the authors, I am increasing my rating below. 

questions: 1. In the last paragraph of Section 2, the authors write that “our idea” is to use an alternate parameterization of state spaces. Isn’t this the original idea in S4?
2. In Section 4.2 (paragraph 4), the authors note that during training, the parameters $\text{Re}(\lambda)$ and $\text{Im}(\lambda)$ move away from the skew-Hippo initialization. Does this mean that this is perhaps not the best initialization for these parameters?

limitations: The authors have discussed the limitations of their work which is sufficient.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper extends prior work on efficient state space models by simplifying the state transition matrix to be diagonal. The authors make the point that diagonal state space models (DSSMs) are almost as expressive as normal state spaces when considering complex diagonal matrices. This is due to the fact that almost all matrices can be Eigen-decomposed (diagonalized) in the complex plane. The paper introduces 2 parameterizations of DSSMs that are based on this insight and show strong empirical performance on the long-range arena as well as raw audio classification; tasks which require processing very long sequences. 

Overall, I think the contributions of this paper a very important for the evolution of the DSSM framework wrt Deep Learning. I have trouble with the presentation of the work though which should be addressed by the authors. It is harder to follow than it should be and the community would benefit from improvements on that end. I am happy to change my score after that accordingly.
strengths_and_weaknesses: **Strengths**

* Mostly well written
* very practical insight and simplification
* strong empirical results
* insightful analysis of parameters after training

**Weaknesses**

I only found one weakness but it has to be addressed (IMO):
*Clarity*: I find the main contribution in proposition 1 very hard to follow. This is due to the fact that it is posited without any derivation or intuition, leaving the "elementary" proof for the appendix. I think this is bad practice and puts the burden on the reader to gather all necessary background to understand how the proposition comes together. Given that this is so central to the paper I do not understand the choice of the authors to put the whole derivation of it in the appendix.

Stating the following at least on a high-level before the proposition would make it much easier to understand proposition 1:
1) Most matrices diagonalize over the complex plane (via eigen-decomposition) —> Assume A is one of those —> almost no loss of expressibility.
2) Matrix exponentials are trivial to compute for diagonalised matrices because (VDV^-1)^n = V D^n V^-1. Analogously this is true for power series e^A.
3) V and V^-1 can be subsumed into B and C, show the resulting equation, show that B, C, V, V^-1 can be squashed into a single vector (w). 

This would go a long way to make the main paper much more understandable and helping the reader understand where Proposition 1 comes from.

questions: * Background discretization: There is no citation on where this comes from.
* Background: Is the paragraph about computing y and u necessary? It breaks the reading flow and adds mental overhead. Maybe push to the appendix.
* Proposition 1: A is not in C, is it?
* Proposition 1: I fail to see how the resulting K is ensured to be real?
* Initialization of delta_log: Why  is delta_log initialised as e^r with r in U(log(.001), log(.1))? An explanation in the paper would help.
* Initialization of A: Why the eigen-values of only the normal part of the Hippo matrix. What's the reasoning behind this decision?
* Figure 5: I don’t understand what are the delta_log values and how they refer to the real and imag parts of the lambda. Please explain and also extend the caption.
* Figure 5: It is stated that the values of delta_log are larger on short-range tasks? How is this “inline with” larger values modeling long-range dependencies?
* It seems like these models need to be tweaked quite a bit and for a practitioner it would be super helpful to see how brittle these models still are. That is, I would love to see a table/graph of results with non-optimal hparams.


limitations: Limitations have been addressed.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper builds on structured state space (S4) models and shows that one can achieve similar or slightly better performance with just diagonal state matrices (avoiding the low-rank structure). This is interesting because then the model becomes conceptually simpler and easy to implement. In addition, the paper provides ablations covering initialization and kernel truncation, and an analysis of the learned parameters.
strengths_and_weaknesses: I like the paper and recommend acceptance. I find it is moderately original, of very good quality, and significant. Clarity can be improved before the final submission.

Strengths:
* The idea is simple and attractive. Simplifications of existing processes is always welcome, especially if they match the accuracy of the previous, more complicated approach.
* I think complexity is reduced. This is perhaps not emphasized enough in the abstract and introduction.
* Good set of evaluations.
* Insightful ablations and analysis.

Weaknesses:
* Presentation. At several points I found that the authors could have made a larger effort in polishing notation or structuring the paper (see some comments below).
* Speed/complexity is only partially assessed. I'd suggest that the authors compare directly with complexities of the original S4 in the small subsection of section 3.2. Alternatively (or in addition), some speed differences can be measured on the same machine.

questions: Questions/suggestions:
- I think the abstract could be largely improved. Some things to consider are: mentioning complexities, mentioning ablation experiments, mentioning analysis, etc.
- Line 17: "denoising objective" --> This is a bit misleading, even more now that denoising diffusion models are in hype. Please find a better terminology.
- Line 39: "DLPR" --> DPLR.
- End of Sec 2: I think it would be better to present here the two views (convolutional and recurrent) instead of just one and then later mentioning the recurrent one.
- Sec. 3: In general, I found it hard to follow with the notation used (and it is not wrong, it is just that it is not presented appropriately). $N$, $H$, and $L$ could be defined beforehand or in Sec 2 (and also give an idea of their relation: like $H<N<L$ if that was the case). I also think that there are some missing "squeeze" and "unsqueeze" operations in the algorithm in the Appendix.
- Algorithm 1: Should there not be an outer product between $\Delta$ and $\Lambda$? Which type of division represents $/$ in line 4? Mention $H=1$?
- Line 137: Isn't $W_{\text{out}}$ the same as $C$ in Eq. 2? (I'm mostly asking for a clarification in the paper)
- Line 152: Any intuition on the initialization of $r$? (same)
- Lines 153-156: So in the end the HiPPO initialization is crucial (according to presented results). Therefore, the model could be considered to still rely on HiPPO theory? More explanation on that would be welcome (in the paper).
- I find there are many differences between the training setups of S4 and DSS (and this can make the reader suspicious). Perhaps it would be instructive to report a dry-run of DSS without those differences (that is, using the S4 setup unaltered) in addition to the reported results.
- Line 192: "images, audio" --> images, and audio.
- Sec 4.1: I think that the section would be much easier to parse if the two ablations were presented one after the other. So first Random Initialization: Question, how, result, and then Truncated Kernels: Question, how, result.
- Line 256: "absolute values" --> Better write "magnitudes"?
- Line 280: Remove "Related work".

Stylistic issues:
- Do not put references in the abstract.
- I am not sure that the way to cite references is correct (not sure about NeurIPS policy here; typically one writes "Surname et al." or just numbers).
- Authors seem to be using "cf." to mean "see". Note that "cf." does not mean "see": https://en.wikipedia.org/wiki/Cf. (that is, cf. should be equivalent to "compare", but the way the authors write it I believe it should be "see").
- A similar thing happens with "provably", where I think they mean "probably". It is not the same and the former implies that it can be proven mathematically: https://wikidiff.com/provably/probably

limitations: I am happy with the section on limitations.
review_title: null

Meta-review: metareview: This paper proposes a simpler alternative to S4 that achieves comparable performance. The method makes sense and the experiments are thorough. All reviewers agreed this is a good paper. I recommend acceptance.
confidence: Certain
Decision: Accept