Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors set out to understand why some diagonal variants of the S4 state-space model work well, then introduce S4D as a specific approach building off of DSS. Their main mathematical result shows that a purely diagonal update reproduces the dynamics of S4’s diagonal plus low rank update in the limit of large latent size.

Based on their theoretical insights, they improve on existing diagonal state-space models (DSS) by carefully choosing initialization, discretization, and related parameters in the S4D family of models. The authors also report detailed ablations to confirm the theoretical conclusions, and demonstrate S4-equivalent quality on a variety of benchmarks.

strengths_and_weaknesses: The paper is clear despite packing information very densely and including both theoretical and empirical results.

The ablations are strong and careful, and presented concisely and clearly (I haven’t seen this particular graphical presentation of ablations before, and I think it works well.)

Overall, I think it was unclear before this paper why diagonal SSMs worked but related methods like linear RNNs didn’t; this paper begins to answer that question.

The approach presented in this paper is to compute the kernel using a Vandermonde matrix, but the authors don't make the connection between this approach and the parallel scan implementation of the linear recurrence. The parallel scan approach has some advantages over explicitly computing the kernel, including support for unbounded sequence length (important for online tasks like reinforcement learning) and variable/dynamic sampling rate.

The abstract mentions “three lines of code”; these should be included in the main paper.

questions: Do you have a guess or narrative for what goes wrong when a parameterization is expressive but doesn’t optimize well? (“[W]e emphasize that Proposition 1 is about expressivity which does not guarantee strong performance of a trained model after optimization. For example, Gu et al. [7] and Gupta [8] show that randomly initialized dense real matrices or diagonal complex matrices, which are both fully expressive, perform much worse than S4.”)

The ability to fully diagonalize the SSM update creates a strongly privileged basis; have the authors explored interpretability of latent states in this basis?

limitations: If there are significant limitations of diagonal SSMs, including S4D, relative to the original S4, it would certainly be worth mentioning them as negative results. But so far it appears not to be the case.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The authors examine a state-space modeling approach that has gained popular in recent literature — the main idea of that line of research concerned with the expressive capability, and ability to capture long-term memory dependence through the use of linear SSMs defined in continuous time.  The authors note that practical use of HiPPo, the structured state-space architecture (S4) and diagonal state-space architectures (DSS) may present complications due to various numerical instabilities associated with bad initializations or parameterizations. To be more specific, they systematically analyze the diagonal approximation to S4 — they prescribe different initializations that perform better numerically for which they also back up theoretically.  Through an ablation study, they are also able to demonstrate the practical utility of their analysis and suggestions.

strengths_and_weaknesses: I think the paper addresses some general issues that would be good for someone looking to employ the previously introduced HiPPo matrix into their modeling schemes.  For example, the authors address numerical concerns that are often brushed over in many papers — even though not addressing them can be catastrophic i.e. parameterization of the dynamics such that the matrix is stable.  The memory saving tip in 3.3 for storing B and C is useful as well.  In section 3.4 I like the initialization prescribed, and while its a nice concept I think the presentation could just be slightly tidied up — a comparison to balancing would be useful too, see [E. E. Osborne, “On pre-conditioning of matrices”].  I thought some of the ablation studies were also very useful; since the point of the paper is to prescribe practical guidelines to follow its nice the authors swept through different variables such as the discretization method, normalization, and parameterizations.  

One of the first difficulties I encountered was with notation.  If inline with the papers cited regarding HiPPo, then it would have been useful to specify that for example, C is a row vector, B is a column vector, and y(t) and u(t) are one-dimensional observations and control inputs respectively.   The lack of conclusion is also somewhat concerning, I think a succinct summary of the experimental and theoretical results would have been a good way to end the paper.  Another big weakness I think lays in the total contributions of the authors; a lot of rehashing of previous results is done, but the authors in my opinion do not devote enough space in the main text to outline their current contributions.

questions: If my statement about the dimensionality of the variables specified was wrong, hopefully you could clear that up.  In, Eq. (5) A^D as defined, is always just the matrix of 1/2 on the diagonal, is that correct?

I don’t think the DSS acronym was specified in text (is it diagonal state-space ?).  Is bar{C} as defined in Eq. (6) just equal to the original C row vector?

In Eq. (7) should the ordering of B and C be swapped? Stemming from my confusion earlier, I thought that B was a column vector and C a row vector.  Maybe subscripts here would help i.e. y_l = (u * K)_l or something more economical that clears up confusion.  A reference for the equality using the Vandermonde matrix would probably help as well in my opinion.  

On page 5, section 4 below prop. 1 you call the randomly initialized matrices fully expressive — in what sense?

limitations: In my opinion, the contributions of the paper are somewhat incremental;  that’s not to say the authors contributions are not novel — but a great deal of writing space is used to describe the previous research on HiPPo.  I think more of the authors contributions could have been thrown into the spotlight, rather than to rehash the cited literature.  Furthermore, since a large part of the paper is about how to select good, in some sense, dynamics matrices — how come there is no discussion on parameter training? I would think for a practitioner, it may be easier to train the parameters of the matrices in question, rather than doing a large amount of model selection.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: Paper simplifies diagonal state space models by i) analysing Eigen-values of Hippo matrices and proposing appropriate initialisation schemes, ii) simplifying the parameterisation of the real parts (to be negative) of the Eigen-values (for stability) and iii) introducing a simple normalisation scheme that ensures variance preservation of the resulting SSM. Experiments are conducted to verify all these simplifications and the authors show that they do not lead to inferior performance. A very important take away message from the paper is that diagonalization alone doesn’t make a performant SSM but it needs proper parameterisation/initialization. Overall the paper is a great extension to the growing literature on SSMs for Deep Learning. 
strengths_and_weaknesses: ## Strengths
* good and mostly clear presentation
* solid mathematical foundation for simplifications
* convincing empirical results

## Weaknesses
* No dicusssion/limitations
* Mostly small scale, toyish tasks evaluated
* The results don't show any advantages of S4D over S4
questions: * “Second, Proposition 1 does not take into account numerical representations of data, which was  the original reason S4 required a low-rank correction term instead of a pure diagonalization.“   —> This is quite vague, pleas elaborate or strike.
* How do you go from conjecture 3 to S4D-Inv?
* Table 1: What S4D variant is used?
* Curious to see what would happen if only A were left untrained.
* S4D-Trap introduced but never ablated?
* More information on how robust S4D is to choice of optimization hparams is would be interesting.
* Simplicity is really nice but it is not clear to me from the paper why S4D should be used over S4 for any practical reasons? Do you see any speed ups? Results seem to still lag behind S4.
limitations: The authors haven't addressed the limitations.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: Detailed analyses of some variants of the so-called S4 model are presented. The focus is on the use of a diagonal state-transition matrix instead of a diagonal plus low-rank matrix, time discretization scheme, spectral radius constraint of $A$, parametrization of $B$ and $C$, normalization of convolution kernels corresponding to SSM, and initialization scheme. The authors provide empirical ablation studies in these regards. They also provide the rationale for why the diagonal approximation of a so-called HiPPO-LegS matrix works well.
strengths_and_weaknesses: Overall I did not find serious shortcomings. The analyses would be of interest to a limited community.

### Strengths

- The experiments are intensive and well organized.
- While I could not check all the correctness of the proofs, the discussion around Theorem 2 is interesting.

### Weaknesses

- The target audience is limited.
- The evaluation is done only in terms of the final performance of classification or regression tasks. Some qualitative and direct analyses of the models' outputs, if possible (and perhaps on much simpler input data), would be interesting.
questions: **1.**
In Figures 1 and 2, please specify what is drawn there not only in the caption but also by using axis labels and legends. The current way of presentation is not so kind.

**2.**
How the proposed models, which are sequence-to-sequence models, were applied to the sequence-to-label problems is not clearly stated. Maybe it is just the same as the previous works, but it would be nice to briefly mention the model architecture somewhere in the main text of the paper.

**3.**
The proof of Theorem 2. I found no obvious errors in the step-by-step reasoning. However, the intermediate result of $u(t) = \mathbf{B}^T x(t)$ is quite counterintuitive. I understand this happens only for $N \to \infty$ with the HiPPO-LegS $\mathbf{B}$ matrix, but I wonder if there is any more intuitive interpretation of why such a relation is the case here.
limitations: I don't have particular comments in this regard.
review_title: null

Meta-review: metareview: All reviewers agree that the paper proposes an interesting approach to examine diagonal state space models. Although some reviewers have some technical concerns at their first reviews, basically those have been resolved by the authors' responses. Thus, although there are some points that should be modified from the current form, I think we can expect the authors modify the paper in the camera-ready by reflecting the discussion. Based on these, I recommend acceptance for this paper.
confidence: Certain
Decision: Accept