Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposed a multi-head state space model (MSSM) and the Stateformer which is a combination of bidirectional MSSM and self-attention. The performance of MSSM and Stateformer are evaluated on Librispeech for ASR. Further, the effectiveness of MSSM is explored on the MLM task.
strength_and_weaknesses: **Strengths**

The paper is generally well-written.

This work is a quick application of the recently proposed Linear State-Space Layer (LSSL). The ideas of multi-head and a combination of bidirectional MSSM and self-attention seem to be novel.

As compared to existing SSM methods, this paper introduces a simpler parameter initialization manner and the multi-head structure. In the librispeech experiment, the Stateformer obtains the SOTA performance.

**Weaknesses**

"Stateformer 25L" performs close to "Gulati et al. 2020" (better only over testclean). So the real benefit taken by Stateformer could be questioned. Can the authors argue against this concern?

Training with the auxiliary classifiers is not common in ASR. As said in the paper, "Our Baselines" and "Our proposed models" in this large configuration are trained using auxiliary classifiers. It brings a further concern that the superior performance of "Stateformer 25L" may be partly due to the use of auxiliary classifiers. It is better to have an ablation study.

How the perplexities are calculated for masked language models?

Compared to the improved results in ASR, the improvement on masked language modeling seems weaker. MSSM needs 20K additional updates to perform better than Sliding window. The authors should compare different models for the same number of training iterations.
clarity,_quality,_novelty_and_reproducibility: see above.
summary_of_the_review: See above.
The major concern is about the experimental evaluations.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes some architectural changes to the S4 model and applies it to speech recognition and language modeling tasks.

strength_and_weaknesses: Amendment after rebuttal:

The original review pointed out that many of the contributions in this work are equivalent to flags available in the public S4 model. The authors have since submitted supplementary code showing that while based on the public model, several of these newer modifications are implemented somewhat differently.

This points to one of the main weaknesses of the submission, that the proposed changes are each not particularly novel. The other main weakness is a lack of discussion and comparison to very closely related prior work.

The strengths of the paper lie in the empirical results, which show strong performance in a number of speech recognition benchmarks as well as language modeling.

A much more detailed review and discussion is contained in the comments, with suggestions for improvement in the experiments and presentation.

clarity,_quality,_novelty_and_reproducibility: Clarity/Novelty: Despite being primarily based on the S4 model, the connection to this and other very closely related models is not clear in the presentation. Each individual contribution to the model is reasonable but not particularly novel, and could also be better described technically and positioned with respect to prior work.

Quality: Aside from the presentation, this paper has fairly strong application results, focusing on speech, as well as some interesting analysis (e.g. visualizations of the learned kernels).

Reproducibility: Code has been provided post-rebuttal, and I believe that the results will be reproducible when fully released.


summary_of_the_review: This paper proposes several modifications to the S4 model and architecture to achieve state-of-the-art results in speech recognition. Weaknesses are a lack of clarity in the details of the model, and a major lack of discussion and comparison to related work.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Tasks: Automatic speech recognition (ASR, measure WER), language modeling (measure PPL)

Two new models are proposed, which are based on state space models (SSMs).

The newly proposed models are: MSSM and Stateformer. The MSSM purely uses the SSM, with residual connections and some layer norm. The Stateformer is like a Transformer, i.e. it uses multi-head attention, and it has a new block type which embeds the MSSM.

On ASR, on Librispeech, without using an external language model, new state-of-the-art results are achieved.

For language modeling, the new model performs just the same as the sliding window attention model.

strength_and_weaknesses: 
Strengths:

- Good state-of-the-art results with the Stateformer on speech recognition.

Weaknesses:

- The code is not released. No information is given on the code.
- No real improvement for language modeling.
- Speech recognition lacks experiments also with an external language model. It's good to also see the numbers without, but using an external language model is the more realistic setting for speech recognition.
- Subsampling seems to be non-standard, and suboptimal. See below.
- How does it compare when replacing the SSM by a RNN or LSTM? It is said that SSMs are equivalent to linear RNNs. So what exactly is the advantage of a SSM? This does not become clear. Such experiments should be done to really see the difference.

clarity,_quality,_novelty_and_reproducibility: The state space model is equivalent to a linear RNN. So what happens if the same experiments are done, but with a RNN or LSTM? Is a SSM or MSSM really adding anything?

Transducer for ASR. Without external language model.

Code?

The subsampling methods for ASR all seem non-standard. A standard subsampling can for example be seen here: https://github.com/espnet/espnet/blob/4138010fb66ad27a43e8bee48a4932829a0847ae/espnet/nets/pytorch_backend/transformer/subsampling.py#L162
It even cites the Conformer for their Conv-based subsampling, but the Conformer uses a different type of subsampling. Yes, it also uses convolution for this subsampling frontend, but it's not just a single layer, and not 1D Conv but 2D Conv.

Table 2, where is the Conformer?

Stateformer and MSSM, how does it perform with Conv subsampling? Where do I see this? Or is this missing? Why is it missing? From table 2, from the Transformer experiments, it looks like Conv subsampling would be much better?

Computational cost? For training and recognition.

What happens with more number of heads for the MSSM? Table 2 suggest it might become even better? Why was this not tested?

How exactly does the final MSSM model look like, e.g. from table 2, 3 and 4? It's not totally clear. I assume, for ASR it is in all cases bidirecitonal? But probably not the LM? This is not really explained and defined well. Where is the SSM defined exactly? In Sec 2.1 I see some equations, but now what exactly is SSM(x)? It's not really clear to me.

In the Stateformer, how does this MSSM block look like exactly? Is this exactly the same as before (figure 2)?

Table 4 on LM, with the same number of updates, there does not seem to be a difference. With more updates, MSSM gets better, but what about the other models? This is missing from the table, and it cannot really be compared directly as is.

summary_of_the_review: It's an interesting extension of the SSM, applied to speech recognition and language modeling, and it achieves state-of-the-art (SOTA) results on Librispeech without external LM.

There are some weaknesses though. Some of them can be improved now by better clarification, but not all of them, where more experiments should be done.

Still, due to the good results for speech recognition, I think it's interesting enough.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper combines continuous time state-space models with transformer themes.  The method is applied to large language and speech modelling tasks, and is compared to some baselines.  The method appears to perform well.  Some analysis of the temporal characteristics of the learned model are explored.  As far as I can tell, no code or supplementary materials were submitted. 
strength_and_weaknesses: # Strengths:
The paper appears to perform favourably on the tasks presented.  

# Weaknesses:
The authors highlight three core contributions of the paper: multi-head parallel attention, head gating, and combination with attention through a bi-directional SSM.  I do not agree with any of these claims.  

(1.1)  I am not convinced that stacking the layers is a better architecture, as the “ablations” you provide are a bit confusing (unclear how performances change with overall depth).  I also don’t understand how you allow multi-head parallel processing where S4 does not? 

(1.2)  I do not believe you have correctly assessed how S4 implements attention, and so I do not believe you have “contributed” this.  (see comment below)

(1.3)  S4 uses a bidirectional SSM, which is not mentioned in your submission.  I also don’t fully understand why a transformer would _require_ bidirectionality in the SSM layer, as the transformer itself is all-to-all.  Ultimately, I find the overall architecture a little confusing.  I’d like Figure 3 to be explained a little more fully, maybe with some accompanying math close to or within the figure, to concretely define the spaces that each element operates on, its computational complexity etc.  


## (2)  Connections with S4:  
I find the presentation and connections to previous work, specifically in the arc of S4 papers, to be chronically lacking.  

(2.1)  _I implore the authors to comment on this prior to me making my final judgement, as I may have misunderstood their claims._  Most significantly, I believe the description of S4 and the use of GLU activation is incorrect.  In Equ. (8), the authors claim that S4 uses GLU activations applied to each two-dimensional output channel per input channel.  To my understanding, this is incorrect, as S4 applies GLU to the entire $y_k^{1:H}$ vector.  The interhead gating proposed in (9) is actually a special case of the GLU activation used by S4, where the number of output dimensions is strictly tied to twice the number of input channels with size-two block-diagonal weights.  As this is the core contribution of the paper, this is a critical point. 

(2.2)  The bidirectional design (Section 3.4) is also proposed by S4.  This is not mentioned.  To the reader, it appears that this bidirectional design is novel.  The parameterization used in Equ (7) is also a special case of the parameterization in Goel+ [2021], but this is not highlighted. 

(2.3)  The block-diagonal interpretation was also explored by Smith+ [2022] (although I accept that this is probably acceptable as concurrent work).  

(2.4)  Was S4 evaluated using the original codebase released by Gu+ [2021], or did you use your own S4 implementation? 

(2.5)  Broader discussion of LSSL, S4, HTTYH, S4D, DSS, Sashimi etc is chronically lacking.

## (3) Core claim on contributions:  
(This also relates to the exposition of S4 above) I do not understand the author's claim that S4 cannot capture different timescales.  The whole premise of S4 is that each SISO SSM is imbued with its own timescale, and hence can integrate information over different timescales.  Recent work, particularly How To Train Your HiPPO [Gu+, June 2022, arXiv], has explored this further, studying the kernel that S4 is implicitly using.  This paper, although recent, should still have been cited, since it tackles very similar core ideas as here.  It is also not clear to me how multi-head attention actually facilitates the claimed improvement in the ability to capture timescales.  Maybe this is true, but with the very limited intuition provided in Section 4.3, I don’t agree that better performance on benchmark tasks can be directly attributed to the new mechanism.  I am not convinced that the authors didn’t simply tune a larger model, with more parameters, flexibility, expressive non-linearities etc, until they got better performance – which then they post-hoc attribute to the capturing of timescales.  

(3.1)  You state: “... long and short term dependency would both be useful in sequence modelling…”.  I do not believe you have backed the claim that yours offers this whereas S4 does not  –  this was pretty much the whole point of S4.  

(3.2)  As an example, I would look at the eigen spectra of each learned latent SSM, for both SSM and the Stateformer, since this describes the temporal dynamics of the filter.

## (4)  Empirical validation:  
I am not overly familiar with the particular benchmarks that the authors present.  I would like to know why they didn’t use the LRA benchmark [Tay+, 2021] or the Speech Commands dataset [Warden, 2018], as these seem like much more standard and widespread benchmarks.  These also don’t require numerous A100’s, and so also help reproducibility.  

(4.1)  How were hyperparameters selected for the different methods? 

(4.2)  Table 1 is very unclear to me.  Does the 16L-2S model actually have 32 layers?  Or are 16 layers arranged into eight blocks of two.  

(4.3)  It is unclear what each variant in the table is exploring.  It appears to me that a soup of different layers and architectures have been tested and the best one is being reported as a substantial scientific contribution.  

(4.4)  The evaluation in Section 4.3 is not explained very well.  I do not understand how the curves in Figure 4 were generated, and so I cannot evaluate the validity of the claims.  I am not even entirely sure what the curves are showing. 



# Minor Weakness / Typographical Errors:
(a) Figures and tables should be floated to the top or bottom of pages.

(b) Footnotes should be avoided if possible.  

(c) States are missing a time index, e.g. in Equ. (8) it should be $y_{k, n1}$, where $k$ is the time index in the sequence.  

(d) I think that the claim in Section 3.2 that the shared recurrence parameters stabilise training is unsupported.  I also don’t think the second half of that claim (that multi-head frameworks give it the flexibility to learn multiple timescales) is fully supported either.  This may be me misunderstanding the analysis presented in Section 4.3.  

clarity,_quality,_novelty_and_reproducibility: I do not think the paper is especially well written.  The differences between the different models is often difficult to tease apart and relate to the content introduced earlier in the paper.  The actual analysis methods for the experiments in Section 4.3 are totally missing.  

I think the work is somewhat original, but the strong and immediate links to existing work are not explored, and make it very unclear to the reader what is a novel contribution.  

summary_of_the_review: Overall I do not think this paper is currently at the requisite level for publication.  Obvious links to existing work are missing.  I am not confident the core contribution itself is actually novel.  Standard evaluation benchmarks are missing.  The clarity of the written communication is poor.  

I am willing to entertain the idea that my understanding of S4, or my understanding of particularly Equ. (8, 9) are incorrect.  If this is the case (and it is indeed how the authors claim it to be) then this would improve the paper slightly in my eyes, but probably not enough for me to rate the paper as acceptance worthy.  

I also note that this paper is one of as many as half a dozen papers submitted to ICLR advancing S4.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: # Summary
This paper builds on the S4 model, proposing a number of changes that enable the model to perform very well on the 960-hour Librispeech speech recognition task when it is used as the acoustic encoder in a neural network transducer model. These changes include stacking of multiple state space models (SSMs) within a residual block, multi-head processing, and inter-head gating. Furthermore, the paper proposes combining this version of an SSM with attention to create a model called the Stateformer, which is similar to a Conformer except that the convolutional block is replaced by an SSM stack (within a residual layer) in the basic Conformer block. The paper also shows that, thanks to their low inference-time cost, SSMs may be included in the downsampling layers that are a common element in the acoustic encoder of many end-to-end speech recognition models. The paper also provides experiments showing that the proposed SSM performs well on a long-context language modeling task.

# Strengths
- The speech recognition results are state-of-the-art, and the fact that they can be obtained with a suitably designed SSM is an important contribution. Reviewers F93h and fosY and the area chair, all of whom are experts in automatic speech recognition, agree on this point.

# Weaknesses
- The paper does not position itself correctly with respect to the S4 model. While the revised paper is much improved over the original, it is still not explicit enough about the commonalities that the proposed model has with S4, most notably the use of the core S4 linear layer as a black box. This could be clarified by the addition of language that explains which parts of S4 are used as is and which are modified and by provision of ablation experiments that illustrate the value of the modifications (e.g., the change from the HiPPO initialization to random initialization). A related problem is that the paper still mischaracterizes the temporal modeling abilities of the S4 model. The introduction in the revised paper says "...in Gu et al. (2022a), the S4 approach equips SSMs with careful options on parameter initialization for long-range modeling ability. **This design would force the model to capture similar temporal dynamics on different features.**" even though in the discussion the authors stated "...we never claimed that S4 cannot model different time-scales" (https://openreview.net/forum?id=hrRNkyyGGgx&noteId=RsDM5EEGnR). _It is significant that the two reviewers who are most deeply familiar with S4, reviewers qqzq and PQ8K, independently raised concerns about the positioning of this work with respect to S4._ I'm certain that the authors will revise this work and submit it to a different venue, and I urge them to be extremely careful to position this work correctly with respect to the prior work, since not doing so poses serious reputational risks.

justification_for_why_not_higher_score: - My rationale for recommending rejection of this paper, despite its excellent empirical results on the Librispeech task, is the need for this work to be more precisely positioned with respect to the prior work on SSMs, most notably the S4 model. This rationale is summarized above in "Weaknesses."

- Much of the at times vitriolic discussion between the authors and reviewers qqzq and PQ8K revolved around the question of how novel different modeling ideas in this paper were, or whether, as the two reviewers argued, they were already implemented in the publicly available S4 code base. The issue was muddied by the fact that the S4 code base is evolving, so in several cases the design ideas in question are likely to be concurrent innovations; by the fact that some of the ideas in question were not mentioned in the original S4 paper, even if they may have been used in some of the experiments; and by the fact that the Sashimi paper did not accurately describe its implementation of bidirectional modeling. The authors of this paper correctly argued that simply having an idea isn't enough, and that it is necessary to explore it in the context of a specific problem to gauge its value. But, in the end, I agree with a comment that Reviewer qqzq made: "...it feels kind of silly to quibble over provenance claims of some 5-line modifications to a 500-line model, and it would all have been circumvented if the model was just described appropriately and points (1) and (2) were executed properly." [Points (1) and (2) being clear description of prior work in isolation and clear description and evaluation through ablation experiments of the differences from the prior work.]
  
- There was also a debate between the authors and reviewers over whether or not evaluations on the long-range arena (LRA) tasks is necessary. I personally do not believe that they are strictly necessary: Librispeech is a challenging, large-scale speech task that is a strong test of any sequence model. However, the paper would be strengthened by inclusion of LRA results, especially if the authors intend to publish the work in a general machine learning venue rather than in a more speech-specific venue.

- Concerns from Reviewer F93h about wanting Librispeech results with an external LM were addressed in the discussion/rebuttal period. Reviewer aXdZ raised some concerns about the evaluation of the MSSM as a language model that should be addressed by providing more specifics on the perplexity computation in the supplementary material. There are two issues: first, the comparison of models trained for a different number of updates is not really fair or reliable, and second, masked language modeling is not necessarily compatible with a perplexity computation, so more details need to be provided on that. See https://huggingface.co/docs/transformers/perplexity for a discussion of the issue.
Decision: Reject