Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposed the H3 architecture to narrow the gap between state space models and attention models on language modeling. Basically, H3 uses a shift SSM and a diagonal SSM to replace the non-linear function in linear attention. The shift SSM is designed to remember tokens in the input sequence while the multiplication between the two SSM outputs for token comparison. 
To further improve H3 efficiency on hardware the authors proposed FlashFFTConv, which uses fused block FFT algorithm. 

Experiments were conducted on training multiple variants of H3 on the PILE dataset, and compared with corresponding GPT2 checkpoints, on OpenWebText and WikiText103. Moreover, the authors also evaluate H3 on zero-shot and few-shot classification.
At last, they evaluate efficiency of the proposed FlashFFTConv algorithm on the LRA benchmark, achieving about 2 times speed up.
strength_and_weaknesses: Strengths: The design of H3 is novel and well-motivated and the paper writing is clear and easy to follow. The authors also clearly claimed their contributions.

Weaknesses: The main concerns are from experiments. I found that the experimental setup on language models is not standard, which makes the comparisons unfair. It is difficult to position H3 among other neural models in language modeling. I elaborated my concerns in the following questions.
clarity,_quality,_novelty_and_reproducibility: The experimental setup in this paper make it difficult to fairly compare H3 with previous models:

1. The authors trained H3 on the PILE data and calculated PPL on OpenWebText and WikiText103. But both the two datasets have their own training data, and several previous models have reported PPL on them. Why not following previous setup and directly comparing H3 with previous models, such as standard Transformer, Transformer-XL and other efficient Transformer architectures. It is strongly recommended to conducted language modeling experiments on standard benchmarks, such as WikiText103 and PG19, and closely follows previous settings for fair comparison.

2. For the comparison in Table4, since GPT2 models are not trained on the same PILE dataset, the PPLs are not directly comparable. 

3. A lot model details are missed, making it hard to check or reproduce experimental results.

Other questions about experiments:
1. Does FlashFFTConv support half precision training?

2. Have you evaluated H3 accuracy on LRA?
summary_of_the_review: To sum up, I am worried that the paper needs to re-organize its experiments and analysis before publication.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper presents a new state space model (SSM) that achieves on par with transformer based language models while inferring up to 2 times faster. The authors first highlight the drawback of previous SSMs compared to attention -- SSMs can't recall earlier tokens in the sequence and can't compare tokens across the sequence. The authors propose an improvement that captures these features by using `shift` and `diagonal` SSM layers. Shift layer shifts dimensions in an input (or steps if the input is a sequence) and diagonal SSM is convenient for computing kernel in the convolution. By borrowing the structure of linear attention, the authors propose a new SSM layer, called H3, that utilizes recurrence structure and can scale to longer sequences better. They show that H3 improves previous SSMs and achieves on par with transformers on synthetic as well as real OpenWebText tasks. They further propose a mechanism, called FastFFTConv, for efficient estimation of convolution where the convolution is replaced with `iFFT (FFT(u) * FFT(f))`. They divide input into chunks -- each chunk fits into the GPU memory -- and keep end-state of each chunk for a more efficient convolution which results in up to 3x speed up against attention.
strength_and_weaknesses: **Strengths** I found the paper interesting and useful for thinking beyond typical transformer architectures. The paper proposes a simple SSM layer that can replace any self-attention module and broadly applicable. Experimental results are also promising and efficiency improvements should help other work in state space models.

**Weaknesses**
1. You mention that SSMs are capable of extrapolating to sequences longer than observed during training. I found no empirical results to support this claim in the paper.

2. There are missing references in long-term dependency line of work, especially *Perceiver* style models. Given that one of the things that the paper claims is capturing long term dependencies, I think these need to be cited and compared against.

3. How would you compare to other models in Table 2 is H3 is trained from data rather than by construction? Please also clarify how other models, excluding attention, are obtained.

4. One thing that Transformers have shown is they scale with data and model sizes. While from 125M to 335M, there is a clear improvement in Table 4, I would like to see if there is a similar pattern for data size as in Table 6.

General-purpose, long-context autoregressive modeling with Perceiver AR. Curtis Hawthorne, Andrew Jaegle, Cătălina Cangea, Sebastian Borgeaud, Charlie Nash, Mateusz Malinowski, Sander Dieleman, Oriol Vinyals, Matthew Botvinick, Ian Simon, Hannah Sheahan, Neil Zeghidour, Jean-Baptiste Alayrac, João Carreira, Jesse Engel.
clarity,_quality,_novelty_and_reproducibility: The paper is written clearly and H3 is a simple and novel extension of previous SSMs. While the implementation is not straightforward to derive from only the paper, the code is also open sourced.
summary_of_the_review: H3 is a simple yet novel extension of previous SSMs. It presents improvements compared to Transformers and can infer 2 times faster. FastFFTConv is useful for efficient training and can be applied to other SSM works.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work aims to improve the state space models (SSMs), which achieve state-of-the-art performance on the LRA benchmark but perform worse on language modeling and large-scale language pre-training. The authors identified two problems related to the model design through two synthetic tasks. Based on the observation, the authors further proposed some modifications which lead to strong LM performance. At the same time, the authors also leverage Flash to improve the efficiency of the inference.
strength_and_weaknesses: Pros:

1. It is interesting to use synthetic tasks to investigate the drawback of the state-space model.
2. It is important to improve S4 on large-scale language modelling tasks. This work make an essential step towards this goal
3. The empirical results is adquante to support the advantage of the model

Cros:

Major questions:

1.The authors proposed two synthetic tasks, Induction Head and Associative Recall. However, the relationship between these two tasks and the language model performance remains unknown. Why does the poor performance of the language model have a strong connection with these two tasks? This connection should be discussed clearly since it motivates the modification, which (as the authors stated) leads to better language model performance. 

2.The change improves the language modelling performance, but will the new model hurt the LRA performance. If I didn't miss anything, there is only an efficiency study on LRA without performance comparison. This work will be problematic if the model fails to work on LRA but only work on LMs.

Minors:

1. The authors claim that standard attention has the ability to complete Induction Head and Associative Recall. Is this a rigorous statement or not? If yes, I recommend the authors give formal proof of the statement. I am not that sure whether attention can succeed #for any length#

2. The authors were motivated by linear attention and proposed an O(nlogn) architecture with FFT. The work "Stable, Fast and Accurate: Kernelized Attention with Relative Positional Encoding" achieves the same efficiency with FFT based on linear attention. I believe the relationship should be discussed as a reference 
clarity,_quality,_novelty_and_reproducibility: The paper is clear and the empirical study is strong.
summary_of_the_review: The authors solved an important problem for state-space models, and the empirical study is solid. I hope the authors can address my concerns, and currently, I give a weak acceptance.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper presents approaches to improve state space models for language models. I think this is a very interesting study and it is great that the proposed method is designed based on preliminary experiments on synthetic data to understand the limitations of SSMs. The results are promising and a step in the right direction for this class of models. All reviewers agree that this is a good paper. I recommend acceptance.
justification_for_why_not_higher_score: Stronger overall results.
justification_for_why_not_lower_score: Useful paper for the community, and the authors have managed to verify it on a 1.3B model.
Decision: Accept: notable-top-25%