Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduces GSS, a general purpose sequence model which leverages gated units and trains significantly faster as shown on several language modeling benchmarks. This simple-to-implement alternative to S4 and DSS which trains 2-3 times faster, and is competitive with Transformer-based baselines on long-range language modeling benchmarks.
strength_and_weaknesses: Strength:
1. The motivation of modeling long range dependencies is clear.
2. The paper is in a good shape and easy following.

Weaknesses:
1. The experimental part lacks of analysis on the proposed Gated State Spaces to show the insight. This part is not convincing.
2. Some minor issue in paper. For instance, I found it is hard to find the alignment on figure and code in Figure 1.

clarity,_quality,_novelty_and_reproducibility: The experimental part lacks of analysis on the proposed Gated State Spaces to show the insight. Without showing that, we are not convinced why Gated State Spaces helps by some intuitive proves.
summary_of_the_review: The paper is well written and easy following but the experimental part lacks analysis. I think a minor revise is needed.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new layer named Gated State Space (GSS) that combines the gate design in Gated Attention Unit (GAU) with a simplified version of Diagonal State Space (DSS) layer. Authors further propose a hybrid model that combines GSS with sparingly interleaved Transformer blocks. The resulted model trains significantly faster than the original DSS and can extrapolate to longer sequences at test time. The proposed model is evaluated on 4 long range language modeling tasks, comparing with a series of strong baseline models.
strength_and_weaknesses: **Strength:**
1. The paper is very clearly written and easy to understand.
2. Authors find that the main bottleneck of DSS comes from the high contextualized input dimension, as the FFT is performed independently on each dimensions of the sequence. With the GAU gating design, GSS enables a 4x reduction in this dimension and increases the throughput when trained on TPUs.
3. GSS can extrapolate naturally to unseen sequence length at test time due to the recurrent nature of DSS layer, which also allows faster inference speed.


**Weaknesses**:
1. GSS is largely motivated by GAU, but GAU is not included as a baseline method in this paper. The hybrid model with interleaved Transformer layer is closely related to the SRU++ (When Attention Meets Fast Recurrence: Training Language Models with Reduced Compute) paper, but it is not considered as a baseline method either.

**Questions**:
1. GSS is only evaluated on the long range language modeling tasks, which is fair enough. But have you evaluated GSS on any other tasks or commonly used LM datasets, e.g. Wikitext-103 and LRA?
2. Have you measured the training speed on GPUs?
clarity,_quality,_novelty_and_reproducibility: Paper is clearly written and reproducible based on the presented experimental details. 
summary_of_the_review: This paper is an established work that is based on some previous works, such as GAU and DSS, speeding up training compared to DSS. However, the empirical improvements seem not be very strong compared to the baseline methods. But it's a good paper that contributes to the state space models and connects with Transformer layers.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The attention mechanism in transformers scales quadratically with the input size that prohibits its application to long sequences. This work proposes a variant of S4 that can scale to long sequences. The key change is to apply S4 to a space of reduced dimensionality before projecting it back to the original space inspired by a recent work Gated Attention Units. This change reduces the time of fast Fourier transform operations which is a bottleneck on TPUs, resulting in a 3X speedup. In addition, this work also found that random initialization works as well as Hippo initialization.  Experiments on long-form text generation datasets found that while the proposed model slightly underperforms baselines such as block-recurrent transformer at the same parameter count, the proposed model is better at a fixed training cost, especially when interleaved with transformer layers. Besides, the model trained on shorter context window sizes is able to generalize to utilize more context at test time.
strength_and_weaknesses: Strengths:
1. Shows that a simple change that applies S4 to a space of reduced dimensionality improves throughput by a factor of two to three.
2. It's interesting that a model trained on shorter context sizes can learn to utilize longer contexts at inference time.

Weaknesses:
1. The main argument is that while the proposed model underperforms the baseline block-recurrent transformer at similar number of parameters, it's better at the same training cost. However, it is not clear to me that the baseline is optimized for limited training cost scenarios: for example, does training the baseline for a shorter period of time significantly degrades its performance in terms of PPL? In this regard, I think showing a curve that plots validation PPL against training time might be more convincing. But even so, the comparison is still not fair if the baseline is not tuned to arrive at the Pareto frontier for the given compute (such as by changing model size, learning rate, and other hyperparameters)
2. The model architecture seems to be optimized for a particular hardware architecture TPUs (which is not that widely accessible compared to GPUs). Does the increased throughput generalize to different hardware architectures such as GPUs?
3. Recent works on training large LMs (such as OPT) show that the attention operation only takes negligible time for large models, and the dominating term is the feedforward operations. Is it still meaningful to address the quadratic attention complexity issue as hardware gets more and more powerful and we can run larger and larger models?
4. Since Gated Attention Units (Hua et al 2022) motivated this work, why don't you compare to it as a baseline? Besides, in Table 1, can you compare to Transformers on other datasets as well (for the 512 setting) to get a sense of how much gain we can get by leveraging more context?

Typos:
1. Page 6 last paragraph: we present results for on a large range of sequence lengths
clarity,_quality,_novelty_and_reproducibility: This paper is clear and provides details on hyperparameter settings. This work is original to my knowledge.
summary_of_the_review: I think this paper is marginally below the acceptance threshold, due to a few concerns: first, at the same parameter count this work seems to underperform baselines; second, I have some concerns about the argument based on limiting the training cost as mentioned above; third, the final model is a hybrid between S4 and transformer layers, and I'm not sure if it will be actually used due to 1) the hybrid architecture is not as easy to implement and tune as pure S4 layers or pure transformer layers; 2) the speedups are only measured on TPUs which is less widely used than GPUs; 3) the recent lesson we learned in training large LMs that attention cost is negligible as we scale the model size. That being said, this paper has its merits in that it fixes a compute bottleneck and observes good empirical speedups. Therefore, I think this paper is marginally below the acceptance threshold.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposed to leverage the gating structure in gated attention unit (GAU) in state space models, named Gated State Space (GSS). By reducing the dimension of the state space module in GSS which requires FFT to compute the output, GSS is faster than the diagonal state space (DSS) model.

Experiments were conducted on four language modeling benchmarks, and the authors demonstrated the effectiveness and efficiency of GSS comparing with DSS and other Transformer-based baselines.
strength_and_weaknesses: Strengths:
The design of GSS is well-motivated and the paper writing is clear and easy to follow. The authors also clearly claimed their contributions.

Weaknesses:
The main concerns are from experiments. I found that the experimental setup makes the comparisons unfair, which makes it difficult to position GSS among other neural models in language modeling. I elaborated my concerns in the following questions.

clarity,_quality,_novelty_and_reproducibility: The experimental setup in this paper make it difficult to fairly compare GSS with previous models:

1. As the authors created their own vocabularies, the results in Table 3 are not directly comparable.  

2. Since in GSS the FFN module is omitted, the authors increases the number of layers to match the number of parameters with DSS. However, we know that deeper neural networks usually deliver better results than shallower ones with higher-hidden dimensions. Have you tried to increases the model dimension instead of increasing the number of layers? I believe it is a more fair comparison.

3. As GSS is directly inspired from GAU and FLASH (Hua, et al, 2022), why the authors have not compared GSS directly with FLASH, but using other variants of Transformers as baselines? Since GSS has very different organization of parameters (more layers and no FFNs), the comparison with FLASH (with the same number of layers and similar model size) would make a more clear position of GSS among previous models, particularly attention-based models.

Other questions about experiments:

1. In GSS, the $N$ is set to 512, while in S4 and DSS, $N$ is usually much smaller, e.g. (64). In the experiments, what is the value of $N$ in DSS? Is it important to use larger $N$?

2. GSS-Hybrid obtained better PPL than GSS, and the authors claimed that the self-attention is modeling local dependencies and the state space model is modeling long-term ones. However, there were no analysis or results to support this. Is it more reasonable that the self-attention is more using to model long-term dependencies (within a chunk of size 512) while the state space model is for local ones across successive chunks?
summary_of_the_review: To sum up, I am worried that the paper needs more experiments and analysis before publication.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes the Gated State Space (GSS) layer for autoregressive sequence modelling. The approach is an alternative to Diagonal State Spaces (DSS), training much faster, in particular on TPUs, while scaling to longer sequence lengths and generalizing better on longer inputs. On perplexity, performance is worse than Transformers evaluating on sequence length 512, but better when on longer sequence lengths (which the Transformer cannot handle), and comparable to or better than FLASH (which incorporated gating in Transformers) on lengths 1024. A GSS-Transformer hybrid performs best on some evaluations. While there is a small performance increase on GPUs, the main performance increase is specific to TPUs, which somewhat limits the potential impact of the method, although hardware specific optimizations are still valuable. The Transformer baseline is not necessarily tuned to be at the Pareto frontier for the given compute, but the GSS still obtains strong performance and enables leveraging longer sequence lengths. 

The contributions of the paper are strong enough to be accepted. However, the paper could benefit from improving the presentation and performing more extensive experiments, such as evaluating on downstream sequence or text generation applications. 

Comments: Please integrate the additional results in the appendix (of the revised version) into the main paper. Please avoid using subjective language in the paper, e.g. “To our delight” (p2), “it is quite relieving” (p9). 

justification_for_why_not_higher_score: The paper's empirical results are not comprehensive enough to justify a spotlight or oral presentation. 
justification_for_why_not_lower_score: The proposed method and the experimental results presented are strong enough to justify acceptance, even though the paper could benefit from some additional improvements.
Decision: Accept: poster