Title: Hierarchically Gated Recurrent Neural Network for Sequence Modeling
Venue: NeurIPS 2023
Accepted: yes
Keywords: RNN

Sequence Modeling

NLP
Abstract: Transformers have surpassed RNNs in popularity due to their superior abilities in parallel training and long-term dependency modeling.
Recently, there has been a renewed interest in using linear RNNs for efficient sequence modeling.
These linear RNNs often employ gating mechanisms in the output of the linear recurrence layer while ignoring the significance of using forget gates within the recurrence. In this paper, we propose a gated linear RNN model dubbed Hierarchically Gated Recurrent Neural Network (HGRN), which includes forget gates that are lower bounded by a learnable value. The lower bound increases monotonically when moving up layers. This allows the upper layers to model long-term dependencies and the lower layers to model more local, short-term dependencies. Experiments on language modeling, image classification, and long-range arena benchmarks showcase the efficiency and effectiveness of our proposed model. The source code is available at https://github.com/OpenNLPLab/HGRN.

Hierarchically Gated Recurrent Neural Network for Sequence Modeling

Abstract

Transformers have surpassed RNNs in popularity due to their superior abilities in parallel training and long-term dependency modeling. Recently, there has been a renewed interest in using linear RNNs for efficient sequence modeling. These linear RNNs often employ gating mechanisms in the output of the linear recurrence layer while ignoring the significance of using forget gates within the recurrence. In this paper, we propose a gated linear RNN model dubbed Hierarchically Gated Recurrent Neural Network (HGRN), which includes forget gates that are lower bounded by a learnable value. The lower bound increases monotonically when moving up layers. This allows the upper layers to model long-term dependencies and the lower layers to model more local, short-term dependencies. Experiments on language modeling, image classification, and long-range arena benchmarks showcase the efficiency and effectiveness of our proposed model. The source code is available at https://github.com/OpenNLPLab/HGRN.

1 Introduction

Sequence modeling is a fundamental problem in various domains such as natural language processing [12, 43, 44, 61, 64], time series analysis [84], computer vision [3, 13, 45, 74], and audio processing [1, 18, 73]. Prior to the invention of Transformers [81], RNN and its variants were the primary selections of architectures for sequence modeling, and have been widely used in machine translation [6], stock price prediction [68], weather forecasting [65], speech recognition [51], and etc.

RNNs have two main drawbacks: slow sequential training and limited capability in modeling longterm dependencies. With the swift development of deep learning and the pervasive use of GPUs, these drawbacks prevent it from flourishing in modern long-sequence modeling tasks. Meanwhile, Transformers [81] have rapidly gained popularity and now dominate various research areas in sequence modeling due to their better abilities in parallel training and long-term dependency modeling. However, Transformer's quadratic time complexity makes long sequence modeling expensive. On the other hand, RNN offers linear complexity and serves as an ideal choice for long sequence modeling. This works aim to addressing these RNN drawbacks, revitalizing their applicability in long-sequence modeling tasks.

To address the training inefficiency problem, we turn to more efficient RNN variants that employ element-wise linear recurrence (ELR) relations [48]. ELR provides two main advantages: (i) By removing nonlinearities in the recurrence, it enables parallelized training. (ii) By assuming independence between distinct hidden states, it enables efficient hidden state updates (through element-wise product instead of matrix multiplication) [20, 40]. Notably, ELR has been used in many modern linear RNN models, including the diagonalized versions of structured state-space models [21] (S4) [20, 26, 71] and RWKV [55]. In recent advancements, numerous studies have incorporated

gating mechanisms into the outputs of linear recurrence layers [11, 46, 49, 55, 82], similar to the output gates in LSTMs and leading to considerable performance gains. However, most current studies overlook the significance of the forget gate, which is often regarded as the most important gate in LSTMs [19, 80]. In this work, we underscore the importance of employing forget gates in linear RNNs and adopt gated linear RNNs for both efficiency and high performance.

To effectively capture long-term dependencies in gated RNNs, it is crucial to maintain high forget gate values close to one [23]. However, gates in saturated regimes (i.e., close to zero or one) suffer from the gradient vanishing issue [23]. Moreover, if all forget gate values are close to one, RNNs will not be able to effectively forget irrelevant information, compromising their ability to model short-term dependencies. To address these challenges, we introduce Hierarchically Gated Recurrent Units( HGRU ). In HGRU , we add an additive learnable value, referred to as the lower bound, to the original forget gate value, effectively mitigating the issue of saturated gates [23] by pushing gate activations away from the saturated regimes. Furthermore, we design the lower bounds to increase monotonically as we move up the layers of the RNN. This ensures that the forget gate values in the lower layers remain relatively small, enabling the necessary forgetting of past information for modeling short-term dependencies. In contrast, in the uppermost layer, the forget gate values approach one, facilitating the effective modeling of long-term dependencies. Our proposed model has proven to be highly efficient and effective, as demonstrated by its outstanding performance in language modeling, image classification, and long-range arena benchmarks.

2 Related work

Efficient token mixing for sequence modeling. [83] abstracts self-attention (SA) as token mixing, thereby transforming the Transformer architecture into MetaFormer. MetaFormer comprises essential components such as token mixer, channel mixer, residual connections, and LayerNorm. This abstraction highlights that the success of Transformers does not solely rely on SA but rather on the holistic integration of these components. Notably, token mixers can be replaced with simpler alternatives like pooling layers without compromising the model's performance in the context of vision transformer. For sequence modeling tasks, [29] provides a comprehensive analysis and discussion of different token mixing strategies. Two prominent contenders, long convolution and linear recurrence, show promise as replacements for SA modules in long sequence modeling due to their superior asymptotic time complexity and competitive performances. In long convolution models [14, 41, 57, 59], the kernel size matches the input sequence length, enabling a broader context compared to traditional convolutions. Training is accomplished using the efficient O ( n log n ) fast Fourier transforms (FFT) algorithm. However, long convolutions face challenges such as the need for causal convolution inference, which requires caching all historical computations similar to the key-value (KV) cache in SA. This can lead to memory limitations when processing long sequences. Moreover, the inference complexity of long convolutions remains higher than that of RNNs. These factors make linear RNNs a more suitable alternative to replace SA in long-sequence modeling. TransNormerLLM [61] scales efficient token mixing in large language models to achieve competitive performance and superior training and inference efficiency compared to transformer-based models.

Element-wise linear recurrence. The slower training speeds of traditional RNNs can be attributed to two main factors: (i) The updating of the hidden state involves full matrix multiplication. (ii) The presence of nonlinearity within the recurrence prevents parallel computation. To tackle the first issue, [40] introduced a simplified interaction between hidden states. This allowed the hidden state update to be performed using an element-wise product instead of full matrix multiplication. They demonstrated that this approach is notably fast when the (nonlinear) recurrence for each dimension is fused within a single CUDA kernel. Likewise, for the linear case, diagonalized versions of S4 [20, 26] have also exhibited speed improvements over S4 by leveraging element-wise recurrence. Regarding the second challenge, the ability to capture nonlinear dependencies on past data can be achieved by stacking multiple linear recurrence layers interleaved with nonlinear MLP blocks. This indicates the potential to eliminate nonlinearity, as suggested by [4, 25, 48]. Empirical support for this strategy's effectiveness came later, as demonstrated by [11, 20, 24, 53, 55, 71]. [52] further highlighted that such an architecture still possesses Universal Approximator properties, thus justifying the employment of linear recurrence. By excluding nonlinearity, [48, 71] showed that the parallel scan algorithm can be used for parallel training.

Linear recurrence can be broadly categorized into exponential moving averages (EMA) and gating schemes, as noted by [48]. The key difference is whether the decay rate is data-dependent. Models such as S4 [21], S4D [20], MEGA [46], RWKV [55], and LRU [53] utilize the EMA approach, where the decay rate is static for all time steps (i.e., data-independent), while our model uses a datadependent dynamic decay rate through the use of the forget gate. We remark on the importance of incorporating a data-dependent decay rate, which is largely ignored by current works in linear RNNs. Although liquid S4 [28] uses a dynamic transition matrix (which amounts to a data-dependent decay rate), it employs a limited form for FFT-based training. Our model does not have the convolutional view and thus cannot use FFT for training but allows the use of parallel scan.

The field of linear Transformers and linear RNNs exhibits a close relationship. [34] shows that linear Transformers can be reformulated as RNNs during auto-regressive decoding, revealing similarities to the update rules observed in fast weight additive outer products [66, 67]. These updates can be seen as a special case of element-wise linear recurrence, where forget gate values are consistently set to one across time and hidden states are two-dimensional. However, this formulation in linear Transformers lacks the ability to forget irrelevant information, resulting in the attention dilution issue [60]. To address this limitation, [66] introduced the delta rule to forget values associated with the current write key by removing the corresponding value before adding the new value. Alternatively, [47, 56] proposed gating mechanisms similar to those in gated RNNs to facilitate the forgetting of irrelevant information.

Long-term dependencies in RNNs. RNNs fall short in long-term dependency modeling, which is commonly attributed to the gradient vanishing issue. Three methods are typically applied to mitigate this issue. (i) Gating mechanisms [9, 17, 23, 30, 70], which are believed to be crucial to the success of LSTMs, use additive (instead of multiplicative) hidden state update rules to improve gradient flow. (ii) Regularizing or initializing the eigenvalues of the recurrent weight matrix (close) to one via identity matrices [38] or unitary matrices [2]. In the diagonal linear RNN case, the eigenvalues coincide with the element-wise decay rates, and LRU [53] uses randomized linear algebra techniques to initialize eigenvalues to be close to one. [53] also interestingly points out that many modern state-space models use a very small time step value on initialization for discretization, resulting in eigenvalues or decay rates close to one. (iii) Adding skip connections between distant time steps to allow shortcuts for gradient flow [5, 8, 37].Our approach combines (i) and (ii), which improves gating mechanisms with a regularized dynamic decay rate that approaches one in the upper layer.

3 Method

3.1 Architecture overview

Our proposed Hierarchically Gated Recurrent Network ( HGRN ) is depicted in Figure 1. It has multiple stacked layers, each of which consists of a token mixing module HGRU and a channel mixing module GLU (Gated Linear Unit [69]).

3.2 HGRU exploration

We begin with a simple gated linear recurrent layer, which is defined as:

where ⊙ denotes the element-wise product. Following the terminology used in the RNN literature, we refer to f t and i t as the forget and input gates, respectively. It is worth noting that f t and i t depend only on x t and not on h t -1 .

Algorithm 1 Recurrent Computing

This characteristic enables the use of the parallel scan algorithm [48, 71], otherwise it is infeasible. We then make the following changes toward our final HGRU step by step.

Input

Linear

*

Output

N

×

Linear

HGRU

GLU

HGRN

Output

HGRU

Input

Linear

Linear

Recurrent Computing

Output

LayerNorm

HRU

SiLU

Sigmoid

𝜃

SiLU

𝑐𝑡

𝜇𝑡

𝛾 𝑘

𝜆𝑡

Input

HRU

Complex-valued recurrence. For linear RNNs with static decay rates, it is common to perform eigendecompositions on the recurrent weight matrix to achieve element-wise linear recurrence. However, if only real-valued eigenvalues are allowed, it restricts the range of the recurrent weight matrix to be symmetric, limiting the model's expressiveness. To overcome this limitation, linear RNNs often employ complex-valued eigenvalues to enhance the model's expressive power [20, 26, 27, 32, 53]. Motivated by this, we extend our model to consider h t , i t , c t ∈ C 1 × d as complex values. For the input c t , we parameterize its real and imaginary parts separately as follows:

Regarding the forget gate values, we find it convenient to use the exponential representation of complex numbers and parameterize f t as follows: f t = λ t ⊙ exp( iθ t ) . Here, i 2 = -1 , λ t , θ t ∈ R d and exp( iθ t ) = cos θ t +sin θ t i . The magnitude argument λ t determines the intensity of remembering historical information, while the phase argument θ t determines the oscillation frequencies. We find that parameterizing θ t in a data-independent manner is preferable, as it allows for a clear interpretation of encoding relative position information (see next subsection for more discussions) , which is reminiscent of Rotary Positional Embedding (RoPE) [72]. We shared θ arcoss times steps, i.e., f t = λ t ⊙ exp( iθ ) , initialize θ as RoPE does, but make it learnable like LRPE [63].

Lower bound on forget gate values. Since the intensity of remembering information is only related to the magnitude argument λ t , we focus on how to add a lower bound to λ t . As mentioned earlier, we want to set a monotonically increasing lower bound on the forget gate (magnitude) values. Inspired by ON-LSTM [70], we employ the cummax activation function to achieve this. Concretely, we allocate Γ ∈ R H × d to parameterize lower bounds independently for all hidden states, where H is the number of layer. Assuming the layer index is k , we have the following calculations:

Here we define [Cumsum( x )] k = ( ∑ k i =1 x i ) -x 1 to prevent the highest layer's lower bound from being one as we still want the ability to forget irrelevant information.

We remark that there is a difference in the use of cummax between our model and ON-LSTM. In ON-LSTM, cummax is applied to the hidden state dimension within a single layer, while in our case, we apply cummax on the layer dimension across different layers to enable upper layers to model long-range dependencies.

Finally, λ t in the k -th layer is parameterized as follows:

Comparing to before (i.e., without lower bounds), to achieve the same forget rate value ¯ γ closed to one, µ t will be pushed away from the Sigmoid activation function's saturated regions (i.e., near one),

thereby mitigating the gradient vanishing issue [23] and making gradient-based optimization easier.

Tying input and forget gates. To reduce the number of parameters, it is common to use leaky units, i.e., tying the input gate with the forget gate using i t = 1 -f t , which has a close relationship to the discretization of continuous-time system [75] and exponential moving average [33], and has been proven effective empirically [9, 19]. To allows for a clear interpretation of encoding relative position information, we only apply this strategy on the magnitude argument:

Output gates and projection. The addition of gates to the output of the recurrence layer has been shown to be effective in state-space models [11, 46, 49, 82]. Motivated by this, we incorporate an output gate before performing the output projection as follows and get HGRU :

3.3 Token mixing perspective of HGRU

We provide the token mixing perspective of HGRU similar to [32]. Expanding Equation 2, we have:

Written in matrix form, we have:

So the token mixing module can be formed as follows:

Note that the token mixing matrix A can be decomposed into two parts A = Λ ⊙ Θ :

This decomposition means that the Token mixing matrix Λ can be decoupled into two independent modules, where Λ models the long-distance dependency and Θ , a Toeplitz matrix, models the relative positional relationship and enhanced expressiveness. Note that if Θ depends on the input, then the matrix Λ will no longer be a Toeplitz matrix, thus unable to capture relative position information. It can be also viewed as a RoPE-enhanced attention mechanism: Λ corresponds to the attention matrix but the attention score here is the cumulative product of data-dependent decay rates; Θ directly corresponds to RoPE.

4 Experiments

We conduct a comparative analysis between our proposed HGRN and four widely adopted sequence modeling structures, i.e., attention-based, MLP-based, FFT-based, and state-space-based. We evaluate HGRN on the WikiText-103 dataset [50] and the Pile [15] dataset for autoregressive language modeling, as well as the length extrapolation ability. To assess the accuracy and efficiency of our model in handling long-term dependencies, we utilize the LRA benchmark [78]. Additionally, we showcase the robustness of HGRN in computer vision task on the ImageNet-1k dataset.

4.1 Setting

We implement our models in Pytorch [54] and train them on 8 Nvidia A100 GPUs. For HGRN , we found that fusing element-wise recurrence into a single CUDA kernel results in fast running speed in practice. [48] also found that unless the sequence length is sufficiently large, the parallel scan's implementation is not necessarily faster than the sequential scan. As such, we use a CUDAbased sequential scan for implementation; however, our model has the potential to model very long sequences through the use of a parallel scan.

We adopt the same training configuration for all competitors, including batch size, learning rate, training epochs or iterations, etc. We list detailed hyper-parameters in the Appendix. For the autoregressive language modeling, we conducted three sets of experiments. Firstly, we validated the performance of two different-scale models on the Wikitext-103 dataset. We used the TNN configuration to verify the performance of the model at around 44m, and the Hyena configuration to verify the performance of the model at around 125m. To evaluate the performance of larger-scale models, we trained a 1b Transformer and HGRN on the Pile dataset using 10b tokens. To assess the performance in downstream tasks, we trained HGRN models of 150m, 350m, and 1b on the Pile dataset using 100b tokens and conducted zero-shot evaluations on downstream tasks.

For the LRA benchmark, We report results on all 6 tasks. For the image classification on the ImageNet-1k dataset, We integrate HGRN into the DeiT [79] structure, we replace the transformer layers with our HGRN modules. It is compared to the performance of the vanilla DeiT on the ImageNet-1K dataset for image classification.

4.2 Results

Autoregressive Language Modeling Autoregressive language modeling stands as a prominent task within the field of natural language processing, as it serves as a measure of a language model's causal inference capability. This task requires the model to estimate the probability distribution of the subsequent token based on the previously seen tokens.

We show the performances of the autoregressive language modeling in table 1 and table 2. Compared to transformer-based methods, HGRN performs favourably than most efficient variants of the vanilla transformer such as FLASH [31], 1+elu [35], Performer [7] and cosFormer [62].

Also, HGRN achieves better results than the MLP-based methods with a notable margin. Nevertheless, HGRN performs similarly to the original transformer [81]. Finally, HGRN shares similar concepts with RNN-based such as S4 [22], DSS [26], GSS [49], RWKV [55], and LRU [53], our HGRN also achieves superior performance to all RNN-based methods. This provide evidence HRGN may be an effective method in LM We also report the extrapolation ability of HGRN compared to previous methods in Table 14.

We also trained a 1b model on the Pile dataset and compared it with LRU and Transformer. Specifically, our training parameters included a sequence length of 1024, batch size of 96, 100k updates, and a learning rate of 5e-4. It can be seen that HGRN still performs better at the 1b scale. Additionally, we trained 100b tokens of

HGRN on the Pile dataset at 150m, 350m, and 1b sizes, and evaluated them against open-source Transformer-based models in downstream tasks. We selected Comparison on Commonsense Reasoning and Super GLUE tasks, and all evaluations were done using the lm-evaluation-harness [16]. HGRN achieves comparable performance to Transformer-based models when consuming only 1/3 of the tokens.

Long Range Arena LRA [77] is proposed as a comprehensive evaluation for assessing the performances of models in processing long-term dependencies in various sequential modeling tasks. We show a performance comparison between HGRN and existing methods in Table 6. HGRN achieves comparable results with other SOTA methods.

Image Classification The image classification results on the ImageNet-1K dataset are presented in Table 7. Notably, with comparable parameter sizes, our proposed HGRN model demonstrates superior performance compared to previous methods such as TNN and the vanilla transformer. It demonstrates the capability of HGRN in modeling visual modalities.

4.3 Ablation Study

We conducted ablation studies in the smallestscaled setting (i.e., TNN[59]'s setting on WikiText103 dataset) to thoroughly verify the effectiveness of each of our proposed components in HGRN . Experiments were conducted on the Pile dataset using a 1b model with 10b tokens for the forget gate experiment.

(i) With lower bound

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Forget rate

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

Percent

Layer 5

(ii) Without lower bound

0.2

0.4

0.6

0.8

1.0

Forget rate

0

2

4

6

8

10

Percent

Layer 5

(iii) LRU

0.6

0.7

0.8

0.9

1.0

Forget rate

0

2

4

6

8

Percent

Layer 5

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Forget rate

0

10

20

30

40

50

Percent

Layer 6

0.0

0.2

0.4

0.6

0.8

1.0

Forget rate

0

2

4

6

8

10

12

14

16

Percent

Layer 6

0.6

0.7

0.8

0.9

1.0

Forget rate

0

2

4

6

8

Percent

Layer 6

The influence of forget gate In table 8, we demonstrate the role of forget gate. From table 8, we observe that removing the forget gate significantly decreases the performance of HGRN , while adding a forget gate to LRU improves performance. On the other hand, using a data-independent forget gate (only lower bound) leads to lower performance compared to a data-dependent forget gate.

The influence of input gate and output gate

Table. 9 validates the effectiveness of using output gates and tying input and forget gates. w/o input gate means to remove the 1 -λ t term. w/o output gate means remove the left branch of HGRN in figure 1. Our design achieves the best performance.

The influence of lower bounds in forget gate

values We demonstrate the effectiveness of introducing a lower bound in Table 10 and Table 13. From Table 10, we observe that gating (i.e., without lower bound) is more critical than the lower bound (i.e., only lower bound). Combining gating and the lower bound consistently provides benefits, but the most significant improvement arises from the monotonically increasing lower bound. This aligns with the intuition that lower layers should primarily focus on nearby tokens, while upper layers should attend more broadly to capture long-term dependencies [58].

Table 13 highlights the essential role of the lower bound in long sequence processing tasks. In these tasks, the model's performance is notably poor and sometimes fails to converge without the lower bound. It is worth noting that language modeling tasks do not require extensive long-term dependencies, which explains why the model performs well even without the lower bound. However, in the task of LRA, the ability to capture long-term dependencies is crucial for achieving satisfactory performance.

The influence of complex-valued recurrence

Table 11 validates the utility of incorporating

complex values in element-wise linear recurrence. Additionally, the experiments show that the phase argument θ should not be data-dependent.

4.4 Analysis on forget gate values

Wepresent the distributions of forget gate values across layers for different methods in Table 12 and visualize the histogram of each layer in Figure 2, trained on the autoregressive language modeling task. The results demonstrate that the addition of lower bounds effectively increases the average forget gate values in higher layers (5-6). Notably, the medium forget gate values in the highest layer reach 0.98, enabling the modeling of long-term dependencies.

Table 11: Ablations of complex-valued recurrence on autoregressive language modeling. w/o complex means remove theta, data-dependent theta means theta is dependent on the input, this makes the matrix Λ not a Toeplitz matrix, which can not capture relative information.

HGRN 24.14 It is interesting to note that the average forget gate values of the LRU model consistently exceed those of our variant model without lower bounds, as per their eigenvalues. However, despite this, the language modeling performance of LRU is lower than that of our variant. Specifically, LRU scored 24.71, while our variant scored 31.12. This suggests that using data-dependent gates to selectively retain relevant information is advantageous, rather than relying on data-independent forget gate values across all time steps.

5 Conclusion

In this work, we have shown that gated linear RNNs could obtain impressive performance across different tasks and modalities without compromising efficiency. We highlighted the significance of the forget gate for linear RNNs in language modeling and emphasized the importance of an additive lower bound on forget gate values for modeling long-term dependencies.

Limitations and broader impact

Our empirical evaluation of HGRN remains on a smaller scale compared to other large-scale models. Potentially negative social consequences include the misuse of brain models for unsuitable purposes or applications, which must be prohibited by appropriate rules. In the era of large language models, the inference cost is the key limitation of transformer-based models. RNNs provide a solution with their lower inference costs. This could potentially lead to a significant evolution in the field.

References

Hassan Akbari, Liangzhe Yuan, Rui Qian, Wei-Hong Chuang, Shih-Fu Chang, Yin Cui, and Boqing Gong. Vatt: Transformers for multimodal self-supervised learning from raw video, audio and text. arXiv preprint arXiv:2104.11178 , 2021.

Martín Arjovsky, Amar Shah, and Yoshua Bengio. Unitary evolution recurrent neural networks. In Maria-Florina Balcan and Kilian Q. Weinberger, editors, Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016 , volume 48 of JMLR Workshop and Conference Proceedings , pages 1120-1128. JMLR.org, 2016.

Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Luˇ ci´ c, and Cordelia Schmid. Vivit: A video vision transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 6836-6846, 2021.

David Balduzzi and Muhammad Ghifary. Strongly-typed recurrent neural networks. In MariaFlorina Balcan and Kilian Q. Weinberger, editors, Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016 , volume 48 of JMLR Workshop and Conference Proceedings , pages 1292-1300. JMLR.org, 2016.

Shiyu Chang, Yang Zhang, Wei Han, Mo Yu, Xiaoxiao Guo, Wei Tan, Xiaodong Cui, Michael Witbrock, Mark A. Hasegawa-Johnson, and Thomas S. Huang. Dilated recurrent neural networks. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA , pages 77-87, 2017.

Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoderdecoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 1724-1734, Doha, Qatar, Oct. 2014. Association for Computational Linguistics.

Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers. arXiv preprint arXiv:2009.14794 , 2020.

Junyoung Chung, Sungjin Ahn, and Yoshua Bengio. Hierarchical multiscale recurrent neural networks. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings . OpenReview.net, 2017.

Junyoung Chung, Çaglar Gülçehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. CoRR , abs/1412.3555, 2014.

Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: Fast and memory-efficient exact attention with IO-awareness. In Advances in Neural Information Processing Systems , 2022.

Tri Dao, Daniel Y. Fu, Khaled Kamal Saab, Armin W. Thomas, Atri Rudra, and Christopher Ré. Hungry hungry hippos: Towards language modeling with state space models. CoRR , abs/2212.14052, 2022.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pages 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 , 2020.

Daniel Y. Fu, Elliot L. Epstein, Eric Nguyen, Armin W. Thomas, Michael Zhang, Tri Dao, Atri Rudra, and Christopher Ré. Simple hardware-efficient long convolutions for sequence modeling. CoRR , abs/2302.06646, 2023.

Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The Pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027 , 2020.

Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric

Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation. Zenodo, Sept. 2021.

Felix A. Gers, Jürgen Schmidhuber, and Fred A. Cummins. Learning to forget: Continual prediction with LSTM. Neural Comput. , 12(10):2451-2471, 2000.

Yuan Gong, Yu-An Chung, and James Glass. AST: Audio Spectrogram Transformer. In Proc. Interspeech 2021 , pages 571-575, 2021.

Klaus Greff, Rupesh Kumar Srivastava, Jan Koutník, Bas R. Steunebrink, and Jürgen Schmidhuber. Lstm: A search space odyssey. IEEE Transactions on Neural Networks and Learning Systems , 28:2222-2232, 2015.

Albert Gu, Karan Goel, Ankit Gupta, and Christopher Ré. On the parameterization and initialization of diagonal state space models. In NeurIPS , 2022.

Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022 . OpenReview.net, 2022.

Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. In The International Conference on Learning Representations (ICLR) , 2022.

Albert Gu, Çaglar Gülçehre, Thomas Paine, Matt Hoffman, and Razvan Pascanu. Improving the gating mechanism of recurrent neural networks. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event , volume 119 of Proceedings of Machine Learning Research , pages 3800-3809. PMLR, 2020.

Albert Gu, Isys Johnson, Karan Goel, Khaled Saab, Tri Dao, Atri Rudra, and Christopher Ré. Combining recurrent, convolutional, and continuous-time models with linear state space layers. In Marc'Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual , pages 572-585, 2021.

Albert Gu, Isys Johnson, Karan Goel, Khaled Saab, Tri Dao, Atri Rudra, and Christopher Ré. Combining recurrent, convolutional, and continuous-time models with linear state-space layers, 2021.

Ankit Gupta, Albert Gu, and Jonathan Berant. Diagonal state spaces are as effective as structured state spaces, 2022.

Ankit Gupta, Harsh Mehta, and Jonathan Berant. Simplifying and understanding state space models with diagonal linear rnns. CoRR , abs/2212.00768, 2022.

Ramin Hasani, Mathias Lechner, Tsun-Hsuan Wang, Makram Chahine, Alexander Amini, and Daniela Rus. Liquid structural state-space models. In The Eleventh International Conference on Learning Representations , 2023.

Hongyu He and Marko Kabic. A unified view of long-sequence models towards modeling million-scale dependencies. CoRR , abs/2302.06218, 2023.

Sepp Hochreiter and Yoshua Bengio. Gradient flow in recurrent nets: the difficulty of learning long-term dependencies. 2001.

Weizhe Hua, Zihang Dai, Hanxiao Liu, and Quoc V Le. Transformer quality in linear time. arXiv preprint arXiv:2202.10447 , 2022.

Feiqing Huang, Kexin Lu, Yuxi CAI, Zhen Qin, Yanwen Fang, Guangjian Tian, and Guodong Li. Encoding recurrence into transformers. In The Eleventh International Conference on Learning Representations , 2023.

J. Stuart Hunter. The exponentially weighted moving average. Journal of Quality Technology , 18:203-210, 1986.

Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event , volume 119 of Proceedings of Machine Learning Research , pages 5156-5165. PMLR, 2020.

Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention. In International Conference on Machine Learning , pages 5156-5165. PMLR, 2020.

Guolin Ke, Di He, and Tie-Yan Liu. Rethinking positional encoding in language pre-training. In International Conference on Learning Representations , 2021.

Jan Koutník, Klaus Greff, Faustino J. Gomez, and Jürgen Schmidhuber. A clockwork RNN. In Proceedings of the 31th International Conference on Machine Learning, ICML 2014, Beijing, China, 21-26 June 2014 , volume 32 of JMLR Workshop and Conference Proceedings , pages 1863-1871. JMLR.org, 2014.

Quoc V. Le, Navdeep Jaitly, and Geoffrey E. Hinton. A simple way to initialize recurrent networks of rectified linear units. ArXiv , abs/1504.00941, 2015.

James Lee-Thorp, Joshua Ainslie, Ilya Eckstein, and Santiago Ontanon. FNet: Mixing tokens with Fourier transforms. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages 4296-4313, Seattle, United States, July 2022. Association for Computational Linguistics.

Tao Lei, Yu Zhang, Sida I. Wang, Hui Dai, and Yoav Artzi. Simple recurrent units for highly parallelizable recurrence. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , pages 4470-4481, Brussels, Belgium, Oct.-Nov. 2018. Association for Computational Linguistics.

Yuhong Li, Tianle Cai, Yi Zhang, De huai Chen, and Debadeepta Dey. What makes convolutional models great on long sequence modeling? ArXiv , abs/2210.09298, 2022.

Hanxiao Liu, Zihang Dai, David So, and Quoc V Le. Pay attention to mlps. Advances in Neural Information Processing Systems , 34:9204-9215, 2021.

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 , 2019.

Zexiang Liu, Dong Li, Kaiyue Lu, Zhen Qin, Weixuan Sun, Jiacheng Xu, and Yiran Zhong. Neural architecture search on efficient transformers and beyond. In arXiv preprint arXiv:2207.13955 , 2022.

Kaiyue Lu, Zexiang Liu, Jianyuan Wang, Weixuan Sun, Zhen Qin, Dong Li, Xuyang Shen, Hui Deng, Xiaodong Han, Yuchao Dai, et al. Linear video transformer with feature fixation. arXiv preprint arXiv:2210.08164 , 2022.

Xuezhe Ma, Chunting Zhou, Xiang Kong, Junxian He, Liangke Gui, Graham Neubig, Jonathan May, and Luke Zettlemoyer. Mega: Moving average equipped gated attention. CoRR , abs/2209.10655, 2022.

Huanru Henry Mao. Fine-tuning pre-trained transformers into decaying fast weights. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , pages 10236-10242, Abu Dhabi, United Arab Emirates, Dec. 2022. Association for Computational Linguistics.

Eric Martin and Chris Cundy. Parallelizing linear recurrent neural nets over sequence length. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings . OpenReview.net, 2018.

Harsh Mehta, Ankit Gupta, Ashok Cutkosky, and Behnam Neyshabur. Long range language modeling via gated state spaces. CoRR , abs/2206.13947, 2022.

Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. 5th International Conference on Learning Representations, ICLR, Toulon, France , 2017.

Yajie Miao, Mohammad Gowayyed, and Florian Metze. Eesen: End-to-end speech recognition using deep rnn models and wfst-based decoding. In 2015 IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) , pages 167-174. IEEE, 2015.

Antonio Orvieto, Soham De, Çaglar Gülçehre, Razvan Pascanu, and Samuel L. Smith. On the universality of linear recurrences followed by nonlinear projections. CoRR , abs/2307.11888, 2023.

Antonio Orvieto, Samuel L. Smith, Albert Gu, Anushan Fernando, Çaglar Gülçehre, Razvan Pascanu, and Soham De. Resurrecting recurrent neural networks for long sequences. CoRR , abs/2303.06349, 2023.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems , 32, 2019.

Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Huanqi Cao, Xin Cheng, Michael Chung, Matteo Grella, Kranthi Kiran G. V., Xuzheng He, Haowen Hou, Przemyslaw Kazienko, Jan Kocon, Jiaming Kong, Bartlomiej Koptyra, Hayden Lau, Krishna Sri Ipsit Mantri, Ferdinand Mom, Atsushi Saito, Xiangru Tang, Bolun Wang, Johan S. Wind, Stanislaw Wozniak, Ruichong Zhang, Zhenyuan Zhang, Qihang Zhao, Peng Zhou, Jian Zhu, and Rui-Jie Zhu. RWKV: reinventing rnns for the transformer era. CoRR , abs/2305.13048, 2023.

Hao Peng, Nikolaos Pappas, Dani Yogatama, Roy Schwartz, Noah A. Smith, and Lingpeng Kong. Random feature attention. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021 . OpenReview.net, 2021.

Michael Poli, Stefano Massaroli, Eric Nguyen, Daniel Y. Fu, Tri Dao, Stephen Baccus, Yoshua Bengio, Stefano Ermon, and Christopher Ré. Hyena hierarchy: Towards larger convolutional language models. CoRR , abs/2302.10866, 2023.

Michael Poli, Stefano Massaroli, Eric Nguyen, Daniel Y Fu, Tri Dao, Stephen Baccus, Yoshua Bengio, Stefano Ermon, and Christopher Ré. Hyena hierarchy: Towards larger convolutional language models. arXiv preprint arXiv:2302.10866 , 2023.

Zhen Qin, Xiaodong Han, Weixuan Sun, Bowen He, Dong Li, Dongxu Li, Yuchao Dai, Lingpeng Kong, and Yiran Zhong. Toeplitz neural network for sequence modeling. In The Eleventh International Conference on Learning Representations (ICLR) , 2023.

Zhen Qin, Xiaodong Han, Weixuan Sun, Dongxu Li, Lingpeng Kong, Nick Barnes, and Yiran Zhong. The devil in linear transformer. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , pages 7025-7041, Abu Dhabi, United Arab Emirates, Dec. 2022. Association for Computational Linguistics.

Zhen Qin, Dong Li, Weigao Sun, Weixuan Sun, Xuyang Shen, Xiaodong Han, Yunshen Wei, Baohong Lv, Fei Yuan, Xiao Luo, Yu Qiao, and Yiran Zhong. Scaling transnormer to 175 billion parameters. arXiv , 2023.

Zhen Qin, Weixuan Sun, Hui Deng, Dongxu Li, Yunshen Wei, Baohong Lv, Junjie Yan, Lingpeng Kong, and Yiran Zhong. cosformer: Rethinking softmax in attention. In ICLR , 2022.

Zhen Qin, Weixuan Sun, Kaiyue Lu, Hui Deng, Dongxu Li, Xiaodong Han, Yuchao Dai, Lingpeng Kong, and Yiran Zhong. Linearized relative positional encoding. Transactions on Machine Learning Research , 2023.

Zhen Qin and Yiran Zhong. Accelerating toeplitz neural network with constant-time inference complexity. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing . Association for Computational Linguistics, Dec. 2023.

Afan Galih Salman, Bayu Kanigoro, and Yaya Heryadi. Weather forecasting using deep learning techniques. In 2015 international conference on advanced computer science and information systems (ICACSIS) , pages 281-285. Ieee, 2015.

Imanol Schlag, Kazuki Irie, and Jürgen Schmidhuber. Linear transformers are secretly fast weight programmers. In International Conference on Machine Learning , 2021.

Jürgen Schmidhuber. Learning to control fast-weight memories: An alternative to dynamic recurrent networks. Neural Computation , 4:131-139, 1992.

Sreelekshmy Selvin, R Vinayakumar, EA Gopalakrishnan, Vijay Krishna Menon, and KP Soman. Stock price prediction using lstm, rnn and cnn-sliding window model. In 2017 international conference on advances in computing, communications and informatics (icacci) , pages 1643-1647. IEEE, 2017.

Noam Shazeer. GLU variants improve transformer. CoRR , abs/2002.05202, 2020.

Yikang Shen, Shawn Tan, Alessandro Sordoni, and Aaron C. Courville. Ordered neurons: Integrating tree structures into recurrent neural networks. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 . OpenReview.net, 2019.

Jimmy T. H. Smith, Andrew Warrington, and Scott W. Linderman. Simplified state space layers for sequence modeling. CoRR , abs/2208.04933, 2022.

Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. CoRR , abs/2104.09864, 2021.

Jingyu Sun, Guiping Zhong, Dinghao Zhou, Baoxiang Li, and Yiran Zhong. Locality matters: A locality-biased linear attention for automatic speech recognition. arXiv preprint arXiv:2203.15609 , 2022.

Weixuan Sun, Zhen Qin, Hui Deng, Jianyuan Wang, Yi Zhang, Kaihao Zhang, Nick Barnes, Stan Birchfield, Lingpeng Kong, and Yiran Zhong. Vicinity vision transformer. IEEE Transactions on Pattern Analysis and Machine Intelligence , 45(10):12635-12649, 2023.

Corentin Tallec and Yann Ollivier. Can recurrent neural networks warp time? In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings . OpenReview.net, 2018.

Yi Tay, Dara Bahri, Donald Metzler, Da-Cheng Juan, Zhe Zhao, and Che Zheng. Synthesizer: Rethinking self-attention for transformer models. In International conference on machine learning , pages 10183-10192. PMLR, 2021.

Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. Long range arena: A benchmark for efficient transformers. In International Conference on Learning Representations , 2020.

Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. Long range arena : A benchmark for efficient

transformers. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021 . OpenReview.net, 2021.

Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herve Jegou. Training data-efficient image transformers &amp; distillation through attention. In International Conference on Machine Learning , volume 139, pages 10347-10357, July 2021.

Jos van der Westhuizen and Joan Lasenby. The unreasonable effectiveness of the forget gate. CoRR , abs/1804.04849, 2018.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Junxiong Wang, Jing Nathan Yan, Albert Gu, and Alexander M. Rush. Pretraining without attention. CoRR , abs/2212.10544, 2022.

Weihao Yu, Mi Luo, Pan Zhou, Chenyang Si, Yichen Zhou, Xinchao Wang, Jiashi Feng, and Shuicheng Yan. Metaformer is actually what you need for vision. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022 , pages 10809-10819. IEEE, 2022.

Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI conference on artificial intelligence , volume 35, pages 11106-11115, 2021.

Chen Zhu, Wei Ping, Chaowei Xiao, Mohammad Shoeybi, Tom Goldstein, Anima Anandkumar, and Bryan Catanzaro. Long-short transformer: Efficient transformers for language and vision. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , 2021.

6 Appendix

In this appendix, we examine the extrapolation ability of HGRN and provide the training and inference speed comparison of HGRN and existing efficient sequence modeling methods. We also illustrate the forget rates of each layer on a trained language model of HGRN .

We also report the extrapolation ability of HGRN compared to previous methods in Table 14.

6.1 Extrapolation test

In this section, we tested HGRN 's extrapolation ability by directly inferring the model with a variety of sequence lengths. As shown in Table 14, our method has the ability to train short and test long.

6.2 Speed comparison

In this section, we benchmark the speed of our method on the LRA benchmark. Our method achieves state-of-the-art training and inference speed.

6.3 Visualization

In this section, we visualize the forget rates of each layer on a model trained on language modeling tasks.

6.4 Configurations

We list detailed hyper-parameters of our experiments here.

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Forget rate

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

Percent

Layer 1

(ii) Without lower bound

0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0

Forget rate

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

Percent

Layer 1

(iii) LRU

0.6

0.7

0.8

0.9

1.0

Forget rate

0

2

4

6

8

10

Percent

Layer 1

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Forget rate

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

Percent

Layer 2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Forget rate

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

Percent

Layer 2

0.650.700.750.800.850.900.951.00

Forget rate

0

2

4

6

8

Percent

Layer 2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Forget rate

0

2

4

6

8

10

12

14

16

Percent

Layer 3

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Forget rate

0

2

4

6

8

10

12

14

16

Percent

Layer 3

0.60 0.65 0.70 0.75 0.80 0.85 0.90 0.95 1.00

Forget rate

0

2

4

6

8

10

Percent

Layer 3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Forget rate

0

2

4

6

8

10

12

Percent

Layer 4

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Forget rate

0

2

4

6

8

10

12

14

16

Percent

Layer 4

0.6

0.7

0.8

0.9

1.0

Forget rate

0

1

2

3

4

5

6

7

8

Percent

Layer 4

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Forget rate

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

Percent

Layer 5

0.2

0.4

0.6

0.8

1.0

Forget rate

0

2

4

6

8

10

Percent

Layer 5

0.6

0.7

0.8

0.9

1.0

Forget rate

0

2

4

6

8

Percent

Layer 5

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Forget rate

0

10

20

30

40

50

Percent

Layer 6

0.0

0.2

0.4

0.6

0.8

1.0

Forget rate

0

2

4

6

8

10

12

14

16

Percent

Layer 6

0.6

0.7

0.8

0.9

1.0

Forget rate

0

2

4

6

8

Percent

Layer 6

17

(i) Layer 1

(ii) Layer 1

(iii) Layer 2

(iv) Layer 4

(v) Layer 5

(vi) Layer 6

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work is concerned with improving linear RNNs (which can be efficiently parallelized) for long-sequence modeling. It proposes to address this problem with gating schemes and complex forget gate values. They introduce a learnable lower bound on the forget gate (or essentially the diagonal dynamics matrix) for each layer to capture both short-term and long-range dependencies. The result is the proposed Hierarchical Gated Recurrent Units (HGRU). This RNN is evaluated on Wikitext, LRA and Imagenet-1K. Ablations are performed to analyze the effectiveness of the design choices.\n\nSTRENGTHS:\n- The area of linear SSMs/RNNs has gained a fair amount of recent attention and the angle this paper takes of considering the linear RNN dynamics matrix from the point of view of the LSTM forget gate is of interest to the community and has the potential to help bridge the gap between prior RNN work and the more modern linear SSM/RNN work\n- While not framed in this way, the forget gate formulation serves as a practical example of input-dependent dynamics, of interest to the deep SSM community\n- The method appears to achieve strong performance on the tasks evaluated in this paper\n- The ablations are helpful to understand the design choices better\n\nWEAKNESSES:\n- While most relevant works are cited in some place in the paper, the results of many of these methods are inexplicably left out of the results tables, leading to misleading statements:\n  - For example SgConv and MEGA are both cited in the paper, but their results are left out of the LRA table. Both of these methods have a better Path-X score (97.83 and 97.98) and overall average (87.17 and 88.21) than the proposed HGRU and both have been released on Arxiv since 2022. This makes the statements in lines 229 that HRGU \"is better than all previous methods\" and lines 231-232 that HRGU \"outperforms all previous methods on this task\"  clearly incorrect. \n  - Similarly, S5 (https://arxiv.org/abs/2208.04933) is never cited at all in the paper despite the fact that it is a linear RNN method that uses the same parallel scan approach for computing a linear recurrence as HGRU (assuming HGRU actually uses a parallel scan though see questions for clarification). The LRU paper that is cited discusses S5 extensively.   In addition, S5's scores are left out of the LRA table despite having a better path-X and overall average than HGRU (98.58 and 87.46), similar to the SgConv and Mega examples mentioned above.  This is particularly relevant since it serves as an ablation showing the forget gate innovations in this paper are not strictly necessary for strong longe-range modeling with a linear RNN.\n  - To clarify, I do not think it would hurt the story of this paper to include these methods that perform slightly better, and in fact I think the statement in line 232 \"It validates the ability of HRGN in handling long sequence tasks\" is completely reasonable and supported by the evidence. It is just the exclusion of these other methods may mislead readers not as familiar with the research on what is strictly necessary for strong performance and the statements listed above regarding \"outperforms all...\" are incorrect.\n  - Finally, given LRU is cited as concurrent work, it would also be relevant to cite DLR (https://arxiv.org/abs/2212.00768) since it explores similar themes.\n\n- It would seem that H3 (cited in the paper) should have been a baseline for the Wikitext-103 experiments. Alternatively, the LRU could have been plugged in for the S4D SSMs used in H3.  As stated in the paper, it is well known in the literature at this point that just using an SSM/Linear RNN such as S4, DSS or LRU without additional gating is not sufficient for language modeling. But additional gating such as in H3 has been shown to work well, such as in the H3 paper and the Wang et al. 2022 papers that are cited. Including this baseline would allow for a better comparison with prior methods on language and allow for a better understanding of the effectiveness of the HGRU innovations vs the gating findings from previous work.\n\n- The overall presentation is poor. Many of the tables say that certain scores are bold or underlined, yet the tables often have no bolds or underlines, or it is inconsistent. Many things are promised to be in the appendix, but then are not there (see questions section below for specific examples). I think some of the sections in the presentation of the method could be improved (see questions section below for suggestions). In addition, Figure 1 is not clearly labeled, leaving one guessing on which gates are located where.\n\n- The authors have checked yes on the author checklist for including training details and reproducibility, but I think this is clearly wrong. First, it is not even clear which methods were implemented by the authors and which methods have numbers reported from other works. Section 4.1 states that \"we adopt the same training configuration for all competitors....\" and that detailed hyperparams are listed in the appendix. However, no hyperparameters are listed in the appendix and many of the performance scores appear to be reported directly from  the prior papers. In addition, no code is included to allow one to try and make sense of this. The authors should make it clear which methods were implemented and evaluated as well as provide detailed hyperparameters to enable reproducibility.\n\nQUESTIONS:\nMajor questions/suggestions:\n\n- Line 186: It is stated here that $\\lambda_t$'s near 1 \"makes HRGN more efficient modeling of global information\". What does efficient mean here? More efficient than what? How is this claim supported? More clarification should be provided here to avoid making an imprecise or unwarranted claim. \n\n- Line 187: The \"visualization of the token mixing matrix\" cannot be found in the appendix\n\n- Section 3.4: I would recommend moving this section up closer to the \"Lower bound on forget gate values\" section. I found myself having to scroll back and forth to remind myself of the definitions. \n\n- Figure 1: needs more labels. How many gates are there? Is the output gate from equation 3 located in the HRU (per the comment in line 174) or in the HGRU (middle figure)? Clearly labeling the location of the f and g gates, as well as the hidden state h would help a ton.  It would also be nice if the full equation for the recurrence (including the computation of $\\lambda_t$) was a part of this diagram, where one would not have to wonder what each of the different symbols mean, or scroll throughout the paper to piece this together. \n\n- Algorithm 1: I would recommend including a line for the explicit computation of $\\lambda_t$. This is an important part of your approach, but the algorithm makes it appear like this is just some learnable parameter. \n\n- Line 208: What does identical training hyperparameters mean? It is not even clear the baselines reported from other papers all use the same hyperparameters. Also is this the right approach? It is not obvious that each method should use the exact same learning rate etc and I would think it would be reasonable to perform a mild sweep over a few different learning rates and other settings. If not, how did you choose the learning rate? Did you choose the learning rate that was best for HRGN? More information here would allow a reader to better assess the reported results. \n\n- Line 224: \"proves the effectiveness of our method in LM.\" I frankly think this is too strong of a claim. It is well known that performance on wikitext-103 can be sensitive to regularization since the dataset is relatively small (at least in 2023). I think a more accurate and defensible statement might read as these results \"provide evidence HRGN may be an effective method in LM\". I also personally think this weakened claim is sufficient for an interesting paper. \n\n- Line 274: I think the discussion of the \"forget gate values\" of LRU is interesting but could be further developed and discussed. This provides an opportunity to connect the forget gates discussed in this paper with the dynamics state matrices used by S4/S5/LRU etc. In addition, I think the point in line 280 about data-dependent gates (or dynamics matrices) is interesting. It would be relevant to cite Liquid-S4   (https://arxiv.org/abs/2209.12951) here since they propose a method to have input-dependent dynamics with an S4-like approach. This approach also has strong performance on LRA and their results should also be included. \n\n- How is the recurrence actually computed? Are parallel scans used? A parallel scan is mentioned in line 135 when talking about the generic equations before the HGRU is fully introduced. Is this how this is computed? Was the parallel scan implemented in Torch? Also, in line 136, Martin and Cundy should probably be cited here for the parallel scan since they were one of the first to suggest their use for linear RNNs. S5 could also be cited since it was the first to actually make this approach work well for challenging long-range problems. LRU could also be cited since they show an effective method that uses a parallel scan.\n\n- More details should be provided for the speed comparisons in Table 10. Also, how does it scale to longer sequences than 5K, e.g. the 16K of Path-X? How does this compare to other methods that use a parallel scan such as S5 or LRU? Or other diagonal RNNS/SSMs that use convolutions such as H3/S4D/DSS? S4 times are listed here, but I wonder if the optimized Cauchy kernel implementation was used? It is also unclear to me why HGRU would be significantly faster than S4 at inference speed? Both are run as RNNs? Is this due to hyperparameter/state size selection? How was this chosen? More details and comparisons would resolve these questions.\n\nOther questions/suggestions:\n- Line 39: the LSSL (https://arxiv.org/abs/2110.13985) and S4 papers should be cited here since these were the first papers that showed this arrangement of linear layers and nonlinear activations can work well on challenging tasks at scale. \n\n- Line 68: S4D should be cited here since it was the first work to show a significant performance increase  for linear SSM/RNN methods over the original S4 release by using GLU activations. \n\n- Line 72: It might make since to cite S4, S4D, DSS, Liquid S4 (https://arxiv.org/abs/2209.12951) here also since they are all long convolution methods\n\n- Line 85: While [3] and [37] both suggested removing the nonlinearity on the recurrence, it really was not until the LSSL and S4 papers that it was shown you can actually do this without losing performance. These methods should probably be cited here.\n\n- Line 91: S5 also uses a diagonal recurrence matrix and could be cited here\n\n- Line 102: MEGA very explicitly uses an EMA approach and could be cited here\n\n- Line 105: in regards to LRU \"we illustrate that the addition of gating scheme substantially enhances its performance\", but this is basically what is already known from H3. So a good baseline for the Wikitext table would have been to either compare to H3 using S4D, or to compare to LRU plugged into H3. This would allow for a comparison with the other differences between this H3-LRU variant and the HGRU. \n\n- Line 142:  Along with the other cited approaches. DLR and S5 should also be cited here as linear RNNs that uses complex-valued eigenvalues.\n\n- Line 143: I think $g_t$ is introduced here without explanation. Presumably this refers to the output gate described around line 172, but is confusing when it has not been introduced yet\n\n- Line 151: I would not use the phrase \"final formulation\" here, when there is much more formulation to be discussed. This confused me at first because I was trying to figure out how $\\lambda_t$ was determined and had to read on to realize it was not a data-independent learnable parameter.\n\n- Line 156: ON-LSTM is mentioned here without much explanation. Why is this relevant? A bit more explanation here might be helpful. \n\n- Line 174: Has $\\tau$ been introduced? What kind of nonlinearity is this?\n\n- Line 292: typo, should be broader impact and not \"border\" impact\n\n\nIn summary, I like the main ideas of this paper, think it has the potential to be a strong submission and would like to see it accepted. But I do not think it can be accepted in its current form. However, I do think it is feasible to incorporate the suggestions above and I am willing to revise my score upward.\n\nLIMITATIONS:\nLimitations are discussed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper uses the token-mixing interpretation of Transformers to address the slow back-propagation problem of RNNs. First, back-propagation is made one-step per layer, rather than T steps, by using linear recurrence.  In order to avoid dramatically exploding the size of the effective weight matrix in back-propagation, the recurrence is also set to be diagonal, with a channel-mixing layer (GLU) just above each HRU in order to permit mixing of the channels. Per-channel recurrence with real-valued weights can only represent exponential decay, whereas gated recurrence needs to represent delayed recall; delayed recall in the proposed network is represented using complex-valued recurrence.  For simplicity, the forget gate is set to one minus the input gate.  For simplicity, because it gives better results in practice, and because it gives results that can be easily interpreted in terms of the recall periodicity of each channel, the recurrent weights are set independent of the input data, so that only the forget gate and output gate depend on the input data.\n\nSTRENGTHS:\nThis paper seems to provide an architecture that seems to have the full representational power of an LSTM, but with the one-step-per-layer back-propagation speed of a Transformer. \n\nThe Transformer has significant training problems on small data: in order to learn useful attention calculations, it needs to see a pretty large dataset.  The LSTM does not have that problem; the inductive bias of the RNN allows the LSTM to converge with somewhat smaller training datasets.  Nevertheless, the LSTM is rarely used now, because of the T-step back-propagation through each recurrent layer.  If HGRU scales well, it could provide an architecture that supercedes the Transformer for small to medium-sized supervised datasets with no useful self-supervised-pretraining dataset available.\n\nWEAKNESSES:\nThe only weakness is the one noted by the authors: Though this is a convincing proof of concept, it has not yet been scaled to GPT scales.\n\nQUESTIONS:\nThese are not really questions, but there are three typos:\n\np. 4: R^{Hxd} -- H has not been defined, is it the number of layers?\n\nTable 4 caption: layer in independent -> layer is independent\n\nTables 5 and 6 are shown on the same page, but referenced in reverse order; should they be swapped?\n\nLIMITATIONS:\nThe only weakness is the one noted by the authors: Though this is a convincing proof of concept, it has not yet been scaled to GPT scales.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-11"}

comments: SUMMARY:
This paper proposed a novel RNN architecture that utilized several existing components: complex(unitary) matrix, and multi-layer. 

The authors also claim stronger long-term dependency ability of the proposed method on several long context benchmarks.

STRENGTHS:
1. The paper is well organized. Experiments on Wikitext, ImageNet etc are good benchmarks for long-term dependency test. 

2. The discussion on the forgetting gate is interesting and demonstrates the effectiveness of the proposed method on learning long-term dependency.

3. The lower bound design is novel. The authors justified the design with ablation experiments.

WEAKNESSES:
1. Figure 1, the most important diagram, is super confusing. Why do you use a dashed arrow to explain a module?

2. Orthogonal/Unitary matrix design has been long discussed in the past. It has already shown strong long-term dependency ability. This paper failed to mention any of them.

3. Experiment results lack detail. No training or architecture specs of other baseline models. Where do these numbers come from? What does `transformer` on `wikitext-103` mean? What sized transformer? ...

QUESTIONS:
N/A

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-04

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe manuscript introduces a novel recurrent neural network architecture with linear recurrent units. The proposed method improves reasoning about long-term dependencies by imposing a monotonically increasing lower bound for the forget gate values of RNN layers, i.e. higher layers retain information longer. Since the proposed method uses linear recurrent units, the authors also employ complex valued activations to increase the models expressive power. Further design choices include tied input and forget gates and the addition of output gates. For evaluation, the authors considered several benchmarks, including WikiText-103, ImageNet-1k and long-range arena (LRA).\n\nSTRENGTHS:\nThe method presents an intriguing approach to addressing difficulties in modeling long-range dependencies with RNNs. The design choices of the proposed method are well-motivated and the performance is mostly better or on par with other methods.\n\nWEAKNESSES:\n1. The reported metrics are not aggregated over multiple runs and thus don't include estimates of variation over random seeds. This is problematic, since it's not clear whether differences in performance are greater than the variation over multiple runs of the same experiment. HRGN outperforms all other baselines in terms of average performance on the LRA benchmark, but LRU shows higher performance for 4 out of 6 tasks. Here an estimate of the variation over random seeds would definitely be helpful in comparing different methods. The same is true for experiments in the ablation study.\n\n2. The manuscript overall is a bit difficult to follow. Several variables are used without being defined, e.g.:\n   1. Line 143: $g_t$ is mentioned before being defined.\n   2. Eq 3: $\\tau$ is not defined. Is it a typo and should've been $\\sigma$ or is it another activation function?\n   3. Line 191: What is $\\mu$ (without subscript $t$)?\n3. I think Section 3.3 could benefit by providing a bit of context, i.e. what the token mixing perspective is supposed to highlight.\n4. The authors write that the same training configuration, including the learning rate, was used for all models. How were these hyperparameters selected? The optimal learning rate may be different for different architectures. A fair comparison should take this into consideration.\n5. The format of the tables with experimental results is inconsistent. In Table 1, the lowest perplexity is highlighted in bold, which should be separate for val and test, same for the underlined second best. In Table 2, the caption says that bold and underlined highlight the best and second best results per task, but this is not the case. Table 3 does not have highlights.\n\nQUESTIONS:\nOverall, I think the method has a lot of potential, but some of the issues mentioned in the weaknesses section prevent me from recommending a higher score at this point.\n\nI have some questions and minor comments:\n1. Line 170 mentions that $\\theta'$ is fixed to 0. Does the equation $i_t=(1-\\lambda_t)\\exp(i\\theta')$ really correspond to $i_t=1-f_t$ for all layers, given that $\\theta$ in $f_t = \\lambda_t\\exp(i\\theta)$ is different across layers? How did you decide on fixing $\\theta'$ to $0$?\n2. Eq 3: How does the dimensionality get reduced to $\\mathbb{R}^{1\\times d}$?\n3. There's a duplicate entry in the references (36 and 37)\n4. Line 176: The reference in \"Expanding the 2\" does not work, I assume it is supposed to refer to Equation 2. It's best to always add the reference type, e.g. \"Equation 2\", instead of just \"2\".\n5. There are two sentences starting with \"Finally,\" in the paragraph of lines 217-225.\n6. Line 292: \"border\" should be \"broader\"\n7. I think it'd be good to include a brief mention of works on deep RNNs that update at different time scales (e.g. clockwork RNN [1] or HM-RNN [2]) in the related work section.\n8. Figure 1 and other places: Sometimes the hidden unit uses the acronym HRGU, sometimes HGRU, the latter of which I think is the correct one. Consider using a tex package for automating the use of acronyms, e.g. glossaries (see https://www.overleaf.com/learn/latex/Glossaries#Acronyms).\n\n## References\n1. Koutnik, Jan, et al. \"A clockwork rnn.\" International conference on machine learning. PMLR, 2014.\n2. Chung, Junyoung, Sungjin Ahn, and Yoshua Bengio. \"Hierarchical multiscale recurrent neural networks.\" arXiv preprint arXiv:1609.01704 (2016).\n\n\n## Acknowledgement of rebuttal\nI have read the rebuttal, the other reviews and engaged actively in discussion with the authors. The authors have addressed several of my concerns, including the addition of aggregated results with an estimate of the variance over random seeds. As a result, I have decided to increase the score.\n\nLIMITATIONS:\nAs limitation, the authors only mention that large-scale testing has not been performed. I would have liked to see a discussion of limitations of the methodology the authors see or anticipate e.g. when scaling the method.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}