Hierarchically Gated Recurrent Neural Network for Sequence Modeling
Authors: Zhen Qin
Songlin Yang
Yiran Zhong
Venue: NeurIPS 2023 spotlight (2023)
Keywords: RNN
Sequence Modeling
NLP
TL;DR: Hierarchically Gated Recurrent Neural Network for Sequence Modeling

Abstract: Transformers have surpassed RNNs in popularity due to their superior abilities in parallel training and long-term dependency modeling.
Recently, there has been a renewed interest in using linear RNNs for efficient sequence modeling.
These linear RNNs often employ gating mechanisms in the output of the linear recurrence layer while ignoring the significance of using forget gates within the recurrence. In this paper, we propose a gated linear RNN model dubbed Hierarchically Gated Recurrent Neural Network (HGRN), which includes forget gates that are lower bounded by a learnable value. The lower bound increases monotonically when moving up layers. This allows the upper layers to model long-term dependencies and the lower layers to model more local, short-term dependencies. Experiments on language modeling, image classification, and long-range arena benchmarks showcase the efficiency and effectiveness of our proposed model. The source code is available at https://github.com/OpenNLPLab/HGRN.

Hierarchically Gated Recurrent Neural Network for Sequence Modeling
Zhen Qin', Songlin Yang', Yiran Zhong
POpenNLPLab, Shanghai Artificial Intelligence Laboratory, "MIT CSAIL https: /tt///gtttt.tt/..//// com/ OpenNL.LLab/HCRR
Abstract
Transformers have surpassed RNNs in popularity due to their superior abilities parallel training and long-term dependency modeling. Recently, there has been a renewed interest in using linear RNNs for efficient sequence modeling. These linear RNNs often employ gating mechanisms in the output of the linear recurrence layer while ignoring the significance of using forget gates within the recurrence. In this paper, propose gated linear RNN model dubbed Hierarchically Gated Recurrent Neural Network (HGRN), which includes forget gates that are lower bounded by learnable value. The lower bound increases monotonically when moving up layers. This allows the upper layers to model long-term dependencies and the lower layers model more local, short-term dependencies. Experiments on language modeling, image classification, and long-range arena benchmarks showcase the efficiency and effectiveness of our proposed model. The source code is available at https:/ggihbb.coomppeeniPaalllle
Introduction
Sequence modeling is fundamental problem in various domains such as natural language pro- cessing [12
44 61 64), time series analysis [84], computer vision R [3 45 74], and audio processing L 73 Prior to the invention of Transformers [8]] RNN and its variants were the
43
primary selections of architectures for sequence modeling, and have been widely used in machine translation (6), stock price prediction [68]. weather forecasting (65) speech recognition (51), and etc.
RNNs have two main drawbacks: slow sequential training and limited capability in modeling long- term dependencies. With the swift development of deep learning and the pervasive use of GPUs, these drawbacks prevent it from flourishing in modern long-sequence modeling tasks. Meanwhile, Transformers [81] have rapidly gained popularity and now dominate various research areas in sequence modeling due their better abilities in parallel training and long-term dependency modeling. However, Transformer's 's quadratic time complexity makes long sequence modeling expensive. On the other hand, RNN offers linear complexity and serves as an ideal choice for long sequence modeling. This works aim to addressing these RNN drawbacks, revitalizing their applicability in long-sequence modeling tasks.
To address the training inefficiency problem, we turn to more efficient RNN variants that employ
element-wise linear recurrence (ELR) relations 48 ELR provides two main advantages: (i)
By removing nonlinearities in the recurrence, it enables parallelized training. (ii) By assuming independence between distinct hidden states, enables efficient hidden state updates (through element-wist product instead of matrix multiplication) [20 Notably, ELR has been used in many modern linear RNN models, including the diagonalized versions structured state-space models and RWKV [55]. In recent advancements, numerous studies have incorporated
21 (S4) 20 26 71
'Equal contribution.
Indicates corresponding author (Email address: zhongyiran @ gmail.com).
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

gating mechanisms into the outputs of linear recurrence layers [1] 46 49 55 82,, similar to the output gates LSTMs and leading considerable performance gains. However, most current studies overlook the significance of the forget gate, which is often regarded as the most important gate in LSTMs [19 [0]. In this work, we underscore the importance of employing forget gates in linear RNNs and adopt gated linear RNNs for both efficiency and high performance.
To effectively capture long-term dependencies in gated RNNs, it is crucial maintain high forget gate values close to one [23] However, gates in saturated regimes (i.e., close to zero or one) suffer from the gradient vanishing issue 2 Moreover, if all forget gate values are close to one, RNNs will not be able to effectively forget irrelevant information, compromising their ability to model short-term dependencies. To address these challenges, we introduce Hierarchically Gated Recurrent Units(HGRU) In HGRU, we add an additive learnable value, referred to as the lower bound, to
the original forget gate value, effectively mitigating the issue of saturated gates [23] by pushing gate activations away from the saturated regimes. Furthermore, we design the lower bounds to increase monotonically as we move up the layers of the RNN. This ensures that the forget gate values in the lower layers remain relatively small, enabling the necessary forgetting of past information for modeling short-term dependencies. In contrast, in the uppermost layer, the forget gate values approach one, facilitating the effective modeling of long-term dependencies. Our proposed model has proven to be highly efficient and effective, as demonstrated by its outstanding performance in language modeling, image classification, and long-range arena benchmarks.
2
Related work
3 abstracts self-attention (SA) as token mixing, thereby transforming the Transformer architecture into MetaFormer. MetaFormer comprises essential components such as token mixer, channel mixer, residual connections, and LayerNorm. This abstraction highlights that the success of Transformers does not solely rely on SA but rather on the holistic integration of these components. Notably, token mixers can be replaced with simpler alternatives like pooling layers without compromising the model's performance in the context of vision transformer. For sequence modeling tasks. (29) provides a comprehensive analysis and discussion of different token mixing strategies. Two prominent contenders, long convolution and linear recurrence, show promise as replacements for SA modules long sequence modeling due to their superior asymptotic time complexity and competitive performances. In long convolution models [14 57 59). the kernel size matches the input sequence length, enabling a broader context compared to traditional convolutions. Training is accomplished using the efficient O(n log fast Fourier transforms (FFT) algorithm. However, long convolutions face challenges such as the need for causal convolution inference, which requires caching all historical computations similar to the key-value (KV) cache in SA. This can lead to memory limitations when processing long sequences. Moreover, the inference complexity of long convolutions remains higher than that of RNNs. These factors make linear RNNs a more suitable alternative to replace SA in long-sequence modeling. TransNormerLLM 61]] scales efficient token mixing in large language models to achieve competitive performance and superior training and inference efficiency compared to transformer- based models.
Efficient token mixing for sequence modeling.
Element-wise linear recurrence. The slower training speeds of traditional RNNs can be attributed to two main factors: (i) The updating of the hidden state involves full matrix multiplication. (ii) The presence of nonlinearity within the recurrence prevents parallel computation. To tackle the first issue, 440 introduced a simplified interaction between hidden states. This allowed the hidden state update to be performed using an element-wise product instead of full matrix multiplication. They demonstrated that this approach notably fast when the (nonlinear) recurrence for each dimension is fused within a single CUDA kernel. Likewise, for the linear case, diagonalized versions of S4 20 26 have also exhibited speed improvements over S4 by leveraging element-wiss recurrence. Regarding the second challenge, the ability to capture nonlinear dependencies on past data can be achieved by stacking multiple linear recurrence layers interleaved with nonlinear MLP blocks. This
indicates the potential to eliminate nonlinearity, as suggested by 4 25 48) Empirical support for
this strategy's 's effectiveness came later, as demonstrated by [1] 20 24 53
55 71.
52 further
highlighted that such an architecture still possesses Universal Approximator properties, thus justifying
the employment of linear recurrence. By excluding nonlinearity, [8 showed that the parallel scan algorithm can be used for parallel training.
2

---

Linear recurrence can be broadly categorized into exponential moving averages (EMA) and gating schemes, as noted by 48 The key difference is whether the decay rate is data-dependent. Models such as $4 [21], S4D 20
MEGA [46]. RWKV (55. and LRU 5 utilize the EMA approach, where the decay rate is static for all time steps (i.e., data-independent), while our model uses a data- dependent dynamic decay rate through the use of the forget gate. We remark on the importance of incorporating a data-dependent decay rate, which is largely ignored by current works in linear RNNs. Although liquid S4 [28] uses a dynamic transition matrix (which amounts to data-dependent decay rate), it employs a limited form for FFT-based training. Our model does not have the convolutional view and thus cannot use FFT for training but allows the use of parallel scan.
The field of linear Transformers and linear RNNs exhibits close relationship. [3] shows that linear Transformers can be reformulated as RNNs during auto-regressive decoding, revealing similarities
to the update rules observed fast weight additive outer products [66] 67 These updates can be seen as special case of element-wise linear recurrence, where forget gate values are consistently set to one across time and hidden states are two-dimensional. However, this formulation in linear Transformers lacks the ability to forget irrelevant information, resulting in the attention dilution issue 660. To address this limitation, [66] introduced the delta rule to forget values associated with the current write key by removing the corresponding value before adding the new value. Alternatively, [7|56] 56] proposed gating mechanisms similar to those in gated RNNs to facilitate the forgetting of irrelevant information.
Long-term dependencies in RNNs. RNNs fall short in long-term dependency modeling, which is commonly attributed to the gradient vanishing issue. Three methods are typically applied to mitigate this issue. (i) Gating mechanisms 9 17
70, which are believed to crucial to the success of LSTMs, use additive (instead of multiplicative) hidden state update rules to improve gradient flow. (ii) Regularizing or initializing the eigenvalues of the recurrent weight matrix (close) one via identity matrices (88 unitary matrices (2). In the diagonal linear RNN case, the eigenvalues coincide with the element-wise decay rates, and LRU [53 uses randomized linear algebra techniques to initialize eigenvalues to be close to one. [53] also interestingly points out that many modern state-space models use a very small time step value on initialization for discretization, resulting in eigenvalues or decay rates close to one. (iii) Adding skip connections between distant time steps to allow shortcuts for gradient flow [5] S 1ur approach combines (i) and (ii), which improves gating mechanisms with regularized dynamic decay rate that approaches one in the upper layer.
3 Method
3.1
Architecture overview
Our proposed Hierarchically Gated Recurrent Network (HGRN) is depicted in Figure | It has multiple stacked layers, each of which consists of token mixing module HGRU and channel mixing module GLU (Gated Linear Unit [69].
3.2 HGRU exploration
We begin with simple gated linear recurrent layer, which defined as:
Algorithm 1 Recurrent Computing
fr = Sigmoid (x, w + by) E Iixd
1: Input: C₁ E Clxd Me. Me, 2
Rixd I I,..., n,kk l,.... H.
i, Sigmoid (x, W; + b.) E gixd
2: Init: h = 0 E cixd HE Caxd.
C₁ SiLU (x, W, b²) E Rlxd
(1)
3: for t ==rt=l 1 to n do
4: begin
h. = f, @ h i, ① c, e R
5:
A = , + (1 7*) ) me.
ho = 0 E R
6:
h X exp((i)) + X.).
7:
[H] = h.
where @ denotes the element-win wise product. Following the terminology used in the RNN literature, we refer to f, and is worth
8:
end
9: return H.
i, as the forget and input gates, respectively. noting that f, and i, depend only on x, and not on h,-1. This characteristic enables the use of the parallel sean algorithm We then make the following changes toward our final HGRU step by step.
48 71. otherwise it infeasible.
3

---

Output
Output
HGRN
HRU
LayerNorm
Recurrent Computing
NX
GLU
F2
SILU
SILU
Sigmold
HGRU
Linear
Input
Input
Input
Figure 1: Illustration of the neural architecture. Each HGRN layer consists of token mixer
HGRU and channel mixer GLU. HGRU employs linear recurrence in the complex domain: h, = Xa @ exp(i0) @ ht-1 + (1 Xe) @ ct. Here e is the input vector, 0 is the rotation angle, fr is the output of the original forget gate, 7tt is the lower bound of the kth layer, À is the resulting data dependent decay rate: X 7
**) 1rc.
Complex-valued recurrence. For linear RNNs with static decay rates, it is common to perform oigendecompositions on the recurrent weight matrix achieve element-wise linear recurrence. However, if only real-valued eigenvalues are allowed, it restricts the range of the recurrent weight matrix to be symmetric, limiting the model's expressiveness. To overcome this limitation, linear RNNs often employ complex-valued eigenvalues to enhance the model's 's expressive power [20] 26 27 32 53]. Motivated by this, we extend our model to consider h, it, C₂ E Clxd as complex values. For the input Ct, parameterize its real and imaginary parts separately as follows:
Ro(c,) = SiLU (x, W + bor) E Rixd Im(c,) = SiLU (x, W ai + bai) R
Regarding the forget gate values, find it convenient to the exponential representation of complex numbers and parameterize f, as follows: f, = Xe @ exp(if;). Here, 12 = -1, A,, 8 E R and exp(i0,) = cos 0g sin 0zi. The magnitude argument determines the intensity of remembering historical information, while the phase argument 0g determines the oscillation frequencies. We find that parameterizing 0, in data-independent manner preferable, it allows for clear interpretation of encoding relative position information (see next subsection for more discussions). which is reminiscent of Rotary Positional Embedding (RoPE) 772]. We shared 0 arcoss times steps, i.e.,
f, = a₁ @ exp(it), initialize as RoPE does, but make it learnable like LRPE [63
Lower bound on forget gate values. Since the intensity of remembering information is only related the magnitude argument A₁, we focus on how to add a lower bound to X₁- As mentioned earlier, we want to set monotonically increasing lower bound on the forget gate (magnitude) values. Inspired by ON-LSTM 770,, we employ cummax activation function to achieve this. Concretely, we allocate T E # to parameterize lower bounds independently for all hidden states, where H is the number of layer. Assuming the layer index is k; we have following calculations:
P (Softmax(T dim 0) E #xd
k [Cumsum (P, dim 0)], E I
Here we define (Cumsum(x)) = EE m;) 21 to prevent the highest layer's lower bound from being still want the ability forget irrelevant information.
We remark that there a difference in the use of cummax between our model and ON-LSTM. In ON-LSTM, cummax is applied to the hidden state dimension within single layer, while in our case, we apply cummax on the layer dimension across different layers enable upper layers to model rangerange dependencies.

---

Finally, X in the k-th layer is parameterized as follows:
A Sigmoid (x, W + bn) ERR
2 =
Comparing to before (i.e., without lower bounds), to achieve the same forget rate value y closed to one, Pe will be pushed away from the Sigmoid activation function's saturated regions (i.e., near one),
14 1
7
thereby mitigating the gradient vanishing issue
23 and making gradient-based optimization easier.
Tying input and forget gates. To reduce the number of parameters, it common i.e., lying the input gate with the forget gate using i, - - f.. which has a close relationship to the discretization of continuous- time system 77 and exponential moving average [33], and has been
leaky units,
proven effective empirically [9]. To allows for clear interpretation of encoding relative position information, we only upply this strategy on the magnitude argument:
h, de a exp(ii) h + (1 X,) & c2 E Cl
Output gates and projection. The addition of gates to the output of the recurrence layer has been shown to be effective state-space models [I] 46 49 82). Motivated by this, we incorporate an output gate before performing the output projection as follows and get HGRU:
g: Sigmoid ( + bg) E 1x2d
o LayerNormi ge + [Re(h:), Im(h;))) E gx2d
(3)
O, of W. + b. E ER lxd
3.3 Token mixing perspective of HGRU
We provide the token mixing perspective of HGRU similar to [32].
Expanding Equation 2 we have:
Σ (1- Xa)
II L
A exp(ii)
c, Σ (1
Xx)
II A
exp(i(l s)0)c, (4)
Jms+1
Written in matrix form, we have:
h
1- A1
0
(1 1- exp(ii)
1- A2
A
C =
(11 A1) ITT Ad exp(i(n 1)0)
(5)
So the token mixing module can formed as follows:
= AC.
(6)
Note that the token mixing matrix A can be decomposed into two parts A =
A O
O
A
0
(1-b1)>2
I X2
exp(if)
1
A
(7)
0
((1 - A1) IT As]
1
Lexp(i(n 1)0)
This decomposition means that the Token mixing matrix A can be decoupled into two independent modules, where models the long-dis dintadistance dependency and θ, Toeplitz matrix, models the relative positional relationship and enhanced expressiveness. Note that if @ depends on the input, then the matrix A will no longer Toeplitz matrix, thus unable to capture relative position information. can be also viewed as RoPE-enhanced attention mechanism: A corresponds to the attention matrix but the attention score here is the cumulative product of data-dependent decay rates; θ directly corresponds to RoPE.
5

---

Table I: Results on Wikitext-103 (TN 's setting). means lower is better.
PPL (val))
PPL
Params (M)
Model
(test)
Attn-bused
Transformer 81
24.40
24.78
44.65
FLASH 10]
25.92
26.70
42.17
1+elu 35
27.44
28.05
44.65
Performer
62.50
63.16
44.65
cosFormer 62
26.53
27.06
44.65
MLP-baser
Syn(D) 76
31.31
32.43
46.75
Syn(R) 76
33.68
34.78
44.65
gMLP[42) [22
28.08
29.13
47.83
[22
38.34
39.66
45.69
DSS 26
39.39
41.07
45.73
GSS 49
29.61
30.74
43.84
RWKV [55
24.31
25.07
46.23
LRU [53]
29.86
31.12
46.24
FFT-hased
TNN [59]
23.98
24.67
48.68
Ours
HGRN
24.14
24.82
46.25
Experiments
We conduct comparative analysis between proposed HGRN and four widely adopted sequence modeling structures, i.e. attention-based, MLP-based, FFT-based, and state-space-based. We evaluate HGRN on the WikiText-103 dataset [50] and the Pile [I5] dataset for autoregressive language modeling, as well as the length extrapolation ability. To assess the accuracy and efficiency of our model in handling long-term dependencies, utilize the LRA benchmark (78). Additionally, showcase the robustness of HGRN in computer vision task on the ImageNet- 1k dataset.
4.1 Setting
We implement our models in Pytorch [54] and train them on 8 Nvidia A100 GPUs. For HGRN, we found that fusing element-wise recurrence into a single CUDA kernel results in fast running speed in practice. [48] also found that unless the sequence length is sufficiently large, the parallel scan's implementation not necessarily faster than the sequential scan. such, we use a CUDA- based sequential scan for implementation; however, sequences through the use a parallel scan.
model has the potential to model very long
We adopt the same training configuration for all competitors, including batch size, learning rate, training epochs or iterations, etc. We list detailed hyper-parameters the Appendix. For the autoregressive language modeling, we conducted three sets of experiments. Firstly, we validated the performance of two different-scale models on the Wikitext- -03 dataset. We used the TNN configuration verify the performance of the model at around 44m, and the Hyena configuration to verify the performance of the model at around 125m. To evaluate the performance of larger-scale models, we trained a 1b Transformer and HGRN on the Pile dataset using 10b tokens. To assess the performance in downstream tasks, we trained HGRN models of 150m, 350m, and 1b on the Pile dataset using 100b tokens and conducted zero-shot evaluations downstream tasks.
For the LRA benchmark, We report results on all 6 tasks. For the image classification on the ImageNet-Ie 1k dataset, We integrate HGRN into the DeiT [79 structure, we replace the transformer layers with our HGRN modules. is compared to the performance of the vanilla DeiT on the ImageNet- IK dataset for image classification.
4.2 Results

---

Table Performance Comparison on Commonsense Reasoning. PS: parameter size (billion).
tokens (billion). HS: HellaSwag. WG: WinoGrande.
Model
Params
Token
BOOLQ
PIQA
HS
WG
ARC-e
ARC-c
OBQA
AVG
GPT-Neo
0.13
300
61.71
63.06
30.40
50.43
43.73
23.12
26.20
42.66
OPT
0.16
300
55,47
62.95
31.35
50.43
43.52
22.70
28.00
42.06
Pythia
0.16
300
55.08
61.32
30.16
51.93
43.18
23.12
26.80
41.66
RWKV
0.17
65.07
32.26
50.83
47.47
24.15
29.60
41.56
HGRN
0.15
100
59,91
65.02
33.33
50.20
46.68
23.81
28.60
43.94
OPT
0.35
300
57.74
64.58
36.69
52,49
44.02
23.89
28.20
43.94
Pythia
0.4
300
60.40
67.08
40.52
$3.59
51.81
24.15
29.40
46.71
BLOOM
0.56
55.14
64.09
36.97
52.80
47.35
23.98
28.20
44.08
RWKV
0.43
67.52
40.90
51.14
52.86
25.17
32.40
45.00
HGRN
0.35
100
59.05
66.70
38.12
51.70
49.20
25.26
30.60
45.80
GPT-Neo
1.3
300
61.99
71.11
48.93
54.93
56.19
25.85
33.60
50.37
OPT
1.3
57.77
71.71
53.70
59.35
57.24
29.69
33.20
51.81
Pythia
1.4
300
60.73
70.67
47.18
53.51
56.99
26.88
31.40
49.62
BLOOM
1.1
350
59.08
67.14
42.98
54.93
51.47
25.68
29.40
47.24
RWKV
1.5
72.36
52.48
54.62
60.48
29.44
34.00
50.56
HGRN
100
58.69
70.89
48.02
51.62
55.64
27.90
31.60
49.19
Autoregressive Language Modeling Autore- gressive language modeling stands as a promi- nent task within the field of natural language processing, as it serves as a measure of a lan- guage model's causal inference capability. This task requires the model to estimate the probabil- ity distribution of the subsequent token based on the previously seen tokens.
Table 2: Results on Wikitext-103 (Hyena(57] setting). All models are in GPT-2 small size (125M). means lower better
Model Transtormer [57
PPL.
18.6
Hybrid H3
[57
18.5
Performer [57
26.8
Reformer 57
25.6
We show the performances of the autoregressive language modeling in table [ and [ Com- pared to transformer-basee methods, HGRN performs favourably than most efficient variants of the vanilla transformer such as FLASH 31
AFT-conv [57
28.2
Linear Attention [57
25.6
Hyena [57
18.6
Hyena-slim 57
18.5
HGRN
18.6
1+elu [35), Performer 7 and cosFormer [62]
Also, HGRN achieves better results than the MLP- methods with notable margin. Never-
theless, HGRN performs similarly to the original transformer [8I]. Finally, HGRN shares similar
concepts with RNN-base such S4 [22]. DSS (66. GSS (49) RWKV (55), and LRU (53),
HGRN also achieves superior performance to all RNN-based methods. This provide evidence HRGN may be effective method in LM We also report the extrapolation ability of HGRN compared to previous methods in Table [4]
We also trained 1b model on the Pile dataset and compared with LRU and Transformer.
Table 3: Results on the Pile. All the model size is 1b. The lower the better.
Specifically, our training parameters included sequence length of 1024, batch size of 96, 100k updates, and learning rate of 5e-4. It can be seen that HGRN still performs better at the lb scale. Additionally,
Model
PPL.
Transformer
4.56
LRU
5.07
HGRN
4.14
trained 100b tokens of HGRN on the Pile dataset at 150m, 350m, and 1b sizes, and evaluated them against open-source Transformer-basee models in downstream tasks. We selected Comparison Commonsense Rea- soning and Super GLUE tasks, and all evaluations were done using the Im-evaluation--haree [16]. HGRN achieves comparable performance Transformer-based models when consuming only 1/3 of the tokens.
Long Range Arena LRA is proposed as comprehensive evaluation for assessing the perfor- mances of models in processing long-term dependencies in various sequential modeling tasks. We show performance comparison between HGRN and existing methods in Table HGRN achieves comparable results with other SOTA methods.

---

Table 5: Performance Comparison on Super GLUE.. PS: parameter size (billion). T: tokens
(billion).
Model
Params
Token
WSC
WIC
RTE
CB
MULTIRC
BOOLQ
COPA
AVG
GPT-Neo
0.13
300
36.54
50.00
54.87
41.07
0.84
61.71
64.00
44.15
OPT
0.16
300
36.54
50.00
49.82
21.43
1.36
55.47
66.00
40.09
Pythia
0.16
300
36.54
50.16
52.71
41.07
2.52
55.08
65.00
43.30
HGRN
0.15
100
38.46
51.10
56.68
42.86
1.47
59.91
65.00
45.07
OPT
0.35
300
36.54
50.00
51.99
46.43
1.36
57.74
72.00
45.15
Pythia
0.4
300
57.69
50.31
52.71
35.71
1.68
60.40
70.00
46.93
BLOOM
0.56
350
40.38
50.00
52.71
41.07
1.05
55.14
61.00
43.05
HGRN
0.35
100
38.46
50.16
52.71
51.79
1.99
59.05
73.00
46.74
GPT-Neo
1.3
300
36.54
50.00
60.29
44.64
1.99
61.99
69.00
46.35
OPT
1.3
300
37.50
51.10
51.99
41.07
3.15
57.77
79.00
45.94
Pythia
1.4
300
36.54
50.00
53.07
35.71
0.94
60.73
72.00
44.14
BLOOM
1.1
350
36.54
50.00
52.71
41.07
0.73
59.08
68.00
44.02
HGRN
100
40.38
50.78
53.43
42.86
3.04
58.69
70.00
45.60
Table 6: Performances Comparison on the Long Range Arena benchmark. The proposed HGRN
achieves the best performances and outperforms all competing methods.
Model
ListOps
Text
Retrieval
Image
Pathfinder
Path-X
AVG.
Transformer [81]
38.37
61.95
80.69
40.57
65,26
47.81
cosFormer 662
36.50
67.70
83.15
51.23
71.96
51.76
FLASH 31
38.70
64.10
86.10
47.40
70.25
51.09
S4
22
59.60
86.82
90.90
88.65
94.20
96.35
86.09
DSS softmax
26
60.60
84.80
87.80
85.70
84.60
87.80
81.88
DSSEXP 266
59.70
84.60
87.60
84.90
84.70
85.60
81.18
DSSEXP-NO-SCAAN
26
59.30
82.40
86.00
81.20
81.30
65.03
TNN (59
61.04
87.90
90.97
88.24
93.00
96.10
86.21
S5 771
62.15
89.31
91.4
88
95.33
98.56
87.46
Mega 46
63.14
90.43
91.25
90.44
96.01
97.98
88.21
SGConv 41
61.45
89.2
91.11
87.97
95.46
97.83
87.17
LRU [53]
60.20
89.40
89.90
89.00
95.10
94.20
86.30
HGRN
59.95
18.14
94.23
88.69
92.92
97.50
86.91
Image Classification The image classification results on the ImageNet- 1K dataset are presented in Table 7 Notably, with comparable parameter sizes, our proposed HGRN model demonstrates superior performance compared to previous methods such as TNN and the vanilla transformer. demonstrates the capability of HGRN in modeling visual modalities.
Table 7: Performances comparison of image classification on ImageNet-lk. HGRN performs
favorably than competing methods with similar parameter sizes.
DeiT-Time
DeiT-Small
Model
Topl Acc
Param (M)
Topl Acc
Parma (M)
Deit
72.20
5.7
79.90
22.0
TNN
72.29
6.4
79.20
23.4
HGRN
74.40
6.1
80.09
23.7
Table 8: Forget gate ablation on an autoregressive language model. The only lower bound means using a iata-inddeendent gate like LRU.
4.3
Ablation Study
We conducted ablation studies in the smallest- scaled setting (i.e., TNN[59]s 's setting on Wiki-
Model
PPL
Text103 dataset) to thoroughly verify the effec-
w forget gate
4.92
tiveness of each of our proposed components in HGRN. Experiments were conducted on the
LRU
5.07
HGRN only lower bound
4.84
Pile dataset using lb model with 10b tokens for the forget gate experiment.
HGRN w/o forget gate
57.42
HGRN
4.14

---

Figure 2: Visualization of forget rates. We plot the forget rates of layers 5 and 6 model trained on language modeling tasks.
(ii) Without lower bound
(iii) LRU
(i) With lower bound
Layer
Layer 5
Layer 5
17.5
15.0
Rercent
Percent
8
Percent
6
2.5
0.0
0.4
0.5
0.6
0.7 0.8
0.9
1.0
0.2
0.4
0.6
0.8
1.0
0.6
0.7
0.8
0.9
1.0
Forget rate
Forget rate
Forget rate
Layer
Layer
Layer 6
50
8-
Percent
40
Percent
Percent
0.4 0.5
0.6
0.7 0.8
0.9
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.6
0.7
0.9
1.0
Forget rate
Forget rate
Forget rate
The influence of forget gate In table we demonstrate the role of forget gate. From table g we observe that removing the forget gate significantly decreases the performance HGRN, while adding a forget gate to LRU improves performance. On the other hand, using inta-inndeendeet forget gate (only lower bound) leads lower performance compared to data-dependent forget gate.
The influence of input gate and output gate Table. 1 validates the effectiveness of using out- put gates and tying input and forget gates. w/o input gate means to remove the - Ae term. w/o output gate means remove the left branch of HGRN in figure I Our design achieves the best performance.
Table 9: Ablations of gates on autoregressive lan- guage modeling. w/o input gate means remove the 1 X term. w/o out gate means remove the left branch HGRU in figure
Model
PPL.
w/o input gate
25.03
w/o output gate
25.50
HGRN
24.14
The influence of lower bounds in forget gate values We demonstrate the effectiveness of introducing a lower bound in Table I0 and Table From Table I0, observe that gating (i.e., without lower bound) is more critical than the lower bound (i.e., only lower bound). Combining gating and the lower bound consistently provides benefits, but the most significant improvement arises from the monotonically increasing lower bound. This aligns with the intuition that lower layers should primarily focus on nearby tokens, while upper layers should attend more broadly to capture long-tern dependencies [58]
Table 13 highlights the essential role the lower bound in long sequence processing tasks. In these tasks, the model's performance is no- tably poor and sometimes fails to converge with- out the lower bound. It is worth noting that lan- guage modeling tasks do not require extensive long-term dependencies, which explains why the model performs well even without the lower bound. However, in the task of LRA, the ability capture long-term dependencies is crucial for achieving satisfactory performance.
Table 10: Lower bound ablation autoregres- sive language modeling. A random lower bound means the lower bound in each layer is indepen- dent. Decrease lower bound means the lower bound is monotonically decreasing with respect to layer k. only the lower bound means the forget rate is independent of input.
Model
PPL
w/o lower bound
24.71
random lower bound
24.60
decrease lower bound
24.63
only lower bound
27.70
The influence complex-valued recurrence Table II validates the utility of incorporating
HGRN
24.14

---

complex values in element-wist linear recurrence. Additionally, the experiments show that the phase argument & should not be data-dependent.
4.4 Analysis on forget gate values
We present the distributions of forget gate values across layers for different methods in Table 12 and visualize the histogram each layer rig- the autoregressive language
Table 11: Ablations of complex-valued recur- rence on autoregressive language modeling. w/o complex means remove theta, data-dependent theta means theta is dependent on the input, this makes
ure 2 trained modeling task. The results demonstrate that the addition of lower bounds effectively increases the average forget gate values in higher layers (5-6). Notably, the medium forget gate values in the highest layer reach 0.98, enabling the mod- cling of long-term dependencies.
Toeplitz matrix, which can not capture relative information.
the matrix
Model
PPL.L
w/o complex
25.34
data dependent 0
28.74
HGRN
24.14
interesting to note that the average forget
gate values of the LRU model consistently exceed those of our variant model without lower bounds, as per their eigenvalues. However, despite this, the language modeling performance of LRU lower than that of our variant. Specifically. LRU scored 24.71, while variant scored 31.12. This suggests that using data-dependent gates to selectively retain relevant information is advantageous, rather than relying on data-independent forget gate values across all time steps.
Table 12: Forget gate values of different methods on language modeling tasks. In each layer,
counted the mean and median of forget gate values.
ours
ours
w/o lower bound
w/o lower bound median
LRU
LRU
Layer
mean
median
mean
mean
median
0.48
0.47
0.52
0.50
0.75
0.72
2
0.55
0.52
0.59
0.55
0.78
0.75
3
0.60
0.57
0.58
0.56
0.78
0.76
4 0.68
0.64
0.58
0.55
0.79
0.78
5
0.79
0,80
0.63
0.63
0.79
0.77
6
0.91
0.98
0.63
0.67
0.79
0.79
Table 13: Lower bound ablation on LRA. We verify the importance of lower bounds long- sequence modeling capabilities.
Model
ListOps
Text
Retrieval
Image
Pathfinder
Path-X
AVG
w/o lower bound
51.41
87.79
88.71
80.17
51.53
HGRN
59.95
88.14
94.23
88.69
92.92
97.50
86.91
Conclusion
In this work, we have shown that gated linear RNNs could obtain impressive performance across different tasks and modalities without compromising efficiency. We highlighted the significance of the forget gate for linear RNNs in language modeling and emphasized the importance of an additive lower bound on forget gate values for modeling long-term dependencies.
Acknowledgement
This work is partially supported by the National Key R&D Program of China (NO.2027ZD0116100))
Limitations and broader impact
Our empirical evaluation of HGRN remains on a smaller scale compared to other large-scale models. Potentially negative social consequences include the misuse of brain models unsuitable purposes or applications, which must be prohibited by appropriate rules. In the era of large language models, the inference cost is the key limitation of transformer-hasee models. RNNs provide a solution with their lower inference costs. This could potentially lead to significant evolution in the field.
10

---

References
[1] Hassan Akbari, Liangzhe Yuan, Rui Qian. Wei-Hong Chuang, Shih-Fu Chang, Yin Cui, and Boqing Gong. Vatt: Transformers for multimodal self-supervised learning from raw video, audio and text. arXiv preprint arXiv:2104. 1.4./1/75 178, 2021.
[2] Martín Arjovsky, Amar Shah, and Yoshua Bengio. Unitary evolution recurrent neural net- works. In Maria-Florina Balean and Kilian Q. Weinherger, editors, Proceedings of the 33nd International Conference on Machine Learning, ICMI. 2016, New York City, NY, USA, June 19.24, 2016, volume 48 of JMLR Workshop and Conference Proceedings, pages 1120-1128. JMLR.org. 2016.
[3] Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučić, and Cordelia Schmid. Vivit: A video vision transformer. In Proceedings of the IEEE/CVF International
Conference Computer Vision, pages 6836-6846, 2021.
[4] David Balduzzi and Muhammad Ghifary. Strongly-typed recurrent neural networks. In Maria- Florina Balcan and Kilian Q. Weinberger, editors, Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016, volume 48 of JMLR Workshop and Conference Proceedings, pages 1292-1300. JMLR.org.
2016.
[5] Shiyu Chang, Yang Zhang, Wei Han, Mo Yu, Xiaoxiao Guo, Wei Tan, Xiaodong Cui, Michael Witbrock, Mark A. Hasegawa-Johnson and Thomas S. Huang. Dilated recurrent neural networks. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 77-87, 2017.
[6] Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares. Holger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder- decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1724 1734, Doha, Qatar, Oct. 2014, Association for Computational Linguistics.
[7] Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers. arXiv preprini arXiv:2009. 14794, 2020.
[8] Junyoung Chung, Sungjin Ahn, and Yoshua Bengio. Hierarchical multiscale recurrent neural networks. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings, OpenReview.net, 2017.
[9] Junyoung Chung, Çaglar Gülgehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation
of gated recurrent neural networks on sequence modeling. CoRR, abs/1442.35555 2014. [10] Tn Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: Fast and memory-efficient exact attention with I0 awarreessss. In Advances in Neural Information Processing Systems, 2022.
[11] Tri Dao, Daniel Y. Fu, Khaled Kamal Saab, Armin W. Thomas, Atri Rudra, and Christopher Ré. Hungry hungry hippos: Towards language modeling with state space models. CoRR, 111/211..44522 2022.
[12] Jacob Devlin, Ming-Wer Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Confer- ence of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume I (Long and Short Papers), pages 4171 Minneapolis, Minnesota, June 2019, Association for Computational Linguistics.
[13] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint
arXiv:2010.1 1999, 2020.
[14] Daniel Y. Fu, Elliot L. Epstein, Eric Nguyen, Armin W. Thomas, Michael Zhang, Tri Dao, Atri Rudra, and Christopher Ré. Simple hardware-efficient long convolutions for sequence modeling. CoRR, abs/2302.00 06646, 2023.
[15] Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang. Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The Pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.000 100.00077 2020.
[16) Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric
11

---

Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation. Zenodo, Sept. 2021.
[17] Felix A. Gers, Jürgen Schmidhuber, and Fred A. Cummins. Learning to forget: Continual prediction with LSTM. Neural Comput., 12(
2000.
[18]
Yuan Gong, Yu-An Chung, and James Glass, AST: Audio Spectrogram Transformer. In Proc. Interspeech 2021, pages 571-575, 2021.
[19] Klaus Greff, Rupesh Kumar Srivastava, Jan Koutník, Bas R. Steunebrink, and Jürgen Schmid- huber. Lstm: A search space odyssey. IEEE Transactions on Neural Networks and Learning
Systems, 21:::22-:222222 2232. 2015.
[20] Albert Gu, Karan Goel, Ankit Gupta, and Christopher Ré. On the parameterization and initialization of diagonal state space models. In NeurlPS, 2022.
[21] Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured
state spaces. In The Tenth International Conference on Learning Representations. ICLR 2022,
Virtual Event, April 25-29, 2022. OpenReview.net, 2022.
[22] Albert Gu. Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. In The International Conference on Learning Representations (ICLR), 2022. [23] Albert Gu, Çaglar Gülçehre. Thomas Paine, Matt Hoffman, and Razvan Pascanu. Improving the gating mechanism of recurrent neural networks. In Proceedings of the 37th International Conference Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 3800-3800 3009.. PMLR, 2020.
[24] Albert Gu, Isys Johnson, Karan Goel, Khaled Saab, Tri Dao, Atri Rudra, and Christopher Ré. Combining recurrent, convolutional, and continuous-time models with linear state space layers. In Mare' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurlPS 2021, December 6-14, 1021, virtual, pages 572-585, 2021.
[25] Albert Gu, Isys Johnson, Karan Goel, Khaled Saab, Tri Dao, Atri Rudra, and Christopher Ré. Combining recurrent, convolutional, and continuous-tima models with linear state- state-space layers,
2021.
[26] Ankit Gupta, Albert Gu, and Jonathan Berant. Diagonal state spaces are as effective as structured state spaces, 2022.
[27]
Ankit Gupta, Harsh Mchta, and Jonathan Berant. Simplifying and understanding state space models with diagonal linear mns. CoRR, abs/22 011221200066.. 2022.
[28] Ramin Hasami, Mathias Lechner, Tsun-Hsuan Wang, Makram Chahine, Alexander Amini, and Daniela Rus. Liquid structural state-space models. In The Eleventh International Conference on Learning Representations, 2023.
[29] Hongyu He and Marko Kabic. A unified view of long-sequence models towards modeling million-scale dependencies. CoRR, abs/2302. 062218 2023.
[30] Sepp Hochreiter and Yoshua Bengio. Gradient flow in recurrent nets: the difficulty of learning long-term dependencies. 2001.
[31] Weizhe Hua, Zihang Dai, Hanxiao Liu, and Quoc V Le. Transformer quality in linear time.
arXiv preprint arXiv:2202. 10447, 2022.
[32] Feiqing Huang, Kexin Lu, Yuxi CAI, Zhen Qin, Yanwen Fang, Guangjian Tian, and Guodong Li. Encoding recurrence into transformers. In The Eleventh International Conference on Learning Representations, 2023.
[33] J. Stuart Hunter. The exponentially weighted moving average. Journal of Quality Technology, 18:203-210, 1986.
[34] Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are mns: Fast autoregressive transformers with linear attention. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 5156-5165. PMLR, 2020. [35] Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention. In International Conference on Machine Learning, pages 5156-5165. PMLR, 2020.
[36] Guolin Ke, Di He, and Tie- Yan Liu. Rethinking positional encoding in language pre-training.
In International Conference on Learning Representations, 2021.
[37] Jan Koutník, Klaus Greff, Faustino J. Gomez, and Jürgen Schmidhuber. A clockwork RNN. In Proceedings of the 31th International Conference on Machine Learning, ICML 2014, Beijing, China, 21-26 June 2014, volume 32 of JMLR Workshop and Conference Proceedings, pages 1871. JMgR.org. 2014.
12

---

[38] Quoc V. Le, Navdeep Jaidy, and Geoffrey E. Hinton. A simple way to initialize recurrent networks of rectified linear units. ArXiv, abs/1 00044,,0011,,, 2015.
[39] James Lee-Thorp, Joshua Ainslie, Ilya Eckstein, and Santiago Ontanon. FNet: Mixing tokens with Fourier transforms. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pages 4296-4315 Seattle, United States, July 2022. Association for Computational Linguistics. [40] Tao Lei, Yu Zhang, Sida I. Wang, Hui Dai, and Yoav Artzi. Simple recurrent units for highly parallelizable recurrence. In Proceedings of the 2018 Conference on Empirical Methods in Nat- ural Language Processing, pages for Computational Linguistics.
Brussels, Belgium, Oct.-Nov. 2018. Association
[41] Yuhong Li, Tianle Cai,, Yi Zhang. De huai Chen, and Dehadeepta Dey. What makes convolu- tional models great on long sequence modeling? ArXiv, abs/2210 10.0229,, 2022.
[42] Hanxiao Liu, Zihang Dai, David So, and Quoc V Le. Pay attention to mlps. Advances in Neural Information Processing Systems, 34:9204 9215, 2021.
[43] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike
Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining
approach. arXiv preprint arxiv:/9077 1692, 2019.
[44] Zexiang Liu, Dong Li, Kaiyue Lu, Zhen Qin, Weixuan Sun, Jiacheng Xu, and Yiran Zhong. Neu- ral architecture search on efficient transformers and beyond. In arXiv preprint arXiv:2207.13955,
2022.
[45] Kaiyue Lu, Zexiang Liu, Jianyuan Wang, Weixuan Sun, Zhen Qin, Dong Li, Xuyang Shen, Hui Deng, Xiaodong Han, Yuchao Dai, et al. Linear video transformer with feature fixation. arXiv preprint arXiv:2210.0811
2022.
[46] Xuezhe Ma, Chunting Zhou, Xiang Kong, Junxian He, Liangke Gui, Graham Neubig. Jonathan May. and Luke Zettlemoyer. Mega: Moving average equipped gated attention. CoRR, abs/2209. 10655, 2022.
[47] Huanru Henry Mao. Fine-tuning pre-trained transformers into decaying fast weights. In Pro- ceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10236-10242, Abu Dhabi, United Arab Emirates, Dec. 2022. Association for Computational Linguistics.
[48] Eric Martin and Chris Cundy. Parallelizing linear recurrent neural nets over sequence length. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018
[49] Harsh Mehta, Ankit Gupta, Ashok Cutkosky, and Behnam Neyshabur. Long range language modeling via gated state spaces. CoRR, abs/2206. 13947, 2022.
[50] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. 5th International Conference on Learning Representations, ICLR, Toulon, France, 2017.
[51] Yajie Miao, Mohammad Gowayyed, and Florian Metze. Eesen: End-to-end speech recognition using deep mn models and wfst-based decoding. In 2015 IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU), pages 167-174. IEEE, 2015.
[52] Antonio Orvieto, Soham De, Çaglar Gülgehre, Razvan Pascanu, and Samuel L. Smith. On the universality of linear recurrences followed by nonlinear projections. CoRR, abs/2307. 1180007.1188... 2023.
[53] Antonio Orvieto, Samuel L. Smith, Albert Gu, Anushan Fernando, Çaglar Gülçehre, Razvan Pascanu, and Soham De. Resurrecting recurrent neural networks for long sequences. CoRR, hts/23033.63499 2023.
[54] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
[55] Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Huanqi Cao, Xin Cheng, Michael Chung, Matteo Grella, Kranthi Kiran G. V., Xuzheng He, Haowen Hou, Przemyslaw Kazienko, Jan Kocon, Jiaming Kong, Bartlomiej Koptyra, Hayden Lau, Krishna Sri Ipsit Mantri, Ferdinand Mom, Atsushi Saito, Xiangru Tang, Bolun Wang, Johan S. Wind, Stanislaw Wozniak, Ruichong Zhang, Zhenyuan Zhang, Qihang Zhao, Peng Zhou, Jian Zhu, and Rui-Jie Zhu. RWKV: reinventing rnns for the transformer era. CoRR, abs/2305. 13048, 2023.
[56] Hao Peng, Nikolaos Pappas, Dani Yogatama, Roy Schwartz, Noah A. Smith, and Lingpeng Kong. Random feature attention. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
13

---

[57] Michael Poli, Stefano Massaroli, Eric Nguyen, Daniel Y. Fu, Tri Dao, Stephen Baccus, Yoshua Bengio, Stefano Ermon, and Christopher Ré. Hyena hierarchy: Towards larger convolutional language models. CoRR, abs/2302. 10866, 2023.
[58]
Michael Poli, Stefano Massaroli, Eric Nguyen, Daniel Y Fu, Tri Dao, Stephen Baccus, Yoshua Bengio, Stefano Ermon, and Christopher Ré. Hyena hierarchy: Towards larger convolutional language models. arXiv preprint arXiv: 10866, 2023.
[59] Zhen Qin, Xiaodong Han, Weixuan Sun, Bowen He. Dong Li, Dongxu Li, Yuchao Dai, Lingpeng Kong, and Yiran Zhong. Toeplitz neural network for sequence modeling. In The Eleventh International Conference on Learning Representations (ICL.R) 2023.
[60] Zhen Qin, Xiaodong Han, Weixuan Sun, Dongxu Li, Lingpeng Kong, Nick Barnes, and Yiran Zhong. The devil in linear transformer. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 7025-7041, Abu Dhabi, United Arab Emirates,
Dec. 2022. Association for Computational Linguistics. [61] Zhen Qin, Dong Li, Weigao Sun, Weixuan Sun, Xuyang Shen, Xiaodong Han, Yunshen Wei, Baohong Lv. Fei Yuan, Xiao Luo, Yu Qiao, and Yiran Zhong. Scaling transnormer to 175
billion parameters. arXiv, 2023.
[62] Zhen Qin, Weixuan Sun, Hui Deng, Dongxu Li, Yunshen Wei, Baohong Lv, Junjie Yan, Lingpeng Kong. and Yiran Zhong. cosformer: Rethinking softmax in attention. In ICLR, 2022.
[63] Zhen Qin, Weixuan Sun, Kaiyue Lu, Hui Deng, Dongxu Li, Xiaodong Han, Yuchao Dai, Lingpeng Kong, and Yiran Zhong. Linearized relative positional encoding, Transactions on Machine Learning Research, 2023.
[64] Zhen Qin and Yiran Zhong. Accelerating toeplitz neural network with constant- tonst inference complexity. In Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Dec. 2023.
[65] Afan Galih Salman, Bayu Kanigoro, and Yaya Heryadi. Weather forecasting using deep learning techniques. In 2015 international conference on advanced computer science and information systems (ICACSIS), pages 281-285. leee, 2015.
[66]
Imanol Schlag, Kazuki Irie, and Jürgen Schmidhuber. Linear transformers secretly fast weight programmers. In International Conference on Machine Learning, 2021.
[67] Jürgen Schmidhuber. Learning to control fast-weight memories: An alternative to dynamic recurrent networks. Neural Computation, 1:3,, 1992.
[68] Sreelekshmy Selvin, R Vinayakumar, EA Gopalakrishnan, Vijay Krishna Menon, and KP Soman. Stock price prediction using Istm, mn and cnn-sliding window model. In 2017 international conference on advances in computing, communications and informatics (icacci), pages 1643-1647. IEEE, 2017.
[69] Noam Shazeer. GLU variants improve transformer. CoRR, abs/2002.05202 2020.
[70] Yikang Shen, Shawn Tan, Alessandro Sordoni, and Aaron C. Courville. Ordered neurons:
Integrating tree structures into recurrent neural networks. In 7th International Conference Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
[71] Jimmy T. H. Smith, Andrew Warrington, and Scott W. Linderman. Simplified state space layers
for sequence modeling. CoRR, abs/2208. 049333. 2022.
[72] Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer
with rotary position embedding. CoRR, abs/2104.09864 2021.
[73] Jingyu Sun, Guiping Zhong, Dinghao Zhou, Baoxiang Li, and Yiran Zhong. Locality mat- ters: A Docality-biased linear attention for automatic speech recognition. arXiv preprint arXiv:2203. / /5609, 2022.
[74] Weixuan Sun, Zhen Qin, Hui Deng, Jianyuan Wang, Yi Zhang, Kaihao Zhang, Nick Barnes, Stan Birchfield, Lingpeng Kong, and Yiran Zhong. Vicinity vision transformer. IEEE Transactions on Pattern Analysis and Machine Intelligence,
12649, 2023.
[75] Corentin Tallec and Yann Ollivier. Can recurrent neural networks warp time? In 6th Interna- tional Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30- May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018
[76] Yi Tay, Dara Bahri, Donald Metzler, Da-Cheng Juan, Zhe Zhao, and Che Zheng. Synthesizer: Rethinking self- celf-attention for transformer models. In International conference machine learning, pages 10183-10192. PMLR, 2021.
[77] Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang. Sebastian Ruder, and Donald Metzler. Long range arena: A benchmark for efficient transformers. In International Conference on Learning Representations, 2020.
[78] Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. Long range arena : A benchmark for efficient
14

---

transformers. In 9th International Conference on Learning Representations, ICLR 2021, Virtual
Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
[79] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herve Jegou. Training data-efficient image transformers &amp; distillation through attention. In International Conference on Machine Learning, volume 139, pages 10347-10357, July 2021. Jos van der Westhuizen and Joan Lasenby. The unreasonable effectiveness of the forget gate.
[80]
CoRR. abs//804. 0404,, 2018.
[81] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez. Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information
processing systems, 30, 2017.
[82] Junxiong Wang. Jing Nathan Yan, Albert Gu, and Alexander M. Rush. Pretraining without attention. CoRR, abs/2212. 10.044,, 2022.
[83] Weihao Yu, Mi Luo, Pan Zhou, Chenyang Si, Yichen Zhou, Xinchao Wang, Jiashi Feng, and Shuicheng Yan. Metaformer is actually what you need for vision. In IEEE/CVF Conference Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 10809- 10099-0089....00 IEEE, 2022.
[84] Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting, In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 11106-11115,
2021.
[85] Chen Zhu, Wei Ping, Chaowei Xiao, Mohammad Shoeybi, Tom Goldstein, Anima Anandkumar, and Bryan Catanzaro. Long-shor short transformer: Efficient transformers for language and vision. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021.
15

---

Appendix
In this appendix, examine the extrapolation ability of HGRN and provide the training and inference speed comparison of HGRN and existing efficient sequence modeling methods. We also illustrate the forget rates of each layer on trained language model of HGRN.
We also report the extrapolation ability of HGRN compared to previous methods in Table 14
6.1
Extrapolation test
In this section, tested HGRN 's extrapolation ability by directly inferring the model with variety of sequence lengths. As shown in Table 14, our method has the ability to train short and test long.
6.2
Speed comparison
In this section, we benchmark the speed of our method on the LRA benchmark. Our method achieves state-of-the-ard training and inference speed.
6.3
Visualization
In this section, we visualize the forget rates of each layer on model trained on language modeling tasks.
6.4
Configurations
We list detailed byper-parameters of our experiments here.
16

---

Figure 3: Visualization forget rates in each layer.
(iii) LRU
(i) With lower bound
(ii) Without lower bound
Layer
Layer
Layer
17.5
17.5
15.0
15.0
Percent
Percent
12.5
6-
10.0
1000
7.5
2-
2.5
25
0.3
0.4
0.5 0.6 0.7 0& 0.9 1.
0.0
0.7 0.3
0.4
0.5
0.6 0.7
0.8 0.9 1.0
0.6
0.7
0.8
0.9
1.0
Forget rate
Forget rate
Forget rate
Layer
Layer 2
Layer 2
17.5
15.0
Percent
Percent
12.5
Percent
10.0
10.0
7.5
5.0
2.5
0.4
0.5
0.6
0.7
0.8
0.9
1.0
0.3
0.4
0.5
0.6
0.7
0.8
0.9 1.0
0.650. 700
850.900. 951.0
Forget rate
Forget rate
Forget rate
Layer
Layer
Layer
10
16-
16
Percent
12.
Percent
Percent
10:
10
0.3
0.4
0.5
0.6 0.8
0.9
100
0.3 0.4
0.5
0.8
0.9
1.0
0.600.650 850.900.991
Forget
Forget
Forget
Layer
Layer 4
Layer
14.
10.
12-
Percent
Percent
10:
Percent
Forget rate
Forget rate
Forget rate
Layer 5
Layer
Layer 5
Percent
Percent
8-
Percent
6
10.0
7.5
5.0
2.5
0.0
0.4
0.5
0.6
1.0
0.2
0.4
0.6
0.8
1.0
0.5
0.7
0.8
0.9
1.0
Forget
Forget rate
Forget rate
Layer
Layer
Layer
Percent
40
Percent
Percent
30
10
20
10
1.0
Forget rate
Forget rate
Forget rate

---

Figure 4: Visualization of token mixing matrix each layer.
(i) Layer 1
Layer
(iii) Layer
(iv) Layer
(v) Layer 5
(vi) Layer
18

---

HGRU
Table 14: The extrapolation performance of competing methods. The best result is highlighted in bold and the second in underline.. means lower is better.
PPL.
24.85
24.4
24.16
24.03
23.94
23.88
23.82
23.71
23.66
23.62
23.6
23.58
23.58
23.56
23.55
23.55
23.55
23.54
23.53
23,80
LRU
PPL.
31.12
30.72
30.5
30.38
30.3
30.24
30.19
30.09
30.04
30.01
29.99
.9
29.97
29.97
29.96
29.94
29.94
29.94
.4
29.93
29.92
30.17
TNN
PPL.
24.67
24.25
24.05
23.91
23.83
23.79
23.73
23.63
23.58
23.54
23.53
23.51
23.51
23.49
23.48
23.48
23.48
23.47
23.46
23.70
ALiBi
PPL.
24.15
23.38
22.98
22.74
22.57
22.52
22,43
22.24
22.17
22.11
22,08
22.07
22.05
22.03
22.02
22.00
22.00
22.00
21.99
22.40
GSS
PPL.
39.66
39.76
39.91
40.82
41.04
41.08
41.53
44.08
48.27
53.32
57.73
60.25
63.36
74.92
81.87
87.67
92.11
96.00
101.47
60.26
DSS
PPL.
41.07
40.50
40.22
40.03
39.94
39.85
39.79
39.64
39.59
39.54
39.51
39.49
39.49
39.46
39.45
39.46
39.44
39.43
39.43
39.75
S4
PPL.
30.74
30.41
30.24
30.15
30.08
30.04
30.00
29.91
29.88
29.85
29.83
29.82
29.82
29.80
29.79
29.79
29.79
29.78
29.78
29.97
8.93E+12
1.58E+15
4.96E+16
5.67E+17
3.59E+18
2.19E+20
1.61E+21
5.08E+21
1.16B+22
1.98E+22
2.78E+22
3.93E+22
4.9 4.93E+22
5. 5.70E+22
7. 78
8.04E+22
9.41E+22
2.41E+22
gMLP
PPL.
1.34B+9
29.13
cosFormer PPL.
102.88
175.26
267.65
368.02
820.77
1335.51
1735.50
2146.19
2494.79
2902.24
3028.72
3247.83 83
3341.91
3644.88 81
3851.70
3951.26
1764.75
27.06
32.90
55.28
Performer
PPL.
63.16
159.74
504.30
1020.28
1568.83
2138.50
2693.89
4945.83
7170.91
8443.15
10234.07
11420.56
12557.09
14847.66
13623.83 832
14661.7 1.7
17959.88 85
20026.35
20971.3 1.31
8684.79 799
1+elu
PPL.
28.05
47.35
70.47
91.88
111.56
129.92
147.09
206.88
267.87
299.31
352.62
389.02
411.50
453.27
457.06
504.19
555.38
584.01
589.83
299.86
FLASH
16950.45
174165.47
346502.88
647788.12
1719873.5
6.25E+6
4.17E+10
2.67E+13
1.26E+15
1.58E+16
8.11E+16
3,403+177
1.22E+18
4.03E+18
1.07E+19
2.52E+19
4.96E+19
1.28E+20
1116++19
PPL1
24.69
LS
PPL.
24.05
23.49
23.21
23.07
22.97
22.97
22.99
23.25
23.83
24.56
25.45
26.42
27.11
28.78
30.13
31.14
33.21
34.72
36.05
26.71
Transformer
PPL.
24.78
41.36
62.35
82.52
100.17
118.42
133.44
188.95
246.06
270.93
311.65
346.58
372.18
387.29
395.94
426.54
463.50 55
506.35
486.86 86
261.36
Seqlen
10240
11264
12288
13312
14336
512
768
1024
1280
1536
1792
2048
3072
4096
5120
6144
7168
8192
9216
Avg
19

---

Table 15: Speed comparison on LRA benchmark. The IK, represent the input sequence length.
We mark with if a method is out memory. The higher the better for all metrics.
Train Steed(steps per secondyt
Inference Speed per second)
Method
IK
2K
3K
4K
5K
IK
2K
3K
4K
5K
Transformer
13.58
4.84
23.67
8.22
Performer [36
18.40
10.77
7.66
6.30
5.64
30.04
17.36
12.80
10.55
9.52
LS
35
20.29
11.24
8.05
6.51
5.89
39.05
21.11
15.02
12.6
11.66
Fnet
39
25.19
15.62
11.24
9.41
8.18
48.81
27.89
19.52
16.27
14.46
cosFormer
22.00
12.80
9.47
7.93
7.13
39.05
22.31
16.62
13.95
12.60
$4 22
13.13
7.33
4.91
3.84
3.04
30.04
16.27
10.85
8.58
6.79
FLASH 31
17.36
9.03
6.54
5.19
4.68
30.04
15.94
11.32
9.19
8.40
TNN [59
17.55
9.89
6.79
5.68
4.54
33.96
17.75
12.40
10.28
8.22
HGRU
22.31
13.58
9.52
7.40
7.44
43.39
25.19
16.62
14.20
13.95
Table 16: Detailed training configurations used in experiments. "Total batch size" means batch_per__ _er__pe__p_______ppp gpu X update_free x mum_gpus "ALM" stands for Autoregressive Language Model. "IM" stands for Image Modeling.
AML
IM
Data
WikiText-
Image Net- 1k
Tokenizer method Sre Vocab size
BPE
50265
Sequence length
Total batch size
128
2048
Number of updates/epochs
50k updates
300 epochs
Warmup steps/epochs
4k steps
5 epochs
Peak learning rate
5c-4
2.5e-4
Learning rate scheduler Optimizer
Inverse sqrt Adam
cosine
Adamw
Adam
le-8
le-8
Adam (8i, .32)
(0.9, 0.98)
(0.9. 0.98)
Weight decay
0.2
0.1
Gradient clipping
1.0
20


Reviews:
[{"review_id": "OCBTYHEiav", "reviewer": "Reviewer_rGhK", "summary": "The manuscript introduces a novel recurrent neural network architecture with linear recurrent units. The proposed method improves reasoning about long-term dependencies by imposing a monotonically increasing lower bound for the forget gate values of RNN layers, i.e. higher layers retain information longer. Since the proposed method uses linear recurrent units, the authors also employ complex valued activations to increase the models expressive power. Further design choices include tied input and forget gates and the addition of output gates. For evaluation, the authors considered several benchmarks, including WikiText-103, ImageNet-1k and long-range arena (LRA).", "questions": "Overall, I think the method has a lot of potential, but some of the issues mentioned in the weaknesses section prevent me from recommending a higher score at this point.\n\nI have some questions and minor comments:\n1. Line 170 mentions that $\\theta'$ is fixed to 0. Does the equation $i_t=(1-\\lambda_t)\\exp(i\\theta')$ really correspond to $i_t=1-f_t$ for all layers, given that $\\theta$ in $f_t = \\lambda_t\\exp(i\\theta)$ is different across layers? How did you decide on fixing $\\theta'$ to $0$?\n2. Eq 3: How does the dimensionality get reduced to $\\mathbb{R}^{1\\times d}$?\n3. There's a duplicate entry in the references (36 and 37)\n4. Line 176: The reference in \"Expanding the 2\" does not work, I assume it is supposed to refer to Equation 2. It's best to always add the reference type, e.g. \"Equation 2\", instead of just \"2\".\n5. There are two sentences starting with \"Finally,\" in the paragraph of lines 217-225.\n6. Line 292: \"border\" should be \"broader\"\n7. I think it'd be good to include a brief mention of works on deep RNNs that update at different time scales (e.g. clockwork RNN [1] or HM-RNN [2]) in the related work section.\n8. Figure 1 and other places: Sometimes the hidden unit uses the acronym HRGU, sometimes HGRU, the latter of which I think is the correct one. Consider using a tex package for automating the use of acronyms, e.g. glossaries (see https://www.overleaf.com/learn/latex/Glossaries#Acronyms).\n\n## References\n1. Koutnik, Jan, et al. \"A clockwork rnn.\" International conference on machine learning. PMLR, 2014.\n2. Chung, Junyoung, Sungjin Ahn, and Yoshua Bengio. \"Hierarchical multiscale recurrent neural networks.\" arXiv preprint arXiv:1609.01704 (2016).\n\n\n## Acknowledgement of rebuttal\nI have read the rebuttal, the other reviews and engaged actively in discussion with the authors. The authors have addressed several of my concerns, including the addition of aggregated results with an estimate of the variance over random seeds. As a result, I have decided to increase the score.", "limitations": "As limitation, the authors only mention that large-scale testing has not been performed. I would have liked to see a discussion of limitations of the methodology the authors see or anticipate e.g. when scaling the method.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 4, "strengths": "The method presents an intriguing approach to addressing difficulties in modeling long-range dependencies with RNNs. The design choices of the proposed method are well-motivated and the performance is mostly better or on par with other methods.", "weaknesses": "1. The reported metrics are not aggregated over multiple runs and thus don't include estimates of variation over random seeds. This is problematic, since it's not clear whether differences in performance are greater than the variation over multiple runs of the same experiment. HRGN outperforms all other baselines in terms of average performance on the LRA benchmark, but LRU shows higher performance for 4 out of 6 tasks. Here an estimate of the variation over random seeds would definitely be helpful in comparing different methods. The same is true for experiments in the ablation study.\n\n2. The manuscript overall is a bit difficult to follow. Several variables are used without being defined, e.g.:\n   1. Line 143: $g_t$ is mentioned before being defined.\n   2. Eq 3: $\\tau$ is not defined. Is it a typo and should've been $\\sigma$ or is it another activation function?\n   3. Line 191: What is $\\mu$ (without subscript $t$)?\n3. I think Section 3.3 could benefit by providing a bit of context, i.e. what the token mixing perspective is supposed to highlight.\n4. The authors write that the same training configuration, including the learning rate, was used for all models. How were these hyperparameters selected? The optimal learning rate may be different for different architectures. A fair comparison should take this into consideration.\n5. The format of the tables with experimental results is inconsistent. In Table 1, the lowest perplexity is highlighted in bold, which should be separate for val and test, same for the underlined second best. In Table 2, the caption says that bold and underlined highlight the best and second best results per task, but this is not the case. Table 3 does not have highlights.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1:  The formulation of the experiment result.**\n\nA1: Thank you for your suggestion. The formulation of the experiment result is identical to that of [1], [2], and [3]. We present the results from Wikitext-103 without any variation since the PPLs on that corpus remain consistent. All publications have reported no variation in the results. Additionally, based on references [1], [2], and [3], all LRA results indicate the highest value.\n\n**Q2: Definition of some notations.**\n\nA2: Thank you for your reminder. We will fix these issues in the revised versions. Regarding the concerns you raised, the notations $g_t$ and $\\tau$ were indeed typographical errors, where $\\tau$  corresponds to the activation function. Additionally, the use of $\\mu$ was a typo, and it should refer to $\\bar \\gamma$.\n\n**Q3: What the token mixing perspective is supposed to highlight?**\n\nA3: The purpose of this section is to explain the distinct roles of lambda and theta. In this context, lambda serves the purpose of capturing the decay rate, while theta's role is to capture the relative positional relationships. R4 appreciates the explanation. We will modify the section to enhance its clarity.\n\n**Q4: Choice of configurations**.\n\nA4: We retained the configuration of the Toeplitz neural network as described in reference [1], without performing any hyperparameter adjustments.\n\n**Q5: The format of the tables with experimental results.**\n\nA5: Thank you for pointing them out. We will address these issues in the updated version.\n\n**Q6: Choice of $\\theta’$.**\n\nA6: We included ablation studies to validate the rationale behind the proposed method. It can be observed that when using $\\theta’=0$ has a better result.\n| method | ppl |\n| --- | --- |\n| $1-\\lambda$ | 24.14 |\n| $(1-\\lambda)\\exp(i\\theta)$ | 24.5 |\n\n**Q7: Eq 3: How does the dimensionality get reduced to \\mathbb{R}^{1\\times d}?**\n\nA7: This is a typo, the dimension should be $\\mathbb{R}^{1 \\times 2d}$. We will fix this in the updated version.\n\n**Q8: There's a duplicate entry in the references (36 and 37)**\n\n**Q9: Line 176: The reference in \"Expanding the 2\" does not work, I assume it is supposed to refer to Equation 2. It's best to always add the reference type, e.g. \"Equation 2\", instead of just \"2\". Q10: There are two sentences starting with \"Finally,\" in the paragraph of lines 217-225. Q11: Line 292: \"border\" should be \"broader\". Q12: I think it'd be good to include a brief mention of works on deep RNNs that update at different time scales (e.g. clockwork RNN [1] or HM-RNN [2]) in the related work section.**\n\nA8-12： In regards to questions 8–12, we will correct the typos in the updated version.\n\n**Q13: Name of the unit in Figure 1.**\n\nA13: This is a typo. The correct name is HGRU. We will fix this issue in the reversion version.\n\nCitations:\n\n- [1]  Zhen Qin, Xiaodong Han, Weixuan Sun, Bowen He, Dong Li, Dongxu Li, Yuchao Dai, Lingpeng Kong, and Yiran Zhong. Toeplitz neural network for sequence modeling. In The Eleventh435 International Conference on Learning Representations, 2023.\n- [2] Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. In The Tenth International Conference on Learning Representations, ICLR 2022,350 Virtual Event, April 25-29, 2022. OpenReview.net, 2022.\n- [3] Xuezhe Ma, Chunting Zhou, Xiang Kong, Junxian He, Liangke Gui, Graham Neubig, Jonathan401 May, and Luke Zettlemoyer. Mega: Moving average equipped gated attention. CoRR, abs/2209.10655, 2022"}, {"review_id": "TcwJBBnL7X", "reviewer": "Reviewer_pKf6", "summary": "This paper proposed a novel RNN architecture that utilized several existing components: complex(unitary) matrix, and multi-layer. \n\nThe authors also claim stronger long-term dependency ability of the proposed method on several long context benchmarks.\n", "questions": "N/A", "limitations": "N/A", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well organized. Experiments on Wikitext, ImageNet etc are good benchmarks for long-term dependency test. \n\n2. The discussion on the forgetting gate is interesting and demonstrates the effectiveness of the proposed method on learning long-term dependency.\n\n3. The lower bound design is novel. The authors justified the design with ablation experiments.", "weaknesses": "1. Figure 1, the most important diagram, is super confusing. Why do you use a dashed arrow to explain a module?\n\n2. Orthogonal/Unitary matrix design has been long discussed in the past. It has already shown strong long-term dependency ability. This paper failed to mention any of them.\n\n3. Experiment results lack detail. No training or architecture specs of other baseline models. Where do these numbers come from? What does `transformer` on `wikitext-103` mean? What sized transformer? ...\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Explanation of Figure 1.**\n\nA1: We will enhance the clarity of Figure 1 in subsequent versions. For the leftmost part of Figure 1 represents the composition of each layer in HRGN involving HGRU and GLU. Dashed arrows point to the middle section of the diagram, explaining HGRU's components: recurrent computing, gate, normalization, and out projection. Here, the dashed arrow leads to recurrent computing, located at the far right of the diagram and corresponding to Algorithm 1. \n\n**Q2: Relation to Orthogonal/Unitary matrix.**\n\nThank you for bringing the Orthogonal/Unitary matrix to our attention. We will include a discussion on this aspect in future versions. However, our method differs significantly from such methods.\n\nAccording to [1], [2], if we disregard nonlinear functions, the recursive form can be expressed as:\n\n$$\nh_{t} = W_t h_{t-1}+ c_t,\n$$\n\nwhere $h_t, c_t \\in \\mathbb{R}^{d\\times 1}$ and $W_t \\in \\mathbb{R}^{d\\times d}$ is a Unitary matrix. Expanding this yields:\n\n$$\nh_{t} = \\sum_{i=1}^t \\left(\\prod_{j=i}^t W_j \\right)c_i.\n$$\n\nSince $\\left(\\prod_{j=i}^t W_j \\right)c_i$ operates on the feature, it can be categorized as Channel mixing [4], and we denoted it as $\\bar c_i$, leading to the equation:\n\n$$\nh_{t} = \\sum_{i=1}^t \\bar c_i.\n$$\n\nFrom the perspective of Token mixing, the above can be expressed as:\n\n$$\n\\begin{bmatrix}\nh_1^{\\top}  \\newline\n\\vdots  \\newline\nh_n^{\\top}\n\\end{bmatrix}=\n\\begin{bmatrix}\n1 & 0 & \\ldots & 0  \\newline\n1 & 1 & \\ldots & 0  \\newline\n\\vdots & \\vdots & \\vdots & \\vdots  \\newline\n1 & 1 & \\ldots & 1 \\newline\n\\end{bmatrix}\n\\begin{bmatrix}\n\\bar c_1^{\\top}  \\newline\n\\vdots  \\newline\n\\bar c_n^{\\top}\n\\end{bmatrix}.\n$$\n\nNotably, [1], [2] assume the Token mixing matrix to be a lower triangular matrix with all ones. In contrast, our method, as per equation 5, takes the form of:\n\n$$\n\\mathbf A =\\left[\\begin{matrix}\n1-\\lambda_1 & 0 & \\cdots  & 0    \\newline\n% (1-\\lambda_1)\\lambda_2 \\exp(j\\theta_2) & 1-\\lambda_2  &  &  \\vdots   \\newline\n(1-\\lambda_1)\\lambda_2 \\exp(i \\theta) &  1-\\lambda_2 &  &  \\vdots   \\newline\n\\vdots &\\vdots   &\\ddots&  0  \\newline\n(1-\\lambda_1) \\left[ \\prod_{s=2}^n \\lambda_k  \\right] \\exp(i(n-1)\\theta)&\n\\ldots  &\n\\ldots  &\n1-\\lambda_{n}\n\\end{matrix}\\right],\n$$\n\nleading to significant distinctions between the two approaches.\n\n**Q3. Explanation of experiment results.**\n\nWe retained the configuration of the Toeplitz neural network as described in reference [1]. In Table 1, we have indicated the model size in the third column, which is 44.65 million parameters. “`transformer` on `wikitext-103`\" means training a transformer language model on wikitex-103 dataset. In the revised version, we will provide the architecture specifications, and we plan to release the code soon.\n\nCitations:\n\n[1]  Zhen Qin, Xiaodong Han, Weixuan Sun, Bowen He, Dong Li, Dongxu Li, Yuchao Dai, Lingpeng Kong, and Yiran Zhong. Toeplitz neural network for sequence modeling. In The Eleventh International Conference on Learning Representations, 2023.\n\n[2] Martin Arjovsky, Amar Shah and Yoshua Bengio. Unitary evolution recurrent neural networks. ICML'16: Proceedings of the 33rd International Conference on International Conference on Machine Learning\n\n[3] Jing, L, Shen, Y, Dubček, T, Peurifoi, J, Skirlo, S, LeCun, Y, Tegmark, M, Soljačić, M. Tunable Efficient Unitary Neural Networks (EUNN) and their application to RNNs. 34th International Conference on Machine Learning\n\n[4] Weihao Yu, Mi Luo, Pan Zhou, Chenyang Si, Yichen Zhou, Xinchao Wang, Jiashi Feng, and Shuicheng Yan. Metaformer is actually what you need for vision. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 10809–10819. IEEE, 2022."}, {"review_id": "GlbGp16Hy7", "reviewer": "Reviewer_TZWK", "summary": "This paper uses the token-mixing interpretation of Transformers to address the slow back-propagation problem of RNNs. First, back-propagation is made one-step per layer, rather than T steps, by using linear recurrence.  In order to avoid dramatically exploding the size of the effective weight matrix in back-propagation, the recurrence is also set to be diagonal, with a channel-mixing layer (GLU) just above each HRU in order to permit mixing of the channels. Per-channel recurrence with real-valued weights can only represent exponential decay, whereas gated recurrence needs to represent delayed recall; delayed recall in the proposed network is represented using complex-valued recurrence.  For simplicity, the forget gate is set to one minus the input gate.  For simplicity, because it gives better results in practice, and because it gives results that can be easily interpreted in terms of the recall periodicity of each channel, the recurrent weights are set independent of the input data, so that only the forget gate and output gate depend on the input data.  ", "questions": "These are not really questions, but there are three typos:\n\np. 4: R^{Hxd} -- H has not been defined, is it the number of layers?\n\nTable 4 caption: layer in independent -> layer is independent\n\nTables 5 and 6 are shown on the same page, but referenced in reverse order; should they be swapped?\n", "limitations": "The only weakness is the one noted by the authors: Though this is a convincing proof of concept, it has not yet been scaled to GPT scales.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "This paper seems to provide an architecture that seems to have the full representational power of an LSTM, but with the one-step-per-layer back-propagation speed of a Transformer. \n\nThe Transformer has significant training problems on small data: in order to learn useful attention calculations, it needs to see a pretty large dataset.  The LSTM does not have that problem; the inductive bias of the RNN allows the LSTM to converge with somewhat smaller training datasets.  Nevertheless, the LSTM is rarely used now, because of the T-step back-propagation through each recurrent layer.  If HGRU scales well, it could provide an architecture that supercedes the Transformer for small to medium-sized supervised datasets with no useful self-supervised-pretraining dataset available.", "weaknesses": "The only weakness is the one noted by the authors: Though this is a convincing proof of concept, it has not yet been scaled to GPT scales.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1:  Scaled to GPT scales.**\n\nAfter training a 1 billion parameter model on the Pile dataset, we have observed that HGRN outperforms the Transformer model as shown below. This result highlights the effectiveness of HGRN in high parameter level.\n\n| method | PPL |\n| --- | --- |\n| Transformer | 4.56 |\n| HGRN | 4.141 |\n\n**Q2: Some typos.**\n\nThe H is the number of layers . We will fix these issues in the revised versions."}, {"review_id": "hYsdAWQCtg", "reviewer": "Reviewer_QJYn", "summary": "This work is concerned with improving linear RNNs (which can be efficiently parallelized) for long-sequence modeling. It proposes to address this problem with gating schemes and complex forget gate values. They introduce a learnable lower bound on the forget gate (or essentially the diagonal dynamics matrix) for each layer to capture both short-term and long-range dependencies. The result is the proposed Hierarchical Gated Recurrent Units (HGRU). This RNN is evaluated on Wikitext, LRA and Imagenet-1K. Ablations are performed to analyze the effectiveness of the design choices.", "questions": "Major questions/suggestions:\n\n- Line 186: It is stated here that $\\lambda_t$'s near 1 \"makes HRGN more efficient modeling of global information\". What does efficient mean here? More efficient than what? How is this claim supported? More clarification should be provided here to avoid making an imprecise or unwarranted claim. \n\n- Line 187: The \"visualization of the token mixing matrix\" cannot be found in the appendix\n\n- Section 3.4: I would recommend moving this section up closer to the \"Lower bound on forget gate values\" section. I found myself having to scroll back and forth to remind myself of the definitions. \n\n- Figure 1: needs more labels. How many gates are there? Is the output gate from equation 3 located in the HRU (per the comment in line 174) or in the HGRU (middle figure)? Clearly labeling the location of the f and g gates, as well as the hidden state h would help a ton.  It would also be nice if the full equation for the recurrence (including the computation of $\\lambda_t$) was a part of this diagram, where one would not have to wonder what each of the different symbols mean, or scroll throughout the paper to piece this together. \n\n- Algorithm 1: I would recommend including a line for the explicit computation of $\\lambda_t$. This is an important part of your approach, but the algorithm makes it appear like this is just some learnable parameter. \n\n- Line 208: What does identical training hyperparameters mean? It is not even clear the baselines reported from other papers all use the same hyperparameters. Also is this the right approach? It is not obvious that each method should use the exact same learning rate etc and I would think it would be reasonable to perform a mild sweep over a few different learning rates and other settings. If not, how did you choose the learning rate? Did you choose the learning rate that was best for HRGN? More information here would allow a reader to better assess the reported results. \n\n- Line 224: \"proves the effectiveness of our method in LM.\" I frankly think this is too strong of a claim. It is well known that performance on wikitext-103 can be sensitive to regularization since the dataset is relatively small (at least in 2023). I think a more accurate and defensible statement might read as these results \"provide evidence HRGN may be an effective method in LM\". I also personally think this weakened claim is sufficient for an interesting paper. \n\n- Line 274: I think the discussion of the \"forget gate values\" of LRU is interesting but could be further developed and discussed. This provides an opportunity to connect the forget gates discussed in this paper with the dynamics state matrices used by S4/S5/LRU etc. In addition, I think the point in line 280 about data-dependent gates (or dynamics matrices) is interesting. It would be relevant to cite Liquid-S4   (https://arxiv.org/abs/2209.12951) here since they propose a method to have input-dependent dynamics with an S4-like approach. This approach also has strong performance on LRA and their results should also be included. \n\n- How is the recurrence actually computed? Are parallel scans used? A parallel scan is mentioned in line 135 when talking about the generic equations before the HGRU is fully introduced. Is this how this is computed? Was the parallel scan implemented in Torch? Also, in line 136, Martin and Cundy should probably be cited here for the parallel scan since they were one of the first to suggest their use for linear RNNs. S5 could also be cited since it was the first to actually make this approach work well for challenging long-range problems. LRU could also be cited since they show an effective method that uses a parallel scan.\n\n- More details should be provided for the speed comparisons in Table 10. Also, how does it scale to longer sequences than 5K, e.g. the 16K of Path-X? How does this compare to other methods that use a parallel scan such as S5 or LRU? Or other diagonal RNNS/SSMs that use convolutions such as H3/S4D/DSS? S4 times are listed here, but I wonder if the optimized Cauchy kernel implementation was used? It is also unclear to me why HGRU would be significantly faster than S4 at inference speed? Both are run as RNNs? Is this due to hyperparameter/state size selection? How was this chosen? More details and comparisons would resolve these questions.\n\nOther questions/suggestions:\n- Line 39: the LSSL (https://arxiv.org/abs/2110.13985) and S4 papers should be cited here since these were the first papers that showed this arrangement of linear layers and nonlinear activations can work well on challenging tasks at scale. \n\n- Line 68: S4D should be cited here since it was the first work to show a significant performance increase  for linear SSM/RNN methods over the original S4 release by using GLU activations. \n\n- Line 72: It might make since to cite S4, S4D, DSS, Liquid S4 (https://arxiv.org/abs/2209.12951) here also since they are all long convolution methods\n\n- Line 85: While [3] and [37] both suggested removing the nonlinearity on the recurrence, it really was not until the LSSL and S4 papers that it was shown you can actually do this without losing performance. These methods should probably be cited here.\n\n- Line 91: S5 also uses a diagonal recurrence matrix and could be cited here\n\n- Line 102: MEGA very explicitly uses an EMA approach and could be cited here\n\n- Line 105: in regards to LRU \"we illustrate that the addition of gating scheme substantially enhances its performance\", but this is basically what is already known from H3. So a good baseline for the Wikitext table would have been to either compare to H3 using S4D, or to compare to LRU plugged into H3. This would allow for a comparison with the other differences between this H3-LRU variant and the HGRU. \n\n- Line 142:  Along with the other cited approaches. DLR and S5 should also be cited here as linear RNNs that uses complex-valued eigenvalues.\n\n- Line 143: I think $g_t$ is introduced here without explanation. Presumably this refers to the output gate described around line 172, but is confusing when it has not been introduced yet\n\n- Line 151: I would not use the phrase \"final formulation\" here, when there is much more formulation to be discussed. This confused me at first because I was trying to figure out how $\\lambda_t$ was determined and had to read on to realize it was not a data-independent learnable parameter.\n\n- Line 156: ON-LSTM is mentioned here without much explanation. Why is this relevant? A bit more explanation here might be helpful. \n\n- Line 174: Has $\\tau$ been introduced? What kind of nonlinearity is this?\n\n- Line 292: typo, should be broader impact and not \"border\" impact\n\n\nIn summary, I like the main ideas of this paper, think it has the potential to be a strong submission and would like to see it accepted. But I do not think it can be accepted in its current form. However, I do think it is feasible to incorporate the suggestions above and I am willing to revise my score upward.", "limitations": "Limitations are discussed.", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "- The area of linear SSMs/RNNs has gained a fair amount of recent attention and the angle this paper takes of considering the linear RNN dynamics matrix from the point of view of the LSTM forget gate is of interest to the community and has the potential to help bridge the gap between prior RNN work and the more modern linear SSM/RNN work\n- While not framed in this way, the forget gate formulation serves as a practical example of input-dependent dynamics, of interest to the deep SSM community\n- The method appears to achieve strong performance on the tasks evaluated in this paper\n- The ablations are helpful to understand the design choices better", "weaknesses": "- While most relevant works are cited in some place in the paper, the results of many of these methods are inexplicably left out of the results tables, leading to misleading statements:\n  - For example SgConv and MEGA are both cited in the paper, but their results are left out of the LRA table. Both of these methods have a better Path-X score (97.83 and 97.98) and overall average (87.17 and 88.21) than the proposed HGRU and both have been released on Arxiv since 2022. This makes the statements in lines 229 that HRGU \"is better than all previous methods\" and lines 231-232 that HRGU \"outperforms all previous methods on this task\"  clearly incorrect. \n  - Similarly, S5 (https://arxiv.org/abs/2208.04933) is never cited at all in the paper despite the fact that it is a linear RNN method that uses the same parallel scan approach for computing a linear recurrence as HGRU (assuming HGRU actually uses a parallel scan though see questions for clarification). The LRU paper that is cited discusses S5 extensively.   In addition, S5's scores are left out of the LRA table despite having a better path-X and overall average than HGRU (98.58 and 87.46), similar to the SgConv and Mega examples mentioned above.  This is particularly relevant since it serves as an ablation showing the forget gate innovations in this paper are not strictly necessary for strong longe-range modeling with a linear RNN.\n  - To clarify, I do not think it would hurt the story of this paper to include these methods that perform slightly better, and in fact I think the statement in line 232 \"It validates the ability of HRGN in handling long sequence tasks\" is completely reasonable and supported by the evidence. It is just the exclusion of these other methods may mislead readers not as familiar with the research on what is strictly necessary for strong performance and the statements listed above regarding \"outperforms all...\" are incorrect.\n  - Finally, given LRU is cited as concurrent work, it would also be relevant to cite DLR (https://arxiv.org/abs/2212.00768) since it explores similar themes.\n\n- It would seem that H3 (cited in the paper) should have been a baseline for the Wikitext-103 experiments. Alternatively, the LRU could have been plugged in for the S4D SSMs used in H3.  As stated in the paper, it is well known in the literature at this point that just using an SSM/Linear RNN such as S4, DSS or LRU without additional gating is not sufficient for language modeling. But additional gating such as in H3 has been shown to work well, such as in the H3 paper and the Wang et al. 2022 papers that are cited. Including this baseline would allow for a better comparison with prior methods on language and allow for a better understanding of the effectiveness of the HGRU innovations vs the gating findings from previous work.\n\n- The overall presentation is poor. Many of the tables say that certain scores are bold or underlined, yet the tables often have no bolds or underlines, or it is inconsistent. Many things are promised to be in the appendix, but then are not there (see questions section below for specific examples). I think some of the sections in the presentation of the method could be improved (see questions section below for suggestions). In addition, Figure 1 is not clearly labeled, leaving one guessing on which gates are located where.\n\n- The authors have checked yes on the author checklist for including training details and reproducibility, but I think this is clearly wrong. First, it is not even clear which methods were implemented by the authors and which methods have numbers reported from other works. Section 4.1 states that \"we adopt the same training configuration for all competitors....\" and that detailed hyperparams are listed in the appendix. However, no hyperparameters are listed in the appendix and many of the performance scores appear to be reported directly from  the prior papers. In addition, no code is included to allow one to try and make sense of this. The authors should make it clear which methods were implemented and evaluated as well as provide detailed hyperparameters to enable reproducibility.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: More citations and experimental results**.\n\nA1: We will include the results for SgConv, MEGA, and S5 in the revised version and rectify our narrative accordingly, and add citation for DLR. We will also include the results for S5 in the upcoming update. Regarding the forget gate, we conducted additional experiments using the LM configuration from Hyena. The previous experimental results for HGRN are from [1]. From the table below, it is evident that our method performs comparably to Transformer, Hyena, and H3 in terms of performance. On the other hand, the performance drop is substantial when the forget gate is removed. This validation underscores the significant role of the forget gate.\n\n| Model | PPL |\n| --- | --- |\n| Transformer | 18.6 |\n| Hybrid H3 | 18.5 |\n| Performer | 26.8 |\n| Reformer | 25.6 |\n| AFT-conv | 28.2 |\n| Linear Attention | 25.6 |\n| Hyena-Convolution | 18.6 |\n| Hyena-slim-Convolution | 18.5 |\n| HGRN | 18.6 |\n| HGRN w/o forget gate | 20.0 |\n| HGRN w/o lower bound | 19.1 |\n\nBesides, the intention of our paper is not to assert an **absolute necessity** for forgetting gates in linear RNNs for long sequence modeling. Rather, our primary objective is to illustrate that simple gated RNNs could exhibit remarkable performance in LRA through the introduction of a lower bound ( in Table 8). **We remark that HGRU is conceptually much simpler than prior state-space models. It obviates the need for meticulously crafted initializations (e.g., Hippos) and yet demonstrates superior performance in real-world tasks such as language modeling.**\n\n**Q2: More experimental results on language modeling.**\n\nA2: We utilize the codebase from H3 to conduct experiments on a GPT-small scale model (～125M parameters) for our HGRN. THe result is listed in R4Q1. Remarkably, HGRN achieves performance parity with Transformers **without incorporating any attention layers**, unlike H3 which employs two such layers. We also scaled our model to 1B parameters and trained on The Pile dataset. As shown in R3 Q1,  HGRN outperforms the Transformer by a clear margin. \n\n**Q3: The presentation quality.**\n\nA3: We will fix these issues in the updated versions.\n\n**Q4: Checklist for including training details and reproducibility**\n\nA4: We will include in the subsequent update a breakdown of which methods were implemented by us and which were referenced. Our configurations are from the Toeplitz neural network [2] and are listed in `one-page PDF`. We will release the code to ensure reproducibility.\n\n**Q5: What meaning of efficient?**\n\nA5: This is a typo. It should be corrected to 'effective'. \n\n**Q6: Visualization of the token mixing matrix.**\n\nA6: We have already added the results of the visualization, as demonstrated in `one-page PDF`.\n\n**Q7: The location of \"Lower bound on forget gate values\" section, more labels on Figure 1 , more detail on Algorithm 1.**\n\nA7: We will fix this issue in the revised versions.\n\n**Q8: Training configurations.**\n\nA8: We retained the configuration of the Toeplitz neural network as described in reference [1], without performing any hyperparameter adjustments.\n\n**Q9: Too strong of a claim.**\n\nA9: Thanks for the suggestion; we will modify this statement in the updated version accordingly.\n\n**Q10: Citation of Liquid-S4.**\n\nA10: We will add the results for Liquid-S4 in the updated versions.\n\n**Q11: Implementation of recurrence.**\n\nA11: We implement linear recurrence by fusing the entire for loop into a single CUDA kernel, akin to RWKV's implementation. Reference to Table 1 in [3] indicates that parallel scan is only faster when sentence lengths exceed 10k, so we did not use parallel scan in training. Still, our model admits the use of the parallel scan, which could be potentially useful in domains such as RNA modeling.\n\n**Q12: More details of the speed comparisons in Table 10.**\n\nA12: In Table 10, we compare various methods based on their speed results with similar parameter counts. Our approach uses a loop-based CUDA implementation, and for the version with a length of 16K, the parallel scan could be faster. The speed of LRU is expected to be similar to ours. Our training speed is notably faster than S4 because of the I/O bottleneck caused by Torch's FFT. We used the official S4 implementation that employs the Cauchy kernel. The speed of S4 is determined by its official code implementation.\n\n**Q13: More citations such as LSSL, S4D, etc.**\n\nA13: We will add the references in the reversion versions.\n\n**Q14: Comparation to H3.**\n\nA14: Please Refer to R4 A1.\n\n**Q15: $g_t$ is introduced here without explanation**.\n\nA15: We will address this issue in the reversion versions.\n\n**Q16: Do not use the phrase \"final formulation\".**\n\nA16: We will address this issue in the reversion versions.\n\n**Q17: Question aboud ON-LSTM.**\n\nA17: The relevance lies in its introduction of the `cummax` function. However, our utilization of the `cummax` function differs, as explicated in line 158: our objective is to drive the forget gate values of higher layers towards one, in contrast to the desire for multiscale forget gate values within a single layer in ONLSTM.\n\n**Q18: Typos.**\n\nA18: We will correct the typo in the revised version.\n\nCitations:\n[1] Albert Gu, Caglar Gulcehre, Thomas Paine, Matt Hoffman, Razvan Pascanu; Proceedings of the 37th International Conference on Machine Learning, PMLR 119:3800-3809\n[2]  Zhen Qin, Xiaodong Han, Weixuan Sun, Bowen He, Dong Li, Dongxu Li, Yuchao Dai, Lingpeng Kong, and Yiran Zhong. Toeplitz neural network for sequence modeling. In The Eleventh International Conference on Learning Representations, 2023.\n[3] Eric Martin and Chris Cundy; Parallelizing Linear Recurrent Neural Nets Over Sequence Length; In The Six International Conference on Learning Representations, 2018."}]

Author rebuttal:
Thank all reviewers for your feedback. We remark that our model is conceptually simple, aligning with the well-established lineage of gated RNN research while obtaining SOTA performance in language modeling. We recognize that the significance of the forget gate has been somewhat overlooked within the current literature on linear RNNs. Our contribution seeks to fill this void, providing the community with a broader and more enlightening point of view. In response to your valuable suggestions, we have also conducted several large-scale language modeling experiments: 125M HGRN on Wiki103 and 1B HGRN on The Pile. Our experiments have demonstrated the significant potential of our work, providing further evidence of its practical applicability and relevance. We acknowledge the concern raised about the presentation quality of our paper. In the revised version, we will enhance the quality of the presentation.

Meta-review: This work proposes an approach to improve linear RNNs with complex forget gate values and learnable lower bounds on the forget gate for each layer. This enables the model to capture both short and long-range dependencies. The new HGRU architecture is evaluated on Wikitext, LRA and Imagenet-1K and thorough ablations are performed on the architecture design. This work will likely impact the architecture field and expand the modern day use-cases for RNNs.

Strengths:
- The architecture has the representational power of an LSTM but with the fast one-step-per-layer backprop speed of the Transformer.
- The inductive bias in the architecture allows it to be more data efficient than the Transformer and so can supercede the Transformer for small/medium sized datasets.
- Linear SSMs and RNNs have gained significant attention. This paper advances that by looking at the linear RNN dynamics matrix from the view of the LSTM forget gate.
- The method achieves strong results against baselines including the Transformer. The new large 1B results the authors added in the rebuttal indicate the architecture has the potential to scale.
- The ablations help the reader understand different design decisions.
- Code is included.

Weaknesses:
- The improvement to perplexity from adding the forget gate to LRU seems small and so the HGRN seems more dependent on the forget gate.
- It would be good to cite and include results from related work such as S5 and SgConv/MEGA, which outperforms this architecture in LRA.
- There are issues with the presentation, especially missing formatting.
Decision: Accept (spotlight)