SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We adopt the same training configuration for all competitors, including batch size, learning rate, training epochs or iterations, etc. We list detailed hyper-parameters in the Appendix. For the autoregressive language modeling, we conducted three sets of experiments. Firstly, we validated the performance of two different-scale models on the Wikitext-103 dataset. We used the TNN configuration to verify the performance of the model at around  $44\mathrm{m}$ , and the Hyena configuration to verify the performance of the model at around  $125\mathrm{m}$ . To evaluate the performance of larger-scale models, we trained a 1b Transformer and HGRN on the Pile dataset using 10b tokens. To assess the performance in downstream tasks, we trained HGRN models of  $150\mathrm{m}$ ,  $350\mathrm{m}$ , and 1b on the Pile dataset using 100b tokens and conducted zero-shot evaluations on downstream tasks.
For the LRA benchmark, We report results on all 6 tasks. For the image classification on the ImageNet-1k dataset, We integrate HGRN into the DeiT [79] structure, we replace the transformer layers with our HGRN modules. It is compared to the performance of the vanilla DeiT on the ImageNet-1K dataset for image classification.
Table 4: Performance Comparison on Commonsense Reasoning. PS: parameter size (billion). T: tokens (billion). HS: HellaSwag. WG: WinoGrande.
Model	Params	Token	BOOLQ	PIQA	HS	WG	ARC-e	ARC-c	OBQA	AVG
GPT-Neo	0.13	300	61.71	63.06	30.40	50.43	43.73	23.12	26.20	42.66
OPT	0.16	300	55.47	62.95	31.35	50.43	43.52	22.70	28.00	42.06
Pythia	0.16	300	55.08	61.32	30.16	51.93	43.18	23.12	26.80	41.66
RWKV	0.17	-	-	65.07	32.26	50.83	47.47	24.15	29.60	41.56
HGRN	0.15	100	59.91	65.02	33.33	50.20	46.68	23.81	28.60	43.94
OPT	0.35	300	57.74	64.58	36.69	52.49	44.02	23.89	28.20	43.94
Pythia	0.4	300	60.40	67.08	40.52	53.59	51.81	24.15	29.40	46.71
BLOOM	0.56	350	55.14	64.09	36.97	52.80	47.35	23.98	28.20	44.08
RWKV	0.43	-	-	67.52	40.90	51.14	52.86	25.17	32.40	45.00
HGRN	0.35	100	59.05	66.70	38.12	51.70	49.20	25.26	30.60	45.80
GPT-Neo	1.3	300	61.99	71.11	48.93	54.93	56.19	25.85	33.60	50.37
OPT	1.3	300	57.77	71.71	53.70	59.35	57.24	29.69	33.20	51.81
Pythia	1.4	300	60.73	70.67	47.18	53.51	56.99	26.88	31.40	49.62
BLOOM	1.1	350	59.08	67.14	42.98	54.93	51.47	25.68	29.40	47.24
RWKV	1.5	-	-	72.36	52.48	54.62	60.48	29.44	34.00	50.56
HGRN	1	100	58.69	70.89	48.02	51.62	55.64	27.90	31.60	49.19
Autoregressive Language Modeling Autoregressive language modeling stands as a prominent task within the field of natural language processing, as it serves as a measure of a language model's causal inference capability. This task requires the model to estimate the probability distribution of the subsequent token based on the previously seen tokens.
We show the performances of the autoregressive language modeling in table 1 and table 2. Compared to transformer-based methods, HGRN performs favourably than most efficient variants of the vanilla transformer such as FLASH [31], 1+elu [35], Performer [7] and cosFormer [62].
Also, HGRN achieves better results than the MLP-based methods with a notable margin. Nevertheless, HGRN performs similarly to the original transformer [81]. Finally, HGRN shares similar concepts with RNN-based such as S4 [22], DSS [26], GSS [49], RWKV [55], and LRU [53], our HGRN also achieves superior performance to all RNN-based methods. This provide evidence HRGN may be an effective method in LM We also report the extrapolation ability of HGRN compared to previous methods in Table 14.
Table 2: Results on Wikitext-103 (Hyena[57]'s setting). All models are in GPT-2 small size (125M).  $\downarrow$  means lower is better
Model	PPL↓
Transformer [57]	18.6
Hybrid H3 [57]	18.5
Performer [57]	26.8
Reformer [57]	25.6
AFT-conv [57]	28.2
Linear Attention [57]	25.6
Hyena [57]	18.6
Hyena-slim [57]	18.5
HGRN	18.6
We also trained a 1b model on the Pile dataset and compared it with LRU and Transformer. Specifically, our training parameters included a sequence length of 1024, batch size of 96, 100k updates, and a learning rate of 5e-4. It can be seen that HGRN still performs better at the 1b scale. Additionally, we trained 100b tokens of
Table 3: Results on the Pile. All the model size is 1b. The lower the better.
Model	PPL↓
Transformer	4.56
LRU	5.07
HGRN	4.14
HGRN on the Pile dataset at  $150\mathrm{m}$ ,  $350\mathrm{m}$ , and 1b sizes, and evaluated them against open-source Transformer-based models in downstream tasks. We selected Comparison on Commonsense Reasoning and Super GLUE tasks, and all evaluations were done using the lm-evaluation-harness [16]. HGRN achieves comparable performance to Transformer-based models when consuming only 1/3 of the tokens.
Long Range Arena LRA [77] is proposed as a comprehensive evaluation for assessing the performances of models in processing long-term dependencies in various sequential modeling tasks. We show a performance comparison between HGRN and existing methods in Table 6. HGRN achieves comparable results with other SOTA methods.
Table 5: Performance Comparison on Super GLUE.. PS: parameter size (billion). T: tokens (billion).
Model	Params	Token	WSC	WIC	RTE	CB	MULTIRC	BOOLQ	COPA	AVG
GPT-Neo	0.13	300	36.54	50.00	54.87	41.07	0.84	61.71	64.00	44.15
OPT	0.16	300	36.54	50.00	49.82	21.43	1.36	55.47	66.00	40.09
Pythia	0.16	300	36.54	50.16	52.71	41.07	2.52	55.08	65.00	43.30
HGRN	0.15	100	38.46	51.10	56.68	42.86	1.47	59.91	65.00	45.07
OPT	0.35	300	36.54	50.00	51.99	46.43	1.36	57.74	72.00	45.15
Pythia	0.4	300	57.69	50.31	52.71	35.71	1.68	60.40	70.00	46.93
BLOOM	0.56	350	40.38	50.00	52.71	41.07	1.05	55.14	61.00	43.05
HGRN	0.35	100	38.46	50.16	52.71	51.79	1.99	59.05	73.00	46.74
GPT-Neo	1.3	300	36.54	50.00	60.29	44.64	1.99	61.99	69.00	46.35
OPT	1.3	300	37.50	51.10	51.99	41.07	3.15	57.77	79.00	45.94
Pythia	1.4	300	36.54	50.00	53.07	35.71	0.94	60.73	72.00	44.14
BLOOM	1.1	350	36.54	50.00	52.71	41.07	0.73	59.08	68.00	44.02
HGRN	1	100	40.38	50.78	53.43	42.86	3.04	58.69	70.00	45.60
Table 6: Performances Comparison on the Long Range Arena benchmark. The proposed HGRN achieves the best performances and outperforms all competing methods.
Model	ListOps	Text	Retrieval	Image	Pathfinder	Path-X	AVG.
Transformer [81]	38.37	61.95	80.69	40.57	65.26	-	47.81
cosFormer [62]	36.50	67.70	83.15	51.23	71.96	-	51.76
FLASH [31]	38.70	64.10	86.10	47.40	70.25	-	51.09
S4 [22]	59.60	86.82	90.90	88.65	94.20	96.35	86.09
DSSSoftmax [26]	60.60	84.80	87.80	85.70	84.60	87.80	81.88
DSSEXP [26]	59.70	84.60	87.60	84.90	84.70	85.60	81.18
DSSEXP-NO-SCALE [26]	59.30	82.40	86.00	81.20	81.30	-	65.03
TNN [59]	61.04	87.90	90.97	88.24	93.00	96.10	86.21
S5 [71]	62.15	89.31	91.4	88	95.33	98.56	87.46
Mega [46]	63.14	90.43	91.25	90.44	96.01	97.98	88.21
SGConv [41]	61.45	89.2	91.11	87.97	95.46	97.83	87.17
LRU [53]	60.20	89.40	89.90	89.00	95.10	94.20	86.30
HGRN	59.95	88.14	94.23	88.69	92.92	97.50	86.91
Image Classification The image classification results on the ImageNet-1K dataset are presented in Table 7. Notably, with comparable parameter sizes, our proposed HGRN model demonstrates superior performance compared to previous methods such as TNN and the vanilla transformer. It demonstrates the capability of HGRN in modeling visual modalities.
Table 7: Performances comparison of image classification on ImageNet-1k. HGRN performs favorably than competing methods with similar parameter sizes.
DeiT-Tiny	DeiT-Small
Model	Top1 Acc	Param (M)	Top1 Acc	Parma (M)
Deit	72.20	5.7	79.90	22.0
TNN	72.29	6.4	79.20	23.4
HGRN	74.40	6.1	80.09	23.7
We conducted ablation studies in the smallest-scaled setting (i.e., TNN[59]) setting on WikiText103 dataset) to thoroughly verify the effectiveness of each of our proposed components in HGRN. Experiments were conducted on the Pile dataset using a 1b model with 10b tokens for the forget gate experiment.
Table 8: Forget gate ablation on an autoregressive language model. The only lower bound means using a data-independent gate like LRU.
Model	PPL↓
LRU w forget gate	4.92
LRU	5.07
HGRN only lower bound	4.84
HGRN w/o forget gate	57.42
HGRN	4.14
(i) With lower bound
(ii) Without lower bound
Figure 2: Visualization of forget rates. We plot the forget rates of layers 5 and 6 on a model trained on language modeling tasks.
(iii) LRU
The influence of forget gate In table 8, we demonstrate the role of forget gate. From table 8, we observe that removing the forget gate significantly decreases the performance of HGRN, while adding a forget gate to LRU improves performance. On the other hand, using a data-independent forget gate (only lower bound) leads to lower performance compared to a data-dependent forget gate.
The influence of input gate and output gate Table. 9 validates the effectiveness of using output gates and tying input and forget gates. w/o input gate means to remove the  $1 - \lambda_{t}$  term. w/o output gate means remove the left branch of HGRN in figure 1. Our design achieves the best performance.
Table 9: Ablations of gates on autoregressive language modeling. w/o input gate means to remove the  $1 - \lambda_{t}$  term. w/o out_gate means remove the left branch of HGRU in figure 1.
Model	PPL↓
w/o input gate	25.03
w/o output gate	25.50
HGRN	24.14
The influence of lower bounds in forget gate
values We demonstrate the effectiveness of introducing a lower bound in Table 10 and Table 13. From Table 10, we observe that gating (i.e., without lower bound) is more critical than the lower bound (i.e., only lower bound). Combining gating and the lower bound consistently provides benefits, but the most significant improvement arises from the monotonically increasing lower bound. This aligns with the intuition that lower layers should primarily focus on nearby tokens, while upper layers should attend more broadly to capture long-term dependencies [58].
Table 13 highlights the essential role of the lower bound in long sequence processing tasks. In these tasks, the model's performance is notably poor and sometimes fails to converge without the lower bound. It is worth noting that language modeling tasks do not require extensive long-term dependencies, which explains why the model performs well even without the lower bound. However, in the task of LRA, the ability to capture long-term dependencies is crucial for achieving satisfactory performance.
The influence of complex-valued recurrence
Table 11 validates the utility of incorporating
Table 10: Lower bound ablation on autoregressive language modeling. A random lower bound means the lower bound in each layer is independent. Decrease lower bound means the lower bound is monotonically decreasing with respect to layer  $k$ , only the lower bound means the forget rate is independent of input.
Model	PPL ↓
w/o lower bound	24.71
random lower bound	24.60
decrease lower bound	24.63
only lower bound	27.70
HGRN	24.14
complex values in element-wise linear recurrence. Additionally, the experiments show that the phase argument  $\theta$  should not be data-dependent.
4.4 Analysis on forget gate values
We present the distributions of forget gate values across layers for different methods in Table 12 and visualize the histogram of each layer in Figure 2, trained on the autoregressive language modeling task. The results demonstrate that the addition of lower bounds effectively increases the average forget gate values in higher layers (5-6). Notably, the medium forget gate values in the highest layer reach 0.98, enabling the modeling of long-term dependencies.
Table 11: Ablations of complex-valued recurrence on autoregressive language modeling. w/o complex means remove theta, data-dependent theta means theta is dependent on the input, this makes the matrix  $\Lambda$  not a Toeplitz matrix, which can not capture relative information.
Model	PPL↓
w/o complex	25.34
data dependent θ	28.74
HGRN	24.14
It is interesting to note that the average forget
gate values of the LRU model consistently exceed those of our variant model without lower bounds, as per their eigenvalues. However, despite this, the language modeling performance of LRU is lower than that of our variant. Specifically, LRU scored 24.71, while our variant scored 31.12. This suggests that using data-dependent gates to selectively retain relevant information is advantageous, rather than relying on data-independent forget gate values across all time steps.
Table 12: Forget gate values of different methods on language modeling tasks. In each layer, we counted the mean and median of forget gate values.
ours	ours	w/o lower bound	w/o lower bound	LRU	LRU
Layer	mean	median	mean	median	mean	median
1	0.48	0.47	0.52	0.50	0.75	0.72
2	0.55	0.52	0.59	0.55	0.78	0.75
3	0.60	0.57	0.58	0.56	0.78	0.76
4	0.68	0.64	0.58	0.55	0.79	0.78
5	0.79	0.80	0.63	0.63	0.79	0.77
6	0.91	0.98	0.63	0.67	0.79	0.79
Table 13: Lower bound ablation on LRA. We verify the importance of lower bounds in long-sequence modeling capabilities.
Model	ListOps	Text	Retrieval	Image	Pathfinder	Path-X	AVG
w/o lower bound	51.41	87.79	88.71	80.17	-	-	51.53
HGRN	59.95	88.14	94.23	88.69	92.92	97.50	86.91
In this work, we have shown that gated linear RNNs could obtain impressive performance across different tasks and modalities without compromising efficiency. We highlighted the significance of the forget gate for linear RNNs in language modeling and emphasized the importance of an additive lower bound on forget gate values for modeling long-term dependencies.
This work is partially supported by the National Key R&D Program of China (NO.2022ZD0160100).
Limitations and broader impact
Our empirical evaluation of HGRN remains on a smaller scale compared to other large-scale models. Potentially negative social consequences include the misuse of brain models for unsuitable purposes or applications, which must be prohibited by appropriate rules. In the era of large language models, the inference cost is the key limitation of transformer-based models. RNNs provide a solution with their lower inference costs. This could potentially lead to a significant evolution in the field.
transformers. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
[79] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herve Jegou. Training data-efficient image transformers &amp; distillation through attention. In International Conference on Machine Learning, volume 139, pages 10347-10357, July 2021.
[80] Jos van der Westhuizen and Joan Lasenby. The unreasonable effectiveness of the forget gate. CoRR, abs/1804.04849, 2018.
[81] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[82] Junxiong Wang, Jing Nathan Yan, Albert Gu, and Alexander M. Rush. Pretraining without attention. CoRR, abs/2212.10544, 2022.
[83] Weihao Yu, Mi Luo, Pan Zhou, Chenyang Si, Yichen Zhou, Xinchao Wang, Jiashi Feng, and Shuicheng Yan. Metaformer is actually what you need for vision. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 10809-10819. IEEE, 2022.
[84] Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 11106-11115, 2021.
[85] Chen Zhu, Wei Ping, Chaowei Xiao, Mohammad Shoeybi, Tom Goldstein, Anima Anandkumar, and Bryan Catanzaro. Long-short transformer: Efficient transformers for language and vision. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021.
In this appendix, we examine the extrapolation ability of HGRN and provide the training and inference speed comparison of HGRN and existing efficient sequence modeling methods. We also illustrate the forget rates of each layer on a trained language model of HGRN.
We also report the extrapolation ability of HGRN compared to previous methods in Table 14.
6.1 Extrapolation test
In this section, we tested HGRN's extrapolation ability by directly inferring the model with a variety of sequence lengths. As shown in Table 14, our method has the ability to train short and test long.
6.2 Speed comparison
In this section, we benchmark the speed of our method on the LRA benchmark. Our method achieves state-of-the-art training and inference speed.
6.3 Visualization
In this section, we visualize the forget rates of each layer on a model trained on language modeling tasks.
6.4 Configurations
We list detailed hyper-parameters of our experiments here.
Figure 3: Visualization forget rates in each layer.
(iii) LRU
(i) Layer 1
Figure 4: Visualization of token mixing matrix in each layer.
(ii) Layer 1
(iii) Layer 2
(iv) Layer 4
(v) Layer 5
(vi) Layer 6
Table 14: The extrapolation performance of competing methods. The best result is highlighted in bold and the second in underline.  $\downarrow$  means lower is better.
Seqlen	Transformer PPL↓	LS PPL↓	FLASH PPL↓	1+elu PPL↓	Performer PPL↓	cosFormer PPL↓	gMLP PPL↓	S4 PPL↓	DSS PPL↓	GSS PPL↓	ALiBi PPL↓	TNN PPL↓	LRU PPL↓	HGRU PPL↓
512	24.78	24.05	24.69	28.05	63.16	27.06	29.13	30.74	41.07	39.66	24.15	24.67	31.12	24.85
768	41.36	23.49	16950.45	47.35	159.74	32.90	1.34E+9	30.41	40.50	39.76	23.38	24.25	30.72	24.4
1024	62.35	23.21	174165.47	70.47	504.30	55.28	8.93E+12	30.24	40.22	39.91	22.98	24.05	30.5	24.16
1280	82.52	23.07	346502.88	91.88	1020.28	102.88	1.58E+15	30.15	40.03	40.82	22.74	23.91	30.38	24.03
1536	100.17	22.97	647788.12	111.56	1568.83	175.26	4.96E+16	30.08	39.94	41.04	22.57	23.83	30.3	23.94
1792	118.42	22.97	1719873.5	129.92	2138.50	267.65	5.67E+17	30.04	39.85	41.08	22.52	23.79	30.24	23.88
2048	133.44	22.99	6.25E+6	147.09	2693.89	368.02	3.59E+18	30.00	39.79	41.53	22.43	23.73	30.19	23.82
3072	188.95	23.25	4.17E+10	206.88	4945.82	820.77	2.19E+20	29.91	39.64	44.08	22.24	23.63	30.09	23.71
4096	246.06	23.83	2.67E+13	267.87	7170.91	1335.51	1.61E+21	29.88	39.59	48.27	22.17	23.58	30.04	23.66
5120	270.93	24.56	1.26E+15	299.31	8443.15	1735.50	5.08E+21	29.85	39.54	53.32	22.11	23.54	30.01	23.62
6144	311.65	25.45	1.58E+16	352.62	10234.07	2146.19	1.16E+22	29.83	39.51	57.73	22.08	23.53	29.99	23.6
7168	346.58	26.42	8.11E+16	389.02	11420.56	2494.79	1.98E+22	29.82	39.49	60.25	22.07	23.51	29.97	23.58
8192	372.18	27.11	3.40E+17	411.50	12557.09	2902.24	2.78E+22	29.82	39.49	63.36	22.05	23.51	29.97	23.58
9216	387.29	28.78	1.22E+18	453.27	14847.66	3028.72	3.93E+22	29.80	39.46	74.92	22.03	23.49	29.96	23.56
10240	395.94	30.13	4.03E+18	457.06	13623.83	3247.83	4.93E+22	29.79	39.45	81.87	22.02	23.48	29.94	23.55
11264	426.54	31.14	1.07E+19	504.19	14661.77	3341.91	5.70E+22	29.79	39.46	87.67	22.00	23.48	29.94	23.55
12288	463.50	33.21	2.52E+19	555.38	17959.85	3644.81	7.18E+22	29.79	39.44	92.11	22.00	23.48	29.94	23.55
13312	506.35	34.72	4.96E+19	584.01	20026.35	3851.70	8.04E+22	29.78	39.43	96.00	22.00	23.47	29.93	23.54
14336	486.86	36.05	1.28E+20	589.83	20971.31	3951.26	9.41E+22	29.78	39.43	101.47	21.99	23.46	29.92	23.53
Avg	261.36	26.71	1.16E+19	299.86	8684.79	1764.75	2.41E+22	29.97	39.75	60.26	22.40	23.70	30.17	23.80
Table 15: Speed comparison on LRA benchmark. The 1K,...,5K represent the input sequence length. We mark it with - if a method is out of memory. The higher the better for all metrics.
Train Speed(steps per second)↑	Inference Speedsteps per second)↑
Method	1K	2K	3K	4K	5K	1K	2K	3K	4K	5K
Transformer [81]	13.58	4.84	-	-	-	23.67	8.22	-	-	-
Performer [36]	18.40	10.77	7.66	6.30	5.64	30.04	17.36	12.80	10.55	9.52
LS [85]	20.29	11.24	8.05	6.51	5.89	39.05	21.11	15.02	12.6	11.66
Fnet [39]	25.19	15.62	11.24	9.41	8.18	48.81	27.89	19.52	16.27	14.46
cosFormer [62]	22.00	12.80	9.47	7.93	7.13	39.05	22.31	16.62	13.95	12.60
S4 [21]	13.13	7.33	4.91	3.84	3.04	30.04	16.27	10.85	8.58	6.79
FLASH [31]	17.36	9.03	6.54	5.19	4.68	30.04	15.94	11.32	9.19	8.40
TNN [59]	17.55	9.89	6.79	5.68	4.54	33.96	17.75	12.40	10.28	8.22
HGRU	22.31	13.58	9.52	7.40	7.44	43.39	25.19	16.62	14.20	13.95
Table 16: Detailed training configurations used in our experiments. "Total batch size" means batch_per_gpu  $\times$  update_freq  $\times$  num_gpus. "ALM" stands for Autoregressive Language Model. "IM" stands for Image Modeling.
AML	IM
Data	WikiText-103	ImageNet-1k
Tokenizer method	BPE	-
Src Vocabulary size	50265	-
Sequence length	512	-
Total batch size	128	2048
Number of updates/epochs	50k updates	300 epochs
Warmup steps/epochs	4k steps	5 epochs
Peak learning rate	5e-4	2.5e-4
Learning rate scheduler	Inverse sqrt	cosine
Optimizer	Adam	Adamw
Adam ε	1e-8	1e-8
Adam (β1, β2)	(0.9, 0.98)	(0.9, 0.98)
Weight decay	0.2	0.1
Gradient clipping	-	1.0