Recurrent Attention Networks for Long-text Modeling
Year: 2023

Abstract:
Self-attention-based models have achieved remarkable progress in short-text mining. However, the quadratic computational complexities restrict their application in long text processing. Prior works have adopted the chunking strategy to divide long documents into chunks and stack a self-attention backbone with the recurrent structure to extract semantic representation. Such an approach disables parallelization of the attention mechanism, significantly increasing the training cost and raising hardware requirements. Revisiting the self-attention mechanism and the recurrent structure, this paper proposes a novel long-document encoding model, Recurrent Attention Network (RAN), to enable the recurrent operation of self-attention. Combining the advantages from both sides, the welldesigned RAN is capable of extracting global semantics in both token-level and documentlevel representations, making it inherently compatible with both sequential and classification tasks, respectively. Furthermore, RAN is computationally scalable as it supports parallelization on long document processing. Extensive experiments demonstrate the long-text encoding ability of the proposed RAN model on both classification and sequential tasks, showing its potential for a wide range of applications.

heading: 1 Introduction
text: Recently, self-attention-based neural networks, such as Transformer (Vaswani et al., 2017), GPT (Radford et al., 2018, 2019; Brown et al., 2020), and BERT family (Devlin et al., 2019; Liu et al., 2019; Lan et al., 2020), have demonstrated superior text encoding ability in many natural language processing (NLP) tasks with the help of large-scale pretraining. These models have set state-of-the-art benchmarks in classification tasks like text categorization (Li et al., 2021a) and sentiment analysis (Naseem et al., 2020; Li et al., 2021c, 2023), and
∗Corresponding author; † Equal contribution.
sequential tasks like question answering (Lee et al., 2019; Karpukhin et al., 2020) and information extraction (Li et al., 2021b; Wu et al., 2022). The time and space complexities of self-attention computation are O(n2) with respect to the sequence length, making it computationally expensive to encode long texts. Therefore, BERT models adopt an absolute positional encoding strategy to manage computational overhead. However, such a setting makes the BERT models unable to handle texts longer than 512 tokens, restricting their application in realistic scenarios like processing user comments, news articles, scientific reports, and legal documents with arbitrary lengths.
Current works focus on two solutions to enable self-attention-based models for handling longer texts. The first solution reduces the computing complexity of self-attention from quadratic to linear by approximating its softmax operation (Beltagy et al., 2020; Choromanski et al., 2021; Hua et al., 2022). These models can handle relatively long texts within the hardware capacity but also suffer from a performance drop (Schlag et al., 2021; Hutchins et al., 2022). Another solution is to divide the long document into chunks shorter than 512 tokens so that pretrained BERT models can be applied (Pappagari et al., 2019; Hutchins et al., 2022). However, as the chunks are individually encoded, the resulted representations do not contain the crucial contextual information for sequential tasks. While a special recurrent mechanism can handle sequential tasks (Hutchins et al., 2022), it cannot produce a document-level representation for classification, limiting their generality as none works for both classification and sequential tasks. Additionally, introducing recurrent modules disables the parallel computing feature, leading to unscalable implementation.
To address the aforementioned issues, this paper proposes the Recurrent Attention Network (RAN)1,
1The code is available at https://github.com/4AI/RAN.
3006
a novel model architecture supporting recurrent self-attention operation over long sequences, enabling global dependency extraction and long-term memory. RAN iterates through the sequence by non-overlapping windows. Unlike token-level recurrent architectures such as LSTM (Hochreiter and Schmidhuber, 1997) and GRU (Chung et al., 2014), RAN applies positional multi-head selfattention (pMHSA) on a window area to extract local dependency. To propagate the information forward, the RAN model extracts the global perception cell (GPC) vector from the self-attention representation of the current window. The GPC vector is then concatenated with tokens in the next window as the input of the self-attention layer. The new GPC vector will be passed to the subsequent windows with residual connection to alleviate the gradient vanishing (He et al., 2016) and updated in the same manner. Figure 1 depicts the difference between the recurrent neural network (RNN) and our proposed RAN.
The function of the GPC vector is twofold. First, like the [CLS] token in BERT, the GPC vector is a window-level contextual representation. But unlike the [CLS] token, the GPC vector is only applied to the self-attention layer, and no special token is inserted during text preprocessing. Second, the GPC vector, resembling the state cell in a recurrent architecture, maintains a long-distance memory over the sequence. For each window, the attended GPC vector encodes an aggregated representation of all the previous windows, which enables the window-level self-attention to perceive global semantics. With the help of a well-designed memory review mechanism, the GPC vector from the last window can be used as a document-level
representation and serve the classification tasks. Meanwhile, the memory review mechanism enhances the token representations of RAN in the sequence, encoding both contextual and global information, which can be leveraged for sequential tasks such as language modeling (LM) and named entity recognition (NER).
We pretrain the RAN model using a masked language modeling (MLM) objective from scratch, which outperforms other pretrained baselines in long document classification. The RAN framework also supports auto-regressive LM and achieves the lowest perplexity score compared with state-of-theart language models on the WikiText-103 dataset. Furthermore, we apply RAN to different downstream tasks via finetuning and observe consistent improvements compared to baseline models.
RAN solely relies on self-attention, and no LSTM-style gate is involved when propagating information via GPC vectors. Therefore, RAN is computationally efficient as it supports parallelized GPU computing. Although the memory complexity is still quadratic, it is regarding the window size W rather than the whole text length L, where W ≪ L. Nevertheless, the window size can be adjusted based on hardware availability to achieve a relatively larger batch size for better training.
In summary, our contribution is to devise the RAN model for long document processing. RAN allows for parallelization on GPU and provides the interfaces for serving both classification and sequential tasks. With pretraining, RAN can outperform the BERT-based models in various tasks.

heading: 2 Related Work
text: This section reviews the relevant works focusing on sequence modeling in NLP, especially long document processing. RNNs are widely used for sequential modeling by recursively updating a state cell to maintain a long-distance memory. Traditional recurrent networks, such as LSTM (Hochreiter and Schmidhuber, 1997) and GRU (Chung et al., 2014), use the fully-connected layer as the basic encoding unit and apply the gate mechanism to update state memory. The recurrent operation is conducted on the token level, which is inefficient as such a framework cannot compute parallelly on GPU. Besides, it might suffer from gradient vanishing for long sequences during the backpropagation phase (Hutchins et al., 2022).
Self-attention models are powerful in global
representation learning. However, applying selfattention in long document processing is intractable due to the quadratic time and memory complexities. To address this issue, some works (Beltagy et al., 2020; Choromanski et al., 2021; Hua et al., 2022) attempt to reduce the computing complexity of self-attention from quadratic to approximately linear complexity. Beltagy et al. (2020) propose a drop-in replacement of the softmax operation in self-attention with a sparse attention mechanism. Similarly, Choromanski et al. (2021) rely on prior knowledge like sparsity and low-rankness to efficiently estimate the full-rank attention. However, these approaches face a trade-off between efficiency and accuracy, as approximations may lead to a performance drop (Schlag et al., 2021; Hutchins et al., 2022).
Other works leverage the power of full-rank selfattention as backbones, such as pretrained BERT and RoBERTa. These works cope with the tokenlength limitation with different strategies. Ding et al. (2020) propose CogLTX framework to generate a brief summary of the document. The short summary is used for the classification task employing BERT. However, it is inevitable to lose information in the length compression. Pappagari et al. (2019) segment the long text into smaller chunks so that BERT can be then used. A recurrent layer is employed to obtain the document-level representation upon chunk-level representations. These models can be applied for the classification task but cannot handle sequential tasks because of losing crucial contextual and sequential information. Hutchins et al. (2022) adopt the chunking strategy and devise a specifically-designed gate mechanism to obtain token-level representations for sequential tasks. Similarly, Didolkar et al. (2022) propose a Transformer-based temporal latent bottleneck for image classification, reinforcement learning, and text classification, in which temporal states are updated using a recurrent function across chunks. In each Transformer block, temporal states update the chunk-level representation by cross-attention layers interleaved with self-attention layers. In general, these models with BERT backbones cannot simultaneously handle classification and sequential tasks for long documents. Meanwhile, the RNN-style gate architecture does not support parallel computing, so the computing efficiency is also impaired.
Our proposed RAN achieves recurrent operation of the self-attention model and hence supports par-
allelization. Moreover, like the traditional RNN architecture, RAN can produce both token-level and document-level representations, which can be leveraged for both sequential and classification tasks.

heading: 3 Methodology
text: This section introduces the proposed RAN framework in terms of its components. Figure 2 depicts the structure of the basic RAN module. In RAN, the primary encoder is the pMHSA, encoding the GPC vector and the current input with the rotary positional information carried (Su et al., 2021). The GPC vector is employed to propagate information through the sequence.

heading: 3.1 Input Layer
text: We first employ the padding operation for the input documents to keep a uniform length L. Then we map each word into a D-dimensional continuous space and obtain the word embedding xi ∈ RD. The word vectors are concatenated to form the model input: X = [x1,x2, . . . ,xL] ∈ RL×D. To feed the text into the RAN, we chunk the input document into m = ceil( LW ) windows, where W is the window size. We use Xi ∈ RW×D to denote the i-th window input. In RAN, the GPC vector G0 ∈ RD is initialized to 0 by default, following the common operation in RNN. We parameterize the GPC vector with the layer normalization (Ba et al., 2016) as follows:
G0 = LayerNorm(WgG0) ∈ RD. (1)

heading: 3.2 Positional Multi-Head Self-Attention
text: As the positional space of long documents is prohibitively large, it is not feasible to use absolute positional embedding following Transformer families (Vaswani et al., 2017) in long document processing. Hence, we follow Su et al. (2021); Chowdhery et al. (2022); Black et al. (2022) to incorporate the local positional information of the current window and leverage rotary position information as follows:
pMHSA(Xi) = W[Att1(Xi); ...; Atth(Xi)] + b, (2) and
Attj(Xi) = SoftMax( RP(Qj) · RP(KTj )√
dk
+M)Vj
Qj = W q jXi + b q j Kj = W k jXi + b k j Vj = W v jXi + b v j ,
(3)
where Attj(·) is the j-th head of pMHSA, h denotes the head size, [; ] means the concatenation operation, M is the attention mask to adapt to different sequential tasks, and RP(·) stands for the rotary position function (Su et al., 2021).

heading: 3.3 Encoding and Updating Layer
text: To encode the i-th window, we concatenate the GPC vector from the previous window Gi−1 and the current window input Xi ∈ RW×D to form the model input Xini = [Gi−1;Xi] ∈ R(1+W )×D. A layer normalization layer is applied to normalize the input.
We then apply the pMHSA to encode the concatenated input to obtain the outputs of the current window:
Oi = pMHSA(X in i ). (4)
After encoding, we extract the updated GPC vector G′i and the output corresponding to the tokens in the window:
G′i = SN(O [1:2] i ) ∈ RD
Owi = SN(O [2:1+W ] i ) ∈ RW×D,
(5)
where [start:end] is the tensor slice operation, and SN(X) = X−Xmeanσ stands for the standard normalization. To alleviate the gradient vanishing issue in modeling long sequences, we employ residual connection to connect the current GPC vector with the previous one, then pass it to a layer normalization layer to normalize the updated GPC vector,
Gi = LayerNorm(G ′ i +Gi−1). (6)
The updated GPC vector Gi ∈ RD will be propagated to the next window.

heading: 3.4 Memory Review and Output Layer
text: After encoding all windows, we can obtain the sequence output by concatenating all window outputs, as follows:
Ow = [Ow1 ;O w 2 ; ...;O w m] ∈ RL×D, (7)
where m is the number of windows. Ow has the same shape as the input X. To prevent history forgetting in handling long sequences, this paper proposes a novel memory review mechanism. Specifically, we first concatenate all updated GPC vectors to produce the history states vector:
S = [G1;G2; ...;Gm] ∈ Rm×D. (8) We compute the cross attention of the concatenated output and the historical memory states to obtain the final output:
O = SoftMax( QKT√
dk )V
Q = WqOw + bq
K = WkS+ bk
V = WvS+ bv.
(9)
This procedure mimics the human behavior of reviewing key points after reading an article, the way that humans naturally consolidate information and reinforce memory.
The sequence output O ≡ Oseq ∈ RL×D can be used for sequential tasks like NER. Although the GPC vector of the last window, Gm, can serve as the document representation, it may lose crucial semantics and long-term memory during the propagation. Therefore, we also add the memory review mechanism to RAN for classification tasks by generating Oclf :
Oclf = W gGm +W oOp + bo, (10)
where Op is the pooling of the output O over time sequence. Our empirical results show that the max pooling works better than the average pooling in classification tasks. Therefore, we adopt max pooling to obtain Op:
Op = MaxPooling(O). (11)
Figure 3 provides a visual illustration of the implementations for both classification and sequential tasks. It is noticeable that the model parameters of RAN are shared across all windows, allowing for efficient computation and reduced memory usage. Particularly, RAN supports multiple sequential tasks with different attention masks. For instance, it employs a causal attention mask (Vaswani et al., 2017) for LM tasks and a prefix causal attention mask (Dong et al., 2019) for the seq2seq tasks to prevent forward information exposure.

heading: 4 Experiment

heading: 4.1 Datasets and Evaluation Metrics
text: To comprehensively evaluate the model performance, we conduct experiments on three major tasks: text classification (TC), NER, and LM.
For the TC task, we attempt to test the model performance on datasets with various document lengths. Specifically, we extend the benchmarks from Park et al. (2022) by adding the long-text dataset Arxiv and the short-text dataset AGNews. The extended benchmarks include (1) AGNews2, (2) 20NewsGroups (Lang, 1995), and (3) Arxiv (He et al., 2019) for multi-class classification; (4) Book Summary (Park et al., 2022; Bamman and Smith, 2013) (abbr. B.S.) and (5) EURLEX-57K
2http://groups.di.unipi.it/~gulli/AG_corpus_o f_news_articles
(Chalkidis et al., 2019) (abbr. EUR.57K) for multilabel classification; and Hyperpartisan (Kiesel et al., 2019) (abbr. Hyper.) for binary classification. Figure 4 depicts the text length distribution and the long-text ratio of the benchmark datasets. For a fair comparison, following Park et al. (2022), we report micro-F1 for multi-label classification and accuracy for binary and multi-class classification.
For the LM task, we adopt the commonly-used dataset WikiText-1033 (Merity et al., 2017) and report the perplexity score following the baselines.
For the NER task, we experiment on two widelyadopted English datasets: OntoNotesV5.04 (abbr. OntoV5, average length is 77.5) and CoNLL2003 (Tjong Kim Sang and De Meulder, 2003) (average length is 63.4). Noted that both datasets consist of short texts with an average length shorter than 100, as there are no available NER datasets of long documents. Accordingly, we adopt a small window size for the NER task to test the effectiveness of the recurrent architecture. We use conlleval5 to measure the model performance and report the F1 score following the baselines.

heading: 4.2 Implementation Details
text: The primary experiments in Section 4.3 were conducted using the NVIDIA A100 GPU, while the remaining experiments were conducted using the NVIDIA Titan X GPU (12G memory). The code was implemented using TensorFlow and Keras. By default, we used two layers of RAN, with a window size of 256 for the TC and LM tasks and 64 for the NER task. The head number of pMHSA is set to 12, and the head size is 768. We trained the models for different tasks using the Adam optimizer (Kingma and Ba, 2015) by optimizing the corresponding objective function. For pretrained and non-pretrained RAN models, we set the learning rate to 2e−5 and 3e− 4, respectively.

heading: 4.3 Main Results

heading: 4.3.1 Long Text Classification
text: We compare RAN with baselines on the long-text classification task, including BiLSTM (Hochreiter and Schmidhuber, 1997) and pretrained language
3https://blog.salesforceairesearch.com/the-wik itext-long-term-dependency-language-modeling-dat aset
4https://catalog.ldc.upenn.edu/LDC2013T19 5https://www.clips.uantwerpen.be/conll2002/ne
r/bin/conlleval.txt
ahttps://nlp.stanford.edu/data/glove.6B.zip
models such as BERT (Devlin et al., 2019), Longformer (Beltagy et al., 2020), ToBERT (Pappagari et al., 2019), and CogLTX (Ding et al., 2020). For a comprehensive review, we adopt different initialization methods for RAN parameters. RAN+Random indicates the weights of RAN are randomly initialized. RAN+GloVe stands for using the GloVe embedding (Pennington et al., 2014) as word representation. RAN+Pretrain is the RAN pretrained on the MLM task, following settings in Devlin et al. (2019); Liu et al. (2019). We pretrained RAN on the BookCorpus (Zhu et al., 2015) (5GB) and C4 (Raffel et al., 2020) (RealNews-like subset, 15GB).
We present the results of long document benchmarks in Table 1. In general, the pretrained RAN achieves the five best results among the six bench-
marks except for the 20NG dataset and outperforms all the baselines regarding the average score. Note that the pretrained RAN has only 96M parameters which are fewer than other pretrained baselines, suggesting that RAN is more efficient and scalable than the baselines. Particularly, the pretrained RAN achieves a 2.2% improvement compared with ToBERT on the super-long text dataset Arxiv, demonstrating the superiority of RAN in handling long documents.
It is worth noticing that the average performance of RAN is higher than that of the chunking-based ToBERT and the document summarization model CogLTX. These two models drop essential information in the chunking and summarizing processes, while RAN can preserve the sequence information
with the help of the well-designed recurrent and memory review mechanisms. Moreover, the pretrained RAN achieves the best result on the shorttext dataset AGNews, indicating that RAN also performs well in short-text tasks.
Remarkably, even without pretraining, RAN can still yield competitive performance. For example, the randomly initialized RAN achieved better results than BiLSTM with pretrained GloVe word embedding. RAN with GloVe embedding outperforms pretrained BERT and ToBERT on the accuracy of the Hyper dataset and Longformer on average score. Such observations illustrate that RAN is effective for text encoding and flexible in adopting different initialization methods for various scenarios. It also suggests that the recurrent attention-based architecture of RAN is more powerful than the recurrent architecture of LSTM in modeling texts.

heading: 4.3.2 Language Modeling
text: The self-attention-based RAN can be employed for LM. Extensive experiments are conducted to evaluate RAN on language modeling. To avoid information exposure, we apply the causal attention mask to ensure the prediction for i-th position only depends on the known outputs before i, following Vaswani et al. (2017). We compare RAN with widely-adopted baselines that are shown in Table 2. The compared models have the same vocabulary and parameter sizes, and the parameters are randomly initialized. The experiment settings follow Zhong et al. (2022). Observing the results, we notice that RAN achieves the state-of-the-art result on the WikiText-103 dataset with 22.76 perplexity. It suggests that RAN is efficient in handling the sequence generation task.

heading: 4.3.3 Named Entity Recognition
text: The NER task is a common information extraction task, and we conduct experiments on the NER task to test RAN for information extraction. As the available NER datasets contain mostly short texts, we set the window size to 64 to test the effectiveness of RAN’s recurrent structure. We compare with the following widely-used baselines: IDCNN (Strubell et al., 2017), LSTM (Langlais et al., 2018), LSTM-CNN (Li et al., 2020), ELMo (Peters et al., 2018), and BERT (Devlin et al., 2019). As shown in Table 3, we notice that RAN consistently outperforms LSTM-based baselines. Specifically, RAN without pretraining achieves 0.5% and 0.3% improvement compared with BERT on both datasets, indicating that the well-designed GPC vector is effective in handling information extraction of long sequences. Both the NER and LM tasks are sequential tasks, and the results demonstrate that RAN is effective in sequence modeling.

heading: 4.4 Ablation Study
text: We conducted an ablation study to investigate the significance of each component of our proposed model on the Arxiv dataset. The results are presented in Table 4.
In the first ablation model, we substituted the max pooling layer depicted in Eq. 11 with an average pooling layer, resulting in a 1.12% drop in Accuracy. Moreover, our findings show that the residual connection between two windows is essential to alleviate gradient vanishing. When we removed it from RAN, the performance drops approximately 1.6%. We also ablated the rotary positional encoding from RAN, which leads to a 1.23% performance drop.
When the memory review mechanism of RAN was removed in the last ablation model, the result shows the most significant drop compared with other ablation models. RAN without the mem-
ory review mechanism suffers a 2.5% performance drop. Such an observation indicates that mitigating information forgetting in processing long documents is crucial, and our proposed memory review mechanism is effective in preserving long-distance memory over the sequence.
In general, the ablation study demonstrates the significance of each component in our proposed RAN model and highlights the importance of the memory review mechanism in processing long documents. Particularly, our observation accentuates the importance of maintaining long-distance memory in long document processing.

heading: 4.5 Discussion

heading: 4.5.1 Scalability Analysis of RAN
text: This section discusses the scalability of RAN in actual implementation. The window size W determines the number of tokens that are encoded by the attention block. In theory, RAN with a larger window size can yield better performance, as we have fewer windows and less information loss when iterating over the windows. However, given the quadratic memory complexity, the hardware capacity limits the maximum batch size that can be used in training and hence imposes a ceiling to the performance improvement. We have conducted additional experiments with RAN+Glove on the Arxiv dataset. Figure 5 depicts the accuracy of the test set and the training time per epoch of RAN with different window sizes. Results of each configuration are obtained with the maximum batch size runnable on the GPU. As expected, when window size increases, the accuracy also gains continuous improvements, albeit minor. The accuracy curve begins to flatten out when the size exceeds 256, partially due to the decreasing maximum batch size. Such an observation indicates the performance is approaching the bottleneck caused by the hardware capacities.
On the other hand, the V-shape curve observed in the training time is an intriguing sign, and we
attribute it to the different time complexities of recurrent and self-attention operations of RAN. Although computing self-attention is of quadratic time, it is significantly accelerated owing to the tensor computation on GPU. In contrast, the recurrent component involves tensor manipulations, such as splitting and concatenation, and thus takes more time. Therefore, a smaller window size will lead to a longer training time as more recurrent operations need to perform. When the window size is large enough, the quadratic time of selfattention becomes significant and dominates the overall spent time. Hence, the green curve bounces back when the window size is 1024. Moreover, when the window size is even larger, such as 2048, the model becomes too large to be loaded on the GPU, and training becomes infeasible.
Furthermore, we compare the training time of pretrained RAN with other pretrained and nonpretrained baselines on the Arxiv dataset. The results in Table 5 indicate that the proposed RAN is highly scalable and efficient. Notably, RAN has six times the parameter size of LSTM but has a shorter training time, which indicates our devised recurrent attention structure is more efficient than the LSTM-style gate mechanism.

heading: 4.5.2 The number of RAN Layers
text: Similar to RNNs, RAN layers can be stacked to build a deep architecture. We adopt a serial manner to pass the previous layer’s GPC output as the input to the subsequent hidden RAN layers. The GPC output at the last RAN layer will be con-
catenated with the following window input. Intuitively, with more RAN layers, the model will contain more parameters and is promising to produce higher performance. We compare the RANs with different depths and list the results in Table 6. As expected, the accuracy improves as the number of layers increases. However, the average training time will significantly increase due to the serial connection between layers, and the improvements become marginal. Therefore, to balance the performance and the time consumption, we adopt the two-layer RAN in this paper by default. This also implies that the results presented in this paper could be further enhanced by increasing the depth of the RAN.

heading: 5 Conclusion & Future Work
text: This paper has presented a novel RAN architecture for long-text modeling that combines the advantages of both recurrent and self-attention networks. The use of a positional multi-head attention mechanism and GPC vector enhances the model’s performance by capturing both local and global dependencies in the input sequence. Our ablation study also highlights the critical role of residual connection and memory review mechanisms in preserving long-distance memory.
With the well-designed recurrent self-attention mechanism, RAN’s training can be accelerated by parallel computing on a GPU, making it highly efficient and scalable. We have conducted extensive
experiments on TC, NER, and LM tasks. The extensive experiments demonstrate the effectiveness of the proposed RAN model on both classification and sequential tasks.
The flexibility and scalability of our proposed RAN make it a promising choice for future research, with broad potential applications in translation, summarization, conversation generation, and large language models. Additionally, we plan to extend the RAN to tasks involving multi-modality input and output like audio and video, to exploit RAN’s long sequence handling capacity in different fields.

heading: 6 Limitations
text: The proposed model, Recurrent Attention Network (RAN), effectively models long sequential data by propagating information window-by-window through the sequence via its well-designed recurrent architecture. However, the multi-head selfattention applied to each window is still limited to local attention, which prevents it from providing a global dependency relationship for the entire sequence. This limitation restricts RAN’s application in scenarios where a global dependency relationship is necessary, such as visualizing attention weights for the entire document via a heatmap. This limitation potentially reduces the interpretability of the model, although it does not affect the model’s performance. Hence, exploring ways to incorporate global attention mechanisms into the RAN architecture is a promising research direction to improve its interpretability and expand its range of applications.

heading: Acknowledgments
text: Xianming Li, Xiaotian Luo, Xing Lee, and Yingbin Zhao’s work has been supported by Ant Group. Zongxi Li’s work has been supported by a grant from Hong Kong Metropolitan University (Project Reference No. CP/2022/02). Haoran Xie’s work has been supported by the Direct Grant (DR23B2) and the Faculty Research Grant (DB23A3) of Lingnan University, Hong Kong. Qing Li’s work has been supported by the Hong Kong Research Grants Council through the Collaborative Research Fund (Project No. C1031-18G). We thank the anonymous reviewers for their careful reading of our manuscript. Their insightful comments and suggestions helped us improve the quality of our manuscript.

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
6
7 A2. Did you discuss any potential risks of your work? This paper focuses on fundamental research of long sequence modeling in natural language processing. The proposed model does not contain any component that is potentially malicious or harmful to human beings. The model can only be applied for classification and information retrieval tasks, and in its current form, the model cannot be used for conducting adversarial attacks. The model was tested on language modeling with sentence generation, in which the pretraining corpus are commonly adopted ones for language model pretraining. The work described does not aim to present any new dataset and hence is free of disseminating any false/biased/unfair/discriminative information. The model and its variants are trained with widely adopted and recognized data for different tasks, thus, to the best of our knowledge, it will not contribute to any fairness issue.
3 A3. Do the abstract and introduction summarize the paper’s main claims? 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts?

heading: 4.1, 4.3

heading: 3 B1. Did you cite the creators of artifacts you used?
text: 4.1
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Most of the datasets we used are open for research purposes. Rest datasets such as OntoNotesV5.0, we have applied to use in this paper on the corresponding official sites, and we have been granted to use in this paper.

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? 4.1
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? The datasets we used have been widely adopted in the literature and are considered safe to use.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? 4.1

heading: 3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader
text: The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. 4.1
C 3 Did you run computational experiments? 4

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? 4.2,4.3

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? 4.2

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? 4.3

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? 4.2
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: The flexibility and scalability of our proposed RAN make it a promising choice for future research, with broad potential applications in translation, summarization, conversation generation, and large language models. Additionally, we plan to extend the RAN to tasks involving multi-modality input and output like audio and video, to exploit RAN’s long sequence handling capacity in different fields