Your task now is to draft a high-quality review for a top-tier Machine Learning (ML) conference for a submission:\n\nTitle:\n```\nEncoding Recurrence into Transformers\n```\n\nKeywords:\n```\nRecurrent models,Transformers,sample efficiency,gated mechanism\n```\n\nAbstract:\n```\nThis paper novelly breaks down with ignorable loss an RNN layer into a sequence of simple RNNs, each of which can be further rewritten into a lightweight positional encoding matrix of a self-attention, named the Recurrence Encoding Matrix (REM). Thus, recurrent dynamics introduced by the RNN layer can be encapsulated into the positional encodings of a multihead self-attention, and this makes it possible to seamlessly incorporate these recurrent dynamics into a Transformer, leading to a new module, Self-Attention with Recurrence (RSA). The proposed module can leverage the recurrent inductive bias of REMs to achieve a better sample efficiency than its corresponding baseline Transformer, while the self-attention is used to model the remaining non-recurrent signals. The relative proportions of these two components are controlled by a data-driven gated mechanism, and the effectiveness of RSA modules are demonstrated by four sequential learning tasks.\n```\n\nFigures/Tables Captions:\n```\nFigure 1: (a) - (c) plot the two data features, namely the sample signal and the sample size as the x and y axis, respectively. The model performance in each data region for the RNN, Transformer and RSA are given, where deeper color implies a better performance. (d) In each attention head, the proposed RSA attaches an REM to a normalized self-attention score via a gated mechanism, with gate value \u03c3(\u00b5). The REM depicts a type of recurrence dependence structure between the tokens in X, and is parameterized by one or two parameters, i.e. \u03bb or (\u03b3, \u03b8), where \u03bb = tanh(\u03b7), \u03b3 = \u03c3(\u03bd).\nFigure 2: Illustration on how an RNN layer can be equivalently represented by a set of simple RNNs and further by a multihead self-attention.\nFigure 3: The boxplot for gate values at different attention layers of RSA models.\nFigure 4: The left panel presents the BPC on Text8 for XL, RSA-XL, TL-XL and FL-XL at different model sizes. The middle panel compares the sample efficiency of RSA-XL on with that of the baseline XL by training them on restricted portions of the Enwik8 training set. Meanwhile, the gate-values at different attention layers of RSA-XL are presented in the boxplot.\nFigure 5: Accuracy for BRT vs RSA-BRT and TLB vs RSA-TLB on the copy task is plotted on the first and second panels respectively, where \"S\" means a small training sample size of 6400, and \"L\" means a large training sample size of 12800. When sequence length = 512, the cross entropy loss of the four models for the first 9000 gradient steps are plotted in the third and fourth panels.\nFigure 8: The training time ber batch for each RSA model, which can be decomposed into the additional computation time due to the REMs (green bar) and the others (blue bar).\nFigure 9: Comparison of the training time of ber batch between RSA-Informer and Informer (left panel), RSA-Logsparse Transformer and Logsparse Transformer (middle panel), RSA-Transformer- XL and Transformer-XL (right panel). The experiment is performed on Weather dataset.\nFigure 10: (a) - (c) report the change in \u03c3(\u00b5) at each attention layer during training, where \u00b5 is initialized at \u22121, 0, and 1, respectively. An 8-headed RSA-Informer is applied to the ETTh 1 dataset with settings reported in Table 5. The prediction window size equals to 24.\nFigure 11: We visualize four recurrent patterns provided by the RSA heads on different languages (top), and their corresponding output, averaged over embedding dimension (bottom). The correspond- ing input sequence is given on x-axis.\nFigure 12: Bit-per-character for BRT and RSA-BRT with respect to the model size (i.e. number of parameters in millions) on Enwik8.\nTable 1: Multivariate long sequence time-series forecasting results reported on different prediction window sizes. We compare the MSE and MAE of RSA models with their baselines under all settings, and highlight the better results in bold letters. Note that XL requires a sequential ordering for training, and the same scheme are adopted for all models to make the results comparable.\nTable 2: Prediction accuracy of the vanilla Transformer, the Transformer with learnable relative positional encoding (\"Transformer \u22c6 \") on the six languages, together with the RSA-Transformer under different hyperparameter settings. All Transformers are 5-headed, and the hyperparameter settings for RSA-Transformer, Cases I-IV, are denoted by the tuple (k 1 , k 2 , k 3 , k 4 , k 5 , k 6 ).\nTable 3: (a) The defect accuracy is reported for the CodeT5-small baseline and the RSA-CodeT5- small. In addition, we report the percentage of the parameters added by the RSA model against its baseline. (b) The number of Bit-per-character (BPC) is reported for Enwik8 and Text8 (lower is better), and the perplexity (PPL) is reported for WikiText-103 (lower is better).\nTable 5: The fine-tuned hyperparameter settings used for the RSA models in Table 1. The tuple (k 1 , k 2 , k 3 , k 4 , k 5 , k 6 ) refer to the number of the six types of REMs. The values of different dilating factors are listed, where those for regular heads are followed by those for the cyclical heads. d = \" \u2212 \" means no dilated REMs are considered.\nTable 6: Ablation study on the hyperparameters for the 8-headed RSA-Informer on ETTh 1 dataset.s and s correspond to the number of dilated and non-dilated cyclical REMs. Ands =\"-\" refers to the case where no dilated REMs are considered. Otherwise, 4 out of 8 dilated REMs are considered. The MSEs are averaged over 5 repetitions.\nTable 7: The hyperparameter settings for the RSA and their baseline models used in Table 3.\nTable 4: Multivariate long sequence time-series forecasting results on ETT and Weather datasets. The standard deviation of the 5 repetitions is included in brackets below the mean value. The results are reported on different prediction window sizes. We compare the MSE and MAE of RSA models with their baselines under all settings (a total of 32 comparisons), and report the number of times each model surpasses its comparing model in the last row.\n```\n\nMain Content:\n```\n\nSection Title: INTRODUCTION\n  INTRODUCTION Sequential data modeling is an important topic in machine learning, and the recurrent networks such as LSTM ( Hochreiter & Schmidhuber, 1997 ) and GRU ( Chung et al., 2014 ) have served as the benchmarks in this area over a long period of time. The success mainly contributes to the variety of recurrent dynamics introduced by these models, referred to as the recurrent inductive bias. More specifically, the dependence between any two inputs can be described by a parametric form, which heavily depends on their relative temporal locations. However, the recurrent models are well known to suffer from two drawbacks. The first one is the gradient vanishing problem ( Hochreiter et al., 2001 ), i.e. the recurrent models have difficulty in depicting the possibly high correlation between distant inputs. This problem cannot be solved fundamentally by the recurrent models themselves, although it can be alleviated to some extent, say by introducing long memory patterns ( Zhao et al., 2020 ). Secondly, the sequential nature renders these models difficult to be trained in parallel ( Vaswani et al., 2017 ). In practice, many techniques were proposed to improve the computational efficiency of recurrent models, while they all come with compromises ( Luo et al., 2020 ;  Lei et al., 2017 ). In recent years, Transformers ( Vaswani et al., 2017 ) have been revolutionizing the field of natural language processing by achieving the state-of-the-art performance on a wide range of tasks, such as language modeling ( Kenton & Toutanova, 2019 ), machine translation ( Dai et al., 2019 ) and text summarization ( Liu & Lapata, 2019 ), etc. They have also demonstrated great potential in other types of sequence learning problems, for instance, time series forecasting ( Zhou et al., 2021 ;  Li et al., 2019 ). The success of Transformers is due to the fact that the similarity between any two tokens is well taken into account ( Vaswani et al., 2017 ) , and hence they can model long range dependence effortlessly. Moreover, contrary to the recurrent models, the self-attention mechanism in Transformers is feed-forward in nature, and thus can be computed in parallel on the GPU infrastructure ( Vaswani et al., 2017 ). However, the flexibility also leads to sample inefficiency in training a Transformer, i.e. much more samples will be needed to guarantee good generalization ability ( d'Ascoli et al., 2021 ). Moreover, the chronological orders are usually ignored by Transformers since they are time-invariant, and some additional efforts, in the form of positional encoding, will be required to further aggregate the temporal information ( Shaw et al., 2018 ;  Vaswani et al., 2017 ;  Dai et al., 2019 ). In short, both recurrent and Transformer models have the pros and cons in modeling sequential data. On one hand, due to inductive bias, the recurrent models excel at capturing the recurrent patterns even with relatively small sample sizes; see Figure 1(a). Meanwhile, sample size is the performance bottleneck for the Transformer models and, when there are sufficient samples, they are supposed to be able to depict any recurrent or non-recurrent patterns in the data; see Figure 1(b). On the other hand, sequential data have recurrent patterns more or less, and Transformers may have an improved performance if the recurrent model can be involved to handle these patterns, especially when the sample size is relatively small. Specifically, if the recurrent and non-recurrent components are separable, then one can apply a parsimonious recurrent model on the recurrent component and a Transformer on the non-recurrent one. As a result, the sample efficiency can be improved comparing to the Transformer-only baseline; see illustration in Figure 1(c). There have been various attempts in the literature to combine the two models. Some earlier works were to simply stack them together in a straightforward manner.  Chen et al. (2018)  mixed and matched a Transformer's encoder with an recurrent-based decoder.  Hao et al. (2019)  introduced an additional recurrent encoder to a Transformer, while  Wang et al. (2019)  stacked a recurrent layer prior to the multihead self-attention. These proposals inherit both the aforementioned shortcomings of Transformer and recurrent models. In particular, for a very long input sequence, the sequential operation in the recurrent layers become extremely expensive. Recent efforts have been spent on integrating recurrence and self-attention systematically. Feedback Transformer ( Fan et al., 2021 ) introduces the memory vectors to aggregate information across layers, and uses them to update the next token in a recursive manner. However, the computationally expensive sequential operation limits its attractiveness. Another line of research applies the recurrent operation only to aggregate the temporal information at a coarser scale, while the token-by-token dependence is learned by self-attention instead. Transformer-XL ( Dai et al., 2019 ) partitions the long inputs into segments and introduces a segment-level recurrence. Meanwhile, Temporal Latent Bottleneck (TLB) ( Didolkar et al., 2022 ) and Block-Recurrent Transformer (BRT) ( Hutchins et al., 2022 ) further divide the segments into smaller chunks, and each chunk is summarized into a few state vectors. A recurrent relation is then formed on the sequence of state vectors. These hierarchical designs are useful to reduce the computational burden, while they overlook recurrent dynamics at a finer scale. In an attempt to simplify the numerical calculation of RNNs, we found surprisingly that an RNN layer with linear activation can be broken down into a series of simple RNNs with scalar hidden coefficients. Each simple RNN induces a distinct recurrent pattern, and their combination forms the recurrent dynamics of the RNN layer. Hence the calculation time can be greatly reduced by training these simple RNNs in parallel. On top of that, it can be equivalently rewritten into the positional encodings of a multihead self-attention (MHSA). This spontaneously inspires a solution, the multihead Self-Attention with Recurrence (RSA), to combine self-attention with RNN into one single operation while maintaining parallel computation. This solution enables our design to preserve the merits from both Transformer and recurrent models, while their respective shortcomings are avoided. More importantly, it can be used to replace the self-attention of existing networks, such as Transformer XL, TLB and BRT, to further explore recurrent dynamics at the finer scale. Our paper makes three main contributions below. 1. With ignorable approximation loss, we demonstrate that an RNN layer with linear activation is equivalent to a multihead self-attention (MHSA); see  Figure 2 . Specifically, each attention head can be used to recover a type of recurrent dependence structure, and multiple heads working in sync replicate the complex temporal dynamics of the RNN layer. 2. In the above MHSA, recurrent dynamics of the RNN layer are encapsulated entirely into the positional encodings, which we name the Recurrence Encoding Matrices (REMs). This makes it possible to add the REMs to any existing Transformer architecture that has the self-attention mechanism, leading to the Self-Attention with Recurrence (RSA) module; see Figure 1(d). A gated mechanism is used to control the proportion of REMs, and the gate value may also be interpreted as the proportion of the recurrent signals in the data. 3. Our experiments on four sequential tasks demonstrate that the proposed RSA module can effectively enhance the baseline Transformers' forecasting power. Moreover, from the fitted gate values, we have an interesting finding that time series data have stronger recurrent signals than those of the regular languages, while the code or natural languages have much weaker recurrent signals.\n\nSection Title: OTHER RELATED WORKS\n  OTHER RELATED WORKS\n\nSection Title: Relative positional encoding\n  Relative positional encoding The proposed REMs can provide complementary positional encoding in the RSA module. Essentially, its recurrent pattern conveys relative location information. This property is inherited from the distance-awareness of the RNN. Compared to the relative positional encoding (RPE) in the existing literature, it is more parsimonious and interpretable. The fully learnable RPE proposed by  Shaw et al. (2018)  requires O(T 2 d) parameters, where T is the number of tokens and d is the token dimension. Later, different variants are introduced by  Huang et al. (2019)  and  Dai et al. (2019) , which reduce the number of parameters to O(T d) and O(d), respectively. In contrast, our REMs have parametric forms of merely one or two parameters. Time series forecasting Recent literature sees a surge in deep learning models designed specially for time series forecasting tasks, which are not Transformer-based. For instance, DLinear ( Zeng et al., 2022 ) and Autoformer ( Wu et al., 2021 ) decompose the series into trend and non-trend components. Autoformer and SCINet ( Liu et al., 2021 ) further exploit the temporal dependence structure of time series via the autocorrelation function and multi-level convolution operations, respectively. FiLM (Zhou et al., 2022a) and FEDformer (Zhou et al., 2022b) transform the series to the frequency domain to capture meaningful signals while removing noises. This paper, however, differs from these literature in both target and method. We aim to add the recurrent inductive bias to general-purpose Transformers, and hence its performance rely heavily on the baseline Transformer that it modifies.\n\nSection Title: RELATIONSHIP BETWEEN RNN AND MULTIHEAD SELF-ATTENTION\n  RELATIONSHIP BETWEEN RNN AND MULTIHEAD SELF-ATTENTION This section demonstrates that, without loss of much generality, an RNN layer can be approximated by a series of simple RNNs with scalar (hidden) coefficients, which can be further represented in the form of a multihead self-attention.\n\nSection Title: BREAKING DOWN AN RNN LAYER\n  BREAKING DOWN AN RNN LAYER Consider an RNN layer with the input variables {x t \u2208 R din , 1 \u2264 t \u2264 T }, and it has the form of h t = g(W h h t\u22121 + W x x t + b), where g(\u00b7) is the activation function, h t \u2208 R d is the output or hidden variable with h 0 = 0, b \u2208 R d is the bias term, W h \u2208 R d\u00d7d and W x \u2208 R d\u00d7din are weights. When the activation function is linear, i.e. g(x) = x, the RNN becomes where the bias term b is suppressed for simplicity. Although it has a feedforward form, the RNN cannot be trained in parallel, and this is mainly caused by the power j of recurrent weights W h , where 1 \u2264 j \u2264 t \u2212 1. This section makes an effort to block diagonalize W h such that the RNN at (1) can be broken down into a sequence of simple RNNs with scalar (hidden) coefficients. Lemma 1 ( Theorem 1 in Hartfiel (1995) ). Real matrices with R distinct nonzero eigenvalues are dense in the set of all d \u00d7 d real matrices with rank at most R, where 0 < R \u2264 d. Suppose that the weight matrix W h has rank R \u2264 d. By Lemma 1, without loss of much generality, we can assume that the nonzero eigenvalues of W h are all distinct. Specifically, W h has r real nonzero eigenvalues \u03bb 1 , . . . , \u03bb r , and s pairs of complex nonzero eigenvalues \u03bb r+1 , . . . , \u03bb r+2s , where (\u03bb r+2k\u22121 , \u03bb r+2k ) = (\u03b3 k e i\u03b8 k , \u03b3 k e \u2212i\u03b8 k ) for 1 \u2264 k \u2264 s, i is the imaginary unit, and R = r + 2s. As a result, we have the real Jordan form, W h = BJ B \u22121 , where B \u2208 R d\u00d7d is invertible and J \u2208 R d\u00d7d is a block diagonal matrix. It holds that W j h = BJ j B \u22121 for all j \u2265 1, and we can then break down the recurrence induced by W h into that of the p \u00d7 p block matrices in J with p = 1 or 2. Similar to (1), we define three types of RNNs with linear activation below, and h C2 t (\u03b3, \u03b8) = t\u22121 j=1 \u03b3 j sin(j\u03b8)W C2 x x t\u2212j , where the first one corresponds to the real eigenvalues, i.e. the 1 \u00d7 1 block matrices in J , while the last two correspond to the complex eigenvalues, i.e. the 2 \u00d7 2 block matrices in J . Note that each of the three RNNs has the recurrent weights of \u03bb or (\u03b3, \u03b8), and its form with a nonlinear activation function is given in the Appendix. Proposition 1. Suppose that W h has rank R = r + 2s \u2264 d, and its eigenvalues are defined above. Let h 0,t = W x x t , and then the RNN with linear activation at (1) can be equivalently rewritten into\n\nSection Title: AN EQUIVALENT MHSA REPRESENTATION\n  AN EQUIVALENT MHSA REPRESENTATION Consider the RNN of {h R t }, and let X = (x 1 , . . . , x T ) \u2032 \u2208 R T \u00d7din be an input matrix consisting of T tokens with dimension d in , where the transpose of a matrix A is denoted by A \u2032 throughout this paper. We first give the value matrix V by projecting X with a linear transformation, i.e. V = XW V with W V = W R\u2032 x \u2208 R din\u00d7d , and the relative positional encoding matrix is set to We call P R mask the recurrence encoding matrix (REM) as it summarizes all the recurrence in {h R t }. For the RNN of {h Ci t }, i = 1 or 2, the REM is denoted by P Ci mask (\u03b3, \u03b8), which has the form of (3) with Thus, these two RNNs at (2) can also be represented into SA forms, The remaining term in Proposition 1, h 0,t depends on x t only, and there is no inter-dependence involved. Mathematically, we can represent it into a SA with the identity relative positional encoding matrix. Finally, all the query and key matrices Q and K are set to zero in the above reformulations. The three simple RNNs at (2) provide different temporal decay patterns: h R t provides the regular exponential decay induced by the real eigenvalues \u03bb k 's, and {h C1 t , h C2 t } provide the cyclical damped cosine or sine decay induced by the pair of complex eigenvalues (\u03b3 k e i\u03b8 k , \u03b3 k e \u2212i\u03b8 k )'s. These tem- poral decay patterns are further summarized into the regular REM, P R mask (\u03bb), and cyclical REMs, P C1 mask (\u03b3, \u03b8) and P C2 mask (\u03b3, \u03b8), respectively; see  Figure 2  for the illustration. From Proposition 2, the combination of these three types of patterns forms the recurrent dynamics of the RNN layer at (1). For each head, the REM has one or two parameters, and W V can be regarded as one d \u00d7 d learnable matrix. This leads to a parameter complexity of O(Rd 2 ), and it is slightly larger than that of the RNN at (1) since R is usually much smaller than d ( Prabhavalkar et al., 2016 ). Moreover, the MHSA representation in Proposition 2 gives us a chance to make use of parallel matrix calculation on the GPU hardware; see Appendix D.3 for an illustration of the computational efficiency of REMs.\n\nSection Title: ENCODING RECURRENCE INTO SELF-ATTENTION\n  ENCODING RECURRENCE INTO SELF-ATTENTION While the query and key matrices are set to zero in the MHSA representation at Proposition 2, they play a central role in a standard Transformer. This motivates us to propose the Self-Attention with Recurrence (RSA) module to seamlessly combine the strengths of RNNs and Transformers: RSA(X) = {[1 \u2212 \u03c3(\u00b5)]softmax(QK \u2032 ) + \u03c3(\u00b5)P }V (4) for each head, where P is a regular or cyclical REM, and \u03c3(\u00b5) \u2208 [0, 1] is a gate with \u03c3 being the sigmoid function and \u00b5 being the learnable gate-control parameter. Figure 1(d) provides a graphical illustration of one RSA head. Note that the REMs in Section 2 are all lower triangular matrices, which correspond to unidirectional RNNs. In the meanwhile, for non-causal sequential learning tasks ( Graves et al., 2005 ; Zhou et al., 2016), bidirectional RNNs are usually applied, and accordingly we can define the unmasked versions of REMs. Specifically, the regular REM is P R unmask (\u03bb) = P R mask (\u03bb) + [P R mask (\u03bb)] \u2032 , and the cyclical REMs are P Ci unmask (\u03b3, \u03b8) = P Ci mask (\u03b3, \u03b8) + [P Ci mask (\u03b3, \u03b8)] \u2032 with i = 1 and 2. In practice, these REMs will be explosive when |\u03bb| > 1 or |\u03b3| > 1. To avoid this problem, we further bound these two parameters by transformations in this paper, i.e. \u03bb = tanh(\u03b7) and \u03b3 = sigmoid(\u03bd), and these notations then become P R k (\u03b7), P C1 k (\u03bd, \u03b8) and P C2 k (\u03bd, \u03b8) with k \u2208 {mask, unmask} accordingly. Sequential data usually have some recurrent patterns, which can be captured by REMs. In the meanwhile, the remaining non-recurrent patterns can be modeled by the baseline Transformer via softmax(QK \u2032 ). The allocation of weights between REMs and self-attention is adjusted by the learnable gate \u03c3(\u00b5). On one hand, comparing with the baseline Transformer, the inclusion of REMs will lead to a significant improvement of sample efficiency in modeling the recurrent patterns, and hence a higher accuracy in prediction can be expected from the proposed RSA. On the other hand, comparing with the recurrent models, the non-recurrent patterns can be taken care of by the flexible Transformers, and hence the representation power of the RSA module is as good as that of the baseline Transformer. In addition, the REM can act as a complementary positional encoding scheme to further provide relative location information. For the multihead RSA, the gate-control parameter \u00b5 only varies across layers, while the parameters controlling the matrix P vary across all heads and layers.\n\nSection Title: Initialization\n  Initialization The gate-control parameter \u00b5 is initialized in the interval [\u22123, 3] for all the layers. For (\u03bb, \u03b3, \u03b8) which determine the recurrent patterns, we initialize \u03bb's at different heads to spread out between [\u22122, \u22121] \u222a [1, 2] and \u03bd's to spread out between [1, 2], and \u03b8 is initialized at \u03c0/4, to encourage REMs to be non-zero and well-diversified.\n\nSection Title: Dilated REM variants\n  Dilated REM variants The dilated REMs can be further obtained by considering the block matrix formed by the first T columns and first T rows of P \u2297 I d , where d is the dilating factor. In fact, the dilated REMs can encapsulate the recurrence patterns of the dilated RNNs ( Chang et al., 2017 ); see Proposition 4 in the Appendix. They describe potentially periodic recurrence over a long-term timespan, and can significantly enrich the temporal dynamics of our RSA. Hyperparameters k i and d. In each multihead RSA layer with H heads, the number of the six types of REMs, namely one regular, two cyclical, one dilated regular and two dilated cyclical REMs, are denoted by k 1 \u2212 k 6 , respectively. Since i=1 k i = H, we can apply constrained hyperparameter search to optimize over the choices of k i 's. For simplicity, we can set k 2 = k 3 and k 5 = k 6 , which indicates that the cyclical REMs come in pairs. The search for d, on the other hand, can be guided by various data analytic tools. For instance, d can be observed from the autocorrelation plots as the seasonal period length for time series data ( Cryer & Chan, 2008 ); while for the language-related data, d can be heuristically deduced from the recurrence plots ( Webber et al., 2016 ). More discussions on REMs. Each type of REMs is basically a masked or unmasked linear aggrega- tion of tokens, and we may alternatively consider a more general Toeplitz, or even a fully learnable, matrix P at (4) such that all kinds of temporal decay patterns can be automatically captured. Although more flexible than REMs, it will need O(T ) or O(T 2 ) additional parameters, where T is the sequence length, while each REM requires merely one or two parameters. Note that the proposed RSA at (4) also includes a standard self-attention, which is flexible enough for the remaining nonlinearities that failed to be captured by REMs, and hence it may not be necessary to consider a more general yet less efficient structure for REMs. see Appendix B for both theoretical and empirical evidences.\n\nSection Title: EXPERIMENTS\n  EXPERIMENTS This section contains four sequential modeling tasks and, for each task, we modify some popular Transformer baselines by adding the REMs to their attention weights via a gated mechanism as in (4). The modified models are named with the prefix \"RSA-\". The trained gate \u03c3(\u00b5) controls the percentage of weights allocated to the REMs, and it hence provides a rough measure for recurrent signals in the data. From  Figure 3 , we may argue that time series have the strongest recurrent signals, followed by the regular language ( Yu, 1997 ) and finally the code or natural languages. The gate values also vary across different layers of Transformer models, indicating that the proportion of recurrent signals may change as the data propagate through layers. Moreover, the data-driven gated mechanism can help maintain the optimal allocation between REM and self-attention to achieve better sample efficiency and improved performance. All experiments are conducted on Nvidia V100 32GB GPUs.\n\nSection Title: TIME SERIES FORECASTING\n  TIME SERIES FORECASTING Time series data possess unique attributes, such as strong recurrent signals, that can improve forecast- ing accuracy. Although Transformers are not currently the state-of-the-art model for this task, it is interesting to explore how the proposed RSA module could enhance their performance. Our experiments are performed on two public benchmark datasets: the ETT 1 dataset is comprised of seven features related to the electric power long-term deployment, where {ETTh 1 , ETTh 2 } are recorded by the hour and ETTm 1 is recorded by 15-minute intervals; and the Weather 2 dataset contains twelve climate indicators collected every 1 hour over a 4-year period. For the baseline models, we adopt Informer ( Zhou et al., 2021 ), LogSparse Transformer ( Li et al., 2019 ) and Transformer-XL ( Dai et al., 2019 ). In particular, to accommodate the cases where the attention map is non-square, we extend the recurrent patterns and adjust the shape of our REMs; see details in Appendix D.2. All hyperparameters in baseline models are set to the optimal setting in  Zhou et al. (2021) , and we also follow their train/val/test division and training schemes to conduct our experiments.  Table 1  summarizes the mean squared error (MSE) and mean absolute error (MAE) for the three baselines against their RSA counterparts, averaged across 5 repetitions. Overall, the revised\n```\n\n======\nYour task:\nCompose a high-quality peer review of an ML paper submitted to a top-tier ML conference on OpenReview.\n\nThe review form is shown below:\n\n1. Summary and contributions: Briefly summarize the paper and its contributions\n\n2. Strengths: Describe the strengths of the work. Typical criteria include: soundness of the claims (theoretical grounding, empirical evaluation), significance and novelty of the contribution, and relevance to the community.\n\n3. Weaknesses: Explain the limitations of this work along the same axes as above.\n\n4. Correctness: Are the claims and method correct? Is the empirical methodology correct?\n\n5. Clarity: Is the paper well written?\n\n6. Relation to prior work: Is it clearly discussed how this work differs from previous contributions?\n\n7. Reproducibility: Are there enough details to reproduce the major results of this work?\n\n8. Additional feedback, comments, suggestions for improvement and questions for the authors\n\n9.  Overall score:\nYou should NOT assume that you were assigned a representative sample of submissions, nor should you adjust your scores to match the overall conference acceptance rates. The \u201cOverall Score\u201d for each submission should reflect your assessment of the submission\u2019s contributions.\n10): Top 5% of accepted papers. Truly groundbreaking work.\n9): Top 15% of accepted NeurIPS papers. An excellent submission; a strong accept.\n8): Top 50% of accepted NeurIPS papers. A very good submission; a clear accept.\n7): A good submission; accept.nI vote for accepting this submission, although I would not be upset if it were rejected.\n6): Marginally above the acceptance threshold. I tend to vote for accepting this submission, but rejecting it would not be that bad.\n5): Marginally below the acceptance threshold. I tend to vote for rejecting this submission, but accepting it would not be that bad.\n4): An okay submission, but not good enough; a reject. I vote for rejecting this submission, although I would not be upset if it were accepted.\n3): A clear reject. I vote and argue for rejecting this submission.\n2): I'm surprised this work was submitted to NeurIPS; a strong reject.\n1): Trivial or wrong or already known.\n\n10.  Confidence score:\n5): You are absolutely certain about your assessment. You are very familiar with the related work.\n4): You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.\n3): You are fairly confident in your assessment.  It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.\n2): You are willing to defend your assessment, but it is quite likely that you did not understand central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.\n1): Your assessment is an educated guess. The submission is not in your area or the submission was difficult to understand. Math/other details were not carefully checked.\n\n11. Have the authors adequately addressed the broader impact of their work, including potential negative ethical and societal implications of their work?\n\n12. Does the submission raise potential ethical concerns? This includes methods, applications, or data that create or reinforce unfair bias or that have a primary purpose of harm or injury. If so, please explain briefly.\n\n