Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper tackles the problem of endowing Transformers with the ability to encode information about the past via recurrence. The proposed architecture can leverage the recurrent connections to improve the sample efficiency while maintaining expressivity due to the use of self-attention. 
strength_and_weaknesses: Strengths:

- The paper is easy to read, and generally well written.
- The paper evaluates the proposed method on various different tasks such as time-series forecasting, code and language modelling. The paper augments the proposed method to various different transformer variants and compares the performance with respect to the unmodified baseline. 

Weakness:

- The problem of integrating recurrence and self-attention is an important research problem. There exists some existing ways on how to augment transformers with recurrence such as Temporal Latent Bottleneck [1] and Block-Recurrent Transformers [2]. The idea behind TLB is to "divide" the sequence into chunks, and within a chunk use self-attention and to access information across chunks the model needs to use recurrence. It would be useful to compare the proposed method to these variants to futher analyse the pros and cons. 
- It may also be useful to study the proposed method by varying the capacity of the network to see how well the underlying idea scales. 

[1] Temporal Latent Bottleneck: Synthesis of Fast and Slow Processing Mechanisms in Sequence Learning, https://arxiv.org/abs/2205.14794 (NeurIPS'22) \
[2] Block Recurrent Transformers, https://arxiv.org/abs/2203.07852 (NeurIPS'22)
clarity,_quality,_novelty_and_reproducibility: Clarity: The paper is easy to read.

Quality: The paper tries to tackle an important problem.

Novelty: Even though the problem is not "new" per se, but the underlying idea is interesting.

Reproducibility: The paper should be easy to reproduce.
summary_of_the_review: The paper proposes a way to incorporate recurrence and self-attention by modifying the positional encoding.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: Transformers have less inductive bias; while it is well generalized. RNNs have large inductive bias; while it is sample-efficient. Authors proposed the REM which combines the advantages of Transformer and RNNs which are famous sequential models. Experimental results have shown the effectiveness of the proposed model.

strength_and_weaknesses: It is novel enough to combine the advantages of two famous models (Transformer, RNN). Also, the combining method looks applicable to a variety of scenarios. The experimental results are impressive, showing superior performance to previous Transformer.

I think the draft would become better if there is a more complete explanation and figures about the self-attention with recurrence (RSA) operation. 

clarity,_quality,_novelty_and_reproducibility: The draft includes details for implementing the methods. 
summary_of_the_review: I think the novelty of this draft is enough for the publication and the experimental results are impressive. English is good enough as well. I recommend weak accept for the draft.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: For small-scaled sequential data, transformers tend to overfit, while RNNs has better inductive bias that prevents the overfitting. However, RNN can't be trained in a parallel way like transformers. In this work, the authors find that a linear RNN has a simple form of masked linear aggregation, which can be formulated as a specific self-attention module, thus can be easily incorporated in to transformers and jointly trained in a parallel way. The authors propose to combine regular self-attention and RNN module together, and use a gating function to decide the weight of the two paths, making it possible to fit data with different scales. Experiments on several sequential modeling tasks show the advantage of the proposed combination of transformers and RNNs.
strength_and_weaknesses: + The motivation is clear and the algorithm is sensible.

+ The proposed method is tested on several benchmarks.

- RNN block discussed in the paper is basically a masked linear aggregation of the tokens (``masked'' means each token can only attend to the previous tokens), with the aggregation weights ($P_{mask}$) specially designed. It would be helpful if the authors can compare to the baseline where the $P_{mask}$ is learnable, i.e., using a learnable masked linear aggregation. Another baseline would be a learnable unmasked linear aggregation. These comparisons can tell us if it's the RNN formulation that matters or just the linear aggregation.

- The authors argue that, transformers on small-scale recurrent data will overfit, thus introducing the RNN module which has a better inductive bias can help prevent the overfitting. However, there is a learnable gate-control parameter to decide whether the model should rely more on self-attention or RNN. Won't that encourage the model to rely more on self-attention in order to overfit the training data?

- Since the authors use a linear RNN, I wonder how much model capacity are we losing by linearizing the RNN, i.e., how huge is the gap between linear and non-linear RNN, performance-wise?

- In Section 3, the authors denote by $p_{total}$ the total temporal patterns in the data. How is this value defined?

- Section 2.1, $b \in \mathbb{R}^{d_{in}}$ -> $b \in \mathbb{R}^{d}.


clarity,_quality,_novelty_and_reproducibility: The presentation is easy to follow. I've read the theoretical justification and found no major issues. The method is somewhat novel in that it shows the equivalence between (linear) RNN and self-attention, although similar results have been demonstrated in other areas such as vision, for example, the equivalence between convolution and self-attention.
summary_of_the_review: Overall the paper is solid in both theoretical and empirical parts, although comparisons to the baseline models are missing.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: 
I Summary:

I.1 Investigated Problem:

Transformers models have the capacity to process large-scale sequential data and tend to overfit on small sequences. At the same time, RNNs inherently possess inductive bias that prevents overfitting and their training can be longer due to their inherent recurrence which hinders the leverage of modern-day parallelism of processing units like GPUs and TPUs. 

- I.2 Proposed Solution:
The Recurrence Encoding Matrix (REM) is proposed to endow positional encodings of a multi-head self-attention with recurrent dynamics leading to a new module named Self-Attention with Recurrence (RSA). The proposed module can leverage the recurrent inductive bias of REMs to achieve a better sample efficiency than its corresponding baseline Transformer, while self-attention is used to model the remaining non-recurrent signals. The relative proportions between the RNN and the transformer are controlled by a data-driven gated mechanism supported by significantly improved performance.

- I.3 Validity Proof of the Proposed Solution:
    - Extensive experiment setting showcase the effectiveness of RSA modules demonstrated by four sequential learning tasks namely:
        - Time series forecasting;
        - Regular language learning;
        - Code language modelling;
        - Natural language modeling.
    - Transformers are augmented to various variants and compared with unmodified benchmarks and Block-Recurrent Transformers (BRT) which integrate recurrence and self-attention mechanisms. The conducted evaluation demonstrates the superiority of the presented method.

II Strengths:

II.1 From a structural (organization) point of view:
- The set is well-structured;
- The method is Cleary presented with descriptive figures. 

II.2 From an analytical (development) point of view:
- The motivation is Clearly presented;
- Experimental setting confirms the benefit of the proposed as the superiority of the solution is illustrated by empirical evidence;
- Theoretical evidence is provided for the design of the solution ;
- The discussion related to the comparison conducted with several features of existing methods is appreciated.

II.3 From a perspective of soundness (unity, and coherence) and completeness (correctness):
- The strength points mentioned above are sufficient evidence of the soundness and completeness of the paper. 
- An additional point reinforcing the strengths mentioned above is the active interaction of the authors during the rebuttal period and their openness to concerns and questions raised by the reviewers. The openness followed by the active interactions and persistence in answering questions and addressing concerns related to the submission demonstrate the author's intellectual honesty and good faith in conveying the details of the proposed solution to the reader as well as its benefits. Such a rebuttal is a great example of how rebuttals can be conducted to reinforce the value of the presented solution in the paper.

III Addressing what can be thought of as weaknesses:

- There are almost no weaknesses to point out. All of the concerns raised by the reviewers were addressed. Something to point out to the authors is the density aspect of the text. The theory behind the design of the method is smoothly introduced and an attentive reader could appreciate this. Yet, the appearance of the presented equations, lemmas, and propositions is dense.
- Most of the points that could be thought of as weaknesses have been and the reviewers agree Unanimously on the acceptance of the submission.

IV. Potential of the paper:

- IV.1 From a Potential perspective (Potential of the paper to the community): The proposed solution has a great potential to be of benefit to the whole community. The application, extension, and investigation of other models based on the presented solution can also be of great benefit to the representation learning community in general.



justification_for_why_not_lower_score: Reviewers unanimously recommended the acceptance of the paper with an average grade of 7.33 and strong average confidence of 4. The proposed solution has the potential to be impactful as it integrates recurrence in self-attention in a very principled way. As stated in Section V.1 of the meta-review, the presented solution has great potential to be of benefit to the whole community. The application, extension, and investigation of other models based on the presented solution can also be of great benefit to the representation learning community in general.
Decision: Accept: notable-top-5%