Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: The paper proposes recurrent connections between two adjacent Transformers, which transfers the previous context to the next step. This is a practically useful technique, improving the performance (perplexity in the experiments), and worth publishing. However, I have some comments and questions about the article.

Section 5.3 is an interesting question. The authors argue that topical information or so between adjacent windows is propagated. Although it is a plausible argument, it seems like it is hardly supported by table 3.

The authors said more complex recurrence modules do not make any significant difference. Then, the authors need to explain why the variations do not matter. For example, the authors fixed l_ins to be 2, without an explanation.  

It is interesting to see the relationship between the overlap length and improvement using the recurrent connection. It would be better to have further discussion about the relation and different roles. 

The Transformer model is also fine-tuned with the recurrent connection. So, I was wondering if the fine-tuning improves the Transformer model too. It would be interesting to compare the updated Transformer to the previous one. 

In Eq. (1), is there 1/T? 
review_title: practically useful method for a long context size 

final_score: rating: 7: Good paper, accept
review_content: Summary:

The paper proposed to add a recurrent component to pretrained transformers. The component pools the hidden states of a context window and passes it to the next context window as an additional input to the self-attention layer. The component reduces the memory usage at both training and inference time, and enables the Transformer model to work on a longer sequence. The component is evaluated on two language modeling datasets and outperforms baseline models.

Reasons for score: 

I vote for accepting the paper. The paper proposed a nice and simple way to make use of the existing pretrained Transformers with reduced memory usage and extended sequence length. This should benefit practitioners who want to apply these language models on a more diverse set of downstream tasks where the sequence length doesn’t fit the one from the original pretrained model. The results presented in the paper are significant. The paper is well-written and easy to follow.

Comments:

1. It would be better to include the failure results stated in the end of section 3.1. I’m surprised that a key-value pair can boost the performance that much. 
2. The paper should add more content on differentiating with Transformer-XL. I believe the difference is more than relative embeddings. For example, each Transformer-XL layer attends to an earlier layer of previous timestep, this convolutional operation making the structure no longer “recurrent”.

Typos: 
- Third line in section 3.1: “at position t” -> “at position i”

review_title: A nice improvement for pretrained Transformers

final_score: rating: 4: Ok but not good enough - rejection
review_content: The goal of this work is to enable existing pre-trained transformers (e.g. GPT-2) to operate over long input contexts. This is achieved by breaking the input sequence into segments and processing each segment through the transformers while allowing tokens in the current segment to attend over a summary vector of the tokens in the previous segment. The summary vector is created as a weighted combination of the tokens in the summarized segment. Thus the summary vector introduces recurrence where each segment can use information from the previous segment. These modifications yield a better language model for long input texts. 

The main benefits of this approach are as follows: (1) The modifications yield a better language model for long input texts, especially when compared to a tiling based approach (2) Potential for reducing memory footprint in these models by shrinking the amount of text that is to be processed in one-go. 

My main concern with the paper is that unfortunately the evaluation is only limited to perplexity numbers on a couple of datasets. While this is a useful metric for evaluation, this alone is inadequate to demonstrate the quality of the model as a text generating system or as a language model that will be fine-tuned for target tasks or to understand how much impact the model will have in these applications.

-- For the model to be considered as a text generation system, there needs to be some human evaluation of the generated outputs. There are a small number of examples in the paper but that is not enough for a quantitative assessment. To clearly establish the benefits of the proposed modification it would be even better to consider generation tasks where conditioning on long inputs is essential. 

-- How will model fare when used in a target task defined over long input contexts? The related work section includes some papers that evaluate on such tasks. For example on target task could be HotpotQA, which requires QA over ten paragraphs which easily exceed the 512 token limits. It is important to know how the proposed recurrence model compares to tiling GPT (disjoint version) or other simpler approaches on these long input tasks.

-- Another key strength of the model is that it potentially allows for processing the input in smaller segments. While perplexity gains are helpful, here again there is a missed opportunity in terms of human evaluation of the generated outputs over shorter segments, and the impact of these choices in different applications. 


To reiterate, this paper presents a very nice idea to a well-motivated problem. The executed experiments show that this idea is likely to work but the gaps in experimentation leave much room for speculation about the potential impact of this approach in end applications. 

review_title: Interesting idea but evaluation is inadequate.

Meta-review: In this paper, the authors propose to add recurrence to pre-trained language models such as GPT-2 or BERT. The idea is similar to the compressive transformer paper: a small module is added to the network, and used to compress the representations from the previous chunk of data from the sequence to a single vector. Then, this vector is added to the keys and values of the self-attention module when processing the next chunk. The main contribution of the paper is to show that this technique can be added to pre-trained models at fine-tuning time.  The main concerns regarding the paper are technical novelty and limited empirical results. The idea of adding recurrence to transformers was previously explored in compressive transformer, and many previous work have considered adding modules with small number of parameters at fine-tuning time. Moreover, I do not believe that the empirical section is strong enough to justify the acceptance of the paper, as the method is only evaluated on two language modeling tasks (and one early experiment on HotpotQA). The baselines are weak, and thus, the results are not convincing. For these reasons, I weakly recommend to reject the paper, and encourage the authors to make the empirical section stronger.
Decision: Reject