Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 3: Clear rejection
review_content: Summary: 
The paper presents a new model for the task of language modeling especially suited for longer sequences. This new model dubbed as Memformer consists of Transformer encoder-decoder and a memory module to store the past information from the encoder outputs. The encoder bidirectionally attends to the immediate previous sequence/segment information and to the memory module, which is designed to capture useful information from the past history of the full sequence. The idea is that by bidirectionally attending simultaneously to the previous input segment and to a memory module, the decoder should be able to improve its generation capabilities.

Pros:
The motivation of the proposed model is interesting, which is to bidirectionally encode the past segment information and a memory module to capture rich signals from the entire history. 

Cons:
- The main drawback of this paper is the lack of controlled experiments in the results section. Due to this, fair comparisons to the previous language modeling results in the literature is not possible and thus merits of the approach are neither convincing nor clear.
- To evaluate performance of the Memformer model, the authors present results on the Wikitext-103 dataset. However, to compare with the perplexity results from baseline models such as Transformer-XL and Compressive Transformer, the authors don't include the results from the original papers and instead re-compute it under simplified settings. However, to report progress in a widely studied task such as language modeling, it is not fair to not compare against highly-cited state-of-the-art results.
- In Section 3.1, it is mentioned that byte-pair encodings are used to represent words. Are the perplexity results also computed over BPE tokens? If so, then I feel this evaluation scheme is inconsistent with the standard language modeling evaluation of tokens which is done over words (or linguistic units).
- Although the results in Table 3.2 show that Memformer models obtains small performance gains in perplexity over baselines, but these results are rather marginal improvements and in the absence of statistical significance testing results, it can't be really understood if these are actual performance gains or due to randomness in the training process.
- As the core of the experimental results are on Wikitext-103, it is unclear if the experimental findings would generalize. Currently, the paper lacks comparisons on other datasets such as PG-19 from Compressive Transformers.
- The authors propose a multi-task training approach for language modeling. However, the motivation and benefits of why doing this is actually needed is not clearly illustrated and the perplexity gains seem to be small.
- In Section 2.1, the motivation of simplified relative position encoding is not presented. 
- In paragraph 3, it is mentioned that due to uni-directional attention is Transformer-XL style language models, the memory may not have enough capacity to retain important information. However, this does not necessarily holds true for different tasks. Can the authors include a more detailed explanation or cite a prior work that illustrates this phenomenon?


Writing Issues:
- abstract 1st line: The mention of "remarkable accomplishments" is very vague in the context of applicability of Transformer models.
- compatible with other self-supervised tasks: compatibility does not necessarily imply that such language models would be useful in self-supervised tasks.
- Introduction section is not well-written. For example, the transition from first paragraph to second paragraph is rather abrupt, paragraph 3: Transformers and its followers: this is a very informal writing style, something that is not really suited when submitting to a publication.
- Sec 2.2.1: The first sentence in third paragraph - "Figure 1b is an assumed language model." is grammatically incorrect. 
review_title: This is very much a work in progress, results are not directly comparable with previous works

final_score: rating: 4: Ok but not good enough - rejection
review_content: Summary:
This paper proposes an encoder-decoder memory-augmented language model, called Memformer. Similar to recurrent neural networks, the memory part is updated as a new input token comes in. The author presents a new optimization method, which is a variant of gradient checkpointing.

Reasons for score:
Contrary to the paper's argument, I am not sure that Memformer improves computational complexity or performance on long-range sequence modeling.

Concerns/Questions:
- Contributions of the newly proposed methods (increased model capacity from memory augmentation, specially designed attention, multi-task self-supervised learning, simplified relative position embedding) to the performance are mixed. A more thorough ablation study is required.
- To evaluate the effectiveness of Memformer in long-range sequence modeling, I would recommend exploring other settings than WikiTxt-103 with Transformer-XL (or Compressive Transformer). I suggest following experimental settings in the Longformer paper [1].
- I feel that the memory size of 32 is unsatisfactory. The author should provide a plausible explanation about why increasing memory size larger than 32 is not helpful. To add, Lample et al. propose product key memory using 512^2 memory slots [2].
- I am curious whether the Memformer using a memory size of 32 is really doing well in long-term modeling even though memory vectors can memorize information from multiple tokens.
- The setting of the main experiment is not valid. Comparison of Transformer-XL and Compressive Transformer with Memformer by reducing memory size (attention length) is not fair because they are designed to capture long-term context.
- The number of parameters in Table 1 is quite far from the values in the original Transformer-XL paper. Could you elaborate on it? 
- Could you provide detail of speed measurement in Table 1?
- Can we try a memory replacement policy like LRU (Least Recently Used) via a sparse update?

Minor comment:
- The name of memory *cross* attention and memory *slot* attention is not intuitive.
- Mention of the text continuation task appears in Section 2.2.3 before the definition in Section 2.3.

[1] Iz Beltagy, Matthew E. Peters, and Arman Cohan. “Longformer: The Long-Document Transformer”
[2] Guillaume Lample, Alexandre Sablayrolles, Marc'Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. “Large Memory Layers with Product Keys”
review_title: Conceptually neat idea but practical usefulness is questionable

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper is clearly written and explains the concept well. Keeping a memory part of transformers network enables Memformer to achieve O(n) complexity in time and O(1) in space. It is particularly helpful for handling long sequences. 
The authors proposed Memory replay Backpropagation scheme which is more effective and efficient than gradient checkpointing. 
The authors draw a fair comparison with long-sequence models (Transformer XL and Compressive Transformer) on Wikitext-103 dataset.
The concept seems to be a mixture of some work done in the past already. However, making it work and slot and cross attentions tricks seems to be a good choice.
 
Weakness:

1. Lack of experimentation: The authors compared other two transformer networks on just one dataset and presented their results with a hypothesis. The hypothesis is hard to be validated on just one task. The claims that the architecture can be scaled to any task is not justified with any empirical evidence. 
2. In continuation with previous point, the idea usage and applicability is not justified. There are sparse transformers already like Longformer, BigBird that can process long documents too. The lack of experimentation doesnot validate the usefulness of the architecture. 
3. The memory scaling seems to be a problem. Also clearly seen from the graph and the experiments. Effect of memory size should further improve the results which is not there. It is left for future work by the authors which doesnot go well with the hypothesis. Further, the restriction on the memory size is too small. With the advent of larger models, the usage of smaller memory seems implausible. 

Minor comments:
* The multi task sampling ratio [0.6, 0.3, 0.1] during training is not justified. Is it randomly chosen or was different ratios tried?

review_title: A good concept but lack experimentation

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes a new style transformer with external memory, which is updated and used through an attention mechanism. They also propose a new algorithm to train the memory, Memory Replay Back-Propagation (MRBP). The memory consists of key-value pair data and is recurrently updated after the segment encoding. Through this memory, it can attend the past knowledge without the limitation of the maximum temporal range. The MRBP algorithm trains the memory through the local back-propagation of loss to reduce memory overhead.

Strengths:
1. They propose a new type of transformer augmented explicit memory, which contains the entire past knowledge, so it can attend the past knowledge without the computational overhead.
2. They also propose a new training algorithm, MRBP to update the memory with the local back-propagation of loss, which makes sense and they showed empirical results about how much this algorithm reduces memory overhead efficiently.
3. They evaluated their method with baselines in the aspect of the number of parameters and training speed and performance for various memory sizes and reported plentiful ablation study results too.

Weaknesses:
1. I think that section 2.1 requires an additional description: (e.g., in (a), what is the meaning of W_rE_{x_j}?, in (c), how W_k E_{x_j} can be the global content bias?)
2. This Memformer requires recurrent processing segment by segment, which can be another limitation on training or processing (can require more time than naive Transformer, which can process parallelly). I think that it is a fundamental weakness on this type of Transformer and if there is the comparison with naive Transformer (if the device can train it), it would be interesting for me (I don't think that it is a critical weakness).

The correctness of their claim and Clarity:

This paper is well written and looks correct except for some parts like requiring more descriptions or related to the previous method.

Additional feedback:

Thank you for submitting it. I enjoyed reading it. 

However for me, section 2.1 is hard to understand, if there is more description, then it can be easier. 

And, you mentioned Transformer XL has a` theoretical maximum temporal range, but by using the hidden states of the previous segment (the authors called a recurrent connection between the segments),  Transformer XL can access the past knowledge. Could you explain more the theoretical maximum temporal range?

This Memformer looks like the combination of the Transformer and Recurrent module through the explicit memory. Like RNN, it doesn't need incremental memory overhead as the sequence is going on through the memory, and like the Transformer, it can predict the output through the attention modules. As I mentioned in the weakness part, exploring the best combination of RNN and the Transformer can be interesting.

Minor things are
On page 6, (L x Nm + c x Ncm -> (L x Nm + c x Ncm)
On page 7, when increasing the memory size from 1 to 8 -> 128


review_title: Official Blind Review by reviewer3

Meta-review: This paper introduces a new model, called Memformer, that combines the strength of transformer networks and recurrent neural networks. While the reviewers found the idea interesting, they also raised issues regarding the experimental section. In particular, they found the results unconvincing, because of weak baselines, non standard experimental settings (eg. using reporting perplexity results on BPE tokens), or evaluating on only one dataset. These concerns were not well addressed by the rebuttal. For these reasons, I recommend to reject the paper.
Decision: Reject