Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: > Summary: The paper proposes to study three formulations (MemTransformer, MemCtrl and MemBottleneck) of memory-augmented self-attention transformers, and investigate the influence of adding memory tokens to the model to its overall performance. The authors claim via some experiments on MT and LM that memory augmentation is able to improve the canonical transformer models.

-------------------------

Post-rebuttal thoughts:

While the rebuttal of the authors did clarify some points, I think it is undeniable that the empirical results presented in the current version of the paper are still far from being convincing enough to claim this form of memory augmentation is a useful addition to the Transformers. The visualization of the memory augmentation via attention map is also too handwavy. 

While I appreciate the authors' effort to address my questions in the rebuttal phase, I will keep my current score.

-------------------------

**My general opinion**: 

I personally think that memory augmentation is a simple but potentially useful technique that explicitly adds memory to a deep network, which, as the authors mentioned in their paper, is not a new topic. Memory augmented networks have been around in many deep models, especially sequence models. However, I find this paper still not strong enough for me to recommend its acceptance in its current form, mainly because 1) the formulations are rather incremental; and 2) the empirical results are too weak to demonstrate (convincingly) the usefulness; and 3) I found some experimental settings uncommon and empirical analysis too handwavy. My detailed question and comments are below.

-------------------------------------

I have some detailed questions/comments.

1. In the MemBottleneck formulation, does the update rule of $A^\text{seq}$ take $X^\text{mem}$ or $H^\text{mem}$? Figure 1d seems to suggest that the memory update at a layer $i$ precedes the update on the sequence tokens. In addition, what is the difference between $H$ and $X$? Are they not the same sequence (i.e., the $H$ from a layer is passed in as $X$ of the next layer)?

2. In the MemBottleneck subsection, the paper claims that its cost "scales linearly with the size of the input sequence $O(N)$" instead of $O(N^2)$. But if the hypothesis of the authors are correct, that these memory modules serve to collect and "re-distribute" the sequence information, shouldn't one generally expect the memory size to be proportionate to the sequence length? For example, I certainly would expect that a sequence of length 8000 would certainly require a different memory size than a sequence length 40. The point is, one cannot simply assume "the size of the memory is constant", just like one cannot assume "the size of the sequence is constant". The more accurate way, for instance, is to say the complexity is $O(NM)$ with $M$ being the memory size.

3. My major concern with this paper is with its various empirical results and experimental settings:
   - i) In machine translation, how exactly does the encoder-decoder self-attention work in the three proposed Mem settings? E.g., does the decoder queries attend to $X^\text{mem}$ only (of the encoder output), or also the $X^\text{seq}$?
   - ii) How does MemCtrl models perform in the small setting of Table 1? (They seem to be slightly better on the larger setting)
   - iii) Overall, I found the numbers reported for WMT'14 de-en in Table 1 **too low** for me to convince of anything. Even though the *much more commonly used setting is WMT'14 en-de*, it should be overall easy to get a >29 BLEU score with a base Transformer model (e.g., [1,2] got >31 for the base Transformer). I am not sure why the authors halted their experiment at 20 epochs and 10 epochs (how many training steps are these though?). Hence, it's not clear to me whether these minor improvements in the below-expectation BLEU scores are truly indicative of the final performance of these models. 
    - iv) The paper says no beam search was used. Why? What are the results if you use beam_size = 5?
    - v) The MemBottleneck Skip Transformer is really just a weak form of a typical Transformer, except that you initialized the first layer of the hidden units to 0, and downsample the sequence from the length of $X^\text{seq}$ to the length of $X^\text{mem}$. So it's not a surprise to me that "MemBottleneck Skip Transformer 20" is significantly better than MemBottleneck 10/20. 
    - vi) The various ablation studies the authors made in the paper, in my opinion, exactly suggest that the usefulness of these memory augmentations are in doubt. For example, the huge gap between MemBottleneck and canonical transformer, as well as the negative gap between MemBottleneck 10 and MemBottleneck 20, I think is good evidence of this. Another example is the Table 2, where inference time with a larger memory size even degrades the performance. I didn't find a satisfactory explanation from the authors on this.
    - vii) What are the values in Table 3? Are they BLEU scores? Why are they <1?
    - viii) The paper says that "BLEU score of MemTransformer 10 with 5 `[mem]` tokens shows it is still able to translate with acceptable quality". This is false. 7 BLEU score is **A LOT** of difference. If you look at the generation result, they are qualitatively very different.
    - ix) How exactly did you use the memory augmentation for a causal/decoder Transformer for the language modeling task in Table 3? This is not explained in detail in the paper, but how did you handle the potential information leakage problem (i.e., we don't want future tokens to flow back to the past)? If you simply do it in the same way as in encoders of MT by appending it to the sequence, then you can either only *read* from the memory without writing (if you append to the start of the sequence), or only *write* to it without reading (if you append to the end of the sequence). I might be missing something here, and hopefully the authors can clarify.
    - x) The fact that there's no consistent improvement in Table 5 of the memory-augmented Transformer on the base Transformer is very concerning to me. Apparently, sometimes BERT-base is better than many of the MemTransformers.

4. One potentially interesting study that I think the authors can look into is the memory slot permutation. In a certain sense, the memory slots have the same representation embedding `[mem]`, and are mainly differentiated (among themselves) by their respective position embedding. However, it also seems from Figure 2 that when writing to memory, the original order of the sequence tokens is not necessarily preserved. So what if, in the intermediate layers of the MemTransformer, you permute the order of the memory slots? Do you expect that to affect the performance of the MemTransformer?

5. The other issue I found about the paper is that, although this is an empirical paper, there are still a bunch of not well-supported claims (i.e., handwavy claims) which the authors should have looked into. For example:
    - i) In section 3.1: "This can be due to the more complex architecture of the MemBottleneck that has twice more layers in the encoder part". But there's no formal investigation of this in the paper... for example, does an 8-layer MemTransformer (which has "twice as many layers as the 4-layer MemBottleneck) also perform badly?
    - i) In the caption of Figure 2: "Activity in the left top corner that involves first four tokens might indicate fusion of neighbour vectors by pairwise summation of `[mem]` tokens." This sounds like a posterior explanation... and it's not clear to me what this "fusion" is doing and why it's doing this. How is this helping MemTransformer? Does it occur in every head in every layer? Does it happen even if you only use memory size 5? These questions should be answered if you would like to claim this phenomenon.
    - ii) In the "memory to memory attention" paragraph: "If the diagonal attention is sharp, then corresponding memory vectors are added to themselves, so their content is amplified and became more error-tolerant." Do you have support for the error tolerance claim? Is it still "amplified" even if there is a LayerNorm immediately afterward, which would shrink things back anyway (i.e., $\text{LN}(x)$ and $\text{LN}(x+x)$ should be the same)?

-------------------------------------

Some minor points that didn't impact the score:

1. In the intro section, 'results in "blurring"' ---> it's not clear to me what this means. Please clarify.
2. In the very beginning of section 3, the authors used $N=4$ to refer to number of layers. But $N$ has been used to refer to sequence length before (e.g., when saying $O(N)$). 

- [1] https://www.ijcai.org/Proceedings/2020/0534.pdf
- [2] https://arxiv.org/pdf/2010.07638v1.pdf

review_title: Review from R5: Interesting exploration but the method and empirical studies are relatively weak

final_score: rating: 3: Clear rejection
review_content: #### Summary:
The paper brought up an interesting limitation of a Transformer network, that information about the context is stored mostly in the same element-wise representation, which might limit the processing of properties related to global information. This work proposed adding memory tokens to store non-local representations and creating memory bottleneck for the global information. The evaluation shows positive results adding memory to a Transformer network.

#### Weakness:
The idea is not entirely new; Memory augmented networks have been proposed in previous work and the paper does not differentiate the method from related work very well. 

-The paper is not clear about how to obtain the memory input token. According to the paper, the memory input token seems like a static vector; however, a memory augmented network can retrieve a dynamic vector for global information, with the ability to read and write to the memory. 

-The naive Mem Transformer Layer is counter intuitive, as there is little differentiation between the memory layer and the sequence layers. How the memory layer contains additional global information is not explained. 
-Figure 1c is confusing, are the arrows crossed or not?

-The formulation of memory representation and sequence representation are exactly the same for the MemCtrl Transformer, which again is very counter intuitive. It is intuitive that we should augment the sequence transformer layer with additional global information, however, we might not want the other way. The memory layers should be updated at coarser granularity. 

-The paper is poorly written with many errors in the paper. Please proof read the paper before submission. 

#### Detailed feedbacks:
The reviewer finds this paper hard to read as there is not a flow of story in the paper. The layout of the paper and the design of experiments seems very arbitrary. 

-The motivation for a memory network like proposed in the paper is not clearly stated. How the memory network can capture long-term dependencies and information is not clearly explained. The design of variants of Memory Transformers seems very arbitrary. 

-The paper also lacks a more detailed comparison with related work, like Transformer-XL, Star-Transformer, Longformer, ETC, etc.

-"global"->''global'' (make sure it looks correct in latex). 

-".Surprisingly"->". Surprisingly"

-Please improve the quality of writing and ask native speakers to proof read the paper.

review_title: Idea is not new and empirical results are not strong.

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper presents transformer extensions with global memory.  The basic idea is to augment the input sequence with special memory token [mem].  The authors explore three variants: (a) MemTransformer - just the augmented sequence, (b) MemCtrlTransformer - separate params for memory update, and (c) MemBottleneck - Bottleneck of memory used to update token representations. The authors show gains over vanilla transformer/pretrained baseline for machine translation, language modeling, and the GLUE setup. 

I think the idea in general is interesting but it's not particularly novel. The authors cite a lot of recent works in this area, and a particularly similar approach has been explored by Gupta et al in "GMAT: Global Memory Augmentation for Transformers" (authors should cite this work). The main problem with the paper is that the empirical results are quite weak and don't show a consistent trend with regards to utility of memory. 

Apart from the machine translation results, it's hard to make a case that the gains on GLUE and language modeling are significant. For the MT results, the baseline results look too weak and the authors don't cite any prior work. Through a quick scan of MT papers, I found papers reporting +6 BLEU scores on the same task and the same architecture (see https://arxiv.org/pdf/1904.09324.pdf).  This is concerning because the baseline model seems highly undertuned. 

Among the three variants, the authors only provide results for MemTransformer on language modeling and GLUE. 
For machine translation, some of the entries in Table 1 are missing, and the small and base model don't have the same entries.

Other minor comments:
* What do the authors mean by 2000 validation "segments"? Are the segments just sentences? Is it the standard dev set? If not, then why not? If it's the standard dev set, why not refer to it at as WMT-14 DE-EN dev set.
* The authors don't specify the number of attention heads in main text. I presume it's 8, as mentioned in appendix.
* In Table 1, the authors add a result with "MemCtrl Shared Transformer" which they say is the MemCtrl model with shared parameters. Isn't this the same as MemTransformer?
* The authors say in conclusion that "the speed of training positively correlates with the memory size". They don't provide any evidence for this, and if they train for the same number of epochs then increasing memory size should only increase the training time. 
* Again in conclusion, the authors say that "the memory controller learned by the model degrades only gradually when memory size is changed during inference." That is clearly not true given the results in Table 2. 

Suggested text edits:
* Section 1 - "They have unspecific [mem] tokens that can store global or copy of local information." -> Rephrase this/Expand on this. 
* Section 2.2 - "It minimally changes the original model architecture." -> There's no architecture change. Get rid of this line.
* Section 3 - " if the opposite is not stated." -> "if otherwise stated."
* Section 3.2 - "Kovaleva et al. (Kovaleva et al., 2019) " -> Use \citet



review_title: Interesting Idea But Weak Empirical Results

final_score: rating: 5: Marginally below acceptance threshold
review_content: The authors propose augmenting transformer architectures by adding trainable memory via introducing special memory tokens to the input sequence. They also explore different architecture extensions (memory bottleneck, dedicated layer for memory updates). The authors’ main hypothesis is that adding memory to Transformer-based architectures should result in better model performance. They present BLEU scores on WMTF-14 DE-EN translation and GLUE benchmark results to support that.

Pros:
+ Good qualitative analysis of attention patterns that emerge when memory tokens are added
+ A practically viable memory extension training scheme that can be used to fine-tune models with added memory tokens

Cons:
- The main claim of the paper is that presence of memory tokens positively correlates with the model performance. At the same time, adding memory tokens increases the model’s capacity, making the comparisons with the baseline unfair. The more rigorous approach would be to compare to a baseline with proportionally increased dimensions as having memory tokens may be akin to having additional space to store sequence token representations. Some of the attention patterns (write-to-memory -> store -> read-from-memory) indicate that this may be the reason for the improved model scores.
- The use of global tokens (like [CLS] token in Longformer as mentioned in the paper) is a well-known technique that undermines the paper’s novelty.
- The authors claim that models’ quality positively correlates with the memory size. However, this statement contradicts the findings in Table 1 and Table 5. The authors briefly discuss the lack of trainability in section 3.1. Still, the authors do not specify conditions under which adding more memory will benefit the model quality.
 
General comments/questions:
* Memory lesions experiments (table 2). Memory size change during inference seems to lead to drastic drops in model quality. At the same time, the models are not trained to perform in the absence of memory. What if the models are trained on different memory sizes from the beginning with memory size sampled per-batch?
* Memory extension experiments (table 3). Does the model forget how to work with smaller memory sizes during the memory extension process? E.g., what will memory lesions results look like for the memory extension model?
* Are any of the models for the GLUE benchmark trained using a memory extension training scheme?
* It’s not clear if all mem tokens use the same embeddings or if they use separate embeddings (e.g., [mem1], [mem2], etc)
* MemBottleneck results lack comparison with other extensions that lower the computational cost (O(N), O(N log N) extensions, etc)
* Relative positional embeddings seem to work better with memory tokens, are there any more experiments with relative positional embeddings except for the ones in table 5?
* Not all results in Table 1 are symmetrically available for both small and base models. This table lacks the results for MemCtrl Transformer for small models and lacks the results for MemBottleneck Transformer for base models which makes it harder to assess the viability of those architecture extensions. 
* No results for MemBottleneck and MemCtrl in table 5. At the moment, both MemBottleneck and MemCtrl look weak; having more results for those extensions may help.
* The authors can add FLOPS/params count to tables 1 and 5 to make a comparison of the models' more convenient.
 
On rating:
 
Although extending transformer architectures with memory tokens looks like a practically viable way to improve models’ quality, the experiments are not rigorous enough to confirm this is due to the memory mechanism and not because of the increased model capacity, which can be achieved by a simple model scaling. The well-documented use of global tokens in literature also diminishes the novelty of this paper.

review_title: Good qualitative analysis of attention patterns on memory tokens but weaker experimental results

final_score: rating: 3: Clear rejection
review_content: This paper proposes to augment transformer architectures with memory components. The high-level idea is to use multiple special “mem” tokens as additional inputs. Depending how the “mem” tokens’ representations interact with the true input sequence, three variants are studied: (a) in MemTransformer [“mem”, input] attends to [“mem”, input]; (b) MemCtrl is similar, but uses a separately parameterized module to calculate the “mem” representations; (c) in MemBottleneck, “mem” attends to [“mem”, input], and [input] attends only to “mem.” Experiments with machine translation, language modeling, and fine-tuning on the GLUE benchmark are conducted.

Overall the presentation of the paper could be significantly improved. For example, it lays out the technical sections in a way that assumes the readers have hands-on expertise with transformer architectures. Further, the motivation is not clear to me. It seems from the experiments that little practical gains can be expected, since MemTransformer never outperforms the transformer baseline in terms of accuracy or efficiency. This is probably fine if the paper studies a set of clear research problems with well-designed experiments and answers some interesting questions. But it is not the case. Last but not least, the experimental setting is flawed: e.g., the MT experiment cuts training at a suspiciously early stage, presumably contributing to the bad performance. In sum I do not think the paper is ready for ICLR.

Pros:
- Studying the transformers through the memory perspective is interesting.

Con:
- Writing can be significantly improved.
- Motivation is not clear: the proposed approach doesn’t seem to bring any interesting practical gain, nor does it answer any interesting research question.
- Experimental design is flawed, making the conclusions less convincing.

Detailed comments:
- I’m not against discussing related works in the introduction. But the current version is probably trying to scramble too much stuff into intro, making it tedious to read.
- I suggest walking through the transformer architecture in a more self-consistent way. I would be very surprised if one without much hands-on experience of transformers can understand what’s going on in section 2.
- The baselines’ performance in the MT experiments is far worse than what we usually see. I’m guessing this is because the training stops too early. Can the authors explain why they choose to do so? If limited computation is the concern, I recommend the authors to work with smaller datasets instead but use a more convincing settings. Same for other experiments.
- The experiments do not show any practical gain from MemTransformer. The paper talked about its linear complexity in input sequence length. This paper would be stronger if it can present convincing experiments and show that MemTransformer can have a better efficiency in practice.
- Section 3.2 would be much more clear and interesting if it opens with the research questions it’s trying to answer.
review_title: Review

Meta-review: The paper studies three kinds of memory-augmented Transformers, focusing on one (the MemTransformer, which adds [MEM] tokens to a document.)  This is a nice clean extension of Transformers and a topic well worth investigating.  Unfortunately, the experimental results were considered unconvincing:

 - The baselines were relatively weak
 - The experimental setting was unusual (eg only 10 epochs)
 - The experiments did not show consistent improvement

Overall the paper was considered below acceptable quality for ICLR.

Decision: Reject