Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: 

This paper mainly proposes a modification of well-studied Transformer architecture that is widely used for the text generation tasks, i.e., language modeling and machine translation.
The main idea is to consist of Transformer architecture only with self-attention layers. In other words, the proposed method discards the feed-forward layers and augment the self-attention layers with persistent memory vectors.
They conduct experiments on character and word-based language modeling and show the on-par or slightly better performance comparing with the standard Transformer language model and other similar Transformer modifications.
 
The proposed method is just a modification of the existing neural network architecture.
Moreover, their method did not significantly improve the performance of language modeling in the experiments.
From these perspectives, the proposed method is not innovative.
 
However, the Transformer architecture is currently applied to a wide variety of tasks in text generation. Therefore, the proposed method can be largely influential to the community. 
Actually, I like an idea discarding the feed-forward (sub-)layers in the Transformer, whose intuitive (and theoretical) role is not much discussed in the literature.


review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: This paper considers an architecture change to the transformer in which they swap the feedforward subcomponent of the standard transformer with an "attention only" variant that includes persistent "memory" vectors. The model is evaluated against a suite of baselines on the tasks of character- and word-level language modeling. Combining this "all attention" approach with adaptive span yields results about equivalent to the SOTA, in some cases with fewer parameters than existing models. The authors do a nice job of presenting ablation results. A key finding here, for example, is that the a model stripped of both persistent vectors and the feedforward sublayer performs poorly. 

Overall, I'm on the fence regarding this submission. This is solid work, and the idea of exploiting persistent representations in the transformer seems promising. But the architecture change here is relatively minor, and the gains seem somewhat minor (the exception may be in Table 2 which shows equivalent performance with half the parameters to previous SOTA, but then no other model is in the range of O(100m) parameters, so hard to know what's going on here). 

One thing I would have liked is more motivation. If equivalence with fewer parameters is the main aim, then the model seems to fair reasonably well but the results are not really compelling. If, on the other hand, the authors are primarily interested in exploiting persistence, then I think this could have been investigated a bit more exhaustively, and perhaps the focus need not be on necessarily replacing the feedforward subcomponent (although that is one reasonable strategy). I do not see a huge inherent advantage to removing the feedforward layer, and it seems like there are alternative strategies --- at least equally as good --- to reduce parameters. 

A question for the authors: did you consider a vanilla Transformer with persistent memory vectors added? This would be something like adding a constant dummy input (independent of the example) that would be passed forward and arbitrarily transformed. I guess the meta-point here is that it doesn't seem to me that the persistent representations and the feedforward sublayer are necessarily mutually exclusive.

As a minor comment, I think Eq 13 is redundant since it literally repeats Eq. 4 save for swapping in $C_t^+$ for $C_t$.
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: This paper proposes a simple modification to the ubiquitous Transformer model. Noticing that the feed-forward layer of a Transformer layer looks a bit like an attention over "persistent" memory vectors, the authors propose to explicitly incorporate this notion directly into the self-attention layer. This involves concatenating the contextual representations with global, learned memory vectors, which are attended over. 

The model is tested on widedly-utilized character/word-level language modeling benchmarks, where it is found to outperform, or be on par with, existing models while using fewer number of parameters. 

Insofar as architectural advancements can translate to general improvements across multiple NLP tasks, this paper could be seen as important. However, I am not sure that in 2019, demonstrating arguably-marginal perplexity improvements on standard datasets is enough. I would love to see if this type of layer can result in better conditional generation models (e.g. translation, summarization), or can train GPT2/BERT/XLNet-style models whose representations better transfer to other tasks.

I had some further questions/comments:

- I found the motivation of the persistent memory vector as replacing the FF-layer somewhat tenuous. Eq(5) is definitely different from Eq(9)! In my opinion this work can be better motivated/presented as just a standalone modification to the Transformer layer.

- While it is impressive that the proposed approach performs better (or on par with, in the case of character-level language modeling) than the previous state-of-the-art models which are larger, to me it is not immediately clear if the benefit is coming from the proposed modifications, or something else (e.g. some of the things mentioned in 4.3). I understand most of this is taking from prior work, but as we all know, in deep models various architectural/hyperparameter modifications can interact in unexpected ways. Therefore, at a minimum, I would like to see the performance of a comparable model with the same exact setting, except for the persistent attention layer (i.e. N = 0 but using the feedforward sublayers). (If I understand the work correctly, this baseline should have comparable number of parameters?)

- The ablation studies are good but it would be good to see them on the word-level task as well.

- What is the performance of the FF-attn baseline with the same depth? (I understand the number of parameters would be  larger, but does this model perform as well as the Dai et al. 2019 work?)

- What if you combine this with the FF sublayer as well?

- Have you tried qualitatively analyzing the attention distributions? What are some examples in which the persistent memory vectors are attended to most? What are some examples in which the attention distribution for this model differs considerly versus regular self-attention layers?

[EDIT after author rebuttal]
Thank you very much for the rebuttal. I have updated my score to reflect the latest iteration of the paper.
review_title: Official Blind Review #3

Meta-review: This paper proposes a modification to the Transformer architecture in which the self-attention and feed-forward layer are merged into a self-attention layer with "persistent" memory vectors. This involves concatenating the contextual representations with global, learned memory vectors, which are attended over. Experiments show slight gains in character and word-level language modeling benchmarks. 

While the proposed architectural changes are interesting, they are also rather minor and had a small impact in performance and in number of model parameters. The motivation of the persistent memory vector as replacing the FF-layer is a bit tenuous since Eqs 5 and 9 are substantially different. Overall the contribution seems a bit thin for a ICLR paper. I suggest more analysis and possibly experimentation in other tasks in a future iteration of this paper.
Decision: Reject