Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper investigates a so-called "compressive transformer" approach. The idea is to compress distant past memories into a coarse-grained representation while keeping a fine-grained representation for close past memories.  A variety of compression techniques and training strategies have been investigated in the paper and verified using tasks from multiple domains including language modeling, speech synthesis and reinforcement learning. Particularly, the authors propose a new benchmark PG-19 for long-term sequence modeling.  

Overall, I found the work interesting and experiments are thorough and strong.   It is always great to see a new benchmark released to the community.  That being said, I have concerns regarding the paper.  The authors put huge amount of effort into the experiments but only describe the proposed technique in a very rough and abstract way, lacking necessary technical details to formulate the technique. What is the mathematical formulation of the problem?  How exactly the compression is carried out on various network architectures is not clear after reading the paper.  Also, I guess many readers including me do not have a perfect understanding of Fig. 1 although it shows something intuitively. (What is the difference between different colors? What is the difference between sequence, memory, and compressed memory?  What do the arrows mean? There is no explanation whatsoever either in the figure or in the caption).  This is the major concern I have regarding the paper.  Despite of the strong experimental presentation, lacking the technical details has significantly hurt the quality of the paper.  

P.S.  Thanks for the rebuttal.  I have lifted my score. 
review_title: Official Blind Review #1

final_score: rating: 8: Accept
review_content: ## Updated review

I have read the rebuttal. First I'd like to thank the authors for the detailled rebuttal. 
The latest version of the paper adressed all my concerns, hence I change my rating to Accept.

## Original review

This paper presents a new variation of the Transformer model, named Compressive Transformer. The key novelty of this model is to preserve long range memory in a compressed form, instead of discarding them as previous models have done. This improves the long-range dependencies modelling capabilities of the approach. The model is evaluated on two common language modelling benchmarks and yields state of the art results in both of them. The paper also introduces a new benchmark for long-range dependencies modelling composed of thousands of books. The paper finally presents an analysis of the compressed memory and provide some insights, including the fact that the attention model uses the compressed memory. The model is also evaluated on two other tasks: speech generation and reinforcement learning on videos.

I think this paper should be accepted, mainly because:
- The proposed model is novel as far as I can tell. 
- The presented approach is significant, as modelling long-range dependencies is an important milestone in sequence modelling.
- The new benchmark is a good addition.
- The comparison with the relevant literature is thorough and well done.
- The experiments are convincing and demonstrate the viability of the approach, although some aspects can be improved (see below).

Detailed comments:
- About the character-level language modelling on Enwik8, the improvement is very small, it seems that the task doesn't benefit from have long-range memory, could it be because character-level modelling is less dependent on the long-range past? can the authors comment on that? It would also been interesting to evaluate the gain of the memory, for instance by varying the size of the compressed memory from 0 to 1152. 
- The WikiText-103 evaluation is interesting, specially Table 6, which shows the advantages of the model. However when comparing with the literature, it's not clear if the performance gain is due to the compressed memory or to the network capacity. A study with different lengths of the compressed memory (starting at 0) would bring some insights about that.
- In Section 5.6.2: can the authors justify why the attention weights were split in only 6 bins? creating a trended curve on only 6 points could be problematic, and I don't see why more bins couldn't be used.
- The speech analysis section (5.7) is not very insightful. It shows that the proposed model is on par with WaveNet on unconstrained speech generation, which is not very useful and feels a bit half-finished. I think that the authors should either commit to this study by constraining the model with linguistic features like in (Oord et al. 2018) and evaluate it in a TTS framework with subjective evaluation or discard this section entirely. 

review_title: Official Blind Review #2

final_score: rating: 8: Accept
review_content: This paper proposes a way to compress past hidden states for modeling long sequences. Attention is used to query the compressed representation. The authors introduce several methods for compression such as convolution, pooling etc. The outcome is a versatile model that enables long-range sequence modeling, achieving strong results on not only language model tasks but also RL and speech. For testing and evaluating the modeling of really long context sequence modeling, the authors introduce PG-19, a new benchmark based on Project Gutenberg narratives. 

The idea is a simple and straightforward one. The choices of compression functions are intuitive and natural. The probably more interesting part of this paper is the training schemes designed to train the memory compression network. 

Results are very strong and there is a pretty diverse set of experiments. That said,  it seems like a huge amount of resources were spent on this work alone. It also seems like these models are not trivial to train (or get them to work). It would be interesting to find out how much resources were spent (in terms of preliminary experiments) to getting these models to start working decently. There are also no reports of parameter counts, which might make the experiments unfair. 

Achieving SOTA is one thing, which could be attributed to large resource pools and maybe larger parameter sizes of models.

Overall, I am voting for a weak accept. While this paper is more incremental and novelty may be slightly lacking, I think the breadth of experiments and competitive results warrants an acceptance. 

Several issues and questions for the authors:

1) Why are the results on PG-19 not reported in a Table format? Why are there no results of the base Transformer on PG-19? I think this is really necessary and should be reported.
2) The authors mention that this memory compression architecture enables long sequence modeling. However, is there an intended way of use for long-text that is not necessarily framed as a LM problem? For instance, results on NarrativeQA benchmark would be nice. 

UPDATE: I have read the author response and other reviewer's comments. I am happy with the efforts made by the authors and I am raising my score to 8 (accept). 


review_title: Official Blind Review #3

Meta-review: The paper proposes a "compressive transformer", an extension of the transformer, that keeps a compressed long term memory in addition to the fixed sized memory.  Both memories can be queried using attention weights.  Unlike TransfomerXL that discards the oldest memories, the authors propose to "compress" those memories.  The main contribution of this work is that that it introduces a model that can handle extremely long sequences. The authors also introduces a new language modeling dataset based on text from Project Gutenberg that has much longer sequences of words than existing datasets.  They provide comprehensive experiments comparing against different compression strategies and compares against previous methods, showing that this method is able to result in lower word-level perplexity. In addition, the authors also present evaluations on speech, and image sequences for RL.

Initially the paper received weak positive responses from the reviewers. The reviewers pointed out some clarity issues with details of the method and figures and some questions about design decisions. After rebuttal, all of the reviewers expressed that they were very satisfied with the authors responses and increased their scores (for a final of 2 accepts and 1 weak accept).

The authors have provided a thorough and well-written paper, with comprehensive and convincing experiments. In addition, the ability to model long-range sequences and dependencies is an important problem and the AC agrees that this paper makes a solid contribution in tackling that problem.  Thus, acceptance is recommended.
Decision: Accept (Poster)