Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: 

This paper explores the effectiveness of the Transformer architecture to the lossless data compression problem.
It also proposes a method to periodically revisit tokens that were already compressed for adopting the task setting of data compression, which is essentially online learning of sequence models. 
 
The authors conduct their experiments on the enwik8 benchmark.
They show that the Transformer architecture obtains state-of-the-art results.
 
This paper is basically easy to follow, but several typos and statements that should be improved.
The problem setting to tackle is interesting.
However, applying a deep neural network approach to data compression problem has already been discussed in several previous studies.
Therefore, the novelty of this paper is somewhat limited.
 
 
My main concern of this paper is that the proposed method was only evaluated on a single benchmark data.
I believe that it is a bit weak to support the effectiveness of the proposed method.
The authors should evaluate their method on several benchmark datasets that have different aspects, such as settings with easy and hard to compress. 
 
 
Minor comment:
In Section 4.2, there is a missing citation.
... we do not use Adaptive Inputs (Baevski & Auli, 2018; ?) ...
Please check and fix it.


review_title: Official Blind Review #2

final_score: rating: 1: Reject
review_content: This paper provides a method for lossless compression of text. It's heavily inspired by the language modelling methods that have been developed for the purposes of predicting the next character/word in a sentence, and it uses this idea as its backbone. The only difference is that the results are presented in the compression setting.

I think we should reject this paper due to the following reasons:
- I don't see enough of a difference between this and previous work
- the results are nowhere near SoTA for compression, despite the method being sold to this community
- there are other papers that do lossless neural compression that could have been used to make a comparison rather than making no comparison at all. For example, "Practical Full Resolution Learned Lossless Image Compression" (CVPR 2019) provides a framework for image rather than text, but that could be adapted to this field without any major changes (predict convolutionally characters, rather than RGB values).
- there's no comparison even with BERT (how well it do to predict the next character vs. this)...
- no runtime numbers
- no reproducibility discussion (i.e., how can I guarantee that my decoder can get exactly the same numbers as my encoder so that I can decompress on a different machine)
- no discussion about whether files were created/decompressed (this is ABSOLUTELY CRUCIAL for compression papers to discuss)

Overall, I am not excited about this paper, and unless the authors put a lot more into it, there's just not enough novelty to justify a publication at ICLR.
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: Summary: 
The paper investigates using the transformer architecture for neural network-based lossless compression of text. The resulting model, obtained through a thorough investigation of the architecture hyper-parameters are on par with standard SOTA compression. The paper is well-written. In particular the authors have done a great job reviewing existing compression literature and positioning their method within the space of prior work.

Recommendation: Weak Reject
While the paper considers an interesting application of the Transformer architecture, and is well-written, it is of limited novelty. Specifically, the bulk of the paper is concerned with describing experimental results of a thorough (but standard) hyper-parameter search - considering things like Transformer context size, learning rate (schedule), number of layers and key, value, query dimensionality; and does not offer any new architectural modifications / insights.

Furthermore, only a single dataset - enwik8 - is considered in the experimental validation and little attention is given to the description of the dataset split and any distribution differences between splits. Taken together, the existing experimental setup potentially creates an unfair advantage for the neural network-based methods - while the standard methods can be expected to perform similarly across a wide range of datasets / texts, the neural-network based methods have been trained and tested on very similar data and could be expected to perform well on these data, but not in case of a distributional shift (e.g. compressing legal texts instead of Wikipedia). The paper does not answer the question of whether or not this is true.

Furthermore, similar to autoregressive models, transformers are known to be slow at inference time. I expect this to lead to very slow decoding. Therefore, methods in table 1 should be compared in compression/decompression time to give a better overview of the practical impact of this work. 

Taken together, in its current form the paper may be better suited for a workshop publication rather than a full conference paper.

Major comments:
1. For reasons mentioned above, the paper should include additional experimental evaluation. In particular, it should consider the effect of training the model on one dataset, but evaluating it on another dataset; and discuss how differences in performance (if any) compare to standard methods.
2. Compression/decompression times of the proposed method should be compared against the other compression methods in table 1. I expect the proposed transformer to be slow at decompressing.
3. The paper does not contain the loss that the transformer model was used to optimize. I assume that it is the softmax cross entropy, but this is worth mentioning explicitly. It would also be worthwhile to explain the training procedure - for how many epochs was the model trained (see also next question), what was the dataset size? 
4. Description of the “training with revisits” is not very clear. My understanding is that it resembles a pass through the data, where some of it is considered again at specific intervals. My first assessment is that this should not be necessary - the data should already be considered multiple times during the training process.
a) The authors should provide a more detailed description of the training-with-revisits procedure, contrasting it specifically with a procedure where revisits are not done (i.e. normal training).
b) If the goal of the revisits training is to observe some training examples more than once, then it would be very interesting if simply training for a longer time (several epochs == passes through the data) has a similar effect.
c) Is there any motivation for the choice of the revisits hyper-parameters F and M? Was a different batch size used during the revisits training? Is the learning rate evolved during the revisits training phase or is it still decayed?

Minor comments:
1. There is some prior work on using Neural Networks for lossless image compression (e.g. [1], [2]. [3] that achieves SOTA compression ratios compared to standard methods. It may be interesting for the readers to mention these results. In particular the authors’ statement that “[...] purely neural network based models are still far from state of the art [...]” may give the wrong impression to the readers.
2. The authors mention that they “[...] propose several improvements to its (the Transformer) architecture and training to accelerate and stabilize [...] training”. In my view, the experiments described in the paper resemble a hyper-parameter search more than architectural improvements. The authors may want to clarify in the text which specific improvements they refer to.
3. Page 1, last paragraph: “[...] of all the important component [...]” -> “[...] of all the important components [...]”
4. Page 3: “[...] attention span size across all layers as it suggested [...]” -> “[...] attention span size across all layers as was suggested [...]”
5. Page 3: Missing references.
6. Page 3: Use of small n and capital N when talking about n-grams. Should be made consistent.
7. Page 8 (Conclusion): “wihtout” -> “without”


[1] F. H. Kingma, P. Abbeel, and J. Ho. Bit-Swap: recursive bits-back coding for lossless compression with hierarchical latent variables. In International Conference on Machine Learning (ICML), 2019.
[2] Emiel Hoogeboom, Jorn W. T. Peters, Rianne van den Berg, and Max Welling. Integer Discrete Flows and Lossless Compression. arXiv e-prints, 2019.
[3] Jonathan Ho, Evan Lohn, and Pieter Abbeel. Compression with Flows via Local Bits-Back Coding. arXiv e-prints, 2019.

review_title: Official Blind Review #3

Meta-review: The paper proposes to use transformers to do lossless data compression. The idea is simple and straightforward (with adding n-gram inputs). The initial submission considered one dataset, a new dataset was added in the rebuttal. Still, there is no runtime in the experiments (and Transformers can take a lot of time to train). Since this is more an experimental paper, this is crucial (and the improvements reports are very small and it is difficult to judge if there are significant).
Overall, there was a positive discussion between the authors and the reviewers. The reviewers commented that concerns have been addressed, but did not change the evaluation which is  unanimous reject.  
Decision: Reject