Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: [Overview]

In this paper, the authors proposed a simple but effective way to augmentation the language model through memorization. Specifically, after obtaining a language model on a dataset, the model further uses the dataset to build a lookup table and then a k-nearest neighbor is used to searching the closest tokens for a token during inference. Based on this, the output distribution of a target token during the inference time would be modified accordingly. Through a comprehensive experiments and ablation studies, the authors showed that the proposed strategy can improve the performance of language models significantly for both the in-domain and out-domain testing scenarios. This is very insightful considering recently a lot of language models are focusing on increasing the size of model and training data.

[Pros]:

Overall I think the paper is well-written and presents clearly. Detailed points below:

1. the authors proposed a simple but effective method for increasing the generalization ability of language model through a memorization strategy. Specifically, the authors proposed to build a lookup table which memorizes the representation and output token pairs which are then used for the inference of language model. Different from conventional way, the proposed strategy does not introduce any more parameters in the model and also does not need any more training or fine-tuning on the target dataset.

2. The authors showed that the proposed strategy can improve the performance of language generation model (i.e., transformer) without any extra training or data, as shown in Table 1. Also, using the continuous caches  with KNN-LM further improve the performance.

3. Besides the main results shown in Table 1 and Table 2, the authors also showed using kNN-LM can probably outperforms the model which is directly trained on it. Also, it also supports domain adaptation from one language domain to another domain.

4. Finally, the authors presented a number of ablation studies to investigate how the performance is affected by the method of building datastore, including the size of nearest neighbor, the interpolation parameter, etc. These results are also insightful and meaningful for the readers to understand the method.

[Cons]:

I think this paper is a solid paper. So I would have some suggestions below:

1. The first concern about the method is the efficiency. At page 3, the authors mentioned that the proposed strategy will bring more time cost. It would be good if the authors can perform more systematical analysis on the time cost of building the datastore and inference for the proposed model. 

2. Second, the authors should not only evaluate the proposed method based on transformers. It would be good to test on various language models to verify the generalization ability across different models, including the old-fashioned one like RNN and CNN.

3. Also, the authors should try to extend the proposed model to other language tasks, such as translation.

[Summary]

In this paper, the authors introduced a simple but effective method to augment the pertained language model through memorizations. Though this is not absolutely new and relatively simple , the authors successfully demonstrate that it can be applied to improve the generation of language model much. The. thorough ablation studies help to understand the property of the proposed strategy. I think this paper overall is insightful and thoughtful. It would be good to see the authors add more analysis on the computational complexity and also evaluate on more type of language models.

review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: 
Summary:
The authors extend a pretrained LM by interpolating its next word distribution with a KNN model. The authors show retrieving nearest neighbor from corpus achieve quite large perplexity decrease in several language modeling benchmarks.

Decision:
Overall, the idea seems simple but is quite effective. Even with some discussions on the related work with cache based LM and the work that use training examples explicitly, I feel it is a simple extension/usage of previous approaches. Hence I am borderline with my decision.

Supporting argument:
1. The proposed idea uses KNN to look up training examples for interpolating the prediction. As discussed by the authors, this approach is effective in factual knowledge, names, and near-duplicate sentences.
2. There are several experiments and ablation study in showing the effectiveness of the approach.
3. The related work that uses training examples explicitly is quite similar to the proposed approach, though the authors claim that one is at the level of individual tokens and the other is the whole training sentences.

Additional feedback:
1. In reference, ‘Bert’ -> ‘BERT’
2. Missing reference: Yogatama et al., Memory Architectures in Recurrent Neural Network Language Models, 2018, https://arxiv.org/abs/1410.3916, https://arxiv.org/abs/1803.02400
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: This work utilizes the kNN method on dense vectors to augment the LMs. The method is simple and straightforward, meanwhile, the performance seems great if only in terms of PPL.
Three of my most concerns:
1)	It seems that this approach heavily relies on the similarity of context distribution between the training and test set. Intuitively, higher performance will be achieved with more similar examples between training and test set. This question should be discussed more in this work. This similarity cannot always satisfied in practice, I thus quite doubt the proposed method can work for general case.
2)	The evaluation is only done for PPL, I notice the LM was trained in a corpus scale as pre-trained BERT, though none of real downstream tasks were evaluated like BERT. Expect to see some MRC or NLI results with the proposed LM.
3)	Furthermore, though FAISS is very fast, it is hard to get great results with only a small datastore which makes the retrieving slow. So it seems not suitable for tasks such as generations but maybe open-domain QA can be the scene for this method. It would be great if there are some experiments on such tasks, and also combining with models such as BERT could be much better and convincing.

Questions
What about other distance functions such as cosine distance? The author only said L2 is better but there is no analysis on it.

review_title: Official Blind Review #1318

Meta-review: This paper proposes an idea of using a pre-trained language model on a potentially smaller set of text, and interpolating it with a k-nearest neighbor model over a large datastore. The authors provide extensive evaluation and insightful results. Two reviewers vote for accepting the paper, and one reviewer is negative. After considering the points made by reviewers, the AC decided that the paper carries value for the community and should be accepted.
Decision: Accept (Poster)