Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a method to model original texts and similar texts in a graph structure for language modeling with graph neural networks. In the graph, the nodes are texts or similar contexts and the edges are connections between the nodes. The new model achieves the new state-of-the-art on WikiText-103 and shows substantial improvements over other language modeling datasets such as One Billion Word and Enwiki8 datasets .
main_review: Strength:
- The authors propose an interesting hypothesis that referring to the training data could be helpful for language modeling, and they showed that the method is able to make considerable improvements over the vanilla LM.
- The method achieves the new SOTA on Wiki103, which is impressive

Weaknesses:
- The paper lacks a discussion part about the actual overhead for retrieval and the time overhead for running the model seems to be significant (8-20X slower). 
- Some hyperparameters seem to be chosen quite arbitrarily like l and r, maybe the authors could provide more insights as in why they chose such a small number. Does it affect the efficiency or the performance much?
summary_of_the_review: The paper proposes a retrieval augmentation to the language models and use GNNs on top of a graph structure of the original input and its neighbors and shows that it consistently improves over the vanilla lm model. The empirical results are good and the authors provide examples showing that the retrieved examples indeed help prediction in language models. The only concern is that through modeling additional neighboring contexts, the method introduces significant overhead in running time.
review_title: null

final_score: rating: 10: strong accept, should be highlighted at the conference
review_content: summary_of_the_paper: This work build a novel GNN-LM to do language modeling by using global context information. The proposed model is novel and quite different from previous LM structures. This work in my view draws the connection between traditional n-gram language model and neural language model. The overall performance is quite impressive in all standard LM datasets. Extensive ablation study is conducted to understand the model. 
main_review: I would recommend  accepting the paper based on the novel idea to build heterogeneous graph to do LM and the impressive model performance on PPL.  This work further extends KNN-LM to utilize not next tokens but all neighborhood information to get global context. This good combination of GNN and LM can be valuable to the community. The improvement on PPL also shows the importance to use global knowledge.  

The heterogeneous GNN is standard. I would like to know if the authors have thought about designing specific graph structure or avoid inter-context edge when building the graph or considering "is" and "are" are the same node when building the graph in Fig 1. 

I would consider KNN-LM is a special version of GNN-LM. Then why adding KNN can greatly further improve the model performance?
summary_of_the_review: Overall, base on the novel idea of creating global context graph, GNN-LM and show the significant improvements on all LM datasets, I would like to recommend accept this paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper presents a GNN based language model where neighbor contexts are retrieved, encoded via a graph neural network, and used to enhance generation. Evaluation on three benchmarks indicates that the proposed approach can outperform a bunch of baseline models. 

Contributions:
1. a new retrieval-augmented language model implemented via GNN techniques.
2. improvements over state-of-the-art models on three benchmarks. 
main_review: The work is among the studies of enhancing language generation by context retrieval, and the new idea is modeling the retrieved neighbor contexts through a graph neural network.  My concerns lie in the following aspects:

1. Comparison: it seems that the proposed model is compared with a retrieval-augmented baseline  only on WikiText-103. Why the comparison is not conducted on the other two benchmarks? Is it because there are no available results on the two other dataset? In this case, can you implement the model and make the comparison? Moreover, there have been many retrieval-augmented models, though some of them are not applied to LM tasks in the initial papers, then can you make adaptations and compare with them as well? Since the major contribution of the work is the GNN module, it would be important to demonstrate how useful the component is. 

2. Clarity: I feel confused at several places during the review process: (1) Eq.(5), the text below explains that f(\cdot) is a neural language model, but afterwards in Table 5, it seems that GNN is a better choice. What is the exact setting for this equation? (2) The first paragraph of Section 3, the authors say "we retrieve k=1024 nearest neighbors for each source token, among them the top 128 neighbors are used in graph, and all of them are used in computing the kNN-baed probability", then can I understand as the graph only involves 128 tokens? If it is, given GNN is better than Transformer in Table 5, how to calculate f(\cdot) in Eq.(5) for other tokens? (3) how do you train the transformer and the GNN？There are some comments above Figure 2, but do you train the two components iteratively? If yes, how do you set up the training procedure? (e.g., how many iterations do you do? how to warm up the parameters of the GNN?) (4) Eq. (6) is really confusing. You only compare w_t with the i-th token of w_{t_i}? Why don't you use soft similarity (e.g., Cosine)? 
summary_of_the_review: 1. Relatively incremental technical contribution to the community. 
2. Relatively weak comparison with baselines.
3. Many vague points that impede us from understanding (reproducing) the work. 

——————————————————————————————————————————————————————————————

The authors' response answered most of my questions. though I still feel that the technical contribution is big enough. I slightly raise my score accordingly.
review_title: null

Meta-review: This paper introduces a new type of language model, the GNN-LM, which uses a graph neural network to allow a language model to reference similar contexts in the training corpus in addition to the input context.  The empirical results are good, and the model sets a new SOTA on the benchmark Wikitext-103 corpus, as well as improving over strong baselines on two other language modeling datasets (enwiki8 and Billion Word Benchmark).  The main drawback, as noted by one reviewer, is the computational expense of the method with significant slowdowns compared to the baseline.

Two reviewers voted strong accept, with a third raising several concerns.  The largest concern was the lack of comparison to prior work, especially prior retrieval based methods on two datasets.  The authors responded with an ablation study comparing their method to KNN-LM and showed their proposed GNN-LM performs better.  Other concerns raised by the reviewer were the paper's lack of clarity (the authors should address the reviewers questions during the next revision) and incremental technical contribution.  Another reviewer highlighted the paper's novelty, and this AC agrees it is sufficient for publication.

Overall, the method is an interesting, if expensive, extension of retrieval based language models, and the empirical results support its effectiveness.
Decision: Accept (Spotlight)