Venue: ICLR 2022 Conference

Reviews:

final_score: rating: null
review_content: I have gone through the reviews by other reviewers, the responses by the authors and the revised submission.

I thank the authors for implementing the following suggestion made by me in my original review:
"It appears that w doesn't matter so much for all non l_0 features. So it would be interesting to set w to 1 for these features and learn only b and for l_0 learn w."
The experimental study done by the authors in the context of this suggestion seem to confirm my hypothesis. One positive implication of this is that fewer parameters need to be learned while not sacrificing the gains (section A.3). Further, the weights seem to be more interpretable and can provide a heuristic for determining the relative importance of different locality features. 

As noted in my original review, there are several positives in this submission and I continue to be appreciative of them. The submission is well written and the engagement of the authors to the reviews has been positive. The main drawback of this work is that it doesn't provide a principled procedure or heuristics for formulating effective locality features for different datasets. The task of formulating the locality features remain outside the scope of the current work and needs domain knowledge. This limits the potential impact of this work as also noted by other reviewers. All that can be concluded from the experimental study is that the carefully designed locality features seem to give minor improvement over language models that don't use such features. 

I'm willing to upgrade my recommendation to marginally above acceptance threshold.


review_title: Response after reading all reviews, responses and revised submission

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work concerns itself about utilizing structural locality inherent in real-world datasets in improving the effectiveness of non-parametric language models. It makes a claim that a) structural locality is not implicitly fully captured by the distance metric used in non-parametric language models and further that b) explicitly plugging in structural locality into non-parametric language models can improve their effectiveness. It validates this claim first by doing analysis of two datasets with the help of custom locality functions and then by plugging in the locality functions into a non-parametric language model with the help of learnable parameters. 

main_review: This work concerns itself about utilizing structural locality inherent in real-world datasets in improving the effectiveness of non-parametric language models. It makes a claim that a) structural locality is not implicitly fully captured by the distance metric used in non-parametric language models and further that b) explicitly plugging in structural locality into non-parametric language models can improve their effectiveness. It validates this claim first by doing analysis of two datasets with the help of custom locality functions and then by plugging in the locality functions into a non-parametric language model with the help of learnable parameters. 


Positives:
	1. Well stated hypothesis and analysis that shows that structural locality is not implicitly fully captured by the distance metric used in K-nearest neighbour non-parametric language model of Khandelwal et al.
	2. Locality features for two datasets - wikipedia and Java projects
	3. Incorporation of locality features in non-parametric language models using simple learnable functions of the distance metric
	4. Analysis that shows that incorporating locality features leads to improved distance distribution among nearest neighbours

Negatives:
	1. Structural locality inherent in datasets need to be captured by a set of custom locality feature functions which requires prior knowledge of the domain of the datasets. 
	2. Marginal improvements in results
	3. No detailed discussion of learned parameters presented in Table 3.
		a. It appears that w doesn't matter so much for all non l_0 features. So it would be interesting to set w to 1 for these features and learn only b and for l_0 learn w. 

In Eqn 4. y should be w_t?


summary_of_the_review: The paper makes an interesting hypothesis and goes about validating the hypothesis. However, the improvements due to the proposed method are marginal.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper is about modelling structural locality in non-parametric language models. The key hypothesis is in modelling not only the co-occurrence characteristics but also structural characteristics such as locality. The paper explains the key claims via case studies conducted on source code data and Wikipedia datasets.

The model paradigm is based on non-parametric language models. A key difference between the non-parametric model and the parametric counterpart is that in the non-parametric model the model parameters are not only determined by the model architecture but also the underlying data.

Structural locality, which is different from just co-occurrence counts, models the structural relationships between pairs of items, e.g., whether they belong to the same or different directory in the case of source code.

The optimisation model is presented in Equation 7 where the authors need a small sample set emanating from the same domain to train the model. The authors then conduct experiments to demonstrate that the method improves upon existing works. Both qualitative and quantitative experimental results are shown.
main_review: The paper models structural information into the non-parametric language models. While the results demonstrate that the method improves upon the existing methods, there are some weaknesses too.

Strengths:
1. The model or the loss function developed by the authors that incorporate structural information is novel. The authors have also clearly explained the model.
2. Results demonstrate that the method improves upon existing methods.

Weaknesses:
1. While there are clear strengths, one weakness is that one may need to define structural properties in different types of datasets that one might use. For instance, it is clear that the model works for source codes and Wikipedia because associated structural information can be mined from the data. It is unclear how does the method generalise across different tasks and datasets, i.e., beyond two datasets. While the authors have addressed these limitations towards the end of the paper, the question is will the work be useful only to a small set of audience, or people from different domains can manually or automatically build such prior knowledge and incorporate it in this model. The key advantages are clear from the paper, this seems to be the weakness that is hard to defend. One possible way to improve the argument so that we could obtain Wikidata-type structure for most datasets is to exploit entity detection and linking including automatically learning their relation (vector) information in a completely unsupervised way. The authors must note that I am simply giving ideas on how to strongly defend this weakness of the model.
2. In terms of experiments, these can be further improved by conducting some downstream application tasks. Can the model be useful for document classification tasks? Currently, it is very difficult to gauge the usefulness of the model through the limited experiments presented in the paper which mainly revolves around perplexity analysis and table 5 in the appendix has additional token prediction results.

summary_of_the_review: Overall, the paper indeed has some merits. The paper can be made stronger by considering some comments mentioned above.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors propose an approach to complement context by adding 'locality' information in examples present in external stores of non-parametric language models. The locality information captures the hierarchical structure, and deems two contexts more similar (or having less distance) if they share common hierarchical structure/attributes. The authors conduct experiments on source code as well as natural language articles. They analyze the results to point out reasons for improvement, and also highlight differences between the domains.
main_review: + The paper is well written and easy to read
+ What the paper is suggesting is simple, yet useful. The example depicted in Figure 1 describes their motivation as well as what they are doing effectively
+The concept of structural locality as used in their paper is defined clearly
+Results are compared with other state of the art models (Table 2)

- The only weakness that I can identify is that the authors have used one dataset of each domain. Since there is minor improvement in the results, it is difficult to gain confidence that the results will indeed improve on other datasets as well





summary_of_the_review: Same as above
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Authors propose a way to use structural locality in language models. Authors demonstrate that the structural locality information improves results for two domains they experiment with. 
main_review: Strengths:
- Authors propose a way to include structural locality into kNN-LM models
- Authors demonstrate improvement on retrieval tasks for Wikipedia and Java source code domains

Weaknesses:
- It seems that the structural locality and its value are known and not new. The paper's contribution then is adding the structural locality to the non-parametric language models. The way to add structural locality is straightforward. It seems to me that this contribution is not significantly novel.
- There is no comparison or discussion of other ways to add structural locality to LMs. It seems authors tried just a single approach that they present in the paper.
- Using locality info in authors' experiments leads to minor improvement. This is not a significant weakness since there might be other tasks where locality will contribute more. However, it would have been interesting to see tasks or domains where improvement is more significant.

Based on discussion with authors and clarifications provided in comments and paper itself, I have increased my score to "marginally above acceptance threshold".
summary_of_the_review: In my opinion, the paper makes a minor contribution by proposing a straightforward way to add structural locality to kNN-LM models. The contribution is possibly useful, but not major. For tested tasks, structural locality information improves results but not significantly. I believe the paper is marginally below acceptance threshold.
review_title: null

Meta-review: Reviewers were in agreement but borderline.  The paper has a nice hypothesis and develops the work using two realistic datasets, Wikipedia and Code.  One reviewer was initially more negative but changed their views based on the authors improvements to the paper.
The idea is fairly simple, but does require modellers come up with the structural features.  There was discussion that more down-stream tasks are needed to highlight the approach.  Moreover, more datasets should be experimented with.  In all, experiments are good but improvement is easily done.
Decision: Accept (Poster)