Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper introduces RetroPrompt, which constructs an open-book knowledge store from training instances and implements a retrieval mechanism during the process of input, training, and inference, thus equipping the model with the ability to retrieve related contexts from the training corpus as cues for enhancement.

strengths_and_weaknesses: Strength:

1. The experiments are extensive and convincing.
2. The idea of using kNN as a grounding demonstration is technically sound and plausible.
3. The choices of datasets and tasks are diverse.

Weakness

1. The idea of using nearest neighbors to train (kNN LM), to inference (FiD), and the general idea of using retrieval modules on memories (Google’s REALM, and Meta’s RAG), have been proposed and well studied. It’s unclear to me what is the novelty of this work. It seems like a combination of different models in different stages. And none of the models mentioned above are included as baselines.

2. What’s the performance of your method on open-ended knowledge-intensive tasks. As the tasks chosen in this work are mainly short-text tasks that can be answered with closed-from short texts (mainly overlapped with LM-BFF). Other tasks such as MSMARCO and Wizard of Wkipedia normally require more complicated reasoning rather than simple keyword memorization.

3. It is important to include some information retrieval systems as baselines, since if you integrate the retrieval system into every stage of LM, what’s the benefit of using an LM for unreliable prediction? I would like to see some analysis in the comparison with some IR-based methods, such as DPR, BM25, etc (+ necessary post-processing modules), if applicable.

4. The writing can be improved, especially some informal expressions.

5. No discussion of limitations
questions: See the comments in weakness. Especially points 2 and 3.

limitations: Discussion of limitations is lacking.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: They propose a method for better zero-shot and few-shot prompt learning called RETROPROMPT. To decoupling knowledge from memorization, they introduce knowledge-store which contains key-value as a [masked] label representation and its corresponding label word for the target data set. Using kNN on this neural knowledge-store, guided training and open-book inference can be implemented. By achieving best performances on 9 NLU tasks, the proposed method is shown to be effective.
strengths_and_weaknesses: [Strengths]
- Well written.
- The paper propose technically sound and smart idea for balancing between generalization and memorization.
- Various experiements prove the effectiveness of the proposed RETROPROMPT for zero-/few-shot learning.

[Weaknesses]
- Experiments on full fine-tuning for all 9 datasets would be better for understanding the proposed work (only 3 results out of 9 datasets are included in the paper).
- The proposed model may suffer from enormous memory usage, especially for the full fine-tuning.
questions: For the full fine-tuning, does the proposed model suffer from tremendous memory usage?
limitations: - Knowledge-store might be managed for full fine-tuning on large training dataset
- As metioned in the conclusion, this paper only deals with NLU tasks with encoder models.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes an idea to retrieve from few-shot examples to prevent the prompt tuning process to bias toward learning from atypical examples. The combination of three techniques - knn-guided training, neural demonstration, and the knn-guided test-time prediction leads to a strong empirical performance across a broad set of tasks. Analysis shows that the proposed method effectively mitigates memorization. 
strengths_and_weaknesses: ### Strengths
- The proposed method shows very strong empirical results compared to strong baselines
- It shows that even in the few-shot setting, retrieval-based approaches would be helpful for stabilizing and improving results without introducing additional parameters, and it could lead to useful application in a broader context.

### Weaknesses
- Details are missing, especially about the baseline and it weakens the compelling results. 1) KPT is a method to expand the verbalizers, what verbalizers did you get? I don't understand why the results would be worse than LM-BFF as it seems to be LM-BFF + additional verbalizers? 2) For LM-BFF, how did you get the scores with demonstrations in a zero-shot setting? 
- Some design choices are not well justified or ablated: 1) Why specifically use FocalLoss? How does it compare to a linear combination of the original cross-entropy loss and a KNN loss similar to KNN-LM and it would be closer to the retrieval setup during test time? 2) Why does neural demonstration happen at the embedding layer? 

questions: - For neural demonstration, I assume that the concatenation happens on the dimension of tokens? I suggest stating it more clearly in the paper.
- For section 4.4, did you just use 16 examples from the source dataset and not fine-tune on the target dataset at all?

limitations: Yes, the authors discuss about efficiency overhead of the proposed method in appendix.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes a new semi-parametric approach for few-shot learning, where the prediction is made based on the combination of softmax probability from the language model (LM) and probability from retrieval over a knowledge store (in this paper, training data, either unlabeled or labeled). The paper proposes a few model components to make this approach work – the LM trained to be a good retriever (along with asynchronous update of the index), neural demonstrations (using nearest neighbors of the query from the knowledge store), and a modified objective (called guiding training in the paper). Experiments are done in a few-shot setup (where a few labeled examples leads to a knowledge store) and in a zero-shot setup (where an unlabeled training data is a knowledge store, and there is no parameter updates), and show strong results on 9 different NLP datasets (classification and information extraction) over competitive zero- and few-shot baselines.

strengths_and_weaknesses: ### Strengths

* The idea of applying retrieval over a knowledge store for language model prompting is of interest in the community and is a timely topic.
* This paper is one of the first that applies the kNN-LM approach for downstream tasks, as far as I know.
* Empirical results are strong, outperforming a range of competitive baselines.
* Experiments also include extensive ablations, showing the impact of each model component.


### Weaknesses

* The idea is strongly based on the kNN-LM model (Khandelwal et al. 2021). In fact, the overall idea of combining softmax probability and probability from retrieval is entirely taken from kNN-LM, and how the model works at inference time is identical with kNN-LM, if I understand correctly. Although this paper is cited, it is cited only briefly (only cited twice, in Section 3.4 and Section 5). It looks to me that the paper should extensively discuss kNN-LM and describe the differences. (To clarify, I do believe there are substantial differences. They are just not discussed in the paper.)
* The model seems named after RETRO (Borgeaud et al. 2021), but the paper does not cite nor discuss the RETRO paper. In fact, I do not think the connection between this paper and RETRO is very tight – they’re only loosely related in the sense that both do retrieval, but what they retrieve, where they retrieve from, how they incorporate retrieval to LM, and what problems they are solving are all different. It is indeed not very intuitive to me why the model is named after RETRO, given that it is significantly more related to kNN-LM.
* It is not entirely clear to me what exactly are giving improvements, e.g., how the model gains from retrieval when the data store is extremely small (e.g., with 4 examples), how the zero-shot model achieves significant gains even with no training (which I believe is the main contribution of the paper), how the model still outperforms most competitive baselines (in Table 1) even without kNN-test that does not actually incorporate retrieval at test time, etc. I discussed them in more detail in the “Questions/comments” section.

(I am giving a borderline score due to the weaknesses mentioned here, but am happy to increase the score if they are resolved during the author rebuttal period.)

Borgeaud et al. 2021: https://arxiv.org/abs/2112.04426
questions: * kNN-based models are known to be effective when the size of the datastore is large. In this context, it is not intuitive to me how the model benefits from the kNN approach in a few-shot setup, where the datastore only consists of 4 or 16 examples.
* Is my understanding correct that, in a zero-shot setup, since there is no training phrase, the only difference from kNN-LM in terms of the model is the use of neural demonstration? Based on Table 1, the proposed model has significantly more improvements in a zero-shot rather than a few-shot? Are all gains here coming from neural demonstrations?
* Questions about Table 4:
    * w/o kNN-test: in this case, since the data store is not actually used, are gains mostly coming from better LM?
    * w/o kNN-train: in this case, is it identical to kNN-LM except there are neural demonstrations? (I know the knowledge store still looks different, but am curious in terms of modeling.) And is my understanding correct that this is identical to the main model in a zero-shot setup?
* Equation (6): It is not entirely clear to me how this objective achieves the problem mentioned in the section, and if this objective is mathematically sound. (e.g. why does it have to be the form of the multiplication between two log probability values?)
* Zero-shot setting: It seems debatable whether the use of training data (even if labels are not used) can be called zero-shot, although I understand authors followed the setup from previous work. Maybe it’s good to add a footnote noting that this is not zero-shot in a stricter sense?
limitations: The paper does not include limitations of work. I suggest authors to indicate limitations of the proposed model (e.g., when the proposed model may not work) as well as broader risk in applying large LMs in downstream tasks.
review_title: null

Meta-review: metareview: The paper proposes RetroPrompt, which builds a knowledge-store with training examples and improves few-shot and zero-shot performance.

All reviewers appreciate the improvements over competitive baselines and the quality of presentation. The main weaknesses are the lack of ablations to clarify where the gains come from and unclear positioning with respect to previous works (KNN-LM, RETRO, REALM, RAG). The reviewers were unanimous in accepting and the authors addressed some of the issues raised.
confidence: Certain
Decision: Accept