Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper addresses the problem in prompt learning that prompts are always included in the input text, which can cause computational and memory overhead.
To this end, this paper proposes prompt injection, a method that injects the prompt into the parameters of a pre-trained language model (PLM).
They show that the prompt injection method can be 280 times more efficient compared with previous approaches.
strengths_and_weaknesses: Strength:
1. This paper proposes an interesting question that many conditional text generation tasks usually include an instance-specific prompt for each data, for example the persona profile for persona dialogue generation.
However, such prompt can cause computational costs and make it difficult for models to capture the key information during inference, in particular when the prompt is very long.
Therefore, this paper proposes a new method call Prompt Inject (PI) to store the prompts in the PLM instead of concating it to the input.
To my knowledge, the idea is very novel and interesting.

2. The experiments are solid.
This paper conducts experiments on persona dialogue generation, Text-2-SQL and zero-shot task generalization.
The experiments show that the PI can speed up the inference time and improve the lower bound of the models.

3. This paper is well written with proper citation and clarification

Weakness:
A minor note would be that the prompt in this paper is different from the prompt in most sense in other NLP paper, such prefix tuning, prompt tuning, etc.
So I  would suggest that the authors could give a more clear explanation on this issue in the introduction.


questions: NA
limitations: NA
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper is about prompt injection, which aims to be an efficient alternative to attaching fixed prompts to the input. Prompt injection is sample efficient and worth it when there are long prompts. This can be useful if there is a detailed description of a personalized chatbot, for example.

**Update after response.**

With the additional experiment, I've increased my score to 5.
strengths_and_weaknesses: Originality: Paper tackles the problem of using fixed prompts many times for different examples.
Quality: Experiments done in the paper are OK but not particularly stunning. One way to improve the experiments could have been to pick a setting where it was crucial for long contexts (like summarization, or long dialogues), and show that with this method it was possible to use longer contexts which lead to substantially better performance. Or show that it works over a broad range of settings.
Clarity: The paper is relatively clear.
Significance: The paper proposes a method for prepending fixed prompts in a computationally efficient manner. 

While the paper is a nice start, I think it still leaves a lot to be desired. There is enough substance to warrant an ACL short paper, but I am not sure about a NeurIPS paper. Some examples of how I think the paper could be stronger:
- Empirical results showing that PING works across a very wide range of tasks with good performance. 
- Comparison with other methods that try to do similar things.
- Show for tasks that require long inputs and long prompts, PING enables substantial performance improvements. 
- Overall, experiments on settings where fast inference is crucial might make it more clear to the reader why it is important to trade off this complexity from using PING for faster inference. 

Minor note: Line 22 should be Sanh et al, not Raffel et al., right?
Some further related work might have been missed, such as context distillation (https://arxiv.org/pdf/2112.00861.pdf)
questions: How does the proposed method compare with other baselines such as context distillation? (Askell et al, https://arxiv.org/pdf/2112.00861.pdf)
Are there other approaches where this approach might be especially useful? As the main benefit is inference-time efficiency, one might imagine that the WSC, RTE, COPA, and SPIDER tasks are not the prime candidates for this approach. The dialogue eval is good but it only uses one dataset. 
limitations: Seems fine to me
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a (self-proclaimed) novel approach called prompt injection, which injects the declarative knowledge of a prompt into the language model parameters by distilling a teacher model with the prompt into a student teacher without prompt on automatically generated inputs. The papers show that the injection is successful across a wide & diverse range of tasks and the approach saves inference time compute. 
strengths_and_weaknesses: Strength:
- This paper elaborates on the impactful idea of "prompt-injection", and show that this is broadly useful for a wide range of tasks. 
- This paper also sets up a reasonable evaluation framework that could be used to test future methods.
- This paper is generally well-written. 

Weakness:
- The idea of prompt-injection is not novel, and various previous works have explored this idea. For example, [1] and "context distillation" in [2]. Please tune down the claim on novelty in the presentation. That said, I would still vote for acceptance of this paper since the other paper did not focus on the framing of prompt-injection, and this paper could be useful for the community to cite. 
- I wished that there were more experiments and ablations, given that the contribution of this paper is more about "studying an existing algorithm (PING) and figure out its basic property", rather than "proposing a drastically novel algorithm". 

[1] Towards Zero-Label Language Learning
[2] A General Language Assistant as a Laboratory for Alignment
questions: - How many pseudo inputs did the paper distill on? I would imagine that the gap between the student and teacher should drastically decrease when you increase the number of pseudo inputs. 
- How does the gap between student & teacher change w.r.t. model size? Should we expect future larger models to become better at this? 
- Can you show some qualitative examples where you claimed that "the low quality of the pseudo generated inputs lead to lower performance"? How about the diversity of the pseudo inputs? 
limitations: Yes.
review_title: null

Meta-review: metareview: The paper proposes a method for distilling prompts into the parameters of a model. Reviewers liked that the method can improve the efficiency of inference by avoiding having to attend over prompts, and the evaluation on the PersonaChat dataset is a good use case for this approach. However, several important concerns were raised. As pointed out by reviewer y47p, similar ideas have been explored in previous work, and claims of novelty need to be toned down. As acknowledged in the author response, most of the experiments are on tasks with short inputs, so gain little benefit from the approach. The difficulty in finding suitable tasks where the approach has a clear benefit might suggest the method has limited applicability. The additional experiment on MSC is a nice addition in the author response, although results here are a bit underwhelming. Overall, this is borderline, leaning reject.


confidence: Less certain
Decision: Reject