Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: * This paper proposes a test-time prompt editing technique with reinforcement learning. Compare to prior methods, TEMPERA can efficiently leverage prior knowledge, adaptive to different queries, and provides an interpretable prompt for every query. This method achieves 5.33x on average improvement in sample efficiency when compared to traditional fine-tuning methods.

strength_and_weaknesses: Strength:
* This paper proposes a novel method: It formulates discrete prompt optimization as using RL to edit an initial prompt. It also proposes carefully designed action space, and a set of techniques to improve the final performance.
Weakness:
* It seems that this method mostly improves sample efficiency but not absolute performance (e.g. not in few-shot setting). 

clarity,_quality,_novelty_and_reproducibility: * This paper is clear, novel and seems to provide a lot of implementation details (in appendices). 

summary_of_the_review: This paper proposes to treat prompt tuning as a test-time editing problem with RL, which can give more flexibility and demonstrate promising empirical results. However, it's novel because it's the first to apply RL on prompt editing, but the RL technique itself is not much novel and seems like it mostly improves sample inefficiency but not absolute accuracy (e.g. not in few-shot setting). So, I'd recommend weak accept.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a new model that is able to construct the instance-dependent prompt through reinforcement learning. It allows the agent to perform different editing techniques to update instructions, few-shot exemplars, and verbalizers at test time to construct query-dependent prompts efficiently. Results on different datasets show the effectiveness of the proposed model.
strength_and_weaknesses: **Strengths**

**1. The proposed method is novel and effective.** 

The paper proposes a novel method to edit the instance-relevant prompt in the test time via RL. The editing/action space covers a comprehensive set of commonly-used components like instructions, few-shot exemplars, and verbalizers. Results show the effectiveness of the proposed method in most datasets.

**2. The paper is well-written and the experiments are comprehensive.** 

**Weaknesses/Feedback**

**The experimental section could be improved.** 

Some experimental details are missing. For example, what are the dataset scales for SST-2, Yelp P., MR, CR, and AG News? And what is the value of |Y| in “Task Settings”?

What is the exact number of training examples for the results in Figure 1? What are the details of the standard fine-tuning method?

In Table 4, only three different editing techniques are included. It could be critical to include more editing techniques.

On datasets like SST-2, Yelp P., and MR, the improvements of TEMPERA are not obvious compared to existing baselines.

I have no idea of the details of the compared method in Figure 1. So I am concerned it might not be fair to claim that TEMPERA has greater data efficiency than the standard fine-tuning model. Look forward to clarifications in the rebuttal phase.

clarity,_quality,_novelty_and_reproducibility: The paper looks good to me since the paper is well written, the proposed model is novel, and the comprehensive results show its effectiveness.
summary_of_the_review: The paper is competitive and I am looking forward to seeing my questions answered in the rebuttal stage and in the revised paper.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes RL-based test-time editing for 
strength_and_weaknesses: ## Strengths
1. The paper is overall well-written and easy to follow.
2. The paper includes a wide range of baselines, both continuous and discrete.

## Weaknesses
1. The scalability of the proposed method is not good. As shown in Figure 3, TEMPERA holds an advantage against fine-tuning but cannot scale. Also, I would like to see other few-shot prompting methods illustrated in the same figure.
1. Figure 1 is misleading. It is the best-performing dataset in Figure 3. It's inappropriate to single it out and put it in the introduction.
1. The editing patterns are not diverse enough. The case study is not very convincing. The improvement doesn't seem explainable. 
1. The proposed method is in essence similar to prompt/verbalizer searching. 
1. How do you justify the use of RL here? Is there a strong reason not to use search instead?
1. I think GLUE or SuperGLUE can be more convincing. Yelp, MR, CR and AG News are considered easy.

-----
After rebuttal: The authors have addressed many concerns mentioned in my review during the response period. I've updated my score to 8.
clarity,_quality,_novelty_and_reproducibility: The paper is clear and has some technical novelty. I recommend the authors release the code.
summary_of_the_review: This paper proposes test-time prompt editing with RL but may have some issues.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This work explores editing prompts in large language models on a per-instance level via reinforcement learning. In contrast to existing works, the editing is conditioned on the input query, rather than a task, and the action space for editing is hand-designed. The approach is tested on different natural language few shot classification tasks, showing better performance than few-shot fine-tuning and discrete prompt optimization methods, being more data-efficient for tuning, interpretable, and robust to different hyperparameters in prompting.
strength_and_weaknesses: **Strengths:**
- Relevance:
	- Given the wide use of prompting for tasks in LLMs, and the effect it can have in final performance of downstream tasks, exploring methods to design better prompts is a very important problem.
- Writing:
	- I like how the related work is structured, motivating first the need for prompt learning, and later providing an overview of prompt learning methods and methods based on retrieval examples from the training set.
	- Formulation in section 4 makes the method clear to understand and connect to RL frameworks.
	- Table 1 is useful to understand action space
- Method:
	- The policy used to modify prompts is constrained, which may allow for more interpretability and sampling efficiency (Fig 1), while at the same rich in some parts (swap example, verbalizer change), allowing different kinds of operations ito the prompts.

- Experiments:
	- Thorough and valuable ablation studies, comparison with solid baselines and in different tasks.
	- SOTA or comparable results in a wide set of tasks.
	- Very good properties in few-shot settings.
	- Very cool example of exemplar editing in Table 5, I would love to see this together with the other exemplars. In general, I think it would be really interesting to see if there are common properties in the exemplars edited by the policy. Does the model tend to balance the number of exemplars per class, or does it try to select more exemplars close to the query class?


**Weaknesses:**


- Writing Clarity 
	- From my understanding of the paper, the prompts are edited at test time using a pre-trained policy, but the policy is not updated at test-time but training time. While this is reasonable because at test time we do not have access to the GT label and therefore cannot compute the reward, the test-time optimization generally makes me think about having a surrogate reward at test time used to update the policy. I would strongly encourage to clarify this more explicitly in the paper, maybe by saying, that the policy is trained first by constructing a training set xyz, and at test time it is deployed to edit the prompts. Referencing directly Algorithm 1 would also help a lot.
	- In line with the previous comment, it is hard for me to understand whether K in the training set corresponds to the number of examples for in-context demonstrations, or an external training set, that contains description examples and query. When I think about a few-shot dataset for NLP I am thinking about in-context learning, but here it seems otherwise?
	- Figure 2 would benefit a lot from a walkthrough example. It is hard to understand what exactly is a verbalizer, E1, ... En etc.
	- Section 3.2 could be condensed, a lot of it is already in related work.
	- Clarify, in sec. 4, "... contains both information", what is both information?
	- How is the initial instruction provided? Is it task specific?
	- A figure of the main methods from prompt-search vs the existing method would be helpful to quickly grasp the main differences, or maybe a table of features.
	- Minor: It would be useful in 3.2 to repeat the difference between He et al. 2022, Deng et al. 2022 for section 3.2 since authors talk about query-dependent prompts and then point at limitations of query-agnostic prompts. This seems addressed in the next paragraph, but it is difficult to follow form there that previous methods are these 2 methods. In fact, from my understanding He at al. 2022 is prompt specific, but Deng et al. 2022 is not. Could authors confirm and correct citation in 3.2 in that case?
	- How is Tempera No-TTE different than promptRL. One main thing is the parametrization of the policy, is there something else?
- Method:
	- I would appreciate more details of the policy, there is attention but how is it done? Is it flat over all the techniques and candidates, or is it hierarchical in that first a technique is selected and then a candidate.
	- Table 4 would be good to include No verb, but Inst
- Results
	- Regarding test-time efficiency in Sec. 5.1, while the claim could be true, it is not validated that the run-time will be smaller. A table with clock time or num ops would be good to validate, and if it cannot be provided, make clear that it is a hypothesis.
	- Few-shot is a very valid setting, but it would be really interesting to see if the current method also improves zero-shot performance.
clarity,_quality,_novelty_and_reproducibility: Clarity:
While the description of related work and where the method is positioned, background, and the discussion of the results is clear, I am missing several details and clarification questions in the method section, which are the main reason for my rating. I would strongly suggest editing the text and figures to reflect those.

Novelty:
While using RL to improve prompts has been used in other works, conditioning the prompt search on the query, and the action space for editing are novel and effective.

Reproducibility:
Clarifying aspects of the method would help in reproducibility, as well as providing the code. On the other hand, the appendix contains details about training hyper-parameters and datasets, which help in reproducibility. Correcting the clarity questions will improve reproducibility.
summary_of_the_review: This paper discussed a relevant topic, has strong ideas, a new method, thorough experiments, and SOTA results. I generally think it would be a valuable contribution to the community, but in the current state there are important details that are difficult to understand for me, and I assume for other readers in the community. If the aspects described above are clarified, I am very open to changing my rating to accept.

[Edit] The authors have addressed my concerns in the revised version. I therefore have updated my score.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes a method for learning (using reinforcement learning) to construct instance-specific discrete prompts in real-time for better task performance. Using their method, an agent learns to perform different editing techniques to update instructions, few-shot samples, and task verbalizers to construct query-dependent prompts.

**Strengths:**
Reviewers noted that the method is novel, important, and empirically effective, demonstrating improvements on a diverse set of classification benchmarks.

**Weaknesses:**
Reviewer complaints centered on two points: writing quality, which the authors have put in effort to fix as part of the rebuttal period, and the conclusions about data efficiency of TEMPERA, which the authors seem to have answered to the reviewer’s satisfaction.

Given the universal recognition of the strengths of the paper and the mitigation of outlined weaknesses during the rebuttal period, I think this paper should be accepted.

justification_for_why_not_higher_score: Reviewers felt the experimental setup could have been stronger
justification_for_why_not_lower_score: Strong, novel solution for an important and interesting topic.
Decision: Accept: notable-top-25%