TEMPERA: Test-Time Prompt Editing via Reinforcement Learning
Authors: Tianjun Zhang
Xuezhi Wang
Denny Zhou
Dale Schuurmans
Joseph E. Gonzalez
Venue: ICLR 2023 notable top 25% (2023)

Abstract: Careful prompt design is critical to the use of large language models in zero-shot or few-shot learning.  As a consequence, there is a growing interest in automated methods to design optimal prompts. In this work, we propose Test-time Prompt Editing using Reinforcement learning (TEMPERA). In contrast to prior prompt generation methods, TEMPERA can efficiently leverage prior knowledge, is adaptive to different queries and provides an interpretable prompt for every query. To achieve this, we design a novel action space that allows flexible editing of the initial prompts covering a wide set of commonly-used components like instructions, few-shot exemplars, and verbalizers. The proposed method achieves significant gains compared with recent SoTA approaches like prompt tuning, AutoPrompt, and RLPrompt, across a variety of tasks including sentiment analysis, topic classification, natural language inference, and reading comprehension. Our method achieves 5.33x on average improvement in sample efficiency when compared to the traditional fine-tuning methods.

Published as a conference paper at ICLR 2023
TEMPERA: TEST-TIME PROMPT EDITING VIA REIN- FORCEMENT LEARNING
Tianjun Zhang' Xuezhi Wang* Denny Zhou² Dale Schuurmans
Joseph E. Gonzalez'
UC Berkeley Google Research, Brain Team University of Alberta t cianjunz&berkelle .ddu edu
ABSTRACT
Careful prompt design is critical to the use of large language models in zero- shot or few-shot learning. As a consequence, there a growing interest in auto- mated methods to design optimal prompts. In this work, we propose TEst-tiMe Prompt Editing using Reinforcement leArning (TEMPERA). In contrast to prior prompt generation methods, TEMPERA can efficiently leverage prior knowledge, is adaptive to different queries, and provides an interpretable prompt for every query. To achieve this, we design novel action space that allows flexible editing compo-
the initial prompts covering a comprehensive set of commonly- nents like instructions, few-shot exemplars, and verbalizers. The proposed method achieves significant gains compared with recent SoTA approaches like prompt tun- ing, AutoPrompt, and RLPrompt, across variety of tasks, including sentiment analysis, topic classification, natural language inference, and reading comprehen- sion. Our method achieves 5.33x on average improvement in sample efficiency when compared to the traditional fine-tuning methods. Our code is available at https://githute com /iar jwn./TEMPEER TeeM/MMPPRR
INTRODUCTION
With the recent advances in pre-training large language models (Brown et al., 2020; Fedus et al.)
2021 Raffel al.) 2020 Chowddery al. [2022). prompting, or in-context learning provides data- efficient framework for performing NLU (Li & Liang 2021 Shin et al. 2020b Gao et al. 2020b).
Such methods achieve impressive zero-shot and few-show performance in many downstream tasks.
However, the prompt often has to be carefully tuned to achieve consistent performance for each task (Lu et al. 2021). For example, prompt tuning aims to optimize a continuous prefix embed- ding via gradient descent and directly takes generated output from the frozen pre-trained language model (Lester et al. 2021| Liu al., 20)1b]a). On the contrary, discrete
optimizztioiiioooon focuses on constructing meaningful instructions, in-context exemplars and verbalizers (Brown et al.) 2020] Gao et al. 2020b). Prior work often performs black-box optimization or applies RL-based methods for direct generation (Deng et al.. 2022, [Sun et al. 2022 Prasad et al., 2022). Recent works in the prompt tuning field have shown that, performing instance-dependent prompt tuning (Wu et al., 2022 Jiang al. 2022 can improve the performance of some downstream tasks. The correspond- ing concept in the discrete prompt optimization domain is intriguing since it allows users to provide different instructions for different inputs and task. Unlike prompt tuning, such instructions can be more human interpretable. However, finding such query-dependent prompts is often overlooked and not feasible given the inefficiency of black-box optimization.
In this paper, we investigate the importance of providing query-dependent discrete prompts and demonstrate how this can be achieved via efficient search. To this end, we propose the concept of test-time editing through reinforcement learning (RL) that allows the agent to perform different editing techniques at test time to construct query-dependent prompts efficiently.
We formulate discrete prompt optimization prompt, which only requires high-level guidance on which part to edit and what tools to use. Dif- ferent from prior work, this formulation strikes good balance between human prior knowledge, flexibility, feasibility and interpretability. The method allows easy incorporation of human knowl- edge since can provide manually chosen initial prompt and allow RL perform editing
RL problem by sequentially editing an initial

---

Published as a conference paper at ICLR 2023
it. It also achieves balance between search flexibility and feasibility because by enabling different editing techniques, the prompt can be transformed to very different forms but the search space is more feasible compared to direct generation. The final prompt is also more interpretable since the editing tools we adopted usually do not change the semantic meaning of the sentence.
To summarize, propose construct query- dependent prompts through test-time editing and formulate this as an RL problem. We carefully
Efficiency TEMPERA
Cassification Performance
0.85
design the action space, enabling the agent to flexibly edit the instructions, in-context exem-
0.80
plars and verbalizers. To better train the RL
agent, we propose using the score difference be- tween consecutive prompts before and after edit- ing as rewards and developing a set of techniques that help improve the final performance (e.g., re- ward normalization). We also adopt an attention- based policy architecture attend over possible candidates or design choices, and show this can be effective for RL training.
0.70
0.65
Finetun ng
TEMPERA
100
200
300
400
50D
Number of Training Examples
Data Efficiency for TEMPERA: We comopare the data efficiency of TEMPERA and standard fine-tuning a few-shot setting- Results are averaged across four tasks: SST2, AG News, RTE and MR. shows that our method achieves comparable performance using 4x fewer examples.
Figure 1:
Following the standard few-shot text classifi- benchmark our algorithm extensively on multiple tasks (including those from GLUE (Wang al. (018)) and Super- GLUE (Wagg et al. 2019)). We show that TEM- PERA can achieve SOTA performance (e.g.. 1.8% better in SST-2 and 3.9% better in CR) compared to few-shot finetuming, prompt tuning and discrete prompt optimization. We also show that TEMPERA is on 4x more data efficient (over the average of 4 tasks SST2, MR, AG News and RTH) compared with traditional finetuning methods (Figure I. In addition, we perform extensive ablations on different aspects of the proposed algorithm. We demon- strate that TEMPERA is robust to the prompt pool size and the number of few-shot exemplars.
cation setting,
RELATED WORK
Prompting in language models and sensitivity to prompts. Recent research has shown that as language models scale up, new capabilities could be unlocked such as in-context learning (Brown) ct al, 2020), where the language model prompted with few in-context demonstrations and learns to perform certain task in samplc-efficient way. However, several works have studied the in- context learning ability more closely and found that the task performance can be highly sensitive to how the in-context prompt is written. For example, Lu et al. (2022) found that the prompt order can have a large effect on the final task performance; Zhao et al. (2021) show that the choice of prompt format, training examples, and prompt order cause the performance vary quite significantly.
Automatic prompt generation and search. To address such sensitivity in language models, mul- tiple approaches have been proposed for better prompt generation. In the continuous space, Lester et al. (2021) propose prompt-tuning add tunable tokens for each task during the fine-tuning stage to improve task performance. [Zhong et al. (2021) propose OptiPrompt that optimizes the prompts in the input embedding space directly for factual probing. More recently. Wu et al. (2022) found performing iestance-indeeender prompt-tuning can further boost the performance. In the discrete space, Gao et al. (2020a) propose prompt- based fine-tuning and utilize pre-trained models to au- tomatically generate prompt templates. SShiick& & Schütze (202T) and [Schick et al. (2020) use a small amount of training data to automatically identify the best label words to use for few-shot clas- sification. [Shin et al.] (2020a) propose AutoPrompt to perform gradient- -uiddd search to find the best tokens in the prompt, although the best prompts found are usually not interpretable by humans. Jiang et al. (2020) propose mining-based and paraphrasing-baset methods to generate meaningful and diverse prompts for factual knowledge probing. Related to our work, Deng et al. (2022) propose an RL-based framework to directly generate better prompts via black-box optimization. Different from existing work, our approach frames the problem as test-time prompt editing with an RL-based framework perform efficient search in the editing space.
2

---

Published as a conference paper at ICLR 2023
(Initial Prompt)
(Final Prompt)
(Edit Agent)
Example
Instruction
Instruction"
[Chesely true laiee given leal.
(Gven tecl, cheselly true or Ioiee
Example k+1
Example
Example 3
E,
E
Verbelizer
...
...
Example
Example
Allention
Verbalizer
Verballizer
Query x
Verballizer
joostive, negative)
(graal, sumiibae)
(Additional Info)
The RI. agent is trained to optimize the performance of downstream task. test-time, given a query, the agent adopts an attention-based policy edit the instructions, in-context exemplars and verbalizers for T rounds.
Figure 2: Test-Time Editing via RL:
Efficient training exemplar retrieval prompts. In addition. existing work has shown the choice of the exemplars can also be critical to the final performance. For example, [Liu et al. (2022) propose to retrieve exemplars from training pool that are semantically similar test example, and show it can significantly boost the performance. Rubin et al.) (2022) trained dense retriever efficiently retrieve good training examples as prompts during time. In this work, we show that an attention-basee exemplar selection process over the embedding space can effectively choose performant training examples within RL framework.
TEST-TIME PROMPT EDITING
We formulate the task of test-time editing in this section. We give some background on the few- shot text classification and how to use prompts for downstream NLP tasks. Then we formalize a new setting called test-fime editing where users are allowed to perform editing a given prompt, depending on the given input and task during test time.
3.1 BACKGROUND
Few-Shot Text Classification. Following the standard few-shot language model classification set- ting (Brown et al. 2020), we assume that we are given pretrained language model L. and wish to perform classification on dataset D with label space V. Assume we are given samples per class from the training set, the new few-shot training set is given as Durain = {z,, } Xx In addition, there hold-out test dataset Drest that we use for evaluation on downstream NLP tasks.
Optimizing Discrete Prompts. Prompt-based few-shot learning considers the following prob- lem: given a piece of text p as a prompt, we use the generative distribution of the language model Pe(vlp. x) to perform various NLP tasks without fine-tuning the model. In particular, for given objective R, we propose to perform the desired optimization over the prompt by finding an optimal p' arg migney R( R(Pr(y/P (x).. In this paper, focus restricting the prompt of text instead of letting p to be any vector in the latent space. This not only provides more inter- pretability of the prompt, but also allows to existing natural language tools (e.g., NLTK (Bird) et al. 2009)) to perform a discrete search for constructing better prompts.
piece
Different Forms of Discrete Prompts. We consider three popular forms of discrete prompts: (1) Instructions, which provide a segment of text describing how the task is performed, usually put at the beginning. (2) In- In-Context Demonstrations {eo, 6 e which selects several examples and their corresponding labels, usually placed before the query. (3) Verbalization, which aims to design how the task asked and which keywords to select as labels. See Figure for an example of different transformations that we perform when editing in our RL-based framework.
3.2 TEST-TIME EDITING

---

Published as a conference paper at ICLR 2023
Algorithm Test-Time Prompt Editing with TEMPERA
Input: Language Model C, Initial Prompt po. Training set Derain Evaluation Deval, Iteration
N, Fix rounds T
2: Initialize (- 8) to be uniform:
3: for episode
1, ..... .N do Derains Set po
4: Random sample batch
5: for step ( - 1..... T do
6:
Get 6, = L(B. pt)
7:
Run editing policy a, (s)),, Get new prompt Pr+1
8:
Get new state sc+1 = CBB, p(+1)
9:
Add transition (8t, U,, St+1) to replay buffer
10:
end for
Update policy parameter 0 of me with the PPO loss
12:
end for
Evaluate policy Th on evaluation dataset D Deval
Prior works have often attempted to identify a query-agnostic prompt (Deng et al. 2022 Sun et al., 2022) attempted to directly generate query-dependent prompt via hyper-networks learning (He) et al. 2022). However, query-agnostic prompting fails to incorporate any query-related information into the prompts and directly generating prompts for each individual query is challenging (due its difficulty to incorporate human prior knowledge or feedback). In addition, by permuting the order of in-context exemplars {eu, 61, .... eg [Lu al., 2022) or searching for the nearest neighbors of the current test instance (Liu et al. 2022) as in-context exemplars yields better performance. These reveal the importance constructing query-dependent prompts.
Unlike prior methods, we perform prompt editing at test-time. The procedure works as follows: at test time, one is given an initial prompt Po. We want to learn function f that takes the initial prompt Po. query x and pool of examples/verbalizers p', and outputs final prompt: Pr [Po. p'). The overall framework of our algorithm is shown Fig. We allow f to make edits (e.g., editing verbalizers and/or swapping examples) over the original prompt make it more suitable for the downstream task and query x. Since the editing function f can depend on the query x, we call it the test-time editing function. Note that train the function in fixed training dataset and directly deploy it at test time without any addition training. This different from the test-time optimization since we don't 't have access to the ground truth label or a surrogate objective. Plese see Algorithm for details.
TEST-TIME EDITING VIA REINFORCEMENT LEARNING
In order to learn the test-time editing function f. we present a novel RL-based framework that naturally maps the editing process to MDP. We will present our framework and discuss how we design the state space, action space and reward in this section.
Reinforcement Learning Formulation. We formulate test-time editing as Markov Decision Process (MDP). Given an initial state, s = (Po, x), consisting of an initial prompt and query, at each time step t, the RL agent selects one of the editing methods from the action space A. We can then define the transition function T S x A + S to be the state of prompt before and after editing (Pr, x) X a + (Pt-1. x). That is, the transition dynamics are deterministic given the editing action. We can either define fixed horizon H or design termination function to stop editing and get the final prompt. The goal is maximize the expected reward R E >/rx where is the reward and 7 is the discount factor. We introduce in detail each component of the state representation, action space and rewards in the following subsections.
State Representation. The RL framework is general and flexible about the representation of states. The only requirement is that such representation contains text information. Instead of di- rectly using the raw text representation, we use the last hidden states of the pretrained language model st L(p,, x) as the state representation and feed it into the policy network.
4

---

Published as a conference paper at ICLR 2023
Table 1: Effect of different editing techniques. For instruction, tokenize it into phrases and perform swapping. addition or deletion. We also allow swapping in-context exemplars or changing different verbalizers.
Before Editing
After Editing
Swap
"Given text, classify whether good or bud."
"Classify whether it is good or bud, given text."
Instruction
Add Delete
"Given text, classify whether good or bad."
"Given text, given text, Classify whether it is good or had." "Classify whether good or had."
"Given text, classify whether good or bad."
Permute
(Example Example 2.... Exmmmll k }
(Example R. Example .., Example } (Example & + I, Example n,... Example 3
Example
Swap
[Example Example 2, Example i
Verhulizer
Change
"positive", "negative")
("greal", "temiile"))
Action Space Design. We include most of the editing actions in our action space. At each stage, the RL. agent can choose the editing objects from instruction, in-context exemplars or verbalizer. For editing the instruction, we provide the initial instruction from natural instructions (Wang et al.) 2022). Then we tokenize the instruction into phrase level using NLTK (Bird et al. 2009) and perform swapping, deletion or addition of different phrases. Suppose we have phrases, the action space size will become (I x (2 1 + 21.
For the in-context exemplars, we keep example pool of N, initialize our prompt by randomly choose n of them as the initial prompt. We then allow the agent to directly perform swapping one example from the current prompt with either another one from the current prompt or from the pool of examples that are not currently used. This results in an action space for the RL. agent of N (n (n 1) since we do not allow swapping with the same example.
For the verbalizer, we allow the RL agent to freely choose which verbalizer to use for each in- context example from PromptSource (Bach al. 2022). We also will enable the agent freely choose which verbalizer to use for each query x. Interestingly we found that this helps boost the performance of our algorithm. We provide some examples of the editing process in Tab. [
Reward Design. We adopt the step reward proposed in RLPrompt (Deng et al. 2022). For each query x, we get the log probability of the output label from the language model log Pa (i) x, Pe) given the proposed prompt Pt with the correct label c. and we define the score difference s(c) as:
scc, x, Pt) A log (a x. Pc) X2 arg max log (o((e//x.p)) c'éc
where we have introduceed the two balancing hyperparameters A > and A2 > for the positive and negative terms respectively. Intuitively, this score gives negative reward when the prediction not correct and a positive reward otherwise. The goal is to optimize the score for the final prompt.
However, RL aims to optimize the accumulated reward during the MDP process while prompt design only cares about the performance of the final prompt. Thus, we propose to use the score difference between successive edits as the immediate reward:
T1 s(c, x, P.) s(c, (cc, 1t
Ignoring the discounting factor y, this makes the accumulated reward from time 0 to T correspond to the score difference between the final and the initial prompt s(c, x, PT) s(c, x, Po). Now the objective of RL is to maximize the score difference.
Attention- Policy Architecture. We adopt an attention-based policy architecture for the reinforcement learning agent. We put attention over graph of possible candidates and let the agent choose which editing technique to perform. We find that the attention- basedd architecture helps the agent to emphasize the important examples (c.g., examples that more semantically similar to the test instance).
We use the PPO (Schulman et al. 2017 7) algorithm in our experiments. The detailed hyperparameter used can be found in Appendix. A We list here a couple of very important techniques we used in experiments. We found these techniques are crucial to the success of our RL-based framework.
Observation Normalization: Since we take the last hidden states of the language model as ob- servation, it might have very small variances between different samples. We keep a running mean
5

---

Published as a conference paper at ICLR 2023
and standard deviation for the observation and normalize it before feeding it to the policy and value network. This is commonly used in RL and we found this boosts the performance of our method.
Reward Normalization: For different training samples, performing editing over prompts may result in significantly different reward scales. For some of the samples, different prompts might have very marginal effects the final prediction, either due to the fact that the model is already confident about the prediction since it is too easy, or the task sample is too hard to predict and the model is confused regardless of what prompt is fed. On the other hand, for other training samples, editing prompts might bring huge difference in terms of the accuracy. Thus, we perform sample-wise reward normalization to ensure that the reward scale between samples is relatively consistent.
Conditioning Policy on Action History: Directly taking the observation from the language model can be inefficient since the policy has no clue about how it has reached the current state. This will bring a loop that the policy will edit prompts PA + Ps and then ps -> PA- To mitigate this effect, we build a policy that not only takes in the current hidden state, but also conditioned on the action history on how it gets to the current state. Thus, we break the loop between two prompts by considering how each state is reached.
5 EXPERIMENTS
Our experiments first reveal the effectiveness of TEMPERA in the few-shot setting. We compare
TEMPERA with prior baselines like Finetuning (Devlin et al. 2019) 9). Soft Prompt Tuning (Lester) et al. 2021), Black-Box Tuning (Sun et al. 2022). RLPrompt (Deng et al. 2022) and other manually tuned prompt methods. On various tasks from GLUE (Wang et al., 2018) and SuperGLUE (Wang et al. 2019), our method achieves impressive performance comparing to prior baselines. This shows that only using small amount of training examples is sufficient for RL and TEMPERA is sample efficient. We also illustrate the data efficiency of our method compared to finetuning, showing that TEMPERA can achieve same performance with 5.33x less data.
In addition to the performance gains, we aim to understand our method from different aspects. In Sec. 5.2 we study how much test-time editing helps compared to query-agnostic prompts. Our experiments demonstrate the importance of test-time editing and the necessity of query-depended prompts. In Sec. [5.] we show that how different editing techniques (e.g, instruction, in-context demonstration and verbalization) affect the final performance of the downstream task. We also ablate the number of in-context demonstrations used and the size of the example pool in Sec. 5.6 and Sec. 5.7 Finally, we show some example prompts after editing to illustrate the editing policy.
Tasks. We conduct our experiments from different categories including single- sentence tasks (e.g., sentiment analysis including SST-2, Yelp reviews, MR, CR, topic classification including AG News). For one-sentence tasks, the goal is to make a prediction based on the sentence. We also include tasks from different types like NLI (e.g.. SST-2) and multiple choices (e.g., AG News). Most of the tasks are from the standard GLUE (Wang et al. 2018).
Task Settings. To ensure a fair comparison, we follow the same setting from LM-BFF (Gao et al., 20206) and RLPrompt (Deng al. 2022), we test TEMPERA few-shot text classification tasks. The setting is devised as follows: We randomly sample 16 training samples per class from the training dataset of each task and use them as the few-shot dataset. This will result in a total of 16 x | training samples (please refer to Appendix. E for the number of classes in each task). We also randomly sample 16 samples per class as the validation dataset. For reporting the final performance, we use the standard test set and the detailed information can be found at Appendix E In addition to the common setup, we also randomly select n examples from the training dataset as the in-context exemplar pool. We average our runs for 4 random seeds and report the average performance and corresponding standard deviation. For the language model, we use L = RoBERTa- large (Liu et al. 2019). For the details of these settings and tasks, please refer to Appendix. E The initial instruction is taken from the Natural Instructions (Mishra et al. (021). The initial in context demonstrations are randomly sampled from fixed example pool of size 16 and the example pool is also randomly sampled from the training dataset, different from the few-shot dataset that used for training the RL policy.

---

Published as a conference paper at ICLR 2023
We compare TEMPERA with several SoTA prompt tuning and discrete prompt opti- mization baselines (including finetuning).
Baselines.
Finetuning:
it finetunes the entire language model with a classification head using the few-shot
dataset.
Manual Prompt: we take the handcrafted prompt from (Bach et al. 2022).
Black-Box Tuning: it is a mixture of discrete and soft prompt. The soft part is trained using gradient descent and the discrete part is optimized using gradient-free tuner.
AutoPrompt: adds the discrete trigger token and updates the prompts by iterative gradient search.
In-Context Demonstration: it randomly selects one training example and concatenates them with the input query.
Instructions: Following Natural Instructions (Wang et al. 2022) prompts are manually created instruction for each task. Each prompt is concatenated with inputs. Details are in Appendix, D
GrIPS: it performs phrase level editing the instructions and selects the best
RLPrompt: it generates discrete prompts using RL framework.
5.1 FEW-SHOT TEXT CLASSIFICATION
Following the settings in existing work, evaluate model some few-shot text classification tasks. In Tab. 2 We compare our method with various baselines including RLPrompt. We can that on most tasks we tested, TEMPERA outperforms previous baselines by a large margin. For example, we have a 1.8% absolute gain on the SST-2 task (over RLPrompt), 3.9% gain on the CR task and the performance almost comparable to finetuning the language model on the AG News task. We also see that our method results in a much smaller variance between runs than Soft Prompt Tuning and AutoPrompt, indicating that it is more stable across different few-shot datasets. Comparing to search-based methods (e.g., Black-Box Tuning or GrIPS), our method avoids the expensive run-time search if one wants to perform test-time editing using one of the black-box optimization methods with surrogate reward. Note since the original Black-Box Tuning or GrIPS paper didn't perform guery-dependent search, this is our conjecture. Thus, out method achieves both test-time efficiency and good performances downstream tasks.
Table 2:
Few-shot classification results. We compare against different baselines in this setting. Results show that TEMPERA surpasses various baselines including finetuning, prompt tuning and discrete prompt search. The standard deviations are shown in brackets.
SST-2
Yelp
MR
CR
AG News
Finetuning
Finetuning (few-shot)
80.6 (3.9)
88.7 (4.7)
67.4 (9.7)
73.3 (7.5)
84.9 (3.6)
Soft Prompt Tuning
73.8 (10.9)
88.6 (2.1)
74.1 (14.6)
75.9 (11.8)
82.6 (0.9)
Continuous Prompt
Black-Box Tuning
89.1 (0.9)
93.2 (0.5)
86.6 (1.3)
87.4 (1.0)
83.5 (0.9)
AutoPrompt
75.6
79.8 (8.3)
62.0 (0.8)
57.5 (5.8)
65.7 (1.9)
Manual Prompt
82.8
80.9
79.6
76.9
In-Context Demo.
85.9 (0.7)
89.6 (0.4)
80.6 (1.4)
85.5 (1.5)
74.9
Discrete Prompt
Instructions
89.0
84.4
85.2
80.8
54.8
GrIPS
87.1 (1.5)
88.2 (0.1)
86.1 (0.3)
80.0 (2.5)
65.4
RLPrompt
90.1 (1.8)
93.9 (1.8)
86.7 (2.4)
87.2 (1.7)
77.2 (2.0)
Discrete Prompt
TEMPERA (ours)
91.9 (2.0)
92.6 (1.7)
(1.1)
(L.6)
85.5 (1.5)
5.2 IMPORTANCE OF TEST-TIME PROMPT EDITING
To illustrate the importance of test-time time prompt editing, we compare our method with various base- lines that do not perform test-time editing. In addition, also construct another baseline where

---

Published as a conference paper at ICLR 2023
Efficinccy 15ST2)
TEMPERA
Efficency
0.90
0.350
Carshigtts Ber Bersmmrce
Claniiication berfomens
0.85
Finguuning
Finetuningg
0.900
Finetuning TEMPERA
TEMPERA
0.80
TEMPERA
0.73
0.7
Black Taninn Sort Frompt Tunii AutsPrompt
Black Tunlig Soft Prompt Tuning
Black Tunling Soft Prompt Tuning AutoPrompt
0.70
0.6
0.63
Auteirropp
150
200
300
400
500
100
200
300
400
500
100
200
300
400
500
Number Traiinn Examples
Number Training Examples
Number of Training Exampies
Figure 3: Data Efficiency for TEMPERA: We compare data efficiency between TEMPERA and few-shot finetuning. Results show that we can achieve good performance with significantly less data (varying from 4x to 8x).
create a RL based method where the policy is not dependent on the input query x, denoted as "TEM- PERA (No TTE)". Results in Tab. show that TEMPERA even without test-time editing can find better query-agnostic prompts comparing to manually construct prompts, in-context demonstration and GrIPS. However, adding test-time editing can further improve the performance when the task is harder: we got 0.8% improvement on MR task and 3.0% improvement at AG News task. On SST-2, the effect of test-time editing is not significant as we suspect that the task is too easy. We found on harder tasks like AG News, the gain of test-time editing is huge.
Table 3: We compare our method against differ- ent methods which do not perform test-time edit- ing. Results show that test-time editing is mostly helpful in harder tasks like AG News.
Table 4: Ablation on different editing techniques. Results show that adding verbalizer-edits helps all the tasks (especially MR and AG News). Adding instruction-edits marginally helps the performance in SST-2 and MR.
SST-2
MR
AG News
Manual Prompt
82.8
80.9
76.9
SST-2
MR
AG News
In-Context Demo
85.9
80.6
74.9
TEMPERA (No Inst & Verb)
91.2
87.2
82.2
Instructions
89.0
85.2
54.8
TEMPERA (No Inst)
91.9
88.2
84.3
GrIPS
87.1
87.1
65.4
TEMPERA
92.4
88,4
85.5
TEMPERA (No TTE)
92.0
87.4
81.3
TEMPERA
91.9
88.2
84.3
5.3
DATA EFFICIENCY FOR TEMPERA
To illustrate the data efficiency of our method, compare the performance of TEMPERA with some few-shot standard finetuning results in Fig. We see that in SST-2, we achieve similar perfor- mance using almost 8x fewer training data. In tasks like Yelp, the gain is about 4x. We see that with fewer examples, TEMPERA strictly dominates fine-tuning methods. This critical when applying TEMPERA in the real-world application since labeled data is expensive to get.
5.4 QUALITATIVE ANALYSIS OF THE EDITS
We also visualize our policy by taking a few examples from the final prompts after editing in Tab. We see that our method mostly does example selection, verbalizer swapping and phrase-level struction editing. Our editing techniques are flexible and the final prompt may take different combi- nations for each query. addition, the resulting final prompt is still interpretable by human, showing that our method achieves flexibility and interpretability at the same time. Note that in the examples provided in Tab. [ our policy choose to modify the example selection and verbalization.
5.5 ABLATION: DIFFERENT EDITING TECHNIQUES
We ablate on the different editing techniques and study how adding removing them can affect the performance. The results are shown in Tab. We can see that adding each component (e.g., verbalizer, instruction) is helpful in terms of the final performance. We also find that verbalizer is es- pecially helpful in some tasks like AG News, resulting in 1.2% difference in the final performance. This indicates that adding more flexibility some extent can help the performance.
8

---

Published as a conference paper at ICLR 2023
Table 5: Qualitative results on the effect of the learned policy, We see that our method both enables the flexibility of various edits and interpretability of the final results. On the contrary, many prior methods produce non-readable prompts. Red text is prior to editing and blue text are the changes.
Refore Edit
"In this task, you are given sentences from movic reviews. The task classify a sentence as "posirive" if the sentiment of the sentence is positive or as "negative" if the sentiment of the sentence is negative. Review; of sauey. Sentiment: posstive, Review; cold movie. Sentiment: negative. Review: herees. Sentiment: <maask>. " "In this task, you are given sentences from movie reviews. The task is to classify a sentence as "greal" the sentiment of the sentence positive saucy. Sentiment; great. Review: cold movic, Sentiment: temible, Review; heroes. Sentiment: <'mask>.""
SST-2
After
Edit
(bettor
"terrible" the sentiment of the sentence is negative. Review; of
verbolizer)
Refore Edit
"Classify the news articles into the categories of World, Sports, Business, and Technology. Article: What's a Name? Well, Matt Sexier Than Paul (Reaters) Reuters Shakespeare said, a rose by any other would smell sweet. Right? Anewer: Technology. Article: Wall St. Bears Claw Back Into the Black (Reaters) Reuters- Pinters-Suie Short-sellers, Wall Street's sooc''swiinn dwindling band of ultra-cynios, are scoing green again. Answer: <_mask>."
AG News
After Edit
"Clasify the news articles into the categories of World, Sports, Business, and Technology- Asticle: Expansion slows Japan Economic growth Japan slows down the country experiences drop domestic and corporate spending. Answer: Rusiness. Article: Wall St. Bears Claw Back Into the Black (Reuters) Reuters- Short-seliers, Wall Street's dwindling band of ultra-cynies, are secing green again. Answer: < <mask>>
(better
exemplar
selection)
Table 6: Ablation on the number of in-context exemplars. Results show that increasing the number of examples results in consistent in- crease of performance except for AG News (which is due to the length limit).
Table 7: Ablation on the size of the prompt pool to select from. We see that the performance does not change too much when changing the size of the pool, indicating that the performance is rela- tively stable.
SST-2
MR
AG News
SST-2
MR
AG News
TEMPERA (2 Examples)
91.6
87.9
84.0
TEMPERA (Pool Size 8)
91.6
87.9
84.1
TEMPERA (4 Examples)
91.9
88.2
84.3
TEMPERA (Ponl Size 16)
91.9
88.2
84.3
TEMPERA (8 Examples)
92.4
88.4
82.2
TEMPERA (Pool Size 32)
92.2
88.4
84.7
5.6 ABLATION: NUMBER OF SHOTS
We also ablate on the number of examples used in the in-context demonstration part of our algorithm. We choose the size of 2, 4 and 8 for the analysis. We see that from Tab. 6 in all the tasks we tested (SST-2, MR and AG News), increasing the number of examples consistently improves the performance. However, the performance improvement is relatively limited. In addition, due to the input length limit constraint by the language model (512 for RoBERTa), longer sequences of input will be truncated. This results in the performance decrease when increasing the number of examples from 4 to 8 for AG News, where the input length is longer than 512.
5.7 ABLATION: SIZE OF THE PROMPT POOL
We also ablate the example size of the prompt pool where we keep the number of examplers of 4. Intuitively, allowing our method to choose in-context demonstrations from a large range of example pool can provide better prompts. From Table. 7 we can see that increasing the example pool size gives the algorithm more flexibility to choose in-context demonstrations, resulting in slightly better final performance.
CONCLUSION
In this paper we present TEMPERA, a test-time prompt editing method for large language models via reinforcement learning. We found that perform test-time editing can greatly improve the perfor- mance of downstream tasks for a pretrained language model. The proposed method only requires little guidance on high-level search space design and can easily incorporate prior human knowledge. achieves SoTA performance on multiple benchmarks including those from GLUE. This intersec- tion area of research between NLP and RL can inspire future research on designing better test-time editing algorithms for practical usage.
9

---

Published as a conference paper at ICLR 2023
REFERENCES
Stephen H. Bach, Victor Sanh, Zheng-Xin Yong, Albert Webson, Colin Rafell, Nihal V. Nayak, Abheesht Sharma, Taewoon Kim, M Saiful Bari, Thibault Fevry, Zaid Alyafeai, Manan Dey, Andrea Santilli, Zhiqing Sun, Srulik Ben-David, Canwen Xu, Gunjan Chhablani, Han Wang, Jason Alan Fries, Maged S. Al-shaibani, Shanya Sharma, Urmish Thakker, Khalid Almubarak, Xiangru Tang, Xiangru Tang, Mike Tian-Jian Jiang, and Alexander M. Rush. Promptsource: An integrated development environment and repository for natural language prompts, 2022.
Steven Bird, Ewan Klein, and Edward Loper. Natural language processing with Python: analyzing text with the natural language toolkir. "O'Reilly Media, Inc.". 2009.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffiee Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems, 2020. URL https: Papeaaaacl1eaaaaaapa.a.eea Paper.
Fille/157ccddbbfcct 967 htttt:::/pprcceeiiiii
neurips.....m cc/ wttpcpppppp///////////ppp
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02331 2022.
Mingkai Deng, Jianyu Wang, Cheng-Ping Hsieh, Yihan Wang, Han Guo, Tianmin Shu, Meng Song, Eric P. Xing, and Zhiting Hu. Riprompt: Optimizing discrete text prompts with reinforcement learning. 2022. doi: 10.4850/AARXI1.225..222 12548. URL https: /atps://arxxixs /aaaavivivv..///..///.aaa./aaavv org/ /
2205. 12548
Jacob Devlin, Ming-Wos Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers). pp. 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/w1/N19- 1423. https::: 7/ //cclannhhology.orr//111
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity, 2021.
Tianyu Gao, Adam Fisch, and Dangi Chen. Making pre-trained language models better few-shot learners, 2020a. URL /tRt::/tts://arr/ www.w/wwwww///////////////// 15722
Tianyu Gao, Adam Fisch, and Danqi Chen. Making pre-trained language models better few-shot learners. arXiv preprint arXiv:2012. 15723, 2020b.
Yun Hc. Steven Zheng, Yi Tay, Jai Gupta, Yu Du, Vamsi Aribandi, Zhe Zhao, Yaguang Li, Zhao Chen, Donald Metzler, Heng-Tze Tiee Cheng. and Ed H. Chi. HyperPrompt: Prompt-based task-conditioning of transformers. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Confer- ence on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 8678-8690. PMLR, 17-23 Jul 2022. URL he22f. html
mlr. peall..pres/vv62.
Yuezihan Jiang, Hao Yang, Junyang Lin, Hanyu Zhao, An Yang, Chang Zhou, Hongxia Yang, Zhi Yang, and Bin Cui. Instance-wise prompt tuning for pretrained language models. arXiv preprint arXiv:2206.0199 2022.
Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. How know what language models know? Transactions of the Association for Computational Linguistics, 8:423-433 2020. doi: 16.1162/ 00324. URI. //acl wwt.aaoooooooo////200200 aaclll0...aaall.m maall11.5 111-1128
10

---

Published as a conference paper at ICLR 2023
Brian Lester, Rami Al- Al-Rfou, and Noah Constant. The power of scale for parameter-efftcient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Lan- guage Processing, pp. 3045-3059, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational I. Linguistics. doi: 10. 00665////222.em///ppl/
URL.
ht.tps://aclantaae logy. wtwp:hhholooooogg////// ort.oogg/2220 http://2222eeem.mmmmemm.. In. 243
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.000 001000 2021.
Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. What
makes good in-context examples for GPT-3? In Proceedings of Deep Learning Inside Out (Dee- IJO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Archi- tectures, pp. 100- 14, Dublin, Ireland and Online, May 2022. Association for Computational Lin- guistics. doi: 10.18633/1//20222deell deelio-1.11
1.10. URL httph: /tt/tttss///eaaaatoooooooooo////
Xiao Liu, Kaixuan Ji, Yicheng Fu, Zhengxiao Du, Zhilin Yang, and Jie Tang. P-tuning v2: Prompt tuning be comparable to fine-tuning universally across scales and tasks, arXiv preprint 10X00:21,..0760, 2021a.
Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujic Qian, Zhilin Yang, and Jie Tang. Gpt understands, too. arXiv preprint arXiv:2 10r.:2103303330 10385, 2021b.
Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 190.::1977.11.... 2019.
Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. arXiv preprint arXiv:2104.0877 2021.
Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically or- dered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In Pro- ceedings of the 60th Anmual Meeting of the Association for Computational Linguistics ( (Volume I: Long Papers), pp. 8086 8098, Dublin, Ireland, May 2022. Association for Computational Lin- guistics. doi:
acl aacl-long: llg.oon..555 556
Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. Cross-task generalization via natural language crowdsourcing instructions. arXiv preprint arXhv:2/04.08773, 2021.
Archiki Prasad, Peter Hase, Xiang Zhou, and Mohit Bansal. Grips: Gradient edit-based in- struction search for prompting large language models. arXiv preprint arXiv:2203.07288, 2022.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. Exploring the limits of transfer learning with a unified text-to-text transformer. Mach. Learn. Res., 21(140):1-61) 2020.
Ohad Rubin, Jonathan Herzig, and Jonathan Berant. Learning to retrieve prompts for in-context learning. In Proceedings of the 2022 Conference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Language Technologies, pp. 2655-2671, Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022 http://mi....lll///a/o//////oooo http://aaaaaatooloorgg/////////a wwwp:////......o.. naacl22aaclaaaaaa 191
Timo Schick and Hinrich Schütze. Exploiting cloze-questions for few-shot text classification and natural language inference. In Proceedings of the 16th Conference of the European Chapter of
the Association for Computational Linguistics: Main Volume, pp. 255-269,, Online, April 2021. Association for Computational Linguistics. doi:
main URL https:
7/ //aclanthology.cooo www.:attwwooooy..ooo2...eeooc aaa.://22n.....lmma.....222.2.
Timo Schick, Helmut Schmid, and Hinrich Schütze. Automatically identifying words that can serve as labels for few-shot text classification. In Proceedings of the 28th International Conference on
11

---

Published as a conference paper at ICLR 2023
Computational Linguistics, pp. 5569-5575 Barcelona, Spain (Online), December 2020. Interna- tional Committee on Computational Linguistics. doi: 10.1865Ww12020.coolngg main.488. URL
https:
wttp:/yy...g///0// col.gggmm....................... 488
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint urXiv:/707.06344 2017.
Taylor Shin. Yasaman Razeghi, Robert L.. Logan IV, Eric Wallace, and Sameer Singh. AutoPrompt: Eliciting knowledge from language models with automatically generated prompts. In Empirical Methods in Natural Language Processing (EMNLP), 2020a.
Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. arXiv preprint arXiv:2010.15998 2020b.
Tianxiang Sun, Yunfan Shao, Hong Qian, Xuanjing Huang, and Xipeng Qiu. Black-box tuning for satpsagg---model-----sse----- arXiv preprint arXiv:2201.0357 2022.
Alex Wang. Amanpreet Singh, Julian Michael, Felix Hill, Omer I Lvy, and Samuel Bowman. GLUE: multi-task benchmark and analysis platform for natural language understanding. Proceed- ings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pp. 353-355, Brussels, Belgium, November 2018, Association for Computational Lin- guistics. doi: 10.1865//11W118550
httsttss:: htt://aclaaan http://aaaaanttoooo/aoao org/ W1 --46--4466
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy. and Samuel Bowman. Superglue: A stickier benchmark for general-purpose language un- derstanding systems. In H. Wallach, H. Larochelle, A. Beygelzimer, d' d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran As- sociates, Inc., 2019. URL htttpp:::/prccceiiggss www.innss.e..uuieeee mwwrrps......................am. wttpc//ppppe..///i.ieie. httpa//44afftaabbbooooooooaaaaaa htt.a//asa.......ooo.oooo///p/ra
Paper. pdf
Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganch Kordi, et al. Benchmarking generalization via in-context instructions on 1, 1,600+ language tasks. arXiv, 2022.
Zhuofeng Wu, Sinong Wang, Jiatao Gu, Rui Hou, Yuxiao Dong, V.G. Vinod Vydiswaran, and Hao Ma. IDPG: An instance-dependent prompt generation method. In Proceedings of the
2022 Conference of the North American Chapter of the Association for Computational Linguis- tics: Human Language Technologies, pp. 5507-5521, Seattle, United States, July 2022, Asso- ciation for Computational Linguistics. doi: 10. 403
URL https:
//aclanthology.u wwtp::y...../02........ www.:22..naacaaaam
Tony Z. Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Im- proving few-shot performance of language models, 2021. URL http:::/aaxiv..or//aa
2IC2.09690
Zexuan Zhong, Dan Friedman, and Danqi Chen. Factual probing is [mask]: Learning vs. learning to recall. In North American Association for Computational Linguistics (NAACL), 2021.
12

---

Published as a conference paper at ICLR 2023
A TRAINING DETAIL
We provide the training details here. We use standard PPO algorithm to do online policy optimiza- tion with GAE. We provide all the hyperparameters here for a reference. We'll specify our neural network architecture in the following section. Note that we perform additional observation normal- ization (i.e., keeping a running mean and std) and reward normalization. We also adopt the same number of parallel environment as the few-shot setting (e.g., 32 in our few-shot experiments). We found large size of parallel environment helps boost the performance.
Table 8: Hyperparameters used for TEMPERA in all the tasks.
Hyperparameter Value
Steps per training
Time limit
Number Parallel Processes
256
Learning rate
0.00005
Entropy Coefficient
0.005
Value loss Coefficient
0.5
Mini Batch Size
32
Gamma
0.99
GAE Lambda
0.95
Number of in-context Exemplars
Number of example pool
16
Positive lambda coefficient (A1)
2.0
Negative lambda coefficient (A2)
1.8
B NETWORK ARCHITECTURE
We follow the GPT (Brown et al. 2020) architecture and use the encoder layer for policy net- work. Note that our policy and baseline network shares the same attention based encoder. The attention is flat over all the possible candidate examples. We heads and 48 latent dimension. We build two different head with 2-layer MLP for each as the policy head and baseline head. We also don't 't dropout for the policy learning part. We found this boost up the performance.
3-layer encoder block with
C ADDITIONAL EXPERIMENTS
We perform additional experiments on some more tasks like RTE, QNLI, SNLI, MNLI and MRPC. Results show that we are consistently better than most of the discrete prompt optimization methods and continuous prompt tuning methods. On several tasks, we are also better than finetuning the entire model.
D NATURAL INSTRUCTIONS AND PROMPTSOURCE
We provide all the instructions we used our experiments from Natural Instructions. Here we just provide a few examples. Please refer to the github for all the instruction they provided. We also provide all the verbalizers we used our experiments from Promptsource. Here we only provide few examples. Please also refer to their github for the full verbalization.
E DATASET DETAIL
For the Finetuning, we use standard finetuning of the RoBERTa model from huggingface for 100 epochs, learning rate of 0.0003 and the optimizer of Adam.
13

---

Published as a conference paper at ICLR 2023
Table 9: Few-shot classification results. We compare against different baselines in this setting. Results show that TEMPERA surpasses various basclines including finetuning, prompt tuning and discrete prompt search. The standard deviations are shown in brackets.
RTE
QNLI
SNLI
MNLI
MRPC
Finetuning
Finetuning (few-shot)
58.6 (3.9)
60.2 (4.7)
54.64 (9.7)
47.8 (7.5)
77.4 (3.6)
Soft Prompt Tuning
54.7 (10.9)
49.7 (0.2)
36.13 (14.6)
33.2 (0.0)
51.6 (0.9)
Continuous Prompt
Black-Box Tuning
52.6 (0.9)
48.8 (0.6)
46.58 (1.3)
42.9 (2.0)
61.6 (0.9)
Manual Prompt
51.6
50.8
31.11
51.7
67.4
Discrete Prompt
In-Context Demo.
60.4 (0.7)
53.8 (0.4)
47.11 (1.4)
53.4 (1.5)
45.8 (0.8)
Discrete Prompt
TEMPERA (ours)
60.3 (2.2)
57.4 (1.5)
56.2
45.2 (2.0)
74.0 (1.0)
Newd
Bffidency teeg)
95
Obsaftcat on Poffmarcc
90
02
70
85
Fine-Tuning TEMPERA
Time Tur-Tu
FineTTnnngg PERA
50
TEMPERA
Data Bliianny TEMHERA (KTE)
Duta Blidiecc TENHERA (CNU)
Data Efficiency for TEMPERA (NP)
57.5
Fine-Tunig TEMPERA
Turinn
65.0
sicatttton Pretorments
TEMPERA
62.5
$0.0
22
65
80-
Fire Tun
TERA
200
300
500
100
400
500
200
300
420
500
Data Efficiency TIMPERA (MKLD)
Data Effictency TEMPRRA (MBPC)
FineTuning
Olouitaton Perttnneeoorre Perfemance
70
TEMPERA
Clsenitaattm Pertermanc
60
28
40
76
TESPERA
202
202
Training
Figure 4: Data Efficiency for TEMPERA: We plot all the finetuning performance for tasks we tested. We see that TEMPERA often achieves the better few-shot performance except for MRPC and QNLI.
F COMPARISON OF DIFFERENT METHOD
We compare the different property of different prompting methods in this section in order to give better understanding of different algorithms.
14

---

Published as a conference paper at ICLR 2023
Table 10: Natural instructions used for TEMPERA in all the tasks.
Task
Natural Instructions
SST-2
"In this task, you are given sentences from movie reviews. The task is to clas- sify sentence as "great" if the sentiment of the sentence is positive or as "terrible" if the sentiment of the sentence is negative."
AG News
"Classify the news articles into the categories of World, Sports, Business, and Technology."
CR
"In this task, you are given sentences from customer reviews. The task to classify a sentence as "great" if the sentiment of the sentence is positive or as "terrible" the sentiment of the sentence negative."
MR
"In this task, you given sentences from movie reviews. The task is to clas- sify a sentence as "great" if the sentiment of the sentence is positive or as "terrible" if the sentiment of the sentence is negative."
Yelp
"In this task, you are given sentences from Yelp reviews. The task is to classify sentence as "great" if the sentiment of the sentence is positive as "terrible" the sentiment of the sentence is negative."
RTE SNLI
N/A
"In this task. you't re given a pair of sentences, sentence 1 and sentence 2. Your job choose whether the two sentences clearly agree (entailment//dsaare) (contradiction) with each other, or if this cannot be determined (neutral). Your answer must be the form of the letters Yes, Maybe, and No respectively." and Sentence2). Answer "yes" if
QNLI
"You are given two sentences(Sentent these sentences are paraphrase of one another, otherwise answer "no"."
MNLI
"In this task, you re given pair of sentences, sentence and sentence 2. Your job to choose whether the two sentences clearly agree (entailment/Vissaree (contradiction) with each other, or if this cannot be determined (neutral). Your answer must be the form of the letters Yes, Maybe, and No respectively."
Table 11: Verbalizers used for TEMPERA in all the tasks. Natural Instructions
Task
SST-2
'Somcone just said to me "[[sentence)]" J Do you think they are {["sad"}} or [("happy"}}* {{ answer choices[label]]}" "What label best describes this news article? [annserrchoicee[[lae]]]]
AG News
{[text)
CR
'Somoone just said to me "[[sentence)]". Do you think they are ( {{"sad")) or {"happy""J" } {{ answer.choices chaeces[[abbel]]
MR
[text)] Did the reviewer find this movie {["good or bad"})? {{ an- wwer.choices/labe]] }}'
Yelp
I{{ text Overall, the experience is {f answer choices[label] }}* 'Does the claim "{ "([sentence2))) follow from the fact that "( ([settencee))"" Please answer either {("yes") or [["no")]- { (hwperrchoices(laee]]]] 'Suppose {[premise}} Can we infer that "I ([[hypothesis)""" Yes, no, or maybe? {{ answer wwwwaanneooiieeeeeeeeeaaaaaasaae chooces/laabel]] }}'
RTE
SNLI
QNLI
'{ [sentence)] } Does that sentence have all you need answer the question "{ "[[question] ""? { (aasserr_chol "
MNLI
'Suppose {premisee)} Can we infer that "( "[(hypothesis))"? Yes, no, or maybe? {{ answer choices|label] }}
MRPC
Does the sentence ([sentence1}) paraphrase (that is, mean the same thing as) this sentence? {[sentence2)) {{ answer.choices/labe }}*
15

---

Published as a conference paper at ICLR 2023
Table 12: Scaling results for TEMPERA in 512 training data per class. Results show that TEMPERA also scales and achieves better results comparing finetuning.
SST2
MR
AG News
RTE
Finetuning
Finctuning (few-shot)
93.4
87.0
89.5
67.9
Discrete Prompt TEMPERA (ours)
93.8
88.6
88.6
71.4
Table 13: Details for the dataset including the type, size of training, evaluation and test. Note that
here all the sizes are shot dataset.
Dataset
Type
C
[Tain
Dev
Test
SST2
Sentiment
2
32
1.8k
AG News
topic
4
64
7 7.6k
CR
Sentiment
2
32
2k
MR
Sentiment
2
32
2k
Yelp
Sentiment
2
32
38k
RTE
NLI
2
32
0.3k
SNLI
NLI
3
48
10k
QNLI
NLI
3
48
9.8k
MNLI
NLI
3
48
9.8k
Gradient-Freo
Guided-Optimnine Optimization
Interprotable
Query-Dependen!
Fine-Tuning
X
Manual Prompt
Instructions
V
X
V
X
Demonstration
Sofl Prompt Tuning
X
X
X
Discrete Prompt
X
X
Search
AuisPrompl
X
RL Prompt
X
X
Tempera (ours]
Figure 5: Comparison of Different Prompting Methods: We compare the different property of different algorithms. We can see that TEMPERA is gradient-free, the resulting prompt is inter- pretable and dependent.
16


Reviews:
[{"review_id": "uOBKeTj938q", "reviewer": "Reviewer_Hybx", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes RL-based test-time editing for ", "summary_of_the_review": "This paper proposes test-time prompt editing with RL but may have some issues.", "strength_and_weaknesses": "## Strengths\n1. The paper is overall well-written and easy to follow.\n2. The paper includes a wide range of baselines, both continuous and discrete.\n\n## Weaknesses\n1. The scalability of the proposed method is not good. As shown in Figure 3, TEMPERA holds an advantage against fine-tuning but cannot scale. Also, I would like to see other few-shot prompting methods illustrated in the same figure.\n1. Figure 1 is misleading. It is the best-performing dataset in Figure 3. It's inappropriate to single it out and put it in the introduction.\n1. The editing patterns are not diverse enough. The case study is not very convincing. The improvement doesn't seem explainable. \n1. The proposed method is in essence similar to prompt/verbalizer searching. \n1. How do you justify the use of RL here? Is there a strong reason not to use search instead?\n1. I think GLUE or SuperGLUE can be more convincing. Yelp, MR, CR and AG News are considered easy.\n\n-----\nAfter rebuttal: The authors have addressed many concerns mentioned in my review during the response period. I've updated my score to 8.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "WsCbjHOMKHr", "reviewer": "Reviewer_Hk2t", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work explores editing prompts in large language models on a per-instance level via reinforcement learning. In contrast to existing works, the editing is conditioned on the input query, rather than a task, and the action space for editing is hand-designed. The approach is tested on different natural language few shot classification tasks, showing better performance than few-shot fine-tuning and discrete prompt optimization methods, being more data-efficient for tuning, interpretable, and robust to different hyperparameters in prompting.", "summary_of_the_review": "This paper discussed a relevant topic, has strong ideas, a new method, thorough experiments, and SOTA results. I generally think it would be a valuable contribution to the community, but in the current state there are important details that are difficult to understand for me, and I assume for other readers in the community. If the aspects described above are clarified, I am very open to changing my rating to accept.\n\n[Edit] The authors have addressed my concerns in the revised version. I therefore have updated my score.", "strength_and_weaknesses": "**Strengths:**\n- Relevance:\n\t- Given the wide use of prompting for tasks in LLMs, and the effect it can have in final performance of downstream tasks, exploring methods to design better prompts is a very important problem.\n- Writing:\n\t- I like how the related work is structured, motivating first the need for prompt learning, and later providing an overview of prompt learning methods and methods based on retrieval examples from the training set.\n\t- Formulation in section 4 makes the method clear to understand and connect to RL frameworks.\n\t- Table 1 is useful to understand action space\n- Method:\n\t- The policy used to modify prompts is constrained, which may allow for more interpretability and sampling efficiency (Fig 1), while at the same rich in some parts (swap example, verbalizer change), allowing different kinds of operations ito the prompts.\n\n- Experiments:\n\t- Thorough and valuable ablation studies, comparison with solid baselines and in different tasks.\n\t- SOTA or comparable results in a wide set of tasks.\n\t- Very good properties in few-shot settings.\n\t- Very cool example of exemplar editing in Table 5, I would love to see this together with the other exemplars. In general, I think it would be really interesting to see if there are common properties in the exemplars edited by the policy. Does the model tend to balance the number of exemplars per class, or does it try to select more exemplars close to the query class?\n\n\n**Weaknesses:**\n\n\n- Writing Clarity \n\t- From my understanding of the paper, the prompts are edited at test time using a pre-trained policy, but the policy is not updated at test-time but training time. While this is reasonable because at test time we do not have access to the GT label and therefore cannot compute the reward, the test-time optimization generally makes me think about having a surrogate reward at test time used to update the policy. I would strongly encourage to clarify this more explicitly in the paper, maybe by saying, that the policy is trained first by constructing a training set xyz, and at test time it is deployed to edit the prompts. Referencing directly Algorithm 1 would also help a lot.\n\t- In line with the previous comment, it is hard for me to understand whether K in the training set corresponds to the number of examples for in-context demonstrations, or an external training set, that contains description examples and query. When I think about a few-shot dataset for NLP I am thinking about in-context learning, but here it seems otherwise?\n\t- Figure 2 would benefit a lot from a walkthrough example. It is hard to understand what exactly is a verbalizer, E1, ... En etc.\n\t- Section 3.2 could be condensed, a lot of it is already in related work.\n\t- Clarify, in sec. 4, \"... contains both information\", what is both information?\n\t- How is the initial instruction provided? Is it task specific?\n\t- A figure of the main methods from prompt-search vs the existing method would be helpful to quickly grasp the main differences, or maybe a table of features.\n\t- Minor: It would be useful in 3.2 to repeat the difference between He et al. 2022, Deng et al. 2022 for section 3.2 since authors talk about query-dependent prompts and then point at limitations of query-agnostic prompts. This seems addressed in the next paragraph, but it is difficult to follow form there that previous methods are these 2 methods. In fact, from my understanding He at al. 2022 is prompt specific, but Deng et al. 2022 is not. Could authors confirm and correct citation in 3.2 in that case?\n\t- How is Tempera No-TTE different than promptRL. One main thing is the parametrization of the policy, is there something else?\n- Method:\n\t- I would appreciate more details of the policy, there is attention but how is it done? Is it flat over all the techniques and candidates, or is it hierarchical in that first a technique is selected and then a candidate.\n\t- Table 4 would be good to include No verb, but Inst\n- Results\n\t- Regarding test-time efficiency in Sec. 5.1, while the claim could be true, it is not validated that the run-time will be smaller. A table with clock time or num ops would be good to validate, and if it cannot be provided, make clear that it is a hypothesis.\n\t- Few-shot is a very valid setting, but it would be really interesting to see if the current method also improves zero-shot performance.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "NjnEsCYKxwJ", "reviewer": "Reviewer_cYCQ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a new model that is able to construct the instance-dependent prompt through reinforcement learning. It allows the agent to perform different editing techniques to update instructions, few-shot exemplars, and verbalizers at test time to construct query-dependent prompts efficiently. Results on different datasets show the effectiveness of the proposed model.", "summary_of_the_review": "The paper is competitive and I am looking forward to seeing my questions answered in the rebuttal stage and in the revised paper.", "strength_and_weaknesses": "**Strengths**\n\n**1. The proposed method is novel and effective.** \n\nThe paper proposes a novel method to edit the instance-relevant prompt in the test time via RL. The editing/action space covers a comprehensive set of commonly-used components like instructions, few-shot exemplars, and verbalizers. Results show the effectiveness of the proposed method in most datasets.\n\n**2. The paper is well-written and the experiments are comprehensive.** \n\n**Weaknesses/Feedback**\n\n**The experimental section could be improved.** \n\nSome experimental details are missing. For example, what are the dataset scales for SST-2, Yelp P., MR, CR, and AG News? And what is the value of |Y| in “Task Settings”?\n\nWhat is the exact number of training examples for the results in Figure 1? What are the details of the standard fine-tuning method?\n\nIn Table 4, only three different editing techniques are included. It could be critical to include more editing techniques.\n\nOn datasets like SST-2, Yelp P., and MR, the improvements of TEMPERA are not obvious compared to existing baselines.\n\nI have no idea of the details of the compared method in Figure 1. So I am concerned it might not be fair to claim that TEMPERA has greater data efficiency than the standard fine-tuning model. Look forward to clarifications in the rebuttal phase.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "E2-D2EDw39", "reviewer": "Reviewer_F772", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "* This paper proposes a test-time prompt editing technique with reinforcement learning. Compare to prior methods, TEMPERA can efficiently leverage prior knowledge, adaptive to different queries, and provides an interpretable prompt for every query. This method achieves 5.33x on average improvement in sample efficiency when compared to traditional fine-tuning methods.\n", "summary_of_the_review": "This paper proposes to treat prompt tuning as a test-time editing problem with RL, which can give more flexibility and demonstrate promising empirical results. However, it's novel because it's the first to apply RL on prompt editing, but the RL technique itself is not much novel and seems like it mostly improves sample inefficiency but not absolute accuracy (e.g. not in few-shot setting). So, I'd recommend weak accept.", "strength_and_weaknesses": "Strength:\n* This paper proposes a novel method: It formulates discrete prompt optimization as using RL to edit an initial prompt. It also proposes carefully designed action space, and a set of techniques to improve the final performance.\nWeakness:\n* It seems that this method mostly improves sample efficiency but not absolute performance (e.g. not in few-shot setting). \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%