Guiding Large Language Models via Directional Stimulus Prompting
Authors: Zekun Li
Baolin Peng
Pengcheng He
Michel Galley
Jianfeng Gao
Xifeng Yan
Venue: NeurIPS 2023 poster (2023)
Keywords: Black-box Large Language Models
Directional Stimulus Prompting
Hint
Reinforcement learning
Prompt optimization
TL;DR: We propose a new prompting framework to guide black-box LLMs toward desired output by introducing directional stimulus (i.e., hints) in the prompt, which is generated by a small model optimized via supervised fine-tuning and reinforcement learning.

Abstract: We introduce Directional Stimulus Prompting, a novel framework for guiding black-box large language models (LLMs) towards specific desired outputs. Instead of directly adjusting LLMs, our method employs a small tunable policy model (e.g., T5) to generate an auxiliary directional stimulus prompt for each input instance. These directional stimulus prompts act as nuanced, instance-specific hints and clues to guide LLMs in generating desired outcomes, such as including specific keywords in the generated summary. Our approach sidesteps the challenges of direct LLM tuning by optimizing the policy model to explore directional stimulus prompts that align LLMs with desired behaviors. The policy model can be optimized through 1) supervised fine-tuning using labeled data and 2) reinforcement learning from offline or online rewards based on the LLM's output. We evaluate our method across various tasks, including summarization, dialogue response generation, and chain-of-thought reasoning. Our experiments indicate a consistent improvement in the performance of LLMs such as ChatGPT, Codex, and InstructGPT on these supervised tasks with minimal labeled data. Remarkably, by utilizing merely 80 dialogues from the MultiWOZ dataset, our approach boosts ChatGPT's performance by a relative 41.4%, achieving or exceeding the performance of some fully supervised state-of-the-art models. Moreover, the instance-specific chain-of-thought prompt generated through our method enhances InstructGPT's reasoning accuracy, outperforming both generalized human-crafted prompts and those generated through automatic prompt engineering. The code and data are publicly available at https://github.com/Leezekun/Directional-Stimulus-Prompting.

Guiding Large Language Models via Directional Stimulus Prompting
Zekun Li"; Baolin Peng", Pengcheng He², Michel Galley', Jianfeng Gao Xifeng Yan University of California Santa Barbara
Microsoft
{zekunli. xyan) @cs ucsu edu
(bapeng, penhe, mgalley, ifgao @microsoft.com
Abstract
We introduce Directional Stimulus Prompting, a novel framework for guiding black-box large language models (LLMs) towards specific desired ouiputs. Instead directly adjusting LLMs, our method employs small tunable policy model (e.g., T5) to generate auxiliary directional stimulus prompt for each input instance. These directional stimulus prompts act as nuanced, instance-specific hints and clues to guide LLMs in generating desired outcomes, such including specific keywords in the generated summary. Our approach sidesteps the challenges direct LLM tuning by optimizing the policy model to explore directional stimulus prompts that align LLMs with desired behaviors. The policy model can be optimized through 1) supervised fine-tuning using labeled data and 2) reinforcement learning from offline or online rewards based on the LLM's output. We evaluate our method across various tasks, including summarization, dialogue response generation, and chain-of-thoughh reasoning. Our experiments indicate a consistent improvement in the performance of LLMSs such ChatGPT, Codex, and InstructGPT these supervised tasks with minimal labeled data. Remarkably, by utilizing merely 80 di- alogues from the MultiWOZ dataset, our approach boosts ChatGPT's performance by relative 4..4%, achieving or exceeding the performance of some fully super- vised state-of-the -fetthrart models. Moreover, the instance-specte chain-of -ofthhugght prompt generated through our method enhances InstructGPTTs reasoning accu- racy, outperforming both generalized human-crafted prompts and those generated through automatic prompt engineering. The code and data are publicly available
Introduction
In recent years, new paradigm has emerged in natural language processing (NLP) with the rise of
large language models (LLMs) such as InstructGPT, ChatGPT (6). GPT-4 45) PaLM [10], and others. These models exhibit emergent abilities [68] such as strong in-context learning and few-shot prompting capabilities, which were not present in previous "smaller" language models (LMs) like BERT 141 RoBERTa 37 GPT-2 (52), and T5 (53). This shift in paradigm has led remarkable advancements in NLP, with LLMs demonstrating impressive general-purposs power. However, due to commercial considerations and the risk of misuse, most LLMs do not publicly release their parameters and only allow users to access them through black-box APIs. While there also exist open-sourced LLMs, fine-tuning them for specific tasks or use cases can be computationally inefficient. In this scenario, the standard approach for utilizing LLMs to perform diverse tasks is crafting generalized
Part the work was done when Zekun Li was interning Microsoft Research.
Co-advise this work.
httpp://gihuub.cmm//eeeeekkk Ditireckkuu/iinettooaallll Stiiualls----------e Stimulas-Prompting
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Input text
Article: (CNN) the first time eight years, legend returned to doing what does best, Contestants down!" on the April 1 edition of 'The Price Is Right" encountered not host Drew Carey but another familiar face in charge of the years before stepping down in 2007. Looking
"come on
proceedings. Instead, there was Boh Barker who hosted TV game show spry 91, Barker handled the first price guessing game the show, the classic "Lucky Seven," " before turning hosting duties over to Carey, who finished up. Despite being away from the show for most the past eight years, Barker didn't seem to miss a beat.
Reference
Bob Barker rerkerre to hast "The Price Is Right" on Wednesday. Barker, 91, had retired as host in 2007.
Directional Stimulus Prompting
Standard Prompting
Q: Summarize the above article briefly in 2-3 sentences based on the hint. Hint: Bob Barker; TV: April 1: "The Price Is Right:: 2007: 91.
Q:
Summarize the above article briefly 2-3 sentences.
Model output
Model output
Bob Barker, the former host of The Price Is Rightt. returned to the show for the first time years. He hosted the first game of the show, and then handed the hosting duties over to Drew Carey.
A:
A: On April Bob Barker returned to the TV. show "The Price Is Right after eight years Looking spry at 911 Barker handled the first guees-gu game Barker stepped down host 2007, but didn't seem to miss a beat.
ROUGE-1: 48.39
ROUGE-1: 34.48
Figure 1: Comparison of our Directional Stimulus Prompting and the standard prompting method using LLMs such as ChatGPT for the summarization task. DSP utilizes directional stimulus/hints (highlighted orange), which are keywords in this case, to provide instance-specifie guidance to LLMs in generating summaries (highlighted in blue) that better align with the desired reference summary with higher ROUGE scores or other measures like human preferences.
task-specific prompts query LLMs. While LLMs have demonstrated considerable performance on a wide range of language tasks, they still struggle to generate outputs that fully align with desired behaviors and directions on some specific tasks and use cases [16]
Optimizing Large Language Models (LLMs) directly for specific tasks can be infeasible or inefficient for many users and developers, leading researchers to shift their focus towards prompt engineering and optimization. Prompt engincering approaches, which involve manually or automatically designing optimal task-specific natural language instructions and selecting appropriate training samples for demonstration in the prompt, have consequently gained the attention of researchers [6] 55 79 39].
Despite these efforts, the majority are centered devising task-specific prompts, often falling short in steering LLMs to generate desired results on a per-instance basis.
To address the challenge, we propose novel framework called Directional Stimulus Prompting (DSP). This framework introduces a new component called the directional stimulus into the prompt to provide nuanced, custance-specifie guidance and control over LLMs. Specifically, the directional stimulus prompt acts as hints and clues for the input query to guide LLMS toward the desired output. Notably, this differs from the methods that augment LLMs with additional knowledge retrieved from external sources 225 60), as the directional stimulus prompt is generated solely based on the input query in our framework. Figure Saaureeeompaace compares our proposed prompting approach, DSP, with standard prompting for the summarization task. Our approach incorporates keywords in the prompt as the directional stimulus prompt to hint at key points the desired summary should cover. By providing this instance-specific guidance through directional stimulus prompts, LLMs can generate outputs that more closely align with the desired reference summary.
We utilize a relatively small and tunable LM (c.g.. T5), as the policy model to generate the directional stimulus prompt for each input query. This approach enables us to sidestep the direct optimization of black-box LLMs by optimizing the small tunable policy model instead. We train the policy model through supervised fine-tuning (SFT) using few collected labeled data. After supervised fine-tuning, we further optimize the policy model to explore better directional stimulus prompts with reinforcement learning (RL). During RL training, we aim maximize the reward defined as downstream performance measures or any other measures of the LLM's output conditioned on the stimulus generated by the policy model.
Figure provides the overview of our framework, using the summarization task as an illustrative example. We employ compact, tunable policy model to generate the directional stimulus prompt,
2

---

(Pseudo) Label Data
Reward
Input Stimulus
Train RL
(KOUGL))
Stimulus (hints)
Output
Article: (GNN) For the first dight years, a TV logend retumed to doing what does best. Contestomss told to "come on down"" on the horil edition of "liee Price Richi"
Keywords:
Summany: Righ""
The Proce in Hagh Rib Harker
2002 R1
Looking 31, Barker handiec
Policy LM (e.g.. T5)
Black
Figure 2: Overview of our proposed framework DSP, where we learn a small tunable policy model to generate the directional stimulus (keywords in this case) that provides input-specific guidance for the LLM toward the desired target. The policy model can be trained with SFT and/or RI., where the reward is defined as the downstream task performance measure, such as the ROUGE score for the summarization task, or other alignment measures like human preferences.
which specifies keywords that should be included in the LLM -LM-gennrated summaries. The policy model can be trained with SFT and RL, where the reward is typically defined as the downstream task performance measure, such the ROUGE score for the summarization task, other alignment measures like human preferences.
Our framework can be flexibly adapted to wide range of LMs and tasks by choosing the ap- propriate directional stimulus prompt, i.e., hints. We conducted experiments on summarization, dialogue response generation, and Sfain-of-tthuuht reasoning tasks to evaluate the effectiveness of our framework. The results demonstrate that our DSP approach can effectively guide LLMs toward the desired targets with small collection of labeled data. Specifically, we conduct experiments with the black-box LLMs: ChatGPT, Codex, and InstructGPT. For the policy model, we employ a 750M Flan-T5-Large [33 II and 220M T5- Base. For the summarization task, directional stimulus, which hints at key points that the desired summary should include. Despite CharGPT's already considerable performance, the policy model trained with only 4,000 samples from the CNN/Daily Mail dataset [33 improved the ROUGE and BLEU scores by 4-13%. For the dialogue response generation task, train the policy model to generate dialogue acts that indicate the underlying intentions behind target responses on dialogues from the MultiWOZ dataset (7]. Guided by the policy model trained with only 80 dialogues. ChatGPT's performance improved by up to 41.4% in combined scores, achieving comparable or even better performance than some ofte-offtheeart models trained on the full dataset with 8,438 dialogues. For the chain-of-thought reasoning, we train the policy model to generate a trigger prompt for each input query to steer the LLM chain-of-thowd offtthuugttthtttthh reasoning, achieving better performance than the generalized hand-crafted prompts and those produced through the automatic prompt engineering approach [79), suggesting the effectiveness of our approach for automatic prompt engineering and optimization.
keywords as the
2 Directional stimulus prompting
For downstream task, there is an input space X, data distribution D over X, and an output space Y. Due to the strong in-context learning and few-shot prompting abilities, LLMs can perform diverse tasks and generate the output y by including instructions that describe the task, a few demonstration examples, and the input query x in the prompt (6]. However, such prompts cannot always steer L.L.s toward desired outputs, especially when it comes to fine-grained instance-specific desired behaviors. For instance, in the case of the summarization task, the input is article, and the output y is the corresponding summary. Different summarizers have distinct styles and emphasize different aspects of an article [16]. In this case, it may not be enough effectively steer LLMs toward generating summaries that closely match reference summaries relying solely on task task-specific instructions or demonstration examples to describe such nuanced differences for each sample.
To this end, our Directional Stimulus Prompting (DSP) approach introduces small piece discrete tokens& z named "directional slimulus" into the prompt, which acts as hints and clues to provide L.Ms with fine-grained guidance toward the desired direction. For example, for the summarization task, the directional stimulus might consist of keywords that should be included in the desired
3

---

summary. To generate this stimulus for each input query, we use small tunable policy language model, (ror(2) We then use this generated stimulus, 2, along with the original input, x, to construct the prompt that steers the L.M toward generating its output, PL at. 2). It's important to note that the parameters of the L.M,, PLLM, are kept frozen, as they are either inaccessible or inefficient to tune. Overall, when using the LLM with DSP to perform a downstream task, the output is obtained via y ~ (LLM z), , 2 ~ (-o).-
2.1 Supervised fine-tuning
To train the policy model that generates directional stimulus for LLMs, we first perform supervised fine-tuning (SFT) on a pre-trained LM (c.g.. T5, GPT-2. etc) on small collection of labeled data. To collect the data, we could heuristically select or annotate the "pseudo-stimulus" == for each input query æ and target output y pair based on the downstream task. For example, for the summarization task, we use keywords that the reference summary includes as pseudo-stimulus, while for the dialogue response generation task, we use dialogue acts that indicate the underlying meaning of the desired system response (see Section 3 for details). The resulting dataset D' {(xx 2*)} consists of input-stimulus pairs. We then fine-tune the policy model by maximizing the log-likelihood:
LSFT = (m, po (z (z))
(I)
Supervised fine-tuning can provide a good initial point for the policy model. However, is important to note that the heuristically selected or annotated pseudo-stimulus may not always be optimal, and the supervised fine-tuned policy model may not generate the most preferred directional stimulus for the LLMs toward the desired outputs. To overcome this limitation, reinforcement learning (RL) to further fine-tune the policy model. By directly optimizing the LLM's output toward desired targets, RL training enables the policy model explore and generate more effective directional stimulus.
also incorporate
2.2 Reinforcement learning
Optimization objective Our goal is to steer the LLM's generation toward the desired target by maxi- mizing an alignment measure R. which can take various forms such as downstream task performance measures (e.g., ROUGE score for summarization), human preferences, or other customized measures. Mathematically, we aim to maximize the below objective:
Beg (-Pxx((x))))) R(x,, y)).
Since the parameters of the black-box LLM are not accessible or tunable, we resort to optimizing the policy model to generate the directional stimulus that guides the LLMs generation toward maximizing the objective. To achieve that, we define another measure RLLM that captures how well the LLM performs when conditioned given stimulus 2:
RLIM(1 2) = R(x, y). PLLM(-12,22
This allows to cast the original objective of maximizing R into optimizing the policy model to generate stimulus that maximizes RIIM- By doing so, the LLM is effectively used as an evaluation function to guide the policy model toward generating more effective directional stimulus. Thus, the optimization objective for LLMs in Equation is equal to the optimization objective for the policy model:
maXpeca E. D.z (-(a)) RRLM (x, =]].
(4)
RL formulation However, the above optimization is intractable for the policy model. To address the issue, we formulate the policy model optimization as an RL problem and employ proximal policy optimization (PPO) (59) We use the policy model to initialize policy network To PPoL and then update π using PPO. The process that the policy model generates a sequence of tokens as stimulus can be seen as a Markov decision process (MDP) (S, A, r. P) with a state space S, action space A. reward function r, and state-transition probability P. In each time step of an episode, the agent selects an action (token) from the vocabulary V according to the distribution of the current policy network T22|2, 2. The episode ends when an ond-of-sequence token is selected, and the stimulus is generated. We can fine-tune the policy network x by optimizing the reward
E., [r] Exw BeeNDeeatti (- (x),
(5)

---

Reward function Recall that our goal is to maximize the objective in Equation which can be used as the reward r. To keep the policy network π from moving too far from the initial policy model Prol. we also add a KI.-divergence penalty reward. Therefore, the final reward becomes:
π (2 ())
r =) = RRIM(æ z) -glog
(6)
PPOL. (z)z)
Following 80 54
we dynamically adapt the coefficient /3 during training:
(T prot.) KL
e, =clip
-0.2,0.2
(7)
Kl Kl
B++1 Be Kges).
(8)
Implementation To optimize the policy network и, we use the NLPO version PPO from [54]. which is specifically designed for language generators. To address the issue of large action spaces in PPO, NLPO learns to mask out less relevant tokens in the vocabulary using top-p sampling. This technique restricts the action space to the smallest set of tokens whose cumulative probability is greater than the given probability parameter p. which we set to 0.9 in our experiments. Both the policy network and value network are initialized from the supervised fine-tuned policy model PPOL, with the final layer of the value network randomly initialized to output a scalar value using regression head.
3 Experiments
Our proposed framework DSP can be flexibly applied to various types of LMs and generation tasks. In this work, we focus on 1) summarization, 2) dialogue response generation, and 3) chain-of- thought reasoning tasks. We mainly pre-trained T5 Flan-T5 [53 M to initialize the policy model and experiment with the black-box LLMs including ChatGPT (gpt-3. 5-turbo), Codex (code-davinci-002), and InstructGPT (text (text-davinci-o02))
3.1 Summarization
have shown that LLMs, such as GPT-3, InstructGPT, and ChatGPT, are capable of generating high-quality summaries with zero- or few-shot prompting. However, their reference-basee based evaluation benchmark performances, such as ROUGE scores, still lag behind fine- tuned methods, indicating that the generated summaries may not completely match the style and emphasis of the reference summaries. summaries that more closely align with the reference summaries by providing keywords that should be mentioned the desired summaries as hints. We evaluate the effectiveness using metrics that compare the generated summarics against reference summaries. Notably, other desired directions, such as better alignment with human preferences, can also be pursued.
Recent studies [16] 75 4
our experiments, we seek to guide LLMs to generate
Dataset and evaluation We conduct our experiments on the CNN/Daily Mail dataset, a widely- used news summarization benchmark. To keep the cost of API usage low, we train on subset of 1,000, 2,000, and 4,000 article-summary pairs from the total 287,1 13 samples in the training set. For evaluation, we randomly select 500 samples, following previous work [16 65), which has been proven to provide sufficient statistical power [8. We use the overlap-based metrics, including ROUGE 33 BLEU (47, and Meteor 3 and similarity-based metric, BERTScore 77.. to compare the generated summaries with the references. The reported evaluation scores are averaged over three inferences of ChatGPT for each query, using temperature of 0.7 and top of 1.0. We use the same three demonstration examples in the prompt for standard prompting and add keywords as directional stimulus the prompt for our approach, DSP. The exact prompts used in our experiments provided in the Appendix.
Supervised fine-tuning details We use keywords as the pseudo-stimulus to train the policy model with supervised fine-tuning as discussed in Section 2.] To collect the data, we employ textrank [ to automatically extract the keywords from the article and summary and only keep those that appear in the reference summary. As result, we obtain list of extracted keywords for each article-summary pair in the dataset. To convert them into a sentence that serves as the stimulus, we concatenate them using split token ".",, resulting in the stimulus formated as "[Keyword]]; [ [Keyword2];
5

---

Standard Prompting
DSP w/ SFT
DSP SFT+RL
28 70
40 MI
40.19
17.45
ROUGE-1
40
ROUGE-2
17.00
ROUGE-L
28.04
25.62
30 45
43
31,41
16.20
18.20
38
1K
2K
4K
1K
2K
4K
1K
4K
#Training
#Training data
#Training data
10
33
0.885
03.99
32.70
0.2545
BLEU
8.99
8.71
METEOR
32.24
BERTScore
0.884
0.892
8832
1X7
2.00
5.06
8.00
31.50
0.802
0.800
C,8507
0,850
0.88
1K
2K
#Training data
#Training data
#Training data
Figure 3: Performance comparison of ChatGPT with standard prompting and DSP trained with SFT and SFT+RL, using varying numbers of training samples from the CNN/Daily Mail dataset.
[KeywordNJ.". We use the constructed article-stimulus pairs to train the policy model via supervised fine-tuning. The input format for training is "Extract the keywords: [Article/", while the output is the target stimulus consisting keywords. The policy model was trained for 5 epochs with a 2 x 10-5 learning rate.
RL training details we aim to guide ChatGPT in generating summaries that more closely match the reference summaries, adopt the automatic reference-based metric scores as the alignment measure reward. Specifically, we calculate the ROUGE- Avg score between the generated summaries and the reference summaries the reward, with rescaling coefficient of 10. We experimentally found that other automatic evaluation metrics, such as BLEU and Meteor, perform similarly. To reduce variance, and compute the average reward. Additionally, we assign a step-wise reward, which we found could improve the efficiency and stability of the training process. Specifically, the policy model generates sequence of keywords in each episode, during which we assign a reward of if a keyword appears in the reference summary and a penalty reward of -0.2 is given otherwise. We train the policy network for 51k episodes, with epochs per batch, a batch size of 8, and learning rate of 2 x 10-6, The Kaaaret and Bg Equation 7 are set 0.5 and 0.005, respectively.
generate four outputs per input query using ChatGPT with temperature of 0.7
Results We evaluate the performance of ChatGPT with standard prompting and approach DSP trained with SFT or SFT and then RL. RSL)+RL) varying sizes of training data and present the results in Figure 3 As can be seen, all the evalu- ation scores improve with our proposed DSP com- pared with standard prompting. Specifically, the supervised fine-tuned policy model generates the stimulus that effectively guides ChatGPT to gener- summaries that closely align with the reference summaries, leading to improved benchmark per- formance. Furthermore, the additional fine-tuning of the policy model with RL results in further per- formance improvement, indicating the effective- ness of RL in exploring better directional stimulus that maximizes the reward. As the size of the train- ing data increases, the performance improvement becomes more significant. Despite using a small collection of only 1,000 to 4,000 samples to keep API usage costs low, our DSP approach still
The training
1000 samples
40.0
5.2
39,9
5.0
4.8
4.6
39.7
Reward
4.4
39.6
4.2-
39,5
4.0-
3.8
39.4
Training Reward
Validation ROUGE-1
3.4
10
Training iters
Figure Training curve on 1000 samples from the CNN/Daily Mail dataset.

---

consistently enhances ChatGPT's ROUGE, BLEU, and Meteor scores by 1-2 in absolute points, even though ChatGPT has already achieved considerable performance. However, due to the discrepancy between the semantic-based metric BERT'Score and the overlap-based metric ROUGE, which are used as the reward, the improvement in BERTSSoree after RI. training may be relatively less significant. Figure presents presents the change of training rewards and ROUGE-1 score on the validation set during the training process on 1,000 samples. We can see that the performance is closely related to the training rewards, and the training is relatively stable using the NLPO algorithm.
3.2 Dialogue response generation
In recent years, there has been a rise in chatbots are typically targeted at open-domain conversations to engage with users on wide range of topics without a specific goal in mind. However, these chatbots still face challenges in handling task-oriented dialogues where they need to assist users in completing specific goals or tasks, such as making reservations or ordering food 4 22]. Unlike open-domain conversations, task- oriented dialogues often require the chatbot to follow task-specific business logic and respond based on reliable information from API calls or database queries. To address this limitation, we train a small policy model to learn the underlying dialogue policy from the training data and thus guide the LLMs in generating reliable system responses that assist users in completing tasks.
chatbots such as ChatGPT and Sparrow These
Dataset and evaluation We conduct experiments on the popular task-oriente dialogue dataset Mul- tiWOZ 77, including both the Multi O (the original version) and MultiWOZ2.1 version [15]. The dataset provides annotations for user utterances, dialogue acts, and system responses for each dialogue turn. The goal is to generate the system response given the history dialogue context as input. We utilize the dialogue act, which represents the communicative intention of the target system response, as the sseudo-stimulig We only use 1% (80 dialogues) and 10% (800 dialogues) to train the policy model and evaluate the performance on the full validation and test set, which contains 1,000 dialogues. We use the standard evaluation metries: Inform, which measures the rate that the appropriate entity that satisfies the user's 's requirements is provided; Success, which measures the rate that all requested attributes are answered; BLEU: the cerpus-level BLEU score with reference responses; and an overall measure Combined score = (Infor++Success)> x BLEU Likewise, we report the average score over three inferences. We use the same three demonstration examples when using DSP or standard prompting.
for our experiment. There are 8,438 dialogues in the training set.
Supervised fine-tuning details To conduct supervised fine-tuning on the policy model, we format the input of each sample as Translate dialogue to dialogue action: (Dialogue context]", with the target being the verbalized dialogue acts in the same format as (77 63]. For instance, a dialogue act <hotel, inform, choice?, <hotel, inform, type>, <hotel, request, area> will be converted to "[hotel] [inform] choice type [request] area", which indicates that the system should inform available hotel choices and their types and ask for the area that the user would like (see the Appendix for examples). Note that the provided dialogue act annotations may not be the only valid dialogue act for the same dialogue content [77], and thus we hope to explore diverse valid dialogue acts (directional stimulus) through RI. training.
RL training details The evaluation metrics Success and Inform rates are defined at the dialogue level, while the BLEU score is computed on the corpus level. However, our training and inference on conducted on the turn level. We thus use the sentence-level SacreBLEU [5]] score the reward. Same as in the summarization experiments, we generate four outputs per input using the LLM with a temperature of 0.7. The policy network is trained 52k episodes. 5 epochs per batch with batch size of 8 and a learning rate of 2 x 10 6 Since the generated dialogue acts should adhere to the business ensure that the updated policy network does not deviate significantly from
logic and ontology, the original policy model. We thus set the KI aarget and 30 in Equation 7 as 0.2 and 0.01, respectively. During training,
top-k sampling and set i to 50 to explore the action space. During inference, we use beam search decoding with a beam size of 5.
Results We evaluate the impact of our approach DSP on Codex and ChatGPT and compare the performance with several representative task-oriented dialogue models trained on the full training set (8438 dialogues), including DAMD 777, MinTL 34 Soloist [4)], SimpleTOD (21, DoTS 233, PP-
7

---

Table 1: Response generation performance of different methods on the MultWWZZ 2.0&2.1 datasets, where Suce. and Comb. denote the Success and Combined Score metrics, respectively.
#Training data
MulrWWZZ..0
Mult
Method
Inform
Suce.
BLEU
Comb.
Inform
Succ.
BLBU
Comb.
Codex
Standard Prompting
76.7
41.5
7.7
66.8
74.2
41.9
7.8
65.9
DSP w/ SFT
1% (80)
74.9
66.3
11.1
81.7
72.0
66.0
11.3
80.1
SFT+RL
(80)
91.0
76.0
9.8
93.3
89.7
78.6
93.4
DSP w/ SET
10% (800)
79.4
71.9
11.3
87.0
72.0
67.0
13.1
82.6
SFT+RI.
10% (800)
10.7
102.2
94.0
ChatGPT
Standard Prompting
71.8
44.1
10.5
68.4
72.8
44.2
10.4
68.9
DSP SFT
(80)
76.6
11.2
82.8
76.0
64.3
11.3
81.4
DSP w/ SFT+RL.
1% (80)
90.9
82,2
10.2
96.7
87.3
78.7
10.7
93.7
DSP SFT
10% (800)
72.7
64.7
11.8
80.5
75.0
67.7
12.6
83.9
DSP w/ SFT+RL
10% (800)
95.3
82.3
10.9
99.6
95.0
84.0
10.7
100.2
Fully supervised TOD models
DAMD
100% (8438)
76.3
60.4
16.6
85.0
MinTL
3
100% (8438)
84.9
74.9
17.9
97.8
Soloist
49
100% (8438)
85.5
72.9
16.5
95.7
SimpleTOD 211
100% (8438)
84.4
70.1
15.0
92.3
85.0
70.5
15.2
93.0
DoTS 28
100% (8438)
86.6
74.1
15.1
95.5
86.7
74.2
15.9
96.3
PPTOD
(8438)
102.9
79.1
102.3
UBAR 72
100% (8438)
95.4
80.7
17.0
105.1
95.7
81.8
16.5
105.3
GALAXY
100% (8438)
94.4
85.3
20.5
110.4
95.3
86.2
20.0
110.8
TOD 63
UBAR 72). and GALAXY 19]. Table summarizes the overall performance comparison, from which we obtain the following observations: (1) Our approach DSP significantly improves the success and inform rates of Codex and ChatGPT, indicating that they better understand the scenario and generate appropriate responses that help users in completing their tasks. (2) However, there is no improvement in the corpus-level BLEU score, possibly because the LLMs generate responses with different speaking styles and vocabulary since they do not see oracle system responses. Nevertheless, the high success and inform rates demonstrate the usefulness of approach in delivering helpful and reliable responses. (3) Increasing the number of supervised fine- tne-ttunig samples does not guarantee performance improvement, but further fine-tuning the policy model using RI consistently provides performance gains. This suggests that RI training encourages the policy model to explore more preef-preferred stimulus, while supervised fine-tuning may merely generate stimulus closely aligned with the pseudo-labeled data, which is not necessarily optimal. (4) Our approach achieves notable success with only 80 dialogues, surpassing several fully trained TOD models, particularly in terms of Success and Inform rates. With 10% of the training data (800 dialogues), our approach delivers comparable performance current SOTA methods trained with full training data (8438 dialogues). We have also provided the performance of these compared methods in the low-resource settings (1% 1% and 10%) and a running example in the Appendix.
3.3 Slain-of-Thoughe reasoning
While current methods primarily utilize generalized task-specific prompts, LLMs exhibit sensitivity to these prompts. Existing studies [69 26 799 illustrate that the performance of LLMs vary significantly based on the prompt used. Consequently, substantial portion of earlier work has been dedicated to either manually (56) or automatically [61 [99 crafting prompts. However, these studies largely concentrate on task-specific prompts, which may not be optimal for every instance of task. In our experiment, Chain-of- Thoughh (CoT) reasoning. Specifically. we train policy model (t 5-base) to generate instance-specific CoT trigger prompts, such as "Let's think step by step". to prompt varying samples.
employ our approach to generate instance-specific prompts elicit
Dataset and evaluation We adopted the experimental setup from previous work (26 79 where we tested zero-shot CoT reasoning abilities of InstructGPT (text-dav- weavvvvviiiie nci-002) with different trigger prompts. There 600 examples in the MultiArith dataset [57]. which divided into

---

Table 2: Zero-shot chain of thoughts reasoning accuracy (%) of text-davinci-002 with different prompts. *Our approach trains a policy model generate ristance-specifer prompt triggers, which are compared to the spec-fpe prompts in 26 79.
No.
Category
Chain-of- Thoough Trigger Prompt
MultiArith
AQuA
Juuman-Designt
Let's s think step by step.
79.6
31.9
We should think about this step by step.
81.2
28.7
3
First,
78.0
38.2
Before we dive into the answer,
54.8
27.2
Proof followed by the answer.
58.4
37.8
Let's 's think step by step in a realistic way.
59.6
33.9
Let's think step by step using common sense and knowledge.
80.0
34.3
Let's think like detective step by step. Let's ss think about this logically.
73.6
24.0
75.2
34.7
10
Let's s think step by step. First,
78.8
32.3
11
Let's think
56.8
38.2
12
Let's 'ss solve this problem by splitting it into steps.
72.4
33.2
The answer is after the proof.
42.8
34.3
14
Let's 's be realistic and think step by step.
69.6
29.9
Let's 's work this out in a step by step way to
15
APE (79
81.6
34.3
sure we have the right answer.
16
DSP w/ SFT
Geenerrtedd instance-specific prompt)
75.2
35.8
17
DSP w/ SFT+RL
(*Generated instance-specific prompt)
84.0
38.6
300/50/250 for training/validationnnes set. for the AQuA dataset 35 we use the standard test set with 254 samples, 300 samples from the standard training set for our training, and 100 samples for the standard validation set for our validation. We report the reasoning accuracy.
Supervised fine-tuning details For supervised fine-tuming (SFT), first run inference the training set with the 14 human-crafted prompts tested in [26]. respectively. We then selected those prompt and query pairs which resulted in correct CoT reasoning outcome to form the training set for SFT. These query-prompp pairs were used to train a t5- base policy model for 2 epochs, with the model input being the query instance and the target output a trigger prompt.
RL training details After SFT, the prompts generated by the policy model were used to trigger InstructGPT for zero-shot CoT prompting. Reasoning accuracy was utilized the reward for reinforcement learning (RL). A reward of 1 was assigned for correct reasoning results and 0 otherwise. We conducted 20 training iterations (106k episodes), with 5 epochs per batch, a batch size of and learning rate of 2e-6. The parameters for KLauget and Bn were set to 0.5 and 0.001, respectively.
Results We compare the performance using our generated instance-specific prompts with using the 14 human-crafted prompts which we used the pseudo-stimulus to constitute the training set for SFT and also the prompt automatically discovered by the APE approach [79]. Note that all these 15 prompts generalized task task-specific and used for the whole test while ours instance- specific. The performance comparison shown the Table g As can be seen, InstructGPTTs performance varies significantly when using different task-specific prompts. Compared the 14 task- specific human-designed prompts, DSP enhances the performance with instance-specific prompts. also outperforms the prompt discovered by the APE approach, suggesting the effectiveness of our approach for automatically prompt engineering and optimization. Solely relying on supervised fine-tuning of the policy model with the dataset comprising the 14 human-designed prompts doesn't lead to its peak performance. After fine-tuning with RL. the policy model is encouraged to explore better instance-spectle specific trigger prompts, further improving performance.
4 Related work
Black-box large language models Recent years have witnessed the emergence of LLMs such as GPT-3 (6), Codex [9) InstructGPT, ChatGPT [46), PaLM [0), and LaMDA (66), which show significant promise in the field of NLP. These L.MSs typically have large number of parameters and require vast amounts of training data. Due their scaling, these models have exhibited many emergent abilities, such as in-context learning, few-shot prompting,
prompting,

---

and instruction following 66, 46 69 However, most LLMs are not open- open-sourced and can only be accessed via black-box APIs, through which the users send prompt queries and receive responses. While there exist open-source LLM such as OPT-175B 73 and Bloom 558] their local execution and fine-tuning require significant computational resources that may be infeasible for most researchers and users. However, despite their considerable performance on various tasks, LLMS often fall short of generating outputs that fully align with desired outputs on specific downstream tasks and use cases 16. 42 18). Our approach seeks to address this limitation by introducing directional stimulus generated by small tunable LM into the prompt to provide more fine-grained guidance and control over black-box LLMs.
Prompt optimization and engineering Efficiently optimizing pre-trained LMs on downstream tasks by finding optimal prompts has been a focus of prior research. One approach involves tuning soft prompts, which are continuous embedding vectors that can be optimized using gradient descent However, the requirements of gradients and the challenge of passing gradients and continuous prompts through black-box APIs, making them less practical for the black- box LLMs. Researchers have also tried to seek optimal prompts by designing task-specific natural language instructions and selecting proper training samples as in-context demonstrations in the prompt. These methods include manual engineering 150, 6 [6], editing [6] 76), reinforcement learning [3 33]. and automatic generation [79]. Despite these efforts, such prompts are not always effective steering LLMs to generate desired outputs, especially for fine-graine instance-specifie behaviors that are difficult to describe using task-specific instructions and demonstration examples. To address this limitation, our approach is able to provide more fine-grained instance-specifie guidance through the directional stimulus prompts (hints) generated by a small tunable policy model which could be optimized with supervised fine-tuning and reinforcement learning.
methods 32 30 67
2 [
Controllable text generation The control of language models (LMs) has been extensively studied. Early approaches fine-tuned LMs on datasets containing desired attributes [17.. [24) proposed class-conditioned LMs, generating text with predefined control codes. However, direct LM training is
costly. To address this, PPLM 12 trains an attribute model and passes gradients to control generation. GeDi [27] and DExperts (36] use class- class-oonditional distributions as generative discriminators to guide generation, reducing computation complexity. These methods require either additional LM training or internal gradients and logistics, making them not applicable to black-box LLMs. Our approach proposes solution to control black-box LLMs by inserting directional stimulus into the input query prompt and optimizing based on the return output.
Reinforcement learning for NLP Reinforcement learning has been successfully applied to various NLP tasks, such as syntactic parsing [44] 29], machine translation 28), summarization [48, 62]. conversational systems 31,, etc. Language models define probability distributions over tokens in their vocabulary, and the text generation problem can be naturally formulated as selecting an action in an RL setting. Therefore, there have been extensive research efforts optimizing LMs with RL, usually by aligning them with human preferences [80] 70, 40 62). For example, the LLM InstructGPT [46] is optimized with RL to better follow users' instructions and intent. In contrast with these works that directly update the LLMs to align with human preferences, our work optimizes a small policy model that generates text (stimulus) guide LLMs generate more human-preferee output instead of directly optimizing the LLMs, bypassing the inefficient LLM's optimization.
Conclusions and future work
In this paper, we introduce Directional Stimulus Prompting (DSP), a new prompting framework to provide black-box LLMs with fine-grained and instance-specific guidance toward the desired outputs. We use a tunable policy model to generate the directional stimulus to provide such guidance and convert the optimization of black-box LLMs to that of the policy model. Experimental results demonstrate the effectiveness of our approach in controlling and guiding black-box LLMs via automatic prompt engineering and optimization. Furthermore, the generated stimulus provides valuable insights and interpretations of LLMs' behaviors. In this work, we use heuristically selected or annotated pseudo-stimulus data for supervised fine-tuning of the policy model. For future work, we hope to explore the possibility of using "machine language" between the policy model and the I.LMs that might not be intuitively preferred by humans but can better convey guidance information, as well as other forms of directional stimulus beyond text.
10

---

Acknowledgments and Disclosure of Funding
This research was partly sponsored by the DARPA PTG program 1ERR0012220009) Any opinions, findings, conclusions, or recommendation. expressed in this paper are those of the authors and do not necessarily reflect the views of funding agencies.
References
[1] Ahn, M., Brohan, A., Brown, N., Chebotar, Y., Cortes, O., David, B., Finn, C., Fu, C., Gopalakrishnan, K., Hausman, K., al. Do i can, not as say: Grounding language robotic affordances. arXiv preprint 2204v:2204.0169) 2022.
[2] An, S., Li, Y.. Lin, Z., Liu, Q., Chen, B., Fu. Q., Chen, W., Zheng, N., and Lou, J.- G. Input-Tuning Adapting unfamiliar inputs to frozen pretrained models. arXiv preprint arXiv:2203.03131, 2022.
[3] Banerjee, S. and Lavie, A. METEOR: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pp. 65-72, 2005.
[4] Bang, Y., Cahyawijaya, S., Lee, N., Dai, W., Su, D., Wilie, B., Lovenia, H., Ji, Z., Yu, T., Chung, W., et al. A multitask, multilingual, multimodal evaluation of ChatGPT on reasoning. hallucination, and interactivity. arXiv preprint 2xXi:23302.04023 2023.
[5] Barrios, F., López, F., Argerich, L., and Wachenchauzer, R. Variations of the similarity function of textrank for automated summarization. arXiv preprint arXiv:1602.03600 2016.
[6] Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877-1901, 2020.
[7] Budzianowski, P., Wen, T.-H., Tseng, B.- Casanueva, I., Ultes, S., Ramadan, O., and Gašié, M. MultiWOZ- large-scale multi-domain Wizard-of-Oz dataset for task-oriented dialogue modelling. arXiv preprint arXiv: /0/0:/08///007 2018.
[8] Card, D., Henderson, P., Khandelwal, U., Jia, R., Mahowald, K., and Jurafsky, D. With little power comes great responsibility. arXiv preprint arXiv:2010.06599 2020.
[9] Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. d. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N.., Brockman, G., et al. Evaluating large language models trained on code. arXiv preprint arwiv::107..033377 2021.
[10] Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. PaLM: Scaling language modeling with pathways. arXiv preprint ar arXiv:2204.02311 2022.
(11) Chung, H. W., Hou, I.., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, E., Wang, X., De- hghani, M., Brahma, S., et al. Scaling instruction-fintuune finetuned language models. arXiv preprins arXiv:22 10. 1416, 2022.
[12] Dathathri, S., Madotto, A., Lan, J., Hung, J., Frank, E., Molino, P., Yosinski, J., and Liu, R. Plug and play language models: A simple approach controlled text generation. arXiv preprint arXiv:1912.021666 2019.
[13] Deng, M., Wang. J., Hsieh, C.-P., Wang, Y., Guo, H., Shu, T., Song, M., Xing, E. P., and Hu, Z. RLPrompt: Optimizing discrete text prompts with reinforcement learning. arXiv preprini arXiv:2205. /2548, 2022.
[14] Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810004805, 2018
[15] Eric, M., Goel, R., Paul, S., Kumar, A., Sethi, A., Ku, P., Goyal, A. K., Agarwal, S., Gao, S., and Hakkani-Tur, Tur, D. MultiWOZ. 2.1: consolidated multi-domain dialogue dataset with state corrections and state tracking baselines. arXiv preprint arXiv: 1et.:.1070..000 2019.
11

---

[16] Goyal, T., J. J., and Durrett, G. News summarization and evaluation in the era of GPT-3. arXiv preprint arXiv:2209.123560 2022.
[17] Gururangan, S., Marasović, A., Swayamdipta, S., Lo, K., Beltagy, I., Downey, D., and Smith, N. A. Don't t stop pretraining: Adapt language models to domains and tasks. arXiv preprint arXiv:2004. 10964, 2020.
[18] Gutiérrez, B. J., McNeal, N., Washington, C., Chen, Y., Li, L.., Sun, H., and Su, Thinking about GPT-3 in-context learning for biomedical IE? Think again. arXiv preprint arXiv:2203.088 00200.00000, 2022.
[19] He, W., Dai, Y. Zheng, Y., Wu, Y., Cao, Z., Liu, D., Jiang, P., Yang, M., Huang, F., Si, L.., et al. Galaxy: A generative pre-trained model for task-oriented dialog with semi-supervised learning and explicit policy injection. Proceedings of the AAAI Conference on Artificial Intelligence, pp. 10749-10757, 2022.
[20] Honnibal, M. and Montani, I. spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing. To appear, 2017.
[21] Hosseini-Asl, E., McCann, B., Wu, C.-S., Yavuz, S., and Socher, R. A simple language model for task-oriented dialogue. Advances in Neural Information Processing Systems, 33: 20179-20191, 2020.
[22] Hudeček, V. and Dušek, O. Are LLMMs all you need for task-oriented dialogue? arXiv preprint arXiv:2304.06556, 2023.
[23] Jeon, H. and Lee, G. G. Domain state tracking for simplified dialogue system. arXiv preprint arXiv:2/03.06645 2021.
[24] Keskar, N. S., McCann, B., Varshney, L. R., Xiong, C., and Socher, R. CTRL: conditional transformer language model for controllable generation. arXiv preprint arXiv: metp::909900090 2019.
[25] Khattab, O., Santhanam, K., Li, X. L., Hall, D., Liang, P., Potts, C., and Zaharia, M. cemmonstrate-searc--rreeee Composing retrieval and language models for knowledge-intensiee nlp. arXiv preprint arXiv:2212. 14024, 2022.
[26] Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., and Iwasawa, Y. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199-22213, 2022.
[27] Krause, B., Gotmare, A. D., McCann, B., Keskar, N. S., Joty, S., Socher, R., and Rajani, N. F. GeDi: Generative discriminator guided sequence generation. arXiv preprint (xXiv:2009.06361 2020.
[28] Kumar, G., Foster, G., Cherry, C., and Krikun, M. Reinforcement learning based curriculum
optimization for neural machine translation. arXiv preprint 1rXhv:1903.0004/ 2019.
[29] Le. M. and Fokkens, A. Tackling emor propagation through reinforcement learning: A case of greedy dependency parsing. arXiv preprint arXivv 2017.
[30] Lester, B., Al-Rfou, R., and Constant, N. The power of scale for parameter-efficient prompt tuning, arXiv preprint arXiv:2104.08660 2021.
[31] Li, J., Monroe, W., Ritter, A., Galley, M., Gao, J., and Jurafsky, D. Deep reinforcement learning for dialogue generation. arXiv preprint arXiv:1606.015:: 2016.
[32] Li, L. and Liang, P. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00199, 2021.
[33] Lin. C.-Y. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pp. 74-81, 2004.
[34] Lin, 7., Madotto, A., Winata, G. I., and Fung, P. Mintl: Minimalist transfer learning for task-oriented dialogue systems. arXiv preprint arXiv:2009.12005, 2020.
12

---

[35] Ling, W., Yogatama, D., Dyer, C., and Blunsom, P. Program induction by rationale generation: Learning to solve and explain algebraic word problems. arXiv preprint arXiv:1705.044 2017.
[36] Liu, A., Sap, M., Lu, X., Swayamdipta, S., Bhagavatula, C., Smith, N. A., and Choi, Y. Dexperts: Decoding-time controlled text generation with experts and anti-experts. arXiv preprint arXiv:2105. 205.02235,, 03023, 2021.
[37] Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L.., and Stoyanov, V. Roberta: robustly optimized bert pretraining approach. arXiv preprint arXiv: 1692, 2019.
[38] Liu, Y., Iter, D., Xu, Y., Wang, S., Xu, R., and Zhu, C. Gpteval: Nig evaluation using gpt-4 with better human alignment. arXiv preprint 2rXiv:2303.16633 2023.
[39] Lu, P., Qiu, L., Chang. K.-W., Wu, N., Zhu, S.-C., Rajpurohit, T., Clark, P., and Kalyan,
A. Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning. arXiv preprint arXiv:2209. /4610, 2022.
[40] Lu, X., Welleck, S., Jiang. L., Hessel. J.. Qin, L.. West, P., Ammanabrolu. P., and Choi. Y. Quark: Controllable text generation with reinforced unlearning. arXiv preprint arXiv:2205. 2022.
[41] Mihalcea, R. and Tarau, Textrank: Bringing order into text. In Proceedings of the 2004 conference on empirical methods in natural language processing, pp. 404-411, 2004.
[42] Moradi, M., Blagec, K., Haberl, F,, and Samwald, M. Gpt-3 models are poor few-shot learners in the biomedical domain. arXiv preprint arXiv:2109.0255 109.0255,, 2021.
[43] Nallapati, R., Zhou, B., Gulcehre, C., Xiang, B., et al. Abstractive text summarization using sequence-to-sequene rnns and beyond. arXiv preprint arXiv: 2016.
[44] Neu, G. and Szepesvári, C. Training parsers by inverse reinforcement learning. Machine Learning. 2009 Dec; 77: 303-37,, 2009.
[45] OpenAl. Gpt-4 technical report, 2023.
[46] Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744, 2022.
[47] Papineni, K., Roukos, S., Ward, T., and Zhu, W.-. BLEU: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311-318, 2002.
[48] Paulus, R., Xiong, C., and Socher, R. A deep reinforced model for abstractive summarization. arXiv preprint arXiv: AXXv:/705.043000 2017.
[49] Peng, B., Li, C., Li, J., Shayandeh, S., Liden, L., and Gao, J. SOLOIST: Building task bots at scale with transfer learning and machine teaching. Transactions of the Association for Computational Linguistics, 9:807-824 824, 2021.
[50] Petroni, F., Rocktäschel, T., Lewis, P., Bakhtin, A., Wu, Y., Miller, A. H., and Riedel, S. Language models as knowledge bases? arXiv preprint 1909:v1109.00000 2019.
[51] Post, M. call for clarity in reporting bleu scores. arXiv preprint 4rX1v:1804.08771,2011
[52] Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
[53] Raffel, C., Shazcer, N., Roberts, A., Lee, K., Narang. S., Matena, M., Zhou, Y., Li, W., and Liu, J. Exploring the limits of transfer learning with unified text-to-text transformer. The Journal of Machine Learning Research, 21(1))5485-555 2020.
[54] Ramamurthy, R., Ammanabrolu, P., Brantley, K., Hessel, J., Sifa, R., Bauckhage, C., Hajishirzi, H., and Choi, Y. Is reinforcement learning (not) for natural language processing?: Bench- marks, baselines, and building blocks for natural language policy optimization. arXiv preprint arXiv:2210.0122. 2022.
13

---

[55] Reynolds, L. and McDonell, K. Prompt programming for large language models: Beyond the few-shot paradigm. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems, pp. 1-7, 2021.
[56] Reynolds, L. and McDonell, K. Prompt programming for large language models: Beyond the few-shot paradigm. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems, pp. 1-7, 2021.
[57] Roy, S. and Roth, D. Solving general arithmetic word problems. arXiv preprint wrXvv:1608.0113, 2016.
[58] Scao, T. L.., Fan, A., Akiki, C., Pavlick, E., Ilié, S., Hesslow, D., Castagné, R., Luccioni, A. S., Yvon, F., Gallé, M., et al. BLOOM: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100 2022.
[59] Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimiza- tion algorithms. arXiv preprint arXiv: 177.027707,06347, 2017.
[60] Shi, W., Min, S., Yasunaga, M., Seo, M., James, R., Lewis, M., Zettlemoyer, L., and Yih, W.-L Replug: Retrieval-augmented black-box language models. arXiv preprint arXiv:2301.12652 2023.
[61] Shin, T., Razeghi, Y., Logan IV, R. L., Wallace, E., and Singh, S. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. arXiv preprint wxXi:22010.19980 2020.
[62] Stiennon, N., Ouyang, L., Wu, J., Ziegler, D., Lowe, R., Voss, C., Radford, A., Amodei, D., and Christiano, P. F. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-303 2020.
[63] Su, Y., Shu, L., Mansimov, E., Gupta, A., Cai, D., Lai, Y.-A,, and Zhang, Y. Multi-task pre-training for plug-and-play task-oriented dialogue system. arXiv preprint arXiv:2109. 119.99999.479, 2021.
[64] Sun, T., Shao, Y., Qian, H., Huang, X., and Qiu, Black-box tuning for language- model- as-a-service. In International Conference on Machine Learning, pp. 2022.
PMLR,
[65] Suzgun, M., Melas-Kyriazia L., and Jurafsky, D. Follow the wisdom of the crowd: Effective text generation via minimum bayes risk decoding. arXiv preprint arXiv:221 1.07634, 2022.
[66] Thoppilan, R., De Freitas, D., Hall, J., Shazeer, N., Kulshreshtha, A., Cheng, H.-T., Jin, A., Bos, T., Baker, L.. Du, Y., et al. LaMDA: Language models for dialog applications. arXiv preprint arXiv: arXiv:2201.08239, 2022.
[67] Vu, T., Lester, B., Constant, N., Al-Rfou, R., and Cer, D. Spot: Better frozen model adaptation through soft prompt transfer. arXiv preprint arXiv:2110.0771 2021.
[68] Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., et al. Emergent abilities of large language models. arXiv preprint vrXiv:2206.07682. 2022.
[69] Wei. J., Wang, X., Schuurmans, D.,, Bosma, M.. Chi, E., Le, Q., and Zhou, D. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201. / 1903, 2022.
[70] Wu, J., Ouyang, L., Ziegler, D. M., Stienmon, N., Lowe, R., Leike, J., and Christiano, P. Recursively summarizing books with human feedback. arXiv preprint arXiv:2109.10886 2021.
[71] Wu, Y.. Schuster, M., Chen, Z., Le, Q. V.. Norouzi, M., Macherey, W., Krikun, M., Cao, Y.,
Gao, Q., Macherey, K., et al. Google's 's neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint 1xX9:1669.001111 2016.
[72] Yang, Y., Li, Y., and Quan, X. UBAR: Towards fully end-to-end task-oriented dialog system with GPT-2. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 14230-14238, 2021.
14

---

[73] Zhang, S., Diab. M., and Zettlemoyer, L. Democratizing access to sarge-scale language models with opt- 175b. Meta AI, 2022.
[74] Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q. and Artzi, Y. Bertscore: Evaluating text 2019.
generation with bert. arXiv preprint
[75] Zhang, T., Ladhak, F., Durmus, E.,, Liang, P., McKeown, K., and Hashimoto, T. B. Bench- marking large language models for news summarization, arXiv preprint arXiv:2301.38845 2023.
[76] Zhang, T., Wang. X., Zhou, D.. Schuurmans, D., and Gonzalez. J. E. TEMPERA: Test- time prompt editing via reinforcement learning. In International Conference on Learning Representations, 2023.
[77] Zhang, Y., Ou, Z., and Yu, Task-oriented dialog systems that consider multiple appropriate responses under the same context. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 9604-9611, 2020.
[78] Zheng, L., Chiang, W.-L., Sheng. Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., et al. Judging llm-as-a-judge with mt-bench and chatbot arena. arXiv preprint arXiv:2306.05663 2023.
[79] Zhou, Y., Muresanu, A. I., Han, Z., Paster, K., Pitis, S., Chan, H., and Ba, J. Large language models human-level prompt engineers. arXiv preprint arXiv:2211.01910 2022.
[80] Ziegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Radford, A., Amodei, D., Christiano, P. and Irving. G. Fine-tuning language models from human preferences. arXiv preprint 1909 08593, 2019.
15

---

A Implementation Details
A.1
Summarization
We use the representative benchmark dataset CNN/Daill Mail for news summarization dataset contains 287,1 13 training examples, 13,368 validation examples, and 1 11,490 test examples. To keep the API usage cost low, we use a subset of 1, 1,000, 2,000, and 4,000 for training. 500 for validation, and 500 for testing. Each example in the dataset consists of news article along with its corresponding highlight/summary written by human authors. In order to train the policy model through supervised fine-tuning, we employed the textrank [41] algorithm to automatically extract keywords from each article and only retained those mentioned the corresponding reference summary. We initialize the policy model using the 780M FLAN-T5-lars
43 This
model [ 33 and use it to guide the black-box LLM ChatGPT. The hyperparameters used in our experiments are detailed in Table All the experiments a server equipped with 8 NVIDIA RTX A6000 GPUs.
Model Params
Hyperparameter values
Supervised fine-tuning (SFT)
batch size: 8
epochs: 5
leaming rate: 0.00002
leaming rate scheduler: linear
weight decay: 0.01
RL (NLPO)
steps per update: 5120
total number of steps: 51200
batch size: 8
epochs per update:
leaminng rate: 0.000002
entropy coefficient: 0.0
initial coeff: 0.005
target kl: 0.5
discount factor: 0.99
gae lambda: 0.95
clip ratio: 0.2
value function coeff' 0.5
rollouts top
top mask ratio: 0.9
target update iterations: 20
Tokenizer
padding side: right
truncation side: right
max length: 512
Policy model decoding
sampling: True
temperature: 0.7
min length: 10
max new tokens: 80
LLM decoding
sampling: True lemperature:
top_p: 1.0
max new tokens: 180
Table 3: Hyperparameters for experiments on the CNN/Daily Mail dataset.
A.2 Dialogue response generation
The MultiWOZ. dataset is widely-used task-oriented dialogue dataset consisting of 8,438 dialogues for training. 1,000 dialogues for validation, and 1,000 dialogues for testing. For each turn of the dialogues, in addition to the user utterances and system response, the annotations of belief state, database query results, and dialogue act are also provided. To process the data, we followed the approach used in UBAR (72). Specifically, we employed delexicalization by replacing specific slot values with corresponding placeholders. These placeholders can be filled based on the results of a database search. The annotated dialogue acts serve as the stimulus in our approach. Table 4 provides information on all the dialogue acts and slots present in the dataset. We converted the structured dialogue acts, originally in the form of <domain, slot, value> triplets, into text format like [domain J/ inforrm] slotl ... [request] slot [domain2 reqmore). where domains, acts, and slot values are all bracketed.
16

---

We used 780M Flan- Plan-Te-Large for our policy model to guide the ChatGPT and Codex LLMs. During the supervised tine-tuning of the policy model, we trained it to generate stimulus converted from the dialogue acts based the given dialogue context. The policy model was trained for 25 epochs using 80 dialogues from the MultiWOZ2.0 and MultiWO72.11 datasets. When 800 dialogues are given, was trained for epochs on the MultWWO22.0 dataset and 20 epochs on the MultiWOO2.1 dataset. All the hyperparameters setup is presented in T
Table 4: Full ontology for all domains in MultiWOZ2.0 [7] dataset. The upper script indicates which domains it belongs to. *; universal, 1: restaurant, 2: hotel, attraction, taxi, train, hospital, 7: police.
inform" / request" / select 1235 recommend//2) nooffer offerbook /
dialogue acts
offerbooked / nohook welcome" greet" / / reemmoee"
address 12367 postcode 12307 phone 123167 12711 n2me area 123 pricerange"
type 23 internet" / parking" stars?
/ departure destination 45 leave arrive /pepple 128 reference 1235 / price /time department"
slots
day 125 stayy? cartt / food!
Model Params
Hyperparameter values
Supervised fine fine-tuning (SFT)
batch size: 8
epochs: 25/25/8/20
leaming rate: 0.00002
leamning rate scheduler: linear
weight decay: 0.01
RI. (NLPO)
steps per update: 5120
total number of steps: 51200
batch size: 8
epochs per update:
leaminn rate: 0.000002
entropy coefficient: 0.0
initial coeff:
target 0.2
discount factor: 0.99
gae lambda: 0.95
clip ratio: 0.2
value function coeff: 0.5
rollouts top
top mask ratio: 0.9
target update iterations: 20
Tokenizer
padding side:
truncation side: left
max length: 512
Policy 1.M decoding
num beams: 5
min length:
tokens;
LLM decoding
sampling: True
temperature: 0.7
top_p: 1.0
tokens: 64
Table 5: Hyperparameters for experiments the MultiWOZ dataset.
A.3 Chain of Thought reasoning
We use our approach to generate instance -penccfc chain of thought (CoT) trigger prompts. Following
previous work 26 79.. we evaluate two widely used arithmetic reasoning datasets MultiArith [57]
and AQuA 355 We compare with the human-crafted chain-of-thought prompts evaluated (26)
part of which are collected from I (6). We also compare with the prompt automatically designed by the APE approach [79]. We use the The hyperparameters used our experiments are detailed in Table
17

---

Model Params
Hyperparumeter values
Supervised fine-tuning (SFT)
batch size:
epochs:
learning rate: 0.00002
learning rate scheduler: linear
weight decay: 0.01
RL (NLPO)
steps per update: 5120
total numher of steps: 51200
batch size:
epochs per update:
leamning rate: 0.000002
entropy coefficient: 0.0
initial kl coeff: 0.001
target kl: 0.5
discount factor: 0.99
gae lambda: 0,95
clip ratio: 0.2
value fincttion cneff:
rollouts top k: 50
top mask ratio: 0.9
target update iterations:
Tokenizer
padding side: right
truncation side: right
max length:
Policy LM decoding
sampling: True temperature: 0.7
max new tokens; 32
LLM decoding
sampling: True temperature: lop_p: 1.0
max new tokens; 32
Table 6: Hyperparameters for experiments on the zero-shot chain-of-thoughe reasoning.
B
Additional results
B.1
Summarization
Number generated clues/keywords
Keyword precision and summary ROUGE-1
total keywords # hit keywords
0.48
39.9
generated keywords
0.46
0.48
Precision
39.7
0.42
39.6
0.40
39.5
0.38
39.4
Keyword precision
0.36
39.3
Summary ROUGE-1
0.34
(a) Number of generated keywords
(b) Keyword precision and ROUGE-1
Figure 5: Number of generated keywords, keyword precision, and summary ROUGE-1 during the training process 4000 samples.
Analysis of generated hints/keywords We outlined changes in the number of generated keywords, hit keywords (those matched in the reference summary), and corresponding ROUGE-1 scores throughout the training process Figure As the training progresses, the policy model appears to generate keywords with increasing precision, which aligns positively with the increasing ROUGE-1 score. However, observed that even when keywords are generated with high precision if their quantity is too limited, the performance doesn" It necessarily improve.
18

---

POS lagging Distribution
NER Tagging Distribution
Thegtn
29.67%
120ging
122351
16.38% 12630
MARE (903)
ADP 1871
ADI H172)
2.00% (2533)
3.40% 1224)
8.00% 1961
2.3006
00%
ADU 1931
AUX 120)
SYM 7.7% 1461
1871
WORE
POINTI T.
LOC OUTAN (24)
QUANTITY 0.8::3
SCONJ 0.05% (3)
CCON 0.22%
(3.00
PRODUCTT 0.20%
PERNTNTT 1756
LAW 0.10% (2:
FHE 0.36%
LAVOURDE 0.309 (4)
(a) POS
(b) NER
Figure 6: Part-of-Speede (POS) and Named Entity Recognition (NER) tagging on the generated hints, ie., keywords.
We also employed the spacy package [20] for Part-of-Speech (POS) and Named Entity Recogni- tion (NER) tagging on the generated keywords. The results are shown the Figure 6 For the POS tagging, we observe that nouns (NOUN) and proper nouns (PROPN) are the most frequently generated keywords, which can serve as informative keywords. As for the NER tagging, the most com- monly generated keywords include persons (PERSON), geopolitical entities (GPE), dates (DATE), organizations (ORG), and numerals (CARDINAL).
GPT-4 Evaluation To gain better understanding of generated summaries guided by keywords, we employed GPT-4 to evaluate the summaries. It has been shown that the LLM, especially GPT-4 is able to produce consistently high-quality assessments of text generation, showing high human alignment and thus being good alternative to human evaluations 778 38). As we employ ROUGE scores as rewards for tuning the policy model to generate keywords that guide the LLM towards generating summaries more aligned with the reference summary, we leveraged GPT-4 to assess the overlap of key points (hints) between generated and reference summaries. Specifically, to compare the summaries generated with our proposed DSP and the original standard prompting. GPT-4 was instructed to first generate explanation, followed by the corresponding answer. We prompt GPT-4 as follows:
GPT-4
You are provided with an article and corresponding reference summary. Additionally, there will be two alternative summaries labeled as 'A' and 'B'. Your task is to identify which of the two summaries (A B) is more similar to the reference summary. This similarity should be evaluated based the presence and accuracy key points from the reference summary in each alternative summary. Please detail your reasoning an explanation. After your explanation, classify the task outcome as: select 'A wins' if Summary A aligns more closely with the reference summary, 'B wins' if Summary B aligns more closely, or 'Tie' if both summaries align equally well with the reference summary.
The GPT-4 evaluation results are shown in Figure 7. We found that GPT-4 can produce reasonable and detailed explanations their assessment. From our test set 500 samples: DSP-genenan OSP-generated summaries were favored 255 times (51.0%), summaries generated with original standard prompting were favored 222 times (44.4%), while a tie was observed in 23 cases (4.6%).
Zero-shot prompting In our main experiments, we employ few-shot prompting with 3 examples in the prompt during training and evaluation. The specific prompt and demonstration examples utilized are detailed in Appendix D To test whether the approach performed well under the zero-shot setting, we evaluated the following two experimental settings on the CNNDM dataset with 4,000 training samples: (1) few(3)-shot during training and zero-shot during evaluation; and (2) zero-shot during both training and evaluation.
As shown in Figure 8, when both training and testing are conducted using zero-shot prompting, the performance improvement over standard prompting is still comparable to the scenario where both are conducted using shw-shot prompting. In addition, we observed that our approach exhibits
19

---

Standard Prompling
DSP (0-shot)
DSP (3-shot)
GPT-4 Evaluation
ROUGE-1
1522
ROUGE:
DSP wins
51.0%
4.6%
14
24
0:5
91.20
1:2
METEOR
BERT Scure
BLEU
0.88
Original wins
0.875
Figure 7: GPT-4 evaluation comparing Figure 8: Zero- shot evaluation results. DSP (0-shot) the summaries generated with our approach denotes that we use O-shot prompting during RL DSP, ie.. with the guidance of gener- training and DSP (3-shot) indicates we use 3-shot ated keywords, and the original standard prompting during RL. training. prompting, i.e., without keyword guidance.
Table 7: Low-resource evaluation on the MultiWOZ 2.0 dataset, where Succ. and Comb. denote the Success and Combined Score metrics, respectively.
1% of training data (80 dialogues)
10% of training data (800 dialogues)
Method
Inform
Suce.
BLEU
Comb.
Inform
Succ.
BLEU
Comb.
DAMD
77
34.4
9.1
8.1
29.9
55.3
30.3
13.0
55.8
Soloist
49
58.4
35.3
10.6
57.4
69.9
51.9
14.6
75.5
PPTOD 63
74.4
52.4
13.0
76.4
84.4
68.4
15.6
92.0
UBAR 72
82.5
66.6
17.7
92.3
GALAXY 19
90.0
75.9
17.5
100.2
Codex
Standard Prompting
76.7
41.5
7.7
66.8
76.7
41.5
7.7
66.8
DSP w/ SFT
74.9
66.3
11.1
81.7
79.4
71.9
11.3
87.0
DSP w/ SFT+RL
91.0
76.0
9.8
93.3
96.0
86.9
10.7
102.2
ChatGPT
Standard Prompting
71.8
44.1
10.5
68.4
71.8
44.1
10.5
68.4
DSP w/ SFT
76.6
66.5
11.2
82.8
72.7
64.7
11.8
80.5
DSP w/ SFT+RL
90.9
82.2
10.2
96.7
95.3
82.3
10.9
99.6
robustness when different numbers of examples are used in prompts during training and evaluation, as our approach with few(3)-shot training still outperforms standard prompting under zero-shot testing.
B.2 Dialogue response generation
Low-resource results In addition the performance compared baseline models with full training data as shown in the main paper, we also present their performance in the low-resource setting in Table 7 is important to note that most of these methods struggle to achieve acceptable performance with only 1% of the training data (80 dialogues), and thus their results in the 1 % setting are not reported. As for those with reported performance with dialogues, their results are significantly worse compared to Codex and ChatGPT guided by the policy model. Furthermore, even with around 800 dialogues, their Inform and Success rates were still much lower than those achieved by ChatGPT and Codex.
20

---

B.3
Chain-of-Thoughe reasoning
Table 8: Some generated trigger prompts by our fine-tuned policy model.
Generated CoT Trigger Prompts
First step:
Let's think like a detective step by step. First, Let's solve this problem by splitting it into steps. First,
Let's ss think step by step using common sense. Let's think step by step using our creative brains.
Let's think step by step using both the above information and the testing.
Let's think step by step using proven methods.
The answer is following the proof.
Newly discovered prompts After fine-tuning with RL, the policy model is encouraged to discover instance-speciff trigger prompts. Table g showcases some of these newly generated trigger prompts, which deviate from those present in the training data for SFT. Some of prompts from the training data, such as "First step:" and "Let's 'et's think like detective step by step. First," " Others include new information, like "Let's think step by step using both the above information and the testing." and "Lef's think step by step using proven methods.".
modifications combinations
C Running examples
We provide two running examples on the CNN/Daily Mail and MultWWZZ dataset in Table respectively. For each example, we present the generations of ChatGPT with standard prompting, DSP trained with SFT, and DSP trained with SFT and RL.
D Prompts
The used prompts of standard prompting and our proposed Directional Stimulus Prompting CNN/Daily Mail and MultiWOZ datasets are given in Figures 9 I0, and Figures II [2
respectively. Both use the same three demonstration examples in standard prompting and DSP. In the case of the CNN/Daily Mail dataset, DSP incorporates additional keywords as hints (stimulus) in the prompts. For the MultiWOZ dataset, DSP includes the dialogue acts for each system turn stimulus, along with explanations for all the dialogue acts.
21

---

Input article
The winter of 2014-15 15 won't be casily forgotten in Boston after the endless snow broke countless records and the city had to pay volunteers $30 an hour to help dig out the battered city. The shere volume of snow that fell earlier this year, nearly 65 inches fell in February alone, means that huge piles of the white stuff still remain. Except the remaining 'snow" isn't t very white any more hut rather disgusting black color riddled with trash including broken pieces of glass, plastic shards and goodness knows what else. Scroll down for video. Vlad Tarasov couldn't It resist filming himself ski down the slopes at Boston's largest snow farm located in the city's 's Seaport District. . The one-minute video gives first-person perspective of pushing through the filthy, trash-filled ice pile that served dumping ground for the snow. To some avid skiers snow is still snow and one in particular couldn't 't resist the urge take to the slopes of Boston's temporary new resort. Vlad Tarasov even filmed his journey down the slopes at Boston's largest snow farm located in the city's Seaport District. 'I've been skiing for 20 years, but never like this,' he told The Boston Globe about the 'surreal' experience of climbing the slopes April and looking down the South Boston urban sprawl. The one- minute video gives viewers first-pert fist-ppesoon perspective of the experience as Tarasov pushes through the filthy, trash-filled ice pile that served as a dumping ground for the historic winter snowfall. Tarasov recalls having to avoid junk including rusted lawn chairs, parking cones, broken bottles, and 'pretty much every kind of trash you's 'dd ever find on Boston street.' He hit a dead seagull, he said. 'I've been skiing for 20 years, but never like this,' said Tarasov about the 'surreal' experience. Junk in the filthy snow included rusted lawn chairs, parking cones, broken bottles and even a dead seagull. 'I'm so accustomed to sking on the white stuff. I was definitely concerned once I climbed the mound about falling debris,' said Tarasov. He has no plans to repeat his daredevil stunt as the warm weather is now finally melting the snow farm and he doesn't want have to clean all his gear again. Boston received 108.6 inches of snow about 9 feet during this last winter, beating the previous seasonal record of 107.9 9 inches that was set in 1995-96, 96. The 65 inches that fell in February crushed the previous one-month record of 43.3 inches in Jamuary 2005. The city of Boston received 108.6 inches of snow about 9 feet- during this last winter, topping scasonal record of 107.9 inches that was set in 1995-96.
Reference sum- mary
Vlad Tarasov couldn't 't resist filming himself ski down the slopes at Boston's 's largest snow farm located in the city's 's Scaport District. His one-minute video gives first-person perspective of pushing through the filthy, trash-filled ice pile that served as dumping ground for the snow. ''ve been skiing for 20 years, but never like this,' he said about the 'surreal' experience. Junk in the filthy snow included rusted lawn chairs, parking cones, broken bottles and even a dead seagull.
Standard prompting
Generated summary (ChatGPT)
A video has emerged a skier named Vlad Tarasov skiing on pile of snow in the Seaport District of Boston. The pile of snow was where the city put the snow that fell during the bcord-breaking winter scason. Unfortunately, the snow is now a disgusting black color riddled with trash, including broken pieces of glass, plastic shards and other debris. 37,58
ROUGE-1
Directional Stimulus Prompting (SFT)
Generated key- words (TS)
Vlad Tarasoy; 65 inches; Boston; Scaport District; the city; $30 hour; volunteers.
Generated summary (ChatGPT)
Vlad Tarasov filmed himself skiing down the slopes of Boston's largest snow farm located in the city's 's Seaport District, which served as dumping ground for the historic winter snowfall. The shere volume of snow that fell earlier this year, nearly 65 inches fell in February alone, means that huge piles of the white stuff still remain and the city had to pay volunteers $30 an hour to help dig out the hattered city.
ROUGE-1
46.91
Directional Stimulus Prompting (SFT+RL)
Generated key-
Vlad Tarasov; ski; Boston; large snow farm; Scaport District; surreal; video; filthy; snowfall.
words (T5)
Generated summary
Vlad Tarasov couldn't it resist filming himself skiing down the slopes of Boston's 's largest snow farm located in the city's Seaport District. To some avid skiers, snow is still snow and Tarasov said it was 'surreal' experience. The one-minute video gives viewers a first-person perspective of the experience as Tarasov pushes through the filthy, trash-filled ice pile that served as a dumping ground for the histaric winter snowfall.
(ChatGPT)
ROUGE-1
71.25
Table 9: An example from the CNN/Daily Mail dataset. The policy model (TS) generates the keywords based on which the ChatGPT generates the summary. The keywords that are mentioned in the reference summary are underlined and highlighted in red."
22

---

User:
want find moderate -ly priced restaurant.
Assistant: have (value_choice) available you there certain area or cuisine that interests you
Dialogue con- text
User: yes would like the restaurant to be located in the center of the attractions. [db__]
Oracle dia-
[restaurntt [inform] choice [request] food
logue acts
Oracle sys-
there are [value choice] restaurant -s available in the [value area] how about specific type of cuisine
tem response
Standard prompting
Generated re-
the [value name] is located in the center of the attractions.. would you like me to book a table ?
sponse (Chat- GPT)
SacreBLEU
14.09
Directional Stimulus Prompting (SFT)
Generated dialogue acts (T5)
restaauant] [inform] food name area choice [recommend] price
Generated re- sponse (Chat- GPT)
would suggest [value_namel in the [value_arces area. they have [value_fooee cuisine and are moderate in price are you interested ?
SacreBLEU
16.40
Directional Stimulus Prompting (SFT+RL)
Generated dialogue acts (T5)
[estaauaat] [inform] choice [request] food
Generated
have vaaue__hhice restaurants in the area. do you have specific cuisine in mind
summary
(ChatGPPT)
SacreBLEU 22.80
Table 10: An example from the MultiWOZ dataset. dataset. The policy model (T5) generates the dialogue acts given the dialog context. With our approach DSP, ChatGPT generates the response conditioned on the generated dialogue acts.
23

---

Standard Prompt (CNN/Daily Mail)
Givan aride, wrlle summary 2-4
Aricle: Secul (CNN) South Korea's Prime Minaser Wan koo offered to resign an Monday amic growing poliical soandal, stay transterred role chairing Cabinet moelings to deputy prime minste ster the time South Korcon President Goun hye accepts his resionation calles "regretiatle"" soctrding being, scronning
until
resign bogan after South Korean tycoon Sung Woon jong was found hanging from a tree in Secul an apparent suxidd on Agri Sung, who was under inwestigation for bribery, mote lising names amounts of cash gien top officials, including those who work for Fresident. and seven other paliiclara with links Bouth Korean President under investgation. special proscoutor's leam
resignation
preaedential
been estabishee investiate case. Lee adamandyy denied the allegation as the scondal escalated: "If there are ony evidence. will give cut my life. As a Prime Minister, will accept Prosecutor Office's Investgation first Park has aed ina ahs leking the accusations very serinusly Baloow depaiting on her trip to Central and South Armrico whe condermeed pollical compution in country. "Comuption and deep-rossed wvil are issuis that can forgtre anyone who responsibee comuptioon wrongdtingg Koma Apri 27, CNNFs Pasle Himcooks contributed this report
taking awey peoplel's liee, We lake this very sarimualy" "We muad make surs to set straight this ieru expested arrve
as matter pottiaal reform.
South
Witt anort summary of the arricte 2-4 senterncss
Calls for Lee Roo resion began after South Korean lycoon Sung Wcan jong was found hanging a in Socul Sung, who was under investigation for froud and bribery, left a note listing names and amounts cash given to top offic alss.
over amalts deceased woman to griaving husband, who believes they prove she bullicd by the Corporations i management it towords the end of her Iff. Mother of swo Marie Cooszar, 45, died last Sepcember following lonn year bottle with brain tumour: Sho has worked for seven years at the BBC's lirancal contre in Caroff as a contracts monager, but according to her husband Paul she was forced out of the post into wnother jub wfler drewing atlandin lo managammnt blanders which hhe says coul licar your buttle
Artile: BBC has refused
payers about 000. Whidselown Marie Casseie died wfter 10- widower has had request her work emall, which beleeees show she was baing bulleed bosses, refused by the BMC. I Lggal eogerts describbed the case as "righly anuuusl bit rted that mould be in fnllowd by simler came as dight decuments auch su emalls and social media posts play an inoreas nglly important part in people's lvee. Facebook users the US have the chance to designate a leggoyy contact who can take over parts of thei ascount after their death Mr Crassor says he belleves the BBC falled in its duty of care to his late wite, and the trestment she rece ved from managoment
brain lumour.
attected her health. He asked the BBC under the Data Protection Act for coples his whel's is emals, the hope they will provide evidece her 'oppallingg treatment
by the Co Corporation, which he has spent months pursuing. However ever, he was sold lost monch by the BBC that under the Act, personal data is dufined as only 'data
which relatos a lvnng indivddal. Despite Mr Csaszar baing his wif''s logal nosd of kin an omail from the BBC - seen by The Mail on Sunday read "Unfor anfortuntlly
your wife has sodly passed away not possible for you exercise the right access her perconal data under the Data Protection Act Mr Csaszar said his
wife gave eviderica to the BBC's a 2013 Respect Work review that wes set up to look into sessem and builying Eeltwing the Jinny Saci apandall He said it was her usidence which led the chainwomen, De Rose GC, to condute some senior Corpont poration stalf were viewed as bniouchaidee Detail of the iddviduala who were the supject of Mr Conszar's complant connot be published for legal reasons but they involve at least one sertor manager on a str-igure satory. Last right, sot-
em ployed businessman Mr Csaszar, 48, sakt: Marie was being ham mered from one side the other sowards the end of her Ho. Nobody stood and objected. Locked out The BBC has infused to hand over the work emeils of Mari Caserar tn her wdhnwrr Paul "The Insemmnnt which Mann eccerienped at the hards of the BSC affected her recovery. She was corstantly upsel and bill lears, sayang "I contt know what I've dooe to deserve being treatedd like liis." "Thyy dián't want
Maria on the staff because she knew too much about what had gone on. She would refese to sign off caraain contracts because they didntt show value for money. but they were signed off anyway This caused her great deal of strees Mrs Caaazar underwent intersive treatment for her cancer over several years, which left her autwwiigg with aptlpey. Howaver, als continued in work in her 150 000aa-eer post, and resed serious monerns that coud have saved lieenpades payers
money, ter husband cleims, In formel grievence propedure with the BBC in 2015, Mix Caeazer reieed compleint about the closure a cash oflice by outside contractor. She was advised by the BBC's legal cepariment that the corporation was not kable under its contrctt pay any money tawards redundancy for statf smp oyed by the contractors. But she told her husband that about E35 000 was pald out anyway. Mr Ceaszar says he is desperate see er enats reterning his wife and now she was treated fallowing ni her wiwii He contacted the Information Commissionns ls Office, only to be told that the relesse of the documants wi the discriiion of the BBC. Mr Caserer resed his request with the BBC Trust, the Corponation's is watchddgg, and consedering lugat wctionn A BBC hevr
Caaazar's death, thorough sisiiigtiinn wes centied into her compleint and the autaome was provided to her in July 2013. We have proviced this response her husband. Mr Csaszar then attended work from October 2013 to May 2014 and dic not raise a further appeal complaint" Wriee summary of the arlicle 2-4 senteross.
spokeaman awid: We were
As Mosher of two died tollowigg 10 yoar bottle with a brain tumour She worked at BBC Cardif where husband claims she was bullied Marie Csaszor gave Protection Act but retused.
evidence BBC's
Respeen work
He asked
under
Artice: (CNN) An American clizzen was wounded by gunfie Thuraday she drove from the medical school Karachi, Pakisdaan where she works, police seid. Dubra Lobo, 55-year ald Calffrnia native was shot in the right cheek and lift arm and is unconsoleus but expected to survive, according to Monamad Shah, Urdu,
should buml," Shah said. had lill the Jiresh Medical and Dantel College, where sll works vide princpell pisk up two deughters from achool. Two axoullarts pessing motoreydl shat her wiike she was driving, Bhah said. "Our U.S. Consulate Gemeral in Kanachi crose contact with Pakistari author liss and is working chiain said Shah. the Karachi's Khan invettgating, Shih sold.
Karachi police spokesman.
found pamphttts
assollants had thrown into Lobo's
information, said U.S. Embassy spokesperson. ed Chrislinn Pakisteni who lbreran at American School Kerschi Karcchi Witte short
lived Pakistan since 1998
of the wrficle 2-4 sentencee.
expected
being she shot.
polic say. She printipel the Jinnah Medical and Dental Collage
Kanachi. Polise
sick up daugtrers
Artidee STIONS Wiite summary of wrticle 244
Figure 9: The prompt for standard prompting on the CNN/Daily Mail dataset.
24

---

Directional Stimulus Prompt (CNN/Daily Mail)
Given srtilee
heywords,
summary that acouralaly inseppratts the provited keywords
Articl:: Seoul (CNN) Korca's Frime Minister Lee Wan-koo offered to resinn on Monday ami growing poliicall scandall
stay
role unill
South Korcon President Park Goun hye acceses his resignatin. has transfered his role chaliing Cabinet meelings the deputy prime minister the time boing, according to his office. Park heard socut the resignation and colled it "regretable. according to the South Kerean presicental office. Calls for Lee to resign began Bauth Korwan lysson Sung Wecnsjong was found hanging from bee Baodl inwesiguiion flaud
appament suidde on Apri Surg. was under brbbar. left a note lasing names and amouniss of cash given to lop officials, incudding those who work for President, Lee and seven other poliicians with inko to the Eouth Korean President are under invesiiaation A special prosecutor's leam has been establishe to investiate the case. Lee had antemently decied the wilegalioss ax the soendal exceatted wledd "Il theme ane any maatenne will gove out my has said that ahs lsking the actusssions very serizusly. Before depariing her trip Central and South America, ahe condarned polticall comupton in her country, "Comuplion and desp-moled avil are issues that can lead to laking away people's lvee. We take this very sernusty. "We must make sure set straight this lssue as a matter at politicl reform. will not targie anyone who responstate for compution or wrongdotigg * Park is Pen: and is expected to arrive back South Kores on April 27, CNN'ss Pauls Hancooks contributed to this report.
Prime Minider, will wocept Prosenutor
Office's investigation ferl."
Or Write short summery of the artide in 2-4 serienoss that soourallly incorporaies the provited keywords.
Keywords:: Lee Wan koo resign, South Korean tyceon: Sung Woan jong: hanging from a free: Investigation: notos; top officats.
A: Calls for Lee Wan-koo resign began after South Korean tycoon Sung Wean-jong was found hanging from a tree in Seoul, Sung, who was under investigation for fruud and arbery left a note listing names and amounts of cash given to top officiate.
The BBC has refused hand over the smalls of decessed women to her griwving haaberd, who belavees they will prove ahe was "bullied" the Corporaten's is menapement lowards the and of her lise Molher-af-loo Mari i Csescer, 45, died last Beplember folkwing lan-puar battle with brain tumoua She had worked for seven years at the BBC's fnnacial centre Cardift as contrccts manager. but according to her husband Paul, the was forced out of the post into another joo after driwing amtntionn to management blunders which 1oyyeer betlle with brain tumour Her widnoer has had regued her work smalls which bellows will show whe was being bulleed bosees, refused by the BBC Legai experts desoribed the case as highty unusual, but predicted that it could be followed simiar da dams as digital documents such ema is and cocial media take parts their ply increasingly important
Artidw:
cost icencedes
socut E150 pio. Whadeatrower: Marie Canarar died after
lives Facebook says believes the BBC tailed cap
the US the
designate "egney contact mecived
atter that death, Ha appa trealment by Conporation, which he
to
wite, the
emails, told last month by BBC they onty data mistes living indviduall. Despite Mr Cearzar his wie''s legal next ih, Protecton Act Untorturately your
manegement
BBC months pursunn,
Protection
personal data
rghtt look pensonal data The Respect Work conolude
the BBC
said wife gave sady passed away is poestble
to
the
Jimmy Savil
He said untouchable" Detalls of the indiidualss who wore subject Mr Ceaszar's compla cannot published legal reasens but they invotre at least senter
her evidence which led chalwoman, Dinah
Corporation
viewed
48, meesgeer as-igure selary. Lest night, self-amployee The BBC has refused objected,
one
her lo. Instmmett whish Merie roparienoed at the hands done deserve being treated like this"" They dion' want on staff secause she know too much about what gone great deal stress. Coaszor underwent intensive trastment for ner cencer over several years, which left her suftering with eptapey. However the continued to work in her 150 000-000--- sost, and raised serious concerns
her BBC ellected her monoery. was consbently upsat and lears, saying don't know
'Tee would refuse slan off
certain contracts because they didn't show value money. they were signed anyway. caused
could have saved loonce-ce payers' money, her husband damss. formal grievance procedure Osaszar raised comp aint about the closure of cash office run by an butside contractor. She was advised by the BBC's legal doparment that the Corperation was not labll under is contrct to pay any money sowards redundancy for staff amployed by the contractors. But she told her husband that about E35,000 was pail oul anyway. Casszer says he is desparaie to see amails laring to wife and how ahe was brssied fallocing
BBC in 2013,
whisleblowing Hu contected the
Information Commissioner' a Office, only to be told that the release of the cocuments was lat the discreson' of the BBC, Mr Caaszar has raised his requeat with the BBC Trusi, the Corporation's wanchdog, and is considering legal action. A BBC spekesman sake: We wore very sorry to hear of Ms Csaszar's death. A thomaugh invesliaatio was carned out into her complant and the outtome was proviced to in Juy 2015. We have providee this response tn her hustand, My Canscer then allended work from Detober 2013 to May 2014 and dil not wise further sppeal to her complein). O Write a short summary of the aride in 2-4 sentenows Bal acouralely incorporalas the provited keywards.
Keywords: Mother, battle brain tumour; BBC: Cardit hutband: bulled: Mare Csaszar: evidence: Respect At Work: emalls: Data Protection Art A Mother-al-ten died followng a 10-yaar battle with brain tumour She worked at BBC in Cardif where husband claims she was bullidd Marin Canszar gave evidence at BBC's 2013 Respect At Work redeew He aoled for emals under the Dats Prolection Act but was relused.
Artide: (ONN] An American ciizen was wounded gunfre Thursday as she drove from the medical school in Karachi, Pakistan, where she works, polios said Debra Lobo, 50-year-odd Callorsia native, was shot in the right check and left arm and unconscauus but expected to survire according Mohamad Shah, a Urdu,
Karachi police sposesman,
pamphiets that the assaliants had thrown into Lobe's
"America should burn." Shah soid. her daughters from school. Two sosallatt
Lobo had lill the Jonain Medical and Derial College, molorcydle ahot her while she was driving Shen sedd "Our U.S. Consulate Gemeral in Kanadii clus contact with Pakiaaani authorties and is warking to ablain informatio,"
works vice printipel, la
Karachis Khan Hospital said Shah. has Pavistan married Chrddta Paccstent who librarian the American School Karach Karachi
Embasay spokesperson.
treated
1966 Q: White sbort Deora Lobo, 55, unconedions, Thunddyy principe, the Medisal poliee
investigaling Shah sald.
of the artide 2-4 awmennos wopurately insorporttes the provited keyworde.
Dollage, Karachi, dasugheess school.
Debra Labo. unconscious but expected College Karachi, Police: She
shot her daughters school when
principal Jinnah Medical and Dental
her way
Artile:: [QUESTION]]
Write summary of the article 2-4 sentences acourately incorporates provided keywords.
Keyworde: [JHNNT]]
Figure 10: The prompt for Directional Stimulus Prompting on the CNN/Daily Mail dataset. The difference compared with the prompts used in standard prompting shown in Figure iss is the stimulus hints (keywords), which are highlighted in red.
25

---

Standard Prompt (MultiWoz)
The following are conversation between user assisiatt. assislant can the booking provising Information restaurant, hotel, attraction, lax,, train, hospital, police and so on.
Converselion:
User do you book trains the bichaps storfford
Aesistani: yes, you deperting g amiiing in Dolon__detinstion
am going there
cambridge on thursday.
Assistant: have (oake chokes] truns going in (vatue deattittittnnii from cateiittddeprrtuueee on [ualua_day what time do you need arrive
User would lkk artive by 18:15.
Aesistant: (vauue di have you (ealuve destnation by /value_aniin
sokets...
Assistant: the (velue [d] will get you there by /value antrel iI can book you (value. people] lckeet on that if you would like?
User tickets be fine, plasse
Assistant: 70277
total (value price) payable at the staton. reference number (valvaee referencee) will be anything else can help
you with today?
User nice place to eat that sarves austrian food in the cantro of town if possible.
Aesistant: am so sorry Ivalue food] restaurant is that area another food type perhops??
User ekay. de you have any good restaurant serving britsh food
there are baale shoice] restaurant is in the (value, aree]] that senve rellue_foodl food, is there a price range you would like to stay in User any is fine, can get the phone number and postcade of you would recommend ?
Assistant:
Aesistant::
would recommend the jealue, name) their phone number is (ralue_phone) their poetoode [ralue_postzoe]
User great thanks. that all need. goodbye
Assistant: awesome, we 're here 24 and If you need us again. have great day!
Convensation:
User i' looking for a hotel in the moderate price range definielly want a hatel not a gueet house
Assistant: of course. there are fvalue choice] (value type] in the fvalue. areal and (valus, cholce) the Ivalue area] of town co you have preference preferably with
User there anything the
reting ? Adsistant: in sony but nothing exits in (vallso amaj) north withuuale,, (valua, starsj 4 star rataing in |sslue_ppic moderato prics rango. could we by anothar araa
User there in the cheap range??
Assisbent: these not
User ph goodnass can you check once again for hotel star 4, in the weat cheap is fine..
Assistant: the (value, name) loeated at jontun adoress] do you nead mom information or would you like to book now
User yeah book for people..
Assistant: for how
days??
User will this start on fiday?
Acsistent: yes would you like issevations 7 what day willl start how many people ?
User would lik reservation for people for4 rights seginning riday, at the samarddge beltly. (vaiue reforonco).
Assistant:
baoked your reservation
User thank you for the reservation.. am also looking for lovely park north cambridge.
Assistent: would suggest (vstus _name)
User what
Assidaant: the phone number |value. phone)
User: alsoo need laxd between the place -4..
Assistant: please
departure location and antival location.
30
Assidiant: what time do you need to loave the restaurant by?
User:
the hotel 04:30
Assistant: booking
you pickid up (value [valuo clepartura) off
destination)
User do their contact number?
Assistant: their contact numberis ohoneel there anything else can you today. thank you. goodbye
Assistant: you'l" gladi could help. goodbye.....
Conversation:
went
the centre that type attractons..
Assistant: think will leee
is sounds please provide the posicado.
book for
Assistant:
you sony. we are
book tickets for this atraction, their postoodl (value postoode) db21y, can provide their address and phone
need. would like help fincing
house though.
have baaue choice) options you. you need parking ? part you stay in
indiudo
should in
Assistant: there seems to be an sponing at haaue name!. would you like me to book you a room
yeu. plasse penple nights starting thursday.
Acsistant: booking was successful your reference number wralve_refeeenne
User thank you very much!
Resistant: you welcome. there anything
for you?
no
appreciate your help.
Assistant: thank you'l gladi could help.
Conversation:
[DDALOG]
Figure 11: The prompt for standard prompting on the MultiWOZ dataset.
26

---

Directional Stimulus Prompt (MultiWoz)
The followigg conversation betwoon and assiatant. Indicated by the dialog acts, the assistant help the usur with booking or providing information |iestaurant, (hotet) [straction] [mea] [sain], (hosp unl (police and so on.
[nform): provide information for trequest request information slot: (nooffert: not offers that satisty requirements: (recommend): recommend
offer; [select]: select offer: (offerbookk] help book a [offerbooked]: have booked an offer [nabook]: can not book offer: (byaa] byw: [grwed]: great: [wulcome]: welcome: [regmorea] aak if need more information help.
Conversation
User de you bock trains for the bishopo stortiord?
Assstant[tiaan] [reques]] departure destlnation: yes, are you deporting or ariving in (volue, destination? ?
ien gaing frw foom cwmbrragg dage on lhuunday.
Assistt ut]inin [pifom] day chocks depasture e deaiiaatin (sequesd] antive) have (owkau choccal Insins going (valuw disfirttionn from (valow departurr) on (oakue day)
time do you need
18:15
Acsistant[Brainl [inform] destiaaton arrivel:
have (vakue destination (value anive))
Uieer need 2 ledeate
Acoritee (ijtrain) [inform] arrive [offerbook] pa optla: the foskoe id] will you thene by (tokuw_ jesiio)) lon book you (owkue, peoplep fiskets on thaet if you would like?
User tollets will be the. plaase
posstartijtrain] (otferbooked] reference price [peneral] [reomone]]: 70277 2 sokets. the total the |vatue_ patuue_roooej_eooooo payable at the station reference number is: feauue rufaraned] will there be anyling slse can help you with talay
User rice place out that serves austiian food in the centre lown if possible.
Monstriilrastaurann [noofter] food (request] food]: am so sorry no (vauun food] meteoumnt that amss ancther food tyoe pertapp
User okay, you have any good restaurant serving brissh 2
Nesstartjpeztturmml [inform] choice area (request) price): are tvalue, choico) restourant the (valao, arca] that serve (valuo, food] food, is there price
ge you
gei
posioode
would recammeed??
ponntartjpnstuuran [recommend] postcode phone
would recommend the [value_rame) their number [vanue_phone) postoode
Jeatue, pretoode]
User great I thanks. that all need goodbye.
Aesistatilleneeal [bye]] awesomo. we tre here 24 and 7 you need us again. have great day!
Convensation:
User m looking a hotel the moderate price range definnoyy want a hotel, not a guest house
Assistar ii]hotel [inform] area choice type (request) arealy course there are (valus cholol) (vukuo lypa] the (salue, area) and (vwlue, choice] the |valux, smoa] of co you have a proforencs?
anything Aossseetijhatel] [nooffer) price (request] area): sory bur nothing
prefernely
rating?
(swaeaame [as]] with pelia_sstonn moking porlon
range?
Acsinata [i[hotel] [nooffer]]:nn Juae
User oh goodness can you checs once again for hotel star in the west cheap is thee
buorsariijhotei [Intorm] name address [ofterbook) [general] [reqmore]]: the jentoe_ namel locaved ot taatue_ sddrrren) do you need more information or would you like to book now
Usor yeah book for people.
Apoaseell (request]] stayl: how many daya?
will his stwrl on fiday? ?
hesssart[jhoee]] [ofterbook] [request] people day': yes, would you like resonvatios day will it start how many people
sishoatt[[holl] reservation people rights, beginning triayy the cambridge belty.
your User thank for reservtiion am siso looking lowely park norhh cambridge.
(boluo referencc))
Aboesteetijaeen (ijetrecciiiinon [recommend] namel: would suggest (owoss (eeme)))
User what the phone number? ?
posttarljpttraaoon (inforn) phonel: the phone number is Jealue phhnn] User slso need isci bellwen 2he2 place
Assistarii [requesi] departure leave destination:: please give me the bavel time, doparture location and artival location..
User want to leave hotel by 06:00
Asoistel((brel) (request]] leavee what
you need foown the resturent by?
User want piskup at the hotel by destination): great your booking
Aooisteti[bbe)) [inform] departure
you will picked in |esdan por] by (ustuw (soco)) (undue_ deeppeturn
dropues
contact number?
[general] [reamone]]: their contact number [vatue_phonn]... anything you with today? User that will be for today. dhank you goodbyo.
Aosntarti[baet] [inform]
Mosistanilgeeeraj [byejj: have great day byo.
Converedions
User wont to and piace to go in the centro of town that are part type attractions Assista [[jetraccion] (inform) price name [offurbook]]: lhink that you will live Joaker, name] is [owlau, prical, may book it for you
User you that sounds good pluase provida me with the poakcoda.
Aesnehttipttractonn (nobook] [Inform] postcodaj:: I' sorry are phone would liee holo fincigg parking Interneti: have bollue cholco) options
tickets this attraction. their postoode tpttle_ppetttteeee poetrodde c521)
thatis
though.
Mosstant[hooeel [inform] choice (request] stay
need wifl parking is there part town
User ahould incldde Fuee will and should be the waat..
hoststarijhotej [inform] name joftebookk]:: there seeme be opening at joatue_ namee would you like me to book you room ?
User you. please. 5 people for rights starting thundaay-
Aosisanttthotel [offerbooked] reforence)) booking g was su coesotul your re reforence rumber is (yalue, reforence).
Userr thank you vey much!
Aosistelllpeereaa]] [vwecomme] [reqmorw]]: you are welcome. thane wnything shew roin do you
User no thank you apprec of your help
Aesstartilgeeerag [welcome]j: thank you gladl
Conversation
[JDALOOG]
The prompt for Directional Stimulus Prompting on the MuliWOZ. Compared with the prompts used in standard prompting shown in Figure 9 each system turn, which are highlighted in red. In addition, we add explanations of dialogue acts at the beginning to help the model understand their meanings.
Figure 12:
we add stimulus hints (dialogue acts) for
27


Reviews:
[{"review_id": "vs4cFZ2CTj", "reviewer": "Reviewer_14f8", "summary": "This paper introduces Directional Stimulus Prompting (DSP), a new prompting framework that introduces directional stimulus into the prompt, which could provide black-box LLMs with fine-grained and query-specific guidance toward the desired outputs.\n\nThe experiments on summarization and dialogue response generation tasks demonstrate the effectiveness of this approach. Notably, on the MultWOZ dataset, our framework enables ChatGPT to achieve a remarkable 41.4% improvement in its combined score with only 80 dialogues.", "questions": "None", "limitations": "None", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.It is quite novel that the paper proposed DSP, a prompting framework for guiding black-box LLMs toward desired outputs, which combined Supervised fine-tuning and Reinforcement learning to further optimize model.\n\n2.The experiment setting is quite detailed, since this paper used varying numbers of training samples from the datasets and evaluation metrics for ease of display and comparison.", "weaknesses": "1.There is too little dataset in the experiment section. Both tasks were conducted on a single dataset and cannot fully demonstrate the effectiveness of the framework.\n\n2.On Page 5 Line 151, the experiment section of the paper is not enough to prove this conclusion. It is not rigorous to say that the framework can be flexibly applied to various types of LMs and generation tasks, just by conducting experiments on two tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful to the reviewer for acknowledging the strengths of our paper and offering valuable feedback. We aim to address your concerns in the following response.\n\nRegarding the concerns about the scope of our experiments, we expanded our experiments with an additional reasoning task on two reasoning datasets MultiArith [1] and AQuA [2] (Details are provided in the [global author rebuttal](https://openreview.net/forum?id=UvIN8oQ4uI&noteId=ztu9uesRIz)). \n\nWhile current prompting methods typically employ general task-specific prompts, LLMs exhibit sensitivity to these prompts. Consequently, a general prompt might not always be the optimal choice for every scenario.  In our extended experiment, we leverage DSP to generate query-specific trigger prompts for chain-of-thought reasoning. Our results showed that text-davinci-002's zero-shot chain-of-thought reasoning performance is indeed highly sensitive to the used trigger prompts. With the query-specific prompts generated by the policy model, DSP improved text-davinci-002's performance compared with using 14 different human-designed task-specific prompts [3] and also a prompt automatically discovered by the APE approach [4]. This study offers further evidence that our framework can provide LLMs with fine-grained query-specific guidance to align them with desired outputs (i.e., conducting chain-of-thought reasoning to derive the correct answer in this case).\n\nWe hope that our response and the expanded experiments can address your concerns.\n\n**References**\n\n[1] Roy, S. and Roth, D. Solving general arithmetic word problems. arXiv preprint arXiv:1608.01413, 2016.\n\n[2] Ling, Wang, et al. \"Program induction by rationale generation: Learning to solve and explain algebraic word problems.\" arXiv preprint arXiv:1705.04146 (2017).\n\n[3] Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., and Iwasawa, Y. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199–22213, 2022.\n\n[4] Zhou, Y., Muresanu, A. I., Han, Z., Paster, K., Pitis, S., Chan, H., and Ba, J. Large language models are human-level prompt engineers. arXiv preprint arXiv:2211.01910, 2022.\n"}, {"review_id": "WOj7qTstfT", "reviewer": "Reviewer_JgjJ", "summary": "This paper introduces a interesting module named 'Directional Stimulus Prompting' (DSP). This innovative module functions by generating cues or hints to aid black-box Large Language Models (LLMs) in response generation. For instance, in a summarization task, providing keywords can guide the LLM towards generating more accurate and relevant answers.\n\nThe uniqueness of this module lies in its ability to be fine-tuned through both supervised learning and reinforcement learning techniques. In reinforcement learning, automatic metrics (reward) are applied to responses generated from black-box LLMs, which have been influenced by the DSP.\n\nThe results from experiments confirm that both the supervised fine-tuning and reinforcement learning approaches to DSP can significantly enhance the performance of black-box LLMs. This finding underscores the potential of DSP as a powerful tool for improving LLM response generation.", "questions": "Could the \"few-shot prompting\" or \"in-context learning\" potentially enhance the effectiveness of the Directional Stimulus Prompting (DSP) module?", "limitations": "N/A", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper introduces a compact yet powerful module known as 'Directional Stimulus Prompting' (DSP). Its function is to generate hints that enhance queries posed to black-box Large Language Models (LLMs).\n\nTwo distinct approaches for fine-tuning this DSP module are presented by the authors: supervised fine-tuning and reinforcement learning.\n\nThe effectiveness of both supervised fine-tuning DSP (SFT DSP) and reinforcement learning DSP (RL DSP) is confirmed through experimental results, underscoring the utility of this novel module in enhancing LLM query performance.", "weaknesses": "Currently, automatic metrics are employed for evaluation, as well as for the reinforcement learning (RL) tuning of the Directional Stimulus Prompting (DSP) module. However, presenting these metrics exclusively in the final analysis could restrict the comprehensive assessment of performance. It may be more insightful to incorporate other evaluation metrics into the analysis. This could include human evaluation, entity word extraction, and GPT-based evaluations. Such a diversified metrics approach could potentially provide a more holistic view of the performance improvements achieved through the use of the DSP module.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate your feedback and insightful suggestions. Below is our response to address these points.\n\n**Response to W1: exclusive use of automatic metrics in the final analysis**\n\nTo address the concern about the exclusive use of automatic metrics in the final analysis, we incorporated GPT-4-based evaluation on the CNNDM summarization dataset. As we employ ROUGE scores as rewards for tuning the policy model to generate keywords that guide the LLM towards generating summaries more aligned with the reference summary, we leveraged GPT-4 to assess the overlap of key points between generated and reference summaries. Specifically, we use GPT-4 to compare the summaries generated with our proposed DSP and the orginal standard prompting. GPT-4 was instructed to first generate an explanation, followed by the corresponding answer. The prompt for the GPT-4 evaluation is as follows:\n```\nYou are provided with an article and a corresponding reference summary. Additionally, there will be two alternative summaries labeled as 'A' and 'B'.\nYour task is to identify which of the two summaries (A or B) is more similar to the reference summary. This similarity should be evaluated based on the presence and accuracy of key points from the reference summary in each alternative summary.\nPlease detail your reasoning in an explanation. After your explanation, classify the task outcome as: select 'A wins' if Summary A aligns more closely with the reference summary, 'B wins' if Summary B aligns more closely, or 'Tie' if both summaries align equally well with the reference summary.\n```\nWe found that GPT-4 can produce reasonable and detailed explanations of their assessment. From our test set of 500 samples: DSP-generated summaries were favored 255 times (**51.0%**), summaries generated with original standard prompting were favored 222 times (**44.4%**), while a tie was observed in 23 cases (**4.6%**). We will release the GPT-4 evaluation results, including explanations.\n\n**Response to Q1: Could the \"few-shot prompting\" or \"in-context learning\" potentially enhance the effectiveness of the Directional Stimulus Prompting (DSP) module?**\n\nIn our current experiments, we employ few-shot prompting with 3 examples in the prompt during training and evaluation. The specific prompt and demonstration examples utilized are detailed in the Appendix. In response to your question, we evaluated two experimental settings on the CNNDM dataset (4,000 training samples):\n1. Few(3)-shot during training and zero-shot during testing.\n2. Zero-shot during both training and testing.\n\nThe zero-shot evaluation results are as follows:\n| Method | ROUGE-1 | ROUGE-2 | ROUGE-L | BLEU | METEOR | BERTScore |\n| --- | --- | --- | --- | --- | --- | --- |\n| Standard Prompting | 37.34 | 14.13 | 24.43 | 4.86 | 29.85 | 0.8798 |\n| DSP w/ SFT+RL(0-shot) | 38.73 | 15.14 | 25.26 | 5.22 | 31.21 | 0.8820 |\n| DSP w/ SFT+RL(3-shot) | 38.47 | 15.20 | 24.82 | 5.17 | 31.29 | 0.8815 |\n\nFrom the results, our approach exhibits robustness when different numbers of examples are used in prompts during training and evaluation. When both training and testing are conducted using zero-shot prompting, the performance improvement over standard prompting is also comparable to the scenario where both are conducted using few-shot prompting (results shown in the current paper).\n"}, {"review_id": "6AbfefPMLY", "reviewer": "Reviewer_RT8N", "summary": "In this paper, a prompting framework called Directional Stimulus Prompting (DSP) is proposed which provides a more fine-grained guidance and control over LLMs by adding directional stimulus into the prompt. These directional stimulus or hints are generated by a small tunable model which is fine-tuned using supervised learning and reinforcement learning. The performance of the model is tested on summarization and dialogue response generation tasks.", "questions": "See weakness section.", "limitations": "One of the good points that they motioned is that their framework could be used to guide LLMs to generate harmful or biased contents.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "S1: It is interesting that they fine-tuned and used a small language model to improve the performance of larger language models such as ChatGPT. This approach cleverly bypasses the constraint of being unable to fine-tune large language models.\n\nS2: In both experiments, they demonstrated the effectiveness of their approach by showing the improvement achieved through their framework.\n\nS3: The paper is well written and has a good flow.\n\nS4: A good number of related works are covered in the Related work section.\n\n", "weaknesses": "In their experiments, they only show the results for the flan-T5-large model as a model for generating directional stimulus. They could have also included the performance of other fine-tuned models in their experiments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We genuinely appreciate your acknowledgment of our work's strengths and your valuable feedback.\n\nRegarding the concern about the exclusive presentation of results for the flan-t5-large model, we would like to underscore that our proposed DSP is a general framework and it is not tailored or confined to a specific task or policy model/LLM. The policy model's role in our evaluated tasks is to generate an output given the query context as input. Hence, we chose the suitable widely-used seq2seq t5 models. Due to the enhanced capabilities of flan-t5 models relative to same-sized models, we prioritized it in our experiments. \n\nAdditionally, we did try t5-base and flant-t5-base in our preliminary tests and the results consistently demonstrated the advantage of using DSP over standard prompting. For instance, in the MultiWOZ dataset, results using t5-base and flan-t5-base are comparable to those of the flan-t5-large model. With the flan-t5-base and Codex as the LLM, after training on 80 dialogues from MultiWOZ 2.0, we can also achieve a BLEU score of 10.32, success rate of 78.33, inform rate of 91.67, and combined score of 95.32. \nIn our new task detailed in the [global author rebuttal](https://openreview.net/forum?id=UvIN8oQ4uI&noteId=ztu9uesRIz), we employed the t5-base as the policy model, and the results also demonstrated the effectiveness of our framework."}, {"review_id": "jowcR7REZZ", "reviewer": "Reviewer_FEWZ", "summary": "This paper introduces a novel approach to effectively guide black-box LLMs towards generating desired outputs. The proposed approach involves utilizing a relatively small policy model to generate \"directional stimulus,\" which serves as specific information to assist LLMs in performing tasks such as summarization and task-oriented dialogue. The policy model is trained using rewards obtained from evaluating LLM outputs. Notably, this approach requires only a small amount of training data and eliminates the need to train the entire model.", "questions": "- It would be valuable to know the exact API cost associated with training the reward model. The authors seem to have made efforts to minimize API calls, and providing this information would offer additional details for other researchers to reproduce the study accurately.\n- In relation to the aforementioned weakness, it would be interesting to investigate how the model's performance is affected by changing the prompt template. Understanding the impact of prompt variations would provide insights into the robustness of the approach.\n- Regarding Figure 1, it appears that the part describing what is \"highlighted in blue\" is missing. Including this information would clarify the intended meaning.\n- As a suggestion, exploring the possibility of allowing the model to \"edit\" the prompt could potentially offer greater flexibility and a wider search space, leading to improved performance. This suggestion aims to enhance the approach by incorporating additional avenues for improvement.", "limitations": "The authors' discussion of the limitations in the paper is not sufficient. Addressing the suggested weaknesses and questions in reviews would significantly enhance the insights provided by this paper.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The writing is clear, making it easy to grasp the motivation and contribution of the paper.\n- The proposed approach is simple, which suggests its potential for widespread adoption across various domains.\n- The experimental results effectively demonstrate the effectiveness of the approach. Notably, in Figure 3, the model leveraging DSP achieves superior performance in terms of BLEU, METEOR, and BERTScore, despite Rouge being used as the reward metric.\n- The fact that the authors have released their code is highly valuable, as it facilitates future research and allows for the replication of their findings.", "weaknesses": "- I have a concern regarding the generalizability of the method, as the experiments primarily rely on a single prompt per task.\n  - It is widely acknowledged that LLMs heavily rely on the structure and format of the prompt, which raises the question of whether this technique would be effective with different prompt formats.\n  - This approach would hold even greater value if it could consistently yield improvements regardless of the choice of prompt format, demonstrating its robustness.\n- Additional analysis about generated clues will be helpful to get lessons from this work. For example,\n  - How does the result change if the number of clues is increased/reduced? \n  - What kind of clues are generated considering the semantics/contents of the input (visualization will be helpful)\n- The method is not intuitive to me, because all the information that is required to perform the task is already contained in the input. Does the result mean that LLMs are not sufficient to extract information from naive and complex inputs? It will be helpful if the authors provide thoughts about this phenomenon.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We deeply appreciate your recognition of our work's strengths and the constructive feedback and suggestions. We hope our response can address your concerns accordingly.\n\n**Response to W1: robustness to different prompt formats.**\n\n1. **Robustness of DSP**: Through the experiment detailed in the \"Response to Q1\" in the [rebuttal to Reviewer JgjJ](https://openreview.net/forum?id=UvIN8oQ4uI&noteId=6sYAdkCX9o), we found that the policy model trained with few(3)-shot prompting can still improve performance when testing with zero-shot prompting. When both training and evaluation are conducted with zero-shot prompting, the performance improvement over standard prompting is also comparable to both using few-shot prompting (the results shown in our current paper). These observations suggest the robustness of our approach to different numbers of examples in the prompt. \n\n2. **Addressing prompt sensitivity of LLMs**: Sensitivity to prompts is a critical and inherent challenge of LLMs. Our proposed DSP could be potentially used to find the suitable prompt for each sample. To prove that, we conducted experiments on the chain-of-thought (CoT) reasoning, as detailed in the [global author rebuttal](https://openreview.net/forum?id=UvIN8oQ4uI&noteId=ztu9uesRIz). Current prompting methods typically rely on task-specific prompts. In our experiment, we use DSP to generate query-specific trigger prompts for chain-of-thought reasoning. Our experimental results demonstrated that text-davinci-002 is highly sensitive to the used prompts. DSP improved its zero-shot CoT reasoning performance over 14 tested human-designed prompts and a prompt discovered by the APE approach [1]. \n\n**Response to W2: Additional analysis about generated clues will be helpful to get lessons from this work.**\nWe included the following analysis of the generated clues/keywords.\n- Number of generated keywords: We've outlined changes in the number of generated keywords, hit keywords (those matched in the reference summary), and corresponding ROUGE-1 scores throughout the training process in the table below. As training progresses, the policy model appears to generate keywords with increasing accuracy, which aligns positively with the increasing ROUGE-1 score. However, it's worth noting that even when keywords are generated with high precision if their quantity is too limited, the performance doesn't necessarily improve.\n| Training iters | #Generated keywords | #Hit keywords | Keyword Precision | ROUGE-1 |\n| ---- | ---- | ---- | ---- | ---- |\n| 0 | 7.986 | 2.936 | 0.367 | 39.30 |\n| 1  | 6.806 | 2.85 | 0.345 | 39.24 |\n| 3 | 7.516 | 3.174 | 0.422 | 39.61 |\n| 5 | 7.262 | 3.31 | 0.456 | 39.63 | \n| 7 | 6.676 | 3.134 | 0.469 | 39.96 |\n| 9 | 6.186 | 2.998 | 0.485 | 39.91 |\n- Type of generated keywords: We employed the spacy package for Part-of-Speech (POS) and Named Entity Recognition (NER) tagging on the generated keywords. The results are shown in the tables below. For the POS tagging, we observe that nouns (NOUN) and proper nouns (PROPN) are the most frequently generated keywords, which can serve as informative keywords. As for the NER tagging, the most commonly generated keywords include persons (PERSON), geopolitical entities (GPE), dates (DATE), organizations (ORG), and numerals (CARDINAL). \n\n| POS Tagging | Appearances | Frequency |\n| ---- | ---- | ---- |\n| NOUN | 2342 | 36.15% |\n| PROPN | 2327 | 35.92% |\n| ADJ | 516 | 7.97% |\n| NUM | 476 | 7.35% |\n| DET | 261 | 4.03% |\n| VERB | 216 | 3.33% |\n| PRON | 81 | 1.25% |\n| ADP | 76 | 1.17% |\n| ADV | 69 | 1.07% |\n| SYM | 42 | 0.65% |\n| AUX | 21 | 0.32% |\n| PART | 18 | 0.28% |\n| CCONJ | 13 | 0.20% |\n| SCONJ | 12 | 0.19% |\n| X | 5 | 0.08% |\n| INTJ | 3 | 0.05% |\n\n| NER Tagging | Appearances | Frequency |\n| ---- | ---- | ---- |\n| PERSON | 567 | 30.32% |\n| GPE | 284 | 15.19% |\n| DATE | 272 | 14.55% |\n| ORG | 247 | 13.21% |\n| CARDINAL | 245 | 13.10% |\n| NORP | 95 | 5.08% |\n| ORDINAL | 49 | 2.62% |\n| MONEY | 37 | 1.98% |\n| TIME | 26 | 1.39% |\n| QUANTITY | 11 | 0.59% |\n| EVENT | 10 | 0.53% |\n| LOC | 9 | 0.48% |\n| FAC | 6 | 0.32% |\n| PRODUCT | 5 | 0.27% |\n| PERCENT | 3 | 0.16% |\n| LANGUAGE | 2 | 0.11% |\n| WORK_OF_ART | 2 | 0.11% |\n\nVisualization (pie chart) of these analyses will be included in the updated version of the paper.\n\n**Response to W3: Does the result mean that LLMs are not sufficient to extract information from naive and complex inputs?**\n\nLLMs have demonstrated their ability to generate high-quality text. However, in open-ended generation tasks, where multiple outputs are valid, LLMs may not consistently produce text that aligns precisely with the desired output for specific tasks and datasets.\nFor instance, in the summarization task, LLMs can produce high-quality summaries. Nevertheless, different summarizers might have distinct styles and emphases. Capturing these nuanced variations and specific emphases in different queries becomes challenging when guiding LLMs with a general task-specific prompt.\n\nTo address this, our approach trains the policy model on labeled data to learn these subtle signals in the dataset and generate query-specific hints to provide LLMs with fine-grained guidance towards desired outputs for the specific tasks and datasets.\n\n**Response to Q1: the exact API cost**\n\nIn our estimation, for experiments on the CNNDM dataset, the training cost per iteration is around \\\\$1.80, while the evaluation cost on the validation/test set is around \\\\$2.37. As for the experiments on the MultiWOZ dataset, the training cost is around \\\\$1.05 per iteration and the evaluation cost is around \\\\$26.7 for the whole test set.\n\n**Response to Q3: highlighted in blue in Figure 1**\n\nThe part highlighted in blue is the generated summary. For DSP, the summary is generated given the provided keywords/hints. For standard prompting, such hints are absent when generating the summary.\n\n**Response to Q4: prompt edit experiments**\n\nPlease refer to the global author rebuttal for details on the experiments."}, {"review_id": "jR8dTNlMlu", "reviewer": "Reviewer_2Z7n", "summary": "This paper proposes a Directional Stimulus Prompting method to provide fine-grained guidance for the output of large language models (LLMs). This method introduces a small trainable policy model (e.g. FLAN-T5) to generate hints for each query to guide LLMs towards desired outputs. This policy model can be trained via a standard paradigm including supervised fine-tuning on pseudo-labeled data and reinforcement learning (RL) to optimize the expected reward. Experiments on summarization and dialogue generation tasks show the effectiveness of the proposed method.", "questions": "I have included my questions in the weaknesses part.", "limitations": "The authors should discuss more on the risk of guiding LLMs to generate unethical contents and how to avoid it via the policy model.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "1. This paper proposes a direct and feasible solution to guide black-box LLMs towards better generation results, which is widely applicable with the rapid development of LLMs.\n2. This paper is well-written and easy to follow.\n", "weaknesses": "1. The technical novelty of the proposed method is limited. The training of the policy model in this paper is a standard paradigm including supervised fine-tuning and reinforcement learning. The techniques used in RL including dynamic adjustment of the coefficient and NLPO are all borrowed from existing works. Thus, I feel that the main difference falls into the usage of LLMs as an evaluation function in the reward model, which is devised for adapting this RL algorithm to the scenario of guiding LLM. Thus, the novelty of this design is mainly on the applicational side.\n\n2. During supervised fine-tuning, the authors heuristically select the “pseudo-stimulus” for each input. The pseudo-stimulus indicates keywords / dialogue acts for summarization / task-oriented dialogue generation, respectively. I wonder whether there is a general principle to select pseudo-stimulus especially in open-ended text generation tasks, since recently proposed LLMs such as ChatGPT are mainly applied to these tasks.\n\n3. The benchmark datasets in this paper only contain CNN/DM for summarization and MultiWOZ for task-oriented dialogue generation, which are not convincing enough to evaluate the performance of the proposed method based on LLMs. The authors should conduct experiments on broader tasks and datasets. Also, I’m curious about the motivation to choose MultiWOZ. In Section 3.2, the authors say that LLM-based chatbots such as ChatGPT face challenges in handling task-oriented dialogue generation because this task requires the chatbot to respond based on reliable information from API calls or database queries. But the proposed method still cannot interact with APIs or databases. The experimental setting of MultiWOZ seems like end-to-end response generation (i.e. generating the system response given the dialogue history), which is similar to open-domain conversations.\n\n4. From [1], automatic evaluation metrics such as ROUGE cannot reliably evaluate the quality of LLM-generated summaries. Thus, human evaluation should be added to strengthen the experimental results.\n\n[1] News Summarization and Evaluation in the Era of GPT-3.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to express our gratitude for recognizing the strengths in our manuscript and for the detailed feedback. We value these suggestions and hope our response can address the concerns accordingly.\n \n**Response to W1: The technical novelty of the proposed method is limited. The training of the policy model in this paper is a standard paradigm including supervised fine-tuning and reinforcement learning.**\n\nThe goal of our work is to tackle the notable challenge of guiding black-box LLM outputs towards desired outcomes for specific tasks, datasets, and even queries. This is a pressing yet challenging issue to address, as it is both inefficient and infeasible to directly fine-tune black-box LLMs. Our proposed framework innovatively proposes to fine-tune a small policy model to guide black-box LLMs, bypassing the constraints of being unable to fine-tune them directly. This can be accomplished with standard SFT/RL training approaches, showcasing the effectiveness and adaptability of our approach. Therefore, introducing new SFT/RL training methods that are not \"standard paradigm\" is not the paper's focus. It is worth mentioning that the novelty and value have gained recognition from other reviewers. \n\n**Response to W2: general principle to select pseudo-stimulus, especially in open-ended text generation tasks**\n\nA principle is to select latent control codes that can influence the generated outputs. These control codes are intrinsic to open-ended text generation tasks, where models have more freedom to decide the textual content they generate. In our experiments, we choose keywords as they can guide the key points that should be incorporated in the generated summary. Likewise, dialogue acts are used for task-oriented dialogues, which could indicate the appropriate responses given the existing dialogue context. As for other open-ended text generation tasks like open-domain chit-chat, the control codes can be emotions, topics, styles, etc. Similarly, for story generation tasks, potential control codes could include key events and the narrative style, both of which steer the direction and tone of the story. Taking it a step further, large language models (LLMs) can be utilized to automatically analyze and determine the latent control codes specific to certain tasks.\n\n**Response to W3: The benchmark datasets in this paper only contain CNN/DM for summarization and MultiWOZ for task-oriented dialogue generation. Also, I’m curious about the motivation to choose MultiWOZ.**\n\nWe have expanded our experiments by incorporating an additional task on the arithmetic reasoning datasets MultiArith [1] and AQuA [2]. The details can be found in the [global author rebuttal](https://openreview.net/forum?id=UvIN8oQ4uI&noteId=ztu9uesRIz). \n\nAs for the reason for the choice of MultiWOZ: Currently, most advances in LLMs and chatbots are predominantly in open-domain conversations where they have demonstrated impressive capabilities. However, LLMs still struggle with task-oriented dialogues where they should follow task-specific dialogue flows and output formats, as observed in [1] [2]. Therefore, we choose to fill the gap and experiment on the widely-used task-oriented dialogue dataset MultiWOZ. In addition, the MultiWOZ dataset provides annotations of dialogue acts, which could be directly used as the stimulus in our DSP framework, avoiding the additional annotation efforts.\n\n**Response to W4: human evaluation should be added to strengthen the experimental results.**\n\nWe appreciate and value the suggestion. In response, we incorporate evaluations based on GPT-4, which is found to be able to provide consistently high-quality assessments of text generation, being a good alternative to human evaluations. Specifically, we leveraged GPT-4 to compare the summaries generated with our proposed DSP and the original standard prompting based on the assessment of overlap of key points between generated and reference summaries. GPT-4 was instructed to first generate an explanation, followed by the corresponding answer (who wins). The prompt used for the GPT-4 evaluation can be found [here](https://openreview.net/forum?id=UvIN8oQ4uI&noteId=6sYAdkCX9o).\n\nFrom the 500 test samples: DSP-generated summaries were favored 255 times (**51.0%**), summaries generated with standard prompting were favored 222 times (**44.4%**), while a tie was observed in 23 cases (**4.6%**). We found that GPT-4 can produce reasonable and detailed explanations of their assessment. We will release the GPT-4 evaluation results, including the explanations.\n\n**References**\n\n[1] Hudeček, Vojtěch, and Ondřej Dušek. \"Are LLMs All You Need for Task-Oriented Dialogue?.\" arXiv preprint arXiv:2304.06556 (2023).\n\n[2] Bang, Yejin, et al. \"A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity.\" arXiv preprint arXiv:2302.04023 (2023)."}]

Author rebuttal:
### Additional experiment
We greatly appreciate all the reviewers' insightful suggestions and feedback. We conducted an additional experiment in which we use DSP to provide query-specific trigger prompts for chain-of-thought reasoning using two widely-used datasets MultiArith [1] and AQuA [2]. Our results showed that our method DSP could improve text-davinci-002's zero-shot chain-of-thought reasoning performance with query-specific prompts, compared with task-specific human-designed prompts and the prompt automatically discovered with the APE approach [4].

**Experimental setup**: We adopted the experimental setup from previous work [3][4]. We tested the zero-shot chain-of-thought reasoning abilities of text-davinci-002 with different trigger prompt templates. There are 600 examples in the MultiArith dataset, which we divided into 300/50/250 for training/validation/test set. As for the AQuA dataset, we use the standard test set with 254 samples, 300 samples from the standard training set for our training, and 100 samples for the standard validation set for our validation.

**Supervised fine-tuning details**: For supervised fine-tuning (SFT), we first run inference on the training set with the 14 human-designed prompts tested in [3], respectively. We then selected those prompt and query pairs which resulted in a correct chain-of-thought reasoning outcome to form the training set for SFT. These query-prompt pairs were used to train a t5-base policy model for 2 epochs, with the model input being the query instance and the target output a trigger prompt.

**RL training details**: After SFT, the prompts generated by the policy model were used to trigger text-davinci-002 for zero-shot CoT prompting. Reasoning accuracy was utilized as the reward for reinforcement learning (RL). A reward of 1 was assigned for correct reasoning results and 0 otherwise. We conducted 20 training iterations (106k episodes), with 5 epochs per batch, a batch size of 8, and a learning rate of 2e-6. The parameters for $KL_{target}$ and $\beta_0$ were set to 0.5 and 0.001, respectively.

**Results**: We compare the performance of using our generated query-specific prompts with using the 14 human-designed prompts which we used as the pseudo-stimulus to constitute the training set for SFT and also a prompt discovered by the APE approach [4]. Note that all these 15 prompts are general task-specific and are used for the whole test set. By contrast, our proposed DSP enables us to generate query-specific prompts to trigger the LLM's CoT reasoning. The performance comparison is shown in the table below.

| No. | Category | Zero-shot CoT Trigger Prompt | MultiArith | AQuA |
| ---- | ---- | ---- | ---- | ---- |
| 1 | Human-Designed | Let's think step by step. | 79.6 | 31.9 |
| 2 | Human-Designed | We should think about this step by step. | 81.2 | 28.7 |
| 3 | Human-Designed | First, | 78.0 | 38.2 |
| 4 | Human-Designed | Before we dive into the answer, | 54.8 | 27.2 |
| 5 | Human-Designed | Proof followed by the answer. | 58.4 | 37.8 |
| 6 | Human-Designed | Let's think step by step in a realistic way. | 59.6 | 33.9 |
| 7 | Human-Designed | Let's think step by step using common sense and knowledge. | 80.0 | 34.3 |
| 8 | Human-Designed | Let's think like a detective step by step. | 73.6 | 24.0 |
| 9 | Human-Designed | Let's think about this logically. | 75.2 | 34.7 |
| 10 | Human-Designed | Let's think step by step. First, | 78.8 | 32.3 |
| 11 | Human-Designed | Let's think | 56.8 | 38.2 |
| 12 | Human-Designed | Let's solve this problem by splitting it into steps. | 72.4 | 32.3 |
| 13 | Human-Designed | The answer is after the proof. | 42.8 | 34.3 |
| 14 | Human-Designed | Let's be realistic and think step by step. | 69.6 | 29.9 |
| 15 | APE [4] | Let's work this out in a step by step way to be sure we have the right answer. | 81.6 | 34.3 |
| 16 | DSP w/ SFT | *Query-specific* | 75.2 | 35.8 |
| 17 | DSP w/ SFT+RL | *Query-specific* | **82.4** | **38.6** |

As can be seen, text-davinci-002's performance varies significantly when using different task-specific prompts. Compared to the 14 task-specific human-designed prompts, DSP enhances the performance of text-davinci-002 with query-specific prompts. It also outperforms the prompt discovered by the APE approach [4]. Solely relying on supervised fine-tuning of the policy model with the dataset comprising the 14 human-designed prompts doesn't lead to its peak performance. After fine-tuning with RL, the policy model is encouraged to explore better query-specific trigger prompts, further improving performance. Some of the newly generated trigger prompts include:
 - "*Let's think like a detective step by step. First,*"
- "*Let's solve this problem by splitting it into steps. First,*" 
- "*First step:*"
- "*Let’s think step by step. First*",
- "*Let's think step by step using our creative brains.*"
- "*Let's think step by step using both the above information and the testing.*"
- "*Let's think step by step using proven methods.*"

Overall, the results provide further evidence that DSP could provide fine-grained query-specific guidance to LLMs to align them better with the desired output (chain-of-thought reasoning with correct answers in this case).
Further details will be provided in the updated version of our paper.


**References**:

[1] Roy, Subhro, and Dan Roth. "Solving general arithmetic word problems." arXiv preprint arXiv:1608.01413 (2016).

[2] Ling, Wang, et al. "Program induction by rationale generation: Learning to solve and explain algebraic word problems." arXiv preprint arXiv:1705.04146 (2017).

[3] Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., and Iwasawa, Y. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199–22213, 2022.

[4] Zhou, Y., Muresanu, A. I., Han, Z., Paster, K., Pitis, S., Chan, H., and Ba, J. Large language models are human-level prompt engineers. arXiv preprint arXiv:2211.01910, 2022.


Meta-review: One consistent strength highlighted by multiple reviewers is the 'Directional Stimulus Prompting' (DSP) module. It offers an approach to guiding black-box Large Language Models (LLMs) by generating hints or cues, and the combination of supervised fine-tuning and reinforcement learning techniques adds to its innovative appeal. The experimental setup in the paper has also been appreciated for its detail, using various training sample sizes and evaluation metrics for clarity and comparison. The authors conducted some addition experiments during rebuttal on MultiArith and AQuA to show its good performance. 

However, there are shared concerns among the reviewers, in order to make the paper more comprehensive. The limited dataset used in the experiments, with a focus on a single dataset for both summarization and dialogue generation, raises questions about the generalizability of DSP to a broader range of tasks. Additionally, some reviewers express doubts about the robustness of the method, as it heavily relies on the format and structure of the prompt, making its effectiveness potentially dependent on specific prompt formats. Reviewers suggest the need for a more diverse set of evaluation metrics, including human evaluation, entity word extraction, and GPT-based evaluations, to provide a more comprehensive assessment of DSP's performance (In my opinion, especially human preference is very important, because we have been observing in summarization field that even though fitting into some formats or styles of a dataset, the output summaries are still not preferred by human). Furthermore, there's a call for additional analysis of the generated cues and clues, exploring variations in the number and content of cues.
Decision: Accept (poster)