SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In recent years, there has been a rise in LLM-based chatbots such as ChatGPT $^{4}$  and Sparrow $^{5}$ . These chatbots are typically targeted at open-domain conversations to engage with users on a wide range of topics without a specific goal in mind. However, these chatbots still face challenges in handling task-oriented dialogues where they need to assist users in completing specific goals or tasks, such as making reservations or ordering food [4, 22]. Unlike open-domain conversations, task-oriented dialogues often require the chatbot to follow task-specific business logic and respond based on reliable information from API calls or database queries. To address this limitation, we train a small policy model to learn the underlying dialogue policy from the training data and thus guide the LLMs in generating reliable system responses that assist users in completing tasks.
Dataset and evaluation We conduct experiments on the popular task-oriented dialogue dataset MultiWOZ [7], including both the MultiWOZ2.0 (the original version) and MultiWOZ2.1 version [15]. The dataset provides annotations for user utterances, dialogue acts, and system responses for each dialogue turn. The goal is to generate the system response given the history dialogue context as input. We utilize the dialogue act, which represents the communicative intention of the target system response, as the pseudo-stimulus for our experiment. There are 8,438 dialogues in the training set. We only use  $1\%$  (80 dialogues) and  $10\%$  (800 dialogues) to train the policy model and evaluate the performance on the full validation and test set, which contains 1,000 dialogues. We use the standard evaluation metrics: Inform, which measures the rate that the appropriate entity that satisfies the user's requirements is provided; Success, which measures the rate that all requested attributes are answered; BLEU: the corpus-level BLEU score with reference responses; and an overall measure Combined score = (Inform+Success)×0.5+BLEU. Likewise, we report the average score over three inferences. We use the same three demonstration examples when using DSP or standard prompting.
Supervised fine-tuning details To conduct supervised fine-tuning on the policy model, we format the input of each sample as Translate dialogue to dialogue action: [Dialogue context]", with the target being the verbalized dialogue acts in the same format as [77, 63]. For instance, a dialogue act <hotel, inform, choice>, <hotel, inform, type>, <hotel, request, area> will be converted to "[hotel] [inform] choice type [request] area", which indicates that the system should inform available hotel choices and their types and ask for the area that the user would like (see the Appendix for examples). Note that the provided dialogue act annotations may not be the only valid dialogue act for the same dialogue content [77], and thus we hope to explore diverse valid dialogue acts (directional stimulus) through RL training.
RL training details The evaluation metrics Success and Inform rates are defined at the dialogue level, while the BLEU score is computed on the corpus level. However, our training and inference on conducted on the turn level. We thus use the sentence-level SacreBLEU [51] score as the reward. Same as in the summarization experiments, we generate four outputs per input using the LLM with a temperature of 0.7. The policy network is trained 52k episodes, 5 epochs per batch with a batch size of 8 and a learning rate of  $2 \times 10^{-6}$ . Since the generated dialogue acts should adhere to the business logic and ontology, we ensure that the updated policy network does not deviate significantly from the original policy model. We thus set the KL_target and  $\beta_0$  in Equation 7 as 0.2 and 0.01, respectively. During training, we use top- $k$  sampling and set  $k$  to 50 to explore the action space. During inference, we use beam search decoding with a beam size of 5.
Results We evaluate the impact of our approach DSP on Codex and ChatGPT and compare the performance with several representative task-oriented dialogue models trained on the full training set (8438 dialogues), including DAMD [77], MinTL [34], Soloist [49], SimpleTOD [21], DoTS [23], PP-
4https://openAI.com/blog/chatgpt
<sup>5</sup>https://www_deepmind.com/blog/building-safer-dialogue-agents
Table 1: Response generation performance of different methods on the MultiWOZ 2.0&2.1 datasets, where Succ. and Comb. denote the Success and Combined Score metrics, respectively.
Method	#Training data	MultiWOZ 2.0	MultiWOZ 2.1
Inform	Succ.	BLEU	Comb.	Inform	Succ.	BLEU	Comb.
Codex
Standard Prompting	-	76.7	41.5	7.7	66.8	74.2	41.9	7.8	65.9
DSP w/ SFT	1% (80)	74.9	66.3	11.1	81.7	72.0	66.0	11.3	80.1
DSP w/ SFT+RL	1% (80)	91.0	76.0	9.8	93.3	89.7	78.6	9.4	93.4
DSP w/ SFT	10% (800)	79.4	71.9	11.3	87.0	72.0	67.0	13.1	82.6
DSP w/ SFT+RL	10% (800)	96.0	86.9	10.7	102.2	94.0	86.0	9.2	99.2
ChatGPT
Standard Prompting	-	71.8	44.1	10.5	68.4	72.8	44.2	10.4	68.9
DSP w/ SFT	1% (80)	76.6	66.5	11.2	82.8	76.0	64.3	11.3	81.4
DSP w/ SFT+RL	1% (80)	90.9	82.2	10.2	96.7	87.3	78.7	10.7	93.7
DSP w/ SFT	10% (800)	72.7	64.7	11.8	80.5	75.0	67.7	12.6	83.9
DSP w/ SFT+RL	10% (800)	95.3	82.3	10.9	99.6	95.0	84.0	10.7	100.2
Fully supervised TOD models
DAMD [77]	100% (8438)	76.3	60.4	16.6	85.0	-	-	-	-
MinTL [34]	100% (8438)	84.9	74.9	17.9	97.8	-	-	-	-
Soloist [49]	100% (8438)	85.5	72.9	16.5	95.7	-	-	-	-
SimpleTOD [21]	100% (8438)	84.4	70.1	15.0	92.3	85.0	70.5	15.2	93.0
DoTS [23]	100% (8438)	86.6	74.1	15.1	95.5	86.7	74.2	15.9	96.3
PPTOD [63]	100% (8438)	89.2	79.4	18.6	102.9	87.1	79.1	19.2	102.3
UBAR [72]	100% (8438)	95.4	80.7	17.0	105.1	95.7	81.8	16.5	105.3
GALAXY [19]	100% (8438)	94.4	85.3	20.5	110.4	95.3	86.2	20.0	110.8
TOD [63], UBAR [72], and GALAXY [19]. Table 1 summarizes the overall performance comparison, from which we obtain the following observations: (1) Our approach DSP significantly improves the success and inform rates of Codex and ChatGPT, indicating that they better understand the scenario and generate appropriate responses that help users in completing their tasks. (2) However, there is no improvement in the corpus-level BLEU score, possibly because the LLMs generate responses with different speaking styles and vocabulary since they do not see oracle system responses. Nevertheless, the high success and inform rates demonstrate the usefulness of our approach in delivering helpful and reliable responses. (3) Increasing the number of supervised fine-tuning samples does not guarantee performance improvement, but further fine-tuning the policy model using RL consistently provides performance gains. This suggests that RL training encourages the policy model to explore more model-preferred stimulus, while supervised fine-tuning may merely generate stimulus closely aligned with the pseudo-labeled data, which is not necessarily optimal. (4) Our approach achieves notable success with only 80 dialogues, surpassing several fully trained TOD models, particularly in terms of Success and Inform rates. With  $10\%$  of the training data (800 dialogues), our approach delivers comparable performance to current SOTA methods trained with full training data (8438 dialogues). We have also provided the performance of these compared methods in the low-resource settings ( $1\%$  and  $10\%$ ) and a running example in the Appendix.
3.3 Chain-of-Thought reasoning
While current methods primarily utilize generalized task-specific prompts, LLMs exhibit sensitivity to these prompts. Existing studies [69, 26, 79] illustrate that the performance of LLMs can vary significantly based on the prompt used. Consequently, a substantial portion of earlier work has been dedicated to either manually [56] or automatically [61, 79] crafting prompts. However, these studies largely concentrate on task-specific prompts, which may not be optimal for every instance of a task. In our experiment, we employ our approach to generate instance-specific prompts to elicit Chain-of-Thought (CoT) reasoning. Specifically, we train a policy model (t5-base) to generate instance-specific CoT trigger prompts, such as "Let's think step by step", to prompt varying samples.
Dataset and evaluation We adopted the experimental setup from previous work [26, 79], where we tested zero-shot CoT reasoning abilities of InstructGPT (text-davinci-002) with different trigger prompts. There are 600 examples in the MultiArith dataset [57], which we divided into
Table 2: Zero-shot chain of thoughts reasoning accuracy  $(\%)$  of text-davinci-002 with different prompts. *Our approach trains a policy model to generate instance-specific prompt triggers, which are compared to the task-specific prompts in [26, 79].
No.	Category	Chain-of-Thought Trigger Prompt	MultiArith	AQuA
1	Human-Designed	Let's think step by step.	79.6	31.9
2	We should think about this step by step.	81.2	28.7
3	First,	78.0	38.2
4	Before we dive into the answer,	54.8	27.2
5	Proof followed by the answer.	58.4	37.8
6	Let's think step by step in a realistic way.	59.6	33.9
7	Let's think step by step using common sense and knowledge.	80.0	34.3
8	Let's think like a detective step by step.	73.6	24.0
9	Let's think about this logically.	75.2	34.7
10	Let's think step by step. First,	78.8	32.3
11	Let's think	56.8	38.2
12	Let's solve this problem by splitting it into steps.	72.4	33.2
13	The answer is after the proof.	42.8	34.3
14	Let's be realistic and think step by step.	69.6	29.9
15	APE [79]	Let's work this out in a step by step way to be sure we have the right answer.	81.6	34.3
16	DSP w/ SFT	(*Generated instance-specific prompt)	75.2	35.8
17	DSP w/ SFT+RL	(*Generated instance-specific prompt)	84.0	38.6
300/50/250 for training/validation/test set. As for the AQuA dataset [35], we use the standard test set with 254 samples, 300 samples from the standard training set for our training, and 100 samples for the standard validation set for our validation. We report the reasoning accuracy.
Supervised fine-tuning details For supervised fine-tuning (SFT), we first run inference on the training set with the 14 human-crafted prompts tested in [26], respectively. We then selected those prompt and query pairs which resulted in a correct CoT reasoning outcome to form the training set for SFT. These query-prompt pairs were used to train a t5-base policy model for 2 epochs, with the model input being the query instance and the target output a trigger prompt.
RL training details After SFT, the prompts generated by the policy model were used to trigger InstructGPT for zero-shot CoT prompting. Reasoning accuracy was utilized as the reward for reinforcement learning (RL). A reward of 1 was assigned for correct reasoning results and 0 otherwise. We conducted 20 training iterations (106k episodes), with 5 epochs per batch, a batch size of 8, and a learning rate of  $2\mathrm{e} - 6$ . The parameters for  $\mathrm{KL}_{\mathrm{target}}$  and  $\beta_0$  were set to 0.5 and 0.001, respectively.
Results We compare the performance of using our generated instance-specific prompts with using the 14 human-crafted prompts which we used as the pseudo-stimulus to constitute the training set for SFT and also the prompt automatically discovered by the APE approach [79]. Note that all these 15 prompts are generalized task-specific and are used for the whole test set while ours are instancespecific. The performance comparison is shown in the Table 8. As can be seen, InstructGPT's performance varies significantly when using different task-specific prompts. Compared to the 14 task-specific human-designed prompts, DSP enhances the performance with instance-specific prompts. It also outperforms the prompt discovered by the APE approach, suggesting the effectiveness of our approach for automatically prompt engineering and optimization. Solely relying on supervised fine-tuning of the policy model with the dataset comprising the 14 human-designed prompts doesn't lead to its peak performance. After fine-tuning with RL, the policy model is encouraged to explore better instance-specific trigger prompts, further improving performance.
Black-box large language models Recent years have witnessed the emergence of LLMs such as GPT-3 [6], Codex [9], InstructGPT, ChatGPT [46], PaLM [10], and LaMDA [66], which show significant promise in the field of NLP. These LLMs typically have a large number of parameters and require vast amounts of training data. Due to their scaling, these models have exhibited many emergent abilities, such as in-context learning, few-shot prompting, chain-of-thought prompting,
and instruction following [6, 46, 69]. However, most LLMs are not open-sourced and can only be accessed via black-box APIs, through which the users send prompt queries and receive responses. While there exist open-source LLMs such as OPT-175B [73] and Bloom [58], their local execution and fine-tuning require significant computational resources that may be infeasible for most researchers and users. However, despite their considerable performance on various tasks, LLMs often fall short of generating outputs that fully align with desired outputs on specific downstream tasks and use cases [16, 42, 18]. Our approach seeks to address this limitation by introducing directional stimulus generated by a small tunable LM into the prompt to provide more fine-grained guidance and control over black-box LLMs.
Prompt optimization and engineering Efficiently optimizing pre-trained LMs on downstream tasks by finding optimal prompts has been a focus of prior research. One approach involves tuning soft prompts, which are continuous embedding vectors that can be optimized using gradient descent methods [32, 30, 67, 2, 64]. However, the requirements of gradients and the challenge of passing gradients and continuous prompts through black-box APIs, making them less practical for the black-box LLMs. Researchers have also tried to seek optimal prompts by designing task-specific natural language instructions and selecting proper training samples as in-context demonstrations in the prompt. These methods include manual engineering [50, 6, 56], editing [61, 76], reinforcement learning [13, 39], and automatic generation [79]. Despite these efforts, such prompts are not always effective at steering LLMs to generate desired outputs, especially for fine-grained instance-specific behaviors that are difficult to describe using task-specific instructions and demonstration examples. To address this limitation, our approach is able to provide more fine-grained instance-specific guidance through the directional stimulus prompts (hints) generated by a small tunable policy model which could be optimized with supervised fine-tuning and reinforcement learning.
Controllable text generation The control of language models (LMs) has been extensively studied. Early approaches fine-tuned LMs on datasets containing desired attributes [17]. [24] proposed class-conditioned LMs, generating text with predefined control codes. However, direct LM training is costly. To address this, PPLM [12] trains an attribute model and passes gradients to control generation. GeDi [27] and DExperts [36] use class-conditional distributions as generative discriminators to guide generation, reducing computation complexity. These methods require either additional LM training or internal gradients and logistics, making them not applicable to black-box LLMs. Our approach proposes a solution to control black-box LLMs by inserting directional stimulus into the input query prompt and optimizing based on the return output.
Reinforcement learning for NLP Reinforcement learning has been successfully applied to various NLP tasks, such as syntactic parsing [44, 29], machine translation [71, 28], summarization [48, 62], conversational systems [31], etc. Language models define probability distributions over tokens in their vocabulary, and the text generation problem can be naturally formulated as selecting an action in an RL setting. Therefore, there have been extensive research efforts on optimizing LMs with RL, usually by aligning them with human preferences [80, 70, 40, 62]. For example, the LLM InstructGPT [46] is optimized with RL to better follow users' instructions and intent. In contrast with these works that directly update the LLMs to align with human preferences, our work optimizes a small policy model that generates text (stimulus) to guide LLMs to generate more human-preferred output instead of directly optimizing the LLMs, bypassing the inefficient LLM's optimization.
In this paper, we introduce Directional Stimulus Prompting (DSP), a new prompting framework to provide black-box LLMs with fine-grained and instance-specific guidance toward the desired outputs. We use a tunable policy model to generate the directional stimulus to provide such guidance and convert the optimization of black-box LLMs to that of the policy model. Experimental results demonstrate the effectiveness of our approach in controlling and guiding black-box LLMs via automatic prompt engineering and optimization. Furthermore, the generated stimulus provides valuable insights and interpretations of LLMs' behaviors. In this work, we use heuristically selected or annotated pseudo-stimulus data for supervised fine-tuning of the policy model. For future work, we hope to explore the possibility of using a "machine language" between the policy model and the LLMs that might not be intuitively preferred by humans but can better convey guidance information, as well as other forms of directional stimulus beyond text.
This research was partly sponsored by the DARPA PTG program (HR001122C0009). Any opinions, findings, conclusions, or recommendations expressed in this paper are those of the authors and do not necessarily reflect the views of funding agencies.
[73] Zhang, S., Diab, M., and Zettlemoyer, L. Democratizing access to large-scale language models with opt-175b. Meta AI, 2022.
[74] Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., and Artzi, Y. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675, 2019.
[75] Zhang, T., Ladhak, F., Durmus, E., Liang, P., McKeown, K., and Hashimoto, T. B. Benchmarking large language models for news summarization. arXiv preprint arXiv:2301.13848, 2023.
[76] Zhang, T., Wang, X., Zhou, D., Schuurmans, D., and Gonzalez, J. E. TEMPERA: Test-time prompt editing via reinforcement learning. In International Conference on Learning Representations, 2023.
[77] Zhang, Y., Ou, Z., and Yu, Z. Task-oriented dialog systems that consider multiple appropriate responses under the same context. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 9604-9611, 2020.
[78] Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., et al. Judging llm-as-a-judge with mt-bench and chatbot arena. arXiv preprint arXiv:2306.05685, 2023.
[79] Zhou, Y., Muresanu, A. I., Han, Z., Paster, K., Pitis, S., Chan, H., and Ba, J. Large language models are human-level prompt engineers. arXiv preprint arXiv:2211.01910, 2022.
[80] Ziegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Radford, A., Amodei, D., Christiano, P., and Irving, G. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593, 2019.
A.1 Summarization
We use the representative benchmark dataset CNN/Daily Mail for news summarization [43]. This dataset contains 287,113 training examples, 13,368 validation examples, and 11,490 test examples. To keep the API usage cost low, we use a subset of 1,000, 2,000, and 4,000 for training, 500 for validation, and 500 for testing. Each example in the dataset consists of a news article along with its corresponding highlight/summary written by human authors. In order to train the policy model through supervised fine-tuning, we employed the textrank [41] algorithm to automatically extract keywords from each article and only retained those mentioned in the corresponding reference summary. We initialize the policy model using the 780M FLAN-T5-large model [11, 53], and use it to guide the black-box LLM ChatGPT. The hyperparameters used in our experiments are detailed in Table 3. All the experiments are run on a server equipped with 8 NVIDIA RTX A6000 GPUs.
Model Params	Hyperparameter values
Supervised fine-tuning (SFT)	batch size: 8
epochs: 5
learning rate: 0.00002
learning rate scheduler: linear
weight decay: 0.01
RL (NLPO)	steps per update: 5120
total number of steps: 51200
batch size: 8
epochs per update: 5
learning rate: 0.000002
entropy coefficient: 0.0
initial kl coeff: 0.005
target kl: 0.5
discount factor: 0.99
gae lambda: 0.95
clip ratio: 0.2
value function coeff: 0.5
rollouts top k: 100
top mask ratio: 0.9
target update iterations: 20
Tokenizer	padding side: right
truncation side: right
max length: 512
Policy model decoding	sampling: True
temperature: 0.7
min length: 10
max new tokens: 80
LLM decoding	sampling: True
temperature: 0.7
top_p: 1.0
max new tokens: 180
Table 3: Hyperparameters for experiments on the CNN/Daily Mail dataset.
A.2 Dialogue response generation
The MultiWOZ dataset is a widely-used task-oriented dialogue dataset consisting of 8,438 dialogues for training, 1,000 dialogues for validation, and 1,000 dialogues for testing. For each turn of the dialogues, in addition to the user utterances and system response, the annotations of belief state, database query results, and dialogue act are also provided. To process the data, we followed the approach used in UBAR [72]. Specifically, we employed delexicalization by replacing specific slot values with corresponding placeholders. These placeholders can be filled based on the results of a database search. The annotated dialogue acts serve as the stimulus in our approach. Table 4 provides information on all the dialogue acts and slots present in the dataset. We converted the structured dialogue acts, originally in the form of <domain, slot, value> triplets, into text format like [domain1][inform] slot1 ... [request] slot1 ... [domain2][reqmore], where domains, acts, and slot values are all bracketed.
We used 780M Flan-T5-Large for our policy model to guide the ChatGPT and Codex LLMs. During the supervised fine-tuning of the policy model, we trained it to generate stimulus converted from the dialogue acts based on the given dialogue context. The policy model was trained for 25 epochs using 80 dialogues from the MultiWOZ2.0 and MultiWOZ2.1 datasets. When 800 dialogues are given, it was trained for 8 epochs on the MultiWOZ2.0 dataset and 20 epochs on the MultiWOZ2.1 dataset. All the hyperparameters setup is presented in Table 5.
Table 4: Full ontology for all domains in MultiWOZ2.0 [7] dataset. The upper script indicates which domains it belongs to. *: universal, 1: restaurant, 2: hotel, 3: attraction, 4: taxi, 5: train, 6: hospital, 7: police.
dialogue acts	inform* / request* / select1235 / recommend/123 / nooffer1235 / offerbook125 / offerbooked125 / nobook12 / welcome* / greet* / bye* / reqmore*
slots	address12367 / postcode12367 / phone123467 / name123 / area123 / pricerange12 / type23 / internet2 / parking2 / stars2 / departure45 / destination45 / leave45 / arrive45 / people123 / reference1235 / id5 / price45 / time15 / department6 / day125 / stay2 / car4 / food1
Model Params	Hyperparameter values
Supervised fine-tuning (SFT)	batch size: 8
epochs: 25/25/8/20
learning rate: 0.00002
learning rate scheduler: linear
weight decay: 0.01
RL (NLPO)	steps per update: 5120
total number of steps: 51200
batch size: 8
epochs per update: 5
learning rate: 0.000002
entropy coefficient: 0.0
initial kl coeff: 0.01
target kl: 0.2
discount factor: 0.99
gae lambda: 0.95
clip ratio: 0.2
value function coeff: 0.5
rollouts top k: 50
top mask ratio: 0.9
target update iterations: 20
Tokenizer	padding side: left
truncation side: left
max length: 512
Policy LM decoding	num_beams: 5
min length: 1
max new tokens: 40
LLM decoding	sampling: True
temperature: 0.7
top_p: 1.0
max new tokens: 64
Table 5: Hyperparameters for experiments on the MultiWOZ dataset.
A.3 Chain of Thought reasoning