SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zekun Li\*, Baolin Peng?, Pengcheng He?, Michel Galley?, Jianfeng Gao?, Xifeng Yan
University of California, Santa Barbara<sup>1</sup>
Microsoft2
{zekunli, xyan}@cs.ucsb.edu
{bapeng,penhe,mgalley,jfgao}@microsoft.com
We introduce Directional Stimulus Prompting, a novel framework for guiding black-box large language models (LLMs) towards specific desired outputs. Instead of directly adjusting LLMs, our method employs a small tunable policy model (e.g., T5) to generate an auxiliary directional stimulus prompt for each input instance. These directional stimulus prompts act as nuanced, instance-specific hints and clues to guide LLMs in generating desired outcomes, such as including specific keywords in the generated summary. Our approach sidesteps the challenges of direct LLM tuning by optimizing the policy model to explore directional stimulus prompts that align LLMs with desired behaviors. The policy model can be optimized through 1) supervised fine-tuning using labeled data and 2) reinforcement learning from offline or online rewards based on the LLM's output. We evaluate our method across various tasks, including summarization, dialogue response generation, and chain-of-thought reasoning. Our experiments indicate a consistent improvement in the performance of LLMs such as ChatGPT, Codex, and InstructGPT on these supervised tasks with minimal labeled data. Remarkably, by utilizing merely 80 dialogues from the MultiWOZ dataset, our approach boosts ChatGPT's performance by a relative  $41.4\%$ , achieving or exceeding the performance of some fully supervised state-of-the-art models. Moreover, the instance-specific chain-of-thought prompt generated through our method enhances InstructGPT's reasoning accuracy, outperforming both generalized human-crafted prompts and those generated through automatic prompt engineering. The code and data are publicly available. $^3$
In recent years, a new paradigm has emerged in natural language processing (NLP) with the rise of large language models (LLMs) such as InstructGPT, ChatGPT [46], GPT-4 [45], PaLM [10], and others. These models exhibit emergent abilities [68] such as strong in-context learning and few-shot prompting capabilities, which were not present in previous "smaller" language models (LMs) like BERT [14], RoBERTa [37], GPT-2 [52], and T5 [53]. This shift in paradigm has led to remarkable advancements in NLP, with LLMs demonstrating impressive general-purpose power. However, due to commercial considerations and the risk of misuse, most LLMs do not publicly release their parameters and only allow users to access them through black-box APIs. While there also exist open-sourced LLMs, fine-tuning them for specific tasks or use cases can be computationally inefficient. In this scenario, the standard approach for utilizing LLMs to perform diverse tasks is crafting generalized
*Part of the work was done when Zekun Li was interning at Microsoft Research.
†Co-advise on this work.
<sup>3</sup>https://github.com/Leezekun/Directional-Stimulus-Prompting
Figure 1: Comparison of our Directional Stimulus Prompting and the standard prompting method using LLMs such as ChatGPT for the summarization task. DSP utilizes directional stimulus/hints (highlighted in orange), which are keywords in this case, to provide instance-specific guidance to LLMs in generating summaries (highlighted in blue) that better align with the desired reference summary with higher ROUGE scores or other measures like human preferences.
task-specific prompts to query LLMs. While LLMs have demonstrated considerable performance on a wide range of language tasks, they still struggle to generate outputs that fully align with desired behaviors and directions on some specific tasks and use cases [16, 4].
Optimizing Large Language Models (LLMs) directly for specific tasks can be infeasible or inefficient for many users and developers, leading researchers to shift their focus towards prompt engineering and optimization. Prompt engineering approaches, which involve manually or automatically designing optimal task-specific natural language instructions and selecting appropriate training samples for demonstration in the prompt, have consequently gained the attention of researchers [6, 55, 79, 39]. Despite these efforts, the majority are centered on devising task-specific prompts, often falling short in steering LLMs to generate desired results on a per-instance basis.
To address the challenge, we propose a novel framework called Directional Stimulus Prompting (DSP). This framework introduces a new component called the directional stimulus into the prompt to provide nuanced, instance-specific guidance and control over LLMs. Specifically, the directional stimulus prompt acts as hints and clues for the input query to guide LLMs toward the desired output. Notably, this differs from the methods that augment LLMs with additional knowledge retrieved from external sources [25, 60], as the directional stimulus prompt is generated solely based on the input query in our framework. Figure 1 compares our proposed prompting approach, DSP, with standard prompting for the summarization task. Our approach incorporates keywords in the prompt as the directional stimulus prompt to hint at key points the desired summary should cover. By providing this instance-specific guidance through directional stimulus prompts, LLMs can generate outputs that more closely align with the desired reference summary.
We utilize a relatively small and tunable LM (e.g., T5), as the policy model to generate the directional stimulus prompt for each input query. This approach enables us to sidestep the direct optimization of black-box LLMs by optimizing the small tunable policy model instead. We train the policy model through supervised fine-tuning (SFT) using a few collected labeled data. After supervised fine-tuning, we further optimize the policy model to explore better directional stimulus prompts with reinforcement learning (RL). During RL training, we aim to maximize the reward defined as downstream performance measures or any other measures of the LLM's output conditioned on the stimulus generated by the policy model.
Figure 2 provides the overview of our framework, using the summarization task as an illustrative example. We employ a compact, tunable policy model to generate the directional stimulus prompt,
Figure 2: Overview of our proposed framework DSP, where we learn a small tunable policy model to generate the directional stimulus (keywords in this case) that provides input-specific guidance for the LLM toward the desired target. The policy model can be trained with SFT and/or RL, where the reward is defined as the downstream task performance measure, such as the ROUGE score for the summarization task, or other alignment measures like human preferences.
which specifies keywords that should be included in the LLM-generated summaries. The policy model can be trained with SFT and RL, where the reward is typically defined as the downstream task performance measure, such as the ROUGE score for the summarization task, or other alignment measures like human preferences.
Our framework can be flexibly adapted to a wide range of LMs and tasks by choosing the appropriate directional stimulus prompt, i.e., hints. We conducted experiments on summarization, dialogue response generation, and chain-of-thought reasoning tasks to evaluate the effectiveness of our framework. The results demonstrate that our DSP approach can effectively guide LLMs toward the desired targets with a small collection of labeled data. Specifically, we conduct experiments with the black-box LLMs: ChatGPT, Codex, and InstructGPT. For the policy model, we employ a 750M Flan-T5-Large [53, 11] and 220M T5-Base. For the summarization task, we use keywords as the directional stimulus, which hints at key points that the desired summary should include. Despite ChatGPT's already considerable performance, the policy model trained with only 4,000 samples from the CNN/Daily Mail dataset [43] improved the ROUGE and BLEU scores by  $4 - 13\%$ . For the dialogue response generation task, we train the policy model to generate dialogue acts that indicate the underlying intentions behind target responses on dialogues from the MultiWOZ dataset [7]. Guided by the policy model trained with only 80 dialogues, ChatGPT's performance improved by up to  $41.4\%$  in combined scores, achieving comparable or even better performance than some state-of-the-art models trained on the full dataset with 8,438 dialogues. For the chain-of-thought reasoning, we train the policy model to generate a trigger prompt for each input query to steer the LLM chain-of-thought reasoning, achieving better performance than the generalized hand-crafted prompts and those produced through the automatic prompt engineering approach [79], suggesting the effectiveness of our approach for automatic prompt engineering and optimization.
2 Directional stimulus prompting
For a downstream task, there is an input space  $X$ , a data distribution  $\mathcal{D}$  over  $X$ , and an output space  $Y$ . Due to the strong in-context learning and few-shot prompting abilities, LLMs can perform diverse tasks and generate the output  $y$  by including instructions that describe the task, a few demonstration examples, and the input query  $x$  in the prompt [6]. However, such prompts cannot always steer LLMs toward desired outputs, especially when it comes to fine-grained instance-specific desired behaviors. For instance, in the case of the summarization task, the input  $x$  is an article, and the output  $y$  is the corresponding summary. Different summarizers have distinct styles and emphasize different aspects of an article [16]. In this case, it may not be enough to effectively steer LLMs toward generating summaries that closely match reference summaries relying solely on task-specific instructions or demonstration examples to describe such nuanced differences for each sample.
To this end, our Directional Stimulus Prompting (DSP) approach introduces a small piece of discrete tokens  $z$  named "directional stimulus" into the prompt, which acts as hints and clues to provide LLMs with fine-grained guidance toward the desired direction. For example, for the summarization task, the directional stimulus  $z$  might consist of keywords that should be included in the desired
summary. To generate this stimulus for each input query, we use a small tunable policy language model,  $p_{\mathrm{POL}}(\boldsymbol{z}|\boldsymbol{x})$ . We then use this generated stimulus,  $\boldsymbol{z}$ , along with the original input,  $\boldsymbol{x}$ , to construct the prompt that steers the LLM toward generating its output,  $p_{\mathrm{LLM}}(\boldsymbol{y}|\boldsymbol{x}, \boldsymbol{z})$ . It's important to note that the parameters of the LLM,  $p_{\mathrm{LLM}}$ , are kept frozen, as they are either inaccessible or inefficient to tune. Overall, when using the LLM with DSP to perform a downstream task, the output is obtained via  $\boldsymbol{y} \sim p_{\mathrm{LLM}}(\cdot|\boldsymbol{x}, \boldsymbol{z})$ ,  $\boldsymbol{z} \sim p_{\mathrm{POL}}(\cdot|\boldsymbol{x})$ .
2.1 Supervised fine-tuning
To train the policy model that generates directional stimulus for LLMs, we first perform supervised fine-tuning (SFT) on a pre-trained LM (e.g., T5, GPT-2, etc) on a small collection of labeled data. To collect the data, we could heuristically select or annotate the "pseudo-stimulus"  $z^{*}$  for each input query  $x$  and target output  $y$  pair based on the downstream task. For example, for the summarization task, we use keywords that the reference summary includes as pseudo-stimulus, while for the dialogue response generation task, we use dialogue acts that indicate the underlying meaning of the desired system response (see Section 3 for details). The resulting dataset  $\mathcal{D}' = \{(x,z^{*})\}$  consists of input-stimulus pairs. We then fine-tune the policy model by maximizing the log-likelihood:
$$
\mathcal {L} _ {\mathrm {S F T}} = - \mathbb {E} _ {\left(\boldsymbol {x}, \boldsymbol {z} ^ {*}\right) \sim \mathcal {D} ^ {\prime}} \log p _ {\mathrm {P O L}} \left(\boldsymbol {z} ^ {*} \mid \boldsymbol {x}\right). \tag {1}
$$
Supervised fine-tuning can provide a good initial point for the policy model. However, it is important to note that the heuristically selected or annotated pseudo-stimulus may not always be optimal, and the supervised fine-tuned policy model may not generate the most preferred directional stimulus for the LLMs toward the desired outputs. To overcome this limitation, we can also incorporate reinforcement learning (RL) to further fine-tune the policy model. By directly optimizing the LLM's output toward desired targets, RL training enables the policy model to explore and generate more effective directional stimulus.
2.2 Reinforcement learning
Optimization objective Our goal is to steer the LLM's generation toward the desired target by maximizing an alignment measure  $\mathcal{R}$ , which can take various forms such as downstream task performance measures (e.g., ROUGE score for summarization), human preferences, or other customized measures. Mathematically, we aim to maximize the below objective:
$$
\mathbb {E} _ {\boldsymbol {x} \sim \mathcal {D}, \boldsymbol {z} \sim p _ {\mathrm {P O L}} (\cdot | \boldsymbol {x}), \boldsymbol {y} \sim p _ {\mathrm {L L M}} (\cdot | \boldsymbol {x}, \boldsymbol {z})} [ \mathcal {R} (\boldsymbol {x}, \boldsymbol {y}) ]. \tag {2}
$$
Since the parameters of the black-box LLM are not accessible or tunable, we resort to optimizing the policy model to generate the directional stimulus that guides the LLMs' generation toward maximizing the objective. To achieve that, we define another measure  $\mathcal{R}_{\mathrm{LLM}}$  that captures how well the LLM performs when conditioned on a given stimulus  $z$ :
$$
\mathcal {R} _ {\mathrm {L L M}} (\boldsymbol {x}, \boldsymbol {z}) = \mathcal {R} (\boldsymbol {x}, \boldsymbol {y}), \boldsymbol {y} \sim p _ {\mathrm {L L M}} (\cdot | \boldsymbol {x}, \boldsymbol {z}). \tag {3}
$$
This allows us to cast the original objective of maximizing  $\mathcal{R}$  into optimizing the policy model to generate stimulus that maximizes  $\mathcal{R}_{\mathrm{LLM}}$ . By doing so, the LLM is effectively used as an evaluation function to guide the policy model toward generating more effective directional stimulus. Thus, the optimization objective for LLMs in Equation 2 is equal to the optimization objective for the policy model:
$$
\max  _ {p _ {\mathrm {P O L}}} \mathbb {E} _ {\boldsymbol {x} \sim \mathcal {D}, \boldsymbol {z} \sim p _ {\mathrm {P O L}} (\cdot | \boldsymbol {x})} \left[ \mathcal {R} _ {\text {L L M}} (\boldsymbol {x}, \boldsymbol {z}) \right]. \tag {4}
$$
RL formulation However, the above optimization is intractable for the policy model. To address the issue, we formulate the policy model optimization as an RL problem and employ proximal policy optimization (PPO) [59]. We use the policy model to initialize a policy network  $\pi_0 = p_{\mathrm{POL}}$  and then update  $\pi$  using PPO. The process that the policy model generates a sequence of tokens as stimulus  $z$  can be seen as a Markov decision process (MDP)  $\langle S, \mathcal{A}, r, \mathcal{P} \rangle$  with a state space  $\mathcal{S}$ , action space  $\mathcal{A}$ , reward function  $r$ , and state-transition probability  $\mathcal{P}$ . In each time step  $t$  of an episode, the agent selects an action (token) from the vocabulary  $\mathcal{V}$  according to the distribution of the current policy network  $\pi(z|x, z_{<t})$ . The episode ends when an end-of-sequence token is selected, and the stimulus  $z$  is generated. We can fine-tune the policy network  $\pi$  by optimizing the reward  $r$ :
$$
\mathbb {E} _ {\pi} [ r ] = \mathbb {E} _ {\boldsymbol {x} \sim \mathcal {D}, \boldsymbol {z} \sim \pi (\cdot | \boldsymbol {x})} [ r (\boldsymbol {x}, \boldsymbol {z}) ]. \tag {5}
$$
Reward function Recall that our goal is to maximize the objective in Equation 4, which can be used as the reward  $r$ . To keep the policy network  $\pi$  from moving too far from the initial policy model  $p_{\mathrm{POL}}$ , we also add a KL-divergence penalty reward. Therefore, the final reward becomes:
$$
r (\boldsymbol {x}, \boldsymbol {z}) = \mathcal {R} _ {\mathrm {L L M}} (\boldsymbol {x}, \boldsymbol {z}) - \beta \log \frac {\pi (\boldsymbol {z} | \boldsymbol {x})}{p _ {\mathrm {P O L}} (\boldsymbol {z} | \boldsymbol {x})}. \tag {6}
$$
Following [80, 54], we dynamically adapt the coefficient  $\beta$  during training:
$$
\boldsymbol {e} _ {t} = \operatorname {c l i p} \left(\frac {\mathrm {K L} \left(\pi_ {t} , p _ {\text {P O L}}\right) - \mathrm {K L} _ {\text {t a r g e t}}}{\mathrm {K L} _ {\text {t a r g e t}}}, - 0. 2, 0. 2\right), \tag {7}
$$
$$
\beta_ {t + 1} = \beta_ {t} \left(1 + K _ {\beta} \boldsymbol {e} _ {t}\right). \tag {8}
$$
Implementation To optimize the policy network  $\pi$ , we use the NLPO version of PPO from [54], which is specifically designed for language generators. To address the issue of large action spaces in PPO, NLPO learns to mask out less relevant tokens in the vocabulary using top- $p$  sampling. This technique restricts the action space to the smallest set of tokens whose cumulative probability is greater than the given probability parameter  $p$ , which we set to 0.9 in our experiments. Both the policy network  $\pi$  and value network are initialized from the supervised fine-tuned policy model  $p_{\mathrm{POL}}$ , with the final layer of the value network randomly initialized to output a scalar value using a regression head.
Our proposed framework DSP can be flexibly applied to various types of LMs and generation tasks. In this work, we focus on 1) summarization, 2) dialogue response generation, and 3) chain-of-thought reasoning tasks. We mainly use pre-trained T5 or Flan-T5 [53, 11] to initialize the policy model and experiment with the black-box LLMs including ChatGPT (gpt-3.5-turbo), Codex (code-davinci-002), and InstructGPT (text-davinci-002).
3.1 Summarization
Recent studies [16, 75, 4] have shown that LLMs, such as GPT-3, InstructGPT, and ChatGPT, are capable of generating high-quality summaries with zero- or few-shot prompting. However, their reference-based evaluation benchmark performances, such as ROUGE scores, still lag behind finetuned methods, indicating that the generated summaries may not completely match the style and emphasis of the reference summaries. In our experiments, we seek to guide LLMs to generate summaries that more closely align with the reference summaries by providing keywords that should be mentioned in the desired summaries as hints. We evaluate the effectiveness using metrics that compare the generated summaries against reference summaries. Notably, other desired directions, such as better alignment with human preferences, can also be pursued.
Dataset and evaluation We conduct our experiments on the CNN/Daily Mail dataset, a widely-used news summarization benchmark. To keep the cost of API usage low, we train on a subset of 1,000, 2,000, and 4,000 article-summary pairs from the total 287,113 samples in the training set. For evaluation, we randomly select 500 samples, following previous work [16, 65], which has been proven to provide sufficient statistical power [8]. We use the overlap-based metrics, including ROUGE [33], BLEU [47], and Meteor [3], and the similarity-based metric, BERTScore [74], to compare the generated summaries with the references. The reported evaluation scores are averaged over three inferences of ChatGPT for each query, using a temperature of 0.7 and top_p of 1.0. We use the same three demonstration examples in the prompt for standard prompting and add keywords as directional stimulus in the prompt for our approach, DSP. The exact prompts used in our experiments are provided in the Appendix.
Supervised fine-tuning details We use keywords as the pseudo-stimulus to train the policy model with supervised fine-tuning as discussed in Section 2.1. To collect the data, we employ textrank [41, 5] to automatically extract the keywords from the article and summary and only keep those that appear in the reference summary. As a result, we obtain a list of extracted keywords for each article-summary pair in the dataset. To convert them into a sentence that serves as the stimulus, we concatenate them using a split token “;”, resulting in the stimulus formatted as “[Keyword1]; [Keyword2]; ...;
Figure 3: Performance comparison of ChatGPT with standard prompting and DSP trained with SFT and SFT+RL, using varying numbers of training samples from the CNN/Daily Mail dataset.
[KeywordN].". We use the constructed article-stimulus pairs to train the policy model via supervised fine-tuning. The input format for training is "Extract the keywords: [Article]", while the output is the target stimulus consisting of keywords. The policy model was trained for 5 epochs with a  $2 \times 10^{-5}$  learning rate.
RL training details As we aim to guide ChatGPT in generating summaries that more closely match the reference summaries, we adopt the automatic reference-based metric scores as the alignment measure reward. Specifically, we calculate the ROUGE-Avg score between the generated summaries and the reference summaries as the reward, with a rescaling coefficient of 10. We experimentally found that other automatic evaluation metrics, such as BLEU and Meteor, perform similarly. To reduce variance, we generate four outputs per input query using ChatGPT with a temperature of 0.7 and compute the average reward. Additionally, we assign a step-wise reward, which we found could improve the efficiency and stability of the training process. Specifically, the policy model generates a sequence of keywords in each episode, during which we assign a reward of 1 if a keyword appears in the reference summary and a penalty reward of -0.2 is given otherwise. We train the policy network for 51k episodes, with 5 epochs per batch, a batch size of 8, and a learning rate of  $2 \times 10^{-6}$ . The KL_target and  $\beta_0$  in Equation 7 are set to 0.5 and 0.005, respectively.
Results We evaluate the performance of ChatGPT with standard prompting and our approach DSP trained with SFT or SFT and then RL (SFT+RL) on varying sizes of training data and present the results in Figure 3. As can be seen, all the evaluation scores improve with our proposed DSP compared with standard prompting. Specifically, the supervised fine-tuned policy model generates the stimulus that effectively guides ChatGPT to generate summaries that closely align with the reference summaries, leading to improved benchmark performance. Furthermore, the additional fine-tuning of the policy model with RL results in further performance improvement, indicating the effectiveness of RL in exploring better directional stimulus that maximizes the reward. As the size of the training data increases, the performance improvement becomes more significant. Despite using a small
Figure 4: Training curve on 1000 samples from the CNN/Daily Mail dataset.
collection of only 1,000 to 4,000 samples to keep API usage costs low, our DSP approach still
consistently enhances ChatGPT's ROUGE, BLEU, and Meteor scores by 1-2 in absolute points, even though ChatGPT has already achieved considerable performance. However, due to the discrepancy between the semantic-based metric BERTScore and the overlap-based metric ROUGE, which are used as the reward, the improvement in BERTScore after RL training may be relatively less significant. Figure 4 presents the change of training rewards and ROUGE-1 score on the validation set during the training process on 1,000 samples. We can see that the performance is closely related to the training rewards, and the training is relatively stable using the NLPO algorithm.
3.2 Dialogue response generation