SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We use our approach to generate instance-specific chain of thought (CoT) trigger prompts. Following previous work [26, 79], we evaluate two widely used arithmetic reasoning datasets MultiArith [57] and AQuA [35]. We compare with the 14 human-crafted chain-of-thought prompts evaluated in [26], part of which are collected from [1, 56]. We also compare with the prompt automatically designed by the APE approach [79]. We use the The hyperparameters used in our experiments are detailed in Table 6.
Model Params	Hyperparameter values
Supervised fine-tuning (SFT)	batch size: 16
epochs: 2
learning rate: 0.00002
learning rate scheduler: linear
weight decay: 0.01
RL (NLPO)	steps per update: 5120
total number of steps: 51200
batch size: 16
epochs per update: 5
learning rate: 0.000002
entropy coefficient: 0.0
initial kl coeff: 0.001
target kl: 0.5
discount factor: 0.99
gae lambda: 0.95
clip ratio: 0.2
value function coeff: 0.5
rollouts top k: 50
top mask ratio: 0.9
target update iterations: 20
Tokenizer	padding side: right
truncation side: right
max length: 128
Policy LM decoding	sampling: True
temperature: 0.7
max new tokens: 32
LLM decoding	sampling: True
temperature: 0.7
top_p: 1.0
max new tokens: 32
Table 6: Hyperparameters for experiments on the zero-shot chain-of-thought reasoning.
B.1 Summarization
(a) Number of generated keywords
(b) Keyword precision and ROUGE-1
Figure 5: Number of generated keywords, keyword precision, and summary ROUGE-1 during the training process on 4000 samples.
Analysis of generated hints/keywords We outlined changes in the number of generated keywords, hit keywords (those matched in the reference summary), and corresponding ROUGE-1 scores throughout the training process in Figure 5. As the training progresses, the policy model appears to generate keywords with increasing precision, which aligns positively with the increasing ROUGE-1 score. However, it is observed that even when keywords are generated with high precision if their quantity is too limited, the performance doesn't necessarily improve.
Figure 6: Part-of-Speech (POS) and Named Entity Recognition (NER) tagging on the generated hints, i.e., keywords.
We also employed the spacy package [20] for Part-of-Speech (POS) and Named Entity Recognition (NER) tagging on the generated keywords. The results are shown in the Figure 6. For the POS tagging, we observe that nouns (NOUN) and proper nouns (PROPN) are the most frequently generated keywords, which can serve as informative keywords. As for the NER tagging, the most commonly generated keywords include persons (PERSON), geopolitical entities (GPE), dates (DATE), organizations (ORG), and numerals (CARDINAL).
GPT-4 Evaluation To gain a better understanding of generated summaries guided by keywords, we employed GPT-4 to evaluate the summaries. It has been shown that the LLM, especially GPT-4 is able to produce consistently high-quality assessments of text generation, showing high human alignment and thus being a good alternative to human evaluations [78, 38]. As we employ ROUGE scores as rewards for tuning the policy model to generate keywords that guide the LLM towards generating summaries more aligned with the reference summary, we leveraged GPT-4 to assess the overlap of key points (hints) between generated and reference summaries. Specifically, we use GPT-4 to compare the summaries generated with our proposed DSP and the original standard prompting. GPT-4 was instructed to first generate an explanation, followed by the corresponding answer. We prompt GPT-4 as follows:
You are provided with an article and a corresponding reference summary. Additionally, there will be two alternative summaries labeled as 'A' and 'B'. Your task is to identify which of the two summaries (A or B) is more similar to the reference summary. This similarity should be evaluated based on the presence and accuracy of key points from the reference summary in each alternative summary. Please detail your reasoning in an explanation. After your explanation, classify the task outcome as: select 'A wins' if Summary A aligns more closely with the reference summary, 'B wins' if Summary B aligns more closely, or 'Tie' if both summaries align equally well with the reference summary.
The GPT-4 evaluation results are shown in Figure 7. We found that GPT-4 can produce reasonable and detailed explanations of their assessment. From our test set of 500 samples: DSP-generated summaries were favored 255 times (51.0%), summaries generated with original standard prompting were favored 222 times (44.4%), while a tie was observed in 23 cases (4.6%).
Zero-shot prompting In our main experiments, we employ few-shot prompting with 3 examples in the prompt during training and evaluation. The specific prompt and demonstration examples utilized are detailed in Appendix D. To test whether the approach performed well under the zero-shot setting, we evaluated the following two experimental settings on the CNNDM dataset with 4,000 training samples: (1) few(3)-shot during training and zero-shot during evaluation; and (2) zero-shot during both training and evaluation.
As shown in Figure 8, when both training and testing are conducted using zero-shot prompting, the performance improvement over standard prompting is still comparable to the scenario where both are conducted using few-shot prompting. In addition, we observed that our approach exhibits
Figure 7: GPT-4 evaluation on comparing Figure 8: Zero-shot evaluation results. DSP (0-shot) the summaries generated with our approach denotes that we use 0-shot prompting during RL DSP, i.e., with the guidance of our gener-training and DSP (3-shot) indicates we use 3-shoted keywords, and the original standard prompting during RL training. prompting, i.e., without keyword guidance.
Table 7: Low-resource evaluation on the MultiWOZ 2.0 dataset, where Succ. and Comb. denote the Success and Combined Score metrics, respectively.
Method	1% of training data (80 dialogues)	10% of training data (800 dialogues)
Inform	Succ.	BLEU	Comb.	Inform	Succ.	BLEU	Comb.
DAMD [77]	34.4	9.1	8.1	29.9	55.3	30.3	13.0	55.8
Soloist [49]	58.4	35.3	10.6	57.4	69.9	51.9	14.6	75.5
PPTOD [63]	74.4	52.4	13.0	76.4	84.4	68.4	15.6	92.0
UBAR [72]	-	-	-	-	82.5	66.6	17.7	92.3
GALAXY [19]	-	-	-	-	90.0	75.9	17.5	100.2
Codex
Standard Prompting	76.7	41.5	7.7	66.8	76.7	41.5	7.7	66.8
DSP w/ SFT	74.9	66.3	11.1	81.7	79.4	71.9	11.3	87.0
DSP w/ SFT+RL	91.0	76.0	9.8	93.3	96.0	86.9	10.7	102.2
ChatGPT
Standard Prompting	71.8	44.1	10.5	68.4	71.8	44.1	10.5	68.4
DSP w/ SFT	76.6	66.5	11.2	82.8	72.7	64.7	11.8	80.5
DSP w/ SFT+RL	90.9	82.2	10.2	96.7	95.3	82.3	10.9	99.6
robustness when different numbers of examples are used in prompts during training and evaluation, as our approach with few(3)-shot training still outperforms standard prompting under zero-shot testing.
B.2 Dialogue response generation
Low-resource results In addition to the performance of compared baseline models with full training data as shown in the main paper, we also present their performance in the low-resource setting in Table 7. It is important to note that most of these methods struggle to achieve acceptable performance with only  $1\%$  of the training data (80 dialogues), and thus their results in the  $1\%$  setting are not reported. As for those with reported performance with 80 dialogues, their results are significantly worse compared to Codex and ChatGPT guided by the policy model. Furthermore, even with around 800 dialogues, their Inform and Success rates were still much lower than those achieved by ChatGPT and Codex.
B.3 Chain-of-Thought reasoning
Table 8: Some generated trigger prompts by our fine-tuned policy model.
Generated CoT Trigger Prompts
First step:
Let's think like a detective step by step. First,
Let's solve this problem by splitting it into steps. First,
Let's think step by step using common sense.
Let's think step by step using our creative brains.
Let's think step by step using both the above information and the testing.
Let's think step by step using proven methods.
The answer is following the proof.
Newly discovered prompts After fine-tuning with RL, the policy model is encouraged to discover instance-specific trigger prompts. Table 8 showcases some of these newly generated trigger prompts, which deviate from those present in the training data for SFT. Some are modifications or combinations of prompts from the training data, such as "First step:" and "Let's think like a detective step by step. First". Others include new information, like "Let's think step by step using both the above information and the testing." and "Let's think step by step using proven methods".
C Running examples
We provide two running examples on the CNN/Daily Mail and MultiWOZ dataset in Table 9 and 10, respectively. For each example, we present the generations of ChatGPT with standard prompting, DSP trained with SFT, and DSP trained with SFT and RL.
The used prompts of standard prompting and our proposed Directional Stimulus Prompting on CNN/Daily Mail and MultiWOZ datasets are given in Figures 9, 10, and Figures 11, 12, respectively. Both use the same three demonstration examples in standard prompting and DSP. In the case of the CNN/Daily Mail dataset, DSP incorporates additional keywords as hints (stimulus) in the prompts. For the MultiWOZ dataset, DSP includes the dialogue acts for each system turn as stimulus, along with explanations for all the dialogue acts.
Dialogue context	User: i want to find a moderate -ly priced restaurant. Assistant: i have [value_choice] available for you! is there a certain area or cuisine that interests you? User: yes i would like the restaurant to be located in the center of the attractions. [db_3]
Oracle dialogue acts	[restaurant] [inform] choice [request] food
Oracle system response	there are [value_choice] restaurant -s available in the [value_area]. how about a specific type of cuisine?
Standard prompting
Generated re-sponse (Chat-GPT)	the [value_name] is located in the center of the attractions . would you like me to book a table ?
SacreBLEU	14.09
Directional Stimulus Prompting (SFT)
Generated dialogue acts (T5)	[restaurant] [inform] food name area choice [recommend] price
Generated re-sponse (Chat-GPT)	i would suggest [value_name] in the [value_area] area . they have [value_food] cuisine and are moderate in price . are you interested ?
SacreBLEU	16.40
Directional Stimulus Prompting (SFT+RL)
Generated dialogue acts (T5)	[restaurant] [inform] choice [request] food
Generated summary (ChatGPT)	i have [value_choice] restaurants in the area . do you have a specific cuisine in mind ?
SacreBLEU	22.80
Table 10: An example from the MultiWOZ dataset. The policy model (T5) generates the dialogue acts given the dialog context. With our approach DSP, ChatGPT generates the response conditioned on the generated dialogue acts.
Standard Prompt (CNN/Daily Mail)
Given an article, write a short summary in 2-4 sentence.
Article: Seoul (CNN) South Korea's Prime Minister Lee Wan-koo offered to resign on Monday amid a growing political scandal. Lee will stay in his official role until South Korean President Park Geun-hye accepts his resignation. He has transferred his role of chairing Cabinet meetings to the deputy prime minister for the time being, according to his office. Park heard about the resignation and called it "regrettable," according to the South Korean presidential office. Calls for Lee to resign began after South Korean tycoon Sung Woon-jong was found hanging from a tree in Seoul in an apparent suicide on April 9. Sung, who was under investigation for fraud and bribery, left a note listing names and amounts of cash given to top officials, including those who work for the President. Lee and seven other politicians with links to the South Korean President are under investigation. A special prosecutor's team has been established to investigate the case. Lee had adamantly denied the allegations as the scandal escalated: "If there are any evidence, I will give out my life. As a Prime Minister, I will accept Prosecutor Office's investigation first." Park has said that she is taking the accusations very seriously. Before departing on her trip to Central and South America, she condemned political corruption in her country. "Corruption and deep-rooted evil are issues that can lead to taking away people's lives. We take this very seriously." "We must make sure to set straight this issue as a matter of political reform. I will not forgive anyone who is responsible for corruption or wrongdoing." Park is in Peru and is expected to arrive back to South Korea on April 27. CNN's Paula Hancocks contributed to this report.
Q: Write a short summary of the article in 2-4 sentences.
A: Calls for Lee Wan-koo to resign began after South Korean tycoon Sung Woan-jong was found hanging from a tree in Seoul . Sung, who was under investigation for fraud and bribery, left a note listing names and amounts of cash given to top officials .
A: Mother-of-two died following a 10-year battle with a brain tumour . She worked at BBC in Cardiff where husband claims she was bullied . Marie Csaszar gave evidence at BBC's 2013 Respect At Work review . He asked for emails under the Data Protection Act but was refused .
Article: (CNN) An American citizen was wounded by gunfire Thursday as she drove from the medical school in Karachi, Pakistan, where she works, police said. Debra Lobo, a 55-year-old California native, was shot in the right cheek and left arm and is unconscious but expected to survive, according to Mohamad Shah, a Karachi police spokesman. Police found pamphlets that the assailants had thrown into Lobo's car, written in Urdu, saying "America should be burnt," Shah said. Lobo had left the Jinnah Medical and Dental College, where she works as vice president, to pick up her two daughters from school. Two assailants on a passing motorcycle shot her while she was driving, Shah said. "Our U.S. Consulate General in Karachi is in close contact with Pakistani authorities and is working to obtain more information," said a U.S. Embassy spokesperson. Lobo is being treated at the Karachi's Aga Khan Hospital, said Shah. She has lived in Pakistan since 1996 and is married to a Christian Pakistani who is a librarian at the American School in Karachi. Karachi police are investigating, Shah said.
A: Debra Lobo, 55, is unconscious but is expected to survive after being shot Thursday, police say . She is vice principal of the Jinnah Medical and Dental College in Karachi . Police: She was on her way to pick up her daughters from school when she was shot .
Article: [[QUESTION]]
Q: Write a short summary of the article in 2-4 sentences. A:
Figure 9: The prompt for standard prompting on the CNN/Daily Mail dataset.
Directional Stimulus Prompt (CNN/Daily Mail)
Given an article and a list of keywords, write a short summary that accurately incorporates the provided keywords into 2-4 sentences.
Article: Seoul (CNN) South Korea's Prime Minister Lee Wan-koo offered to resign on Monday amid a growing political scandal. Lee will stay in his official role until South Korean President Park Geun-hye accepts his resignation. He has transferred his role of chairing Cabinet meetings to the deputy prime minister for the time being, according to his office. Park heard about the resignation and called it "regrettable," according to the South Korean presidential office. Calls for Lee to resign began after South Korean tycoon Sung Woan-jong was found hanging from a tree in Seoul in an apparent suicide on April 9. Sung, who was under investigation for fraud and bribery, left a note listing names and amounts of cash given to top officials, including those who work for the President. Lee and seven other politicians with links to the South Korean President are under investigation. A special prosecutor's team has been established to investigate the case. Lee had adamantly denied the allegations as the scandal escalated: "If there are any evidence, I will give out my life. As a Prime Minister, I will accept Prosecutor Office's investigation first." Park has said that she is taking the accusations very seriously. Before departing on her trip to Central and South America, she condemned political corruption in her country. "Corruption and deep-rooted evil are issues that can lead to taking away people's lives. We take this very seriously." "We must make sure to set straight this issue as a matter of political reform. I will not forgive anyone who is responsible for corruption or wrongdoing." Park is in Peru and is expected to arrive back to South Korea on April 27. CNN's Paula Hancocks contributed to this report.
Q: Write a short summary of the article in 2-4 sentences that accurately incorporates the provided keywords.
Keywords: Lee Wan-koo; resign, South Korean tycoon; Sung Woan-jong; hanging from a tree; investigation; notes; top officials.
A: Calls for Lee Wan-koo to resign began after South Korean tycoon Sung Woan-jong was found hanging from a tree in Seoul. Sung, who was under
investigation for fraud and bribery, left a note listing names and amounts of cash given to top officials
Keywords: Mother, battle; brain tumour; BBC; Cardiff; husband; bullied; Marie Csaszar; evidence; Respect At Work; emails; Data Protection Act.
Article: (CNN) An American citizen was wounded by gunfire Thursday as she drove from the medical school in Karachi, Pakistan, where she works, police said. Debra Lobo, a 55-year-old California native, was shot in the right cheek and left arm and is unconscious but expected to survive, according to Mohamad Shah, a Karachi police spokesman. Police found pamphlets that the assailants had thrown into Lobo's car, written in Urdu, saying "America should be burnt," Shah said. Lobo had left the Jinnah Medical and Dental College, where she works as vice principal, to pick up her two daughters from school. Two assailants on a passing motorcycle shot her while she was driving, Shah said. "Our U.S. Consulate General in Karachi is in close contact with Pakistani authorities and is working to obtain more information," said a U.S. Embassy spokesperson. Lobo is being treated at the Karachi's Aga Khan Hospital, said Shah. She has lived in Pakistan since 1996 and is married to a Christian Pakistani who is a librarian at the American School in Karachi. Karachi police are investigating, Shah said. © Writtle Publishing Co., Ltd.
Q: Write a short summary of the article in 2-4 sentences that accurately incorporates the provided keywords. Keywords: Debra Lobo; 55; unconscious; Thursday; vice principal; the Jinnah Medical and Dental College; K
A: Debra Lobo, 55, is unconscious but is expected to survive after being shot Thursday, police say . She is vice principal of the Jinnah Medical and Dental
College in Karachi . Police: She was on her way to pick up her daughters from school when she was shot
Keywords: [[HINT]]
Figure 10: The prompt for Directional Stimulus Prompting on the CNN/Daily Mail dataset. The difference compared with the prompts used in standard prompting shown in Figure 9 is the stimulus hints (keywords), which are highlighted in red.
Standard Prompt (MultiWOZ)
The following are conversations between a user and an assistant. The assistant can help the user with booking or providing information of restaurant, hotel, attraction, taxi, train, hospital, police, and so on.
Conversation:
User: do you book trains for the bishops stortford?
Assistant: yes. are you departing or arriving in [valuedestination]?
User: i am going there from cambridge on thursday
Assistant: i have [value choice] trains going to [value destination] from [value departure] on [value day]. what time do you need to arrive ?
User: i would like to arrive by 18:15
Assistant: [value_id] will have you in [valuedestination] by [value_arry]
User: i need 2 tickets
Assistant: the [value_id] will get you there by [value_arry]. i can book you [value_people] tickets on that if you would like?
User: 2 tickets will be fine, please
Assistant: tr0277 2 tickets'. the total fee is [value_price] payable at the station . reference number is : [value_reference]. will there be anything else i can help you with today ?
User: a nice place to eat that serves austrian food . in the centre of town if possible
Assistant: i am so sorry no [value_food] restaurant -s in that area , another food type perhaps ?
User: okay, do you have any good restaurant -s serving British food?
Assistant: there are [value_choice] restaurant -s in the [value_area] that serve [value_food] food , is there a price range you would like to stay in ?
User: any is fine , can i get the phone number and postcode of 1 you would recommend ?
Assistant: i would recommend the [value_name] . their phone number is [value_phone] . their postcode is [value_postcode]
User: great! thanks. that's all i need. goodbye
Assistant: awesome . we 're here 24 and 7 if you need us again . have a great day !
User: i ' m looking for a hotel in the moderate price range . i definitely want a hotel , not a guest house
Assistant: of course . there are [value_choice] [value_type] in the [value_area] and [value_choice] in the [value_area] of town . do you have a preference ?
User: is there anything to the west, preferably with a 4 star rating?
Assistant: i ' m sorry but nothing exists in [value_area] north with [valuestars] 4 star rateing in [value_price] moderate price range . could we try another area ?
User: is there 1 in the cheap range?
Assistant: no, there is not.
User: oh goodness . can you check once again for a hotel , star 4 , in the west . cheap is fine .
Assistant: the [value_name] is located at [value_address]. do you need more information or would you like to book now?
User: yeah book for 5 people
Assistant: for how many days?
User: will this start on friday ?
Assistant: yes . would you like reservations ? what day will it start ? how many people ?
User: i would like a reservation for 5 people for 4 nights , beginning friday , at the cambridge belfry .
Assistant: you are all booked and your reservation number is [value reference].
User: thank you for the reservation . i am also looking for a lovely park in north cambridge .
Assistant: i would suggest [value_name]
User: what is the phone number?
Assistant: the phone number is [value_phone]
User: i also need a taxi between the 2 place -s
Assistant: please give me the travel time , departure location and arrival location .
User: i want to leave the hotel by 04:30
Assistant: what time do you need to leave the restaurant by?
User: i want to be pickup at at the hotel by 04:30
Assistant: great your booking is complete . you will be picked up in a [value_car] by [value Leave] at [value departed] and dropped off at the
[valuedestination]
User: do you have their contact number?
Assistant: their contact number is [value_phone] . is there anything else i can help you with today ?
User: that will be all for today . thank you . goodbye
Assistant: thank you i'm glad i could help. goodbye
User: i want to find place -s to go in the centre of town that are park type attractions
Assistant: i think that you will like [value_name] . it is [value_price] , may i book it for you ?
User: yes, that sounds good! please provide me with the postcode
Assistant: i ' m sorry , we are n't able to book tickets for this attraction , but their postcode [value_postcode] cb21jf , and i can provide their address and phone
if you like?
User: no . that is all i need . i would like help finding a cheap guest house though
Assistant: i have [value_choice] options for you . do you need wifi or parking ? is there a part of town you prefer to stay in ?
User: it should include free wifi and should be in the east
Assistant: there seems to be an opening at [value_name]. would you like me to book you a room?
User: yes , please . 5 people for 5 nights starting thursday
Assistant: booking was successful. your reference number is : [value_reference]
User: thank you very much!
Assistant: you are welcome . is there anything else i can do for you ?
User: no thank you . i appreciate all of your help
Assistant: thank you i'm glad i could help.
[DIALOG]
Figure 11: The prompt for standard prompting on the MultiWOZ dataset.
Directional Stimulus Prompt (MultiWOZ)