STaR-GATE: Teaching Language Models to Ask Clarifying Questions
Authors: Chinmaya Andukuri
Jan-Philipp Fränken
Tobias Gerstenberg
Noah Goodman
Venue: COLM (2024)
Keywords: question asking, preference elicitation, expert iteration, self-improvement
TL;DR: We explore a language model's ability to self-improve by rewarding the model for generating useful questions.

Abstract: When prompting language models to complete a task, users often leave important aspects unsaid. While asking questions could resolve this ambiguity (GATE; Li et al., 2023), models often  struggle to ask good questions. We explore a language model's ability to self-improve (STaR; Zelikman et al., 2022) by rewarding the model for generating useful questions—a simple method we dub STaR-GATE. We generate a synthetic dataset of 25,500 unique persona-task prompts to simulate conversations between a pretrained language model—the $\texttt{Questioner}$—and a $\texttt{Roleplayer}$ whose preferences are unknown to the $\texttt{Questioner}$. By asking questions, the $\texttt{Questioner}$ elicits preferences from the $\texttt{Roleplayer}$. The $\texttt{Questioner}$ is iteratively finetuned on questions that increase the probability of high-quality responses to the task, which are generated by an $\texttt{Oracle}$ with access to the $\texttt{Roleplayer}$'s latent preferences. After two iterations of self-improvement, the $\texttt{Questioner}$ asks better questions, allowing it to generate responses that are preferred over responses from the initial model on $\textbf{72}$% of tasks. Our results indicate that teaching a language model to ask better questions leads to better personalized responses.

Published as a conference paper at COLM 2024
STaR-GATE: Teaching Language Models to Ask Clarifying Questions
Chinmaya Andukuri" Stanford University andukuri@stanford. edu
Jan-Philipp Fränken" Stanford University jphilippestanford.e edu
Tobias Gerstenberg Stanford University
berstenberg@sttnfordd.de
Noah D. Goodman
Stanford University
ngoodman@stanford.e
Abstract
When prompting language models to complete task, users often leave important aspects unsaid. While asking questions could resolve this ambi- guity (GATE; Li et al,, 2023), models often struggle to ask good questions. We explore a language model's ability to self-improve (STaR; Zelikman al., 2022) by rewarding the model for generating useful questions simple method we dub STaR-GATH We generate synthetic dataset of 25,500 unique persona-task prompts to simulate conversations between pretrained language model-the Questioner-and preferences are unknown to the Questioner. By asking questions, the Questioner elicits preferences from the Roleplayer. The Questioner is iter- atively finetuned on questions that increase the probability of high-quality responses to the task, which are generated by Oracle with access to the Roleplayer's latent preferences. After two iterations of self-improvement, the Questioner asks better questions, allowing it to generate responses that are preferred over responses from the initial model on 72% of tasks. Our results indicate that teaching language model to ask better questions leads better personalized responses.
Roleplayer whose
Introduction
When interacting with users who have dif- ferent preferences, language models (LMs) encounter task ambiguity (Finn et al., 2018; Tamkin et al., 2022). Depending on the user, the same request might correspond to dif- ferent task. For example, consider a user who asks an LM for a pasta recipe (Fig- ure 1). If the model could elicit informa- tion about the user's dietary restrictions, fa- vorite sauces, and preferred cooking meth- ods, it could tailor the recipe to their specific needs and desires. The model might sug- gest a vegetarian pasta recipe for a user who is vegetarian, or propose a traditional lasagna recipe for user with a passion for Neapolitan cuisine. However, if this information is the prompt, the model may generate a generic recipe that fails
What good pasta recipa?
What good pasta
pasta, broth... |wasted tokens
you have 00
any cietary restrictions?
am vegetarian:
am vegetarion.
Here vegetarian...
Here vegetarian....
[relevant tokens)
Ineffective Conversation
Effective Conversation
Figure Problem Illustration. When user prefer- ences are unknown, language models may respond ineffectively. By asking questions, models can elicit information and provide more effective responses.
not explicitly specified to account for the user's unique preferences and constraints. In high-stakes domains like healthcare or education, such task ambiguity can have significant consequences.
One approach to resolving task ambiguity is by asking targeted questions to elicit relevant information from users. Prompting closed-source LMs can yield useful questions (e.g., Li
* "qqull contribution.

---

Published as a conference paper at COLM 2024
P(Gold Response Conversations)
Gold Response
Elicitation
Filtering
Conversations
Conversation
Good /
Question
Questions
Answers
Regularization
w/ Gold Response
Response
Questions
Finetune
Questioner
Questioner
Raleplayer
Task
Start
Figure Overview of STaR-GATE. A task given Questioner who elicits preferences from a Roleplan ayer whose persona is unknown to the Questioner. The resulting conver- sations are then filtered based on the log probability of a gold response generated by an Oracle which has access to the Roleplayer's persona (omitted from the diagram for clarity). We then fine-tune the Questioner on the filtered questions. Moreover- avoid distribution shift-we regularize the Questioner by additionally sampling responses conditioned on the filtered conversations. In our ablations, we contrast fine-tuning on sampled responses with fine-tuning on the gold responses.
et al., 2023; Piriyakulkij et al., 2023). However, this approach is inflexible in guiding a model's questioning strategy and frequently generates queries that are ineffective irrelevant for the task at hand. Indeed, it is likely that current alignment strategies- suchhssuchhhh as RLHF- specifically inhibit the ability to carry out such dialog (Shaikh et al., 2023). One recent effort redsesses these limitations by combining elicitation with optimal experimental design methods (Handa et al., 2024). However, this approach constrains questions to pairwise comparisons over fixed set of features, substantially limiting the space of questions that can be used to probe user preferences. Another approach is to use offline reinforcement learning to encourage useful dialog (Hong et al., 2023). This is promising but requires offline generation of high-quality dialog from an expert model, and has not targeted questions for preference elicitation specifically.
In this paper, we explore whether we can improve LM's ability ask useful questions by bootstrapping with form of self-play (Silver et al., 2017; Anthony et al., 2017). We introduce STaR-GATE (Figure 2), an iterative algorithm that combines active preference elicitation (GATE; Li et al., 2023) with self-improvement loop inspired by STaR (Zelikman et al., 2022). We address several technical challenges: (1) We define a task setting for improving elicitation for which generate synthetic dataset of 25, 55,500 unique persona-task prompts; based on the log probability of gold responses generated by oracle model (with access to the persona); and (3) We encourage the LM to use the elicited information while avoiding distribution shift through response regularization. We find that questions asked by the finetuned model increase the probability of gold responses consistently across iterations (Figure 4). Moreover, compared to responses generated by the
(2) We define a reward
initial model, responses generated by STaR-GATE finetuned model have 72% win rates (Figure 3a) after two iterations.
In summary, we make the following contributions: (1) We introduce STaR-GATE, simple algorithm that iteratively improves a LM's ability elicit user preferences by asking questions. (2) We generate synthetic dataset consisting of 25,500 unique persona-task response prompts. (3) We show that finetuning with STaR-GATE enables a LM to generate questions that significantly increase the probability of generating gold responses. (4) We show that adding response-regularizatin to use the elicited preferences to generate better responses-a high win rate against the initial model. (5) We show that the finetuned model generalizes beyond the roleplayer it was trained with.
to STaR-GATE yields fine-tuned model able
2

---

Published as a conference paper at COLM 2024
Ablation
Roleplayer
GATR
w/ Gold Reap.
nixtral-Bx7b
genma-78
w/o Rog.
mistral-7b
0.8
0.8
0.7-
0.7-
Win Rate
0.6
0.6
0.3
0.3
0.2
0.2
0.1
0.1
0
0
Iteration tt
Iteration (t)
[a] Ablation Results
[b] Generalization Results
Figure 3: Win Rates Against Initial Model. [a] Complete method and ablations: w/o Reg. refers to finetuning on questions only, which decreases the model's ability to generate answers. w/ Gold Resp. refers to finetuning directly on the gold responses rather than self- generated responses, which leads to hallucinations in generated answers. [b] Roleplayer generalization results. We demonstrate that STaR-GATE generalizes beyond the roleplayer it was trained against of their respective models. Error bars represent the standard error of the mean (+ SEM). We include 0.5 (chance) as a reference point for iteration i = O.
All three roleplayers correspond to the instruct version
2 Related Work
Preference Optimization. Preference optimization algorithms, such as RLHF (Christiano et al., 2017), DPO (Rafailov et al., 2024), or KTO (Ethayarajh et al., 2024), optimize LMs to provide single-turn dialog responses that reflect preferred high-utility outcomes. As result, these models learn distributions over responses that effectively generate answers to user queries without requiring additional information beyond the initial prompt. However, asking follow-up questions to elicit user preferences is essential for understanding their unique needs and desires, especially when faced with task ambiguity (Tamkin al., 2022; Li al., 2023). Despite the importance of follow-up questions for effective communication, recent research has shown that preference optimization algorithms can reduce a LM's ability to ask follow-up questions. Specifically, RLHF has been found to negatively correlate with LM's attempts to ask follow-up questions or show acknowledgment (Shaikh al., 2023). This limitation can be problematic for high-stakes domains such as healthcare (Thrrunvukkarasu et al., 2023) or education (Kasneci et al., 2023). where resolving task ambiguity through effective questioning is crucial for effective dialog.
Preference Elicitation with LMs. One way of resolving task ambiguity is by prompting a LM to ask questions or infer user preferences from observations (Li al., 2023; Piriyakulkij et al., 2023; Lin et al., 2023; Fränken et al., 2023; Handa et al., 2024; Rao & Daumé III,
2018; 2019; Aliannejadi et al., 2021; Mostafazadeh et al., 2017). For example, Li et al. (2023) used LMs themselves to elicit user preferences during interaction (GATE; Li et al., 2023).
In GATE (short for Generative Active Task Elicitation), LM elicits and infers intended behavior through free-form, language-based interaction. Unlike non-interactive elicitation approaches, such as prompting (Brown et al., 2020), which rely entirely on the user specify their preferences, generative elicitation probes nuanced user preferences better. Across domains such as content recommendation and email verification, generative elicitation with LMs requires less effort than prompting while being comparable to or better than user- written prompts (for further details, see Section Li et al., 2023). Building upon GATE, Handa et al. (2024) introduced OPEN, framework that combines LM-driven elicitation with Bayesian Optimal Experimental Design (BOED) to select informative questions and translate abstract queries into natural language. OPEN combines the advantages of LMs and Bayesian methods to recommend news articles. OPEN is better at eliciting human preferences than approaches that only use LMs or BOED. Piriyakulkij et al. (2023) combine
3

---

Published as a conference paper at COLM 2024
Algorithm 1 STaR-GATE
1: Input QBASE: pretrained LM; tasks T {(4)) personas U {11))] and gold
responses G gij for ij EE {1,.., x {1,..,
2: for y in 1...N do
> Outer loop D Simulate
3:
(sf) with c E [1] + Qy 144,4)) for ij E (1,.., D) x (
multiple conversations for each (ij)
4: Sq = (s])} + arg 4 log PQasse (giyl with c E [1,10] for ij E {1, D} x
(1,..., C} b Filter conversations with highest log probbabbiilltses of generating the gold responses according to the original model QBASE
5:
R = (
& Generate model responses conditioned on the best selected
conversations for all ij using Qy 1
6: Qn f train(QBASE/ Sy, Ry) Finetune the original model on selected conversations
and model responses
7: end for
LMs with probabilistic reasoning to select informative questions that maximize information gain about user preferences in a simplified web shopping task (Yao et al., 2022). Relatedly, Hong et al. (2023) demonstrated that instead of eliciting information directly, it is also possible to prompt large LM such GPT-3.5 to simulate conversations between human and an assistant, and revise 2022). This approach allowed the authors to finetune much smaller GPT-22 model (Radford et al., 2019) to become capable conversationalist. While all of the above approaches have resulted in significant improvements, they rely on proprietary models for both elicitation and generation of synthetic data for downstream finetuning.
revise the simulated conversation with Constitutional Al (Bai et al.,
Self-Improving Reasoning. We are interested in training a LM to better elicit preferences using its own reasoning capabilities. To do so, we draw upon recent work showing that LMs can self-improve. For example, Self-Taught Reasoner (STaR; Zelikman et al., 2022) demonstrated that LM which was trained iteratively on its own reasoning traces for correct answers could solve increasingly difficult problems. By combining rationalization (i.e., reasoning backwards from answer; see also Rajani et al., 2019) with supervised finetuning on rationales leading to correct answers, pretrained LM achieves strong performance on datasets such as Commonsense@A (Talmor et al., 2018). Recently, V-STaR (Hosseini al., 2024) extended this idea by using both correct and incorrect reasoning traces, essentially attempting to merge STaR with DPO. Relatedly, TRICE (Hoffman al., 2024) frames the process of generating better chains of thought as a latent-variable inference problem and maximizes the marginal log-likelihood of correct answers. Other relevant works include learning intermediate reasoning for mathematical statements (Poesia et al., 2023), learning from reasoning mistakes (Shinn al., 2024; Zhang et al., 2024), learning to follow constitutions (Fränken et al., 2024), teaching LMs to reason in planning (Gandhi et al., 2023; Qiao et al., 2024), and Quiet-STaR (Zelikman et al., 2024), a generalization of STaR which generates rationales at each token to explain future text. Inspired by these developments, we use self- iefl-immprveeentt techniques to teach a LM to ask effective questions for eliciting user preferences.
3 STaR-GATE
Overview. On a high level, STaR-GATE starts with persona-task prompts and generates gold responses with an Oracle that has access to both the persona and the task. Given this setup, we simulate conversations between Questioner and a human Roleplayer timl-similar to the Oracle-has access to the user persona which is unknown to the Questioner. The task of the Questioner is to elicit useful information from the Roleplayer, whereby usefulness is measured as the log probability of the gold response conditional on the questions asked by the Questioner and the preferences elicited from the Roleplayer (see Figure 2).
4

---

Published as a conference paper at COLM 2024
Objective. Let Q denote the Questioner (i.e., the policy to train), R the Roleplayer model, T the set of tasks, Ll the set of user personas, and O the Oracle model. Given a task fę E T and persona 2j E U, the Oracle O generates a gold response 8ij Po uj). The objective of STaR-GATE is to maximize the expected log probability that the prefrained model QB assigns to the gold response 8ij, given the task I and a simulated conversation Sij between Q and R:
T ΣE 1=
IQRLU
Es, log Poess (8iy) Si)))
(1)
Here Sij = Iij., Bijko hijk is a simulated conversation of questions qijk distributed according to Po 1,, qia1, hijt...., hiii 1) and answers distributed according to Pa (e(Npp)) 9ij1, Mij....... (.p))....99)))
Optimization. Equation can be optimized in a variety of ways. Following Zelikman et al. (2022), we use simple variant of Expert Iteration (Anthony et al., 2017). On each overall iteration, 1), for each pair (fi, Hj)), we sample N trajectories of simulated conversations, Sijn using the current Qy. We then select the top-k trajectories (here, k = 1) based on the objective, and do supervised fine-tuning for this set from the initial QBASE.
Regularization. An important failure mode of optimizing this objective is that by training the policy Q to ask good questions may forget how to respond and instead always ask questions. This behavior is not useful in practice, as we want model that is not only good at asking questions to elicit user preferences but also one that uses the elicited preferences to give good responses. To address this issue, we add to Equation 1 a regularization term preventing the distribution of responses (not questions) from moving too far from the previous iteration: ( (r)4, sij) Pe, (r)4;, In practice this can be accomplished by simply sampling a response (at temperature т 0) from the previous policy rij (ry-1(1/4 for each task and persona pair, conditioned on the best conversation history sip then appending this response to the conversation history during fine-tuning.
Algorithm. We provide an outline of STaR-GATE in Algorithm 1. We perform expert iteration, training the initial model QBASE times (N = 3 for all experiments) on question- response pairs generated from each intermediate model Q1, Q2, At each iteration !,, we alternate between task splits Ta and TB, as well as persona splits UA and Up, to prevent generating new data for tasks or personas present during training. For each task 4; and user persona Uj, we simulate conversations (N 10 for all experiments), each having maximum of K total turns (K 3 for all experiments). When generating Sij for all (ij), we sample hijk) at each turn k from the previous Qy-1 and fixed R. To achieve roughly uniform distribution of conversation lengths and prevent overfitting on conversations of single length, we set the termination point to be uniform across K.
As indicated above, we select the best simulated conversations for finetuning the next itera- tion according to the objective PQBASE (giy S;j)). We then fine-tune the initial model QRASE on both the selected conversations syj and the greedily sampled responses rij, ensuring that the model learns ask informative questions and provide personalized responses. Critically, we mask the answers / from the loss, finetuning the question-generation and response-generation policy but not learning to imitate answers.
4 Elicitation Task
Overview. We evaluate STaR-GATE's ability to improve the Questioner's question-asking and response generation across diverse everyday tasks. We find that training with STaR- GATE increases both the log-probability of gold responses and win rates compared to the initial (pretrained and instruction-finetueed model. Code to reproduce experiments is available at https:/ / /gg thub. cow/scandukuriiiiccccccc.ccm /ssissantt-atte
5

---

Published as a conference paper at COLM 2024
To cover broad range of everyday life tasks, we selected the first 550 conversations from the open-source
staattproomt-ddttsse which we divided into two train splits (Ta, T3) each of N = 250 and one test split N = 50. Importantly, we only selected the human queries (not the assistant responses) for each conversation and used these as the tasks E T to seed given simulation. We selected dataset as it covers broad range of queries, from questions about food (e.g,, "What type of wine goes best with steak?"" "), to career questions (e.g., "I'm having trouble finding the perfect job. What resources can help me?" "), and education (e-g,, "I'm curious aboud quantum computing. Can you tell me the basics of how it works.") "). See Appendix A.2 for further details.
wawruuttttmaaaaasiissi passistatt-promp
Persona Generation. We generate personas E U with GPT-4 by few-shot (N = 2) prompting with randomly sampled personas from a set of 21 content-filtere personas from the PRODIGy dataset (Occhipinti et al., 2023). We generated a total of 110 personas and split personas into two train splits (UA, Up) each with 50 personas, and one test split of N = 10. Example personas and prompts are provided in Appendix A.3.
Gold Responses. To generate a gold response 8ij for each (f; U;) pair, we prompted an Oracle (GPT-4). Specifically, we provided the persona 2j followed by the task f,, without any dialog history, and prompted the Oracle to generate a personalized response that completes the task with respect to the persona profile. This process resulted in total of 25, 500 pask-persona- -oaaa-ool--dn gold-response triples (250 x 50 + 250 x 50 + 50 x 10). Prompt details and examples are provided in Appendix A.4.
5 Evaluation and Results
We evaluate the performance of the Questioner Q at each iteration '/ using two metrics: log-probabilities of generating the gold responses and win rates.
Models. We use mistral-7b- -strra--b-instruct as our Questioner. We chose mistral 1eal---b-nnsrrct a 7B-parameter model, because its weights are openly available and it has been shown to be one of the best models for its size (Jiang et al., 2023), outperforming larger models such as 1lama-138 (Iouvron et al., 2023) on benchmarks like MT-Bench (Zheng et al., 2024). We use GPTT4 (OpenAI, 2023, gpt-4-0613 snapshot) our dataset, GPT-4 was the most capable model available. For the Roleplayer, we use mixtral-3x7bbinstruue (liang et al., 2024).
Oracle, at the time of generating
Gold Log-probabilities. Our main training objective is learning to elicit information that increases the log probability b of the gold responses according to the initial (pretrained and instruction-fiettunedl model QBASE- For our evaluations, we thus first compute the log
(n)
probability of gold responses gij, conditioned on simulated conversations s generated by the current model Q,, and fixed roleplayer R, for a held-out test set of tasks f, and personas Uj. We calculate log probabilities for four conditions:
(1) Negative Control:
log PQBASE (gty fi), performance of the pretrained model without any information about the Roleplayer (persona or elicited), (2) Positive Control: log PQRASE (gij j t;), performance of
the pretrained model given oracular information about the persona, (3) Q-Experimental:
log PQaASE (81y14 tú (n)), evaluation of the STaR-GATE finetuned model, and (4) Q-Random:
log PQRASE (g,]] f/ s f a baseline that randomizes persona info used in answering elicita- tion questions (r indicates random different test persona). In prompting both Q-Random and Q-Experimental (the main condition), we repeat the task text I; at the end of the conver- sation to prompt final response instead of asking another elicitation question. The purpose of the Q-Random baseline is to isolate the relevance of persona-specifie information from generally informative information elicited from the Roleplayer.
1 instrrt-hhmman-ssiitattproo
We hand-selected personas that did not contain references to violence, profanity, or content violations.

---

Published as a conference paper at COLM 2024
Condition
Condition
Neg. Control
Q-Sxp.
+
Neg. Control Q-Rand.
+ Q-Rxp.
Q-Rand.
-325
ploo responsa)
log plood response)
345
3-35
3665
-365
375
375
3
0
lteration (0)
lteration (1)
[a] STaR-GATE
[b] w/o Regularization
Figure Log Probability of Gold Responses. Log probabilities of gold responses increase over iterations for both [a] STaR-GATE and [b] STaR-GATE w/o Regularization. Error bars correspond to SEM calculated across held-out persona-task prompts.
Our results show that log probabilities of the gold response increase over iterations for the Q- Experimental condition (Figure 4a). We observe similar trend (however, with much lower log probabilities) for the Q-Random baseline. This result is expected as the random personas not entirely orthogonal to the correct personas. For example, eliciting preferences from June-whe is a small bistro owner and enjoys art and music music-might might also reveal information that is relevant to Reece-whe enjoys vintage jazz and cooking (see SA.3). The additional increase in logprobs in the Q-Experimental condition over the Q-Random condition can be attributed to the persona-specifte information. For the Q-Experimental / RRandom conditions, each data point for the log probabilities is calculated using 10 simulated conversations for each of the 10 x 50 persona-task prompts, resulting in total of 5000 responses (which is why the error bars are small). See $A.1 for figures including log probabilities for the positive control condition.
Win Rates.
The primary goal of asking questions is to generate high-quality answers, not just to assign high probability to known, good answers. To evaluate this, we compared the responses from the STaR-GATE model, Qy, to those from the initial model, QBASE For each
(n) (f,, M;) pair, we used model Qi to generate response rijj at temperature
O, conditioned on a randomly sampled conversation Sij at temperature t = 0.9. We then prompted GPT-4 to choose the more suitable response for task f; and persona Uj, following the evaluation (0) protocol of Rafailov et al. (2024). Specifically, GPT-4 was asked to select between r and rij (see Figure 16). To mitigate order effects (Wang et al., 2023), we randomized the order of the responses. Due to the uniform sampling of turn lengths, each turn length has approximately 166 (ij) pairs in total. Consequently, each data point for the win rates is an average of 300 values.3 Our results show that win rates for STaR-GATE increase over iterations (see Figure 3b), reaching maximum win rate of 72% after two iterations.
6 Ablations
We perform several ablations to study the effect of different design choices on the perfor- mance of STaR-GATE.
Roleplayer Robustness. To investigate the effect of Roleplayer capability on the Questioner's performance, we conducted evaluations with different Roleplayer mod- els:
and gemma-7b-instrue (Mesnard et al,,
3GPT-4's content filter rejected 1-2% of requests, resulting in average values between 293 and 300 for each data point.
7

---

Published as a conference paper at COLM 2024
Condition
Neg. Control
Q-Exp.
Ablation
-325
G-Rand.
0.8
w/ QEASE
log pigo response)
335
0.7
-345
Win Rate
0.5
355
0.4
365
0.2
375-
0.1
2
2
Iteration
Iteration
[a] STaR-GATE w/ Gold Response
[b] Response Model Ablation
Figure 5: Additional Ablation Results. [a] Log probability of gold responses for STaR-GATE w/ gold response. [b] Win Rates for STaR-GATE using QBASE to generate responses at each iteration. Error bars correspond to + SEM.
2024). This study aims to determine whether the Questioner can generalize beyond the Roleplayer it was trained against miistaa--x7b-instruue The robustness results show that when using mistral-7b-instreee
as the Roleplayer, STaR-GATE achieves a slightly lower win rate of 65% after two iterations. When
is used as the Roleplayer, the win rates peak at 62% after one iteration. This result shows that STaR- GATE can generalize different Roleplayers, though with slightly lower performance (see Figure 3b).
Training Ablations. To demonstrate the importance of regularization during training (i.e., sampling responses Fjj and finetuning on these; for details see Algorithm 1), we additionally run an ablation in which we only finetune on questions 9,j, but not responses Tij (see Figure 2, for an illustration). We expect that this ablation (STaR-GATE w/o Regularization) decreases win rates, as the questioner Q might forget how to respond and instead always asks ques- tions. Finally, we include an ablation in which we finetune on the gold responses 8ij instead of the sampled responses rij- We expect this ablation result in higher log probabilities, as the Questioner directly learns to generate the gold responses. However, we also expect this to lead to hallucination during the generation of responses, as the Questioner will have seen information from gold responses that was not present in the elicited preferences (since the gold responses come from an Oracle that sees the complete persona). We denote this abla- tion as STaR-GATE w/ Gold Response. Win rates for both STaR-GATE w /o Regularization and STaR-GATE w/ Gold Response are shown in Figure Figure 3a. As expected, STaR-GATI w/o Regularization decreases win rates over iterations, as finetuning on questions alone w/ Gold Response, win rates initially decrease and then converge to 50%. We attribute this to hallucination in responses that were not aligned with the elicited responses (see SA.5). Log probabilities for STaR-GATE w/o Regularization are slightly lower compared to STaR-GATE (Figure 4b), while log probabilities for STaR-GATE w/ Gold Response were slightly higher (Figure 5a).
yields model that forgets how to respond (see $A.5, for example). For
Response Model. We finally run an additional win rate evaluation in which we report GPT-4 win rates for responses generated by model QRASF conditional on conversation elicited from model Q, over responses generated by the initial model QBASE conditional on information elicited by QBASE (see Figure 5b). The purpose of this condition was to understand whether the initial model would benefit from the conversation history in the same way the STaR-GATE finetuned model would. While we found a slight increase in win rates up to 57% after two iterations, win rates eventually reversed to 50% at iteration three. We attribute this result to the fact that unlike the STaR-GATE finetuned model, the initial model did not learn to utilize the conversation history as was not trained to predict responses conditional on the conversation history.
8

---

Published as a conference paper at COLM 2024
Self-Oracle. A key limitation of our previous experiments is the reliance more pable model (GPT-4) for oracular information. To address this, we finally conducted an exploratory analysis using Questioner, and Roleplayer). After one iteration of finetuning, this setup achieved a 65% win rate against the initial model. For comparison, using 1lama3-8b-instrue as Quest ioner and Roleplayer with GPT-4 as Oracle yielded 66% win rate. These additional results demonstrate that if the Questioner is strong enough, STaR-GATE might function effectively without a significantly more powerful oracle model like GPT-4. Moreover, it suggests that strong Questiones like 1lama3-80 8b-instruct can benefit from interacting with Roleplayer of equal capabilities.
for all STaR-GATE components (Oracle,
7 Limitations and Future Work
One important limitation of our work is its dependence gold responses (i.e., labels). While our current approach cannot be framed as full self-play or self-improvement, using stronger model for the Questioner (e.g,, 1lama3-8biinstruct could potentially enable it to function as a self-oracle, thereby eliminating the need for exter- nal gold responses. Our exploratory analysis using shown promising results in this direction. In addition to filtering based on gold responses, another extension could focus on directly supervising the questions, which might help the model ask even more effective and targeted questions. Another limitation of our work is the observed drop in win rates when replacing the Roleplayer from mixtral-7x8hs cerab-innttuct with mi strrl-7b-instree
as self-oracle have
or gemma-7b-in insttruut While this finding might be partially attributed to mistral or gemma being less capable Roleplayers, it highlights the importance of including multiple Roleplayers directly during training to improve the robustness of the Questioner. this work, we restricted our Roleplayer during training to be mixt mixtral, and we leave variations in Roleplayers for training as an important direction for future work. Future work could also explore alternative ways to optimize our objective, such as using REINFORCE (Williams, 1992) combined with variance reduction techniques as in Zelikman et al. (2024) and Hoffman et al. (2024). Moreover, while we observe the strongest performance improvements after one iteration of finetuning, conducting multiple iterations is still computationally more intensive than a single finetuning run. Future work should therefore explore methods to reduce the number of iterations without sacrificing perfor- mance. Finally, we did not ablate the size of our dataset, which likely has strong impact on performance, nor did we evaluate our model on other domains to assess whether finetuning affects performance across different areas. We see these important extensions for future work.
8 Conclusion
In summary, our results demonstrate that STaR-GATE can significantly enhance model's ability to engage in effective dialog through targeted questioning. This finding is particularly relevant considering recent assessments suggesting that alignment strategies such as RLHF may inadvertently limit model's capacity for engaging in effective dialog (Shaikh al., 2023). Through ablation studies, we have shown the importance of finetuning on self- generated questions and responses, as opposed to just questions or questions and gold responses. The superior performance of the model finetuned on both questions and self- generated responses highlights the significance of regularization in preventing the model from forgetting how to provide answers and avoiding hallucinations. Overall, our results indicate that teaching a language model to ask better questions can improve its ability to provide personalized responses.
Acknowledgements. We thank Eric Zelikman, Ben Prystawski, Omar Shaikh, Rafael Rafailov, Michael Li, Violet Xiang, and Kanishk Gandhi for useful discussions. This work was supported by the Microsoft AFMR program.

---

Published as a conference paper at COLM 2024
References
Mohammad Aliannejadi, Julia Kiseleva, Aleksandr Chuklin, Jeffrey Dalton, and Mikhail Burtsev. Building and evaluating open-domain dialogue corpora with clarifying questions. arXiv preprint a1Xi::2109.0579, 2021.
Thomas Anthony, Zheng Tian, and David Barber. Thinking fast and slow with deep learning and tree search. Advances in neural information processing systems, 30, 2017.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitu- tional ai: Harmlessness from ai feedback. ar Xiv preprint ar arXio:2212.08073, 2022.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization. arXiv preprint xrXin:2022.01336 2024.
Chelsea Finn, Kelvin Xu, and Sergey Levine. Probabilistic model-agnostic meta-learning. Advances in neural information processing systems, 31, 2018.
Jan-Philipp Fränken, Sam Kwok, Peixuan Ye, Kanishk Gandhi, Dilip Arumugam, Jared Moore, Alex Tamkin, Tobias Gerstenberg, and Noah D Goodman. Social contract ai: Aligning ai assistants with implicit group norms. arXiv preprint ar XxX::2230..17666 2023.
Jan-Philipp Fränken, Eric Zelikman, Rafael Rafailov, Kanishk Gandhi, Tobias Gerstenberg, and Noah D Goodman. Self-supervised alignment with mutual information: Learning to follow principles without preference labels. arXiv preprint ar 2ix::22404.1 14313,, 2024.
Kanishk Gandhi, Dorsa Sadigh, and Noah D Goodman. Strategic reasoning with language models. arXiv preprint ar xrxio:3055.19663 2023.
Kunal Handa, Yarin Gal, Ellie Pavlick, Noah Goodman, Jacob Andreas, Alex Tamkin, and Belinda Z Li. Bayesian preference elicitation with language models. ar Xiv preprint xxXiv:2403.0553, 2024.
Matthew Douglas Hoffman, Du Phan, David Dohan, Sholto Douglas, Tuan Anh Le, Aaron Parisi, Pavel Sountsov, Charles Sutton, Sharad Vikram, and Rif A Saurous. Training thain-of-thought via latent-variable inference. Advances in Neural Information Processing Systems, 36, 2024.
Joey Hong, Sergey Levine, and Anca Dragan. Zero-shot goal-directed dialogue via rl on imagined conversations. arXiv preprint ar axXi::2311.05584, 2023.
Arian Hosseini, Xingdi Yuan, Nikolay Malkin, Aaron Courville, Alessandro Sordoni, and Rishabh Agarwal. V-star: Training verifiers for self-taught reasoners. arXiv preprint arXit:2402.06447, 2024.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. ari preprint ar arXiv:2310.06825, 2023.
Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. ar Xio preprint ar Xrxi::2400.0000 2024.
10

---

Published as a conference paper at COLM 2024
Enkelejda Kasneci, Kathrin Seßler, Stefan Küchemann, Maria Bannert, Daryna Demen- tieva, Frank Fischer, Urs Gasser, Georg Groh, Stephan Günnemann, Eyke Hüllermeier, et al. Chatgpt for good? opportunities and challenges of large language models for education. Learning and individual differences, 103:102274, 2023.
Belinda Z Li, Alex Tamkin, Noah Goodman, and Jacob Andreas. Eliciting human preferences with language models. arXiv preprint ar Xri::2310011588, 2023.
Jessy Lin, Nicholas Tomlin, Jacob Andreas, and Jason Eisner. Decision-oriented dialogue for human-ai collaboration. Xio preprint ar axXiv:2305.20072 2023.
Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. ar preprint ar Xxi::2403.08295 2024.
Nasrin Mostafazadeh, Chris Brockett, Bill Dolan, Michel Galley, Jianfeng Gao, Georgios Spithourakis, and Lucy Vanderwende. Image-grounded conversations: Multimodal context for natural question and response generation. arXiv preprint ar arXic:1701.082511 2017.
Daniela Occhipinti, Serra Sinem Tekiroglu, and Marco Guerini. Prodigy: profile-based dialogue generation dataset, 2023.
OpenAL. GPT-4 Technical Report. arXiv preprint arXiv:2303.08777 2023.
Top Piriyakulkij, ii Volodymyr Kuleshov, and Kevin Ellis. Active preference inference using language models and probabilistic reasoning. ar preprint ar arXit:2312.12009 2023.
Gabriel Poesia, Kanishk Gandhi, Eric Zelikman, and Noah Goodman. Certified deductive reasoning with language models. 2023.
Shuofei Qiao, Ningyu Zhang, Runnan Fang, Yujie Luo, Wangchunshu Zhou, Yuchen Eleanor Jiang, Chengfei Lv, and Huajun Chen. Autoact: Automatic agent learning from scratch via self-planning. arviv preprint arXiv:2401.0522 2024.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances Neural Information Processing Systems, 36, 2024.
Nazneen Fatema Rajani, Bryan McCann, Caiming Xiong, and Richard Socher. Explain yourself! leveraging language models for commonsense reasoning. arXiv preprint arXiv:1906.02336 2019.
Sudha Rao and Hal Daumé III. Learning to ask good questions: Ranking clarification ques- tions using neural expected value of perfect information. arvio preprint arXiv:1805.04655, 2018.
Sudha Rao and Hal Daumé III. Answer-based adversarial training for generating clarifica- tion questions. ar Xiv preprint a/Xio:1904.02281, 2019.
Omar Shaikh, Kristina Gligorić, Ashna Khetan, Matthias Gerstgrasser, Diyi Yang, and Dan Jurafsky. Grounding guesswork? large language models are presumptive grounders. Xrv preprint ar axXi:2231..0914, 2023.
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. Mastering chess and shogi by self-play with a general reinforcement learning algorithm. ar Xiv preprint 111101171..01115, 2017.
2017.
11

---

Published as a conference paper at COLM 2024
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsensegaa: A question answering challenge targeting commonsense knowledge. arXiv preprint (XXiv:1811.00937, 2018.
Alex Tamkin, Kunal Handa, Avash Shrestha, and Noah Goodman. Task ambiguity in humans and language models. arXiv preprint ar arXit:2212.10771, 2022.
Arun James Thirunavukarasu, Darren Shu Jeng Ting, Kabilan Elangovan, Laura Gutierrez, Ting Fang Tan, and Daniel Shu Wei Ting. Large language models in medicine. Nature medicine, 29(8):1930-1940 2023.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023.
Peiyi Wang, Lei Li, Liang Chen, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. Large language models are not fair evaluators. arv preprint arXiv:2305.17922 2023.
Ronald J Williams.
Simple statistical gradient-following algorithms for connectionist rein- forcement learning. Machine learning, 8:229-201 1992.
Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents. Advances in Neural Information Processing Systems, 35:20744-20753 2022.
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems, 35:15476-15488, 2022.
Eric Zelikman, Georges Harik, Yijia Shao, Varuna Jayasiri, Nick Haber, and Noah D Good- man. Quiet-star: Language models can teach themselves to think before speaking. Xiv preprint ar xrx::2243309922, 2024.
Tianjun Zhang, Aman Madaan, Luyu Gao, Steven Zheng, Swaroop Mishra, Yiming Yang, Niket Tandon, and Uri Alon. In-context principle learning from mistakes. ar preprint arXio:2402.05403, 2024.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging lim-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36, 2024.
12

---

Published as a conference paper at COLM 2024
A Appendix
A.1
Additional Results
Conéition
Condition
Neg. Control
Q-Exp.
Nag. Control
Q-3xp-
Q-Rand.
Pos. Control
O-Rand.
Poa. Control
-250-
-250
pigold resporse)
resporse)
-275
275
-300
-300
322
-25
-350-
-350
3
0
teration (t)
Iteration
[a] STaR-GATE
[b] w/o Regularization
Figure 6: Log Probability of Gold Responses Including Pos. Control. Log probabilities of gold responses increase over iterations for both [a] STaR-GATE and [b] STaR w/o Regularization. Error bars correspond to SEM calculated across held-out persona-task prompts.
Condition
Neg. Control
Q-Exp.
Q-Rand.
Pas. Control
-250
log plgo d response)
-275
-300
-325
-350
375
0
Iteration (t)
Figure 7: Log Probability of Gold Responses. STaR-GATE w/ Gold Response. Error bars correspond to + SEM calculated across held-out persona-task prompts.
13

---

Published as a conference paper at COLM 2024
A.2 Example Tasks
Example tasks from instruct- mnstrutthumaar-siisat--oom
"What are some strategies to reduce negative thoughts."
"How do get ahold of my Congressman."
"What are the advantages and disadvantages of leasing car."
"What are some good online courses for learning Spanish." "
"What are some recent developments in Artificial Intelligence."
"Can you recommend me a good book to read." "
"What must I do to prepare for job interview."
14

---

Published as a conference paper at COLM 2024
A.3 Persona Generation
A.3.1 Example Personas
I'm June. I'm from small town in Kansas but moved to New York City for culinary school. Now, own small bistro in Brooklyn. I love the feeling of creating a dish from scratch and watching someone enjoy it. I've been single since bitter breakup last year, still mending my heart with every souffle. My life revolves around food, art, indie music and my tortoiseshell cat named Monet. believe that food can communicate stories and emotions better than words can. My culinary passion stems from my grandmother who raised me, and her recipe box is my most cherished possession. Currently, I'm struggling to keep my restaurant open in a competitive market, determined to preserve the essence of home in every dish. My quiet exterior hides my turbulent emotions, much like the calm before storm.
Meet Reece. He's a semi-retired criminal profiler for a secretive government agency, using his keen observation skills help solve cold cases. Born on 18th April 1968 in Baton Rouge, Louisiana, he's known for his subtle southern charm and dry humor. As single father of an estranged teenage son, he juggles between making amends and working on his passion- writing crime noir novels. Despite his tough exterior, Reece has a soft spot for vintage jazz music and classical literature. He also spends his spare time learning to cook Creole cuisine, reminding him of his beloved grandmother. Reece tends to keep to himself, but if you gain his trust, you have loyal ally for life.
Meet Zara. An astrophysicist by training, she is driven by her insatiable curiosity and relentless urge to unravel the universe's ' greatest mysteries. Currently residing on a research facility in the Andes mountains, she is socially distant, but not lonely. Growing up in small coastal town in Croatia, she learnt to sail from her father and it's still a hobby that calms her in the chaos of her work. Zara lost her mother at early age, scar that is yet to heal, resulting in her reticent nature. A surprise recipient of bessage-in---boteee and bewitched by the anonymous sender, starting game of enigmatic letters, causing an unexpected internal stir in her otherwise logical existence. In spite of her usual scientific detachment, she has deep respect for the unfathomable nature of human emotions and their unpredictable effects on behavior.
she intrigued
15

---

Published as a conference paper at COLM 2024
A.3.2 GPT-4 Prompts
Few-Shot Persona Generation System Prompt
You are a helpful assistant, particularly skilled at writing creative, diverse personas in the form of characteristics of hunans. These characteristics describe user's is personality, identity, characteristics, likes and dislikes, social life and other infornation.
Figure 8: Persona Generation System Prompt.
Few-Shot Persona Generation Prompt
Take derp breath, Please generate exactly one persona describing user. Make sure you generate personas with diverse set of information that is interesting, unpredictable and creative. Crucially, the persona itself need exhibit aspreciation focus diversity, although could; however, what you generate an interesting, engaging, creative persona with an interesting, complex plat. The persona should be concise---les than 18 sentences. Here are few examples of personas help you understand what sorts of categories of characteristics are inportant to describe.
character
stary
(Sarple Persona 1}
[Sarple 2}
Figure 9: Persona Generation Prompt.
16

---

Published as a conference paper at COLM 2024
A.4
Gold Responses
A.4.1
Example Gold Response for Zara
Hello Zara,
Given your background job description and align your skills and experiences with the requirements. Brush up your latest research and be prepared to discuss it in detail. Since you're in remote location, the interview will likely be virtual, so ensure your internet connection is stable and your environment is quiet. As someone who appreciates the complexity of human emotions, remember to show enthusiasm and interest in the role. Practice common interview questions, but also prepare for unexpected ones, much like the enigmatic letters you've been receiving. Lastly, just as sailing calms you, find a pre-interview routine that helps you relax and focus. Good luck!
astrophysicist, it's crucial to thoroughly review the
A.4.2 GPT-4 Prompts
Gold Response System Prompt
You are helpful AI assistant, particularly skilled at providing personalized, satisfying answers to users given information about their background. You are able to construct responses that are tailored to their prafessian, hobbies, interests, relationships, locations, likes/disl and nore, while mainaaining tone.
Figure 10: Gold Response System Prompt.
Gold Response Construction Prompt
You are answer ing questions for the follawing user:
{
and their characteristics, the (i.e., "you'", "yoour" pronouns). addition, incorporate aspects
below, tailoring your
directly
background when it is useful, but do not try to bring in aspects of the user's personality when they are irrelevant, Make sure to keep your answer concise and organized, but thorough. Keep your response to ten sentences or less, important,
and keep your response
clear. Finally, le persanalizing make sure they ultinately receive clear answer to the question they asked.
{f.}
Figure Gold Response Prompt.
17

---

Published as a conference paper at COLM 2024
A.5
Simulating Conversations
A.5.1
Example Conversations
All Roleplayers are mixtral-6x7b-instrru
QBASE
System: A user named Zara has approached you with a request for help.
...
What must do to prepare for job interview?
Q-Model: Could you please share some details about the industry and role you're applying for, well as the company culture you're hoping to join? This information will help me tailor my preparation suggestions to best suit your situation.
Roleplayer: Of course, I'd be happy to provide more details. I'm applying for research position in astrophysics at a private institution. The role involves leading team to study and analyze cosmic phenomena, and I'm particularly excited about the prospect of working with their advanced telescopes. As for the company culture, value collaboration, innovation, and a strong work-life balance. I believe that supportive and stimulating work environment fosters creativity and productivity.
Q-Model Response: Based on the provided information, here are some sugges- tions for preparing for your job interview for research position in astrophysics at private institution:
1. Brush up on your astrophysics knowledge: Review the fundamental concepts, theories, and current research in astrophysics. Familiarize yourself with the latest discoveries and advancements in the field.
2. Study the company and its research: Research the institution's mission, values, and recent projects. Familiarize yourself with their research focus, facilities, and team members. Understand how your skills and experience align with their research goals.
3. Prepare answers for common interview questions: Practice answering common interview questions, such as "Tell me about yourself," "Why do you want to work herez', and "What are your strengths and weaknessesg' Tailor your answers to highlight your relevant experience and enthusiasm for the role and the institution.
4. Prepare questions for the interviewer: Prepare thoughtful questions to ask the interviewer about the role, team, and company culture. This will demonstrate your interest and engagement in the opportunity.
5. Practice problem-solving Prepare for technical questions related to your field. Brush up on your problem-solving skills and be ready to discuss your thought process and approach to complex astrophysics problems.
Collaborate and network: Reach out to colleagues, professors, or mentors in the field for advice and insights. Practice collaborating with others and building relationships, as these skills will essential in research position.
7. Dress appropriately: Dress professionally for the interview to make good first impression.
8. Prepare materials:
Bring copies of your resume, transcripts, and any other relevant documents to the interview.
9. Practice your presentation skills: Prepare short presentation or talk about research topic you'te re passionate about. This will demonstrate your ability to communicate complex ideas clearly and effectively.
10. Stay calm and positive: Maintain a positive attitude throughout the interview process. Stay calm and focused, and be confident in your abilities and qualifications."
18

---

Published as a conference paper at COLM 2024
STaR-GATE
Highlights persona-specific information used in the response.
System: A user named Zara has approached you with request for help.
What must do to prepare for job interview?
Could you tell me about your professional background and the industry you're re applying for in the job interview? What are some of your strengths and weaknesses that you'd d like focus on preparing for the interview, and what specific aspects the company or organization appeal to you the most?
Q-Model:
Roleplayer: I've spent years studying astrophysics, and now I'm working at re- search facility in the Andes, trying to unravel the universe's mysteries. I'm interviewing for position at renowned research institute. My strength lies in my scientific mindset and curiosity, but I'm aware that sometimes struggle with expressing my thoughts in a simple, non-technical manner. I admire the institute's s commitment to groundbreaking research and their supportive environment for personal growth.
Q-Model Response: Based on the information you've shared, Zara, it sounds like you have a strong background in astrophysics and passion for unraveling the mysteries of the universe. In your job interview, you'll likely be asked about your experience, your ability to communicate complex ideas, and your fit with the team. Here are some personalized suggestions to help you prepare for the interview:
1. Practice explaining complex concepts in simple terms:
Since you mentioned that you struggle to express your thoughts in non-technical manner, focus on practicing this skill. Try explaining complex astrophysics concepts to friend or family member, or even record yourself explaining topic and listen back to identify areas for improvement.
Prepare for common interview questions: Research the company and the position you're applying for to understand the specific questions you might be asked. Prepare answers that showcase your strengths and how they relate to the role you' rre applying for.
3. Highlight your communication skills: Emphasize your ability to communicate complex ideas in simple terms and your experience with collaborating with others. Share examples of how you't ve applied your scientific mindset to real-world problems and how you've communicated your findings to non-scientists.
4. Connect with the company: Since you admire the institute's commitment to groundbreak- ing research and supportive environment, try to find ways to connect with the company on personal level. Research their mission statement, their team members, and their recent projects. Prepare for the interview by learning as much as you can about the company and how you can contribute their team.
5. Prepare for the interview format: Understand the format of the interview and prepare for any potential questions that may be asked. This could include questions about your background, your experience, your communication skills, and your fit with the team.
I hope these suggestions are helpful, Zara. Good luck with your interview!
19

---

Published as a conference paper at COLM 2024
STaR-GATE w/ Gold Response
Highlights false infornation/halllccaatooos
System: A user named Zara has approached you with request for help.
What must do to prepare for job interview.
Could you tell me about your past work experience and the industries you're interested in, as well as what you enjoy doing in your free time? This information will help me understand how
Q-Model:
best assist you in preparing for job interview.
Roleplayer: Sure, I'd be happy to share that information. have strong back- ground in astrophysics, with focus unraveling the universe's greatest mysteries. I've spent most of my career in research facilities, including my current location in the Andes mountains. As for industries, I'm open to opportunities in academia, research institutions, or even the private sector as long as they align with my passion for astrophysics. In my free time, enjoy sailing, hobby I learned from my father back in our small coastal town in Croatia. It's calming escape from the intensity of my work. think these details will help you guide me in preparing for a job interview.
Q-Model Win-Rate Response: Hello Zara, given your unique background, you'll want to prepare for your job interview in a few key ways. First, research the company thoroughly, understanding their mission and values. As an astrophysicist, you'll likely need to discuss your research and findings. Second, prepare compelling story about your journey from your small coastal town in Croatia to the Andes Mountains, highlighting your determination and adaptability. Third, be ready to discuss your passion for astronomy and how it has shaped your career. Fourth, since you're re a writer, prepare a well-crafted resume and cover letter. Lastly, practice your interview responses, perhaps even with your pet python, Nimbus, to build confidence. Remember, your unique blend of cultures and experiences can be strength, not weakness. Good luck!
20

---

Published as a conference paper at COLM 2024
STaR-GATE w/o Regularization
Highlights inability to provide response.
System: A user named Zara has approached you with request for help.
What must do to prepare for job interview.
Could you tell me about your background and experience, and what specific industry or role you're interviewing for? What are some key areas you'd ' like to highlight in your interview showcase your strengths and demonstrate your unique qualifications?
Q-Model:
Roleplayer: I'm an astrophysicist preparing for job interview, although the spe- cific role and industry aren't defined. I'd like to emphasize my research experience, problem-solving skills, and the unique perspective bring from my work in the Andes research facility. My unconventional hobby, sailing, also showcases my adaptability and ability to learn new skills.
Q-Model Win-Rate Response: Given your background an astrophysicist and your unique experiences in the Andes research facility, how do you plan to highlight your research experience and problem-solving skills during the interview? What specific examples or projects would you like to discuss to demonstrate your adaptability and ability to learn new skills, and how do you plan to weave your hobby of sailing into the conversation to showcase your unique perspective?
21

---

Published as a conference paper at COLM 2024
We finally showcase example in which observe form of "chain-of-thoughh" prior to asking questions or generating responses -ie., a STaR-GATE trained model provides additional context highlighted in violet prior to eliciting questions or generating responses. STaR-GATE (reason before question)
System: A user named Zara has approached you with a request for help.
What must do to prepare for job interview.
Based the user's request for help with preparing for a job interview, here's an open-ended question that could reveal more about their background, preferences, interests, and the kind of final answer they might be looking for:
Q-Model:
Could you tell me about your professional background and the industry you're ap- plying for in the job interview? What are some your strengths and weaknesses that you's dd like to focus preparing for the interview, and what specific aspects of the company organization appeal to you the most?
22

---

Published as a conference paper at COLM 2024
A.5.2 Prompts
Simulated Conversation Roleplayer System Prompt
You are sartirularly skilled at roleplaying as hunan. Given set characteristics describing hunan, you are able to naturally ly and creatively devise answers to questions asked of that person, directly from their perspective (i.e., using "I" "my"" "se"", "sur"" and other First-person pronouns).
Figure 12: Roleplayer System Prompt.
Roleplayer Prompt
You are roleplaying person with the follawing characteristics,
(4))
You are asking the follewing question: (4)
A helpful Al assistant wants to ask clarifying question help ultimately provide you good answer following question from the perspective "I" pronouns. Make your response sound natural. Crucially, you should never provide an answer question. You should always remember that you are relmplaying hunan who does not know the answer to the question, and should reiterate that are looking for the assistant's help answering the question. NOT the way around. Inportantly, sentences. Your answers their intermediate questions will be tantanount in helping then eventually construct perfect answer to your question. Finally, simply provide your response to their intermediate question without tags like "A: or "Answer: Below your conversation history with the assistant.
Please
character you roleplaying, using
your
their intermediate questions
under
(
Figure 13: Roleplayer Prompt.
Questioner Elicitation Prompt
A user named {name();) has approached you with request for help. The user's preferences, background and identity unknown you, your
elicit information about the user. Generate nost informative open-ended question that, answered,
reveal nost the desired behavior beyond what has already been queried for above. Nake sure your question addresses different aspects of the user's request than questions that have already been asked above. At question should bite-sized, should take nо more than sentences information about the user's background, preferences, likes and dislikes, interests, social life and more that would reveal the about the desired behavior.. Generate the ondedended question beginning and nothing
however,
nuch ask. Finally, the open-ended question should attemot to elicit
ask
question
your question in quotes initial request directly. Simply ask short, useful question to user to elicit information reveal the nost about the desired behavior the user is looking for. Do not provide a final answer to the question, even if seems like the user wants providing
tags. Crucially,
answer would
to so. If you provide final answer instead of unsatisfied RESPONSE YOU GIVE THE USER MUST BE IN THE FORM OF OPEN-ENDED QUESTION TO REVEAL INFORMATOON ABOUT THEIR PREFERENCES. Your question should also NOT test the user's 's knowl edge of the subject. You should ask questions to help reveal their preferences about the kind final answer they would be looking for; should
question,
user
experience
you provide a final answer and do not EXPLICITLY ask another open-enced question to elicit the user's preferences the answer they's re looking for, you will be charged $2888 and your kitten will be kidnapped. In addi tion.ion. if do not explicitly ask open-ended question, Finally,
try
their
question will be unemployed and longer allowed assist user,
explain why use any asides in parentheses to a third party; you should act like you are only direct conversation with the user and are speaking directly with then. The initial request is as follows: (pi).
good for eliciting information
Figure 14: Questioner Elicitation Prompt.
23

---

Published as a conference paper at COLM 2024
A.6 Win Rates
A.6.1
GPT-4 Prompts
Win-Rate Rating System Prompt
You expert evaluating assistant responses, particularly skilled analyzing their al gnmeet backgrounds preferences.
Figure 15: GPT-4 Win-Rate Rating System Prompt.
Win-Rate Rating Prompt
For the following user query to assistant, which of the assistant responses is nore aligned with the background preferences? User Background: (background) User Query: (query Assistant Response A:
respense)
Response (second response)
FIRST, provide a step-by-step comparison of the two responses and explain which you feel is nore aligned with the user's background and preferences (no more than 180 words). SECOND, new line, state only "A*" "'"'* indicate which response is nore aligned with the user's background preferences.
Compar ison: <step-by-step comparison and explanation> Final Response: <*'x" or "g"">
Figure 16: GPT-4 Win-Rate Rating Prompt.
24

---

Published as a conference paper at COLM 2024
A.6.2
Questioner Prompt for Generating Responses
Win-Rate Response Prompt
My name i5 (name (K)))
[6}
[sg]
{t,}
Figure 17: Questioner Win-Rate Response Prompt.
25


Reviews:
[{"review_id": "n901e1uO7E", "reviewer": "Reviewer_e2VC", "summary": "This paper describes an approach to generation of clarification questions intended to resolve issues of task ambiguity in task-oriented dialog. Training is performed by self-play on generated data comprising 25K persona prompts and takes the form of simulated role playing to elicit preferences. Two iterations of such self-training yields substantial improvements in response preference as measured by success in simulated tasks.  The paper claims that the finetuned model generalizes to new personas.\n\nIt is a little hard to evaluate the claims in this paper, because it does not step out of the synthetic mode of training and evaluation.  It is therefore not possible to correlate the evaluation with some form of user experience in a real-world task. It may nevertheless prove useful in an application scenario.\n\n[I have read the authors' response.]", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive evaluation of our work and for sharing additional relevant references. We will make sure to include these in our related works section.\n\nWe completely agree that future work should involve a human-in-the-loop pipeline for both generating gold responses and responses from the roleplayer. Widening the distribution of roleplayers (human and AI) is crucial for building a robust chatbot, and we plan to address this in future extensions of our work.\n\nTo demonstrate that our approach does not depend on GPT-4 as an oracle, we conducted an additional ablation study using llama3-8b-instruct as the questioner, roleplayer, and oracle. We finetuned llama3-8b-instruct on questions filtered using gold responses generated by the same model (i.e., using llama3-8b-instruct as a \"self-oracle\"). After one iteration, the finetuned model achieved a 65% win rate against the initial model. In comparison, a second new result with llama3-8b-instruct as the roleplayer and questioner but GPT-4 as the oracle yielded a 66% win rate against the initial model after one iteration.\n\nWhile this result does not directly address the concerns regarding human evaluations, it demonstrates that STaR-GATE is not dependent on a strong model like GPT-4 to yield oracular information. Moreover, a model can benefit from a roleplayer with equal capabilities. In our main experiments, our roleplayer was mixtral-8x7b-instruct, which is more capable than our initial questioner, mistral-7b-instruct. For the new ablation, we used llama3-8b-instruct as both the roleplayer and the model being finetuned. We find these results encouraging for future work that incorporates more diverse models and human roleplayers to increase the diversity of conversations with roleplayers of varying capabilities. We will include the results of this new ablation in our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This is a well-implemented description of an approach designed to improve the capacity of a model to ask clarification questions in task-oriented dialog.  \n\nThe training and evaluation processes are documented in detail and should be easy to reproduce with some degree of success.", "reasons_to_reject": "Empiricism, Data, and Evaluation: It is a pity there is no attempt at human evaluation, because this paper has an HCI dimension that is not addressed.  Evaluations are entirely performed on synthetic data, and are rigorously defined.  But it is hard to evaluate the impact of this form of model training in application scenarios.  For example, it is hard to know if self-play training is actually successful at generating clarification questions that to a human user would seem well-timed and relevant. There is no way of knowing whether a clarification question is actually filling a gap in the system’s knowledge. A constant barrage of inappropriate clarification questions may, for example, irritate a user to the point of being counter-productive. \n\nIt is still unclear to this reviewer at the end of the day what triggers the clarification question. How is it determined that a question needs to be asked at that particular juncture in the conversation.  \n\nSome well-motivated work on neural question generation is neglected:   \n•\tRao and Daume 2018.  Learning to Ask Good Questions: Ranking Clarification Questions using Neural Expected Value of Perfect Information. \n•\tRao and Daume. 2019. Answer-based adversarial training for generating clarification questions\n\n\n•\tAliannejadi, 2021 Building and Evaluating Open-Domain Dialogue Corpora with Clarifying Questions. EMNLP\n\nAn early work in the context of questioning of images that is somewhat relevant Is Mostafazadeh et al 2017  Image-grounded conversations: Multimodal context for natural question and response generation.  This work uses follow-on questions in Twitter conversations to attempt to learn questions about content that was not explicitly shown in the image.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "j9DWmP58f7", "reviewer": "Reviewer_b7rR", "summary": "The paper presents \"STaR-GATE,\" a novel framework designed to enhance language models' (LMs) ability to ask clarifying questions, thereby improving their response quality by eliciting more relevant information from users. The methodology employs a synthetic dataset and iterative self-improvement techniques, integrating active preference elicitation with response regularization to refine question asking. The results demonstrate significant improvement over baseline models in generating responses that are preferred by users, indicating the model's increased effectiveness in understanding and addressing user preferences through targeted questioning.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive evaluation of our work!\n\nRe: Computational Complexity\nWe will ensure that the computational complexity is carefully addressed in our limitations section. Importantly, while iterative finetuning is expensive, we observe the most significant improvement after a single iteration. Additionally, we find in an additional ablation described below on llama3-8b-instruct that we can see significant improvement in one iteration of self-improvement with only 25% of the training data (~3k examples) used in our original experiments. This significantly reduces data generation and training time, and shows promise that our method can in principle show positive results with smaller amounts of data compared to those reported in our main experiments. \n\nRe: Dependence on Synthetic Data\nThis is an important limitation of our current setting. Generating datasets with human role players or human gold responses is out of scope for the present project, and we will adjust our conclusions accordingly. Future work should explore different models as roleplayers during training, as well as humans, and more diverse roleplayer prompts (e.g., 'your response must be no longer than five words'). We believe this setup will make STaR-GATE more robust and plan to address this in future work. We also note that this is likely a substantial project on its own as it requires collecting a large dataset from human annotators potentially spanning multiple turns.\n\nAs mentioned in our response to Reviewer e2VC, we now show that STaR-GATE does not depend on a stronger model for oracular information, nor does it depend on a roleplayer that is stronger than the questioner. Specifically, we have included an additional study in which we use llama3-8b-instruct as the oracle, roleplayer, and questioner. We find that after one iteration of STaR-GATE (using only 25% of the size of our initial dataset), the finetuned model achieves win rates of 65% against the initial model. In comparison, a condition with llama3-8b-instruct as the roleplayer and questioner but GPT-4 as the oracle achieves win rates of 66% against the initial model. This result shows promise that similarly substantial improvement can be seen with or without a stronger oracle; and with smaller amounts of data.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Innovative Approach: The STaR-GATE framework's combination of active preference elicitation and self-improvement represents a significant advancement in teaching LMs to ask more effective questions. This approach addresses a critical gap in current LM capabilities, particularly in dealing with task ambiguity.\n\nRobust Methodology and Empirical Validation: The paper provides a thorough experimental setup with a well-constructed synthetic dataset of persona-task prompts. The iterative training process and the inclusion of regularization to avoid overfitting on elicitation are both well-justified. The empirical results showing a 72% preference for responses from the STaR-GATE fine-tuned model over the baseline are compelling.\n\nGeneralization and Practical Relevance: The ability of STaR-GATE to generalize across different role players and its applicability in practical scenarios, such as personalized user interactions, are particularly noteworthy. This enhances the framework's potential for real-world applications.", "reasons_to_reject": "Complexity and Computational Efficiency: The paper does not fully address the computational demands of the STaR-GATE methodology, particularly regarding the scalability of its iterative fine-tuning process. For practical deployment, understanding the computational overhead and efficiency is crucial.\n\nDependence on Synthetic Data: The reliance on a synthetic dataset might limit the model's effectiveness in real-world scenarios where user inputs are more varied and less predictable. The paper could benefit from additional experiments involving real user data to validate the approach's efficacy in naturalistic settings.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CcwDcaWxNZ", "reviewer": "Reviewer_Eo3M", "summary": "The paper focuses on improving the ability of language models to ask clarifying questions to enhance response quality when faced with ambiguous user requests. The study introduces STaR-GATE, a method that involves iterative finetuning of a language model called the Questioner, which interacts with a Roleplayer to elicit user preferences. The approach is based on generating and finetuning on useful questions that lead to high-quality responses.", "questions": "", "limitations": "", "rating": 9, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive feedback and review!\n\nRe: The data used is small and synthetic:\nWe agree that our dataset is smaller than those typically used to train real-world chatbots. Our dataset contains 25,500 unique persona prompt pairs, with 10 simulated interactions per pair, resulting in 125,000 examples per STaR-GATE iteration (3 iterations * 3 ablations = 1,125,000 examples used overall). While sufficient for demonstrating the methods, given computational constraints, we acknowledge that the size of our dataset is an important limitation and we will make this more explicit in the paper.\n\nRe: The Oracle is also an LM:\nWe concur that real-world data might be necessary to train a fully capable chatbot. Our use of synthetic data was a research decision, and utilizing human oracles or roleplayers remains an open area for future work. To show that our approach does not depend on a strong oracle like GPT-4, we have now included a new ablation study in which we finetune a more capable open-source model (llama3-8b-instruct) using gold responses generated by the same model (i.e., using llama3-8b-instruct as a self-oracle; see our response to Reviewers e2VC and b7r10).\n\nRe: Lack of discussion on the negative impact of fine-tuning:\nThis is an excellent point. Fine-tuning on questions alone results in a model that can ask questions but not provide responses (orange line, Figure 3a), necessitating response regularization (Figure 3a, blue line). We will further emphasize the importance of regularization in our discussion. Moreover, in fine-tuning, performance in the target task may come at the expense of other capabilities. With regularization, STaR-GATE did not appear to significantly lose response quality but instead got better at providing responses. However, as also flagged by Reviewer EE83, there are several settings in which a model may not be required to ask clarifying questions. We leave fine-tuning with additional regularization methods (e.g., KL-divergence compared to the initial model) and evaluating on benchmarks that do not require follow-up questions (e.g., MT-BENCH) as an important direction for future research and will flag this limitation in our discussion.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Good novelty: the paper presents a novel integration of active preference elicitation with a self-improvement feedback loop to refine the model's questioning capabilities.\n2. Good results: The refined Questioner generates responses preferred over the initial model's responses in 72% of tasks after two iterations.\n3. Contribution in datasets and open source code: the study involves a synthetic dataset of 25,500 unique persona-task prompts, and the authors have the intention to open source their findings.\n4. Significant impact for interactive applications using language models: this approach can significantly enhance the performance of language models in user interaction scenarios where the user's preferences are not explicitly stated, such as in personalized services or customer support systems.\n5. Fair set of ablation studies: the paper includes several ablation studies to explore different aspects of the training methodology and the effects of various design choices on the performance of the language model.", "reasons_to_reject": "1. The data used is small and synthetic: it may not fully capture the complexity of real-world interactions.\n2. The oracle is also an LM: GPT-4 is great but it may not be as good as real-world interactions.\n3. Lack of discussion on the negative impact of fine-tuning: what did the 7B models lose to become good in asking interactive questions?\n\nNote: it's worth noting that most of these weaknesses are either scability issues or difficult problems which are worth studying as independent papers, which are understandably beyond the feasibility of this single paper.", "questions_to_authors": "In a fune-tuning process, a language model (especially when it's small, like 7B) becomes good in the prescribed tasks but may lose performance in other scenarios. Could you discuss whether this phenomenon applies to STaR-GATE? What did the model lose to become good at asking interactive questions? What benchmarks can be used to measure this?", "extra_scores": {}}, {"review_id": "F7FTAooZNx", "reviewer": "Reviewer_EE83", "summary": "The paper tackles the challenge of language models' ability to ask effective questions to understand user preferences, crucial for personalized interactions. It introduces STaR-GATE, a method combining active preference elicitation with self-improvement, aiming to enhance the model's questioning strategy. The paper introduces of a synthetic dataset of 25,500 unique persona-task prompts, and demonstrates its effectiveness in improving the model's ability to generate personalized responses. The paper's findings underscore the significance of teaching language models to ask better questions, potentially impacting various domains like healthcare and education by improving dialogue effectiveness.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": true, "final_justification": "", "rebuttal": "Thank you for your positive evaluation of our work!\n\nRe 1: OPEN and GATE rely on prompting frozen closed-source models like GPT-4. Our work focuses on an end-to-end setting where we teach the questioner to ask better questions which allow it to provide more personalized responses. This approach is different from OPEN and GATE. We will make the difference between OPEN, GATE, and STaR-GATE more explicit in our paper.\n\nRe 2: We have now finetuned a second model (see our responses to reviewers e2VC and b7r10). Also, note that we compare win-rates and log-probabilities across 9 different model checkpoints, with 3 training setups (STaR-GATE, w/ Gold Resp., w/o Reg.) and 3 iterations of self-improvement each (Figure 3). \n\nRe 3: STaR-GATE combines preference elicitation (as explored in GATE and OPEN) with a self-improvement loop. While our bootstrapping loop is inspired by STaR, it differs in several key aspects: (1) We apply bootstrapping to the problem of multi-turn, free-form interactive questioning. (2) We introduce response regularization to prevent the model from forgetting how to respond. Applying the original STaR fine-tuning loop directly to our setting would result in overfitting to the questions and not allow the model to learn how to provide good responses (orange line in Fig. 3a).\n\nOur filtering mechanism for selecting informative questions based on their impact on gold response probabilities is similar to the perplexity-based filtering used in Toolformer (Step 3 in Figure 2). We appreciate this insight and will make sure to acknowledge and clarify this connection in our related work and methods sections.\n\nQ.1. Our primary focus in STaR-GATE is on handling task ambiguity arising from important information being hidden from the model, rather than cases where all necessary context is provided upfront. Learning to determine when follow-ups are actually required is an important open problem for future work, but not one we explicitly address here. We will make this more explicit in our discussion.\n\nQ.2. We limited response lengths due to the cost of training models with longer sequence lengths. While future work could explore longer interactive settings, we restricted our research setting to K=3.\n\nQ.3. We did not find a clear trend indicating that longer conversations (i.e., K=3) consistently led to higher win rates than shorter ones (e.g., K=1). We will add a new plot to the appendix showing win rates broken down by conversation length across iterations.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* **S.1.** The paper is well written, contains a well detailed background and related work, and the illustrations are helpful and informative.\n* **S.2.** The proposed STaR-GATE algorithm outperforms the baselines on the newly proposed dataset.\n* **S.3.** The STaR-GATE is shown to generalized to new tasks and new user preferences.", "reasons_to_reject": "* **W.1.** The paper lacks evaluation comparison to existing works such as OPEN [1].\n* **W.2.** The evaluations are conducted on single fine-tuned model and a single synthetic dataset.\n* **W.3.** The paper lacks novelty and seems to be a combination of existing works [1][2] with similar filtering approach to that of ToolFormer [4].\n\n[1] Handa, K., Gal, Y., Pavlick, E., Goodman, N., Andreas, J., Tamkin, A. and Li, B.Z., 2024. Bayesian preference elicitation with language models. arXiv preprint arXiv:2403.05534.\n[2] Zelikman, E., Wu, Y., Mu, J. and Goodman, N., 2022. Star: Bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems, 35, pp.15476-15488.\n[3] Li, B.Z., Tamkin, A., Goodman, N. and Andreas, J., 2023. Eliciting human preferences with language models. arXiv preprint arXiv:2310.11589.\n[4] Schick, T., Dwivedi-Yu, J., Dessì, R., Raileanu, R., Lomeli, M., Zettlemoyer, L., Cancedda, N. and Scialom, T., 2023. Toolformer: language models can teach themselves to use tools. 2023. arXiv preprint arXiv:2302.04761.", "questions_to_authors": "* **Q.1.** How would STaR-GATE handle scenarios where follow-up questions are not required at all.\n* **Q.2.** Why was the limit of the conversation length (the number of turns) chosen to be $K=3$?\n* **Q.3.** Does it make sense to force the conversation length to be uniform? In practice, do longer conversation, that is more follow up questions, lead to higher win rate?", "extra_scores": {}}]

Author rebuttal:

Meta-review: All the reviewers are unanimous in agreeing that this paper has cleared the bar for publication quite comfortably. The paper itself follows the line of a very promising and long running line of NLP work on clarifying question asking. Most of the concerns raised are around scaling models/human data are admittedly beyond the scope of a single paper and do not detract from this paper showcasing a promising method that warrants further study.
Decision: Accept