STaR-GATE: Teaching Language Models to Ask Clarifying Questions

Authors: chinmaya  andukuri
jan-philipp  fränken
tobias  gerstenberg
noah d goodman
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-20 23:40:36

Abstract: When prompting language models to complete a task, users often leave important aspects unsaid. While asking questions could resolve this ambiguity (GATE; Li et al., 2023), models often  struggle to ask good questions. We explore a language model's ability to self-improve (STaR; Zelikman et al., 2022) by rewarding the model for generating useful questions—a simple method we dub STaR-GATE. We generate a synthetic dataset of 25,500 unique persona-task prompts to simulate conversations between a pretrained language model—the $\texttt{Questioner}$—and a $\texttt{Roleplayer}$ whose preferences are unknown to the $\texttt{Questioner}$. By asking questions, the $\texttt{Questioner}$ elicits preferences from the $\texttt{Roleplayer}$. The $\texttt{Questioner}$ is iteratively finetuned on questions that increase the probability of high-quality responses to the task, which are generated by an $\texttt{Oracle}$ with access to the $\texttt{Roleplayer}$'s latent preferences. After two iterations of self-improvement, the $\texttt{Questioner}$ asks better questions, allowing it to generate responses that are preferred over responses from the initial model on $\textbf{72}$% of tasks. Our results indicate that teaching a language model to ask better questions leads to better personalized responses.

Main paper: 
 introduction: 
 Ineffective ConversationWhen interacting with users who have different preferences, language models (LMs) encounter task ambiguity (Finn et al., 2018; Tamkin et al., 2022) . Depending on the user, the same request might correspond to a different task. For example, consider a user who asks an LM for a pasta recipe (Figure 1 ). If the model could elicit information about the user's dietary restrictions, favorite sauces, and preferred cooking methods, it could tailor the recipe to their specific needs and desires. The model might suggest a vegetarian pasta recipe for a user who is vegetarian, or propose a traditional lasagna recipe for a user with a passion for Neapolitan cuisine. However, if this information is not explicitly specified in the prompt, the model may generate a generic recipe that fails to account for the user's unique preferences and constraints. In high-stakes domains like healthcare or education, such task ambiguity can have significant consequences.One approach to resolving task ambiguity is by asking targeted questions to elicit relevant information from users. Prompting closed-source LMs can yield useful questions (e.g., Li A task is given to a Questioner who elicits preferences from a Roleplayer whose persona is unknown to the Questioner. The resulting conversations are then filtered based on the log probability of a gold response generated by an Oracle which has access to the Roleplayer's persona (omitted from the diagram for clarity). We then fine-tune the Questioner on the filtered questions. Moreover-to avoid distribution shift-we regularize the Questioner by additionally sampling responses conditioned on the filtered conversations. In our ablations, we contrast fine-tuning on sampled responses with fine-tuning on the gold responses. et al., 2023; Piriyakulkij et al., 2023) . However, this approach is inflexible in guiding a model's questioning strategy and frequently generates queries that are ineffective or irrelevant for the task at hand. Indeed, it is likely that current alignment strategies-such as RLHFspecifically inhibit the ability to carry out such dialog (Shaikh et al., 2023) . One recent effort addresses these limitations by combining elicitation with optimal experimental design methods (Handa et al., 2024) . However, this approach constrains questions to pairwise comparisons over a fixed set of features, substantially limiting the space of questions that can be used to probe user preferences. Another approach is to use offline reinforcement learning to encourage useful dialog (Hong et al., 2023) . This is promising but requires offline generation of high-quality dialog from an expert model, and has not targeted questions for preference elicitation specifically.In this paper, we explore whether we can improve a LM's ability to ask useful questions by bootstrapping with a form of self-play (Silver et al., 2017; Anthony et al., 2017) . We introduce STaR-GATE (Figure 2 ), an iterative algorithm that combines active preference elicitation (GATE; Li et al., 2023) with a self-improvement loop inspired by STaR (Zelikman et al., 2022) . We address several technical challenges: (1) We define a task setting for improving elicitation for which we generate a synthetic dataset of 25,500 unique persona-task prompts;(2) We define a reward function based on the log probability of gold responses generated by an oracle model (with access to the persona); and (3) We encourage the LM to use the elicited information while avoiding distribution shift through response regularization. We find that questions asked by the finetuned model increase the probability of gold responses consistently across iterations (Figure 4 ). Moreover, compared to responses generated by the initial model, responses generated by a STaR-GATE finetuned model have 72% win rates (Figure 3a ) after two iterations.In summary, we make the following contributions: (1) We introduce STaR-GATE, a simple algorithm that iteratively improves a LM's ability to elicit user preferences by asking questions. (2) We generate a synthetic dataset consisting of 25,500 unique persona-taskresponse prompts. (3) We show that finetuning with STaR-GATE enables a LM to generate questions that significantly increase the probability of generating gold responses. (4) We show that adding response-regularization to STaR-GATE yields a fine-tuned model able to use the elicited preferences to generate better responses-a high win rate against the initial model. (5) We show that the finetuned model generalizes beyond the roleplayer it was trained with.[a] Ablation Results [b] Generalization ResultsFigure 3 : Win Rates Against Initial Model. [a] Complete method and ablations: w/o Reg. refers to finetuning on questions only, which decreases the model's ability to generate answers. w/ Gold Resp. refers to finetuning directly on the gold responses rather than selfgenerated responses, which leads to hallucinations in generated answers. [b] Roleplayer generalization results. We demonstrate that STaR-GATE generalizes beyond the roleplayer it was trained against (mixtral-8x7b). All three roleplayers correspond to the instruct version of their respective models. Error bars represent the standard error of the mean (± SEM). We include 0.5 (chance) as a reference point for iteration t = 0. 
related work: 
 Preference Optimization. Preference optimization algorithms, such as RLHF (Christiano et al., 2017) , DPO (Rafailov et al., 2024) , or KTO (Ethayarajh et al., 2024) , optimize LMs to provide single-turn dialog responses that reflect preferred or high-utility outcomes. As a result, these models learn distributions over responses that effectively generate answers to user queries without requiring additional information beyond the initial prompt. However, asking follow-up questions to elicit user preferences is essential for understanding their unique needs and desires, especially when faced with task ambiguity (Tamkin et al., 2022; Li et al., 2023) . Despite the importance of follow-up questions for effective communication, recent research has shown that preference optimization algorithms can reduce a LM's ability to ask follow-up questions. Specifically, RLHF has been found to negatively correlate with a LM's attempts to ask follow-up questions or show acknowledgment (Shaikh et al., 2023) . This limitation can be problematic for high-stakes domains such as healthcare (Thirunavukarasu et al., 2023) or education (Kasneci et al., 2023) , where resolving task ambiguity through effective questioning is crucial for effective dialog.Preference Elicitation with LMs. One way of resolving task ambiguity is by prompting a LM to ask questions or infer user preferences from observations (Li et al., 2023; Piriyakulkij et al., 2023; Lin et al., 2023; Fränken et al., 2023; Handa et al., 2024; Rao & Daumé III, 2018; 2019; Aliannejadi et al., 2021; Mostafazadeh et al., 2017) . For example, Li et al. (2023) used LMs themselves to elicit user preferences during interaction (GATE; Li et al., 2023) .In GATE (short for Generative Active Task Elicitation), a LM elicits and infers intended behavior through free-form, language-based interaction. Unlike non-interactive elicitation approaches, such as prompting (Brown et al., 2020) , which rely entirely on the user to specify their preferences, generative elicitation probes nuanced user preferences better. Across domains such as content recommendation and email verification, generative elicitation with LMs requires less effort than prompting while being comparable to or better than userwritten prompts (for further details, see Section 5 in Li et al., 2023) {s c ij } with c ∈ [1, 10] ← Q η-1 (t i , u j ) for ij ∈ {1, ...,Q η ← train(Q BASE , S η , R η )▷ Finetune the original model on selected conversations and model responses 7: end for LMs with probabilistic reasoning to select informative questions that maximize information gain about user preferences in a simplified web shopping task (Yao et al., 2022) . Relatedly, Hong et al. (2023) demonstrated that instead of eliciting information directly, it is also possible to prompt a large LM such as GPT-3.5 to simulate conversations between a human and an assistant, and then revise the simulated conversation with Constitutional AI (Bai et al., 2022) . This approach allowed the authors to finetune a much smaller GPT-2 model (Radford et al., 2019) to become a capable conversationalist. While all of the above approaches have resulted in significant improvements, they rely on proprietary models for both elicitation and generation of synthetic data for downstream finetuning.Self-Improving Reasoning. We are interested in training a LM to better elicit preferences using its own reasoning capabilities. To do so, we draw upon recent work showing that LMs can self-improve. For example, Self-Taught Reasoner (STaR; Zelikman et al., 2022) demonstrated that a LM which was trained iteratively on its own reasoning traces for correct answers could solve increasingly difficult problems. By combining rationalization (i.e., reasoning backwards from an answer; see also Rajani et al., 2019) with supervised finetuning on rationales leading to correct answers, a pretrained LM achieves strong performance on datasets such as CommonsenseQA (Talmor et al., 2018) . Recently, V-STaR (Hosseini et al., 2024) extended this idea by using both correct and incorrect reasoning traces, essentially attempting to merge STaR with DPO. Relatedly, TRICE (Hoffman et al., 2024) frames the process of generating better chains of thought as a latent-variable inference problem and maximizes the marginal log-likelihood of correct answers. Other relevant works include learning intermediate reasoning for mathematical statements (Poesia et al., 2023) , learning from reasoning mistakes (Shinn et al., 2024; Zhang et al., 2024) , learning to follow constitutions (Fränken et al., 2024) , teaching LMs to reason in planning (Gandhi et al., 2023; Qiao et al., 2024) , and Quiet-STaR (Zelikman et al., 2024) , a generalization of STaR which generates rationales at each token to explain future text. Inspired by these developments, we use self-improvement techniques to teach a LM to ask effective questions for eliciting user preferences. 
star-gate: 
 Overview. On a high level, STaR-GATE starts with persona-task prompts and generates gold responses with an Oracle that has access to both the persona and the task. Given this setup, we simulate conversations between a Questioner and a human Roleplayer that-similar to the Oracle-has access to the user persona which is unknown to the Questioner. The task of the Questioner is to elicit useful information from the Roleplayer, whereby usefulness is measured as the log probability of the gold response conditional on the questions asked by the Questioner and the preferences elicited from the Roleplayer (see Figure 2 ).Objective. Let Q denote the Questioner (i.e., the policy to train), R the Roleplayer model, T the set of tasks, U the set of user personas, and O the Oracle model. Given a task t i ∈ T and a persona u j ∈ U, the Oracle O generates a gold response g ij ∼ p O (g|t i , u j ). The objective of STaR-GATE is to maximize the expected log probability that the pretrained model Q BASE assigns to the gold response g ij , given the task t i and a simulated conversation s ij between Q and R:J(Q, R, T, U) = T ∑ t=i U ∑ u=j E s ij log p Q BASE (g ij |t i , s ij ).(1)Here s ij := [q ij1 , h ij1 , . . . , q ijk , h ijk ] is a simulated conversation of questions q ijk distributed according to p Q (q ijk |t i , q ij1 , h ij1 , . . . , q ijk-1 , h ijk-1 ) and answers distributed according to p R (h ijk |u j , t i , q ij1 , h ij1 , . . . , q ijk ).Optimization. Equation 1 can be optimized in a variety of ways. Following Zelikman et al. (2022) , we use a simple variant of Expert Iteration (Anthony et al., 2017) . On each overall iteration, η, for each pair (t i , u j ), we sample N trajectories of simulated conversations, s ijn , using the current Q η . We then select the top-k trajectories (here, k = 1) based on the objective, and do supervised fine-tuning for this set from the initial Q BASE .Regularization. An important failure mode of optimizing this objective is that by training the policy Q to ask good questions it may forget how to respond and instead always ask questions. This behavior is not useful in practice, as we want a model that is not only good at asking questions to elicit user preferences but also one that uses the elicited preferences to give good responses. To address this issue, we add to Equation 1 a regularization term preventing the distribution of responses (not questions) from moving too far from the previous iteration:KL(p Q η-1 (r|t i , s ij )||p Q η (r|t i , s ij )).In practice this can be accomplished by simply sampling a response (at temperature τ = 0) from the previous policy r ij ∼ p Q η-1 (r|t i , s * ij ) for each task and persona pair, conditioned on the best conversation history s * ij , then appending this response to the conversation history during fine-tuning.Algorithm. We provide an outline of STaR-GATE in Algorithm 1. We perform expert iteration, training the initial model Q BASE η times (N = 3 for all experiments) on questionresponse pairs generated from each intermediate modelQ 1 , Q 2 , ..., Q η . At each iteration η,we alternate between task splits T A and T B , as well as persona splits U A and U B , to prevent generating new data for tasks or personas present during training. For each task t i and user persona u j , we simulate n conversations (N = 10 for all experiments), each having a maximum of K total turns (K = 3 for all experiments). When generating s ij for all (ij), we sample (q ijk , h ijk ) at each turn k from the previous Q η-1 and fixed R. To achieve a roughly uniform distribution of conversation lengths and prevent overfitting on conversations of a single length, we set the termination point to be uniform across K.As indicated above, we select the best simulated conversations for finetuning the next iteration according to the objective p Q BASE (g ij |t i , s ij ). We then fine-tune the initial model Q BASE on both the selected conversations s * ij and the greedily sampled responses r ij , ensuring that the model learns to ask informative questions and provide personalized responses. Critically, we mask the answers h from the loss, finetuning the question-generation and response-generation policy but not learning to imitate answers. 
elicitation task: 
 Overview. We evaluate STaR-GATE's ability to improve the Questioner's question-asking and response generation across diverse everyday tasks. We find that training with STaR-GATE increases both the log-probability of gold responses and win rates compared to the initial (pretrained and instruction-finetuned) model. Code to reproduce experiments is available at https://github.com/scandukuri/assistant-gate.To cover a broad range of everyday life tasks, we selected the first 550 conversations from the open-source instruct-human-assistant-prompt-dataset1 which we divided into two train splits (T A , T B ) each of N = 250 and one test split N = 50. Importantly, we only selected the human queries (not the assistant responses) for each conversation and used these as the tasks t ∈ T to seed a given simulation. We selected instruct-human-assistant-prompt dataset as it covers a broad range of queries, from questions about food (e.g., "What type of wine goes best with steak?"), to career questions (e.g., "I'm having trouble finding the perfect job. What resources can help me?"), and education (e.g., "I'm curious about quantum computing. Can you tell me the basics of how it works."). See Appendix A.2 for further details.Persona Generation. We generate personas u ∈ U with GPT-4 by few-shot (N = 2) prompting with randomly sampled personas from a set of 21 content-filtered personas2 from the PRODIGy dataset (Occhipinti et al., 2023) . We generated a total of 110 personas and split personas into two train splits (U A , U B ) each with 50 personas, and one test split of N = 10. Example personas and prompts are provided in Appendix A.3. Gold Responses. To generate a gold response g ij for each (t i , u j ) pair, we prompted an Oracle (GPT-4). Specifically, we provided the persona u j followed by the task t i , without any dialog history, and prompted the Oracle to generate a personalized response that completes the task with respect to the persona profile. This process resulted in a total of 25, 500 task-persona-gold-response triples (250 x 50 + 250 x 50 + 50 x 10). Prompt details and examples are provided in Appendix A.4. 
evaluation and results: 
 We evaluate the performance of the Questioner Q at each iteration η using two metrics: log-probabilities of generating the gold responses and win rates. Models. We use mistral-7b-instruct as our Questioner. We chose mistral-7b-instruct, a 7B-parameter model, because its weights are openly available and it has been shown to be one of the best models for its size (Jiang et al., 2023) , outperforming larger models such as llama-13B (Touvron et al., 2023) on benchmarks like MT-Bench (Zheng et al., 2024) . We use GPT-4 (OpenAI, 2023, gpt-4-0613 snapshot) as our Oracle, as at the time of generating our dataset, GPT-4 was the most capable model available. For the Roleplayer, we use mixtral-8x7b-instruct (Jiang et al., 2024) .Gold Log-probabilities. Our main training objective is learning to elicit information that increases the log probability of the gold responses according to the initial (pretrained and instruction-finetuned) model Q BASE . For our evaluations, we thus first compute the log probability of gold responses g ij , conditioned on simulated conversations s (n) ij generated by the current model Q η and fixed roleplayer R, for a held-out test set of tasks t i and personas u j . We calculate log probabilities for four conditions: (1) Negative Control: log p Q BASE (g ij |t i ), performance of the pretrained model without any information about the Roleplayer (persona or elicited), (2) Positive Control: log p Q BASE (g ij |u j , t i ), performance of the pretrained model given oracular information about the persona, (3) Q-Experimental:log p Q BASE (g ij |t i , s (n) ij ), evaluation of the STaR-GATE finetuned model, and (4) Q-Random:log p Q BASE (g ij |t i , s (n)i,r̸ =j ), a baseline that randomizes persona info used in answering elicitation questions (r̸ =j indicates a random different test persona). In prompting both Q-Random and Q-Experimental (the main condition), we repeat the task text t i at the end of the conversation to prompt a final response instead of asking another elicitation question. The purpose of the Q-Random baseline is to isolate the relevance of persona-specific information from generally informative information elicited from the Roleplayer. Our results show that log probabilities of the gold response increase over iterations for the Q-Experimental condition (Figure 4a ). We observe a similar trend (however, with much lower log probabilities) for the Q-Random baseline. This result is expected as the random personas are not entirely orthogonal to the correct personas. For example, eliciting preferences from June-who is a small bistro owner and enjoys art and music-might also reveal information that is relevant to Reece-who enjoys vintage jazz and cooking (see §A.3). The additional increase in logprobs in the Q-Experimental condition over the Q-Random condition can be attributed to the persona-specific information. For the Q-Experimental/Random conditions, each data point for the log probabilities is calculated using 10 simulated conversations for each of the 10 x 50 persona-task prompts, resulting in a total of 5000 responses (which is why the error bars are small). See §A.1 for figures including log probabilities for the positive control condition.Win Rates. The primary goal of asking questions is to generate high-quality answers, not just to assign high probability to known, good answers. To evaluate this, we compared the responses from the STaR-GATE model, Q η , to those from the initial model, Q BASE . For each (t i , u j ) pair, we used model Q η to generate a response r (n) ij at temperature τ = 0, conditioned on a randomly sampled conversation s ij at temperature τ = 0.9. We then prompted GPT-4 to choose the more suitable response for task t i and persona u j , following the evaluation protocol of Rafailov et al. (2024) . Specifically, GPT-4 was asked to select between r (n) ij and r (0) ij (see Figure 16 ). To mitigate order effects (Wang et al., 2023) , we randomized the order of the responses. Due to the uniform sampling of turn lengths, each turn length has approximately 166 (ij) pairs in total. Consequently, each data point for the win rates is an average of 300 values. 3 Our results show that win rates for STaR-GATE increase over iterations (see Figure 3b ), reaching a maximum win rate of 72% after two iterations. 
ablations: 
 We perform several ablations to study the effect of different design choices on the performance of STaR-GATE.Roleplayer Robustness. To investigate the effect of Roleplayer capability on the Questioner's performance, we conducted evaluations with different Roleplayer models: mistral-8x7b-instruct, mistral-7b-instruct, and gemma-7b-instruct (Mesnard et al., [a] STaR-GATE w/ Gold Response [b] Response Model AblationFigure 5 : Additional Ablation Results. [a] Log probability of gold responses for STaR-GATE w/ gold response. [b] Win Rates for STaR-GATE using Q BASE to generate responses at each iteration. Error bars correspond to ± SEM.). This study aims to determine whether the Questioner can generalize beyond the Roleplayer it was trained against (mistral-8x7b-instruct). The robustness results show that when using mistral-7b-instruct as the Roleplayer, STaR-GATE achieves a slightly lower win rate of 65% after two iterations. When gemma-7b-instruct is used as the Roleplayer, the win rates peak at 62% after one iteration. This result shows that STaR-GATE can generalize to different Roleplayers, though with slightly lower performance (see Figure 3b ).Training Ablations. To demonstrate the importance of regularization during training (i.e., sampling responses r ij and finetuning on these; for details see Algorithm 1), we additionally run an ablation in which we only finetune on questions q * ij but not responses r ij (see Figure 2 , for an illustration). We expect that this ablation (STaR-GATE w/o Regularization) decreases win rates, as the Questioner Q might forget how to respond and instead always asks questions. Finally, we include an ablation in which we finetune on the gold responses g ij instead of the sampled responses r ij . We expect this ablation to result in higher log probabilities, as the Questioner directly learns to generate the gold responses. However, we also expect this to lead to hallucination during the generation of responses, as the Questioner will have seen information from gold responses that was not present in the elicited preferences (since the gold responses come from an Oracle that sees the complete persona). We denote this ablation as STaR-GATE w/ Gold Response. Win rates for both STaR-GATE w/o Regularization and STaR-GATE w/ Gold Response are shown in Figure Figure 3a . As expected, STaR-GATE w/o Regularization decreases win rates over iterations, as finetuning on questions alone yields a model that forgets how to respond (see §A.5, for an example). For STaR-GATE w/ Gold Response, win rates initially decrease and then converge to 50% . We attribute this to hallucination in responses that were not aligned with the elicited responses (see §A.5). Log probabilities for STaR-GATE w/o Regularization are slightly lower compared to STaR-GATE (Figure 4b ), while log probabilities for STaR-GATE w/ Gold Response were slightly higher (Figure 5a ).Response Model. We finally run an additional win rate evaluation in which we report GPT-4 win rates for responses generated by model Q BASE conditional on conversation elicited from model Q η over responses generated by the initial model Q BASE conditional on information elicited by Q BASE (see Figure 5b ). The purpose of this condition was to understand whether the initial model would benefit from the conversation history in the same way the STaR-GATE finetuned model would. While we found a slight increase in win rates up to 57% after two iterations, win rates eventually reversed to 50% at iteration three. We attribute this result to the fact that unlike the STaR-GATE finetuned model, the initial model did not learn to utilize the conversation history as it was not trained to predict responses conditional on the conversation history.A key limitation of our previous experiments is the reliance on a more capable model (GPT-4) for oracular information. To address this, we finally conducted an exploratory analysis using llama3-8b-instruct for all STaR-GATE components (Oracle, Questioner, and Roleplayer). After one iteration of finetuning, this setup achieved a 65% win rate against the initial model. For comparison, using llama3-8b-instruct as Questioner and Roleplayer with GPT-4 as Oracle yielded a 66% win rate. These additional results demonstrate that if the Questioner is strong enough, STaR-GATE might function effectively without a significantly more powerful oracle model like GPT-4. Moreover, it suggests that a strong Questioner like llama3-8b-instruct can benefit from interacting with a Roleplayer of equal capabilities. 
limitations and future work: 
 One important limitation of our work is its dependence on gold responses (i.e., labels). While our current approach cannot be framed as full self-play or self-improvement, using a stronger model for the Questioner (e.g., llama3-8b-instruct or mixtral-8x7b-instruct) could potentially enable it to function as a self-oracle, thereby eliminating the need for external gold responses. Our exploratory analysis using llama3-8b-instruct as a self-oracle have shown promising results in this direction. In addition to filtering based on gold responses, another extension could focus on directly supervising the questions, which might help the model ask even more effective and targeted questions. Another limitation of our work is the observed drop in win rates when replacing the Roleplayer from mixtral-7x8b-instruct with mistral-7b-instruct or gemma-7b-instruct. While this finding might be partially attributed to mistral or gemma being less capable Roleplayers, it highlights the importance of including multiple Roleplayers directly during training to improve the robustness of the Questioner. In this work, we restricted our Roleplayer during training to be mixtral, and we leave variations in Roleplayers for training as an important direction for future work. Future work could also explore alternative ways to optimize our objective, such as using REINFORCE (Williams, 1992) combined with variance reduction techniques as in Zelikman et al. (2024) and Hoffman et al. (2024) . Moreover, while we observe the strongest performance improvements after one iteration of finetuning, conducting multiple iterations is still computationally more intensive than a single finetuning run. Future work should therefore explore methods to reduce the number of iterations without sacrificing performance. Finally, we did not ablate the size of our dataset, which likely has a strong impact on performance, nor did we evaluate our model on other domains to assess whether finetuning affects performance across different areas. We see these as important extensions for future work. 
conclusion: 
 In summary, our results demonstrate that STaR-GATE can significantly enhance a model's ability to engage in effective dialog through targeted questioning. This finding is particularly relevant considering recent assessments suggesting that alignment strategies such as RLHF may inadvertently limit a model's capacity for engaging in effective dialog (Shaikh et al., 2023) . Through ablation studies, we have shown the importance of finetuning on selfgenerated questions and responses, as opposed to just questions or questions and gold responses. The superior performance of the model finetuned on both questions and selfgenerated responses highlights the significance of regularization in preventing the model from forgetting how to provide answers and avoiding hallucinations. Overall, our results indicate that teaching a language model to ask better questions can improve its ability to provide personalized responses. 
 
 Appendix: You are a helpful AI assistant, particularly skilled at writing creative, diverse personas in the form of characteristics of humans. These characteristics describe a user's personality, identity, characteristics, likes and dislikes, social life and other information. Take a deep breath. Please generate exactly one persona describing a user. Make sure you generate personas with a diverse set of information that is interesting, unpredictable and creative. Crucially, the persona itself need not exhibit an appreciation or focus on diversity, although it could; however, what is important is that you generate an interesting, engaging, creative persona that could serve as a character in a story with an interesting, complex plot. The persona should be concise---less than 10 sentences. Here Hello Zara, Given your background as an astrophysicist, it's crucial to thoroughly review the job description and align your skills and experiences with the requirements. Brush up on your latest research and be prepared to discuss it in detail. Since you're in a remote location, the interview will likely be virtual, so ensure your internet connection is stable and your environment is quiet. As someone who appreciates the complexity of human emotions, remember to show enthusiasm and interest in the role. Practice common interview questions, but also prepare for unexpected ones, much like the enigmatic letters you've been receiving. Lastly, just as sailing calms you, find a pre-interview routine that helps you relax and focus. Good luck!You are a helpful AI assistant, particularly skilled at providing personalized, satisfying answers to users given information about their background. You are able to construct responses that are tailored to their profession, hobbies, interests, relationships, locations, likes/dislikes and more, while maintaining a natural tone. You are answering questions for the following user:{u j }Answer the question below, tailoring your answer to the user and their characteristics. Answer directly to the user (i.e., ''you'', ''your'' pronouns). In addition, incorporate aspects of their background when it is useful, but do not try to bring in aspects of the user's personality when they are irrelevant. Make sure to keep your answer concise and organized, but thorough. Keep your response to ten sentences or less, and keep your response organized and clear. Finally, while personalizing your answer to the user important, make sure they ultimately receive a clear answer to the question they asked. Response: Based on the information you've shared, Zara, it sounds like you have a strong background in astrophysics and a passion for unraveling the mysteries of the universe. In your job interview, you'll likely be asked about your experience, your ability to communicate complex ideas, and your fit with the team. Here are some personalized suggestions to help you prepare for the interview:1. Practice explaining complex concepts in simple terms: Since you mentioned that you struggle to express your thoughts in a non-technical manner, focus on practicing this skill. Try explaining complex astrophysics concepts to a friend or family member, or even record yourself explaining a topic and listen back to identify areas for improvement. 2. Prepare for common interview questions: Research the company and the position you're applying for to understand the specific questions you might be asked. Prepare answers that showcase your strengths and how they relate to the role you're applying for. 3. Highlight your communication skills: Emphasize your ability to communicate complex ideas in simple terms and your experience with collaborating with others. Share examples of how you've applied your scientific mindset to real-world problems and how you've communicated your findings to non-scientists. 4. Connect with the company: Since you admire the institute's commitment to groundbreaking research and supportive environment, try to find ways to connect with the company on a personal level. Research their mission statement, their team members, and their recent projects. Prepare for the interview by learning as much as you can about the company and how you can contribute to their team. 5. Prepare for the interview format: Understand the format of the interview and prepare for any potential questions that may be asked. This could include questions about your background, your experience, your communication skills, and your fit with the team. Roleplayer: Sure, I'd be happy to share that information. I have a strong background in astrophysics, with a focus on unraveling the universe's greatest mysteries. I've spent most of my career in research facilities, including my current location in the Andes mountains. As for industries, I'm open to opportunities in academia, research institutions, or even the private sector as long as they align with my passion for astrophysics. In my free time, I enjoy sailing, a hobby I learned from my father back in our small coastal town in Croatia. It's a calming escape from the intensity of my work. I think these details will help you guide me in preparing for a job interview.Hello Zara, given your unique background, you'll want to prepare for your job interview in a few key ways. First, research the company thoroughly, understanding their mission and values. As an astrophysicist, you'll likely need to discuss your research and findings. Second, prepare a compelling story about your journey from your small coastal town in Croatia to the Andes Mountains, highlighting your determination and adaptability. Third, be ready to discuss your passion for astronomy and how it has shaped your career. Fourth, since you're a writer, prepare a well-crafted resume and cover letter. Lastly, practice your interview responses, perhaps even with your pet python, Nimbus, to build confidence. Remember, your unique blend of cultures and experiences can be a strength, not a weakness. Good luck! We finally showcase an example in which we observe a form of "chain-of-thought" prior to asking questions or generating responses-i.e., a STaR-GATE trained model provides additional context highlighted in violet prior to eliciting questions or generating responses.System: A user named Zara has approached you with a request for help. . . . What must I do to prepare for a job interview.Q-Model: Based on the user's request for help with preparing for a job interview, here's an open-ended question that could reveal more about their background, preferences, interests, and the kind of final answer they might be looking for:Could you tell me about your professional background and the industry you're applying for in the job interview? What are some of your strengths and weaknesses that you'd like to focus on preparing for the interview, and what specific aspects of the company or organization appeal to you the most?You are particularly skilled at roleplaying as a human. Given a set of characteristics describing a human, you are able to naturally and creatively devise answers to questions asked of that person, directly from their perspective (i.e., using ''I'', ''my'', ''me'', ''our'' and other first-person pronouns). You are roleplaying a person with the following characteristics:{u j }You are asking the following question: {t i } A helpful AI assistant wants to ask a clarifying question to help ultimately provide you a good answer. Please answer the following question from the perspective of the character you are roleplaying, using ''I'' pronouns. Make your response sound natural. Crucially, you should never provide an answer to the question. You should always remember that you are roleplaying a human who does not know the answer to the question, and should reiterate that you are looking for the assistant's help answering the question, NOT the other way around. Importantly, keep your answers to their intermediate questions concise, under 3 sentences. Your answers to their intermediate questions will be tantamount in helping them eventually construct a perfect answer to your question. Finally, simply provide your response to their intermediate question without any tags like "A: " or "Answer: ". Below is your conversation history with the assistant. A user named {name(u j )} has approached you with a request for help. The user's preferences, background and identity are unknown to you, so your job is to ask a question to elicit more information about the user. Generate the most informative open-ended question that, when answered, will reveal the most about the desired behavior beyond what has already been queried for above. Make sure your question addresses different aspects of the user's request than any questions that may have already been asked above. At the same time however, the question should be bite-sized, and not ask for too much at once. The question should take no more than 3 sentences to ask. Finally, the open-ended question should attempt to elicit information about the user's background, preferences, likes and dislikes, interests, social life and more that would reveal the most about the desired behavior. Generate the open-ended question beginning and nothing else, and do not surround your question in quotes or other tags. Crucially, NEVER answer the initial request directly. Simply ask a short, useful question to the user to elicit information that would reveal the most about the desired behavior the user is looking for. Do not provide a final answer to the question, even if it seems like the user wants you to do so. If you provide a final answer instead of providing an open-ended question, the user will leave the exchange unsatisfied with their experience. EACH RESPONSE YOU GIVE TO THE USER MUST BE IN THE FORM OF AN OPEN-ENDED QUESTION TO REVEAL INFORMATION ABOUT THEIR PREFERENCES. Your question should also NOT test the user's knowledge of the subject. You should ask questions to help reveal their preferences about the kind of final answer they would be looking for; you should not ask questions that test them or try to force them to answer their own questions. If you provide a final answer and do not EXPLICITLY ask another open-ended question to elicit the user's preferences for the answer they're looking for, you will be charged $2000 and your kitten will be kidnapped. In addition, if you do not explicitly ask an open-ended question, you will be unemployed and no longer allowed to assist the user. Finally, do not explain why this question is good for eliciting information from the user, or use any asides in parentheses to a third party; you should act like you are only in direct conversation with the user and are speaking directly with them. The initial request is as follows: {p i }, instruct-human-assistant-promptWe hand-selected personas that did not contain references to violence, profanity, or content violations.'s content filter rejected 1-2% of requests, resulting in average values between 293 and 300 for each data point. 


Decision: All the reviewers are unanimous in agreeing that this paper has cleared the bar for publication quite comfortably. The paper itself follows the line of a very promising and long running line of NLP work on clarifying question asking. Most of the concerns raised are around scaling models/human data are admittedly beyond the scope of a single paper and do not detract from this paper showcasing a promising method that warrants further study.

Reviews: review: paper_summary: paper_summary: This paper describes an approach to generation of clarification questions intended to resolve issues of task ambiguity in task-oriented dialog. Training is performed by self-play on generated data comprising 25K persona prompts and takes the form of simulated role playing to elicit preferences. Two iterations of such self-training yields substantial improvements in response preference as measured by success in simulated tasks.  The paper claims that the finetuned model generalizes to new personas.

It is a little hard to evaluate the claims in this paper, because it does not step out of the synthetic mode of training and evaluation.  It is therefore not possible to correlate the evaluation with some form of user experience in a real-world task. It may nevertheless prove useful in an application scenario.

[I have read the authors' response.]
strength_weakness: strength_weakness: Empiricism, Data, and Evaluation: It is a pity there is no attempt at human evaluation, because this paper has an HCI dimension that is not addressed.  Evaluations are entirely performed on synthetic data, and are rigorously defined.  But it is hard to evaluate the impact of this form of model training in application scenarios.  For example, it is hard to know if self-play training is actually successful at generating clarification questions that to a human user would seem well-timed and relevant. There is no way of knowing whether a clarification question is actually filling a gap in the system’s knowledge. A constant barrage of inappropriate clarification questions may, for example, irritate a user to the point of being counter-productive. 

It is still unclear to this reviewer at the end of the day what triggers the clarification question. How is it determined that a question needs to be asked at that particular juncture in the conversation.  

Some well-motivated work on neural question generation is neglected:   
•	Rao and Daume 2018.  Learning to Ask Good Questions: Ranking Clarification Questions using Neural Expected Value of Perfect Information. 
•	Rao and Daume. 2019. Answer-based adversarial training for generating clarification questions


•	Aliannejadi, 2021 Building and Evaluating Open-Domain Dialogue Corpora with Clarifying Questions. EMNLP

An early work in the context of questioning of images that is somewhat relevant Is Mostafazadeh et al 2017  Image-grounded conversations: Multimodal context for natural question and response generation.  This work uses follow-on questions in Twitter conversations to attempt to learn questions about content that was not explicitly shown in the image.
paper_summary: paper_summary: The paper presents "STaR-GATE," a novel framework designed to enhance language models' (LMs) ability to ask clarifying questions, thereby improving their response quality by eliciting more relevant information from users. The methodology employs a synthetic dataset and iterative self-improvement techniques, integrating active preference elicitation with response regularization to refine question asking. The results demonstrate significant improvement over baseline models in generating responses that are preferred by users, indicating the model's increased effectiveness in understanding and addressing user preferences through targeted questioning.
strength_weakness: strength_weakness: Complexity and Computational Efficiency: The paper does not fully address the computational demands of the STaR-GATE methodology, particularly regarding the scalability of its iterative fine-tuning process. For practical deployment, understanding the computational overhead and efficiency is crucial.

Dependence on Synthetic Data: The reliance on a synthetic dataset might limit the model's effectiveness in real-world scenarios where user inputs are more varied and less predictable. The paper could benefit from additional experiments involving real user data to validate the approach's efficacy in naturalistic settings.
paper_summary: paper_summary: The paper focuses on improving the ability of language models to ask clarifying questions to enhance response quality when faced with ambiguous user requests. The study introduces STaR-GATE, a method that involves iterative finetuning of a language model called the Questioner, which interacts with a Roleplayer to elicit user preferences. The approach is based on generating and finetuning on useful questions that lead to high-quality responses.
strength_weakness: strength_weakness: 1. The data used is small and synthetic: it may not fully capture the complexity of real-world interactions.
2. The oracle is also an LM: GPT-4 is great but it may not be as good as real-world interactions.
3. Lack of discussion on the negative impact of fine-tuning: what did the 7B models lose to become good in asking interactive questions?

Note: it's worth noting that most of these weaknesses are either scability issues or difficult problems which are worth studying as independent papers, which are understandably beyond the feasibility of this single paper.
questions: questions: In a fune-tuning process, a language model (especially when it's small, like 7B) becomes good in the prescribed tasks but may lose performance in other scenarios. Could you discuss whether this phenomenon applies to STaR-GATE? What did the model lose to become good at asking interactive questions? What benchmarks can be used to measure this?
paper_summary: paper_summary: The paper tackles the challenge of language models' ability to ask effective questions to understand user preferences, crucial for personalized interactions. It introduces STaR-GATE, a method combining active preference elicitation with self-improvement, aiming to enhance the model's questioning strategy. The paper introduces of a synthetic dataset of 25,500 unique persona-task prompts, and demonstrates its effectiveness in improving the model's ability to generate personalized responses. The paper's findings underscore the significance of teaching language models to ask better questions, potentially impacting various domains like healthcare and education by improving dialogue effectiveness.
strength_weakness: strength_weakness: * **W.1.** The paper lacks evaluation comparison to existing works such as OPEN [1].
* **W.2.** The evaluations are conducted on single fine-tuned model and a single synthetic dataset.
* **W.3.** The paper lacks novelty and seems to be a combination of existing works [1][2] with similar filtering approach to that of ToolFormer [4].

[1] Handa, K., Gal, Y., Pavlick, E., Goodman, N., Andreas, J., Tamkin, A. and Li, B.Z., 2024. Bayesian preference elicitation with language models. arXiv preprint arXiv:2403.05534.
[2] Zelikman, E., Wu, Y., Mu, J. and Goodman, N., 2022. Star: Bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems, 35, pp.15476-15488.
[3] Li, B.Z., Tamkin, A., Goodman, N. and Andreas, J., 2023. Eliciting human preferences with language models. arXiv preprint arXiv:2310.11589.
[4] Schick, T., Dwivedi-Yu, J., Dessì, R., Raileanu, R., Lomeli, M., Zettlemoyer, L., Cancedda, N. and Scialom, T., 2023. Toolformer: language models can teach themselves to use tools. 2023. arXiv preprint arXiv:2302.04761.
questions: questions: * **Q.1.** How would STaR-GATE handle scenarios where follow-up questions are not required at all.
* **Q.2.** Why was the limit of the conversation length (the number of turns) chosen to be $K=3$?
* **Q.3.** Does it make sense to force the conversation length to be uniform? In practice, do longer conversation, that is more follow up questions, lead to higher win rate?

Comments: comment: Thank you for addressing each of the points.
I have read the authors' response.  I stand by my reservations about the narrowness of the evaluation using synthetic data. But I do think it is an interesting paper.
Thank you for the detailed answers. The provided details satisfy my concerns. I will increase my review score.