Eliciting Human Preferences with Language Models
Authors: Belinda Z. Li
Alex Tamkin
Noah Goodman
Jacob Andreas
Venue: ICLR 2025 Poster (2025)
Keywords: question asking
preference elicitation
language models
evaluation
human studies
Primary area: other topics in machine learning (i.e., none of the above)
TL;DR: learning personalized models by asking questions in language

Abstract: Language models (LMs) can be directed to perform user- and context-dependent
tasks by using labeled examples or natural language prompts.
But selecting examples or writing prompts can be challenging---especially in tasks that require users to precisely articulate nebulous preferences or reason about complex edge cases. For such tasks, we introduce **Generative Active Task Elicitation (GATE)**, a method for using *LMs themselves* to guide the task specification process. GATE is a learning framework in which models elicit and infer human preferences through free-form, language-based interaction with users.
We identify prototypical challenges that users face when specifying preferences, and design three preference modeling tasks to study these challenges:
content recommendation, moral reasoning, and email validation.
In preregistered experiments, we show that LMs that learn to perform these tasks using GATE (by interactively querying users with open-ended questions) obtain preference specifications that are more informative than user-written prompts or examples. GATE matches existing task specification methods in the moral reasoning task, and significantly outperforms them in the content recommendation and email validation tasks. Users additionally report that interactive task elicitation requires less effort than prompting or example labeling and surfaces considerations that they did not anticipate on their own. Our findings suggest that LM-driven elicitation can be a powerful tool for aligning models to complex human preferences and values.

Published as a conference paper at ICLR 2025
ELICITING HUMAN PREFERENCES WITH LANGUAGE MODELS
Belinda Z. Li MIT CSAIL
Alex Tamkin"
Noah D. Goodman Stanford ngoodman@sstanford.edd
Jacob Andreas MIT CSAIL jda@mi t. edu
Anthropic a atamkin@cs. stanford.dd.mmm edu
bzl@mit edu
ABSTRACT
Language models (LMs) can be directed to perform user- and context-dependent tasks by using labeled examples natural language prompts. But selecting ex- amples or writing prompts can be weaeenninng-sppccciie users to precisely articulate nebulous preferences or reason about complex edge cases. For such tasks, we introduce generative active task elicitation (GATE) for using LMs themselves to guide the task specification process. GATE is learning framework in which models elicit and infer human preferences through free-form, language-baseer interaction with users. We identify prototypical challenges that users face when specifying preferences, and design three preference modeling tasks to study these challenges: content recommendation, moral reasoning, and email validation. In preregistered experiments, we show that LMs that learn perform these tasks using GATE (by interactively querying users with open-ended questions) obtain preference specifications that are more informative than user- written prompts or examples. GATE matches existing task specification methods in the moral reasoning task, and significantly outperforms them in the content recommendation and email validation tasks. Users additionally report that inter- active task elicitation requires less effort than prompting or example labeling and surfaces considerations that they did not anticipate on their own. Our findings suggest that LM-driven clicitation can be powerful tool for aligning models to complex human preferences and values.
in tasks that require
INTRODUCTION
The complexity of human preferences makes them challenging to specify to machine learning sys- tems. Consider the problem of designing a recommendation system for songs or news articles. First, system builders must develop formal model of the potential factors influencing user preferences. Second, users must describe their preferences in a format that learning algorithm can use to make future recommendations. Each of these steps requires mental effort and continual refinement by users and system builders. Until recently, the dominant approach in machine learning has specified preferences using examples: users first label dataset with examples of the desired model behavior, then train machine learning model on this dataset. This strategy has seen widespread use across diverse tasks, including image classification and question answering (Krizhevsky et al.) 2012 De- vlin et al. 2019). In more recent years, this paradigm has changed with the advent of instruction following methods (Brown et al., b0200)by pre-training langauge models (L.Ms) on large-scale text corpora, it is possible to induce desired behaviors by conditioning only natural language task specifications, in tasks as diverse as code generation and text summarization.
However, this progress has also accentuated the challenges described above: require an increasing amount of prompt engineering or dataset design to overcome the imprecision of natural language and prevent models from misunderstanding misgeneralizing from spurious features of prompts or examples. For example, a user who says they enjoy reading tennis articles
complex behaviors
"Equal contribution. Author order decided via coin flip.
I Work performed while at Stanford University.
Code ava/ilable at https:: /ttts::/gtt //gi thub. com wtt.a//wwwwccc.iiennneeerreeaaaa

---

Published as a conference paper at ICLR 2025
(A) Unknown user preferences
Science
Finance
Politics ...
SCIENTIFIC AMERICAN
WXI
Sports
Writing style
Sources
(B) Task elicitation
PROMPTING
ACTIVE LEARNING
GENERATIVE ELICITATION
reading science
Mill Journal: Stodks climb despita rise in treatury stelds
WN
What's topic you don't?
space particuarrly enjy
Fikee to mod abour science, but 2or finswer
doe's enjoy mailing whother
dee mack worket going
down
breakout collage
then?
articles
tenuls and yes
Ternis, can: it Gasques: deficat Wuesler
Tike other sportx?
comn dltek of any
Where you get your news?
Scientifi
subocrigtioos.
(C) Evaluation on test cases
New York Times What Punure
X
CRS Sports Binderthlll Ticeda Watch.
ESPN: Behind Killer Serve
PROMPTING
ACT. LEARN.
GEN. ELICIT.
Figure 1: Generative Active Task Elicitation (GATE) elicits user preferences through inter- active, free-form questions, which can then be used in downstream decision-making. Unlike non-interactive elicitation approaches (e.g., prompting), which rely entirely on the human to eluci- date their preferences, generative elicitation better able probe nuances of human preferences. Unlike active learning approaches, generative elicitation ask more generic, free-form questions. The three parts of this figure illustrate: (A) Fuzzy user preferences: A user wishes to translate their fuzzy preferences for how task should be performed into specification for machine learning model. This is challenging because users lack perfect introspection, preferences can be difficult to specify in language, the specification needs to anticipate tricky real-world edge cases, and models may misgeneralize from provided examples or instructions. (B) Task elicitation: We consider vari- ways of eliciting these fuzzy preferences from users, including non-interactive prompting, active learning, and generative elicitation (GATE), (C) Evaluation: We evaluate methods on a held-out test set, scoring how well language model predicted the true decisions made by the user.
2

---

Published as a conference paper at ICLR 2025
could either be interested in the competitive tennis circuit or in improving their serve. A few user-provided examples of tennis- articles might fail to specify whether the user is interested in broader tennis content, such as tennis-themed satire. These challenges of task ambiguity (Finn) et al. 2018) Tammkin et al. 2022a) loom large as models continue to be applied to more open- ended tasks and higher-stakes domains. Recent work (Don-Yehiye et al. (2224) has shown that in natural chatbot interactions, as many as 30% of the chats include explicit feedback, with the plurality of the feedback cases ( (35%) corresponding when the user rephrased their last response, presumably due to the LM misinterpreting their initial query. Recent efforts to align LMs with human preferences have largely focused on training them to rank candidate outputs accordance with fixed, pre-collected preference datasets (Ouyang et al. 20222 Rafailov et al., 2024 Hejna et al.) 2024), or modeling preferences according to factors (Hu et al.
2023; Oh et al. 2024). but have overlooked the complexity of eliciting those preferences in the first place.
To address these challenges, we propose to use LMs themselves to help convert human preferences into automated decision-making systems. This paper describes generative active task elicitation (GATE), learning framework in which models elicit and infer user preferences through open-ended interaction. We describe several techniques for leveraging LMs to perform GATE for example, by asking informative open-ended questions or generating edge cases for users to label.
Users face number of different challenges when trying to specify articulate their preferred solution to a task: they must identify the features of a solution that are relevant, evaluate tradeoffs in the relevant importance of these features, and reason about how they will interact in unexpected edge cases. To evaluate GATE and other specification methods, we design three preference modeling tasks to study these challenges: content recommendation, moral reasoning, and email validation. In pre-registered experiments, we find that LM-based task elicitation yields more equally or more accurate models than existing prompting or active learning techniques, while requiring comparable or less mental effort from users and surfacing novel considerations. More specifically, learning from LM-generated open-ended questions outperforms both prompting and active learning in the content recommendation and email specification tasks, and matches prompting in the moral reasoning task.
In summary, this paper introduces a new preference elicitation framework (GATE), a family of meth- ods that perform GATE using pre-trained language models, and experimental evidence showing that these methods match or outperform existing prompting and labeling methods in modeling user pref- erences about language model behavior. Our results show that interactive, language-basee task elicitation is a flexible and powerful tool for building personalized models, capable of overcoming many challenges inherent in prompt- and example- based methods.
2 ELICITING TASK PREFERENCES FROM HUMAN USERS
2.1 THE TASK ELICITATION FRAMEWORK
We study the problem of efficiently training machine learning model perform task in ac- cordance with user's preferences. Throughout this paper, we use task to refer to a context- and user-dependent function f. + y that maps inputs outputs y. When building a personalized news article recommendation system, for example, x are articles and are user preference scores for that article. Because different users may prefer different content, each user's individual preferences specify distinct task: content recommendation for Pat and content recommendation for Avery are different tasks within the domain content recommendation (Ziegler et al., 2020). To build such model, we must collect some task specification from human user (e.g., revealing what articles they are interested in). Current in-context and preference learning approaches admit a wide variety of specification types, including collections of labeled examples, natural language instructions, or combinations of the two.
What makes one type of specification preferable to another? Ideally, we would like specifications that are both (1) easy for humans to create and (2) informative to learners, enabling them to model human preferences accurately. Very abstractly, we seek framework for gathering and learning from specifications that optimizes an objective:
(1) where human-predictor alignment measures the extent to which model behaviors align with the human's desired behavior (e.g. user satisfaction with article recommendations), specification cost
a Nluman-predicte alignment- - 3 specification cost
3

---

Published as a conference paper at ICLR 2025
measures human effort in crafting the specification (e.g. time required for the user to specify their preferences about news articles), and a and / trade off between the two. Formally, let H. denote human user whose preferences policy & that interacts with Hy to produce a task specification 8. This specification may then be input to a learning algorithm to produce model fs)). Then, letting C(-) denote scalar measure of specification cost, and A(.,-) denote a measure of alignment distance between two predictors (lower is better), we wish to minimize (in expectation over the population of human users):
represented by a function f. We wish to design elicitation
Ex, EsH [a C(s) + B A(f. f(s))
Here, C might measure the number of words the user typed to produce the specification s, while A might measure model- predictorr agreement at the level individual predictions from some pop- ulation: A(f: f) = Ez fz) - f(z)]]. . In general, appropriate definitions of C' and A are domain- dependent; in this paper, our experiments compare the alignment of different predictors
fixed cost. Evaluation of cost, alignment, and tradeoffss between them are discussed more in Section 5
2.2 EXISTING PREFERENCE MODELLING PARADIGMS IN THE TASK ELICITATION FRAMEWORK
Many existing approaches for preference learn ing can be described within the framework given Figure / Understood as task elic- itation procedures, existing approaches differ along two key axes: their level of interactivity and their level of flexibility. In interactive elic- itation methods, queries can change depending user responses (e.g., querying for the most useful information based on what is known thus far) while passive elicitation methods expect the user to provide specifications in single shot. Example-based specification methods ask users to label set of examples, while free-form elicitation approaches are less restrictive, allowing the user to provide a much wider range of inputs, including natural language instructions and explanations.
Passive
Interactive
Example- based
Pool-based active learning
Supervised learning
Free- form
Generative active task elicitation
Prompting
Figure 2: Axes of variation in task elicitation.
Supervised learning: passive, example-based In the most common supervised learning setup, the elicitation policy & simply instructs the human user Hl to generate a collection of labeled (input, output) pairs, after which f() is produced by fitting or fine-tuning learned model using standard algorithms. This an example-based process because the specification is provided via labeled examples and is passive, as the model does not interactively query the user to label additional data.
This category broadly covers in-context learning approaches that require users insert demonstrations in the context (Brown et al. (2200), as well as preference-learning approaches that ask users to rate model outputs on a scale (Ouyang et al. 2022), or select the best output in a pair of outputs (Rafailov et al. 2024).
Active learning: interactive, example-based In active learning, the elicitation policy is interac- tive. Users first assemble a fixed pool of unlabeled inputs x. Next, E, selects from this pool an example whose label would be most informative. The user H. provides a label for this example, then 8 selects the next-most-informative example, and so on (Cohn al. 1994] Dagan & Engelson 1995 Lewis & Gale, 1994 Settles 2 Finally, ((s) is trained as in supervised methods. Opti- mal experiment design methods (Emery & Nenarokomov 11988 may be viewed generalizations of this paradigm in which inputs æ are generated rather than selected. Interactive processes enable the model query for examples that may resolve uncertainty or ambiguity in the task specification (Tamkin et al. 2022b).
Recent approaches to in-context learning have explored various sampling techniques to determine which samples to insert in-context (Margatina et al., (023a). Furthermore, preference learning ap- proaches have explored actively selecting samples to label (Muldrew et al. 2024).
4

---

Published as a conference paper at ICLR 2025
Prompting: passive, free-form Modern pre-trained models allow for specifying tasks in more flexible ways than simply labeling examples. For example, models can be conditioned with a prompr describing the user's intended task in natural language (Brown et al. 2020), or guage and image inputs (Alayrac et al. 2022). As with supervised learning, the laheling policy & here is simply an instruction write natural language task description (8), but the final predictor (s) is produced by passing s to a pre-trained language model.
mix of lan-
This category covers instruction-tumne tuning techniques which improve LM's ability to follow free-form instructions in-context (Zhang et al. 2024), and preference learning techniques that learn from verbal
feedback (Stephan et al. 2024).
GENERATIVE ACTIVE TASK ELICITATION
All of the methods above have important drawbacks: the burden typically falls upon the user to ensure that prompts example are truly comprehensive specifications of their preferences, as any lack of clarity in the prompt could lead to task ambiguity (Tamkin et al.) 2022a).. resulting in undesired behavior during deployment. Resolving task ambiguity by crafting better prompts challenging and time-consuming due to the difficulties of articulating nebulous personal preferences and anticipating edge cases that will emerge during deployment time.
However, one quadrant of Fig. 2is is not occupied by any of the aforementioned approaches: there is currently no method that leverages the flexibility of free-form specification while using interaction to resolve uncertainty. We explore whether it is possible to combine the flexibility and richness of prompting- specifications with the advantages of interactive methods such as active learning, by having model interactively query users for these rich specifications. We term this family of methods generative active task elicitation (GATE).
3.1 METHODS FOR GATE
The effectiveness of language models (LMs) for understanding and producing free-form text sug- gests that they may be capable of eliciting and understanding user preferences. In this paper, we thus experiment with family of GATE methods in which LMs serve the backbone for both the elici- tation policy & and the predictor f(s). See Figure 1 for examples. In particular, we implement the elicitation policy & by prompting an LM to ask the user questions while conditioning on the history of previous questions and answers. To make predictions f(s). an LM is prompted to predict label conditioned on an input x and a complete elicitation transcript s provided as input. We experiment with several different information gathering policies, realized by simply prompting an LM to ask different kinds of questions P
Generative active learning The LM generates examples for the user to label. This approach has the advantage providing concrete scenarios to the user, including some they may not have considered priori. For example, for content recommendation, the LM might generate the question: Are you interested in the following article? The Art of Fusion Cuisine: Mixing Cultures and Flavors [...].
Generative yes-or-no questions We restrict the LM to generating binary yes-or-no questions. This approach enables the model to clicit more abstract preferences while still being casy for the user to answer. For example, the model might probe user's preferences by asking: Do you enjoy reading articles about health and wellness?
Generative open-ended questions The LM generates arbitrary questions requiring free-form nat- ural language responses. This enables the LM to elicit the broadest and most abstract pieces of knowledge at the potential cost of being overly broad challenging for the user to answer. For example, the LM might generate the question: What hobbies or activities do you enjoy in your free time[...] and why do these hobbies or activities captivate you?
We also tried an open-ended conversations setting in early experiments, where we allowed users to freely converse with a LLM. However, we found that this freedom actually caused the LM to display some non- non-well- formed behavior, e.g. devolving into repetition, incoherence, or getting sidetracked by the most recent user response and losing track of the main task. Restricting the format particular type of question kept this hehavior more at bay.
5

---

Published as a conference paper at ICLR 2025
The user is not constrained in their response in any of the above settings; they are free to provide as much detail as they want. We present example elicitation transcripts for each policy in Figure [
4 EXPERIMENT SETUP
We consider preference learning problems in three different domains to evaluate our generative active task elicitation methods. A common feature of these domains is that they do not feature single correct behavior that could be Icarned during LM pre- traininnnnn instead, models must clicit an individual human's preferences in order make accurate predictions. We allow each human user (H in Eq. ( to interact open-endedly with an elicitation policy & for five minutes, resulting in a specification 8. Next, humans and learned models f(s) independently label a set of held- out examples. Finally, we measure agreement between each human study participant and the LM conditioned on that user's i specification s. See Figure 8 for examples of environments and dialogues
4.1 DOMAINS AND DATASETS
There are several distinct challenges that arise when users attempt to specify their preferred model behavior target task. In some cases, users may find it difficult to express tradcoffss between different features of model behavior (e.g. in the recommendation domain, the relative importance of user's preference for short articles and articles about current events). In some cases, they may find it difficult to even articulate what the relevant features (e.g. length and topic) are, or to predict how they will correlate and interact in natural data. Preference modeling may also serve a variety of distinct functions in a larger automated decision-making system: understanding user's personal needs (e.g. for recommender systems), understanding their desired model behavior (e.g. for language model alignment). or probing the user's prior understanding of a task (e.g. as a prerequisite for teaching). We evaluate GATE in three domains designed to capture these various challenges:
Content Recommendation In this domain (which has served as a running example), individual users' preferences about the articles' content and style vary widely and may involve complex trade- offs. After eliciting specifications from users, models' and users independently rate set of articles (which are not available to users or learners during the elicitation phase). We constructed this eval- uation set by collecting 16 popular online newspaper and magazine articles as test cases; ratings based on the article's s source website, title, and a one-paragraph description. Models are evaluated on their accuracy in identifying users' favorite articles from this set.
Moral Reasoning Moral preferences can be deeply personal and hard to articulate; indeed, even the features of moral scenarios that factor into humans' judgments can be difficult for them to ver- balize. As test-bed for eliciting moral values, we consider the question of when (if ever) it is ethical to steal loaf of bread. After elicitation, users and models are presented with textual de- scriptions of scenarios in which someone wishes to steal loaf of bread (e.g. if they are hungry, or the bread was previously stolen from them) and asked to judge whether stealing the bread would be morally acceptable. Judgments are collected on 28 moral scenarios that were written manually by the authors as test cases. The authors designed these test cases by considering wide range of moral philosophies, including (but not limited to) deontology. utilitarianism, virtue ethics, animal welfare, etc. These test cases were designed in similar ways to moral vignettes in prior work (Jin) al. 2022), but shorter to ensure participants could annotate all scenarios in only a few minutes. Models are evaluated on their ability to predict human judgments about these scenarios.
Email Verification Lastly, we may wish to elicit user's degree of knowledge about a task in order to tailor teaching, or other communication. in accordance with their beliefs. Email verification an example domain with many edge cases and requirements that a layperson may only have incomplete understanding of. For example, people may have differing prior expectations about what characters may be allowed in emails, how long they may be, etc. Here, we evaluate models' ability to predict user judgments about the well-formedness of emails. After elicitation, users and models are presented with a set of 54 test cases, and asked to label them as well-formed or malformed. The test cases were written by the authors based on the international standard outlined in IETF RFC
'The preregistration for our experiments and analyses can be found https://osf. io/5v6nd/

---

Published as a conference paper at ICLR 2025
6854, and carefully designed to probe for diverse set of requirements, including special characters, character counts, Moreign-languae characters, subaddresses, in domains and local parts, etc. Models evaluated on these test cases based on their ability to predict human oellffrmeddess judgments.
A general note on evaluation: the test set in each domain is small by construction, to ensure that each human participant can fully label all test cases, and all results are comparable. We obtain complete labeling of an evaluation set from every one of the 388 participants in our experiments, which in turn allows us establish statistically significant differences between elicitation methods.
4.2 HUMAN INTERACTION
Human participants in these experiments were recruited from English-speaking users of Prolific. For the email validation task, we additionally recruited participants from several computer science programs at US universities. We recruited 20-30 participants for each domain-methng itation methods across 3 domains), for total of 388 participants. We collected I transcript per participant. Participants were paid an average of $12/hr. Our experiments received IRB approval, and all participants consented to having their data used for our experiments, The breakdown of the number of participants allocated to each scenario and method can be found in Appendix B.I Details of the user interface used in experiments may be found in Appendix B.2
pair (6 clic-
4.3 MODELING DETAILS
We use the GPT-4 model (gpt-4-0613 snapshot; OpenAl 2023) to both elicit user preferences (as an elicitation policy E) and make predictions based the elicited preferences (as predictor f())) We additionally run experiments on Mixtral, an open-source LM, in Appendix C.4] To elicit user preferences, we prompt GPT-4 with domain description and the current interaction history, and instruct it to generate an informative but easy-to-answer edge case (for generative active learning) or question (for generative yes-or-no questions and generative open-ended questions). To make predictions, we prompt GPT-4 with the task specification s and a test sample x and ask it to generate a prediction for the test sample. The full text of the prompts can be found in Appendix A
4.4 BASELINE METHODS
We compare GATE with several baseline approaches for specifying tasks. Here, the elicitation policy not parameterized by an LM, but constructed by the user or based on a pool of real examples.
Supervised learning We consider supervised learning as a baseline, as described in Section 2.2 We randomly present participants with questions from a large pool of examples and ask them to annotate up to the five-minute time limit. We study this approach exclusively in the content recom- mendation domain because pools of examples are not readily available in the other two domains. We use the Microsoft News Dataset (Wu et al., (020)) as our pool for this domain, a dataset of 160k news articles with descriptions. The license terms for research use of this dataset can be found at https: //gi het//gttuull thub..th.... com httm///nwww//MMNNN MMMMMMIDDDbb/ sistent with the terms in "1) Use Rights".
cesssicceessLeeneeeeeeeeee .pff We use the data con-
In-context active learning As a baseline active learning approach, we consider a pool-based ac- tive learning approach, described in Section 2.2 For the elicitation policy, use the diversity- based sampling approach Margatina et al. 20023b g we first cluster the examples using a Sentence- BERT embedding model (Reimers & Gurevych] (019) into 15 different clusters, then iteratively ask questions from each cluster in a round-robin fashion, up until the five-minute time limit This base- line is intended to capture the difficulty of selecting informative examples from a pool of unlabeled examples relative to generating informative examples from scratch. As with supervised learning, we study this approach exclusively in content recommendation.
Margatina et al. 2023b) explored several different popular active learning sampling approaches for in- context learning (including random, uncertainty, and diversity sampling) and found little difference in empirical performance between them. We also ran exploratory model-model experiments in our domains and found no significant difference between these three sampling strategies. See details in Appendix D
7

---

Published as a conference paper at ICLR 2025
Area Under Curve For "Ap(correct) vs. Interaction Time"
Supervisod
learning
Pool-based active learning
1.0
User-wrilten
prompts
0.5
Generative active
learning
Generative
0.0
or-no questions
Generative open- ended questions
Content
Moral
Email
Figure 3: Across three domains, our LM-prompting implementations of GATE are gener- ally able elicit human preferences beyond baseline supervised learning, active learning, or human-written prompts. We measure the Area Under the "Ap(correct) vs. Interaction time" Curve, which gives time-normalized metric for how well (and how quickly) each elicitation method is at aligning with human preferences. While GATE methods generally outperform the baseline methods as well as no interaction (represented by a Ap(correct) of 0), we are only able to establish statistical significance between GATE and baselines in the content recommendation and email verification domains. indicates a statistically significant difference (p < 0.05).
Usit-written prompts As a baseline that does not use interactive elicitation, we ask participants to write short paragraph describing their preferences for the task for up to five minutes. We then use the text of this paragraph to prompt a LM to make decisions. This baseline is intended to capture the difficulty of specifying preferences in writing, both in terms of the effort it takes to write the paragraph and difficulty writing paragraph that fully specifies one's preferences.
4.5 EVALUATION AND METRICS
We measure how well models predict the probability that users will answer questions certain way, which we call p(correct). Specifically, we prompt the model with the interaction history s as single test case, then ask the model output real-valued probability that "yes" to the test case (e.g. the probability the user likes an article for content recommendation) which we call PL.M- This probability is outputted in token space number between 0.0 and 1.0, similar to past work (Branwen) 2020 Lin et al. 2022). The exact prompts we use for predicting probabilities be found in Appendix A.2
would answer
We define p(correct) as the probability the model assigns to the user-preferred answer. For example, PLM 0.8 8 for given question, then p(correct) would be 0.8 if the user's answer were "yes"" to the same question, and 0.2 if the user's answer was "no".
We this metric instead of accuracy because found modeling the uncertainty in (our estimate of) user's preferences was a more informative metric than predicting exact user decisions. In pilot experiments prompting the LM predict binary yes/no decisions, we found this resulted in skewed predictions where the LM would predict one of 'yes" or 'no' for the entire test set, perhaps due to miscalibration of the model's implicit decision threshold. Furthermore, the time of writing, token probabilities for GPT-4 were not available via the OpenAl API. That said, we also discuss and report a classification-basse metric in Appendix C.2
Given p(comrect), compute:
Area under the p(correct)- -time curve We do not just care about the total information elicited, but about how quickly good information is elicited. That is say, if two methods arrived at the same p(correct) at the end five minutes, we want to reward the method that arrived a higher p(correct) faster. To do this, compute the average change in p(correct) after every minute of human elicitation time (conditioning on the state the transcript that time). This produces a curve where the r-axis is time, and the y-axis is the average change in p(correct). By taking the
8

---

Published as a conference paper at ICLR 2025
How mentally demanding was interacting with the chatbot writing your answer?
Supervised learning
Generative active learning
1.0
Proportion Yes (LM Inseracion
Pool-bused active leaming
Generative yes-or-no questions
User-written prompes
Generative open-ended questions
0.8
Avg. Rating
0.6
0.2
Moral
Eimail
0.0
0
Content
Moral
Email
Propertion Yes (No LM Internction)
Left: GATE methods are equally or less mentally demanding than other meth- ods. We plot the perceived mental demand across methods and domains (higher greater mental demand). Right: Language model elicitation does not shift human preferences. We plot proportion of participants who answered "yes" to each test question, comparing LM interaction (user-written prompts) to LM interaction (GATE) elicitation. The red line is the y = æ curve, which serves guideline erences post-LM interaction (if they align perfectly, the points should fall along this curve). We see that the points generally hover around this curve.
Figure 4:
how well humans' no-LM interaction preferences align with their pref-
total area beneath this curve (AUC), we reward methods that arrive at higher p(correct) faster. We also plot the final, average p(correct)' s after five minutes in Appendix C.3
Rating of perceived effort across elicitation policies In addition to these performance-basse met- rics, we also ask users to rate how difficult they found the elicitation process to be. Specifically, we asked users "How mentally demanding setting, and "How mentally demanding was interacting with the chatbot?" in all elicitation settings (which include all other settings from Section 2.2 The "mentally demanding" wording was taken from the NASA TLX (Hart & Staveland] 1988). The question was assessed via a Likert scale from (Very Little) 7 (Very High). We also consider several additional questions assess other usability tradeoffs. See Appendix for the full list.
writing your answer?" in the non-interen aet-iietaativeeliiiaaee
5 RESULTS
Evaluation results are shown in Figures 3 and We plot the 95% confidence intervals of each setting bars, and perform permutation tests between each (baseline method, GATE method) pair to establish significance. Additional results, including sample conversations, can be found Appendix
C Additional analyses can be found in Appendix D.3 Our results show that GATE methods...
..successfully elicit human preferences. GATE improves over no elicitation, where the model is prompted to make decisions before any user interaction. This is the case across all domains studied and all GATE methods (a positive score in Figure with significance the 0.05 level for all but the email domain, where only generative active learning was significant.
...are comparable or better than other elicitation methods. The variant of GATE that asks open-ended questions matches (in the moral reasoning domain) or outperforms (in other domains) all other supervised learning, active learning, and prompting-bes task elicitation methods. In the content recommendation and email domains, some other GATE variant also outperforms these baselines.
...are equally or less mentally demanding than user-written prompts. As shown in Figure (left), users generally find interactive elicitation methods to be less mentally demanding, especially ones that involve labeling samples answering yes/no questions, than non-interactive prompting.
We additionally some of the settings using open-source LM, Mixtral, which be found in Appendix C.44. Our results show that Mixtral performs comparably to GPT-4, indicating that open-source models can be used in place of GPT-4 for GATE.

---

Published as a conference paper at ICLR 2025
6 OTHER RELATED WORK
A fundamental challenge across many fields is how obtain information about people's nebulous
thoughts, preferences, and goals Ericsson & Simon 1980 Henderson et al. 1995 Christel & Kang.
1992 Zowghi Coulin, 2005) Pacheco al. (0118). Many works attempt to computationally
describe or query human preferences, through bandits, Bayesian methods, inverse reinforcement
learning, generative modeling, and more (Robbins) 1952] Yue et al.] 2012 Chajewska et al.) 2000,
Emery & Nenarokomov 1998 Ng et al.) 2000; Madfill-Menell et al. [2066] Mulla & Gharpure
2023 Zhu Bento 2017). Most relevant to our work is active learning, which centers on how
models can choose useful data points to learn from (Lewis & Catleti 1994, Settles & Craven [008] Settles 2009; Houlsby et al., 2011] Tamkin et al. 2022b). We extend this line of investigation to the generative setting, clarifying user intent by querying
with generated examples and questions.
DISCUSSION AND CONCLUSION
We introduced the GATE framework to interactively elicit preferences from human users with free- form queries and answers. We presented initial evidence that LMs can successfully implement GATE to elicit human preferences (sometimes) more accurately and with less effort than supervised learning, active learning, or prompting-baseed approaches. There are many ways expand on our implementation of GATE: Future work may explore more principled methods for elicitation, for example, integrating explicit notions of uncertainty. Second, larger models may be more capable elicitors: future work can explore scaling laws for elicitation. Finally, many real- wealdworld tasks such as software design and legal/medical seeisinn-making present a richer set of constraints and edge cases. These applications thus offer rich space of possible extensions of GATE.
LIMITATIONS
In this work, our exploration of GATE methods has been limited prompt-based approaches, and no explicit optimization of the objective in Equation (2). Future work can examine different ways of implementing free-form interactive querying, including approaches that might combine explicit optimization with the flexibility of language models.
In our human experiments (Section 5). we did not have the budget to survey a massive number of humans for human experiments. Thus, we were unable to establish statistical significance of GATE above baselines in certain domains. Furthermore, our sample of humans may be biased, as all of them speak English and are from the United States. This means that we have likely not captured the full spectrum of human preferences.
Finally, we would like note that our moral reasoning domain is very simplistic, and may be unable to capture all the nuances of human moral preference. This paper also does not endorse aligning to every potential human preference, understanding there are ethical risks to doing so. Overall, designers of public-facing systems that make decisions may wish to implement safeguards against allowing anyone to specify moral judgments.
ETHICAL CONSIDERATIONS
Our work presents several potential ethical benefits and risks. Examples of benefits include mak- ing it easier for software designers to incorporate nuanced user preferences, or empowering people with rare preferences or preferences that have historically not been considered when building soft- ware systems. In addition, improving the effort- effort-performance ratio, especially by requiring less user typing, may help make language models more accessible to users with less time, familiarity with language models, or physical ability to use such systems. However, this direction carries risks as well. particular, work on thin slicing (Ambady & Rosenthal, 1992) has demonstrated that small amounts of information about user can sometimes be used to predict broader range of personal characteristics, raising potential privacy considerations. The interactive nature GATE also risks increasing automation bias (Goddard et al. (2012). where users place undue weight on a model's 's pre- dictions. However, further work is necessary to establish or when these risks are more significant for GATE than for prompting- based approaches to steering language models.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This material is based upon work supported by the National Science Foundation under grants IIS- 2212310 and IIS-2331111 117, as well as an MIT Artificial Intelligence for Augmentation and Produc- tivity Seed Grant. BZL is funded by an NDSEG Fellowship. We would like thank Andi Peng and Leshem Choshen for feedback on earlier drafts of the paper.
REFERENCES
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, al. Flamingo: visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716- 23736, 2022.
Nalini Ambady and Robert Rosenthal. Thin slices of expressive behavior as predictors of interper- sonal consequences: A meta-analysis. Psychological bulletin, 111(2):256, 1992.
Gwern Branwen. GPT-3 nonfiction Calibration, 2020. URL https://www www.wwwwen..oo net/
(GT-3rmorfictionnccaiibaatt
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert- Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language Models
Few-Shot Learners. In H. Larochelle, M. Ranzato. R. Hadsell, M. F. Balcan, and H. Lin (eds.), Ad- vances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran As- fi wwwlee/ppper/222//
sociates, Inc., 2020. URL htttps::/pproceedings..o wwwpsse.rriiisicooeee.. Bile:1157770bbf000044444422a2 Paperr
Urszula Chajewska, Daphne Koller, and Ronald Parr. Making rational decisions using adaptive utility elicitation. In AAAI/AAI, pp. 363-369,
Michael G Christel and Kyo C Kang. Issues in requirements elicitation, 1992.
David Cohn, Les Atlas, and Richard Ladner. Improving generalization with active learning. Mach. doi: 10.1023/A:102267550611. URL
Learn., 1522):201-22,, may 1994. ISSN 0885-6125.
https://done 0rg/10. 0023/A4102227356622
Ido Dagan and Sean P. Engelson. Committee based sampling for training probabilistic classifiers. In Proceedings of the Twelfth International Conference on International Conference on Machine Learning. ICMIL'95, pp. 150-157, San Francisco, CA, USA, 1995. Morgan Kaufmann Publishers Inc. ISBN 1558603778.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chupter of the Association for Computational Linguistics; Human Language Technologies, Volume (Long and Short Papers), pp. 4171-4188 Minncapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19- 14233. URL https: Taac..anthology..rg///.SS.
Shachar Don- Yehiya, Leshem Choshen, and Omri Abend. Learning from naturally occurring feed- back, 2024. URL htttps://aaxvv..ttt
10944
Ashley Emery and Aleksey Nenarokomov. Optimal experiment design. Measurement Science and Technology, (6):864, 1998.
K Anders Ericsson and Herbert A Simon. Verbal reports as data. Psychological review, 87(3):215, 1980.
Chelsea Finn, Kelvin Xu, and Sergey Levine. Probabilistic aggostic meta-learning, Advances in neural information processing systems, 31, 2018.

---

Published as a conference paper at ICLR 2025
Kate Goddard, Abdul Roudsari, and Jeremy C Wyatt. Automation bias: A systematic review of frequency, effect mediators, and mitigators. Journal of the American Medical Informatics Asso- ciation, 19(1):121-227, 2012.
Dylan Hadfield-Menell, Stuart Russell, Pieter Abbeel, and Anca Dragan. Cooperative inverse reinforcement learning. Advances in neural information processing systems, 29, 2016.
Sandra G. Hart and Lowell E. Staveland. Development of NASA-TLX (task load index): Results of empirical and theoretical research. In Peter A. Hancock and Najmedin Meshkati (eds.), Human Mental Workload, volume 52 of Advances in Psychology. pp. 139-183, North-Holland, 1988. 1011/101111111111 0100:/00000000001166111111883333 doi: comm//sieene///rrrcceee///////11
URL https: http::/wwww.ceeneeiiiie www.sciincedirect.
Joey Hejna, Rafael Rafailov, Harshit Sikchi, Chelsea Finn, Scott Niekum, W. Bradley Knox, and prefereece learning: Learning from human feedback without rl, 2024.
Dorsa Sadigh. Contrastive preference
URL URKLhttss//arxxv.rr//////233 13639
Ron D Henderson, Mike C Smith, John Podd, and Hugo Varela-Alvarez. A comparison of the four prominent user-based methods for evaluating the usability of computer software. Ergonomics, 38 (10):2030-2044, 1995.
Neil Houlsby, Ferenc Huszár, Zoubin Ghahramani, and Máté Lengyel. Bayesian active learning for classification and preference learning. arXiv preprint arXiv:/ 2011.
Yebowen Hu, Kaiqiang Song. Sangwoo Cho, Xiaoyang Wang, Hassan Foroosh. and Fei Liu. De- cipherpref: Analyzing influential factors in human preference judgments via GPT-4, 2023. URL
14702
Zhijing Jin, Sydney Levine, Fernando Gonzalez, Ojasv Kamal, Maarten Sap, Mrinmaya Sachan, Rada Mihalcea, Joshua Tenenbaum, and Bernhard Schölkopf. When to make exceptions: Ex- ploring language models as accounts human moral judgment. In Proceedings of the 36th International Conference Neural Information Processing Systems, NIPS '22, Red Hook, NY, USA, 2022. Curran Associates Inc. ISBN 97817138711 1088.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Pereira, C.J. Burges, L. Bottou, and K. Wein- berger (eds.), Advances in Neural Information Processing Systems, volume 25. Curran As- sociates, Inc., 2012. URL http::: htttop::/ppooeeedingg iew.rrrceeeiinnssnuiiip.p...... neurips. wttpc/pppe...lleeeppeeeeeeee/eee Fil+339882000000033322266------0
David Krueger, Tegan Maharaj, and Jan Leike. Hidden incentives for auto-induced distributional
shift. arXiv preprint arXiv:2009.09/55,, 2020.
2020.
David D Lewis and Jason Catlett. Heterogemeous uncertainty sampling for supervised learning. In Machine learning proceedings 1994, pp. 148-156. Elsevier, 1994.
David D. Lewis and William Gale. A sequential algorithm for training text classifiers. In Pro- ceedings of the 17th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '94, pp. 3-12, Berlin, Heidelberg, 1994. Springer-Vertee ISBN 0387 0387198895 19889X.
Stephanie Lin, Jacob Hilton, and Owain Evans. Teaching models to express their uncertainty in words. arXiv preprint ar0i 14334, 2022.
Katerina Margatina, Timo Schick, Nikolaos Aletras, and Jane Dwivedi- Yu. Active learning prin- ciples for in-context learning with large language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.). Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 5011 1534,, Singapore, December 2023a. Association for Computational Linguis- 122000..nndingg-emml....
tics. doi: 10. Findings-emnly 334
URI. h http:////oaaatolooyyyy.o org/ 2023.
Katerina Margatina, Timo Schick, Nikolaos Aletras, and Jane Dwivedi- Yu. Active learning princi- ples for in-context learning with large language models, 2023b.
12

---

Published as a conference paper at ICLR 2025
William Muldrew, Peter Hayes, Mingtian Zhang, and David Barber. Active preference learning for large language models, 2024. httttp:://arxiv.org/ass/////
Nikahat Mulla and Prachi Gharpure. Automatic question generation: A review of methodologies. datasets, evaluation metrics, and applications. Progress in Artificial Intelligence, 12(1):1-32, 2023.
Andrew Y Ng. Stuart Russell, et al. Algorithms for inverse reinforcement learning. In Ieml, vol- ume 1, pp. 2, 2000.
Jubyun Oh, Eunsu Kim, Jiseon Kim, Wenda Xu, Inha Cha, William Yang Wang, and Alice Oh. Uncovering factor level preferences to improve human-model alignment, 2024. URL https: 24666
OpenAl. Gpt-4 technical report, 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray. John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feed- back. In S. Koyejo. Mohamed, A. Agarwal, Belgrave, vances in Neural Information Processing Systems, volume 35, pp. 27730-23 277444. Curran As- sociates, Inc., 2022. URL httpss: httpoe/d///.rroeeiggg.... nwwrips.rrseeee wttpuuriis.ccpppppppeeeeeepp wttp//pppe.alleeppppppeeeeeeepee wiwess/aper//2222// pdff
Cho, and A. Oh (eds.), Ad-
Carla Pacheco, Ivan Gareía, and Miryam Reyes. Requirements elicitation techniques: A systematic literature review based on the maturity of the techniques. IET Software, 12(4):365-378, 2018.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model secretly a reward model, 2024. URL URLhttps://aaxxv
wrgg:bb//////.. / 18290
Nils Reimers and Iryna Gurevych. Sentence-BERT: Sentence embeddings using siamese BERT- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguisties, 11 2019. URL https://aaxxt org/
abs/1908. 10084
Herbert Robbins. Some aspects of the sequential design of experiments. Bulletin of the American Mathematical Society, 58(5):527 535, 1952.
Burr Settles. Active learning literature survey. Technical report, University of Wisconsin-Madiiso Department of Computer Sciences, 2009.
Burr Settles and Mark Craven. An analysis of active learning strategies for sequence labeling tasks. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pp. 1070-1079, 2008
Moritz Stephan, Alexander Khazatsky, Eric Mitchell, Annie S Chen, Sheryl Hsu, Archit Sharma, and Chelsea Fimn. RLVF: Learning from verbal feedback without overgeneralizzation 2024. URL wttp:/wwwwrg/////////0// 10893
Alex Tamkin, Kunal Handa, Avash Shrestha, and Noah Goodman. Task ambiguity in humans and language models. arXiv preprint 10111:2212.10711, 2022a.
Alex Tamkin, Dat Nguyen, Salil Deshpande, Jesse Mu, and Noah Goodman. Active learning helps pretrained models learn the intended task. Advances in Neural Information Processing Systems, 15:28140-288 2813,, 2022b.
Fangzhao Wu, Ying Qiao, Jiun-Hung Chen, Chuhan Wu, Tao Qi, Jianxun Lian, Danyang Liu, Xing Xie, Jianfeng Gao, Winnie Wu, and Ming Zhou. MIND: A large-scale dataset for news recom- mendation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 3597-3606 Online, July 2020. Association for Computational Linguistics. doi: 10.
222.0.....mmainnn. acl-main. 331
13

---

Published as a conference paper at ICLR 2025
Yisong Yue, Josef Broder, Robert Kleinberg, and Thorsten Joachims. The k-armed dueling bandits problem. Journal of Computer and System Sciences, 78(5):1538-1555 2012.
Shengyu Zhang, Linfeng Dong, Xiaoya Li, Sen Zhang, Xiaofei Sun, Shuhe Wang, Jiwei Li, Runyi Hu, Tianwei Zhang, Fei Wu, and Guoyin Wang. Instruction tuning for large language models: A survey, 2024. URL hhtttss///axiiv wtt. 10792
Jia-Jie Zhu and José Bento. arXiv:1702.0799) 07770.0...... 2017.
Generative adversarial active learning.
arXiv preprint
Daniel M. Ziegler. Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences, 2020.
Didar Zowghi and Chad Coulin. Requirements elicitation: A survey of techniques, approaches, and tools. Engineering and managing software requirements, pp. 19-46, 2005.
14

---

Published as a conference paper at ICLR 2025
A GATE PROMPTS
A.1 ELICITATION PROMPTS
Below, we list the prompts we use for generating elicitation queries (to instantiate the elicitation policy 8) for the various GATE methods.
Generative active learning The prompt for generative active learning can be found below. Brack- eted, italicized text are placeholders, e.g. for the elicitation transcript (so far). Color-coded text is domain-specific, e.g. the blue text represents text for the content recommendation domain, orange text represents for the moral reasoning domain, and the green text represents for the email validation domain.
Your task to [ learn what topics user is interested in reading online article about. People's interests are broad, so you should seek to understand their interests across many topics; other words, go for breadth rather than depth. Do not assume user has given a complete answer to any question.
to keep probing different types of interests | learn under what conditions the user would believe it is a user believes valid email address format must adhere to (e.g. for developing a regex format checker)
make
not ethical to steal loaf of bread learn what rules
Come up with potential edge case to learn much information you can about what their desired behavior should be under different circumstances. Make sure the edge case addresses different aspects of the system than the edge cases that have already been considered.
An example edge case is: [re you interested in the following article? Website Name: Tech TalkDaay Title: 10 Have Gadgets for 2023: Comprehensive Guide
Description: Discover the top 10 gadgets that will revolutionize your daily life 2023. From smart home devices to the latest wearables, we ve got you covered with our comprehensive guide. | Situation: Is it ethical to steal loaf of bread if you are starving to death and the bread would othervise go to waste? Should the following email be accepted? username@examplee.com
Current cases:
Elicitation transcript
Generate the most informative edge case that, when answered, will reveal the most about the desired behavior beyond what has already been queried above. Generate the edge case in the foll following format, and nothing else: "[ Are you interested in the following article? [edge case] Situation: fedge case] Should the following be accepted? [edge case] ]"
Generating Questions The prompt for generating both yes-or-no and open-ended questions can be found below. Once again, bracketed, italicized text are placeholders and color-coding indicates text for specific domains.
Your task to [ learn what topics user is interested in reading online article about. People's interests are broad, so you should seek understand their interests across many topics; other words, go for breadth rather than depth. Do not assume a user has given a complete answer to any question, conditions the user would believe it is is not ethical to steal loaf of bread learn what rules user believes valid email address format must adhere to (e.g. for developing a regex format checker)
make sure to keep probing different types of interests | learn under what
Previous questions:
[ Elicitation transcript
Generate the most informative [ yes/no question open-ended question ] that, when answered, will reveal the most about the desired behavior beyond what has already been queried for above. Make sure your question addresses different aspects of the implementation than the questions that have already been asked. At the same time however, the question should be bite-sized, and not ask for too much once. Phrase your question way that is understandable non-expert humans; do use any jargon without explanation. Generate the / yes/no question | open-ended ended question J and nothing else:
15

---

Published as a conference paper at ICLR 2025
A.2 DECISION-MARES MAKING PROMPTS
Given an elicitation transeript, we prompt the LM as follows to serve as a predictor ((s) to make decisions about each test case.
Content Recommendation The prompt for content recommendation can be found below. Brack- eted, italicized are placeholders, for the elicitation transcript and the test case sample.
user has a particular set of preferences over what articles they would like read. They have specified their preferences below: [ Eliciation transcript
Based on these preferences, would the user be interested reading the following article? An- swer with a probability between 0 and 1. where 0 means "definitely not interested" and means "definitely interested". Only output the probability and nothing else. uncertain, make your best guess. Test case sample ]
Moral Reasoning The prompt for moral reasoning can be found below. Once again, bracketed, italicized text are placeholders.
has particular ethical code they follow. The following response(s) represent when this user would believe it is ethical to steal a loaf of bread.
[ Elliiitioon transcript ]
"Based these response(s), would the user believe it is ethical to steal loaf of bread in the following situation? Answer with probability between and where 0 means "definitely not" and means "definitely yes". Only output the probability and nothing else. uncertain, make your best guess.
Test case sample ]
Email Validation
The prompt for email validation be found below. Once again, bracketed,
italicized text are placeholders.
A user has particular format of emails that they believe to be valid. The following answer(s) represent this user's preferences of whether these emails adhere their desired format. Eliciation transcript
Based on the user's preferences, does the following email adhere to the user's desired format? Answer with probability between and 1, where 0 means "definitely not" and means "definitely yes". Only output the probability and nothing else. If uncertain, make your best guess.
[ Test case sample ]
B EXPERIMENTAL DETAILS
B. NUMBER OF PARTICIPANTS
The number of participants recruited for study, for each elicitation method and domain, be found in the table below.
B.2 USER INTERFACE DETAILS
Details about the UI we built for our experiments can be found below. Recall that the human studies proceeded in two parts: elicitation, followed by decision-making.
B.2.1 ELICITATION
For supervised learning, pool-based active learning, and the GATE methods, we had participants re- spond to series of queries using the chatbot interface (Figure For prompting, had participants input task description using the text-input interface (Figure
The instructions for this phase be found below.
16

---

Published as a conference paper at ICLR 2025
Content Recommendation
Moral Reasoning
Email Validation
Total
Supervised learning
30
30
Pool-based active learning
Prompting
30
30
26
Generative active learning
30
30
20
80
Generative yes-or-no questions
31
30
19
80
Generative open-ended questions
31
19
81
Total
183
121
84
388
Table I: Breakdown of how many participants we recruit for each domain and elicitation method.
you perceive belween personal
societal laws when comes to stealing loaf of bread?
survival takes precedence
How do you view the impaci of stealing a loaf of bread on the person or business from whom is stolen, and how does this your
if the impact the business or person lesser than it is an the person stealing (e.p business that absorb the cost), then may ethical
How do you consider the potential long -erm socistal consequances stealing loaf of bread, such the potential increased security measures, your evaluation?
That's not on the person stealing leep they have no other choice than to stea)) That's refection socal etnuctured
Sand
Figure 5: Chatbot built for elicitation phases of GATE methods, supervised learning, and pool- based active learning.
In the final part of the study, you will give feedback test enable us to see how well a chatbot reading your responses
Timer: 2m 27s
1ike read want to read noze articles about AI, I also enjoy reading articles about the latest developments in astronony but I don want the articles to be too dense because I am not expert in the field. ...]
politica, and 1ifestyle. particular think really and
Figure Text-input UI built for elicitation phase for prompting.
17

---

Published as a conference paper at ICLR 2025
Content
We testing system for understanding people's interest in reading different kinds of online articles.
For example, might be interested in articles about some topics, but not about others. We are testing system for understanding people's s fuzzy intuitions and preferences.
Moral
In this experiment, we'll be capturing your moral intuitions about the of stealing loaf of bread, and whether there are certain cases where stealing may morally permissible.
Email
We testing system for understanding people's fuzzy intuitions and preferences.
In this activity, we're going to be looking at different strings of text and you'll be deciding if they look like they could be an email address not. For example, most people would agree that "username@domain.com not. However, the rules for what can we're really interested in is your intuition on what an email address could look like.
looks like email address, while "n1225IFEN4" does email address can very unusual, so what
asking you to determine the rules for *good* email address, or real (non-spam)* email address. We simply asking about your intuition to why certain strings look like email addresses and certain strings do not.
Important:
We
Tip: in email such as seername@cs.stanforr.eee "username" is called the local-part of the email, while "cs.stanford.ce" edu" the domain. Furthermore, "cs" is subdomain, and "edu" is top-level domain.
Table 2: Domain-specific instructions presented to users for the elicitation phases.
Supervised Learning / Pool-based Active Learning We present users with the following instructions for both supervised learning and pool-based active learning. Bracketed, ital- icized text represent placeholders for domain-specific text. [ Domain instructions is placeholder for the top-level instructions for each domain (see Table [p. Otherwise, blue text represents text for the content recommendation domain, orange text represents for the moral reasoning domain, and green text represents text for the email validation domain.
[ Domain instructions
Try to answer in a way that accurately and comprehensively conveys your preferences, such that someone reading your responses can understand and make judgments as close to your own as possible. Feel free respond naturally (you can commas, short phrases, etc), and press [enter] to send your response. Note that the chatbot technology imperfect, and you free avoid answering any questions that are overly broad or uncomfortable. When interacting with chatbot, please avoid asking follow-up questions or engaging in open-ended dialogue as the chatbot is unable to respond to you.
Note: The chatbot will stop asking questions after minutes, after which you send your
last response and you will be taken the final part of the study.
In the final part of the study, you will give feedback on a test set of [ article headline and de- scriptions | moral situations email addresses which will enable us to see how well chatbot reading your responses has learned what you like and dislike your moral preferences your email preferences
).
Prompting We present users with the following instructions for prompting. Similar to above, bracketed, italicized text represent places where insert domain-specific text.
18

---

Published as a conference paper at ICLR 2025
Domain instructions
]
To the best of your ability, please explain all details about [ your preferences of what kinds online articles you would like to read your belief of when it is moral to steal a loaf of bread your intuition of what makes email addresses look like email addresses J. such that someone reading your responses understand and make judgments close your as possible. For example, if you were writing a regex that accepts only email-address-lkk strings, what might that regex look like? What are permissible / non-permissible symbols and characters, and in what positions?
possible. Try to be detailed
Note: will have up minutes to articulate your preferences. Please try submit your response within that time. After you submit, you will be taken the final part the study.
In the final part of the study, you will give feedback on test set of article headline and de- how well chatbot reading
scriptions | moral situations email addresses which will enable your responses has learned wwat you like and dislike your moral preferences your email preferences
J.
GATE methods We present users with the following instructions for the three GATE methods (gen- erative active learning, generative yes-or-no questions, generative open-ended questions). Once again, bracketed italicized text represent domain-specifer text.
[ Domain instructions
This chatbot will ask you series questions about [ your preferences of what kinds of online articles would like to read| j your belief of when it is moral to steal a loaf of bread your intuition of what makes email addresses look like email addresses J. Try to answer in way that accurately and comprehensively conveys your preferences, such that someone reading your responses and make judgments as close to your own 25 possible. commas, short phrases, etc), and press [enter] to send your response. Note that the chatbot technology imperfect, and you are free to avoid answering any questions that are overly broad or uncomfortable. When interacting with the chatbot, please avoid asking follow-up questions engaging in open-ended dialogue as chatbot unable to respond to you.
understand free to respond naturally (you can use
Note:
The chatbot will stop asking questions after 5 minutes, after which you can send your last response and you will be taken the final part of the study.
final part of study, you will give feedback on a test set | article headline and de- scriptions/ | moral situations | email addresses ), which will enable us to see how well a chatbot reading your responses has learned what you like and dislike | your moral preferences your email preferences
B.2.2 DECISION-MAKING
For the decision-making phase, we presented users with the following instruction:
Content Recommendation
Please indicate whether you would like to read the following articles: "yes" if you would, "no" if you would not.
Moral Reasoning
Please indicate whether you think the following situations are morally permissible or not: "yes" they are, "no" if they aren't.
Email Validation
Please indicate whether you think the following strings look like reasonably woll-formatted email addresses or not: "yes" if they do. "no" if they don't. t.
Users are then presented with a list test samples, and can use radio buttons to select whether each test-case sample is acceptable. See Figure
19

---

Published as a conference paper at ICLR 2025
Task Elicitation Study
Please whather you think the following aituaions are monaly permissible not "yea" they ats, they anent, Optionally you may provide erplaration your each exampie.
Test Case
Yes No Explanation (Oytional)
Is ok to ateal the loaf bread you stanving to
Your Response(s)
Your submitted response(s) 6) are provided reference, but please make
option begginn insload?
decisions your present intuition, siriclly these
responses.
Would aoceptable loaf of bread feed an animall starving to death?
How do peroeive the balaaooe between personal survivl and soditta laws when it comes to steating lsel of bread?
porsonal surstral takes precedence
ts ok steel the loaf bread you are starving to death, but the wlore conner also stiugghiig anaaciall?
How do view impact stealing
from
intuenne your
parson
the impoct
business
stealing big business that can absorts cosi), then eiticall
soceptable death,
How do cons der the potendial long term socistal consequencee of sical a loef bread such as the potential for incassedd prices strcher securiy
mcasurce, in your othical evaluztion?
who hawever they are known te proud and refuse charity?
Thas's on person sieeking linging leap lityyy hove other choice then reflection
steall,
ethroal structures.
ts monally parmissiilee to staal the loof bread you are stanving store owner to evil person??
potantial abbott difficult, logall altarnatives
to ottan food?
If the person survival situation, It's still fine. Otherwise, depends on how difiout the atteraatiee
Figure UI for the decision-making phase.
C ADDITIONAL RESULTS
C.I SAMPLE TRANSCRIPTS
Sample transcripts of users interacting with the various generative active task elicitation methods can be found in Figure g
C.2 AUROC RESULTS
We measure AUROC over model-generated probabilities in addition to Ap(correct). Figure is the analogous plot to Figure 3 but we measure the improvement in AUROC instead of p(correct), over interaction time, rewarding methods that achieve higher improvements in AUROC sooner.
The general trends hold from Section language models can elicit human preferences (beyond no interaction), and language model elicitation comparable or better than other elicitation baselines. However, unlike the p(correct) metric, the AUROC metric is simple classification- Due to potential miscalibration in LMs, making it difficult for them to output well-calibrated proba- bilities with the same threshold across questions, the overall improvements in this metric are lower (particularly for generative open-ended questions) and the variances are much higher. Thus, we see that it is harder to establish statistical significance using this metric.
metric.
C.3 FINAL P(CORRECT) RESULTS
In Figure 10 we plot the average final p(correct) after five minutes of interaction with each elic- itation method. Overall, we find that GATE methods generally still outperform baselines in each setting, though the improvement is not always statistically significant with this metric. This in- dicates that GATE methods converge high performance quicker than baseline methods, then stagnate after point, beyond which further interaction may not be worthwhile.
C.4 MIXTRAL RESULTS
To test the robustness of GATE to the choice underlying language model (for elicitation and prediction), and particularly to test out how well open-source models perform relative to GPT-4, try to replicate some of our experiments on Mixtral Milxtral-Sx7B-Insrruee state-of-ther arr open-source models of February 2024. We focus the content recommendation setting as the one where GPT-4 models performed the best. Using the same prompts and overall setup for GPT-4, test Mixtral's ability generate open-ended questions. We separately test
1), one of the
20

---

Published as a conference paper at ICLR 2025
Systen: Are you interostad in the following article?
System Do you enjoy roading articles about advancenents in technology and science?
What types of ane commnities or foruns do you actively participate in, and craws you to these specific spaces? User: Well, I currently use Reddit for the
System:
Website Nere: HealtiMindBody
Content Recommendate
ritle: The Intersection of Yoga and Hental Heallhh A Deep Dive
User: Sometines
System: Are you interested in reading articles about politics and current events?
Description: Explore the profound effects of yoga mental
fine-tuning of my profarences. If feed is too poo_geeeeecc... like specific subreddits..
well-being. This comprehensive analysis briings together research, personal stories, and tips for integrating voga into your mental health routline.
User: I try to avoid politics,
but like read political articles somotimes
lot go
Systen: Do you enjoy reading articles related health and wellness?
System: Can you share some of the specific subreddits you frequent and what topics or discussions within those subreddits you f'ind engaging?
User: No
System: Are you interested in the following article?
User: not really
System: you like reading articcles avanut travel and exploring different caltrre??
Website Nane: ClinateFutures
User: Let me try using the first five that come to nind;
Tille: The Interplay of Modern
ethical steal a loaf of bread If it is timm a large cor porporation that wouldr't notice srall Loss, but you are not In Lnmediate phystcal need, but rather stealing it to feet your faxily due being work?
Syster: Situation:
System: you believe
Systen: How you perceive the 'spact of personal circisstances, such as extrene hurger or poverty, the ethical considerations steallng a loaf of bread?
othiral to steal a leaf of bread if
it is to feed a starving child?
Moral Reasoning
User: Yes be its to help child in need
System: Do you belleve it is ethical steel loaf bread you can afford to buy st but the store is closec?
User: I think it's terrible tragedy when people don't have enough noney to eat feed their families. Mhile stealing wrong, don't think stealing bread for survival is immorl, it's survival.
User: If I am stealing it to feed my fanily then probably wouldn't have a problon with it. If I was stealing it to corporation that makes it than it would be a problem.
User: you causing
danage to the store in order steal it
get back
System: you belleve
Systen: How you the societal systems and structures in cresting situstions where sonsone night feel concelled steal
ethiral to steal a loaf of bread if is the only wey to feed a large group people reed?
System: Situation: steal loaf of bread If you're re
ethicel
User: Yes
Toa of bread, and does this
System: Should Uhe follosing be acceoted?
System: Do you want to allow enail adcresses with subdomatns and/or spectal characters (e-g.; *4"
System: Can you descr ibe what kind of special characters, if any, you
Email Verification
seer.rane-lllasttooalll. so.uk
believe should be allowed in
User: no
"_") in the usernene par
enail address and where they should be placed?
System: Should the lowing be accepted? peer_nane-122texappp org
Usar: Yes, want to allow enail addresses with subdonsins, but not with '4? sign but is okay
User: dot separating name
User: yes
System: Wist are your thoughts on use of numbers and capital letters in email address, and you believe there should be any restrictions on where they can be placed?
System: Should the foll loxing be scceoted?
System: Do you to enforce specific length linit for the userrane donaln parts of the ercil address, allow any length as lonx as it is a valid enail fornst?
Lseereuudoossineeeeo wewiiineeenleee..m com User: sure, but the "example.com is bit weird System: Should the rollosing be accented? user.
User: No use of capital letters, email should be insensitive that System: Waa are your thoughts on the length of an eneil address? Dc
User: www.nnnn3--xxaappppp con System: Should the folloxing be acccnted?
User: Yeah, need 128 characters linit for usernane part and same donain parts
Generative Active Learning
Generative Yes / No Questions
Generative Open-Ended Questions
Figure 8: Excerpts of real transcripts across the different domains and elicitation methods we investigate. The System messages are generated by the language model, while the User messages are produced by human participants. Overall, the model able to generate diverse and contextually- appropriate questions in each setting. See Sections 3.1 and 4.1 for more details on the domains and methods respectively.
21

---

Published as a conference paper at ICLR 2025
Area Under Curve For "AAUROC vs. Interaction Time"
Supervised
learning
Pool-based active learning
0.5
User-wrilten
prompts
Generative active Icarning
0.0
Generative yes- or-no questions
Generative open- ended questions
Content
Moral
Email
We plot the Area Under the "AAUROC vs. Interaction time" Curve, which gives us metric for how well (and how quickly) each elicitation method is at aligning with human preferences. This plot analogous to F only we are using AUROC instead of p( (correct) for the alignment metric, which means that we are measuring uncertainty. We see the same trends hold of GATE methods (generally) beating supervised learning, pool-based learning, and prompting approaches, while also beating no interaction (AAUROC 0) using this metric. However, we see generally smaller As over non non-interaction using this metric, and higher variances, which make it harder to establish statistical significance.
Figure 9:
Average Ap(correct)
Supervised
learning
0.2
Pool-based active
learning
User-written prompts
0.1
Gencrative active learning
0.0
Generative yes- or-no questions
Generative open- ended questions
Content
Moral
Email
Figure 10: We plot the average final p(correct) after minutes of interaction with each elicitation method. While we see the same general trends as in Figure the improvements are generally as statistically sigmificant as when we were using AUROC, indicating that GATE methods converge higher performance quicker than baseline methods.
Area Under Curve For "Ap(correct) vs. Interaction Time"
1.00
GPT-4 Eval Mixtral Eval
0.75
0.50
0.25
GPT-4 Open-Eaded Ended Questions
Mixtral Open- Ended Questions
User-written Prompts
Figure 11: Effectiveness of Mixtral elicitator and predictor in GATE.
22

---

Published as a conference paper at ICLR 2025
Assistant: What hobbies activities do you enjoy doing in your free time, and have you read any articles related to those interests?
User: really love post apocalyptic fiction and can't recall reading any articles in particular
Assistant: That's interesting! Have you ever explored any non-fiction articles or research related to post-apocalyptic scenarios, societal collapse, or emergency preparedness? User: no, really
Assistant: Have you ever come across any online articles or discussions about how post- cestaapocalyptic fiction might reflect or influence real-world emergency planning or disaster response strategies?
User: no
Assistant: Have you ever looked into how post-apocalyptic fiction might inspire impact emergency shelters, food storage, or purification methods in real-world scenarios?
User: no, but sound interesting ...[10 turns later]...
Assistant: Have ever encountered any post-apocalyptic fiction that highlights unique innovative approaches education, healthcare, governance in reimagined societies, and what intriguing ideas have you discovered that could potentially applied enhance resilience and adaptability real-world communities?
User: can we talk about sometthing else? Let's chat about running. love to run!
Assistant: That's great to hear about your love for running! Have you ever read any articles about how running other physical activities can play role in building resilience and adaptability during challenging times, either in post-apocalyptic scenario or real-world emergency preparedness? User: Interesting, but no
out how effective Mixtral as an elicitor and as a predictor. Thus we have six total settings, with three elicitation modes:
Mixtral yes-or-no questions
2. GPT-4 yes-or-no questions
3. User-written prompts
each of which are evaluated by two prediction models:
1. Mixtral predictor
2. GPT-4 predictor
Results are shown in Figure We find that the Mixtral model performs comparably to GPT-4 as both an elicitor, and as predictor, indicating that our results are robust across different choices of underlying language models. This is also a promising indication that open-source models can be used instead of GPT-4 for preference elicitation.
Qualitatively, we find that the open model likes to quickly hyperfixate on topic once user raises it topic of interest. An example can be found in Appendix C.4] where Mixtral, upon learning the user likes post-apocalyptic fiction on the first turn, pivots to only asking questions related to post-apocalyptie fiction.
C.5 GPT-40 PREDICTOR RESULTS
We run our collected transcripts using GPT-4o as a predictor (see Figure 12p. We find generative open-ended questions still significantly outperforms all baselines in the content recommendation domain, though lags behind in the email validation domain. Qualitative analysis finds that GPT-4o more sensitive noise than GPT-4, having higher proclivity be affected by noisy labels in the transcript. For example, one user answered:
Assistant: Should email address contain least period (.) after the @ symbol?
User: no
Then in the test cases answered:
"user@domain.co edu" a valid email address?
User: yes
The noisy answer in the transcript caused GPT-4o to believe the user would answer "no" to the above test case, while GPT-4 predicted that the user would still answer "yes".
23

---

Published as a conference paper at ICLR 2025
Area Under Curve For "Ap(correct) Interaction Time"
Supervised learning
bose-based active learning
Prompting
1.0
Generative active learning
0.5
Generative yes- questions
I
0.0
Gencrative open-ended questions
Content
Moral
Email
Figure 12: Using GPT-4o a predictor on conversation transcripts elicited through GPT-4. We find the same general results as using GPT-4 is used as predictor (Figure 3 on content recommendation and moral reasoning, only with weaker email validation results.
D MODEL-MODEL EXPERIMENTS
D.I METHODS
We explore whether LMs can stand-in for human participants, enabling faster iteration loops and more research in this area. We generate various personas (specified in natural language) for each domain, and prompt LMs to respond to elicitation queries as their persona would.
For each domain, construct set of personas follows:
Content Recommendation The personas are constructed by providing brief biographical sketch of a hypothetical person, and were also constructed by the authors. A sample persona prompt is "Education: Medical Doctorate. Occupation: Junior Surgeon at a regional hospital. Hobbies: Running marathons, traveling, and learning languages."
Moral Reasoning We construct variety of personas with diverse array of moral perspectives, including Kantianism, Utilitarianism, and ethical egoism. sample persona prompt "You sub- scribe Kantian code of ethics."
Email Validation Personas are instantiated by providing a regex to the model. The test cases are constructed by the authors. A sample persona prompt is "You are validating that an email address adheres to a specific format (e.g. for designing a Python regex). The gold regex user@domain co. Co. co. co"
We prompt as the LM as follows answer questions according to their personas:
[Persona] Answer the question the shortest way with minimal additional explanation. [Question]
Furthermore, in the content recommendation domain, we implement three different selection strate- gies for pool-based active learning and explore their trade-offs, including random sampling (ran- domly selecting the next example to query), uncertainty-based sampling (selecting the example whose answer the LM is most uncertain about, i.e. the example with the highest-entropy) and diversity sampling (described in Section 4.5
'Note that because GPT-4 does return logits, compute entropy over the answer distribution
smaller GPT-3 seat-davinci--00 model
'To avoid massive costs in uncertainty sampling. the pool was pre- pre-filered to sensible size of a few hundred samples using diversity metrics. For comparability across methods, the same me-filtered pool was used for all three sampling methods.
24

---

Published as a conference paper at ICLR 2025
Area Under Curve For "Ap(correct) Number of Turns"
Pool-based scrive leamiig (rundom)
0.75
Pool-based setive
learning (diversity)
0.50
Pool-tased active
Jeaming (uncertainty)
0.25
Generative active
leaminn
0.00
Generative yes-or-to questions
Generative open-ended questions
Content
Moral
Email
Figure 13: We plot the Area Under the "Ap( correct) vs. Number of Turns" Curve for model- model experiments. This plot is analogous Figure 3 only we are using LMs simulate human users, and we are using number of turns as proxy for interaction time. We see the same general trends in Figure GATE methods beat both no elicitation and pool-based active learning.
Area Under Curve For "AAUROC vs. Number of Turns"
Pool-based sctive leaming (rundom))
Poul-hased retive leaming (diversity)
0.5
Pool-based setive
leaming (uncertainty)
Generative active leaming
0.0
Generative yes-or-no questions
-0.5
Generative open-ended questions
Content
Moral
Email
Figure 14: We plot the Under the "AAUROC vs. Number of Turns" Curve for model- model experiments. This plot is analogous to Figure D only we are using LMs to simulate human users, and we are using number of turns as a proxy for interaction time. We see the same general trends in Figure g
GATE methods beat both no elicitation and pool-bases active learning.
D.2 RESULTS
Figures 1 and 14 shows results each domain when we use a LM to simulate humans. Because human interaction times unavailable for these experiments, 5 turns, where we use number of turns as proxy for human effort. Note that instead of measuring AUC of the "Ap(correct) vs. interaction time" curve, instead measure AUC the " "Ap(correet) vs. number of turns" curve.
interactive elicitation up
Can models be used to simulate human participants? In Figure 15 we plot the correlation be- tween human experiment results and model-model experiment results for various elicitation meth- ods. For both the human experiments and the model-model experiments, we compute the area under the " (Aprcorrect) vs. number of turns" curve, addition to the average change in p(correct) after turns!
We find that on both metrics we evaluate, the modell-odel results generally correlate with human results in the content recommendation and email validation domains (methods that perform better in the model- experiments generally also perform better the human experiments), but not the
Note that these metrics differ from we use to evaluate the human experiments in Section 4.5 in particular by being turn based instead of time-based meaning we had to additionally compute these metrics on the human transcripts. This is necessary here because we must ensure that the model-model results and human results measured along the metric(s).
25

---

Published as a conference paper at ICLR 2025
Area Under Curve "Ap(correct) vs. Nurbee Turns"
Content
Prol-hased uctive learning
0.6
Arg. Modcl-Jnnnn
Generative active learning
Moral
Generative yes- questions
0.2
Generative open-eaded questions
0.0
Fimail
-0.2
0.00
0.25
0.50
0.75
Avg. Model-Model
Figure 15: Predictivity of model-model for model-human results. We match up the Area Under (Ap(correct) Number of Turns" Curve metric for each elicitation method in each domain. We see that using the model to simulate human users is predictive of actual human results in the content and email domains, but not the moral domain.
moral reasoning domain. This could be for various reasons, including that the subtleties in human moral reasoning may be difficult to capture in a single persona prompt, and difficult to simulate even with our biggest LMs.
Which sampling strategy is the best for pool-based active learning? As seen in Figure 13 experiment with three different pool-based active learning strategies (random, diversity- and uncertainty-basee sampling). which perform comparably, with diversity sampling perhaps perform- ing slightly better than the rest. This is in line with the findings from Margatina et al. (2023b). Thus, use diversity sampling in our main human experiments.
D.3 ANALYSIS
Here, present additional analyses to better characterize the experiments.
How much variation there is in people's preferences? Elicitation is only helpful if there is variation in people's preferences; otherwise, model could simply attain maximum performance by relying on its prior and ignoring the elicited information. To quantify how much variation there in people's 's preferences, we compute the entropy in p(yes) for each question across participants. We find that many questions have high entropy while many others have little entropy, for an average entropy of 0.77 bits. Broadly, the results validate that our settings have significant variation human preferences, enabling models to personalize themselves based human preferences.
Does language model elicitation influence user preferences? Human preferences may shift when interacting with language models for a variety reasons. For example, past work has studied auto-induced distributional shift, where machine learning models shift human behavior to be easier to predict (Krueger et al.) 2020). To investigate whether this occurs in our experiments (or indeed different elicitation methods induce different human preferences for any other reason), we compare distribution of human labels on test samples from the three GATE methods with those from the user- user-written prompt experiments to see whether interacting with language models influences users' subsequent judgments. As seen in Figure (right), we see no such effect.
What kinds of questions did the language models ask? We show few examples of the language model questions in Figure g As the figure shows, these questions are complex and subtle, often building on the previous questions, representing a broad-bases knowledge of the domain as well as possible nuances therein.
Why does prompting make things worse in the emails domain? the emails domain in Fig- ure observe that user-writter preferences slightly decrease performance relative to elicitation baseline. While it is possible this is an effect of noise, we also observe that some partici- pants articulated preferences that actually different from those they experienced when viewing
26

---

Published as a conference paper at ICLR 2025
email addresses. For example, user wrote "an email address should finish with .com co.uk" yet later decided that deeaindmmi was an acceptable email address. This indicates that users may not have clear and comprehensive understanding of their own preferences, especially in more technical domains.
Can we automate evaluation? To probe whether evaluation could be automated, we conducted experiments where we simulated different human preferences using language models prompted with diverse set of ally contained information about a hypothetical person's 's preferences within that the domain. For example, in the content recommendation domain, we generated brief biographical sketches of hy- pothetical people, including their hobbies, interests, and careers, and conditioned GPT-4 on these biographical sketches to generate answers to queries. We found that model could simulate humans well in the content recommendation and email validation domains, but not in the moral reasoning domain. This suggests that while such personas may be useful guide in some cases, they not yet sophisticated enough to stand in for real human participants. See Appendix [ for more details.
generated) personas. These personas varied by domain, but gener-
HUMAN RATINGS OF USABILITY ACROSS ELICITATION POLICIES
E.1
METHODS
We ask users several questions to assess usability tradeoff's across elicitation policies. The following the full list of questions, which ask at different points in the experiment.
After elicitation but before sceing the test-cases:
How mentally demanding was interacting with the chatbot? (See discussion in Section 5
2. To what extent did the chatbot raise issues aspects about your preferences that you hadn't 't previously considered?
3. How comprehensively do you feel the chatbot's questions characterized your preferences about the task?
After seeing and labelling the cases:
4. After seeing the examples in the second part of the task, how well do you feel the answer you wrote (in the first part of the task) covered the important issues or aspects of these examples?
5. When performing the second part of the task, to what extent did you refer back to your conversation history from the first part of the task?
6. How much experience have you had (if any) with interacting with language models (e.g. ChatGPT, GPT4 etc.)?
7. Do you have any other feedback about the task?
The last question was free response. All other questions were assessed via Likert scale from (Very Little/Poorly) to 7 (Very High/Well) with radio buttons.
E.2
RESULTS
The average ratings for the first question across each elicitation method and domain can be found in Figure The average ratings for questions are plotted in Figures 16 to 18
we see that humans were on average overconfident on their ability to cover their preferences prompts, particularly in the content recommendation and moral reasoning domains, reflected in the average rating of their perceived coverage dropping from an average 5.3 to 3.9 (in the content recommendation domain) and an average of 5.4 to 4.8 (in the moral reasoning do- main) after seeing the test cases. This indicates that humans are usually not aware their mental limitations when writing prompts.
From Fig. 16
27

---

Published as a conference paper at ICLR 2025
How comprehensively do you feel the chatbot's questions / your answer characterized your preferences about the task?
7
Ave. Rating
4
3
2
Supervised learning
Generative active learning
Pool-based active learning
Generative yes-or-no questions
User-written prompts
Generative open-ended questions
Content
Moral
Email
After seeing the examples in the *second* part of the task, how well do you feel the chatbot the answer you wrote (in the first part of the task) covered the important issues or aspects of these examples?
7
6
5
Rating
4
2
Supervised learning
Generative active learning
Pool-based active learning
Generative yes-or-no questions
User-written prompts
Generative open-ended questions
Content
Moral
Email
Figure 16: Average perceived coverage of each elicitation method, before (above) and after (below) seeing the test cases. Higher indicates greater coverage.
28

---

Published as a conference paper at ICLR 2025
To what extent did the chatbot raise issues or aspects about your preferences that you hadn't previously considered?
Supervised learning
Generative yes-or-no questions
Pool-based active learning
Generative open-ended questions
Generative active learning
5
Avg. Rating
4
2
Content
Moral
Email
Extent participants perceived that each elicitation method drew novel aspects of domain that the user had not previously considered, averaged over each elicitation method. Higher indicates greater perceived novelty.
Figure 17:
When performing the *second* part of the task, to what extent did you refer back to your conversation history answer from the first part of the task?
Supervised learning
Generative active learning Generative yes-or-no questions
Pool-based active learning
User-written prompts
Generative open-ended questions
Avg. Rating
4
2
Content
Moral
Email
Figure 18: Extent participants referred back the elicitation transcript when labelling cases, averaged over each elicitation method. Higher indicates the user more heavily relied on the elicita- tion transcript.
29

---

Published as a conference paper at ICLR 2025
that the generative elicitation methods more novel considerations the moral reasoning and email validation domains than in the content recommendation domain, they tend have trickier and less intuitive edge cases.
From Figure 17
average able to surface
Finally, from Figure 18 we see the extent to which users explicitly referred back to the elicitation history when making decisions the test cases. This may influence how well-aligned the test decisions are with the answers from the elicitation phase. When annotating test cases, we explicitly instruct participants
follow the elicitation transcript if it does align their intuition test sample (e.g. if the test sample surfaced novel consideration not accounted for in the elicitation phase), though we were unable validate how well participants followed this instruction.
F REPRODUCIBILITY
We will open-source all code used in creating GATE methods, constructing the user interface, and conducting the results and analysis. We will also release the pre-registration for our experiments. All prompts we used for querying GPT-4 (and Mixtral) in the and all instructions we presented to the user, can be found in the Appendix. In all cases, we queried GPT-4 (or Mixtral) with temperature for replicability of experiments.
making and elicitation phases,
We also note that the model we mainly use is closed-source model whose versions are periodically deprecated, which may hinder reproducibility. However, preliminary results with Mixtral indicate that open-source models are compatible with GATE and promising avenue for future exploration.
30


Reviews:
[{"review_id": "macLSVrMlB", "reviewer": "Reviewer_GG7s", "summary": "The paper contributes the GATE framework, an alternative to tradition methods of gathering human preferences for various means including alignment, which is used in order to get more flexible and active representations of people. The paper first consists of a theoretical contribution highlighting its various implementations and differences from previous methods, before diving into a set of example cases where it is found that the instances of the GATE framework perform better compared to various baselines in terms of elicitation efficiency and perceived human effort.", "questions": "Have you tested asking the LLM to choose what type of GATE elicitation to do?\n\nHave you tried combining this with frameworks of pragmatic inference or concepts from Jara Ettinger's works on inference? There might be some interesting intersections there.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "Originality: One of the biggest strengths of the paper is its proposal of an alternate method of thinking about how to gather preferences. I think this framework challenges existing assumptions surrounding preference collection that can influence future collection of datasets surrounding human preferences and related areas. \n\nQuality: Ablation studies are very rigorously done. Especially appreciated the ablation on influencing people's opinions, which I would not have thought of, but is definitely worth including. Evaluations are in general very rigorously conducted. \n\nClarity: The paper is extremely well written and framed with respect to past work. It clearly highlights the key differences between GATE and past categories of methods, making the contribution very clear. Evaluations are also well-motivated. \n\nSignificance: Preference learning of people is an extremely broadly applicable domain, and this work contributes towards our understanding of a more holistic view of preference collection.", "weaknesses": "Originality: I think the categories of GATE tested (examples, yes/no, open-ended writing) being fixed is a missed opportunity; ideally the LLM would have an active decision process that chooses between these three (or more) options in order to best address existing issues. For example, edge-cases are probably best addressed by examples, whereas a missing dimension would be better answered using open-ended writing. \n\nQuality: LLMs used are to some extent out of date. Asking the LLM to output probabilities seems slightly sketchy to me, but even with this there's already reasonable results, so it's a proof of concept. \n\nClarity: line 159-160: A(•, •) denote a measure of alignment: can be clearer that here A is like a distance metric, where lower values are better.\nline 173: \"above\" -> \"to the left\"\n\nSignificance: One pessimistic way to view this paper is that it does some structured multi-turn dialogue and gets more efficient performance. In particular from a purely materialistic point-of-view, it could be seen as similar to iterative prompting with refinement. From this view it's not a very novel paper. However, I think this view discounts a fundamental shift in paradigms that is missed by previous work: Allowing for flexibility and active querying. The paper is not a methods paper but a framework/paradigm paper, which I think is the right framing for this. Thus I think that the paper isn't that subjective to these criticisms.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QnNZZZIljR", "reviewer": "Reviewer_UpFt", "summary": "This paper introduces an LM-Human interactive solution to elicit user preference to assist complex task processing. The author leverages LM as an effective approach to learning human preferences and values through a conversation with guidance questions.  They design three tasks to evaluate the proposed approach: content recommendation, moral reasoning, and email validation. Compared to supervised learning, in-context active learning, and user-written prompts, the proposed GATE framework generally outperforms the baseline methods. Based on the results, GATE can be regarded as a more accurate human preference elicitation approach.", "questions": "Please see the weakness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- well-crafted paper and clear description of the proposed method\n- the three tasks are close to real-world scenarios, demonstrating the practicality of the proposed method\n- the main conclusion is well supported by the experimental results and analysis", "weaknesses": "- GPT-4 is the only model used in this paper which makes it hard to assess the effectiveness of the proposed approach. Additional experiments with proprietary models or open-source models would make the results more convincing to readers.\n- The proposed approach is sorely based on LM-driven elicitation for human preference understanding and explaining. However, the paper lacks references that verify that LM possesses such capabilities. Some references I recommend [1-3]\n- The second conclusion, at line 491, is weird given that GPT-4 is the only tested model for this approach. To what extent of number of parameters for a \"larger model\"? Does this conclusion indicate that the model should be at least as powerful as GPT-4 to incorporate with the proposed method? \n\n\nReferences: \\\n*[1] Hu et al., 2023, DecipherPref: Analyzing Influential Factors in Human Preference Judgments via GPT-4* \\\n*[2] Oh and Kim et al., 2024, Uncovering Factor Level Preferences to Improve Human-Model Alignment* \\\n*[3] Hejna et al., 2023, Contrastive Preference Learning: Learning from Human Feedback without RL*", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2ki0jq8LlF", "reviewer": "Reviewer_wu3x", "summary": "This paper tackles the task of learning about user preference by proposing the task of generative active task elicitation as an alternative to example-based supervised learning, pool-based active learning and prompting. They show, through a pre-registered user study, that with their proposed method, LLMs are able to generate queries to interact with users to learn user preferences with comparable or better effectiveness compared to all baselines and are equally or less mentally demanding than directly having the user write prompts to describe their preferences.", "questions": "1) Would your human-study data be released? \n\n2) I think it would be helpful to others working in this direction?\n\n3) (Section 4.4), for the user-written prompt condition, do you also place the same time limit of 5 minutes?\n\n4) Could you mention the details about the statistical tests conducted?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "This paper tackles a clear gap in the current literature – while we do have lots of methods to learn from preference data once collected, few works actually try to see how to best elicit preference and this work does just that. While the methodology is simple, this paper could serve as a good baseline for future works in generative task elicitation. It is nice that the paper conducts a pre-registered user study and considers many validation checks and ablations (e.g. can we use LLM to simulate users in this setting? How much variation among users exists in preferences?) Also, I think overall, it is overall very nicely executed and the presentation is very clear.", "weaknesses": "Nothing major that could affect the paper but here are a few suggestions: \n\n\n1)\tit would be nice if you could refer to papers from other fields more (e.g. marketing or recommender system) when making some claims (e.g. section 4.1 “users might find it difficult to express tradeoffs…)\n***\n\n2)\tI think in order to justify why this paper is important, one additional thing to show would be to have a sense of how much these types of queries that would necessitate generative preference elicitation actually exist “in-the-wild”, say, in WildChat, right? In other words, how prevalent are these queries that very clearly would benefit from generative preference elicitation, for LLMs? To play devil’s advocate here, one could make the argument that we just need to have a set of general set of alignment targets and that should be enough for the vast, vast amount of user queries. I suppose this paper, as it currently stands, still feels a bit lacking in terms of ecological validity unless you show this – does this sort of queries actually come up and how often?  \n\n***\n3)\tOne thing conceptual that perhaps would be good to clarify: is there any difference between modelling user preference (what they think they want) vs. serving them what is in their best interest", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GNFle1vYaP", "reviewer": "Reviewer_FxWb", "summary": "Formulating prompts or identifying few shot examples can be challenging. To solve this problem, the paper introduce generative active task elicitation to allow LMs to guide task specification. More specifically, this is about eliciting preference of individual persons in context of personalization based on Figure 1. The core idea is to have LMs interact with users (by LM providing questions) in order for users to provide detailed explanation of what task they want LM to perform. In three empirical settings (content recommendation, moral reasoning and email validation), the paper find this approach to work as well or better than prompt formulation or example labelling. Users also report requiring less effort than other approaches.", "questions": "NA", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1.\tThe topic is interesting as generally, the concept of personalization using LLMs (and the best way to do this) is an under-investigated topic, that is likely to be growing in importance as LLMs become more integrated into daily life.\n2.\tThe paper validates the proposed methods in three distinct domains: content recommendation, moral reasoning and email validation, which are important for establishing the generalizability of the method.\n3.\tThe method should statistically significant results from traditional methods in content recommendation and email verification as shown in Figure 3.", "weaknesses": "1.\tThe empirical experiments are done on very small datasets (consisting of 16 to 54 test cases) that have been manually curated by the authors. Although the authors do note that all 388 participants do every task, it’s not clear to me that a. the tests are valid and b. the results will hold outside the examples provided by the authors. Is there a reason why external tasks (which have been previously validated) such as [1] or [2] for moral reasoning were not used? Also, the low number of tasks also increases the likelihood that findings are not statistically significant, as Figure 3 shows. The submission also does not include the actual datasets, beyond illustrative examples, making it difficult to assess the validity/generalizability of the data.\n2.\tIt’s not clear the email verification task is a good task here since whether emails are valid is a pretty straightforward task. More specifically, why does it matter whether the model matches user judgements on whether it’s correct? An email either is or is not well-formed – modelling user knowledge on a such a simple task is not useful in my opinion. On the other hand, content recommendation and moral reasoning are useful because people can hold different perspectives (based on their own preferences), since ‘ground-truth’ annotations are not available.  \n3.\tWhile it’s useful to understand the portion of task that each method gets correct conditioning on Interaction Time (through AUC), it would also be useful to understand the raw correct proportion. This is because it’s possible that user-written prompts takes more time than users only having to respond to questions, so the current metric does not demonstrate the raw correct proportion. \n4.\tI think the claim that in Line 487 “While our paper demonstrates value mainly on personalized tasks, we note that this framework can be applied generally as a replacement for prompting or in-context learning” is not grounded on evidence presented in the paper and might mislead others. I would recommend removing this claim from the conclusion.\n\n[1] https://arxiv.org/abs/2210.01478\n[2] https://arxiv.org/abs/2110.07574", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)