ADAPT: Actively Discovering and Adapting to Preferences for any Task
Authors: Maithili Patel
Xavier Puig
Ruta Desai
Roozbeh Mottaghi
Sonia Chernova
Joanne Truong
Akshara Rai
Venue: COLM 2025 (2025)
Keywords: Task Oriented Agents
Interactive Learning
Active Dialog
Personalization
Task Planning
TL;DR: A benchmark, ADAPT, and a training mechanism, ReflectionDPO, to create and evaluate a grounded task planner that can actively elicit user preferences by asking questions, and adapt execution accordingly.

Abstract: Assistive agents should be able to perform under-specified long-horizon tasks while respecting user preferences. We introduce Actively Discovering and Adapting to Preferences for any Task (ADAPT) – a benchmark designed to evaluate agents’ ability to adhere to user preferences across various household tasks through active questioning. Next, we propose Reflection-DPO, a novel training approach for adapting large language models (LLMs) to the task of active questioning. Reflection-DPO finetunes a ‘student’ LLM to follow the actions of a privileged ‘teacher’ LLM, and optionally ask a question to gather necessary information to better predict the teacher action. We find that prior approaches that use state-of-the-art LLMs fail to sufficiently follow user preferences in ADAPT due to insufficient questioning and poor adherence to elicited preferences. In contrast, Reflection-DPO achieves a higher rate of satisfying user preferences, outperforming a zero-shot chain-of-thought baseline by 6.1% on unseen users.

Published as a conference paper at COLM 2025
ADAPT: Actively Discovering and Adapting to Preferences for any Task
Maithili Patel', Xavier Puig?, Ruta Desai², Roozbeh Mottaghi", Sonia Chernova', Joanne Truong"; Akshara Rai2*
Georrgia Institute of Technology
?FAIR Labs, Meta maithil@@atechh.em
Abstract
Assistive agents should be able to perform under-specified long-horizon tasks while respecting user preferences. We introduce Actively Discover- ing and Adapting to Preferences for any Task (ADAPT) benchmark designed to evaluate agents' ability to adhere to user preferences across various household tasks through active questioning. Next, we propose Reflection-DPO, a novel training approach for adapting large language models (LLMs) to the task of active questioning. Reflection-DPO finetunes a 'student' LLM to follow the actions of a privileged 'teacher' LLM, and optionally ask question to gather necessary information to better predict the teacher action. We find that prior approaches that use state-of-the-an LLMs fail to sufficiently follow user preferences in ADAPT due to insuffi- cient questioning and poor adherence to elicited preferences. In contrast, Reflection-DPO achieves higher rate of satisfying user preferences, out- performing zero-shot shain-of-thougee baseline by 6.1 on unseen users.
1 Introduction
Consider an embodied agent that assists users at tasks, such as "organize incoming stock" in warehouse, or "prepare an omelet" at home. These tasks are high-level, under-specified and lack information about user preferences. For example, a warehouse user might prefer to store larger boxes towards the back or to keep frequently used items at the front for easy ac- cess, and the user in a home setting might want an omelet made in olive oil instead of butter. In contrast, grounded planning works typically assume detailed instructions (Yenamandra et al., 2023; Liang et al., 2023; Huang et al., 2023a), such as "move the small box to the first shelf". tedious for a user to break down high-level tasks into unambiguous step-by-step instructions, especially for multi-step, long-horizon tasks. Instead, we argue that the agent should be able to autonomously plan towards the high-level task, and actively ask questions to uncover user's 's preferences that are not explicitly stated in the goal command.
Actively asking questions towards an open-world long-horizon task challenging, because it requires creating a plan in a large search space, understanding possible variations of that plan, and determining when and what to ask the user to execute their preferred variation. Recent works have showcased the abilities of large language models (LLMs) in open world planning (Chang et al., 2025; Rana et al., 2023; Huang et al., 2023b; Liu et al., 2023; Brohan et al., 2023). However, LLMs have been shown to assume, rather than ask for user preferences (Shaikh et al., 2023). Works that learn user preferences for embodied tasks (Wang al., 2024; Wu al., 2023; Kapelyukh & Johns, 2022; Jain et al., 2023) typically rely on task demonstrations. However, providing multiple demonstrations for all everyday tasks, such as preparing breakfast, is difficult and not scalable. Ideally, an assistive agent should be able to execute tasks autonomously, and ask questions when needed.
Indicates equal contribution

---

Published as a conference paper at COLM 2025
ADAPT
Reflection-DPO
Con you make me ommlet for brecklest?
Suro thing!
Hove eBg 0 free eax. car boxl, Fillings
counter
Teacher Planner
Student Planner
you any Iilingo
Loox creese
preferencee
found cheese_ (a bleck mezzarelia cheese),
Yes. would like cheese my smalee.
Reflection-DPO
DPO training
cheese), and theese_ (oox of rets theese crumbles)
Data Generation
Which mozzerdio, steds and fato cheecse would ths?
cheese
Nove suiss, cheese_sitc free cheese_ to
Swiss chesse, piecse.
Figure 1: ADAPT (left): A new benchmark that requires an agent to actively elicit user preferences through questions. Reflection-DPO (right): novel approach for training an LLM for active question- ing using a privileged teacher, and a reflection mechanism that introduces questions into the dataset.
We contribute Reflection-DPO (Figure 1, right), an approach for teaching LLMs to intelli- gently ask questions regarding user preferences, where needed. Our key insight is that LLMs are not good at actively asking questions to uncover hidden preferences, but given access to preferences, they can adapt task execution to align with them. Reflection-DPO uses a privileged LLM (teacher), primed with knowledge about preferences of a particular user, to train a student LLM to adhere to preferences without privileged knowledge. However, as we show in Section 6.3, directly trying to imitate the teacher actions isn't enough, because the student might lack key information needed predict the teacher's actions. To account for missing information, we introduce a reflection step that generates a candidate question to help the student predict the teacher's action. Ultimately, this teaches the student to adhere to preferences (as demonstrated by the teacher) by learning to acquire the necessary information through active questioning (by reflection), enabling it to fulfill ambiguous goals, while adhering to user preferences by proactively asking questions.
Our second contribution is a benchmark Actively Discovering and Adapting to Preferences for any Task (ADAPT)- designed to evaluate closed loop performance of assistive agents on long-horizon, ambiguous everyday tasks (Figure 1, left). ADAPT contains a set of high-level tasks and 16 personas. We instantiate a total of 128 persona-tasi combinations, each characterized by up to 14 preferences and grounded in scenes with on average 162 movable objects, in a text-based planning domain. We focus on common daily activities (e.g,, making eggs or cereal) and support text-based actions, which can be state-changing actions or open-ended questions, such 'Open cabinet' or 'How would you like your eggs cooked?'. An LLM is prompted to be a 'human' user that answers the agent's questions. This results in an open-ended communication benchmark with challenging, long-horizon, ambiguous tasks, useful for benchmarking adaptive assistive agents.
Our results show that SoTA LLMs such as Llama3.1 1oma3.117000 1-70,, when used control agents, do not adhere to preferences in ADAPT. Instead, training using Reflection-DPO enables an L.L.M to ask informative questions and adapt to preferences, improving upon an untrained L.M by achieving 8.6% higher rate of preference satisfaction for unseen users. On the same metric, Reflection-DPO outperforms a zero-shot tfain-of-thought LLM, our leading baseline, by 6.1%. Our key contributions are: (1) A novel approach, Reflection-DPO, for training LLMs to actively ask questions and adapt task execution to align with elicited preferences. (2) A benchmark, ADAPT, for evaluating assistive agents on active questions that operates within a grounded text planning domain, and includes large action space and user models capable of answering open-ended questions.
2 Related Work
Interactive learning for behavior adaptation has been widely studied. Interactive feedback includes action corrections (Li et al., 2025; Korkmaz & BIyIk, 2025; Pérez-Dattari et al., 2019; Liu et al., 2022), preference feedback (Yang et al., 2024; Sadigh et al., 2017; Christiano et al.,
Code for ADAPT and Reflection-DPOss openly available at https:// //gi thub. com/Mai thili/Adapt.
2

---

Published as a conference paper at COLM 2025
2017; Bıyak et al., 2022), and language-based corrections (Liang et al., 2024; Han et al., 2025; Sharma et al., 2022; Campos & Shem, 2022), using feedback on agent-generated trajectories to understand user preferences. Some methods personalize by predicting feedback (Liang et al., 2024), and co-embedding corrective language feedback with trajectory (Yang et al., 2024). These approaches focus on single actions and struggle with long-horizon tasks due to context-length limits and learning shared embedding for multi-step trajectories. Prior work Bärmann al. (2024) has used memory capture online feedback, but relies explicit preferences. LLM- LLM-Personalize (Han et al., 2025) iteratively improves at multi-step object rearrangement by learning from user ratings, but relies on demonstrations and dense feedback. Natural language has also been used to express preferences (Wu et al., 2023; Wang et al., 2024), but these works also rely on prior task demonstrations. Instead of demonstrations or dense user feedback, we propose adapting through active questioning during task execution to elicit preferences.
Task-oriented dialog is crucial for instruction, guidance, collaboration, and clarification. Prior works have explored dialog for collaborative tasks (Bara et al., 2021; Narayan-Cher et al., 2019), paired commander-followee agents (Padmakumar et al., 2022; Thomason et al., 2019; Banerjee et al., 2020), and agent asking for help (Singh et al., 2022; Zhang et al., 2023). When the task is not explicitly defined, some works optimize directly for user attention (Yoshino & Kawahara, 2015) or enable responses to their implicit requests (Tanaka et al., 2024). Our approach differs by focusing on exchanging information about hidden user preferences, not task instructions or execution help. Follow-up work (Gella et al., 2022) that analyzes a commander-follower setup (Padmakumare et al., 2022) shows that only 1.6% of the utterances asked for additional information, and 2.7% more provided information relevant to learning hidden preferences. Methods that seek to efficiently ask for help either separately train model through RL (Singh et al., 2022), directly eliciting an expert action rather than obtaining information, or use a measure of entropy lower-bound of the goal (Zhang et al., 2023), requiring an explicit estimate of goal uncertainty, which is difficult to obtain in open- set preferences as in our setup. Prior work (Hong et al., 2023) has utilized offline RL from experts, focusing on dialogue efficiency rather than user preferences. Instead, we gather information about hidden user preferences through dialogue.
asking
Information-seeking dialog has been employed to clarify user preferences (Wang et al., 2024), disambiguate instructions (Park et al., 2023), and improve predictions when uncer- tain (Ren et al., 2023). Prior work on active clarifications for personalization (Wang et al., 2024) rely on preference hypotheses generated from demonstrations. Research on asking for clarification between multiple probable LLM predictions (Ren et al., 2023) is not geared to long horizon tasks, and struggles to distinguish between action-related and preference- related ambiguities. In contrast, we use a privileged teacher to teach a student LLM how to follow preferences. Most related to our work is STaR-GATE (Andukuri et al., 2024), which enables a virtual agent to act based on hidden preferences. However, this agent can only take one action based on questions, whereas our tasks involves sequential decision making, Prior benchmarks that study preference elicitation through dialog operate on domains such online shopping assistance, evaluated on datasets of shopping prompts and target items (Andukuri et al., 2024; Piriyakulkij et al., 2023), game-like decision-making (Lin et al., 2024) or price negotiation (Verma et al., 2022), evaluated on human-human dialog datasets, and content recommendation, evaluated through user studies (Handa et al., 2024). However these domains are either purely conversational or oow-dimeniional game-like environments. They do not support complex tasks, such as those expected of assistive household agents, requiring grounded planning, diverse states and actions and long-term interactions.
3 Problem definition
We consider agent controlled by policy tasked with achieving goal g while adhering to a user's preferences I. At each step k, the agent takes an action 4t given the task goal g, and the history of agent interactions, including previous actions Agg 1, observations 0okk 1, and user responses Mo 1. The actions available to the agent include physical actions, questions for the user, and a termination action. Physical actions involve object manipulation
3

---

Published as a conference paper at COLM 2025
and exploration, and result in an updated state of the environment and an observation. User responses to questions provide feedback, and help uncover user preferences I.
Each episode results in trajectory (g, Ro:K, 0o.K, Hoox) of K steps. A reward r E {0,1} is assigned to the trajectory depending on how well the agent's actions satisfy the user preferences: 8, [, aq:K, 0o.K r. User preferences are task-related constraints, not captured in the goal 8, and a priori unknown to the agent, such as preference for dairy-free milk. The policy π aims to maximize the reward At each step π predicts the next action a = .ig, a...110k110000 10.-11 4o which is executed in the environment. If the policy generates a question, the agent receives a user response. A termination action ends the trajectory.
4 The ADAPT benchmark
We develop agents capable of adhering to user preferences for diverse, long-horizon house- hold tasks, such as "Preparing eggs for breakfast". We focus on meal preparation tasks, since they involve complex preferences beyond object locations. Setting up such tasks requires an environment with: 1) a grounded simulation supporting diverse actions, 2) large environmental diversity to accommodate realistic preferences, and 3) diverse user models that are characterized by preferences and can interact with the agent. We present benchmark ADAPT that exhibits these characteristics.
Grounded text-based simulation: To support complex tasks, such as "make an omelet for breakfast" we develop a grounded interactive text-based simulation environment. This simulator allows for an open set of state changes, enabling complex actions such as mixing eggs and chopped vegetables to get omelet batter. The environment represented using textual scene graph, similar to prior work (Puig et al., 2018). Each node in the graph corresponds to an entity, such as movable objects, furniture, or room and is characterized by: 1) unique identifier (e.g., milk 1), 2) freeform text description (e.g., carton of oat milk), 3) state description as a list of state tags and contents (e.g. open, contains oat milk), and 4) an optional type, designating whether the object is edible or if it is container which can hold food. Each interaction begins with the agent receiving an scene layout, consisting of the rooms and furniture, but no objects. The agent must take text-based actions, to discover the moveable objects and their locations. Each action results in an observation, provided to the agent as feedback. For exploration actions, the observation includes list of observed objects. For manipulation actions, the observation includes summary of changes made to the environment, the action was successful, else the reason for action failure. Failures can occur due to impossible actions such as heating something that is not on a cooking appliance, mixing non-food objects into a mixture etc.
Supporting open set of actions: Our simulator supports an open set of actions necessary for simulating a wide range of everyday tasks, alongside a base set of actions. For open-set actions, we create two templates: "<action> items in <container> to get <object>" and "<action> the <object> to get <object>. " These templates allow for actions like "mix items in bowl 0 to get omelet batter" or "crack the egg 0 to get cracked egg 0.". We rely from
on the agent to provide semantically meaningful open set actions (e.g." "cra"crrccceeeeeee egg.0) .), and do not explicitly check the validity of each action. Using LLMs as the policy to control agents ensures that open set actions are diverse yet semantically meaningful. Given an action, the simulator transforms the nodes involved in the action into the expected output node, as specified in the template. For example, the bowl.0 node now contains omelet batter. We also provide a base set of actions that enable state-changes, exploration and relocation, including open, close, turn on, turn off, move, move from, pour, search and look for. Successful execution of actions require certain pre-conditions actions require the referred entities to exist in the environment. move from and pour require the referred object being in the mentioned container. Certain freeform actions require the referred object to be in the required location as a pre-condition for successful execution. See re Appendix C.1 for a full list actions and their pre-conditions.
hold. All
Task, object and scene diversity: To support diverse user preferences, we offer wide vari- ety of objects, including several variations within categories, summing up to 232 movable objects, over 6 rooms and 37 static appliances and furniture. Variations within each category,
4

---

Published as a conference paper at COLM 2025
Task Command & Instructions
Student Model
Teacher Model Intruct
Actions and Observationn
Privileged Knowledge User Preferences
203 Intrec
ReflectionDPO
Rellccior Pronee
Porret
Progress Summory
Student Modal 200 Iarrcc
Teacher
Student Act
Candidate Question
Teacher Action
a
Danapooint
Reflection tharing
Pronct
ReflectionDDPO
Detaxen
Repooted
Donapoint
Figure 2: Training mechanism of Reflection-DPO, using a teacher model and probability-based reflection to create dataset of desirable actions for finetuning the student using DPO trainer
such tea, are identified through object descriptions, such as box of green tea bags, box of chamomile tea, etc. We randomly generate each scene by including all rooms, furniture and set of mandatory Sheeetra__O0P0 -0), and adding each of the other moveable objects with a 70% probability. We include 8 high-level tasks, related to breakfast preparation, such "Make toast and coffee for breakfast" and "Prepare omelet for breakfast" (full list in Appendix C.2). Due to the large combinatorial space of scenes, generated by sampling each object, each scene used in train and test interactions is unique, though composed of the same set of objects and tasks. The high-level nature of tasks, coupled with the presence of broad range of objects, allows several variations in each task, supporting diverse preferences, such adding a specific category of ingredient, and preferring one ingredient variation over another.
saakesshake..ae 0', 'pepper.shaker shaker 00' 'water bottle bottle 00'
Modeling user personas: Finally, we create persona models to simulate users, powered by LLMs and characterized by a set of preferences. ADAPT includes 16 personas, each with an average of 14.25 preferences, 5.5 of which apply average to given task (see AppendixC.3 for a complete list). Preferences include: 1) preferring certain food variations, such as non- dairy milk, 2) adding sides or toppings, such as herbs to eggs nuts to yogurt, 3) excluding add-ons, such as no cream or sugar with coffee, 4) temporal ordering preferences, such as pouring cereal before milk, or preparing beverages before food, 5) preferred object usage, such as preferring pour-over for coffee, 6) preferred locations for serving meals, 7) preferring certain preparation alternatives, such as preferring tea iced over hot.
Each preference includes a textual description, and a verification function. We prompt an LLM with the user preferences to enable answer questions during task execution. The verification functions are python functions that can evaluate whether the linked preference is satisfied, violated or inapplicable be conditioned on the task at hand, such as 'when making toast use whole grain bread', and on the availability of objects, such as 'use the espresso machine, if available, else pour-over'. We track entities through an interaction to verify preference satisfaction, keeping track of object usage, additions during preparation, temporal order of usage, and final placement. Both preferences and verification functions manually created.
given interaction. The applicability of preference can
Learning to plan with user preferences
To enable an LLM to learn actively elicit user preferences while performing a task, we introduce Reflection-DPO, a training mechanism which leverages a privileged teacher model with access to user preferences. Reflection-DPO finetunes an LLM using a corrective dataset of desirable predictions, which can be physical actions or questions (Figure 2: left). To create the corrective dataset, we choose between physical actions demonstrated by the privileged teacher model, and questions demonstrated by reflection mechanism at every timestep (Figure 2: right). At its core, Reflection-DPO has three main components: 1) Dagger-style (Ross et al., 2011) demonstration of physical actions by privileged teacher model, 2) demonstration of informative questions by a novel reflection mechanism, and 3) Direct Preference Optimization (DPO) (Rafailov et al., 2024) for finetuning the student L.M.M
5

---

Published as a conference paper at COLM 2025
Generating data with a privileged teacher: We use LLM with privileged informa- tion about user preferences to create a teacher πt that generates data to train student LLM n². First, given a set of personas and tasks, we generate interaction trajectories &g 00:K, M0:K, using the student model. Next, we use the teacher model, which has access to privileged preferences r, to predict an action at each time-step k in the student trajectory: of = πt (g, ,, fox 1 Pok Mook 1). Similar to Dagger (Ross et al., 2011), this results in corrective dataset, where teacher actions a provide corrections to the student actions of. We skip datapoints where teacher and student predict the same action.
The student model cannot directly imitate the teacher, because the teacher model has access to privileged information regarding user preferences, which the student lacks. To teach the student model to actively elicit missing information in an efficient manner, we introduce a novel reflection mechanism, outlined in Algorithm 1. The reflection mechanism queries the student model with a reflection prompt that includes the student and teacher predictions, along with the interaction history. The reflection prompt asks the student model to generate candidate question at that might help the student predict the teacher's ss action. For example, if teacher action adds almonds to the user's 's cereal, a candidate question might ask about desired toppings.
Reflection to generate questions:
However, when prompted in this way, the student model tends to produce a question even if there no missing information, and often the predicted ques-
Algorithm 1 Reflection mechanism
function REFLECTION(Pr: (.), a², at)
tion might not be relevant to the user preference. To ensure that the questions are useful, we compute the
atl +- getQuestion(P,, (Prr,, a, at)
A + Pr q)) Prr (a')
utility of a question in predicting the teacher action. Given interaction history x, calculate question utility Ag as the increase in probability of predicting teacher action under the student distribution given
A + Pyr (a') ) - Pre (at)
if A. < 0 then: Rehosenn
else if Aq > 61 then: #chosen 119
else if A < C2 then: Achosen a¹
the question: Ag P,ee (a') ( x) - Pae (a" [x), where Par (a) is the probability of predicting an action a by the student LLM given context x, calculated as average over tokenwise probabilities. If the question is deemed useful (A, > ε1), we add the question to the training dataset 4chosen = a9. Other- wise we add the teacher action to the training dataset, as long as the teacher action is not significantly out of distribution for the student. To determine this, we measure the difference
else: Achosen
None
end if
return (ghosse
end function
in probabilities of a" and a' under the student distribution: A = P [x) Px (a' (a')x), and
if A; < E2, we choose the teacher action 4chosen = a². If neither conditions apply, we conclude that some information was missing but the question did not elicit it effectively and skip that datapoint. Additionally, if greedy decoding causes the teacher action to be more probable under the student distribution, i.e. A, < O, we choose the teacher action achosen = a'. The original student action a, becomes the rejected action arejected = a² in all cases.
Direct Preference Optimization (DPO) for policy optimization: To train the student model using the corrective data generated from the reflection process, we employ Direct Preference Optimization (Rafailov et al., 2024), a variant of reinforcement learning from human feedback (RLHF). DPO optimizes the policy using closed-form objective that assumes a Bradley-Terry model (Bradley & Terry, 1952) of the user's reward function, enabling more stable training compared to traditional RLHF methods, which involve training a reward model and policy separately (Rafailov et al., 2024). We leverage DPO training over direct supervised finetuning, because we empirically find that it results in better performance (Section 6.3). We attribute this improvement to the KL-divergence constraint inherent in DPO, which prevents the model from deviating significantly from the base policy. This constraint is particularly beneficial in our context, as it helps avoid model forgetting when training on relatively small dataset. For further regularization, we shorten the prompt to only contain a summary of rollout history, instead of the entire history. Each element of our dataset [(x, Achosen #rejected)) includes a chosen action achosen, a rejected action Arejected, and a prompt x, which comprises the goal g, and a history of student actions fxx1 and corresponding observations 0 and any user feedback U0x-1- This ensures
6

---

Published as a conference paper at COLM 2025
that the student model is effectively aligned with user preferences, demonstrated by the teacher, and asks questions when needed, while maintaining its foundational capabilities. In our experiments, the training data for Reflection-DPO contains about 37% questions, and 48% teacher actions. The rest of the datapoints deem the student action as appropriate.
6 Experiments
In this section, we analyze the performance of Reflection-DPO, state-of-the-ar baselines, and ablations of Reflection-DPO. Our experiments highlight the importance of active questioning and its role in improving preference satisfaction in ADAPT. We also analyze ablations of Reflection-DPO, where we remove the reflection step, and replace DPO with supervised fine-tuning. We benchmark Reflection-DPO against the following baselines:
ReAct (Yao et al., 2023): A chain-of-thoughh style prompting by interleaving action predic- tion with thought prediction, popular approach for using LLMs in planning.
STaR-GATE (Andukuri et al., 2024): Finetunes the student model using the top one-third of student rollouts that fulfill the highest fraction of user preferences. This reflects a common method of leveraging rollout rewards for finetuning LLMs.
Baseline LLM: Untrained zero-shot LLM, serves as baseline to measure the benefit of training and question-asking capabilities.
'Always Ask LLM: Variant of baseline LLM that asks question before every action.
'Never Ask' LLM: Variant of the baseline LLM that never asks a question, serving as lower bound of performance without any preference information on ADAPT tasks.
Teacher LLM: LLM with access to privileged user preferences, providing an upper bound on expected performance when learning from it. Does not need to ask any questions.
Both -.55--2.--7BB and Llama 3.1-rob-Instruct show comparable performance on ReAct, our strongest baseline, shown in Table 1. We choose Llama 3.1-70b-Instruct for all our experiments. We add decoder constraints, in the form of context-free grammar (Geng et al., 2023), to limit outputs to valid scene nodes and actions to all baselines in ADAPT tasks. This minimizes hallucinations and improves performance across the board for all approaches. While ADAPT allows open-set action verbs, limit them to 82 plausible meal-related actions, like dice and peel, while keeping arguments and results open-set. Despite these constraints, nearly 60k valid actions remain available at any timestep. We evaluate gener- alization to unseen environments over both seen and unseen personas. We cross-validate across the 16 personas available in ADAPT, using 12 personas for training and for evalu- ating generalization to unseen persona. The training mechanisms for Reflection-DPO, its ablations, and STaR-GATE, use a total of 192 trajectories, composed of 16 interactions with each of the 12 personas. This results in 1500 training datapoints for STaR-GATE, and about 5000 datapoints for Reflection-DPOand its variants. Note that STaR-GATE has about 1/3 of the datapoints by design (top 1/3 trajectories). We use Low Rank Optimization (Hu et al., 2022) with a rank of for all models for efficient and regularized training. We set the 0.5 to preserve informative questions with- out allowing them to dominate over physical actions. This configuration yields a dataset composed of 37% questions and 48% physical actions.
reflection hyperparameters to E = 0.2 and
6.1 Quantitative Analysis
Metrics: We measure performance using two metrics: preference satisfaction rate and the number of questions asked. In the ADAPT tasks, we evaluate trajectory by the preferences satisfied p and violated p-, resulting in preference satisfaction rate p+-p- A preference p+
is considered satisfied only if the relevant subtask is completed, such as ensuring tea bag is added to water to make tea. Subtask failure, such as not making tea when instructed to "make tea and toast for breakfast," results in all related preferences being violated. Thus the preference satisfaction rate reflects both task completion and percentage of preferences satisfied. Additionally, we measure the number of questions each approach asks.
7

---

Published as a conference paper at COLM 2025
Preference Satisfaction Rate
Num. Questions Asked
Method
Seen Persona
Unseen Persona
Seen Persona
Unseen Persona
'Never Ask LLM
27.5%
28.6%=1.2%%
0.0
0.0
Baseline LLM
34.7%:20.88%
34.3%==1.%%
2.3+0.0
2.3=0.1
ReAct
36.1%11..80
36.8%=14%%
2.3+0.0
24=0011
ReAct (Qwen)
36.4%%:t1. 1.5%%
39.5%=266%
2.640.1
2.7=0.2
STaR-GATE
34.1 +-.8%%
33.5%++144%
2.0+0.0
2.0+0.0
Reflection-DPO
44.1 % +.00% 1.0%
42.9%+1.6%
9.8+0.1
9.5+0.2
AAwwys Ask LLM
52.1% 9.1..%
50.8%=1.6%
22.24:011
22.4=02
Teacher LLM
65.5%++1..1%
65.5%=+1.99 1.9%
0.0+0.0
0.0+0.0
Table 1: Comparison Reflection-DPO and baselines for average number of questions asked and preference satisfaction rate.
Reflection-DPO is able to elicit user preferences even for unseen personas. Table 1 shows a comparison of Reflection-DPO against all baselines, reported through the mean and standard error for both seen and unseen persona. Reflection-DPO achieves preference satisfaction rate of 44.1% for seen personas and 42.9% for unseen personas, outperforming all baselines, including ReAct (36.1% seen, 36.8% unseen) and STaR-GATE (34.1% seen, 33.5% unseen). This indicates that Reflection-DPO effectively generalizes without overfitting to the training personas. This shows that Reflection-DPO able elicit and adapt to user preferences better than leading zero-shot baseline ReAct, as well as STaR-GATE which is finetuned on successful past trajectories. Note that the performance is similar between seen and unseen persona because the training data contains no information about the specific persona is being trained on. As a result, Reflection DPO is able to learn how to elicit and adapt to preferences for any user, seen unseen equally well.
Asking questions is critical to high-performance on ADAPT. The 'Always Ask' model shows that asking questions significantly improves performance, achieving satisfaction rate of 52.1% for seen personas and 50.8% for unseen personas. However, it asks an average of 22 questions per interaction, which is impractical and overwhelming in real-world scenarios. In contrast, Reflection-DPO asks about 10 questions per interaction, efficiently eliciting preferences and approaching the performance of Always Ask with fewer questions. Baseline models, such as the Baseline LLM and ReAct, ask only about 2-3 questions, which is insufficient for uncovering preferences and 'Never Ask' performs the worst. This indicates that ADAPT requires active questioning, and Reflection-DPO is able to elicit preferences.
Reflection-DPO learns from an imperfect Teacher. Even the privileged Teacher LLM, with satisfaction rate of 65.5% for both seen and unseen personas, does not achieve perfect performance due to the long-hord long-horizon nature tasks. For example, we observe that the Teacher might miss some steps, resulting in failures, such as retrieve butter and toast the bread but forget to butter the toast. Additionally, the freedom given to the planner to name entities resulting from freeform actions often mislead it into believing that step is taken. Baseline- without any preferences performs significantly worse (34.7% seen, 34.3% unseen) compared to Teacher, but Reflection-DPC is able to bridge this gap (44.1% 1% seen, 42.9% unseen). This shows that even with an imperfect Teacher, Reflection-DPO is able to learn to follow and elicit preferences in complex, long-horizon ADAPT tasks. There may be further improvements through reinforcement learning and learning from experience, bridging the gap to the Teacher and even outperforming it, which we leave for future work.
6.2 Qualitative Analysis of Questions asked
Reflection-DPO adapts questioning to preferences. Reflection-DPO learns to ask more questions when more preferences are applicable, probing further when users have more task-related preferences. In contrast, baselines ask a similar number of questions regardless of the interaction context (Figure 3).
On average, each task has 5-6 relevant preferences, and Reflection-DPO asks 4-5 extra ques- tions, while baselines fall short by 2-3 questions. Moreover, we observe that while Reflection- DPO asks extra questions (examples discussed in Appendix A), they are reasonable prefer- ences that other personas could have had. However we observe that Reflection-DPP asks
8

---

Published as a conference paper at COLM 2025
Preference Satisfaction Rate Unseen Persona
Num. Questions Asked
Method
Seen Persona
Seen Persona
Unseen Persona
Reflection-DPo
44.1% 1.0%%
42.9% 1.6%%
9.1
9.5.10.2
Ablation: no Reflection
17.5%=20.7%
16.5%1122%
1.0=0.0
0.9=0.0
Ablation: no DPO
3..3%=2088%
34.9%+14%%
2.9=0.1
2.9=0.1
Table 2: Comparison of Reflection-DPO against ablated versions without the reflection mechanism DPO training, and against student model that asks a question before each action.
more questions than strictly needed, which usually helps adapt to user preferences that the agent has no prior knowledge about. We leave it to future work to use knowledge from past interactions to further reduce questions.
Reflection-DPO asks informative questions. Qualitatively, observe that Reflection-DPO asks higher-quality, and more informa- tive questions than baselines, For example, start of an interaction, it asks open-ended questions, such as "Do you want any toppings on your cereal?" and provides exam- ples to clarify, such as "Would you like your coffee with any creamers, sweeteners, or flavorings, such as sugar, milk, or vanilla syrup?". When fewer options exist, it sim- plifies questions to binary responses, such "Do you want milk in your coffee?". In contrast, baselines struggle to ask sufficient and prioritized questions. They can ask disambiguation questions when multiple options are present, but often fail to elicit all relevant user prefer- ences. The Always Ask' LLM covers more preferences but gets sidetracked by questions about the world state or object availability, such as "Is the stove set to medium heat?". This information does not need user-feedback and is already available in the environment, resulting in missed opportunities to elicit actual preferences, which prevent the model from closing the gap to the Teacher. We include a discussion of when during trajectory different methods adapt to preferences in Appendix A.
Sudert
Relet
Stan GATE
Nom. Queettons
Ieffectionbi
8.8
2.2
1.9
2.4
2.4
2.1
2.3
2.3
2.0
preferences (39% interactions)
5-6 preferences interactions)
7:14 preferences interactions)
Figure 3: Num. questions asked over interactions with vary- ing number of preferences show that Reflection-DPO asks more questions when more preferences exist.
6.3 Ablations
Reflection-DPO differs from classical Dagger in two ways: the use of a reflection mechanism and DPO-based training. Table 2 studies the impact of both of these components.
Reflection mechanism enhances questioning. We compare Reflection-DPO to a "no Reflection" version that uses teacher actions directly for training. Without reflection, the preference satisfaction rate drops to 17.5% for seen personas and 16.5% for unseen personas, compared 44.1% and 42.9% with reflection. The teacher data contains no questions, hence "no Reflection" asks few questions, failing to uncover preferences.
DPO training improves generalization with small training dataset. We also evaluate "no DPO" version of Reflection-DPO, trained with supervised finetuning the chosen actions. This version achieves a satisfaction rate 35.3% for seen personas and 34.9% for unseen personas, lower than Reflection-DPO. The large action space and long-horizon tasks in ADAPT create a vast input prompt space, leading to out-of-distribution evaluations, even for seen personas and tasks. The "no DPO" variant overfits to the small training dataset, despite extensive hyperparameter tuning. In contrast, DPO's KL-constraint regularizes training, preventing model forgetting and enabling better generalization.

---

Published as a conference paper at COLM 2025
7 Summary and Limitations
We present ADAPT, benchmark for evaluating assistive agents at adapting long-horizon tasks hidden user preferences, requiring preference elicitation by actively asking questions. We also propose novel approach Reflection-DPO that finetunes an LLM for active ques- tioning using a privileged teacher to identify preference-adherinn step to identify when to ask questions. Reflection-DPO demonstrates strong performance in our experiments, eliciting user preferences for both seen and unseen personas and achieving higher satisfaction rates than baselines like chain-of-thought and supervised finetuning. This success is due to its ability to ask informative questions, efficiently uncovering user preferences with relatively few questions. Additionally, our ablation experiments demon- strate that the reflection mechanism significantly improves the LLM's questioning ability, and DPO training improves generalization, even with small training dataset.
actions, and a reflection
Ultimately, all question-asking models, including Reflection-DPO fall short of a privileged teacher's performance, indicating room for improvement on ADAPT. Most importantly, even the privileged teacher makes mistakes, in both preference satisfaction and task comple- tion, indicating the need for learning stronger demonstrators, or learning from experience. Additionally, Reflection-DPO does not penalize questions, and as result asks more ques- tions than strictly needed. Future work can optimize the number of questions by introducing penalties and learning from prior experience with the same user.
References
Chinmaya Andukuri, Jan- ahiPpii Fränken, Tobias Gerstenberg, and Noah Goodman. STar- GATE: Teaching language models ask clarifying questions. In First Conference on Language Modeling, 2024.
Shurjo Banerjee, Jesse Thomason, and Jason J. Corso. The RobotSlang Benchmark: Dialog- guided Robot Localization and Navigation. In The Conference on Robot Learning (CORL), 2020. URL https: /ttp///axvvvv//a////a/ http://aaaww...oaabssbss// orgg 12639.
Cristian-Paul Bara, Sky CH-Wang, and Joyce Chai. MindCraft: Theory of mind modeling for situated dialogue in collaborative tasks. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 1112-1125, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653/vv 2021 2021. wee.eemllp-ain... URL https:///allanthology haw.aaanthooogyorrg////000 org/2021 emn1p-main.85..
Erdem Bryık, Dylan P Losey, Malayandi Palan, Nicholas Landolfi, Gleb Shevchuk, and Dorsa Sadigh. Learning reward functions from diverse sources of human feedback: Optimally integrating demonstrations and preferences. The International Journal of Robotics Research, 41(1):45-67, 2022.
Ralph Allan Bradley and Milton E. Terry. Rank analysis of incomplete block designs: 1. the method of paired comparisons. Biometrika, (1033//44444333/3 1952. ISSN 00063444, 14643510. URL http://www. wttp:/jstor.or//ttabbbooooo wwt.:///aaae//3342222222
Anthony Brohan, Yevgen Chebotar, Chelsea Finn, Karol Hausman, Alexander Herzog, Daniel Ho, Julian lbarz, Alex Irpan, Eric Jang, Ryan Julian, et al. Do as can, not as i say: Grounding language in robotic affordances. In Conference on robot learning, pp. 287-318. PMLR, 2023.
Leonard Bärmann, Rainer Kartmann, Fabian Peller-Konrad, Jan Niehues, Alex Waibel, and Tamim Asfour. Incremental learning of humanoid robot behavior from natural interaction and large language models. Frontiers Robotics and AI, 11, 2024. ISSN 2296-9144. doi:
Jon Ander Campos and Jun Shern. Training language models with language feedback. In ACL Workshop on Learning with Natural Language Superaision. 2022., 2022.
10

---

Published as a conference paper at COLM 2025
Matthew Chang, Gunjan Chhablani, Alexander Clegg, Mikael Dallaire Cote, Ruta Desai, Michal Hlavac, Vladimir Karashchuk, Jacob Krantz, Roozbeh Mottaghi, Priyam Parashar, Siddharth Patki, Ishita Prasad, Xavier Puig, Akshara Rai, Ram Ramrakhya, Daniel Tran, Joanne Truong, John M Turner, Eric Undersander, and Tsung-Yer Yen Yang. PARTNR: A benchmark for planning and reasoning in embodied multi-agent tasks. In The Thirteenth International Conference on Learning Representations, 2025. URL forum?ide=5QLRRRyyL..
net/
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei.
Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
Spandana Gella, Aishwarya Padmakumar, Patrick Lange, and Dilek Hakkani-Tur. Dialog
Acts for Task-Driven Embodied Agents. In Proceedings of the 23nd Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDial), pp. 111-123, 2022.
Saibo Geng, Martin Josifoski, Maxime Peyrard, and Robert West. Skammar-constraiing
decoding for structured NLP tasks without finetuning. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Singapore, December 2023. Association for Computational Linguistics. URL https://aclantiology.or org. 2023. emnlp-main. 674.
Dongge Han, Trevor McInroe, Adam Jelley, Stefano V. Albrecht, Peter Bell, and Amos Storkey, LLM-personalize: Aligning LLM planners with human preferences via reinforced self- training for housekeeping robots. In Owen Rambow, Leo Wanner, Marianna Apidianaki, Hend Ål-Khalifa, Barbara Di Eugenio, and Steven Schockaert (eds.), Proceedings of the 31st International Conference on Computational Linguistics, pp. 1465-1474, Abu Dhabi, UAE, January 2025, Association for Computational Linguistics. URL https: hacps////oo/..ooog..... ht122225.colig. main. 98/.
Kunal Handa, Yarin Gal, Ellie Pavlick, Noah Goodman, Jacob Andreas, Alex Tamkin, and Belinda Li. Bayesian preference elicitation with language models. arXiv preprint arXiv:2403.05534, 2024.
Joey Hong, Sergey Levine, and Anca Dragan. Zero-shot goal-directed dialogue via rl imagined conversations. arXiv preprint ar arXi::2311.05584 2023.
Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022.
Wenlong Huang, Chen Wang, Ruohan Zhang, Yunzhu Li, Jiajun Wu, and Li Fei-Fei. Vox- poser: Composable 3d value maps for robotic manipulation with language models. In 7th Annual Conference on Robot Learning, 2023a. URL BLF30mOC.
serenreeiew.net/foumiiiiii net/forum?id=
Wenlong Huang, Fei Xia, Dhruv Shah, Danny Driess, Andy Zeng, Yao Lu, Pete Florence, Igor Mordatch, Sergey Levine, Karol Hausman, et al. Grounded decoding: Guiding text generation with grounded models for embodied agents. Advances in Neural Information Processing Systems, 36:5933--5961 2023b.
Vidhi Jain, Yixin Lin, Eric Undersander, Yonatan Bisk, and Akshara Rai. Transformers are adaptable task planners. In Conference on Robot Learning, pp. 1011-1037. PMLR, 2023.
Ivan Kapelyukh and Edward Johns. My house, my rules: Learning tidying preferences with graph neural networks. In Conference on robot learning, pp. 740-749. PMLR, 2022.
Yigit Korkmaz and Erdem B1yIk. Mile: Model-based intervention learning. arviv preprint arXiv:2502.13519, 2025.
Zhaoting Li, Rodrigo Pérez-Dattari, Robert Babuska, Cosimo Della Santina, and Jens Kober. Beyond behavior cloning: Robustness through interactive imitation and contrastive learning. arXi preprint arXto=2502.07645, 2025.
11

---

Published as a conference paper at COLM 2025
Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control. In 2023 IEEÉ International Conference Robotics and Automation (ICRA), pp. 9493-9500. IEEE, 2023.
Jacky Liang, Fei Xia, Wenhao Yu, Andy Zeng, Montserrat Gonzalez Arenas, Maria Attarian, Maria Bauza, Matthew Bennice, Alex Bewley, Adil Dostmohamed, Chuyuan Kelly Fu, Nimrod Gileadi, Marissa Giustina, Keerthana Gopalakrishnan, Leonard Hasenclever, Jan Humplik, Jasmine Hsu, Nikhil Joshi, Ben Jyenis, Chase Kew, Sean Kirmani, Tsang- Wei Edward Lee, Kuang-Huei Lee, Assaf Hurwitz Michaely, Joss Moore, Ken Oslund, Dushyant Rao, Allen Ren, Baruch Tabanpour, Quan Vuong, Ayzaan Wahid, Ted Xiao, Ying Xu, Vincent Zhuang, Peng Xu, Erik Frey, Ken Caluwaerts, Tingnan Zhang, Brian Ichter, Jonathan Tompson, Leila Takayama, Vincent Vanhoucke, Izhak Shafran, Maja Mataric, Dorsa Sadigh, Nicolas Heess, Kanishka Rao, Nik Stewart, Jie Tan, and Carolina Parada. Learning to learn faster from human feedback with language model predictive control. 2024.
Jessy Lin, Nicholas Tomlin, Jacob Andreas, and Jason Eisner. Decision-oriented dialogue for human-Al collaboration. Transactions of the Association for Computational Linguistics, 12:892- 911, 2024. doi:
URL https: / http///aaaaaaaalaaaee/ee org/2024. tacl-1. 100/
Bo Liu, Yuqian Jiang, Xiaohan Zhang, Qiang Liu, Shiqi Zhang, Joydeep Biswas, and Peter Stone. Llm+ p: Empowering large language models with optimal planning proficiency. arxiv preprint arXiv:2304.11477, 2023.
Huihan Liu, Soroush Nasiriany, Lance Zhang, Zhiyao Bao, and Yuke Zhu. Robot learning the job: Human-in-the-loop autonomy and learning during deployment. The International Journal of Robotics Research, pp. 02783649241273901, 2022.
Anjali Narayan-Chen, Prashant Jayannavar, and Julia Hockenmaier. Collaborative dialogue in Minecraft. In Anna Korhonen, David Traum, and Lluis Mårquez (eds.), Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 5405-5415, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653 (0118633vv1/P11 /v11119-37/ URL httts://aallanthoooogy .wceegy..rrg//.. -
Aishwarya Padmakumar, Jesse Thomason, Ayush Shrivastava, Patrick Lange, Anjali Narayan-Chen, Spandana Gella, Robinson Piramuthu, Gokhan Tur, and Dilek Hakkani- Tur. TEACh: Task-driven Embodied Agents that Chat. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 2017-2025, 2022.
Jeongeun Park, Seungwon Lim, Joonhyung Lee, Sangbeom Park, Minsuk Chang, Youngjae Yu, and Sungjoon Choi. Clara: classifying and disambiguating user commands for reliable interactive robotic agents. IEF.F. Robotics and Aulomation Letters, 9(2):1059-1066, 2023.
Rodrigo Pérez-Dattari, Carlos Celemin, Javier Ruiz-del Solar, and Jens Kober. Continuous
control for high-dimensional state spaces: An interactive learning approach. In 2019 International Conference on Robotics and Automation (ICRA), pp. 7611-7617 IEEE, 2019.
Wasu Top Piriyakulkij, Volodymyr Kuleshov, and Kevin Ellis. Active preference inference using language models and probabilistic reasoning. arxiv preprint arXiv:2312.12009, 2023.
Xavier Puig, Kevin Ra, Marko Boben, Jiaman Li, Tingwu Wang. Sanja Fidler, and Antonio
Torralba. Virtualhome: Simulating household activities via programs. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8494,8502, 2018
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in Neural Information Processing Systems, 36, 2024.
Krishan Rana, Jesse Haviland, Sourav Garg, Jad Abou-Chakra, Ian Reid, and Niko Suen- derhauf. Sayplan: Grounding large language models using 3d scene graphs for scal- able robot task planning. In 7th Anmual Conference on Robot Learning, 2023. URL https://openrevias mett/forumiidd:-mo00000ssss
12

---

Published as a conference paper at COLM 2025
Allen Z. Ren, Anushri Dixit, Alexandra Bodrova, Sumeet Singh, Stephen Tu, Noah Brown, Peng Xu, Leila Takayama, Fei Xia, Jake Varley, Zhenjia Xu, Dorsa Sadigh, Andy Zeng, and Anirudha Majumdar. Robots that ask for help: Uncertainty alignment for large language model planners. In 7th Annual Conference on Robot Learning, 2023. URL https: wttp:/wwwnnn/ooouni///4///-////
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp. 627 6655. JMLR Workshop and Conference Proceedings, 2011.
Dorsa Sadigh, Anca Dragan, Shankar Sastry, and Sanjit Seshia. Active preference-based learning of reward functions. 2017.
Omar Shaikh, Kristina Gligorić, Ashna Khetan, Matthias Gerstgrasser, Diyi Yang, and Dan Jurafsky. Grounding or guesswork? large language models are presumptive grounders. 2023.
Pratyusha Sharma, Balakumar Sundaralingam, Valts Blukis, Chris Paxton, Tucker Hermans, Antonio Torralba, Jacob Andreas, and Dieter Fox. Correcting Robot Plans with Natural Language Feedback. In Proceedings of Robotics: Science and Systems, New York City, NY, USA, June 2022. doi: 10. /T2.///S..22.....1.66666
Kunal Pratap Singh, Luca Weihs, Alvaro Herrasti, Jonghyun Choi, Aniruddha Kembhavi, and Roozbeh Mottaghi. Ask4help: Learning to leverage an expert for embodied tasks. Advances in Neural Information Processing Systems, 35:16221-16232, 2022.
Shohei Tanaka, Konosuke Yamasaki, Akishige Yuguchi, Seiya Kawano, Satoshi Nakamura, and Koichiro Yoshino. Do demand, not as say: dataset for developing a reflective life-support robot. IEEE Access, 12:11774-11784, 2024.
Jesse Thomason, Michael Murray, Maya Cakmak, and Luke Zettlemoyer. Vision-and-dialog navigation. In Conference on Robot Learning (CoRL), 2019.
M
Siddharth Verma, Justin Fu, Mengjiao Yang, and Sergey Levine. Chai: A chatbot ai for task-oriented dialogue with offline reinforcement learning. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), 2022.
Huaxiaoyue Wang, Nathaniel Chin, Gonzalo Gonzalez-Pumariian Xiangwan Sun, Neha Sunkara, Maximus Adrian Pace, Jeannette Bohg, and Sanjiban Choudhury. APRICOT: Active preference learning and constraint-aware task planning with LLMs. In 8th Annual Conference on Robot Learning, 2024.
Jimmy Wu, Rika Antonova, Adam Kan, Marion Lepert, Andy Zeng, Shuran Song, Jeannette Bohg, Szymon Rusinkiewicz, and Thomas Funkhouser. Tidybot: Personalized robot assistance with large language models. Autonomous Robots, 2023.
Zhaojing Yang, Miru Jun, Jeremy Tien, Stuart J. Russell, Anca Dragan, and Erdem Biyik. Trajectory improvement and reward learning from comparative language feedback. In 8th Annial Conference on Robot Learning, 2024.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations, 2023.
Sriram Yenamandra, Arun Ramachandran, Karmesh Yadav, Austin S Wang, Mukul Khanna, Theophile Gervet, Tsung- Yen Yang, Vidhi Jain, Alexander Clegg, John M Turner, Zsolt Kira, Manolis Savva, Angel X Chang, Devendra Singh Chaplot, Dhruv Batra, Roozbeh Mottaghi, Yonatan Bisk, and Chris Paxton. Homerobot: Open-vocabulary mobile manipulation. In 7th Annual Conference on Robot Learning, 2023.
13

---

Published as a conference paper at COLM 2025
Koichiro Yoshino and Tatsuya Kawahara. Conversational system for information navigation based on pomdp with user focus tracking. Computer Speech & Language, 34(1):275-291, 2015.
Jenny Zhang, Samson Yu, Jiafei Duan, and Cheston Tan. Good time to ask: A learning framework for asking for help in embodied visual navigation. In 2023 20th International Conference on Llbquuitous Robots (UR), pp. 503-509. IEEE, 2023.
14

---

Published as a conference paper at COLM 2025
Appendix
A Preference elicitation performance over time
In this section, examine when during interaction each model adapts to various pref- erences. Figure 4 shows the average performance of each model throughout the course of an interaction, ultimaaely achieving the preference satisfaction rates reported in Section at 100% completion. Our evaluation function is meant to be used for a complete trajectory, because preferences such not adding ingredients and action ordering can wrongly appear to be satisfied when looking at partial trajectory. To avoid such misleading effects, we mea- sure the preference satisfaction rate each step of an interaction by evaluating specifically those preferences which were satisfied at the end of the interaction. Doing so effectively captures the fraction final performance achieved at all times over the span trajectory.
RoAct
STaR GATE
Reflectior-DPP
Preference Sottsfaction
100
% Rollout Completed
Figure 4: Comparison of preference satisfaction rate of different models over the interaction, showcasing when they adapt to differnt preferences.
Reflection DPO shows a slower improvement at the beginning, but continuously improves thereafter, surpassing all baselines. A subjective inspection shows that Reflection-DPO prioritizes exploration actions at the beginning, causing delay in asking about and adapting to preferences. Later in the interaction, keeps improving as result of asking more questions. Reflection-DPO continues to probe for more fine the-grained preferences, such as asking about which kind of nuts to use, once it knows that the user prefers to top their cereal with nuts. In contrast, all baselines show steady improvement through the initial part of the interaction, but their improvement tapers off at the end.
Soo 1996
Make cereal and coffee for breakfast
X
X
K
X
X
X
Student
X
X
X
ReAct
coffee brewingg meshodl
coffee bypa
cereal 1y24
milespp
caffee
Inlk type
X
SToR-GATE
Geneed type
whohholoodd inãl peveerstinn
RoflectionDPO
sevetener coffee
Inspenn)
CUt preference
done
Figure 5: Example actions taken through the course a cereal and coffee task, with task component that each action is related to indicated by colors, and what different methods asked about. All baselines ask similar set of 3-4 questions, but Reflection-DPO is able to probe further for more preferences.
Figure 5 shows example comparing trajectories for all methods task of making cereal and coffee. Reflection-DPO is able elicit a wide range of preferences by asking more general question at the beginning, followed by fine-grained follow-up questions towards
15

---

Published as a conference paper at COLM 2025
the end. We observe that the baselines consistently ask generic questions at the beginning, such as what kind coffee the user would like, or they would like their eggs, but later on in the interaction their questions are between instances of objects uncovered during exploration, such as various types of milk. Finally, the Alwayys Ask and Teacher LLM models show steady improvement till the end, as they steadily adapt to preferences known a priori or elicited continuously throughout the course of the interaction.
sparse, and are mostly related to disambiguating
B Satisfaction Rate per Question
Measuring satisfaction rate per question is challenging due to the lack of "ground truth" for whether question directly relates to preference satisfies multiple preferences. For instance, if the agent asks, "Do you like low-fat or whole milk?" and the user responds, "I am trying to be healthy, low fat," the LLM might use this to offer healthier options like whole-wheat cereal in subsequent interactions. this case, one question and response are responsible for multiple preference satisfaction.
To approximate question efficiency, we consider the ratio of preferences satisfied to user input, i.e. the number of questions and initial goal specification P+ NT+ We report results comparing Reflection-DPO against baselines on this metric Table 3. Note that this metric can favor models that satisfy preferences by chance without asking questions, asking very few questions.
Method
Seen Persona
Unseen Persona
Baseline LLM
0.55
0.54
ReAct
0.57
0.56
STaR-GATE
0.59
0.59
Reflection DPO
0.26
0.25
Always Ask
0.12
0.12
Table 3: Satisfaction Rate per question of Reflection DPO compared against baselines
As expected, baselines that ask few questions like ReAct are more efficient this metric, while Reflection-DPO and Always Ask are less efficient. Note that we currently don't explicitly penalize for asking extra questions, and merely aim to satisfy preferences (noted as a limitation in Section 7). In future work, we aim to increase the efficiency of our approach, aiming for models that can ask fewer questions while satisfying preferences.
C Details of ADAPT benchmark
The ADAPT benchmark includes 8 tasks and 16 personas in a simulation environment, where the agent can perform an open-set of actions. In this section we list out all the tasks, action templates and list of persona preferences in ADAPT.
C.1 Actions
The full list of actions available to the planner, along with templates that the planner can use to invoke them, and the action-specific pre-conditions that must hold for successful execution of the action enumerated in Table 4. Note that the existence of each entity referred in an action, excluding the result entities (those preceded by 'to get'), is a pre- condition for all actions.
16

---

Published as a conference paper at COLM 2025
Action
Action Template
Conditions for success
ask
Ask (X)
open
Open X
close
Close X
turn_on
Turn on X
X must be cooking appliance
turn off
Turn off X)
must be a cooking appliance
heat
Heat X
either X the container containing
must be on cooking appliance
search
Search X)
look_for
Look for X
move_from Move X from (Y) to (Z)
must contain X
move
Move X to { Y ) Serve the
object
at Y Place the
object X) at ( YY)
pour
Pour X from ( (Y)
} to Z
must contain X
Pour X from Y into ( (
mix
Mix all items in X to get Y
must be a container
cook
Cook items in ( ) to get (Y)
must be on cooking appliance
chop
Chop (X) to get (X)
X must be chopping surface
freeform
( (X) items in ( Y to get Z
Y contain edible contents, and Z must
be container
freeform
(X) the object Y to get Z
Y must be edible, and Z must be con- tainer
done
Declare Done
Table 4: All actions available to the planner, the associated templates expected from the planner, and action-specific pre-conditions.
C.2 Tasks
The 8 tasks in ADAPT are:
1. Prepare eggs for breakfast
2. Prepare omelette for breakfast
3. Prepare cereal for breakfast
4. Make toast and coffee for breakfast
5. Make yoghurt parfait for breakfast
6. Make tea and eggs for breakfast
7. Make cereal and coffee for breakfast
8. Prepare tea and toast for breakfast
C.3 Persona Preferences
ADAPT consists of total of 16 personas. Each persona is composed of set of 10-19 preferences. Following is the list of preferences for each persona:
Persona 1:
1. Prefer hearty scrambled eggs or omelettes fitting a high-protein high-calorie diet
2. Add whole milk while preparing eggs
3. Beverages such as coffee should be black, without milk or sugar
Beverages such as tea should be black, without milk or sugar
Beverages such as coffee should served iced
Beverages such as tea should be served iced
7. Use full-fat milk whipped cream maintain high-fat high-calorie diet
17

---

Published as a conference paper at COLM 2025
8. Top sweet breakfasts, such as french toast, cereal, pancakes, oatmeal, yoghurt parfait, with whipped cream to add some extra calories
9. Add cheese avocado to savory breakfasts add some extra calories
10. Stainless steel or cast iron cookware and durable utensils
11. Cream or avocado are great sides when making eggs
12. Dining room or patio dining locations
Persona 2:
1. Eggs must be loaded with toppings such as vegetables
Eggs must be loaded with toppings such cheese
3. Pair toast with spread
4. Add nuts topping to sweet breakfasts, such oatmeal, yoghurt parfait, pancakes, french toast, cereal
5. Sprinkle cinnamon sweet breakfasts, such as oatmeal, yoghurt parfait, pancakes, french toast, cereal
6. Drizzle honey maple syrup sweet breakfasts, such as oatmeal, yoghurt parfait, pancakes, french toast, cereal
7. Add fresh herbs to eggs
8. Always use cheese and vegetables together, if making eggs
9. When preparing cereal, add cereal first then milk, in that order
10. Prefers dine on the patio
Persona 3:
Vegan diet using egg substitutes
Eggs should prepared with side of vegetables
3. Avocados favorite side, and especially necessary on toast
4. Beverages like coffee and tea preferred is preferred black without any added sugars creamers
5. To maintain vegan diet, vegan alternatives to milk
6. To maintain a vegan diet, use vegan alternatives to cheese when cooking eggs
7. To maintain a vegan diet, use vegan alternatives to yoghurts
side of berries a nice complement for any breakfast meal, no matter whether is savory or sweet
side nuts is nice complement for any sweet breakfast
10. Serve food directly the standing desk start the work day right away
Persona 4:
eggs are eaten, make them into an omelette with at least 3 spices
2. Add at least two vegetables when making eggs
3. Prefer using butter, for example toast
4. Prefer using butter, for example when cooking eggs
5. Prefer beverages like tea and coffee served hot Prefer beverages like tea and coffee served hot with milk and sugar
7. Prefer hint of cardamom in tea and coffee
8. If yogurt is eaten, prefer it plain
9. Add a hint of cardamom to sweet breakfasts, such as french toast, cereal, pancakes, oatmeal, yoghurt parfait
10. Add nuts topping to sweet breakfasts, such french toast, cereal, pancakes, oatmeal, yoghurt parfait
11. Top eggs and toast with cheese
12. Use chili as garnish or provide hot sauce condiment
13. Fresh herbs like cilantro or mint make lovely garnish for eggs
14. Serve breakfast at the formal dining room or outdoor patio
Persona 5:
Eggs should accompanied by crispy bacon
2. Eggs should be accompanied by toast
3. Only use traditional full calorie sugars in sweet breakfasts, such as french toast, cereal, pancakes, oatmeal, yoghurt parfait
18

---

Published as a conference paper at COLM 2025
4. Only use full fat dairy milk
5. Oatmeal cooked with milk
Add touch of sweetness to non-savory breakfasts such as pancakes, waffles, oatmeal, yoghurt parfait, etc.
7. Sweet breakfasts like french toast, cereal, yoghurt parfait, oatmeal, pancakes should be topped with maple syrup or honey
8. Sweet breakfasts french toast, cereal, yoghurt parfait, oatmeal, pancakes topped with nuts
9. Prefer flavored yoghurt
10. Yoghurt mixed with granola
11. Beverages such as tea and coffee preferred
12. Beverages such as tea and coffee prefemed with milk and sugar
13. Melted cheese is favorite addition to savory breakfast dishes, such eggs
14. A sprinkle of chopped parsley or chives would be nice touch to eggs
15. Serving beverages first would be a good approach, so it ready to be had as soon as possible
16. Use butter to cook eggs and top toast
17. Prepare one of sausage, hash browns, or toast as side when making eggs
18. Prefer to have condiment as side with savory breakfasts
19. Do not put vegetables in savory breakfasts
Persona 6:
1. Eggs must be scrambled, unless otherwise specified
2. Incorporate fresh herbs for garnish on savory breakfasts
Ron likes vegetables with eggs for added flavor
Toast should be lightly buttered
5. Toast should be topped with spread of natural jam or honey
6. Prefer lactose-free milk alternatives with cereal and oatmeal
7. When it comes tofrench toast, cereal, yoghurt parfait, oatmeal, or pancakes, he has sweet tooth and enjoys them topped with sweet ingredients like syrup, honey, or fresh fruit
8. He likes sweet breakfasts, such french toast, cereal, yoghurt parfait, oatmeal, pancakes topped with sprinkle of cinnamon
9. Yoghurt must be lactose free
10. Yoghurt or oatmeal must be topped with fruit
11. A sprinkle of fresh herbs is appreciated on savory breakfasts
12. Add lactose-free cheese to eggs
13. Add vegetables eggs
14. He prefers add milk first, then cereal rather than the other way around
15. Sides like bacon sausage, if available, are preferred with sweet breakfasts, like french
toast, cereal, yoghurt parfait, oatmeal, pancakes
16. He prefers his coffee to be served first to savor as he awaits his meal
Persona 7:
1. Poach eggs unless a different cooking method is specified
2. Add milk when making scrambled eggs or omelette
Sprinkle cheese on eggs
Serve least two condiments with eggs
5. Top toast with cheese
6. Use herbs toast
7. Use dairy milk when using any milk
Add an extra splash of dairy milk when cooking sweet breakfasts, like cereal, yoghurt parfait, pancakes, french toast, oatmeal
9. Use fruits as a side for sweet breakfasts, such as cereal, yoghurt parfait, pancakes, french toast, oatmeal
10. Serve beverages like coffee and tea iced, not hot
11. Use pour over coffee method
12. Beverages served black, without any added sugars or creamers
19

---

Published as a conference paper at COLM 2025
13. Use stainless steel cast iron cookware
Persona 8:
1. Add only salt and pepper to eggs aside from oil
2. Prefer healthy fats, such as olive oil to cook with
3. Sweet breakfasts, such as pancake, waffles, french toast, oatmeal or yoghurt parfait, and even cereals, topped with fresh fruits instead of syrup sugar
4. Sweet breakfasts, such as pancake, waffles, french toast, oatmeal or yoghurt parfait, and even cereals, topped with nuts
5. Sweet breakfasts, such as pancake, waffles, french toast, oatmeal or yoghurt parfait, and even cereals, topped with granola
6. Always add fruits before nuts or granola
7. Always add nuts before granola
Always use dairy-based milks alternatives, and avoid plant-based milks
9. Always use dairy-based yoghurts alternatives, and avoid plant-based yoghurts
10. Beverages like tea and coffee are preferred hot, not iceds
11. Use the espresso machine, if available, else pour over to make coffee
12. Serve beverages such as tea and coffee with milk next to it, but not mixed in
13. Have breakfast on the kitchen island
Persona 9:
1. Eggs scrambled or made into an omelette
Eggs with vegetables like bell peppers, onions, and mushrooms
3. Prefer side of fruits avocado with eggs
4. Toast toasted with spreads like avocado or hummus; instead of nut-based spreads
5. add butter on toast
Prefer herbal teas like peppermint or chamomile
Beverages such as tea or coffee should black, without milk or sugar
Avoid nuts and nut based spreads on toast
When adding milk to cereal, avoid nut-based milk, dairy-based alternative instead
10. When using yoghurt, avoid nut-based yoghurt
11. Add fruits as topping to sweet breakfasts, such as oatmeal, yoghurt parfait, pancakes, french toast, cereal
12. Serve items in a certain order, starting with the food first, then beverages
13. Eat breakfast at a standing desk or kitchen island
Persona 10:
1. When cooking eggs, prefer them scrambled or made into an omelette
Prefer sprinkle of salt on eggs
3. Prefer sprinkle of pepper on eggs
4. Prefer topping of cheese eggs
5. Prefer side of avocado with eggs
Prefer toast made by toasting bread in a toaster
Prefer bread with savory toppings like avocado, cheese or hummus
Prefer oatmeal cooked combination of water and non-dairy milk alternative
9. Prefer even sweet breakfasts, such yoghurt parfait, pancakes, oatmeal, cereal, french toast, to be topped with savory toppings like nuts, seeds, cheese or avocado
10. Prefer slight sprinkle of salt even on sweet breakfasts, such as yoghurt parfait, pan- cakes, oatmeal, cereal, french toast
11. Beverages like and coffee are preferred hot, not iced
12. Beverages like coffee and tea preferred is preferred black without any added sugars or creamers
13. Prefers teas and coffees with caffeine, instead of decaf options herbal teas
14. Prefers a squeeze of lemon in her tea
15. Prefers dairy-free cheese when topping eggs and toast
16. Prefers the use of stainless steel or cast iron cookware
17. Prefers dine on the patio
18. When preparing cereal, add cereal first then milk, in that order
20

---

Published as a conference paper at COLM 2025
Persona 11:
1. Prefer eggs made into an omelette, unless otherwise specified
Add in some veggies like bell peppers or spinach to eggs
My toast topped with nut butter or avocado spread
Coffee should strong and iced
Top pancakes or French toast with fresh fruit drizzle of honey
6. Yogurt must be topped with granola
7. Incorporate foods like nuts, and seeds in sweet breakfasts, such as french toast, cereal, yoghurt parfait, oatmeal, pancakes
8. Add melted cheese to eggs, it's casier digest and adds rich, creamy flavor
9. Serve beverages first before foods
10. When mixing cereal and milk, prefer to pour the cereal in first - it helps prevent spills
11. A side of fresh fruit or nuts with savory breakfast is appreciated to satisfy my sweet tooth
12. Serve breakfast at my desk or on the outdoor patio
Persona 12:
1. Prefer gluten-free cereals
2. Prefer gluten-free alternatives, when using bread for savory toast or french toast
3. Prefer lactose-free milk alternatives
4. Prefer toast without any spreads
Do not use butter on toast, when cooking eggs, etc. since is not lactose-free
Sweet breakfasts, such as yoghurt parfait, pancakes, oatmeal, cereal, french toast etc. are preferred without any sugar, syrups or spreads
7. Prefer yogurt alternatives made from non-dairy sources to avoid lactose
Prefer beverages such coffee served hot Prefer beverages such served hot
10. Prefer beverages such without milk and sugar
11. Prefer beverages such coffee without milk and sugar
12. Prefer cereal served either with lactose-free milk eaten dry
13. Prefer lactose-free alternatives, and adding cheese to breakfast dishes like eggs
14. Prefer beverages to be served first, followed by food
15. Prefer at least two vegetables like bell peppers, onions, or mushrooms as sides with eggs
16. Prefer to first pour milk then add cereal, rather than the other way around
17. Prefer to eat breakfast in his dining area or kitchen
Persona 13:
1. Eggs should be scrambled or over easy unless otherwise specified
2. Does not take caffeine, prefers decaf coffee
Does not take caffeine, prefers non-caffeinated teas
Prefer butter, and nothing else, on toast
Toast preferred side with any savory breakfast
6. No jams toast
No spreads toast except butter
8. Add just drizzle of honey, and nothing else, to sweet breakfasts, such yoghurt parfait, pancakes, oatmeal, cereal, french toast
No toppings (fruits, nuts, etc.) sweet breakfasts such yoghurt parfait, pancakes, oatmeal, cereal, french toast
10. Beverages served hot, not iced
11. Beverages served with hint of honey
12. Vegetables in omelettes or scrambled eggs
13. Simple dishes rather than fancy ones
14. Food served first, followed by beverages
15. Breakfast served in the backyard or on the patio
Persona 14:
21

---

Published as a conference paper at COLM 2025
1. Toast with sweet spreads like jams, marmalades, and nut butters
2. Add honey maple syrup to sweet breakfasts, like oatmeal, yoghurt parfait, pancakes, french toast, cereal
3. Add fresh fruit to sweet breakfasts, like french toast, cereal, yoghurt parfait, oatmeal, pancakes
Add nuts to sweet breakfasts, like french toast, cereal, yoghurt parfait, oatmeal, pancakes
5. Add granola yoghurt if available
6. Savory items served with side of sautéed vegetables hash browns
7. Beverages like tea and coffee served hot
8. Beverages like tea and coffee served with honey, when available else bit of sugar
9. Tea served with lemon
10. Melted cheese is a staple in her diet, especially when paired with eggs and veggies
11. Add at least two spices to savory breakfast dishes
12. Add herbs to savory breakfast dishes
13. Add side of sauteed vegetables to savory breakfast
14. Use cast-iron cookware for breakfast dishes
15. Use expensive glassware for breakfast
16. Prefers small dessert of fresh fruit and whipped cream with savory breakfasts
Persona 15:
Use whole-grain bread when making toast
Make toast without butter or spreads
Make coffee black, without milk or sweetner
4. Sugar-free pancakes, waffles or French toast
5. Add fresh fruits toppings sweet breakfasts, such pancake, waffles, french toast, oatmeal or yoghurt parfait
6. Prefer Plain yogurt over sweetened or flavored
7. Prefer berries the fruit topping with yoghurt cereal
8. Prefer eggs with side of vegetables
9. Add salt and pepper to eggs and vegetables
10. Prefer eggs and vegetables seasoned with no other spices except salt and pepper
11. Use non-stick pans when cooking
12. Use silicone spatulas when cooking
Persona 16:
1. When cooking savory ingredients, such as eggs vegetables, use at least two spices
2. Use fresh herbs if cooking savory dishes like eggs, meats and vegetables
3. When making eggs, scramble them or make an omelet
4. When cooking eggs serve with a side of toast When making toast use whole grain bread When making coffee, serve black or with plant-based milk, if available, else black
7. Mix oatmeal either only with water or with water and plant-based milk
Add honey or maple syrup to oatmeal or sweet breakfasts, such as oatmeal, yoghurt parfait, pancakes, french toast, cereal
9. Add fresh fruits, nuts, or seeds as toppings on sweet breakfasts, such as oatmeal, yoghurt parfait, pancakes, french toast, cereal
10. Add cinnamon or cardamom on sweet breakfasts, such as oatmeal, yoghurt parfait, pancakes, french toast, cereal
11. When using yoghurt prefer plant-based yoghurts
12. When cooking or sauteeing breakfast meats, eggs, vegetables, etc. prefer using olive oil
13. Serve tea without milk and sweetners
14. Plant-based alternatives are always preferred, so use plant-based milk with cereal
15. Use vegan cheese when making eggs, plant-based alternatives are always preferred
16. Serve beverages first, when making breakfast
17. Add milk first, then cereal, that order
18. Finally, serve food at dining location
22

---

Published as a conference paper at COLM 2025
D Model Prompts
D.1 Planner Prompt
The following is an example of the prompt the planner receives to predict the next action.
Example Planner Prompt
system Cutting Knowledge Date: December 2023
Source:
Today Date: 26 Jul 2824
provide ussistance nanner Person! wants for preparing and serving breakfasts such as naking cereal, pancakes, toast, waffles, french teast, coffee, tea, etc. You have the ability take the
You expert
planning.
following actions:
Open <X>; open an instance an articulsted furniture or object. t.g. Open cabinet Clase <x>: close instance of articulated furniture object. e.g. Close cabinet
Heat <t>; heat a container, which is located on heating appliance. e.g. Heat pan_ appliance. e-g.
on <b>; turn
stove_
Turn off <x>: turn off an appliance. e.g Turn off stave_l
Search <X>: search a container in the house. e.g. Search counter_8
Look for et>; Look for object in the whole house. Use this with generic category
name of an object, and not an object instance or phrase. Be sure to look for abjects one-at-a-tine. e.g. Search apple
Move <o to <Y>; nove an object x from wherever it currently is to the furniture or location e.g.
plate_@ table_2
Mix all itens in <x> get <Y>; nix itens that exist in container, typically to create a new entity. e.g. Mix all items bowl_0 to get cake_batte
Cook itens in <x>
<t>; cook sonething in
other such appliance
to create a cocked version of that entity. e.g. Cook itens in pan_å to get scrambled_eges get
chop itens entity. e.g. Chop apple_0 to get _oppped_apple
cutting board create chopped version of that
pour an entity
container
conteiner
2. e.g. Pour milk from milk_carton_@ to mug_a
Move <о> from <t> <z>: move content x container to another container Z. e.g. Hove apple from apple_bos_6 ta bowl_l
<x> itens in <t> to get <2>: freefarn actian to change object state, such as whisk, heat, blend, etc. e.g. whisk
pan_
custard, brew coffee_grounds
brewed_coffes etc.
object <t> get peel, crack, wash, wipe, etc. e.g. chop the object tomato_1 1 get chopyycccopeeettmaaeee
freeform action change object state,
chop,
chop the object onion_8 to get sliced_onion. crack the object get cracked_egg, etc.
Ask <x>; ask a freeforn question to the user to decide between multiple options and preferences. e.g. "Would like the eggs
adhere sunny side upp or scrambled?"
23

---

Published as a conference paper at COLM 2025
Example Planner Prompt
Declare Done: indicate that the task is complete. Make sure to use this exactly once the
For a given task, you will provide the next action required achieve given task. Before each step you will provide your e-g.
intention behind your action.
Thought: need find eggs prepare onelet
Action: Find eggs.
NOT repeat your
action.
must do the task in that the user prefers. Think of different variations, modifications, sides, etc. applicable to the given task, and do the task in way that you think the which you should extrapolate from when possible, and if you dan have any related information, you preferences. Think about whether multiple options are available for particular think creatively about toppings and sides that person night prefer. Do not ask for help in finding things; the user environment and where they night be located. Ask general questions to learn about the preferences which the one at hand.
Note that
would prefer. You might knew
things about the
only
their
ingredient,
know what objects exist the
useful in preparing future breakfasts in addi tion
You have prior information about user.
You will perforning tasks house with the following layout:
kitchen (kitchen)
stave_d (stove)
oven_ (oven)
microwave_d (nicrowave)
coffee_nschinge (a drip coffee machine) espresso machins) kettle_@ (an electric kettle)
espresso_nachines
toaster_0 (a classic 2-bread toaster) dishwasher_@ (dishwasher)
island_e (kitchen island next to countertop with two seats) island_cheir__ (bar
island)
island_choir_] (bar height chair next to kitchen island)
counter_@ (kitchen countertop
stave)
counter_1 (kitchen countertop next to the sink)
cabinet_@ (left kitchen cabinet above the counter) (right kitchen cabinet
counter)
cabinet_2 (tall kitchen cabinet)
cabinet_3 (kitchen
counter)
sink_1 (standard kitchen sink, centains water) fridge_0 (fridge)
drawer_@ (top kitchen drawer ta left of the stove).
drawer_1 (top kitchen drawer to right of the stave) spice_cabinet__ (snall cabinet for spice bottles and related items) dining_room (dining roon)
table_ (weoden dining table
chairs)
chair_1 (wooden dining chair)
chair_2 (wooden dining chair)
chair_3 (wooden dining chair) bedroom (bedroon)
(queen-sized bed with wooden frame)
dresser_0 (wooden dresser with six drawers)
nightstand_@ (wooden nightstand
drawers)
lamp_0 (bedside lamp with white shade)
desk_0 (work desk)
living_roon (living raan)
chair_ (recliner with a black leather upholstery) sofa_e (three-seater
leather upholstery)
EiFFee_tabbe_. (wonden coffee table with glass top)
tv_0 (flat-sereen
55-inch screen)
bathroon (bathroon)
toilet_@ (white porcelain toilet)
sink_0 (white porcelain sink with chrone faucet, contains water)
mirror_e e (large nirror with a wooden frame) shower_e (glass-enclesed patio (autdoor patio area) table_1 (small patio table)
showerhead)
What is the next action required to achieve the task: Prepare omelette for breakfast?
24

---

Published as a conference paper at COLM 2025
Example Planner Prompt
Source:
assistant
Action: Ask "Person1, would like any fillings in your onelette, such as vegetables, cheese, or meats. and 1f so, what would be your top choices?"
environment
Observation:
Source:
user
would like cheese my onelette.
Source:
assistant
Action: Ask "user, what type cheese would you prefer in your onelette, such as cheddar, mozzarella, feta, or something else?"
Source:
environment
Observation:
would prefer mozzarella cheese in my onelette.
Source:
assistant
Action: eggs
Source: environment
observation: Found egg-carton_e (carton of eggs) in/on fridge_0, eg8_6 (brown egs) in/on egg.carton_@, egg_5 (brown egg) in/on agg_carton_0, egg.3 (brown egg) in/on egg_carton_8, egg_2 (brown egg) egg_carton_e, egg_4 (brown egg) in/on egg_carton_@
Source:
You have made anything yet.
What is the next step to complete the task: Prepare omelette for breakfast?
Source:
Action:"
D.2 Reflection Prompt
The following is an example of the prompt the student receives to predict a candidate question the reflection step.
Example Reflection Prompt
Source: systen
You expert task planning, and guide rebot how provide assistance a nanner that user wants to Make cereal and coffee for breakfast.
difference between actually predicted. Mas there some knowledge about user's 's preferences, which if the about, would have predicted cannot predict that it should use almonds when making chia pudding. when it doesn 't know that user wants their chia pudding topped with almonds. If so. could ask user a question clarify their preference first, such on your sweet breakfasts, like chia pudding?' no question needs to be asked to predict expected action, you say 'Question: and do not provide any additianal information explanation."
Reflect
action
predicted,
expected action?NnExample:
"Question: What toppings you want
Answer
single
Source: assistant
Action: Look for cereal
25

---

not the avilability encourage the robat to search and explore different lscations. Even the robot nakes repeat Published as a conference paper at COLM 2025
Example Reflection Prompt
environment
Observation: Found cereal_boxe (a box of cocoa puffs, contains cocoa.puffs_ceres in/on cabinet_0, cereal_box_5 (a box classic corn flakes, contains forn_flakes_cereall in/on cabinet_@, cereal_box_@ (carton of fruit loaps, contains fruit__lops_cerral) in/an cabinet_@
Source: user
You have
anything yet.
What is the next step to complete the task: Make cereal and coffee for breakfast?
Source: assistant
Instead of Ask "What type of cereal would you prefer for breakfast: cocoa puffs, classic flakes, fruit loops?", the robot expected perforn coffee_nachine_e. 8. because if the robot knew user better, it would have thought that 'Since prefers his coffee to served should start by making coffee. To nake coffee, I need What question could the robot have asked user so that it could predict Turn on coffee_nachine_@ its next step?"
savor as awaits his meal, the coffee machine."
D.3 Persona Prompt
The following is an example of the prompt the persona model receives to answer the planner's questions.
Example Persona Prompt
Source: system
teaching household manner user would like. The robot may net know user 's preferences, so your jeb is to guide rabot to perforn
performing
assistive
user. sure guide robat only those dishes that the task calls for, e.g. if the task is make waffle do not ask the rebot to nake other things, such as coffee. Answer direct questions regarding preferences,
given
lecation of objects. In the latter case.
irreversible error, it's nistakes the next tine.
that
Given the current state of the house and what know about user and the task at hand, you will respond to the robot's 's last question concisely, and in first person, as if you
Environment
kitchen (kitchen)
stove_8 (stove)
oven_0 (oven)
microwave_@ (microwave)
(a drip coffee naching)
espresso_machine, nachine_ (an espresso nachins) espresso,
kettle_0 electric kettle)
toastar_0 (a classic 2-bread toaster) dishwasher__ (dishwasher)
island_e (kitchen island next countertop with two seats) islend_chair, (bar height chair next to kitchen island) island_chair_1 (bar height chair
kitchen island)
counter_@ (kitchen countertop next to the stave)
oil_bottle__ (bottle of olive sil,
olive_oil)
salt_box_a (box of iodized salt, contains salt)
knife_block. (woode knife block with space for knives) knife_3 (bread knife with serrated edge)
counter_1 (kitchen countertop next to the sink)
bread_e sliced loaf white sandwich bread, cantains white_bread_slice
bbrea__1 (a loaf of sprouted whole grain bread, contains wtteeeerrra.bbdddsllii..e
bbread_2 sliced loaf
walnut
banana.bread_slice)
banana_8 (a ripe banana)
apple_8 honeyerisp apple)
cabinet_0 (left kitchen cabinet above the counter)
26

---

Published as a conference paper at COLM 2025
Example Persona Prompt
sereal_box_@ (carton of fruit loops, contains fruit__oops_cereal
tereal_box_1 (carton
wheat shreds, contains whole_wheat__hhreb_.co
cereal_box_4 (a box of cocoa puffs, contains socoa-puffs_cerent
cereal_box_5 (a box classic corn flakes, contains forn_flakes_cerea
pancake_nix_0 bag of classic pancake
contains pancake_mix)
pancake_mix_1 bag of low-sugar
stweciillly-seeeeedd pancake mix, contains
lov_caloriee.annaae.nttt
waffle__ix__ (a bag of classic waffle nix, contains waffle_nix)
oatmeal_0 (packet overnight oats, contains overnight_.ats)
oatmeal_1 (box of Quaker 1-minute instant eats, contains instant_oatt)
oatmeal_2 (plain rolled oats, contains plain_rolled_oats)
(smull bottle of white sugar, cantains white_sugar)
sugar_1 (packet of brown cane sugar, contains broun_cane_sugar)
sugar_2 (small bottle of loose calorie-free
contains stevia)
honey_8 (bottle of honey, contains honey)
tea_bags_0 (box of green tea bags, contains green_tea_bag)
tea_bags_1 (bex of herbal peppernint tea bags, contains peppermint_tea_bar
tea_bags_2 (box of chanonile tea, centains chamomile_tea.bar
tea_bags_3 (box of earl grey
bags, contains cari_grey_tea_bang
tea_bags_4 (box of english breakfast tea bags, contains ogglish.breakfast.eeo..gg
coffee_0 (bag regular ground coffee,
coffee_grounds)
coffee_ (bag of decaf ground coffee. contains decaf_ sece__coffee_grounde
coffee_2 (jar
instant coffee, contains instant.coffee.powl
beans_8 (can of black beans, contains black_beans)
beans_ (can of pinto beans, contains pinto_beans)
brans_2 (can of garbanzo
contains garbanzo_beans)
pasta_ (box of classic penne pasta, contains penne_pasta)
pasta_ of whole
spaghetti pasta, contains spaghetti_pasta)
pasta_2 (box of lentil pasta, contains lentil_pasta)
alnands_0 (pack raw almands, contains raw_almands)
simands_1 (pack of reasted and salted almonds, contains salted_slmonds)
pistachias_@ (pack of roasted and salted pistachios, contains salted_pistachios
pistachios_1 (pack of chilli line coated shelled pistachios,
salted_
wainuts_@ (pack raw wainuts. contains rem_walnuts)
punpkin_seeds_0 (pack of punpkin seeds, contains pumpkin_seeds)
pine_nuts_@ (pack pine nuts, contains pine_nuts) cranberries_0 (pack
dry cranberries, contains dry_eranberries) figs_e (pack of dry figs, contains dry_figs)
(pack of raisins, pecans_0 (pack of raw pecans, contains raw_pecans)
cabinet_1 (right kitchen cabinet
counter)
bowl_8 (microwaveable glass boul)
bowl_2 (microwaveable glass bowl)
bowl_3 (microwaveable glass bowl)
bowl_4 (white ceramic salad bowl)
bowl_9 (fancy porcelain china bowl with pink flowers)
bowl_19 (small steel bowl)
bowl_11 (small steel bowl)
plate_@ (flat white ceramic plate)
plate__ (deep white ceramic plate)
plate_5 (deap white ceranic plate)
plate_6 (deep white ceramic plate)
plate_8 (fancy porcelain
flowers)
cup_8 (Fancy porcelain teacup with saucer)
cup_1 (fancy porcelain teacup
saucer)
cup_3 (simple white ceramic cup)
mug_3 (plain white nug)
french_press_e glass french press)
pour_ wwwr__oee_.oofeeenkee.
(a chenex pour over coffee maker)
cabinet_2
cabinet)
glass_0 (tall glass for drinking water)
glass_? (tall glass
drinking water)
glass_3 (tall glass for drinking water)
glass_4 (short Fancy cocktsil glass)
wine_glass_0 (standard wine glass)
wine_glass_1 (standard wine glass)
wine_glass_3 (standard
glass)
wine_glass_4 (tall chanpagne flute)
wine_glass__ (tall champagne flute)
cabinet_3 (kitchen cabinet below the counter)
pan_a (classic non non-stick pan)
pan_1 (nettl pan)
pot_0 (small aluminium pot)
p (large
pot)
27

---

Published as a conference paper at COLM 2025
Example Persona Prompt
cutting.board_.0 (plastic cutting board)
cutting_hoard_1 (wooden cutting beard)
tray_0 (serving tray)
box_1 (plastic contianer for leftovers)
box_2 (small nierowavesble glass contianer for leftovers)
box_3 (large microwaveoble glass contianer for leftovers)
sink_1 (standard kitchen sink, contains water)
sponge_ e (sponge for cleaning) soap, contains dish_soap)
soap_ (bottle fridge_0 (fridge)
water_bottle, (bottle still water. conteins water) with
ice_tray_@ (ice
contains ise_cube)
milk_e (carton of whole dairy milk, contains whole_nilk)
milk_3 (bottle of almond milk, contains almond_nilk)
milk_4 (jug of skin dairy nilk, contains skin_dairy_nilk
yoghurt_0 (pack of plain non-fat greek yoghurt, contains yoghury)
yoghurt_2 (cup of vegan cashew-based yoghurt, contains vegan_yoghurt)
egs_carton_@ (carton of 12 eggs)
egg_0 (brown egg)
egg-1 (brown egg)
egg_5 (brown egg)
egg_7 (brown egg)
liquid_egg_0 (a box vegan liquid egg substitute, contains rean.egg..uustiitur
butter_e (block of butter. contains butter)
butter_1 (box of vegan butter, contains vegan_butter)
cheese_8 (slices af
american_chesse_lleee
cheese_1 (block of part skin mozzarella cheese, contains mozzarella_cheese) tonato)
tonato_@
bell_pepper__ (red bell pepper)
anian_e (yellow onion)
potato_@ (sweet potato)
jalepeno_ (spicy jalepeno peppers)
lettuce_0 (head of iceberg lettuce)
strawberries_box. (a pint-sized box of organic strawberries, contains strawberry) navel_orange)
oranges. avocada_ 0 (a ripe avacado)
tranges_bag___ bag
xed_berriis__ (a bag
herries, contains frazen_berry)
_aos_broows__ bag of ready-to-cook frazen hash browns, cantains hash_brown) hunmus_8 8 (hox humnus, contains hunmus)
salsa_0 of
contains salsa)
cheese_dip_e (Jar of cheese dip, contains cheese_dip)
mayonnaise.J (jar mayonnsise, contains mayenneise)
jam_1 (jar of strawberry jam, contains strawberry-jan)
spread_8 (jar orange marnalade, contains orange_narmalade) spread_2 (jar of peanut butter spread, contains peanut_butter_speee
hot_sauce_0 (bottle habanero hot sauce, contains habanero_hot_sauce
hot__auce_1 (bottle
tabasco hot sauce, bacon_8 (pack of breakfast bacon. contains bacon_strip)
tabasco_hot_sauce)
parsley
of fresh parsley, contains fresh_parsley)
chives (box of fresh chives, contains fresh_chives)
basil (box
fresh basil, contains fresh_basil)
oregana (box of fresh oregane,
frash_oregano)
dill (box of fresh dill, contains fresh_dill) fresh
cilantro
fresh_cllantro)
Shaple_syrup_1 (small bottle of artificially sweetened low-calorie naple syrup,
contains low_calorie, www.calorie.male__syeeeee maple_syrup)
drawer_e (top kitchen drawer to left of the stove)
knife_e (simple netal butter knife)
knife_1 (simple netal buttar knife)
spoon__ (simple netal dinner spoon)
spoon_2 (simple
spoon)
Fork_2 (simple metal dinner fork)
fork_4 (simple
fork)
knife_7 (simple steak knife)
knife_8 (simple steak knife)
drawer_1 (top kitchen drawer to right of the stave)
ladle_9 (ladle to serve food)
serving_scoap_e (scoop serve food)
spatula_2 (plastic spatula)
spatula_3 (woaden spatula)
spice_cabinet__ (snall cabinet for spice bottles and related items)
pepper_0 8 (snall bottle pepper. contains pepper)
mixed_herbs_@ (snall battle of nixed herbs, contains nixed_herbs)
28

---

Published as a conference paper at COLM 2025
Example Persona Prompt
gerlic_powder__ (small bottle of garlic powder, contains garlic_powder) powder, contains onion_powder) cinnamon_@ (small bottle of cinnamon, contains cinnamon) nutmeg, contains nutneg)
anian_powder__ (small bottle
nutmeeg (snall bottle
clove_ê (small bottle of clove, contains clove)
dining_room (dining roan)
table_0 (wooden dining table with four chairs)
napkins_@ (set of four cloth napkins)
salt_shaker_e (table salt shaker, contains salt) [haperr_ (pepper shaker, contains pepper) chair_1 (wooden dining chair)
chair_2 (wooden dining chair) chair_3 (wooden dining chair) bedroom (bedroon)
bed_e (queen-sized bed with a wooden frame) dresser_0 (wooden dresser with six drawers) nightstand_ 8 (wooden nightstand with two drawers)
lamp_0 (bedside lamp with white shade)
desk_@ (work desk)
living_room (living roan)
chair_e (recliner with black leather upholstery) sofa_e (three-sead seater sofa with a brown leather upholstery) coffee_table_e (wooden coffee table with glass top)
tv_ (flat-screen TV with 55-inch screen) bathroon (bathroon)
toilet_ (white porcelain toilet)
sink_9 (white porcelain sink with a chrome faucet, contains water)
mirror_e (large nirror with
frane) shower with chrone showerhead)
shower_@ 8 (glass-enclese) patio (outdoor patio area)
table_ (small patio table)
Task: Make cereal and coffee for breakfast
User has the following preferences.
eggs vegetables, Use fresh herbs if cooking savory dishes like eggs, neats and vegetables when naking eggs, scramble then or nake an onelet. with When naking toast use whole grain bread naking coffee, serve Mix oatmeal either only with water with water and plant-basee milk Add honey or maple syrup to oatmeal or sweet breakfasts, such as oatmeal, yoghurt parfait, pancakes, french toost, cereal
cooking savory ingredients,
two spices
when cooking eggs
of
available,
Add fresh fruits, nuts, or seeds as toppings on sweet breakfasts, such as oatmeal, yoghurt parfalt, pancakes, french toast, Add cinnamon or cardamon on sweet breakfasts, such as oatmeal, yoghurt parfeit, pancakes,
toast,
when using yoghurt prefer plant-based yoghurts
when cooking or sauteeing breakfast meats, eggs, vegetables, etc. prefer using olive oil Serve without milk and Plant-based alternatives are always preferred, so use plant-based milk with cereal naking eggs, plant-based alternatives always preferred Serve beverages first, when making breakfast cereal,
vegan
Add milk first, Finally, serve food at dining location
Look at the following interaction and provide a short answer to the rebot 's last question
user's preferences. user flexible their preference choice arbitrarily, but make sure to tell the robot that usually user is flexible. and okay with. sure consistent your previous
options they
feedback.
Source: rabot
Action: Look for cereal
Saurce: environnent
observation: Found cereal_box_4 (a of cocoa puffs, contains cocoa_puffs_cerral) in/on cabinet_0, cereal_box_5 (a box of classic corn flakes, contains corn_flakes_ceredd in/on cabinet_8. cereal_box_f (carton of fruit loops. contains frust_llops_cereal) in/an cabinet_@
Ask "What type of cereal would you prefer for breakfast: cocoa puffs, classic corn flakes, or fruit loops?"
29


Reviews:
[{"review_id": "pxvwiib95s", "reviewer": "Reviewer_tqzp", "summary": "## Empiricism, Data, and Evaluation \n- Table 1 is informative - showing the different results with error bars with different methodologies\n- Table 1 has satisfaction rate and num questions asked, it seems like a metric the paper would like to measure is also the increase in satisfaction rate per question, since the \"Always Ask\" LLM is considered \"question inefficient\". Analysis of the additional questions asked during Reflection-DPO indicate the model is learning which questions are more valuable to ask, and how to better formulate those questions.\n\n## Technological Impact \n- Including prompts, open sourcing code, and detailed descriptions of setup all help future researchers with experiment design by providing reference materials for reproduction.\n- Developing a reliable and well curated benchmark for preference adherence in long-horizon tasks is valuable to the research community. \n\n## Ambition, Vision, Forward-outlook Progress\n- The paper provides a benchmark which increases the complexity of task measured by increasing the length and complexity of interaction.\n\n## Understanding Depth, Principled Approach \n- It would be nice to see performance results from a wide array of SoTA LLM as agents on the ADAPT benchmark.\n- One criticism of the paper may be that the benchmark uses an LLM simulated user, but this seems to be acceptable and necessary for the proliferation and reproducibility of the benchmark. Real-user preferences were transcribed\n\n## Clarity, Honesty, and Trust \n- Applaud the authors for releasing code which can reproduce the results in the paper and run the benchmark.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Publishing new benchmark with coverage over a more complex set of long horizon tasks is a welcome contribution.\n- The paper adds to the library of work discussing how to systematically evaluate assistive agents. \n- Reflection-DPO description and methodology are informative and novel. Analysis of the additional questions asked during Reflection-DPO indicate the model is learning which questions are more valuable to ask, and how to better formulate those questions.", "reasons_to_reject": "- Authors may want to increase complexity or difficulty of the ADAPT benchmark. \n- Authors may want to make the ADAPT benchmark more relevant by adding more tasks in varying domains.\n- Authors may want to add more baseline SoTA model results on the ADAPT benchmark", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nfrSEBkTJo", "reviewer": "Reviewer_ojsc", "summary": "This paper proposes the ADAPT task, which involves adapting an agent’s action selection to user preferences by referencing dialogue history. It also introduces Reflection-DPO, which updates the agent’s behavior to reflect user preferences. The results show that the policy trained with Reflection-DPO can implicitly adapt to user preferences using only dialogue history.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Overall, the paper was a unique and well-evaluated contribution. \nThe design of the ADAPT task is interesting.", "reasons_to_reject": "There are some lacks of references.\nThere are some questions about data construction and implementation details.", "questions_to_authors": "There are some areas where the related work and technical details could be expanded or clarified. \n\nThe design of the ADAPT task is interesting. However, the paper lacks an adequate review and contrast with prior work. For example, approaches that adapt dialogue policies based on the focus of user utterances have existed for quite some time. Additionally, frameworks have been proposed in the context of agent action selection in which appropriate actions are chosen based on dialogue history or surrounding context, especially in the absence of explicit user requests.\n\nThe ADAPT dataset is based on 16 manually created personas. It would be helpful to include a discussion of how these personas were selected. Presumably, they were designed to cover a diverse range of user types, but is there any rationale for selecting exactly 16 personas? Is there any justification that this number is sufficient?\n\nI also have a question about the teacher model. Is the teacher model the same as the user simulator, or is it simply a language model with access to the gold persona? A naive implementation suggests it might be a dialogue model that includes the user simulator. Could the authors clarify this point?\n\nRegarding the ε hyperparameter, were there any methods to avoid data leakage or heuristics when setting its value?\n\nWhen evaluating the usefulness of questions, one could consider using action sampling results. Although this may increase training time, it could allow evaluation without the influence of the hyperparameters.\n\nReferences\n\nYoshino, Koichiro, and Tatsuya Kawahara. \"Conversational system for information navigation based on POMDP with user focus tracking.\" Computer Speech & Language 34.1 (2015): 275-291.\n\nTanaka, Shohei, et al. \"Do as I Demand, Not as I Say: A Dataset for Developing a Reflective Life-Support Robot.\" IEEE Access 12 (2024): 11774-11784.", "extra_scores": {}}, {"review_id": "rq6FQNvHOG", "reviewer": "Reviewer_bZvY", "summary": "The paper is about adapting language models for any tasks by training it to actively ask preference questions. While this has been studied in previous research for different contexts (behavior adaptation, task oriented dialog, information seeking questioning), paper contributes (1) designed a benchmark to evaluate agent's capability to adhere to preferences (2) introduced reflection-DPO to produce \"reflection\" data  to fine-tune a student model to mimic a privileged teacher model. Proposed method is compared to the performance of existing methods (zero shot cot baseline, ReAct, star-gate (most related).", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Paper presents an interesting approach to adapting agents behavior. The choice of using preference and reflection-DPO (at steps) are interesting and novel to me.\n     Strategies to obtain preference data are interesting.\n- Experiment analysis and results shows the method works better (less questions asked, better preference satisfaction rate.\n- Compared with various baselines (basic zero-shot cot, or adapted ones).\n- Paper writing is clear and easy to understand.", "reasons_to_reject": "na", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a benchmark ADAPT where a student LLM agent needs to jointly work with a teacher LLM agent, which has privileged information in the form of ground-truth preferences. The student agent must actively ask questions to learn the preferences of the teacher and act accordingly. The paper also proposes an algorithm, Reflection DPO, that uses preference data generated via students' active questioning and uses it to fine-tune the student. Results in Table 1 show that ReflectionDPO achieves good accuracy while not asking a lot of questions.

Overall, all reviewers found this paper to be good and recommended acceptance. Reviewers overall found the paper to be interesting. One potential issue can be that the baseline that always asks questions does 8% better than ReflectionDPO and may still be preferred when the number of questions asked is less important than quality. It would have been a much stronger result if ReflectionDPO could match its performance while keeping the number of questions lower.

However, I think this is an interesting benchmark and the field will benefit from these results, so I am still recommending acceptance.
Decision: Accept