APRICOT: Active Preference Learning and Constraint-Aware Task Planning with LLMs
Authors: Huaxiaoyue Wang
Nathaniel Chin
Gonzalo Gonzalez-Pumariega
Xiangwan Sun
Neha Sunkara
Maximus Adrian Pace
Jeannette Bohg
Sanjiban Choudhury
Venue: CoRL 2024 (2024)
Keywords: Active Preference Learning
Task Planning
Large Language Models

Abstract: Home robots performing personalized tasks must adeptly balance user preferences with environmental affordances.
We focus on organization tasks within constrained spaces, such as arranging items into a refrigerator, where preferences for placement collide with physical limitations.
The robot must infer user preferences based on a small set of demonstrations, which is easier for users to provide than extensively defining all their requirements.
While recent works use Large Language Models (LLMs) to learn preferences from user demonstrations, they encounter two fundamental challenges.
First, there is inherent ambiguity in interpreting user actions, as multiple preferences can often explain a single observed behavior.
Second, not all user preferences are practically feasible due to geometric constraints in the environment.
To address these challenges, we introduce APRICOT, a novel approach that merges LLM-based Bayesian active preference learning with constraint-aware task planning. 
APRICOT refines its generated preferences by actively querying the user and dynamically adapts its plan to respect environmental constraints.
We evaluate APRICOT on a dataset of diverse organization tasks and demonstrate its effectiveness in real-world scenarios, showing significant improvements in both preference satisfaction and plan feasibility.

APRICOT: Active Preference Learning and Constraint-Aware Task Planning with LLMs
Huaxiaoyue Wang', Nathaniel Chin', Gonzalo Gonzalez--umarrega' Xiangwan Sun', Neha Sunkara', Maximus Adrian Pace Jeannette Bohg', Sanjiban Choudhury? *Cornell University, Stannford University
Abstract: Home robots performing personalized tasks must adeptly balance user preferences with environmental affordances. We focus on organization tasks within constrained spaces, such as arranging items into a refrigerator, where pref- erences for placement collide with physical limitations. The robot must infer user preferences based on small set of demonstrations, which is easier for users provide than extensively defining all their requirements. While recent works use Large Language Models (LLMs) to learn preferences from user demonstrations, they encounter two fundamental challenges. First, there is inherent ambiguity in interpreting user actions, as multiple preferences can often explain single ob- served behavior. Second, not all user preferences are practically feasible due to ge- ometric constraints the environment. To address these challenges, we introduce APRICOT, novel approach that merges LLM- Bayesian active preference learning with constraint-award task planning. APRICOT refines its generated pref- erences by actively querying the user and dynamically adapts its plan respect environmental constraints. We evaluate APRICOT on dataset of diverse orga- nization tasks and demonstrate its effectiveness in real-world scenarios, showing significant improvements in both preference satisfaction and plan feasibility. The project website ttttps://portal-eee
Keywords: Active Preference Learning, Task Planning, Large Language Models
Introduction
For robots to perform personalized household tasks, they need to balance a user's preference with constraints the user's household. Concretely, organizational tasks, such as putting away grocery items into a refrigerator, require the robot place items based on where the user prefers and avoid colliding with the fridge or knocking other items over.
Large Language Models (LLMs) provide an effective interface for users to communicate what they want via natural language [1, 2, 3], but carefully articulating the task and their specific preferences can also become tedious. Conversely, strations of the task, from which Vision- LaiiuunL Models (VLMs) can extract relevant states and actions. Thus, recent works [4, 6, have studied using LLMs infer user preferences from small set of such demonstrations. However, current approaches face two challenges. First, multi- ple preferences can consistently explain user behaviors in demonstrations. Randomly choosing one preference to generate a plan will fail to solve unseen initial conditions. Second, environmental constraints can invalidate some placement locations that satisfy the preference. Naively converting a preference to placement locations can cause the robot to collide with objects in the environment.
often more straightforward for users to provide demon-
Our key insight is close the loop LLMs, enabling them to refine preferences and plans through efficient interactions with both the user and the environment. We combine the generative
Correspondence to: Huaxiaoyue Wang. yukiwang Bolkiwang@es.comme .os.cooellleddu
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

Visual Demonstrations
Active Preference Learning
%
N Candidate Preferences as Natural Language
Query the User Optimal Questions
1. 'Vegetables middle shelf right top shelf. Drinks can be on..."
Bayes.
Perception Medule
Demos I'n Text
Framework
Do just reed vagetables be placed togotiner?
Terminate?
70
N.1 want vegetables to placed together. the right side the fridge...
anywhere
Best Preferance
Task Planner
påck (pppp11")
place 10.9, 9, 0.33F1
* mant vegetables to just
be placed together. Drinkk side
Feedback
World Model
fridge...
Planner
Initial State fu
Task T7
Objs to put tway:
pickandplace "pepoi", *right side middle sheir" 10.9, 0.3, 0,1611 picbandplace ("piceapppe"" shele",
X
0.9511,
Geometrically Feasible
Replanning
Preference Satisfied
Overview of APRICOT that (1) converts user visual demonstrations into anguuae-basee demon- strations, (2) given demonstrations, determines the preference that best approximates the ground-truth user preference by minimally querying the user, (3) generates and refines plan based world models' feedback satisfy preferences and respect constraints, (4) executes the plan a real robot system.
Figure 1:
question-asking capabilities of LLMs with Bayesian Active Learning [8] explicitly capture uncer- tainty in user preferences and ask informative questions that collapse uncertainty. We also combine the generative planning capabilities of LLMs with feedback from the world model to iteratively improve plans, ensuring they respect environmental constraints while satisfying user preferences.
We present APRICOT (Active Preference Learning with Constraint-Aww Task Planner) that uses: (1) VLM convert visual demonstrations into language-based demonstrations; (2) an LLM-based Bayesian active preference learning module ask the user a small number of questions and identify the preference that most closely approximates the user's preference; (3) task planner that refines its robot task plan based on feedback from world models to satisfy preferences and respect environ- mental constraints; (4) a real robot system to execute the generated plan. Our contributions are:
A new algorithm for LLMM-bsed Bayesian active preference learning that can efficiently learn a preference from a small set demonstrations and minimal online user-querying.
A real robot system with a constraint-awarr task planner that can generate and execute plans that satisfy user preferences and respect environmental constraints.
Evaluations of (1) the active preference learning approach on benchmark dataset of 50 different ground-truth preferences and 100 test cases, and (2) real robot experiments on 9 scenarios.
2 Related Works
Active Preference Learning. Active preference learning focuses on estimating user preferences or reward functions by asking the user a set of queries. robotics, classical approaches focus on synthesizing the optimal comparative queries (e.g., pairs of trajectories/features) [9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19]. A subgroup of works [20. 21, 22, 23] explores using teleoperated demonstrations learn prior over reward functions expedite the process. In contrast, our work uses visual demonstrations. Instead of using linear combination of features, choose to represent preferences natural language because language can capture complex preferences. Prior works [5, 4] have also studied using LLMs to infer user preferences based on demonstrations, but they lack an interactive mechanism to refine incorrect preferences based on user feedback.
Recent works explore incorporating LLMs into the active preference learning framework [24, 25, 26, 27J. [28] uses Bayesian optimal experimental design [29, 30] to select the optimal comparative pair of feature vectors and uses LLMs to translate the selected pair into natural language questions,

---

but feature vectors will fail to scale to our setting where placement preferences are combinatorial. Meanwhile, [31] uses L.MSs to generate questions directly, but it relies solely on LLMs to pick ef- fective questions to ask. Our approach bridges the two paradigms by proposing candidate questions to ask via LLMs and selecting the best questions that maximize information gain. Another tangential work is [32], which uses conformal prediction to quantify uncertainty. but it does not focus on active learning, and it requires diverse calibration dataset, which is difficult to acquire for our setting.
LLMs for Planning. Work can be categorized as interactive task planners [33, 34, 3] that adapt the plan through user interactions, affordance planners [1, 35, 36] that generate feasible plans, and code planners [37, 4, 36, 2, 5] that generate plans as a sequence of function calls. Affordance planning is significant our work since we must generate geometrically feasible plans to respect environmental constraints. Prior works like [38] verify its LLM plans with an external tool solve structured planning domains. [35] verifies the feasibility of its LLM plan, which sequences robot skills, with learned Q-functions. Similar to these works, we use world model to verify geometric feasibility. However, our work differs from both because the LLM planner incorporates and reflects on the world model's feedback for geometric feasibility to improve the plan over time [39, 40).
3 Problem Formulation
We frame the fridge organization task as an MDP. State & E S is the set of objects and their locations. Action a E are high-level actions pi ckandplace (obj name, xyz.loc). Given a plan 5 = {so, ,T, ar)) the reward function (E, 0) evaluates whether object placements in È satisfy the preference 0. We represent these preferences as natural language, and 0 is the (latent) ground- truth preference. We compute the reward R(ç, 0) through an LLM that outputs the percentage of object placements in that satisfy the preference 0. We assume access to a world model, which computes geometric constraints C(g) = 0/1 based on whether object placements cause collisions.
Given an initial condition 8o (e.g., objects in the fridge), a task T (e.g., a set of objects to place), a preference prior P(0). the goal is find a feasible plan that maximizes the expected reward:
arg max Eo- R(E, 0) s.t. C(E)= = 00
&
To learn the prior P(0), we leverage small set of demonstrations D from the user similar to prior works [20, 21, 22, 23]. Given a plan &D E DD we model the prior as Boltzmann distribution P(O[kD) & exp(R(ED, 0)). Intuitively, preferences 0 sampled from P(0) are consistent with the demonstrations, i.c. rewards given these preferences are high: R(ED. 0) = 1. Hence, given demon- strations D, we construct a preference prior P(0 = IkLe P(O(s)). We make an important assumption that this prior has support over the ground truth preference 0"
One challenge is that demonstrations alone may not be sufficient to construct a good prior that helps solve (1). For example, multiple candidate preferences can explain the same demonstrations D, but given a new initial condition unseen in the demonstrations, there may not exist a plan & where RIC 0) is high for all preferences 0 ~ P)))D). Thus, we need an approach that can actively collapse uncertainty in P(0) until a solution ş to (1), which has sufficiently high rewards given all plausible preferences, has been found. Another challenge is that if the LLM is directly used open-loop to generate a plan, may not satisfy the constraints. We address both of these in the next section.
4 Approach
We introduce APRICOT, a three-stage approach, to tackle the problem of (1) learning user preferences from visual user demonstrations and a minimum number of queries (2) generating a plan that satisfies preferences and respects environmental constraints (3) executing the plan with a real-robot system.
4.1 Active Preference Learning
We present APRICOT's LLM-based Bayesian active preference learning module, which takes as input visual user demonstrations and converts them to language-based demonstrations D via VLM
We can relax this to prior has support over preference i that is value cquivalent to the ground truth preference 0", i.e., the optimal plan E" according to 0 is also optimal according to 8"
3

---

Plae Preference
(4.1.3) Select Optimal Question
Fruits the botton top sheff Drinis must on...
4 pickeropi hagekkco"" cap stal en) plakaripl p"pppi
Do prefer fruits be specifi
(4.1.2) Terminate? RPOD)
the right dide of middle shef fall, do drinks be placed
S
Candidate Preforences
Cardilate
Plans
Initialize
Update
State
P(D) P(O 00000"000=0000000000000000000000
argmax MiPlo) E.Hii
P(O)
Oljjccs to pet away
(4.1.1) Propose Candidates
<*: Do you prefer ...?
Ask the Optimal Question
Figure 2: LLM--Based Bayesian Active Preference Learning Approach. Given a set of Linguage-ased demonstrations, APRICOT (1) proposes candidate preferences and corresponding candidate plans, (2) deter- mines whether terminate by evaluating whether the prior over candidate preferences P(O) is sufficient, (3) select the optimal question that maximizes information gain before updating its prior based on user answers.
(Section 4.3). APRICOT outputs preference that approximates the ground-truth user preference. Because user demonstrations alone cannot construct a sufficient preference prior (Section 3), the goal is to reduce uncertainty in this prior P(0) by asking the user small number of questions. This module contains three key components: (1) Propose Candidate Preferences, which generates candidates based on user demonstrations, (2) Determine whether Query, which determines whether the module has sufficiently updated the prior, (3) Select and Ask the Optimal Question, which selects the optimal question to reduce uncertainty and updates the prior given user answers.
4.1.1 Propose Candidate Preferences
Directly maintaining the preference prior P(0) as a continuous density function is challenging be- cause preferences are represented as frre-form natural language. Instead, APRICOT maintains a set of N diverse candidate preferences 0 ~ P(O given a small set of user demonstrations D. Con- cretely, we construct an LLM prompt that keeps sampling until getting preferences {0.) such that they are all consistent with the demonstrations, i.e. R(ED, 0,) = 1, vED E DD 1,..., N. Having sampled N preferences, we set the prior as P(O,) = ./- With this initial prior, given new initial condition 8n unseen in the demonstrations and task T, there is no guarantee that there exists a plan & that satisfies all preferences, i.c., R(E, 0.) = 1. This necessitates actively querying the user.
4.1.2 Determine Whether To Query
Given the current prior over candidate preferences P(0), APRICOT determines whether it needs to ask more questions to reduce uncertainty or the current prior is sufficient to solve (1), i.e., there exists a plan & that has sufficiently high rewards given all preferences [0.]
To do so, we first construct library of candidate plans. Given an initial condition so and task T, for each candidate 8,, we use the task planner (Section 4.2) to generate plan & that maximizes RR,, 0,). We accumulate N plans in total =\ ( corresponding to the N preferences (0, j2 We also define the disadvantage function DIS DISADV (E,- 0z), which quantifies how bad a plan E, is compared to any other plan in the library =\t for a given preference 0;:
DISADV 0,) = max R(g 0,) RR(E, 0.).
(2)
EEE
Then, the condition on whether to terminate querying S(P(0)) checks the following: does there exists a plan & whose expected disadvantage across all candidate preferences is below a threshold e,
N
S(P(0)) E s.t Σ P(O.)DISADV((.0) S.
(3)
Intuitively, APRICOT will terminate querying the user either (1) when a plan & has low disadvantages across all candidate preferences or (2) when the probability of candidate preferences that give a high disadvantage to & approaches O. Once the terminating condition f(P()) is true, we get the best plan & that satisfies (3) and, by proxy, the corresponding preference 0: that generates the plan. Appendix 1.4 prove that APRICOT's performance is bounded given this terminating condition.

---

4.1.3 Select and Ask Optimal Query
To collapse uncertainty in P(0), APRICOT asks the user question 9 and receives an answer o E O that provides additional information about latent user preferences beyond the demonstrations. ef- ficiently query the user with informative questions, we choose questions q" that greedily maximizes information gain [29, 30]: g + arg max, H [[O)] E. H [ q, o)]. Unlike the typical Bayesian approach that relies on a fixed corpus of questions, we use an LLM to generate question set Q.
For APRICOT, we assume that the robot will only ask yes-or-no questions, user answers are defined as o E {yes, no). We use an LLM to generate M questions for each preference pair (6,, 8y), which we accumulate to construct Q. Empirically, the LLM often asks about specific aspects that are different between the pair instead of directly asking which preference is better (Fig. 4).
Given candidate question set Q, can find the optimal question q" by solving the equation:
N
9" arg max
H[P(0)]
Σ
P, (.)H)(((())())))))))))
(4)
OE [yes,no)
To calculate the likelihood P(olg, 0z), we use the Bradley- Terry Model [41], where the score the logit of a L.M prokelay (-lg, 0.) that pretends be user with preference 0g answering the question q-
Thus, Pllg 0,) o (Poollepay (o yes|g, 0,) pmoleplay (o no/g. 0,)). The posterior P(O.lg. o) P(.)) the normalized product of the likelihood and prior. Once the user answers to the P(o.g)
optimal question q", we update the prior with the posterior p(0]q". o) and re-determine whether to terminate. Hyperparameters, computation cost, and LLM prompts are in Appendix 7.1.
4.2 Preference and Geometric Constraint Aware Task Planner
Given an initial condition a task T, and preference 0, APRICOT's task planner must output plan & that maximizes the reward function R(E, 0.) and satisfies constraints C(G.) = 0 (Section 3).
Semantic Plan. We first prompt an LLM output a semantic plan, which is sequence of high-level semantic actions (e.g.. pickandplacce ("apricot", "top shelf")).
Geometric Plan Based on World Model. Given semantic plan, the task planner needs to ground it in geometrically feasible XYZ locations. We assume access to a world model, which, for the real-robot system, is represented as the 3D point cloud of the fridge and 3D bounding boxes for all objects. When calculating C(E,), the world model collision checks an object's XYZ placement location with the point cloud. The task planner performs beam search, where each node is an object's XYZ placement coordinate sampled based on the semantic plan's 's location. It selects the final & based on the beam with the highest reward R(, 0.) and lowest constraint violation C(E.).
Reflect and Refine Based on Feedback. To refine the generated plan & we construct a Reflex- ion [40] style prompt that takes feedback both from the reward R(E,, 0,) and constraint violations C(G.). We prompt the task planner to regenerate until finds a feasible placement location for all objects or it exhausts its maximal refinement steps. See Appendix 7.2 for details.
4.3 Execution on Real Mobile Manipulator
We introduce two additional components to implement APRICOT in real-robotic system.
Perception. Given image, the perception system detects objects and determines semantic loca- tions for each object. We assume a list of all possible grocery items that can appear in fridge, static camera pointed at the fridge, and ArUco tags for shelf localization. We use an open- vocabvocyyylary object-detector Grounding-DDin DINO [42] that uses the prompt "grocery item" to draw bounding boxes on all of the objects. After filtering with Non-Maximum Suppression, CLIP similarity score [43] determines the most likely label for a cropped object image. The semantic location (e.g. "top shelf", "middle shelf") for each object determined by the pixel location of its bounding box center. The perception system is used to (1) parse visual user demonstration into before and after states for grounded, language-base based demonstration and (2) track the current state of the fridge in real time.
5

---

Execution Policy. A high-level action pi cickaddlacce<<bp)) <xyz loc>) is converted into sequence of low-level skills. (1) For pick (<obj>) and place (<xyz_loc>), we train a RL, policy a simple low-dimensional simulator that knows the robot's dynamic model. See details in Ap- pendix 7.3. (2) For move O), we use simple path planner. (3) ensure the system responsive to changes in the environment, we regenerate the plan & after completing each high-level action.
5
Experiments
1.00
Dand q/A
0.75
0.70
0.70 0,70
0.55
0.50
0.p
0.39
0.25
0.25
0.30
Speafii Location
Location
Position
Subcategory Excephion
10.0
8:05
7.5
0.15
Qveree
5.0
3,65
420
3.50
3.00
2,15
0.02
Specific Location
Location
Subcategory Exception
Specific Location: Evayshing must be placed at specis, location
eg Dalry con placed
APRICOT
Democ
Candilate
Quesion Candidatee
Boyesian Framewore
Queetics
unyynfore et top shiff
Bust Prefs
Relative Positions #
Fruits must be placed
CandeLIM-QU Q/A
Demoe
Cand
Pres
Ganeroted Quention
vegetablie
Exception: heaey fruits must
Q/A
Demoe
S
right ride bottom
Generated Preference
Interactive Approaches
Conditiona: ta.
Non-Interactive Approach
lelt side of top shell is full place drinks lift sids middle sheff
Non-Intezactive
Prefeuncs
Figure 3: Active Preference Learning Results on Benchmark Dataset. APRICOT achieves the highest pref- erence accuracy 58%, which the percentage outputted preferences that are equivalent the ground-truth preference, while asking the user the smallest amount of questions (2. on average).
5.1 Active Preference Learning Experiment
Benchmark Dataset. To evaluate our active preference learning approach, we programmatically generate a dataset of 100 test cases containing a ground truth preference, 2 demonstrations, and a test scenario. The preferences always refer to object categories (e.g., fruits, vegetables), and they are split into 5 groups, each with 20 cases. Fig. 3 shows examples and Appendex 8. 6.1.1 contains details.
Setup. We conduct the experiments in 2D version of the fridge environment. The task planner generates a plan & as a sequence XY placement locations, and a 2D world model computes C(g) by collision checking object assets against the environment. To simulate user, we use an LLM that can answer questions based on the ground- -ruth user preference. Details are in Appendix 8.1.2.
Baselines. We compare APRICOT against two categories of baselines: interactive and non- interactive ones. Fig. 3 shows visualizations of the differences between APRICOT and these base- lines. Compared to APRICOT's Bayesian framework, interactive baselines directly use LLMs to control the active preference learning process. LLM-Q/A relies on one LLM prompt to determine whether to stop asking questions, generate question, or generate the preference when termi- nates. Cand+LLM-Q/A splits the process into two steps, where it first generates candidate pref- erences similar APRICOT. It then uses another LLM to determine whether terminate, gener- ate question, or select the best preference from the candidates when it terminates. Mcanwhile, Non-Interactive [5] uses LLMs to directly summarize the demonstrations into a preference.
Metrics. The preference accuracy represents the percentage of preferences determined by an approach that are equivalent to the ground-truth preference. Practically, based on Section 3, pref- erence 0; is equivalent to the ground-truth preference 0* if its corresponding plan is optimal according to 6* such that R(E,, 0.) R(g,, 0*) 1. The number of queries quantifies the average number of questions approach asks the user, lower is better.

---

113-0/A
under doocc miais
Dar de C.NN- D/A
Luliove dull?
ACRICOE
del's
Fe die the niddl de
1000
De ae la ta ploat supeler
slds
Jasd the ug cide of bation
C Ho
Queries: 10
Queria Reschal)
Queriee:
* of Querics:
Generaed Peduarcc
Choan Prefeanes Cand
c frudts luft udd of the wille sheff bes space
G chaneiee iiht side
inat.
Vegetallic sia nul
p Dairy Productss side widile
Daiyy Products right edu al
3 Dairy Preducts right wibde sholl
Drekkr right sid wide
Briaka: laft luttas
O Deek lok oide batter shetl
Candirants: right batars shsf
Agh of
Figure 4: Example Queries From Each Approach, APRICOT correctly infers the trouh user preference with the least number of queries because it selects informative questions directly about the category with the most complex requirement. In contrast, LLM-Q/A A exhausts the number of queries, while Cand+LLX-Q// A termi- nates carly but infers the preference incorrectly. Preferences here are simplified as bullet points for readability.
Q1: What is the quality of preferences determined by APRICOT compared to base- lines? In Fig, 3, APRICUT overall achieves the highest preference accuracy of 58.0%. while Non -en-Interactiv performs the worst with only 35.0% accuracy, LLM-Q/A A with 39.0% accuracy, and Cand+LLM-Q/A with 43.0% accuracy. For easy test cases where demonstrations consistently show the same category at the same location, Non-Interactive performs equally well compared to the interactive baselines. However, as the demonstrations have more variability and the pref- erences become more complex, Interattrsssse''s performance worsens. This trend suggests that interactive approaches, which learn additional information by querying the user, are useful in uncovering complex preferences. We analyze APRICOT's failure cases in depth in Appendix 8.1.3.
Q2: How query-efficient is APRICOT compared to interactive baselines? Fig. 3 shows that APRICOT consistently requires 71.9% less queries compared to LLM-Q/A A and 46.25% less queries compared Cand+LLM-Q/A. A. Fig visualizes a qualitative example of each approach's behavior given the same demonstrations. LLM-Q/A A struggles to perform well because its LLM prompt is bur- dened with simultaneously reasoning about the entire active preference learning process and about what preference to generate at the end. Thus, LLM-Q/A A simply asks every possible preference for a category, causing it to exhaust the allocated number of queries. Meanwhile, Cand+LLM Q/ Aasks fewer queries compared to LLM-Q/A, A, but its L.M incorrectly interprets the user's answer "no" and chooses the wrong candidate. In contrast, APRICOT's maximum entropy reduction objective (4) helps it select the most informative question to reduce uncertainty in the candidate preferences, while its terminating condition (3) helps it determine when it can terminate without excessive querying.
5.2 Real robot experiments
Setup. The task planner is tested on 9 different scenarios, categorized into 3 difficulty levels. Fig- ure 5 shows examples of these scenarios. Each scenario is accompanied by 2 visual demonstrations, which we convert into danguage-based demonstrations and run APRICOT's 's active preference learning module to output a preference. We evaluate how well a task planning approach can generate a plan to satisfy this preference while respecting constraints. Details are in Appendix 8.2.1.
Baselines. The Feasibility planner disregards user preferences and focuses only on geometric feasibility. It uses beam search to find collision-free placement coordinates for each object. The Preference-one -nll planner focuses on satisfying user preferences without considering geometric constraints, and its semantic plan is directly converted to XYZ coordinates.
The percentage of feasible plans is the percentage of objects that have collision-fre free placement locations, which checks how well a planner is able to respect environmental constraints. Meanwhile, the percentage of preference satisfied is the percentage of objects that are placed at location that satisfies the preference outputted by APRICOT's s active preference learning module.
Metrics.
7

---

Paabb Aaramatts (S))
Initial State
2.0
0.96
333
0.4
0.2
Expected Final State
0.0
Easy
3.0
0.W.
0.8
0.6
0,21
auccc 2:00:1 per Easy
oble fillge cble to awyy
aigg dee so
Medium
Hard
Easy
Peasibility Eoyy
APRICOT
Preference-Coue -all
Preference
"I uaut srgitation scad edkiing condinents Diaia oepobire keg an they the die bettur shalt
Fezsible Plan:
Preference Sotisfied
Figure 5: Task Planner Results Real-Robot Scenarios. Evaluated on 9 scenarios with 3 difficulty levels. The qualitative example below APRICOT generating plan that satisfies preferences and respect constraints
Q3: How does APRICOT balance satisfying preference and respecting constraints? In Fig. 5, even as the fridge space becomes more constrained, APRICOT is able to maintain a high percentage of feasible plan (96.0% for the Hard case) and preference satisfied (89.0%). Fig. also shows qualitative example of different planners' behavior. For this scenario, Feasibility-Only ignores the preference and just focuses placing everything in the fridge. Preference-Only incorrectly assumes that the bottom shelf can fit both the red bell pepper and the pincapple, thereby causing the bell pepper to not receive feasible placement location. In contrast, APRICOT initially makes the same incorrect assumption, but it is able to improve based on the world model's feedback and adjust its plan to place the red bell pepper on the right side of the top shelf. Appendix 8.2.2 shows detailed traces of APRICOT planner's improvements as it receives feedhack.
Q5: How does APRICOT regenerate plan to account for changes in the environment? Fig. 6 shows that, because of its closed-loop planning, APRICOT can adapt its plan to environmental changes and continue satisfying user pref- erences. See Appendix 8.2.3 for other examples.
Inéia State
Intiaal Plan
pickandplees ("cohe"t *Light side
pactrandpa Diganaady
packandplace ("Eantx", " abele")
Robot Placnui "coke"
User moved drinka
Sussealully
Discussion and Limitation
We explore the problem of solving personalized or- ganizational task with environmental constraints. We present APRICOT, a three-stage approach that learns the user's preference based on small set of demonstrations and minimal online querying, generates robot task plan that accounts for the learned preference and geometric constraints, and executes that plan on a real-robot sys- tem. Limitations include: (I) Active preference learn- ing assumes one of the candidate preferences is equiva- lent to the ground-truth preference. Future work ex- plore dynamically expanding prior based on the question- answering and the initial demonstrations. (2) LLM task planners cannot guarantee satisfying hard constraints after our beam search and reflection steps. Future work will embed our L.M planner as fast heuristics within a complete
Figure 6: APRICOT Adapts to Changes in Environments. The user's preference re- quires drinks to be placed together. After the user disrupts the seene and moves the drinks the right side of top shelf, APRICOT's closed-loop planner is able adjust its plan and properly satisfy the user's preference in the newly changed environments.
planner.

---

Acknowledgments
This work was supported in part by the National Science Foundation FRR (#2327973). Sanjiban Choudhury is supported in part by the Google Faculty Research Award and the OpenAl Superalign- ment Grant. We thank Marion Lepert and Jimmy Wu for discussing ideas for this project in its early stages.
References
[1] M. Ahn, A. Brohan, N. Brown, Y. Chebotar, O. Cortes, B. David, C. Finn, C. Fu, K. Gopalakr- ishman, K. Hausman, A. Herzog, D. Ho, J. Hsu, J. Ibarz, B. Ichter, A. Irpan, E. Jang, R. J. Ruano, K. Jeffrey, Jesmonth, N. Joshi, R. Julian, D. Kalashnikov, Y. Kuang, K.-H. Lee, S. Levine, Y. Lu, L. Luu, C. Parada, P. Pastor, J. Quiambao, K. Rao, Rettinghouse, D. Reyes, Sermanet, N. Sievers, C. Tan, A. Toshev, V. Vanhoucke, F. Xia, T. Xiao, P. Xu, S. Xu, M. Yan, and A. Zeng. Do as can, not as say: Grounding language in robotic affordances, 2022.
[2] J. Liang, W. Huang, F. Xia, P. Xu, K. Hausman, B. Ichter, P. Florence, and A. Zeng. Code as policies: Language model programs for embodied control, 2023.
[3] B. Li, P. Wu, P. Abbeel, and J. Malik. Interactive task planning with language models, 2023.
[4] H. Wang, G. Gonzalez-Pumariee Y. Sharma, and S. Choudhury. Demo2code: From summa- rizing demonstrations to synthesizing code via extended of-thuht,, 2023.
[5] J. Wu, R. Antonova, A. Kan, M. Lepert, A. Zeng, S. Song, J. Bohg, S. Rusinkiewicz, and T. Funkhouser. Tidybot: personalized robot assistance with large language mod- els. Autonomous Robots, 4788):108711122, Nov. 2023. ISSN 1573-7527. doi:10.1007/ $10514-023-10139-- URL http:://dd..dd..org/00.000000 0
[6] G. Gao, A. Taymanov, E. Salinas, Minciro, and D. Misra. Aligning Ilm agents by learning latent preference from user edits, 2024.
[7] D. Han, McInroe, A. Jelley, Albrecht, P. Bell, and Storkey. ..mm-personaliz:: Aligning IIm planners with human preferences via reinforced self-training for housekeeping robots, 2024.
[8] D. Golovin, A. Krause, and D. Ray. Near-optimal bayesian active learning with noisy obser- vations, 2013.
[9] R. Akrour, M. Schoenaue, and M. Sehag. April: Active preference learning- based reinforce- ment learning, 2012.
[10] D. Sadigh, A. D. Dragan, S. Sastry, and S. A. Seshia. Active preference-based learning of reward functions. 2017.
[11] E. Biyik and D. Sadigh. Batch active preference-biie based learning of reward functions. In A. Bil- lard, A. Dragan, J. Peters, and J. Morimoto, editors, Proceedings of The 2nd Conference on Robot Learning, volume 87 of Proceedings of Machine Learning Research, pages 519-528.
PMLR, 29-31 Oct 2018.
[12] C. Basu, E. Bryik, Z. He, M. Singhal, and D. Sadigh. Active learning of reward dynamics from hierarchical queries. In 2019 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 120-127. IEEE, 2019.
[13] A. Wilson, A. Fem,, and P. Tadepalli. A bayesian approach for policy learning from trajectory preference queries.
Pereira, Burges, Bottou, and K. Weinberger, editors, Advances in Neural Information Processing Systems, volume 25. Curran Asso- ciates, Inc., 2012. URL https: https: https://proceeding..nttt neurips.cc ppaperffies//pppll 2212::/iwhee2222119ee00ssseeeee2

---

[14] M. Katz, A. C. Le Bihan, and M. J. Kochenderfer. Learning an urban air mobility encounter model from expert preferences. In 2019 IEEE/AIAA 38th Digital Avionics Systems Conference (DASC), pages 1-8. IEEE, 2019.
[15] N. Wilde, A. Blidaru, S. L. Smith, and D. Kulić. Improving user specifications for robot behavior through active preference learning: Framework and evaluation. The International Journal of Robotics Research, 396))651 2020. doi: 161:1001177077788549900009000000 https://dos http::://oo.oor//1... 1177 117777007783644209100000
URL
[16] N. Wilde, D. Kulic, and S. L. Smith. Active preference learning using maximum regret, 2020.
[17] C. Basu, M. Singhal, and A. D. Dragan. Learning from richer human guidance: Augment-
ing comparison-barn learning with feature queries. Proceedings of the 2018 ACM/IEEE International Conference on Slaman-Robol Interaction, pages 132-140, 2018.
[18] E. BIyIk, M. Palan, N. C. Landolfi, D. P. Losey, and D. Sadigh. Asking easy questions: A user-friendly approach to active reward learning. arXiv preprint arXiv:1910.04366. 2019.
[19] R. Holladay, S. Javdani, A. Dragan, and S. Srinivasa. Active comparison based learning incor- porating user uncertainty and noise. In RSS Workshop on Model Learning for Human-Robor Communication, 2016.
[20] M. Palan, N. C. Landolfi, G. Shevchuk, and D). Sadigh. Learning reward functions by integrat- ing human demonstrations and preferences. arXiv preprint arXiv:1906. 089228 2019.
[21] E. BIyIk, D. Losey, M. Palan, N. C. Landolfi, Shevchuk, and D. Sadigh. Learning reward functions from diverse sources of human feedback: Optimally integrating demonstrations and preferences. The International Journal of Robotics Research, 41(1):45-67, 2022
[22] A. Avaei, L. van der Spaa, L. Peternel, and Kober. An incremental inverse reinforcement learning approach for motion planning with separated path and velocity preferences. Robotics, 12(2):61, 2023.
[23] R. Wang, W. Wang, and B.-C. Min. Feedback-efficient active preference learning for socially aware robot navigation. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 11336-1 1343. IEEE, 2022.
[24] W. Muldrew, P. Hayes, M. Zhang, and D. Barber. Active preference learning for large language models, arXiv preprint arXiv:2402.081144 2024.
[25] Lin, N. Tomlin, J. Andreas, and J. Eisner. Decision-oriented dialogue for human-al ai collabo- ration. arXiv preprint 230.5h::00000076 2023.
[26] T. Piriyakulkij. V. Kuleshov, and K. Ellis. Active preference inference using language models and probabilistic reasoning. 2023.
[27] C. Andukuri, J.-P. Fränken, T. Gerstenberg, and N. D. Goodman. Star-gate: Teaching language models to ask clarifying questions, 2024.
[28] K. Handa, Y. Gal, E. Pavlick, N. Goodman, J. Andreas, A. Tamkin, and B. Z. Li. Bayesian preference elicitation with language models, 2024.
[29] D. V. Lindley. On measure the information provided by an experiment. The Annals of Mathematical Statisties, 27(4):986-1055, 1956.
[30] T. Rainforth, A. Foster, D. R. Ivanova, and F. Bickford Smith. Modern bayesian experimental design. Statistical Science, 39(1): 100-114, 2024.
[31] B. Z. Li, A. Tamkin, N. Goodman, and J. Andreas. Eliciting human preferences with language models, 2023.
10

---

[32] A. Z. Ren, A. Dixit, A. Bodrova, S. Singh, S. Tu, N. Brown, P. Xu, L. Takayama, F. Xia, J. Varley, et al. Robots that ask for help: Uncertainty alignment for large language model planners. arXiv preprint arXiv:3307.01922 2023.
[33] W. Huang, F. Xia, T. Xiao, H. Chan, J. Liang, P. Florence, A. Zeng. J. Tompson, I. Mordatch, Y. Chebotar, P. Sermanet, N. Brown, T. Jackson, L. Luu, S. Levine, K. Hausman, and B. Ichter. Inner monologue: Embodied reasoning through planning with language models, 2022.
[34] H. Wang, K. Kedia, J. Ren, R. Abdullah, A. Bhardwaj, A. Chao, K. Y. Chen, N. Chin, P. Dan, X. Fan, G.
A. Kompella, M. A. Pace, Y. Sharma, X. Sun, N. Sunkara, and S. Choudhury. Mosaic: A modular system for assistive and interactive cooking, 2024.
[35] K. Lin, C. Agia, T. Migimatsu, M. Pavone, and J. Bohg. Text2motion: from natural lan- guage instructions to feasible plans. Autonomous Robots, 47(8):1345-1365, Nov. 2023. ISSN 1573-7527.
URL http://dx www.ddi.oo wto.....000/.....11.............
s 110514-023-10111
[36] W. Huang, C. Wang. R. Zhang, Y. Li, J. Wu, and L. Fei-Fei. Voxposer: Composable 3d value maps for robotic manipulation with language models, 2023.
[37] I. Singh, V. Blukis, A. Mousavian, A. Goyal, D. Xu, J. Tremblay, D. Fox, J. Thomason, and A. Garg. Progprompt: Generating situated robot task plans using large language models, 2022.
[38] K. Valmeekam, M. Marquez, Sreedharan, and S. Kambhampati. On the planning abilities of large language models-a critical investigation. Advances in Neural Information Processing Systems, 36:75993-76005, 2023.
[39] S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.0362,, 2022.
[40] N. Shinn, F. Cassano, A. Gopinath, K. Narasimhan, and S. Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
[41] R. A. Bradley and M. E. Terry Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324-455) 1952.
[42] S. Liu, Z. Zeng. T. Ren, F. Li, H. Zhang, J. Yang. C. Li, J. Yang, H. Su, J. Zhu, and L. Zhang. Grounding dino: Marrying dino with grounded pre-training for open-set object detection, 2023.
[43] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever. Learning transferable visual models from natural language supervision, 2021.
[44] Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.06347, 2017.
[45] A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan, Goyal, A. Hartshom,, A. Yang. A. Mitra, A. Sravankumar, A. Korenev, A. Hinsvark, A. Rao, A. Zhang, A. Rodriguez, A. Gregerson, A. Spataru, Roziere, B. Biron, B. Tang, B. Chern, C. Caucheteux, C. Nayak, C. Bi, C. Marra, C. McConnell, C. Keller, C. Touret, C. Wu, C. Wong, C. C. Ferrer, C. Nikolaidis, D. Allonsius, D. Song, D. Pintz, D. Livshits, D. Esiobu, D. Choudhary, D. Mahajan, D. Garcia-Olano, D. Perino, D. Hupkes, E. Lakomkin, E. AlBadawy, E. Lobanova, E. Dinan, E. M. Smith, F. Radenovic, F. Zhang, G. Synnaeve, G. Lee, G. L. Anderson, G. Nail, G. Mialon, G. Pang, G. Cucurell, H. Nguyen, H. Korevaar, H. Xu, H. Touvron, Zarov, I. A. Ibarra, Kloumann, I. Misra, Evtimov, Copet, Lee, J. Geffert, Vranes, Park, Mahadeokar, J. Shah, J. van der Linde, Bil- lock, J. Hong. J. Lee, J. Fu, J. Chi. J. Huang, J. Liu, J. Wang, J. Yu, J. Bitton, J. Spisak, J. Park, J. Rocca, J. Johnstun, J. Saxe, J. Jia, K. V. Alwala, K. Upasani, K. Plawiak, K. Li, K. Heafield,
11

---

K. Stone, K. El-Arini, K. Iyer, K. Malik, K. Chiu, K. Bhalla, L. Rantala- Yeary, L. van der Maaten, L. Chen, I.. Tan, L.. Jenkins, L. Martin, L. Madaan, L. Malo, L.. Blecher, I.. Landzaat, L. de Oliveira, M. Muzzi, M. Pasupuleti, M. Singh, M. Paluri, M. Kardas, M. Oldham, M. Rita, M. Pavlova, M. Kambadur, M. Lewis, M. Si, M. K. Singh, M. Hassan, N. Goyal, N. Torabi, N. Bashlykow, N. Bogoychev. N. Chatterji, O. Duchenne, O. Çelebi, P. Alrassy, P. Zhang, P. Li, P. Vasie, P. Weng. P. Bhargava, P. Dubal, P. Krishnan, P. S. Koura, P. Xu, Q. He, Q. Dong, R. Srinivasan, R. Ganapathy, R. Calderer, R. S. Cabral, R. Stojnic, R. Raileanu, R. Gird- har, R. Patel, R. Sauvestre, R. Polidoro, R. Sumbaly, R. Taylor, R. Silva, R. Hou, R. Wang, S. Hosseini, S. Chennabasappa, S. Singh, S. Bell, S. S. Kim, S. Edunov, S. Nie, S. Narang, S. Raparthy, S. Shen, S. Wan, S. Bhosale, S. Zhang, S. Vandenhende, S. Batra, S. Whitman, S. Sootla, S. Collot, S. Gururangan, S. Borodinsky, T. Herman, T. Fowler, T. Sheasha, T. Geor- giou, T. Scialom, T. Speckbacher, T. Mihaylov, T. Xiao, U. Karn, V. Goswami, V. Gupta, V. Ra- manathan, V. Kerkez, V. Gonguet, V. Do, V. Vogeti, V. Petrovic, W. Chu, W. Xiong, W. Fu, W. Meers, X. Martinet, X. Wang. X. E. Tan, X. Xie, X. Jia, X. Wang, Y. Goldschlag, Y. Gaur, Y. Babaei, Y. Wen, Y. Song, Y. Zhang, Y. Li, Y. Mao, Z. D. Coudert, Z. Yan, Z. Chen, 7. Pa- pakipos, A. Singh, A. Grattafiori, A. Jain, A. Kelsey, A. Shajnfeld, A. Gangidi, A. Victoria, A. Goldstand, A. Menon, A. Sharma, A. Boesenberg, A. Vaughan, A. Baevski, A. Feinstein, A. Kallet, A. Sangani, A. Yunus, A. Lupu, A. Alvarado, A. Caples, A. Gu, A. Ho, A. Poulton, A. Ryan, A. Ramchandani, A. Franco, A. Saraf, A. Chowdhury, A. Gabriel, A. Bharambe, A. Eisenman, A. Yazdan, B. James, B. Maurer, B. Leonhardi, B. Huang. B. Loyd, B. D. Paola, B. Paranjape, B. Liu, B. Wu, B. Ni, B. Hancock, B. Wasti, B. Spence, B. Stojkovic, B. Gamido, B. Montalvo, C. Parker, C. Burton, C. Mejia, C. Wang, C. Kim, C. Zhou, C. Hu, C.-. Chu, C. Cai, C. Tindal, C. Feichtenhofer, D. Civin, D. Beaty, D. Kreymer, D. Li, D. Wy- att, D. Adkins, D. Xu, D. Testuggine, D. David, D. Parikh, D. Liskovich, D. FOss, D. Wang, D. Le, D. Holland, E. Dowling, E. Jamil, E. Montgomery, E. Presani, E. Hahn, E. Wood, E. Brinkman, E. Arcaute, E. Dunbar, E. Smothers, F. Sun, F. Kreuk, F. Tian, F. Ozgenel, Caggioni, F. Guzmán, F. Kanayet, F. Seide, G. M. Florez, G. Schwarz, G. Badeer, G. Swee, G. Halpern, G. Thattai, G. Herman, G. Sizov, Guangyi, Zhang, G. Lakshminarayanan, H. Sho- janazeri, H. Zou, H. Wang, H. Zha, H. Habeeb, H. Rudolph, H. Suk, H. Aspegren, H. Goldman, I. Molybog, I. Tufanov, I.-E. Veliche, I. Gat. J. Weissman, J. Geboski, J. Kohli, J. Asher, J.-B. Gaya, J. Marcus, J. Tang, J. Chan, J. Zhen, J. Reizenstein, J. Teboul, J. Zhong, J. Jin, J. Yang, J. Cummings, J. Carvill, J. Shepard, J. McPhie, J. Torres, J. Ginsburg. J. Wang, K. Wu, K. H. U, K. Saxena, K. Prasad, K. Khandelwal, K. Zand, K. Matosich, K. Veeraraghavan, K. Miche- lena, K. Li, K. Huang, K. Chawla, K. Lakhotia. K. Huang. Chen, L. Garg, L. A, L. Silva, L. Bell, L. Zhang, L. Guo, L. Yu, L. Moshkovich, L. Wehrstedt, M. Khabsa, M. Avalani, M. Bhatt, M. Tsimpoukelli, M. Mankus, M. Hasson, M. Lennie, M. Reso, M. Groshev, M. Nau- mov, M. Lathi, M. Keneally, M. I. Seltzer, M. Valko, M. Restrepo, M. Patel, M. Vyatskov, M. Samvelyan, M. Clark, M. Macey, M. Wang, M. J. Hermoso, M. Metanat, M. Rastegari, M. Bansal, N. Santhanam, N. Parks, N. White, N. Bawa, N. Singhal, N. Egebo, N. Usunier, N. P. Laptev, N. Dong, N. Zhang, N. Cheng, O. Chernoguz, O. Hart, O. Salpekar, O. Kalinli, Kent, P. Parekh, Saab, P. Balaji, P. Rittner, P. Bontrager, Roux, P. Dollar, P. Zvyagina, Ratanchandani, P. Yuvraj, Q. Liang, R. Alao, R. Rodriguez, R. Ayub, R. Murthy, R. Nayani, R. Mitra, R. Li, R. Hogan, R. Battey, R. Wang, R. Maheswari, R. Howes, R. Rinott, S. J. Bondu, S. Datta, S. Chugh, S. Hunt, S. Dhillon, S. Sidorov, S. Pan, S. Verma, S. Yamamoto, S. Ramaswamy, S. Lindsay, S. Lindsay, S. Feng, S. Lin, S. C. Zha, S. Shankar, S. Zhang, S. Zhang, S. Wang, S. Agarwal, S. Sajuyigbe, S. Chintala, S. Max, S. Chen, S. Kehoe, S. Sat- terfield, S. Govindaprasad, S. Gupta, S. Cho, S. Virk, S. Subramanian, S. Choudhury, S. Gold- man, T. Remez, T. Glaser, T. Best, T. Kohler,
Matthews, Robinson, Li, T. Zhang, T. Chou, T. Shaked, V. Vontimitta, V. Ajayi, V. Montanez, V. Mohan, S. Kumar, V. Mangla, V. Ionescu, V. Poenaru, V. T. Mihailescu, V. Ivanov, W. Li, W. Wang, W. Jiang, W. Bouaziz, W. Constable, X. Tang, X. Wang. X. WU, X. Wang, X. Xia, X. Wu, X. Gao, Y. Chen, Y. Hu, Y. Jia, Y. Qi, Y. Li, Y. Zhang, Y. Zhang, Y. Adi, Y. Nam, Yu, Wang, Y. Hao, Y. Qian, Y. He, Z. Rait, Z. DeVito, Z. Rosnbrick, Z. Wen, Z. Yang, and Z. Zhao. The Ilaaa herd of models, 2024. URL https: // //twxviivvv.////////a///aaaa/ wwwaaii.org///a.aae...l..a orgo htrg/aba/2400 21783.
12

---

[46] A. Madaan, N. Tandon, P. Gupta, S. Hallinan, L. Gao, S. Wiegreffe, U. Alon, N. Dziri, S. Prabhumoye, Y. Yang, S. Gupta, B. Majumder, K. Hermann, S. Welleck, A. Yaz- danbakhsh, and P. Clark. Self-refine: Iterative refinement with self-feedback, 2023, URL https://arxiee argg/bs//303/ 17651.
[47] Z. Ling, Y. Fang, X. Li, Z. Huang, M. Lee, R. Memisevic, and H. Su. Deductive verification of chain-of-thought reasoning, 2023. URL https::////// //ap:ii//axxvv..// wwg/oog//////e ab3//306...... 22822.00777....
[48] X. Chen, R. Aksitov, U. Alon, J. Ren, K. Xiao, P. Yin, S. Prakash, C. Sutton, X. Wang, and
D. Zhou. Universal self-consistency for large language model generation, 2023. URL https: // //arxiv.or abs atss/231111 2211. 17311.
[49] C. Kemp, A. Edsinger, M. Clever, and B. Matulevich. The design of stretch: A compact, lightweight mobile manipulator for indoor human environments, 2022.
[50] Zed stereo camera. variant=4137992996338
URL https://store wtw.rrlaaset....bb........o. heareollbb..cmmpprrouuttt////d2d
13

---

Appendix
Table of Contents
7 Approach Details
14
7.1 Active Preference Learning
14
7.2 Task Planner
17
7.3
Real Robot System
8
Experiments Details
18
8.1 Active Preference Learning Experiments Details
18
8.2 Task Planner Real Robot Experiments
19
Prompts
23
9.1
Active Preference Learning Prompts
23
9.2 Task Planner Prompts
43
9.3 Baseline Prompts
46
Approach Details
7.1 Active Preference Learning
7.1.1 Detailed Algorithm.
Algorithm 1 shows the detailed workflow APRICOT's active preference learning module. We note that, to expedite computation, we generate the candidate questions before entering the main active preference learning loop. For each candidate preference pair (0,, 0j), the question-generating LLM also receives the initial state so, the task T, the candidate plans corresponding to the pair &, E,. and each plan's rewards given one of the candidate preferences (R(E,, 0,), R(E,, f2)) and (R(E,, 0,), R(E 0;)). During the main loop, once that question is asked, it is removed from the list of candidate questions.
7.1.2 Hyperparameters.
We set the number of candidate preferences to N 5 and the number of questions to generate for each preference pair to M = 2, so we generate 20 questions in total. For the terminating function, which determines when the preference prior P(0) is sufficient, we set the terminating threshold to 0.07.
The components that require an LLM prompt are: generating candidate preferences, calculating the reward of plan given preference, generating candidate questions, estimating the likelihood of an answer given preference and a question. All LLM calls use temperature of 0.7. We use gpt-4 for "generating candidate preferences" because this task requires reasoning about the similarities and differences across multiple demonstrations and proposing a diverse set of preferences that are consistent with the demonstrations. We use gpt-3-turbo for "estimating the likelihood of answer" because this task calls the L.LM the most number of times (N M x (2) times), and this task has simple input-output space (inputs include preference to roleplay and yes-or-no question to answer, outputs include reasoning and yes/no answer). We gpt-4o for the rest of the tasks. Prompts can be found in Appendix 9.1.
14

---

Steps
# LLM calls
Avg. Tokens
Avg.
Time (min)
Gen preferences
13000
2.5
Gen N plans
5
2800
0.25
Calculate plans' rewards
25
90000
2.5
Propose candidate questions
10
27000
1.16
Calculate the likelihood
100
200000
3.33
One iteration asking clarification questions
N/A
N/A
0.0001
Table 1: Given N = 5, we report the number of LLM calls, the average number of tokens used, and the average wallclock time for each step in APRICOT's 's active preference learning module.
Algorithm APRICOT LLM-based Bayesian Active Preference Learning
Input:
Demonstrations D, Initial State sp. Task T, Terminating Function f(P(0))
LLM Prompts: LLM to generate candidate preferences pgen-pre LLM to generate plans pplan LLM to generate questions Peerq
Output: Best plan &. Corresponding preference of the best preference 0,
// Generate candidates
Generate candidate preferences {0: J.V 0 paen-preffeeeee pref (0
Initialize Prior P(0,)
Generate candidate plans {{:} prlnn (E|su, T, 0,), ,,...,
Construct Q by generating M questions for all preference pairs: q ~ pgen (q
// While not exist plan & that has sufficiently high rewards given all preferences in [0.)-- while not (()()) do
H[P(0)] - E. (yes,no) ΣN P(olg, HP) 9 9.
Find the best question: 9"' < arg m Query the user with qº and gets answer On
Remove 9º from Q
Update prior: P(0) P(O
Return & that satisfies the Terminating Function f. the corresponding 0;
7.1.3 Computation overhead.
For smooth user interaction, crucial that a system fast when it actually asks user clarification questions.
APRICOT complete iteration of question-asking quickly (about 0.0001 min/iter) because it can find the optimal question by simply doing matrix calculations with the newly updated prior. All the necessary LLM calls, which naturally take longer, are completed before APRICOT starts asking questions. We also quantify the estimated token usage and wall-clock time for each step of the active preference learning algorithm in Table 1.
In addition, because APRICOT combines LLMs with active preference learning, it is able scale well with increasing complexity (e.g. in preference types) because adding new preference types just requires adding short description to the prompt. It can also scale well with an increasing number of user demonstrations. Because these demonstrations can become too long if they are simultaneously processed together, APRICOT can adapt the approach in [4], which summarizes each demonstration individually first before analyzing common patterns across summaries.
7.1.4 Discussion About the Terminating Condition
We derive our termination condition as a principled approximation of the ideal termination condi- tion, and provide additional analysis to quantify the performance gap due to the approximations.
The Ideal Terminating Condition.
15

---

Recall R(C.0) is the reward of a plan & on whether it satisfies the preference 0. We define de- terministic function (g. 0) = arg max oE (yes, (yes,o)) P(olq, 0) that outputs the answer to an question given a preference 8. Let be the ground-trune truth preference unknown to the system, D be a small set of demonstrated plans ED, and H - {(g, o), ..... corresponding answers from the user with 0* We define the of consistent preferences as
a history the questions asked so far and the
+ (0: (VED R,ED
1) (v(g, o) H, Hg 0)
consistent with demos
consistent with user answers
The ideal terminating condition requires the module keep asking questions until
E s.t v0 θ R(5.0) =.
However, it is computationally intractable enumerate through all the possible preferences and compute optimal plans for each preference. We introduce a number of approximations.
Approximation Assumptions.
Approximating the aforementioned ideal terminating condition requires the following assumptions:
1. When we sample a set of N candidate preferences consistent with the demonstrations θ = {0, : VED E R(ED, 8,) = 1) one of the sampled candidates 0 is value- equivalent to the ground-truth preference 0" This candidate must be consistent with the user's answers, i.c., V(g.o) o) H. (g 8) O. Also,
vg, [R({,0)) R(E, 0)1| Crealization
2. The task planner has a suboptimality of (plamner- This planner's suboptimality lower bounds the suboptimality of the outputted plan. Given a preference 0, let the optimal plan be 5* arg max; ((E,0) The performance difference between this optimal plan &* and the planner's plan Ç is:
RiE", 0) R(E,0) 0) S Eplanner-
(8)
Performance Bound Proof.
Theorem 7.1. Under assumptions (7) and (8), given problem with the ground-truths 0*
and E" APRICOT outputs & when it follows (3) the terminating condition f(P(O))
E s.t EN P(e.)DDSADV((, DISADV(S,, 0:) maxxe= R(E,0.) - -R(EE, 0:). Its performance is bounded by R(E",0" - R(É,0") s Ne 2erealization + Eplamer-
I based (2) the disadvantage function
Proof. Let 9 be the candidate preference that is value-equivalent to l*. Note that probabilities are nen-negatives (i.e., vo è, P(0) z 0) and outputs of the disadvantage function are also non- negative (i.e., VE, 40 ê, DisADVE, #) 0). Based on the terminating condition (3), the worst case on is:
(10) In other words, this scenario is when only one candidate is value-equivalent to 0*, while all the remaining N preferences θ {0} agree on an incorrect plan (i.e. vO E O (0), R(É, 0) 1).
P(O DISDDIV c
Expanding (10), we show:
max REE, 0) R) < SES
via (2)
(11)
P(0)
max R(C, 8*) RIE, 0*) 5 EEB
+ 2'
via (7)
(12)
P(O)
R(E", 0*) R(S, 00) < P())
+ 2Egealization + Eplanner
via (8)
(13)
R(E", 0*) RIC 0*) < No + 2erelizztion + fllanerr-
(14)
The last inequality is via the fact that P(0) 2 $. Because the initial prior is y and the answers of Ô are consistent with the user's answers, its probability will never decrease.
16

---

Choose the terminating threshold e in practice. Similar to the proof, we consider the worst-case scenario before APRICOT starts asking any questions: 0 is the only candidate preference that is value-equivalent to the ground-truth preference 8* and 4A E ê \ (0), DISADV(É, 6) = 0 for an incorrect plan 5.
Ideally, APRICOT should not pick this incorrect plan, and it should ask clarification questions in- stead. To avoid satisfying the terminating condition (3), the following must be true
e W P(@)DiSADV(E..)
via (10)
(15)
8 S 1 N
1
DISADV((, 0).
via P(O) 2 explained in the previous section
(16)
Thus, to set the terminating condition, we need to determine the minimum disadvantage that an incorrect plan can get given the correct candidate preference 0. We empirically find setting 0.35 has worked well, thereby resulting in threshold 0.07.
7.2 Task Planner
Beam Search. Recall that APRICOT's task planner first uses an LLM to output semantic plan given a task T (a list of objects to put away) and an initial state so (a dictionary of objects initially at each semantic location in the fridge). Then, uses a beam search to find the XYZ placement location for each object by sequentially following the semantic plan.
The world model contains mapping from semantic location (e.g. "left side of top shelf") range of XYZ coordinates that corresponds to that region. With this pogaanti-locaiioo-t----ooooo range mapping, the beam search samples candidate XYZ placement location for each object based on its planned semantic place location. Specifically, it samples in XY space since objects can be placed anywhere length-wise and depth-wise on shelf. Z location is based on the height of the shelf because we do not consider stacking objects atop one another. The best candidates are those where the placement is geometrically feasible (no fridge or object collisions). These candidates are maintained in the beam search nodes.
The beam search continues until all objects are placed into the fridge. An optimal plan is one where all objects acquire geometrically-fesii in the search only places a subset objects, the semantic plan is assumed to be geometrically infeasible and the LLM reprompted with information about which objects lack a feasible XYZ placement location.
collision-free XYZ placement location. If the best plan
Hyperparameters. For semantic plan generation, we use gpt- gpt-4-turbo with a temperature of 0.7. For the beam search, we maintain 10 best candidates and sample 10 evenly-spaced points with the location regions. For feedback retries, we allow up to 4 attempts. Prompts can be found in Appendix 9.2.
7.3 Real Robot System
RL Policy Training. The policy pick(sobj>) and place(<loc>) can both be abstracted as the problem of given a goal XYZ position and the current robot joint state, output sequence of actions that allows the gripper to reach the goal straight on. We simulate the robot's kinematics and include the J, norm between the goal and current positions as the observation space. We define the robot's teleoperation commands as the discrete action space. To guide the agent to learn a suitable grasp policy, we implemented four-phase reward function:
1. Align the Gripper horizontally: The goal object should align in between the robot's gripper (X coordinate).
2. Align the Gripper height: The gripper should align with the goal height (Z coordinate).
3. Extend the gripper: The gripper should extend to pick/place the object (Y coordinate).
4. Goal Achievement: A reward of +1 is given if the robot successfully reaches the goal.
17

---

AX max(AY) + m
AX Z cs
AZ AX + max
Z 2 E2! Ar 2 Ex
(17)
AY AZ AX,
AY 2 by AZ. 2 62 Ax 2 6z
AY Ens AZ c Au Ez
With this reward function, we train a Proximal Policy Optimization agent using the implementation from [44], to predict actions that will lead suitable grasp on the goal position.
Experiments Details
8.1 Active Preference Learning Experiments Details
8.1.1 Benchmark Dataset
The dataset contains 100 test cases, each containing a ground-truth preference, two demonstrations with before and after state of user putting objects into fridge, one scenario to solve. The ground-truth preference is created by (I) randomly generating the special preferences for 1 or 2 object categories and (2) randomly assigning simpler preferences for the remaining object categories. For the demon- strations and the test scenario, we randomly sample objects and specific placement coordinates that satisfy the generated ground-truth preference. Each demonstration has 3 objects initially in the fridge and to place into the fridge. Mcanwhile, each test scenario has objects initially in the fridge and to place into the fridge. The objects in the fridge can be categorized into 5 categories: fruits, vegetables, condiments, dairy products, juice and soft drinks. The cases are also grouped into 5 categories, where each categories contain 20 test cases: specific location, general location, relative position, subcategory exceptions, and conditionals. Fig 3 shows examples of each category.
The test cases "specific location" require each category to be placed at a specific location in the fridge (e.g. "fruit on the left side of top shelf"), while rest of the categories have special requirements on subset of the categories and only specific location requirements on the remaining categories. For the 20 "general location" test cases, half of them require only one category to be general location (e.g. "fruits the left side of the fridge"), and the other half require two categories to be at general location (e.g. "fruits on the left side of the fridge, and vegetables on the right side of the fridge). In addition, for the 20 "relative position" test cases, half of them require one category to be together with no specific location (e.g. "fruits should be placed together regardless of what shelf they are on"), and the other half require two categories to be together (e.g. "fruits and vegetables should be placed together regardless of what shelf they are on.")
8.1.2
Setup
The prompts for the bascline approaches are in Appendix 9.3. We run each approach on a test case once. We make APRICOT generate N 1 candidate preferences and add the preference generated by Ion-Interactive as the fifth candidate preference. We also make Cand+LLM-Q/A have the same set of candidate preferences and candidate plans and APRICOT for given test case.
8.1.3 Failure Modes of APRICOT
We find that APRICOT fails mainly because the LLMs hallucinate and make mistakes when they are called used for evaluation.
After analyzing all the instances where APRICOT fails, we identify the following failure modes (with the most common failure modes sorted at the top):
1. Incorrect question answering: The LLMs can answer a question incorrectly due to hal-
lucinating or interpreting the question incorrectly. This can be further categorized as:
Mistakes during likelihood estimation which is used to select the optimal questions.
18

---

Incorrect QA
No GT Pref
Mismatched
Incorrect reward
Demo
During likelihood est
During evaluation
14.3%
21 1144%
26.2%
19.0%
11.9%
7.20%
Table 2: Among the 42/100 test that APRICOT fails to solve, each column shows the percentage of failures that caused by specific failure mode.
Mistakes during evaluation when the LLM pretends to be the user with the ground- truth preference and interacts with APRICOT.
2. Ground-truth preference not one of the candidate preferences: Sometimes, the LLMs do not generate candidate preference that fully captures the same requirements the ground-truth preference.
3. Mismatched understanding of object categories: For example, some demonstrations use "tomatoes" as examples for vegetable placements because we treated tomatoes as vegeta- bles, but the LLM classifies "tomatoes" as fruit.
4. Incorrect reward calculation: The LLMs can make mistakes when calculating the reward of plan and determining the percentage of object placements that satisfy preference.
5. Hallucinations when interpreting demonstrations: The LLMs occasionally ignore some objects that appeared in the demonstrations.
Among the 42/100 test cases that APRICOT fails to solve, Table 2 reports the percentage failures that are caused by each failure mode.
The most common failure mode, "Incorrect question answering" (causing 35.7% of the failures in total), explains why APRICOT's performance is not higher. The LLMs that approximate the model of the human are imperfect. The subcategory "Mistakes during evaluation" causes 21.4% .144% of the failures. When evaluation uses the LLMs to mimic the actual user, the LLMs can answer APRICOT's question incorrectly, thereby causing APRICOT also update its prior incorrectly.
For future work, plan improve the LLMs by (1) Switching more powerful models: We currently use gpt-3.5-turbe 5-turbo to answer the questions, but we can switch to more powerful models such as gpt-4o or Llama3 [45] to boost performances. (2) Aligning the LLMs' understanding of object categories with the user's: In the prompt, we can include information about how the user typically categorizes the objects that have ambiguous categories (c.g. "tomato," "bell pepper"). (3) Adding self-consistency verification: A verification module (which can be another L.L) can verify the output of the model [46, 47, 48].
8.1.4 Tradeoff between query efficiency and preference accuracy
We hypothesize that, ideally, the number of queries increases, preference accuracy should also increase and eventually plateau.
However, because the most common failure mode is that the LLMs make mistakes when answering questions (see Section 8.1.3), we find that empirically asking too little and too many questions would both harm the performance. Specifically, we test on terminating threshold ranging from 0.09 0.03 and plot the number of queries v.s. preference accuracy Figure 7. For the test cases under the more complex preference types ("subcategory exception" and "condition"), asking too many questions actually lowers the preference accuracy.
8.2 Task Planner Real Robot Experiments
8.2.1 Setup
Similar to the benchmark dataset for active preference learning experiments, the 9 real-world test cases (a ground-trui -ruth preference, two demonstrations, a test scenario) are randomly generated while
19

---

Number of Queries v.s. Preference Accuracy
subcategory exceppionnoceiionnnoooooo
conditional
Preference Accuracy
0.4
0.2
0.0
1.5
2,0
2.5
3.0
3.5
4.0
Number Quaries
Figure 7: The number of queries v.s. preference accuracy. We test APRICOT on terminating threshold rang- ing from 0.09 to 0.03. Contrary to the hypothesis, large number of questions actually lowers the preference accuracy.
Tker Preferance
Planner
Vigetaaias -->h Tegatter
Condinents Tepether
Mese 2iee of deundiiaee receree she strete aeo 21)
Left sife
falle "cadeess
viice
button shall
Initial State fs
Task
Reward
World Model
rickas Prrmer doft side of the (ar
Oltys unady.
The provices plan was goometricaly irlousille.
Reward
World Modil
RCS.
The hhess dil were [Ind bell pepper].
R 0)
pnpeac Dler Mick 7%, 3.30 19.34
Figure 8: Example APRICOT's task planner reflecting and refining its plan based on feedback.
following dataset generation rules to create scenarios where the geometric constraints are in tension with the preference. Each test case's ground-truth preferences are generated by first assigning some special preference for object categories. For the demonstrations and the initial condition to test on, randomly sample objects from list of objects that have, and we design each object's placement coordinates to be close to each other to create cluttered environment. We show the visual demonstration and the ground-truth preference in Fig. 9.
For all real robot experiments, we use a 6-DoF Stretch Robot REI [49] as the mobile manipulator. A static RBG-D camera (ZED-2i [50]) mounted to have a direct view of the fridge. A table is placed the left of the fridge, on which objects that need to be put away by the robots are placed in row with occlusions. Before the experiments, we have mapped the environment and determined the location of the table and the fridge in the map. Images from the static camera are used by the perception module to track the current state of the world for the task planner and to convert user demonstrations into text-based demonstrations. The point cloud from the static camera is used by the world model to collision-check potential XYZ placement location. Meanwhile, the Stretch Robot's onboard RGB-D camera is used by the pick policy identify the grasp position of an object.
We report the performance of APRICOT's active preference learning module in Table. 3.
8.2.2 APRICOT Plan Refinement Example
Fig. shows an example of APRICOT's task planner refining its plan based on feedback. When the LLM generates the semantic plan, because it does not have knowledge about the environmental constraints of the fridge, decides to place both the pineapple and the red bell pepper on the left side of the bottom shelf, which would satisfy the user's preference. However, only one object can be placed on the left side of the bottom shelf. Given this plan, although the reward function determines that the semantic plan maximally satisfies the preference R(E, 0) 1. the world model determines
20

---

Demonstrations
Ground-Truse Preference
Scenario Initial State
Objs to Put Away
prder condinerss be put neet to each other and drinks to be each other.
Morard Sottle Sprke Can
Easy Scenarios
prefer fruits side right cheff
Pineapple Juice bentle
orffer drinks ocitiig some regarilems of which sheff or which slde of the shalf they
Bottle
Sprite
vegitablec
Pinsoople
together on ane shalt Dairy products ouild be the left side fridge. and drinks fridge.
Chopse
Pepsi Can
Medium Scenarios
tegether on the side of shelt and
Pincupii
Katchup
vegetables should tegather the lefi side another right side cheff
Mustard bottle
smar la conimmats be plose sughter and déels to
Relish Speie
objess, caeaed peservad releradde faod and le the sido the
the hummnn dulf
Pesper
Orange jaice betile
Fanta can
cune ragetabica placed next to evisting vegrtalles the same shelf regandiess whar whell what side Fruits should
Hard Scenarios
Red Bell Peoper
Ketchup
Turtar Soute
Pinesople
right the fridge want meetss to be placod next cxiting cand ientss the shelf
Sprite
Apple Juice
Pireappe
Figure 9: Real world demonstrations and scenarios.
21

---

Scenario
Preference Accuracy
Number of Queries
1
1.00
0.00
Easy
2
0.00
5.00
3
1.00
0.00
Medium
1
0.00
0.00
2
1.00
1.00
3
0.00
0.00
1.00
0.00
Hard
2
1.00
2.00
3
1.00
1.00
Overall
0.56
1.00
Table 3: APRICOT active preference learning module's performance real-world scenarios. Note that the difficulty level is based on how difficult the task is for the task planner (i.e. the number of objects initially in the fridge the number objects to put away.)
that red bell pepper will have collision when being placed on the left side of the bottom shelf next the pineapple. This feedback gets included in the input for the LLM, the LLM is able reflect on why its current plan is geometrically infeasible and determine alternative semantic placement location for the red bell pepper ('right side of the top shelf"). The new plan satisfies the preference by maximizing the reward and is geometrically feasible by converting the semantic plans into XYZ placement locations for all the objects.
Replan when the user moves sth
Replan when the user adds sth
Replan when the user removes sth
Prefermcne:
topeberr
Prefennco:: rigroblle
tidgs
Proferecc the right of top shafl les specs condine protered thene, Odhanitas marts right mádila
Iesil Suas
beita Plr
Deplooed Senortic
Repllaed Seruniic
Medtoed Semsrtic
Sacour's auelity
Robor Thad
Uve Mowel Drann
Secscari
Robas Flaaed "siieapplin
Une Added
Saccodls
Roaoe Flnze
Uspe
Uabile
Figure 10: Examples of how APRICOT's task planner can replan and handle different ways user could affect and change the environment.
8.2.3 APRICOT Replanning Example
Fig. 10 show all 3 examples of how APRICOT's task planner is able to replan and handle changes in the environment.
Replanning after additions. The user's preference requires vegetables to be placed on the left side of the fridge. The planner initially plans to place the bell pepper on the left side of the middle shelf. However, the user disrupts the scene and adds various vegetables there. APRICOT's closed-loop planner adjusts to its plan by placing the bell pepper on the left side of the bottom shelf which still has space.
Replanning after removals. The user's preference requires placing condiments on the right of the top shelf if that space is empty. If that location full, condiments can be placed on the right of the middle shelf. The planner initially plans to place the mustard and relish the right side of the middle shelf since the right side of the top shelf is filled with coke bottles. However, the user
22

---

disrupts the scene and removes the bottles. APRICOT's closed-loop planner adjusts to this by placing the relish in the freed space to best satisfy the user's preference.
8.2.4 Perception System Performance
To quantify the performance our perception module, we evaluate the real-world scenarios used in the task planner real-robot experiments.
Baselines. Recall that APRICOT's perception module (GD+CLIP) uses Grounding-Ding [42] to detect the objects in the fridge and CLIP [43] to label the detected object name with an object name. Compared to this approach, we define the baseline GD-Only, which runs Grounding-DINO,, given each object label from the list of all possible grocery items.
Metrics. We define two metrics. The percentage of correctly identified object calculates the erage ratio between the detected objects that have the correct object label over the expected number of objects. The number of incorrectly detected objects calculates the average number of incorrect labels, so lower is better.
GD-Only
0.7
GD-Only
CUP
0.6
Correctly Objects
of incorrectly Identified Objects
0.6
0.4
0.2
0,32
0.0
OD-Only
6D
GD-Only
Figure 11: Result perception module on real-robot scenarios. also report the standard error the plot.
Results. Fig. 11, we observe that, for objects that are actually in the fridge, GD+CLIP able as- sign more correct labels (87.09 (87.0%) compared GD-Only (74.0%). In addition, GD-Only detects more objects incorrectly and has much more false positives compared to GD+CLIP. Because ID-Only calls Grounding DINO for each possible object label, it repeatedly labels the same region in the image with different object labels. This behavior is undesirable because APRICOT's task planner relies on the perception system to exactly detect the right number objects in the fridge and assign each detected object with the correct label. GD-Only would cause the task planner to hallucinate what objects are currently in the fridge.
Prompts
9.1
Active Preference Learning Prompts
The prompts for APRICOT's ' active preference learning modules are the following:
Generate Candidate Preferences Given Demonstrations (Sec. 9.1.1)
Generate Plan Given Preference, an Initial Condition, and Task (Sec. 9.2)
Calculate the Reward Given a Preference and a Plan (Sec. 9.1.2)
Generate Candidate Preferences Given Pair of Preferences (Sec. 9.1.3)
Answer Question Given Preference (Sec. 9.1.4)
23

---

Goal 9.1.1 Generate Candidate Preferences
The LLM first analyzes the demonstrations in detail before proposing candidate preferences that can explain user behaviors in the demonstrations. We gpt-4 for these prompts because this is complex task.
System Message
You are an assistant who sees someone demonstrating how the fridge is organized and analyzes patterns in the demonstrations that you can + infer the potential preferences that the user might have.
Instruction
# Input
You are given 2 demonstrations that show before and after when set - of objects gets put into the fridge. For each demonstration:
"Objects that got put away" describes the objects that user will + demonstrate how they would like to put in the fridge.
"Initial state of the fridge" describes the objects that are initially
+ in the fridge before the user starts the demonstration.
"Final state the fridge" describes what the fridge looks like after
+ the demonstration. All the objects in "Objects that got put away"
+ should be in the fridge now.
Your goal is to fill the reasoning worksheet below that analyze the + demonstrations and help you infer the preferences that the user could have.
The demonstrations are complementary, and they are shown by the user with the same preference. Thus, even though they are different, their difference is due to the difference in the initial state of the fridge and the objects that got put away. They are not contradictory, and there exist preferences that can explain why the demonstrations are + the way they are.
# Instructions and useful information
## Specific locations the fridge
The fridge can be segmented into 3 shelves (top shelf, middle shelf, bottom shelf); each shelf has 2 sides (left side, right side). The
+ fridge has 6 specific locations:
left side the top shelf, right side the top shelf
left side of the middle shelf, right side of the middle shelf
left side of the bottom shelf, right side of the botton shelf
## General locations in the fridge
general location contains multiple specific locations. There
general locations:
"left side of fridge" contains: "left side of top shelf", "left side
+ of middle shelf", or "left side of bottom shelf"
"right side of fridge" contains: "right side of top shelf", "right
+ side of middle shelf", or "right side of bottom shelf"
"top shelf" contains: "left side top shelf", "right side of top
+ shelf"
"middle shelf" contains: "left side middle shelf", "right side of
+ middle shelf"
"bottom shelf" contains: "left side bottom shelf", "right side of
+ botton shelf"
## Details about the preferences that you need to output
preference is short paragraph that specifies requirements for each
4 category of grocery items. There must be least requirement for

---

7 each category. The type of requirement for each category can be
different. The categories are: "Fruits", "Vegetables", "Juice-and-soft -drinks", "Dairy-Products", and "Condiments".
The requirement needs to be one of the following:
**Type-1. Specific Locations. These represent that the object must +
place at this specific location. The options are:
"left side of top shelf"
"right side of top shelf"
"left side of middle shelf"
"right side of middle shelf"
"left side of bottom shelf"
"right side of bottom shelf"
**Type-2. General Locations. ** These are vaguer locations that contain multiple specific locations. The options are:
"left side of fridge": which means that the user is ok if the object is placed on "left side of top shelf", "left side of middle shelf", + or "left side of bottom shelf"
the object - is placed on "right side of top shelf", "right side of middle shelf",
"right side of fridge": which means that the user is
or "right side bottom shelf"
"top shelf": which means that the user is ok if the object is either placed on the "left side top shelf" or "right side top shelf" "middle shelf": which means that the user is ok if the object is either placed on the "left side middle shelf" or "right side
middle shelf"
"botton shelf": which means that the user is ok if the object is 4 either placed on the "left side of bottom shelf" or "right side of bottom shelf"
* Typee- Relative Positions.** ** The options are:
"<category> must be placed together next to existing <category>
regardless of which shelf they
on.". ": This means that the
- only cares that a category of objects are placed together next to existing objects of the
type. The user does not which specific shelf or which side of the shelf the objects are placed on. For example: "fruits must be placed together next existing fruits A regardless which shelf they are on."
the same shelf next <another category + of objects>, and which specific shelf does not matter.". ": This means objects are placed that the user only cares that a category category of objects. It
" "<category> must be placed
4 together on the same shelf next to another
does not matter which specific shelf the objects are on. For example: "fruits must be placed on the same shelf next to condiments, and which specific shelf does not matter."
In addition to giving specific requirements for each category of grocery + items, sometimes you may choose to add additional requirements. The options are:
*+ype-44 Exception For Attribute**
"<category> needs to be placed at <specific location 1>, but < attribute of category> needs to be placed at <specific location 2>.": This means that the user has a different specific requirement for object with certain attribute compared to the main category. An attribute includes a subcategory of the object, the size/veight of For example,
the object, specific feature of the object,
Dairy product needs to be placed at the right side of top shelf, but
cheese needs to be placed at left side of middle shelf." Here, "dairy product" is the main category, and "cheese" is the attribute, which
is specific type dairy product. Another example is, "Fruits
25

---

- needs to be placed at the left side of middle shelf, but big fruits
needs placed right side of bottom shelf. " Here, "fruits" is
- the main category, and "big fruit" is the attribute, which is about
the size the fruit.
** ** Conditional On Space**
"If there are less than <N> objects at <priminary specific location>,
+ want <category> to be placed at <priminary specific location>. Else
4 I want <category> to be placed at <second choice specific location
4 > This means that there is maximum number of objects that be
7 placed at top choice specific location. If that top choice specific location does not have less than N number of objects, the user wants
+ the a category of objects to be placed at the second choice specific the right side of top shelf, want dairy products to be placed at right side of top shelf. Else, want dairy products to be placed at left side
location. For example, "if there are less than 3 objects
4 of middle shelf."
Below the reasoning worksheet and output that you must follow. + contains parts that you must fill out and instructions on how to fill out that part.
Reasoning worksheet
## (Fruits) Reasoning about fruits
### (Fruits. Summarize where fruits are in the demonstrations
In demonstration 1:
Fruits initially in the fridge:
<You must write down an unordered list of fruits that are initially + in the fridge. For each fruit, you must write down the specific
locations it occupy, and what other grocery items next to it> Fruits that got placed in the fridge:
<You must write down unordered list of fruits that initially + in the fridge, and the specific locations they occupy>
<You must copy the same format and follow the process for + demonstration 2.>
### (Fruits. B) When fruits are in the fridge, are they only appearing in one specific location? If that is the case, what is that specific
+ location?
<You must look the summary you wrote in (Fruits.A) You must carefully analyze each fruit's location in verbose detail. If you answer yes, you must write down the specific location that fruits are placed at.>
### (Fruits.C) C) Only if you answered no in (Fruits.B), you should answer this section. When fruits appear at different locations in the fridge, you must group the fruits based on their specific locations they appear at.
<Based on the summary you wrote in (Fruits.A), you must write down json
(a dictionary) that map each specific location to fruits at that + specific location> json
"demonstration 1":
"<ppccific__ocation__ ; [<You must write down the fruits at specific.loccation_. as list of strings>],
"<specific_locatioo. :> [<You must write down the fruits at
specific_locationn_
list of strings>], <You must do this for each specific location that appeared in your summary A))
26

---

},
<You must do this for each demonstration.>
} ...
### (Fruits.D) Potential preferences for fruits
#### (rruits.D.1) **Type-1. Specific Locations. <Based on section (Fruits.B), what are some specific locations that the user might like the fruits to be placed at?>
#*## (Fruits.D.c **Type-2. Group Locations.**
<Based on the description of group locations explained at **Type-2.
General Locations. and the specific locations in mentioned in Fruits. A) and Cfruits.C,, what are some general locations that the user might like the fruits to be placed at? For example, if the specific locations is "left side of top shelf", there are two possible general locations for fruits: "top shelf" or "left side the fridge You must write down your reasoning in verbose detail, and you must specify general locations that fruits might in.>
#### (Fruits. D.3) **Type-3. Relative Positions.
<Based on the section Ar) what type of objects are fruits typically placed next to? What kind of objects you think user might like the fruits to be placed next to? You must write down your reasoning verbose detail.>
#*## (Fruits. 4.4) **Type-4. Exception For Attribute.**
<Only if your answer in (Fruits.B) is no, you should proceed to answer this section. Based on the json in (Fruits.C). for each specific location, what kind of attribute can you identify the fruits at that specific location share? You must write down your reasoning in verbose detail.>
#### (Pruits.D.55 **Type-5. Conditional On Space.
<Only if your answer in (Fruits.B) is no, you should proceed to answer this section. Based on the json (Fruits. C), for each specific location, what kind of conclusion can you make about the number of
objects there? What is primary location that the user likes to put fruits at? What is secondary location that the user would put fruits at? What is the condition for user to go from putting fruits at the primary location to the secondary location?>
<You must now copy from ## (Fruits) Reasoning about fruits to #### ( Fruits. D. **Type-5. Conditional Space. ** and repeat reasoning
process for the rest of the categories: ## (Vegetables) Reasoning
about vegetables, ## (Iuice-and-soff-drrink Reasoning about juice and
soft drinks, ## (Dairy-Products) Reasoning about dairy products, ## ( Condiments) Reasoning about condiments. You must write down potential preferences for all 5 categories. You cannot omit any category and you must write down detailed reasoning for all of them.
System Message
You are assistant who sees an analysis of the demonstrations that the
+ user have provided and propose potential preferences that the user might
+ have.
Instruction
# Input
You are given a reasoning worksheet written markdown formatthat has
4 already carefully analyzed the demonstrations that the user has
27

---

provided to represent their personal preference. You must pay close attention to this reasoning worksheet.
Goal
Your goal is to generate <num_ taum.prfference> fundamentally different and diverse preferences that are consistent with the reasoning worksheet and explain what the user want.
Instructions and useful information
## Specific locations in the fridge
The fridge be segmented into 3 shelves (top shelf, middle shelf,
+ bottom shelf); each shelf has 2 sides (left side, right side). The
fridge has 6 specific locations:
left side of the top shelf, right side of the top shelf
left side of the middle shelf, right side of the middle shelf
left side of the bottom shelf, right side of the botton shelf
## Details about the preferences that you need to output preference is short paragraph that specifies requirements for each 7 category of grocery items. There must be at least one requirement for each category present the demonstrations. The type requirement for each category can be different. The categories are: "Fruits", "Dairy-Products", and
Vegetables", "Juice-and-sfft-drine Condiments".
If no objects from given category are present any of the demonstrations, you should omit that category from your planning. Do not randomly choose a rule. Your preferences should also not re- explain what the categories are include.
The requirement needs to be of the following:
**Type-1. Specific Locations. These represent that the object must
place this specific location. The options
"left side of top shelf"
"right side of top shelf"
"left side of middle shelf"
"right side of middle shelf"
"left side of bottom shelf"
"right side of bottom shelf"
**Type-2. General Locations. ** These are vaguer locations that contain multiple specific locations. The options are:
"left side of fridge": which means that the user is ok if the object placed on "left side of top shelf", "left side of middle shelf", - or "left side of bottom shelf"
the object + is placed on "right side of top shelf", "right side of middle shelf",
"right side of fridge": which means that the user is
or "right side of bottom shelf"
"top shelf": which means that the user is ok if the object is either placed on the "left side top shelf" or "right side top shelf" "middle shelf"; which means that the user is ok if the object is either placed on the "left side of middle shelf" or "right side of middle shelf"
"bottom shelf": which means that the user is ok if the object is either placed the "left side of bottom shelf" or "right side of + bottom shelf"
Typee- Relative Poaitions.** The options are: "<category> must be placed together next to existing <category>
regardless of which shelf they on.". "; This
that the
+ only cares that category of objects are placed together next to
existing objects of the same type. The user does not care which
28

---

"If there are less than <N> objects at spriminary specific location>, want <category> to be placed <priminary specific location>, Else + I want <category> to be placed at <second choice specific location - specific shelf or which side of the shelf the objects are placed on. For example: "fruits must placed together next to existing fruits regardless which shelf they are on."
"<category> must be placed on the same shelf next to <another category - of objects>, and which specific shelf does not matter.". This means that the user only cares that a category of objects are placed together the same shelf next to another category of objects. It
does not matter which specific shelf the objects are on. For example: shelf next to condiments, and which specific shelf does not matter."
"fruits must be placed the
In addition to giving specific requirementa for each category of grocery items, sometimes you may choose add additional requirements. The options are:
**ype--4 Exception For Attribute**
"<category> needs to be placed at <specific location 1>, but < attribute of category> needs be placed <specific location 2>.": + This means that the user has a different specific requirement for main category. An 4 attribute includes subcategory of the object, the size/weight of the object, specific feature of the object, etc. For example,
object with certain attribute compared
Dairy product needs to be placed at the right side of top shelf, but cheese needs to placed at left side of middle shelf." Here, "dairy product" is the main category, and "cheese" is the attribute, which specific type dairy product. Another example is, "Fruits
needs to be placed at the left side of middle shelf, but big fruits
needs to placed at right side of bottom shelf." Here, "fruits" is
+ the main category, and "big fruit" is the attribute, which is about
4 the size of the fruit.
**yee- Conditional On Space**
4 This means that there is maximum number of objects that be + placed at top choice specific location. If that top choice specific have less than number of objects, the user wants 4 the a category of objects to be placed at the second choice specific the right side of top shelf, I want dairy products to be placed at right side of top shelf, Else, want dairy products to placed at left side 4 of middle shelf."
location does
location. For example, "if there are less than 3 objects
Below is the output that you must follow. < contains parts that you must fill out and instructions on how fill out that part. You must only copy and output what is under # Potential Preference Generation Process.
Potential Preference Generation Process.
## (I) Propose potential preferences based reasoning worksheet <You must write <num.preferences> potential preferences that are diverse
+ in the requirements for each category. The potential preferences
+ CANNOT only differ in the wording they have. They must be fundamentally different with different requirements for at least object.
### (I.1) Potential preference
#### (I.1.a) What type of requirements are you picking for each category?
You must pick different types of requirements compared to previous preferences, so how are types that you picked different from previous ones?
29

---

<For each category, you must pick at least one or more type of
requirements based on your answers (Fruits.D), (Vegetables.D) .D) ( Juice-and-soft-
, (Dairy-Products. D), (Condiments.D). The type of requirements for each category can be the same, but they do not have to be the same for all categories. For each category, you must verbosely explain in detail what is the type of requirement that you picked and what exactly is the preference for that category. If this is the first potential preference that you are writing, you not
need to answer how the types that you picked are different from previous preferences. However, for subsequent potential preferences that you will write, you must make sure that you are picking
different type of requirements for the categories. You must explain different from previous preferences. You must cite the previous potential preferences by their number, for example (I.1.b).>
in detail how the types that you pick
#### (I.1.b) Preference
<Based on the types of requirements that you picked in (I.1.a), your goal paragraph a potential user preference that consistent with the demonstrations and the reasoning worksheet. You must write in natural langauge and there must be at least one requirement for each category. You must not make reference to other potetial preferences or reasoning that you have written. Each preference be understandable on their own without referring to another
is to write
potential preference.
<You must now copy from ### (I.1) Potential preference and repeat the writing process for the remaining <num_preferences sncefferenee_mmnus__oo _nne>> potential preferences. You must write <num potential
preferences in total.>
(II) Final valid JSON output
You must copy the <num <num_preett pemepreffeeness>>> diverse potential preferences that list of strings in valid json format. You
you wrote in (II) must only copy the preference, which must not refer to other preferences parts of the writing process. Each preference must define requirements for all the categories, and it must be understanable and interpretable
its own without referring other potential preferences or other writing that you have done before. Each preference should You must make sure that when writing each preference, you do not use single quotes or double quotes or citations
json
"<You must copy the first potential preference in (I.1.b) here. For + subsequent potential preferences, you must not make reference to
+ previous potential preferences.>",
<You must copy the remaining <num,preferences, pngreferrncee_mmmus_ommm minus__on>> preferences here>
]
9.1.2 Calculate the Reward
The LLM is prompted to output a json yes/no on whether an object's semantic placement location plan satisfies given preference 0. The reward is calculated by the ratio between the sum of the responses (yes is and no is 0) and the total number of objects in the plan. We gpt-4o for this prompt.
30

---

System Message
You are roleplaying as user with specific preference. Your goal is to + answer whether not an object placement has satisfied your preference + and the initial configuration of the fridge.
Instruction
You are roleplaying as user with specific preference. You are shown 4 the objects that are already initially in the fridge. The fridge has different shelves, which divided into smaller sub-sections. The objects in a specific section are listed from left to right.
You are also shown plan to place an object at specific part of the fridge. Your goal is to answer "yes" "no" whether the object + placement has satisfied your preference. You must also pay attention to objects that are already initially in the fridge because where you want objects placed could depend that initial condition.
You must respect the following rules when you make your decision:
fridge. You + must not make your decision based on you judging if shelf is already
not need worry about the amount space full.
If the preference for a category is specific location, you must only check
object is being placed at that specific location.
+ must ignore other objects that are at that specific location.
For example, consider when your preference is to put drinks together regardless of the shelf and to put fruits the left side of top shelf, and the plan has placed a drink on the left side of the top shelf then a fruit also on the left side of top shelf. When you answer whether the plan of placing the fruit on the left side of top shelf, you should answer "yes" because it satisfies the specific + placement location for fruit.
If the object placement plan is more general than your preference, that + plan does not satisfy your preference, and you must reply "no". For example, if your preference is to put fruits the left side of
- the middle shelf, and the plan tries to put a fruit on the middle shelf, the plan is too general, and it will not always satisfy your - preference. Thus, you must reply "no".
When your preference has conditionals, you must carefully check if that conditional is true first when you make your decision.
your preference wants vegetables + if there are already N objects at location 1, you must check the objects that are already intially in the fridge
For example,
location 2
make sure that + the condition (N objects at location 1) is true.
When your preference depends on initial condition the fridge, + you must carefully check what is desirable placement location based on the inital condition
fridge.
For example, if your preference wants vegetables to be placed together you must check if there are already vegetables the fridge. there are already vegetables at the right side of top shelf, the object placement plan should also place new objects at the right side of top shelf.
The preference, the objects already initially in the fridge, the object + placement plan that you receive will be in markdown format: # Your preference
Your preference will be stated here
# Objects already initially in the fridge
31

---

The objects already initially in the fridge will be stated here as an json
# Object placement plan
"..
... The plan will be stated here python code: pickandplace ( 4 +bbject_to_placet placement_location))
You must reply in valid json format, each item corresponds to of + the pickandplace() () function in the object placement plan. You must
only double quote. You cannot apostrophy single quote. See
below for the format that you must follow.
jjson
"Reasoning":
"(You must not use any more of quotation mark inside your resoning. must not write apostrophe, double quote, or
+ single quote. You must put your reasoning on whether user with your preference would happy with the object placement given
+ the objects already initially in the fridge.}",
"Does this plan satisfy your preference (yes/no)": "[you must only
+ reply yes or no}"
}. ... ...<there should be one dictionary per line pickandplace()
]
..r
Example Input
Your preference
like putting dairy the top shelf and vegetables the left side of the bottom shelf.
Objects already initially in the fridge
"top shelf":
"left side of top shelf": D "right side of top shelf": ["cheese", "yogurt"]
},
"middle shelf":
"left side of middle shelf": 0] "right side of middle shelf": []
}.
"bottom shelf":
"left side of bottom shelf": ["carrot"], "right side of bottom shelf": []
}
<<c
Object placement plan
pickandplace ( (ooat milk", "right side of top shelf")
pickandplace (wwhole milk", "right side of middle shelf") <er
Example Response
32

---

json
like fruits on the left side middle shelf, and vegetables on "Reasoning": "Oat milk dairy product. prefer dairy on the + top shelf. The plan was to put oat milk on the right side of top shelf, which falls under top shelf, so the plan satisfies my preference.",
"Does this plan satisfy your preference (yes/no)": "yes"
"Reasoning": "Whole milk is dairy product. prefer dairy
+ the top shelf. The plan was to put whole milk on the right
side of middle shelf, which is not top shelf, the plan + does not satisfy my preference.",
"Does this plan satisfy your preference (yes/no)": "no"
}
] <<<
Example Input
# Your preference
> the right side of the middle shelf.
Objects already initially in the fridge
"top shelf":
"left side of top shelf": ["whole milk"], "right side of top shelf": []
},
"middle shelf":
"left side of middle shelf": D
"right side of middle shelf": [
}.
"bottom shelf":
"left side of bottom shelf": 0, "right side of bottom shelf": []
<cc
Object placement plan
'cr
pickandplace ("applee, "middle shelf")
Example Response
" 'json
"Reasoning": "Apple is fruit. prefer fruits the left side + of the middle shelf. The plan is placing apple in a more general location: middle shelf. This means that it could
+ potentially place apple not specifically at the left side of
the middle shelf, so the plan does not satisfy my preference
+ .",
"Does this plan satisfy your preference (yes/no)": "; "no"
33

---

}
] <c.
Example Input
# Your preference
prefer vegetables be placed together. I also want fruits on the middle shelf.
Objects already initially in the fridge <
"top shelf":
"left side of top shelf": 0 "right side of top shelf": 0
},
"middle shelf":
"left side of middle shelf": D "right side of middle shelf": ["cucumber", "carrot"]
},
"bottom shelf":
"left side of bottom shelf": 0 "right side of bottom shelf": 0
Object placement plan
pickandplace ("appinaccce"aiaachh.. ("spinach", "right side of top shelf")
Example Response "json
"<*
"Spinach is vegetable. I want vegetables to be + placed together, so need to check if there are vegetables initially the fridge. There are vegetables (cucumber,
"Reasoning":
carrot) at the right side of middle shelf, so new vegetables
should also get placed at the right side of middle shelf.
However, the plan decides to place spinach to the right side the top shelf, which is not right side middle shelf. + Thus, this plan does not satisfy my preference.", "Does this plan satisfy your preference (yes/no)": "no"
}
]
Example Input
Your preference
If there are less than 2 dairy product on the right side of top shelf,
you can put dairy products the right side of top shelf; else, I want the rest of the dairy product to be on the right side of middle shelf.
+
Objects already initially in the fridge
<<c
34

---

"top shelf":
"left side of top shelf": D, "right side of top shelf"; ["oat milk"]
"middle shelf":
"left side of middle shelf": D], "right side of middle shelf": [
}, "bottom shelf":
"left side of bottom shelf": 0 "right side of bottom shelf": []
}
}
Object placement plan
<c.
pickandplace ("whole milk", "right side of middle shelf")
'<"
<<' Example Response
jesjon
"Reasoning";
"Whole milk a dairy product. My preference has a condition: whether there are less than 2 dairy products
the right side of top shelf, so I must pay close attention to
the objects initially in the fridge. There is 1 dairy
product (oat milk), so the condition (less than 2 dairy
product the right side of top shelf) is true. My
preference would want whole milk also on the right side of top shelf. However, the plan put whole milk right side of middle shelf, so it does not satisfy my preference.", "Does this plan satisfy your preference (yea/no)": "no"
] <cc
Example Input
# Your preference
If there are less than dairy product on the right side top shelf, - you can put dairy products on the right side of top shelf; else, I on right side of middle want rest of the dairy product shelf.
Objects already initially in the fridge
<<r
toop shelf":
"left side of top shelf": 0]
"right side of top shelf": ["oat milk", "whole milk"]
},
"middle shelf":
"left side of middle shelf": D] "right side of middle shelf": []
35

---

"bottom shelf";
"left side of bottom shelf": 0
"right side of bottom shelf": 0]
}
Object placement plan
<..
pickandplace ("cheese", "right side of top shelf")
"cr
Example Response
" jsonn
"Reasoning": "Cheese is dairy product. My preference has + condition: whether there are less than 2 dairy products on the right side of top shelf, so must pay close attention to the objects initially in the fridge. There are 2 dairy
product (oat milk, whole milk), so the condition (less than 2
dairy product on the right side of top shelf) is false. I
should look at the else case my prefernece, which wants
+ the cheese to be on the right side of middle shelf. However,
the plan put cheese on right side of top shelf, so does
not satisfy my preference.",
"Does this plan satisfy your preference (yes/no)": "no"
]
Example Input
Your preference
I prefer vegetables be placed on the right side of middle shelf. I want fruits the right side of top shelf, condiments the left side of top shelf.
Objects already initially in the fridge
"top shelf":
"left side of top shelf": 0] "right side of top shelf": [
},
"middle shelf":
"left side of middle shelf": D. "right side of middle shelf": ["cucumber", "carrot", "corn"]
}.
"bottom shelf":
"left side of bottom shelf": 0] "right side of bottom shelf": 0
<<c
Object placement plan
<cc
36

---

pickandplace (weaaaccesssppiiie ("spinach", "right side of bottom shelf")
Example Response
" jssoo
"Spinach is vegetable. I prefer vegetables the right side of middle shelf. The plan is placing spinach at the wrong location: right side of bottom shelf. Thus, this + plan does not satisfy my preference.""
"Reasoning":
"Does this plan satisfy your preference (yes/no)": "no"
}
]
<<<
Example Input
# Your preference
prefer vegetables be placed together next exsiting vegetables
+ regardless of which shelf they are on. want fruits on the right
side of top shelf.
Objects already initially fridge <<"
"top shelf":
"left side of top shelf": D. "right side of top shelf"; ["cucumber"]
}.
"middle shelf":
"left side of middle shelf": [] "right side of middle shelf": [
},
"bottom shelf":
{
"left side of bottom shelf": 0
"right side of bottom shelf": 0]
}
<cc
Object placement plan
<c"
pickandplace("pphachh "right side of top shelf")
pickandplace ("apple", "right side of top shelf")
"cr
Example Response
"<
jsson
"Reasoning":
"Spinach is vegetable. I want vegetables to be placed together, so need to check if there are vegetables initially in the fridge. There
vegetables (cucumber) at + the right side of top shelf, so new vegetables should also
get placed at the right side of top shelf. Since the plan put
+ spinach at the top shelf, it has satisfied my preference.",
"Does this plan satisfy your preference (yea/no)": "yes"
},
37

---

"Reasoning"; "Apple is fruit. I prefer fruits on the right side of top shelf. The plan is placing apple right + side of top shelf, which matches the requirement and + satisfies my preference.",
"Does this plan satisfy your preference (yes/no)": "yes"
]
9.1.3 Generate Candidate Questions
The LLM receives input pair of preferences, the corresponding plans for these preferences, the reward of each plan given these preferences, the initial condition, and the tasks that these plans are generated to solve. We use gpt-4o for this prompt.
System Message
You are assistant that ask informative yes-or-no questions to try to differentiate between two potential preference candidates.
Instruction
Goal
You trying learn what is user's personal preference. You receivea new initial state that you are trying to solve, two possible candidates of the user's preference, the corresponding task plan generated based on the candidate preferences.
Your goal is to analyze the differences between the two candidates and their plans and ask iuum_qestioons> informative, effective yes-or-no questions such that the user's s answers to your questions can reveal which candidate is closer the user's preference.
Input
You will receive the input in the following format:
New Initial Condition Solve
## Objects initially in the fridge < jsson
The objects already initially in the fridge will be stated here json
## Objects that must be put away into the fridge
< json
... The objects that need to be put away into the fridge will be stated + here as a list
Preference Candidate
The first candidate will be stated here
## Plan 1 based on Preference Candidate 1
The plan will put away the objects that need be put away
### Preference Candidate 1's Score on Plan
Numerical score here. The higher a score, the better Plan 1 is at satisfying the preference candidate 1...
### Preference Candidate 2's Score on Plan 1
Numerical score here. The higher a score, the better Plan 1 is at satisfying the preference candidate 2
Preference Candidate 2
38

---

Your questions must not be about overall objects or items. You must never questions like "Do you prefer to items to be ...?" or "Do The second candidate will be stated here
## Plan 2 based on Preference Candidate 2
The plan will put away
objects that need to put away
### Preference Candidate 1's Score on Plan 2
Numerical score here. The higher score, the better Plan 2 is at satisfying the preference candidate 1...
### Preference Candidate 2's Score Plan 2
Numerical score here. The higher a score, the better Plan 2 is at satisfying the preference candidate 2
You must follow these rules when coming up with questions ask:
You must analyze the difference in the candidate preferences and their plans. The differences will inform you what question to ask. The 4 scores can inform you how much each plan is liked by each preference candidate. The scores can help guide you understand the main
4 difference in the candidate preferences.
You must ask questions about the preferences. You must never
+ question about that mention the new initial condition to solve or the
plan that got generated.
Your questions must just be about what the user prefer. They must be
the user can just look at the question and be able + to answer that question.
standalone so
You must
effective, useful questions where user's answer will help you figure out which preference candidate is better.
You must ask simple yes or no questions.
Your questions cannot ask if the user prefers B. However, you
+
ask if the user prefers A rather than B.
Your questions should try
wording terminology that also used in the candidate preferences.
+
Your question must be about specifc categories attributes under specific category. The categories are for example: ["fruits", vegetables", "dairy product", "condiments", "juice and soft drinks"].
+
Some examples of attributes are: "heavy fruits", which is an attribute under "fruits"; "sweet conditments" which is attribute under condiments"; "soft drinks", which is an attribute under "juice and soft drinks".
+ you prefer items of the same category ...?"
Your questions must not be asking about specific objects. You must
+ never ask questions like "Do you prefer chocolate milk to be placed at specific category (e.g
...?" Instead, you can ask questions about
"Do you prefer dairy products to be placed at ...?" "d specific attribute (e.g. "Do you prefer milk to be placed at", where "milk" the specific attribute.)
You must reply in this format: Reasoning
You should analyze the differences between preference candidates and their plans. You should strategize what questions you would ask here
# Questions
You must put your enum_questions> questions as unordered list here
39

---

9.1.4
Answering Question Given a Preference
The LLM is asked roleplay user who has preference 0 answering the question g. We gpt-4o for this prompt. This prompt is used to:
Estimate the likelihood P(olg, 0): We use OpenAl's API, which outputs the log-likelihood of outputting a specific token, to extract the log-likelihood of the LLM outputring "yes" or "no".
Mimic the user's answer with ground-truth preference: When the interactive approaches need query the user, the LLM supplied with the ground-truth preference (not known the approaches) answer the question that an approach asks.
System Message
You are roleplaying as user with specific preference. Your goal is to + answer yes-or-no question based on the preference of the user whom you are roleplaying. When you put your answer in the json, you must only
+ write "yes" "no"
Instruction
You are roleplaying as user with specific preference. You are asked + one yes-or-no question, and you goal is answer "yes" or "no" based on your preference.
The preference and the questions that you receive will be in markdown 4 format:
# Your preference
Your preference will be stated here
# Question for you to answer
The question will be stated here unordered list
You must follow these rules when answering:
When you write down your reasoning, you must not use apostrophy, single
quote, or double quote.
You must only answer "yes" or "no". Even if it does not apply to your preference or you are not sure, you must answer "yes" or "no". You cannot answer anything else. You must not write "N/A" "na" or "n/a
+ ". You must only output "yes" or "no".
If the question is asking whether the user prefers specific thing,
+ but the user's preference is more general, the user will likely respond because they do not care about the specific thing and they
+ only care about the general requirement.
If the question is asking whether the user prefers general thing, but
+ the user's preference is more specific, the user will likely respond
+ because the general thing can include cases that violate the user's specific requirement.
You must reply in a json, which includes your reasoning process and to question. You must follow this format:
"Reasoning": "{put your reasoning on user with your preference + would answer to this question. You must not use single quote or double quote inside your reasoning. }",
"Answer (yes/no)": "[you must only reply yes or no)"
Example Input
Your preference
Fruits should be placed on left side of the fridge.
Questions for you to answer
40

---

Do you prefer fruits to be placed on the left side of the top shelf?
Example Response
on left side the + fridge, which is general requirement. The user does not specifically perfer fruits to be on the left side of the top shelf, the user would answer no.", Answwr (yez/no)": "no"
"Reasoning": "The user prefers fruits
Example Input
# Your preference
Fruits should be placed on left side of the fridge.
Questions for you to answer Is it important to you have fruits specifically the left side of + the middle shelf rather than just on the right left of the fridge? Example Response
on left side the - fridge, which is general requirement and the user will not care about the specific shelf that fruits are placed on long as it the left side of the fridge. Thus, it is not important for fruits user would
"Reasoning":
"The user prefers fruits
-
on left side of the middle shelf, - answer no.",
"Answer (yes/no)); "no"
Example Input
Your preference
Fruits should be placed on left side of the top shelf.
Questions for you to answer Do you prefer fruits to be placed the left side of the fridge? Example Response
"The user prefers that fruits are on the left side of the top shelf, which is specific. Left side of the fridge includes + left side of the middle shelf and left side of the bottom shelf, which does match preference of the user. The user would
"Reasoning":
+ no.",
"Answer" (yes/no)": "; "no"
Example Input
# Your preference
Fruits should be placed on left side middle or bottom shelf.
Questions for you to answer Do you prefer to place fruits next to other fruits already in the fridge? Example Response
"Reasoning": "The prefers that fruits are placed specific locations (left side of the middle or bottom shelf). The question is more explicitly asking if the user prefers fruits to be placed next to other fruits already in the fridge, which does not have specfic locations requirement, so the user would say no.".
Answer (yes/no)": "no"
Example Input Your preference
41

---

Fruits should be placed on the left side of the middle or bottom shelf.
# Questions for you to answer you prefer to have similar items grouped together on the same shelf
- Example Response
"Reasoning": "The user prefers that fruits are placed at specific locations (left side of the middle bottom shelf). The question + is implying that the user only cares about similar items being on the same shelf without specific requirements which shelf and which side of the shelf, so the user would answer no.",
AAnswr (yes/no)": "no"
Example Input
# Your preference
want fruits to be placed together next fruits that are already in + the fridge.
# Questions for you to answer you prefer to place fruits next to other fruits already the - fridge? Example Response
"The user prefers that fruits are placed together next to fruits that are already in the fridge. The question also asks if the user prefers to place fruits next to other fruits already in + the fridge, the anawer is yes.", "Answer (yes/no)"; "yes"
"Reasoning":
Example Input
Your preference
Fruits should be placed on the left side of the middle shelf.
Questions for you to answer
Do you prefer if fruits are placed specifically on the middle shelf? Example Response
"Reasoning":
"The user prefers that fruits are placed at specific locations (left side of the middle shelf). The question is asking
+ about more general requirement because middle shelf is more
general than the left side of middle shelf. Based on the rules
+
above, the general location contains locations that violate the user's preference (right side of the middle shelf), so answer is no
"Answer (yes/no)): "; "no"
Example Input
Your preference
Fruits should be placed on the left side of the middle shelf. Vegetables should be the right side of middle shelf.
Questions for you to answer
Do you prefer to have items of the same category placed together on the same shelf?
Example Response
"Reasoning": "The user prefers that fruits are placed at specific locations (left side of the middle shelf) and vegetables are also
42

---

- placed at specific locations (right side of the middle shelf). The question asking there any category objects that needs to be placed together the same shelf. Since the requirements for fruits is about specific location instead putting fruits together, and the requirements for vegetables is also about a specific location instead of putting vegetables togetheer, the answer to this question is no", "Answer (yes/no)": "no"
Example Input
# Your preference
Fruits should be placed on the left side of the middle shelf. Condiments the right side of the top shelf. Vegetables should be placed 4 together next to other existing vegetables.
Questions for you to answer you prefer to have items of the same category placed together on + the same shelf?
Example Response
"The user prefers that fruits are placed specific locations (left side of the middle shelf), condiments are placed at vegetables need to be placed together. The question asks if there is any category objects that needs to be placed together on the same shelf. In this preference, vegetables need to be placed together the same shelf. Since there exists one category that are placed + together the same shelf, the anawer to this question is yes", "Answer (yes/no)": "yes"
"Reasoning":
specific locations (right side the top shelf),
9.2 Task Planner Prompts
The LLM prompt will generate semantic plan that satisfies the given user preferences. When it receives feedback, it will reflect on its plan and revise its original plan. We use gpt-4-turbo because we empirically find that gpt- go tends to make more mistakes (typos) when generating the code.
System Message
You are an assistant that comes up with a plan for putting items into a + fridge given list of items and human's preference.
Instruction
You must analyze human's preferences and then come up with plan to + put items into a fridge.
You will receive the following as input: Optional [Feedback]]
Objects:
Locations:
Initial State:
Preference:
where
"Feedback" appears if the previous plan was geometrically infeasible.
will contain items that did not fit the previous plan.
It is possible that the item could fit but the pickandplace function call was incorrect (e.g. misspelled item or location)
"Objects" is list of items that need to be placed in the fridge.
43

---

"Locations" is list of locations in the fridge.
"Initial State" a dictionary whose keys are the locations -
fridge and the values are list of items currently in that location.
"Preference" a description of the human's preferences for where
- they like things in a fridge.
The preference should always be satisfied, even when attempting to + place items that did not fit in the previous plan.
You must respond in the following format: Reflect:
# Reasoning:
pickandplace (iteml,, (item1, pickandplace (item2, location2)
location1)
where
"Reflect" should contain reasoning about the previous plan if it was + geometrically infeasible.
Reflect on what went wrong and how you plan to fix
The plan must abide by the human's preference.
"Reasoning" should contain the reasoning for your plan.
Reason about how best to place the items in the fridge based on the human's preference.
If the reflection involves repositioning items, ensure that the human's preference still satisfied.
"pickandplace (item, location)" is function call that places the item in the location in the fridge.
This is your plan of action.
Each time you prompted to generate new plan, the fridge is reset to + its initial state. Use this up with better plan. It is absolutely + necessary to satisfy the human's
to your advantage
preference; geometric infeasibility only suggests that objects should be + placed in different locations
that still satisfy the preference.
Example Input Objects: ["milk", "cheese", "apple", "orange"]
Locations: ["top shelf", "left side of top shelf", "right side of top shelf", "middle shelf", "left side middle shelf", "right side of 4 middle shelf", "bottom shelf", "left side of botton shelf", "right + side bottom shelf"]
Initial State: O
Preference: like putting dairy on top shelf and fruits on the + right side of middle shelf."
Example Response # Reasoning: Milk and cheese are dairy product, which based the preference needs to be on the top shelf. Apples and oranges are fruits, which needs to be the right side of middle shelf. pickandplace ("milk", ("milkkk""""""""" "top shelf")
pickandplace ("cheese", "top shelf")
pickandplace (arellacccappppppppa ("apple", "right side of middle shelf") pickandplace ("orange", "right side of middle shelf") Example Input
Objects: ["milk", "cheese", "apple", "orange"]
Locations: ["top shelf", "left side of top shelf", "right side of top shelf", "middle shelf", "left side of middle shelf", "right side of
+ middle shelf", "bottom shelf", "left side of bottom shelf", "right
side of bottom shelf"]
44

---

Initial State: {"left side of middle shelf": ["peach", "cherries"]} Preference: "Fruits must placed next to other exisiting fruits - regardless of which shelf they are on. Dairy products the be the right side of bottom shelf.
Example Response
Reasoning: Apples and oranges are fruits, which needs to be placed next to other exisiting fruits. Existing fruits in the fridge are peach" and "cherries", which are at the left side of middle shelf, so apple and oranges should be placed there well. Milk and cheese are diary product, which needs to be placed on the right side of botton shelf.
pi accandppaaeeeecc (aaaaccc"ppppp ("apple", "left side of middle shelf") pickandplace
(arolllacee""raaaaaa" ("orange", "left side of middle shelf") pickandplace
a"mmiilaaeemmmmm"" ("milk", "right side of bottom shelf") pickandplace ("cheese", "right side of bottom shelf")
Example Input
Objects: ("milk", "cheese", "apple", "orange"]
Locations: ("top shelf", "left side of top shelf", "right side of top shelf", "middle shelf", "left side middle shelf", "right side of + middle shelf", "bottom shelf", "left side of bottom shelf", "right + side of bottom shelf"]
Initial State: ("right side of top shelf": ("cabbage", "corn"]}
Preference: "Fruits must placed next to other exisiting vegetables 4 regardless of which shelf they are on. Dairy products on the be the right side of bottom shelf.
Example Response
Reasoning: Apples and oranges are fruits, which needs to be placed next to other exisiting vegetables. Existing vegetables in the fridge are "cabbage" and "corn", which are at the right side of top shelf, apple and oranges should be placed there well. Milk and cheese are diary product, which needs to be placed on the right side of botton shelf.
pickandplace aace aaallacce"""pee"" ("apple", "right side of top shelf") pickandplace ace cechhpppaleeeeraaaaeeea ("orange" "right side of top shelf") pickandplace ("miiiccee"mmm"" ("milk", "right side of bottom shelf") pickandplace (" ("cheese", "right side of bottom shelf")
Example Input
Objects: ["milk", "cheese", "apple", "orange"]
Locations: ["top shelf", "left side of top shelf", "right side of top shelf", "middle shelf", "left side middle shelf", "right side of 4 middle shelf", "bottom shelf", "left side of botton shelf", "right b side of bottom shelf"]
Initial State: ("left side of middle shelf": ["peach", "cherries"]} Preference: "Most dairy product should be placed on the left side of top shelf, but cheese product should be placed the right side of middle shelf. Fruits should placed on the left side of middle shelf."
Example Response
Reasoning: Cheese is cheese product, it needs to be placed at 4 right side of middle shelf. Milk is a dairy product, so it should be
+ placed the left side of top shelf. Apples and oranges are fruits, + which needs to be on the left side of middle shelf. pickandplace
("cheese", A"ckandplace(ccheeee"" "right side of middle shelf")
pickandplace ace ("milk", "left side of top shelf")
pickandplas ace ("apple", "left side of middle shelf") pickandplace ("orange", "left side of middle shelf") Example Input Objects: ["milk", "cheese", "apple", "orange"]
"leeft
45

---

Locations: ["top shelf", "left side of top shelf", "right side of top shelf", "middle shelf", "left side middle shelf", "right side of middle shelf", "bottom shelf", "left side of bottom shelf", "right side bottom shelf"]
+
Initial State: {"right side of bottom shelf": ["yogurt", "butter"]] Preference: "If the right side of bottom shelf has less than items, be placed there. Else, you must place them at the left side of top shelf. Fruits should placed on the left side of middle shelf."
+ dairy products
Example Response
# Reasoning: Right side of bottom shelf fit
dairy product, + so will put milk there. Since now there are 3 items at the right
side of bottom shelf, I must put cheese the left side of top shelf - Apples and oranges are fruits, which needs to be on the left side of middle shelf.
pickandplace ("milk", "right side of bottom shelf") pickandplace
(wt.lpalaaccccccseeeeeeeee ("cheese", "left side of top shelf") pickandplace (aaaaacceeaappa ("apple", "left side of middle shelf") pickandplace ("orange", "left side of middle shelf") Example Input
Objects: ["milk", "cheese", "apple", "melon"]
Locations: ["top shelf", "middle shelf", "bottom shelf"]
Initial State: {
"top shelf": ["yogurt", "butter"],
"middle shelf"; ["watermelon", "pizza box"]
Preference: "I don't want any of the fridge shelves to be too crowded.
Example Response
pickandplace ("milk", "top shelf") pickandplace ace
hwapopaaacceeccccceeee ("cheese", "top shelf")
aackandplace (aacce""pppll" ("apple", "middle shelf")
pickandplace ("melon", "middle shelf")
Example Input
Feedback: The previous plan geometrically infeasible. The items that - did not fit were ["melon", "apple"]. Objects: ["milk", "cheese", "apple", "melon"]
Locations: ["top shelf", "middle shelf", "bottom shelf"]
Initial State: {
"top shelf": ["yogurt", "butter"],
"middle shelf"; ["watermelon", "pizza box"]
}
Preference: don't want any of the fridge shelves to be too crouded." Example Response
Reflect: melon and apple did not fit in the previous plan. This must mean that the middle shelf is too crowded.
Reasoning: Instead of placing the apple and melon on the middle shelf, + they can be placed on the bottom shelf which is empty. pickandplace ("milk", ("mmillk "top shelf")
pickandplace cecppppacceccchhheehee ("cheese", "top shelf") pickandplace acee ("apple", "bottom shelf") pickandplace ("melon", "bottom shelf")
9.3 Baseline Prompts
9.3.1 Cand+LLM-Q/A Prompts
Cand+LLM-QU A/A use the prompts APRICOT to generate candidate preferences (Sec 9.1.1) and plans (Sec 9.2). Below is the prompt that it used determine whether to terminate, what is the best
46

---

question to ask if not terminating, and what is the best preference out of the candidate preference list terminating. We use gpt-4o.
System Message
You are a helpful, thoughtful assistant whose task is to select the user's s + preference from the demonstrations that the user provides and asking + users for yea/no clarification question. Instruction
# Input
You are giving
preferences, new initial condition of the + fridge that your generated preference will get used at, and chat log of what you have asked bo far.
## Preferences
You are given a list of preferences that show how the user wants their + fridge organized.
A preference is short paragraph that specifies requirements for each category grocery items.
There will be at least one requirement for each category. The type of
+ requirement for each category can be different.
## New initial state of the fridge
This is the new initial state that your preference will get used to determine how new set of objects will get placed into the fridge and satisfy the user's preference.
## Objects to put away
This is the list of object that will get put away in this new initial 4 state of the fridge.
## Chat log
This a json that contains history of your thought and your
+ conversation with the user.
When the "role" "thought" this your internal reasoning about
+ what you were going to do. The user never sees your thought.
When the "role" "assistant", this is the yes/no clarification question that you ask the user about.
When the "role" "user", this the yes/no answer that the user + provided to your yes/no clarification question.
## Can you still ask question?
There is a maximum number of questions that you can ask. This field will tell you if you still ask questions if you must terminate and generate your best guess at the user's preference.
# Goal
Your goal is select the best of the given preference in how the + grocery items should be placed into the fridge. You can do this inference by analyzing the given preferences and asking users yes/no clarification questions. Doing this process, you can choose betwwen 2
actions:
(1) Ask the user yes/no clarification question
(2) Select the index of the best preference from the and terminate
Instructions and useful information
## Set up of the fridge
The fridge can segmented into 3 shelves (top shelf, middle shelf, botton shelf); each shelf has 2 sides (left side, right side)
47

---

left side of the top shelf, right side of the top shelf left side the middle shelf, right side the middle shelf left side of the bottom shelf, right side of the bottom shelf
## Concrete steps that you must follow
To successfully determine the user's 's preference, you must do the following steps.
### Step 1: Reasoning about what actions to do
You can choose between 2 actions:
(1) Ask the user a yes/no clarification question
(2) Select the best preference, output its index, and terminate
You must follow these rules when you deciding what to do: If you are still uncertain about the preference, you should ask an informative yes/no clarification question.
can select which + you think the right preference is and terminate.
If you are 100% confident about the preference,
### Step 2: Take the action that you decided to You must reply in a valid json format: json
"terminate? (yes/no)": "<you must only reply yes or no>", "reasoning": "<you must write down the reasoning behind either why you generated specific question (if you choose the first action) or why you think the user's preference is what you are generating (if you choose the second action)>",
"question": "<you must only fill out this string if you chose action (1) ask yes/no clarification question. Otherwise, you must leave this as an empty string.>",
<you must only fill out this if you chose action (2) select index of best preference. Otherwise, you must leave 4 this as null.>
"index_of_best bew.roff__btttppefeeeeeeeee
}
<.r
#### If you chose (1) Ask the user yes/no clarification question in + Step 3
You must follow these rules when coming up with questions to ask: simple yes or no questions.
You must
Your questions cannot ask if the user prefers A or B. However, you can ask if the user prefers A rather than B.
Your questions must not be about overall objects or items. You must never ask questions like "Do you prefer to items to be ...?" or "Do you prefer items of the same category ...?"
Your question must about specifc categories or type of objects. The + categories are for example: ["fruits", "vegetables", "dairy product", "condiments", "juice and drinks"]
Your output must have siteee_o_beet_peetteeee
mull
Remember that you must reply in valid json format: " 'json
"terminate? (yes/no)": "no", "reasoning": "<you must write down the reasoning behind why you
+ generated a specific question (because you choose the first action)
>",
"question": "<you must fill out this string because you chose action (1)
ask yes/no clarification question.>",
48

---

"indee__f_bbsttprrfferre mull
} <er
#### If you chose (2) Select the best preference index
You must follow these rules.
terminate you should answer "yes" under the key 4 terminate? (yes/no)"
You must
You must output the index of the best preference, with referring to + the first listed preference
Remember that you must reply in valid json format: jsson
"terminate? (yes/no)": "yes",
"reasoning": "<you must write down the reasoning behind why you think the user's preference what you selecting (because you choose + the second action)>",
"question": "",
"index_ot of ofl ses__bes__p_____ post__preferenee. <you must output 0-indexed integer corresponding to the preference you picked from the given preference list because you chose action (2) select best preference.>
cc.
9.3.2 LLM-Q/A Prompts
LLM-Q/A only uses the prompt below to determine whether terminate, determine what is the best question ask terminating. and generate the preference based on demonstrations and its queries with the user if terminating. We gpt-4o.
System Message
You are a helpful, thoughtful assistant whose task select the user's + preference from the demonstrations that the user provides and asking users for yes/no clarification question.
Instruction
Input
You are giving list of preferences,
initial condition of the + fridge that your generated preference will get used at, and chat log
4 of what you have asked so far.
## Preferences
You are given a list of preferences that show how the user wants their + fridge organized.
A preference is a short paragraph that specifies requirements for each category of grocery items.
There will be at least one requirement for each category. The type of requirement for each category be different.
## New initial state of the fridge
This the new initial state that your preference will get used to
+ determine how a new set of objects will get placed into the fridge and satisfy the user's preference.
Objects to put away
This is the list of object that will get put away in this new initial state the fridge.
49

---

## Chat log
This a json that contains history of your thought and your
conversation with the user.
"thought"; this your internal reasoning about > what you were going to do. The user never sees your thought.
When the "role"
When the "role" is "assistant", this is the yes/no clarification
question that you ask the user about.
When the "role" "user", this is the yes/no answer that the user provided to your yes/no clarification question.
## Can you still ask question?
There is maximum number of questions that you can ask This field will tell you if you still ask questions if you must terminate and generate your best guess at the user's preference.
# Goal
Your goal is to select the best of the given preference in how the 4 grocery items should be placed into the fridge. You can do this
inference by analyzing the given preferences and asking users yes/no clarification questions. Doing this process, you can choose betwwen 2
actions:
(1) Ask the user yes/no clarification question
(2) Select the index of the best preference from the and terminate
Instructions ## Set up of the fridge
useful information
The fridge can segmented into 3 shelves (top shelf, middle shelf,
+ bottom shelf); each shelf has 2 sides (left side, right side):
left side of the top shelf, right side of the top shelf left side of the middle shelf, right side of the middle shelf left side of the bottom shelf, right side of the botton shelf
## Concrete steps that you must follow
To successfully determine the user's preference, you must do the following steps.
### Step 1: Reasoning about what actions do
You can choose between 2 actions:
(1) Ask the user a yes/no clarification question
(2) Select the best preference, output its index, and terminate
You must follow these rules when you are deciding what to do: you are still uncertain about the preference, you should ask an + informative yes/no clarification question.
you are 100% confident about preference, you can select which + you think the right preference is and terminate.
### Step 2: Take the action that you decided to do You must reply a valid jaon format: csjoo
"terminate? (yes/no)": "<you must only reply yes no>", "reasoning": "<you must write down the reasoning behind either why you generated specific question (if you choose the first action) why
+ you think the user's preference is what you are generating (if you choose the second action)>".
"question": "<you must only fill out this string if you chose action (1) ask yes/no clarification question. Otherwise, you must leave this an empty string.>".
50

---

"index_of_ Baabe__ff_bbtt_preeerrceee <you must only fill out this if you chose action (2) select index of best preference. Otherwise, you must leave
+ this mull.>
<.r
yes/no clarification question in #### If you chose (1) Ask the 4 Step 3
You must follow these rules when coming up with questions to ask:
You must ask simple yes or no questions.
Your questions cannot ask if the user prefers
B. However, you
+
ask if the user prefers A rather than B.
Your questions must not be about overall objects items. You must
+ never ask questions like "Do you prefer to items to be ...?" or "Do category ...?"
you prefer items of the
Your question must be about specifc categories or type of objects. The categories are for example: ["fruits", "vegetables", "dairy product", + "ccnddments", "juice and drinks"]
Your output must have sttee__o_best_prefeeeeee
mull
Remember that you must reply in valid json format: ''sjs
"terminate? (yes/no)": "no",
"reasoning": "<you must write down the reasoning behind why you + generated specific question (because you choose the first action)
+ >",
"question": "<you must fill out this string because you chose action (1) 4 ask yes/no clarification question.>", wtwpr_oo__estt_peeeeeeeeee null
}
#### If you chose (2) Select the best preference index
You must follow these rules.
You must now terminate you should answer "yes" under the key
+ terminate? (yes/no)"
You must output the index of the best preference, with referring to + the first listed preference
Remember that you must reply in valid json format: json
{
"terminate? (yes/no)": "yes",
"reasoning": "<you must write down the reasoning behind why you think + the user's preference what you are selecting (because you choose
+ the second action)>",
"question": "",
<you must output 0-indexed integer + corresponding to the preference you picked from the given preference list because you chose action (2) select best preference.>
"index_of" of wtwpr_ooooesttppeeeeeeeeeee
}
51

---

9.3.3 Non Von-Interactive Prompts
Mon-Interactive the prompl below generate preference from demonstrations, and uses the same prompt APRICOT to generate the task plan. We use gpt-4 because this is complex task to reason about all the demonstrations.
System Message
You are an assistant who sees someone demonstrating how the fridge is organized and summarizes that person's preference. Instruction
Input
You are given 2 demonstrations that show the before and after when set each demonstration:
+ of objects gets put into fridge.
"Objects that got put away" describes the objects that the user will demonstrate how they would like to put in the fridge.
"Initial state of the fridge" describes the objects that are initially user starts the demonstration,
the fridge before
"Final state of the fridge" describes what the fridge looks like after the demonstration. All the objects "Objects that got put away" should be in the fridge now.
Goal
Your goal is to generate one preferences that are consistent with the demonstrations and explain what the want.
Instructions and useful information
## Specific locations in the fridge
The fridge can be segmented into 3 shelves (top shelf, middle shelf,
4 bottom shelf); each shelf has 2 sides (left side, right side). The
fridge has 6 specific locations:
left side of the top shelf, right side of the top shelf middle shelf, right side the middle shelf left side of the bottom shelf, right side of the botton shelf
left side
## Details about the preferences that you need to output preference is short paragraph that specifies requirements for each
category of grocery items. There must be at least one requirement for each category. The type of requirement for each category can
different. The categories are: "Fruits", "Vegetables", "Juice-and-soft "Condiments".
-drinks", "Dairy-Products",
The requirement needs to be one of the following:
++yype-1. Specific Locations. ** These represent that the object must place at this specific location. The options are:
+
"left side of top shelf"
"right side of top shelf"
"left side of middle shelf"
"right side of middle shelf"
"left side of bottom shelf"
"right side of bottom shelf"
**Type-2. General Locations. These are vaguer locations that contain + multiple specific locations. The options are:
"left side of fridge"
"right side of fridge"
"top shelf"
"middle shelf"
"bottom shelf"
*+yype-3. Relative Positions.#* ** The options are:
"<category> must placed together next to existing <category> + regardless of which shelf they on."
52

---

"<category> must be placed on the same shelf next to <another category of objects>, and which specific shelf does not matter."
addition giving specific requirements for each category of grocery + items, sometimes you may choose to add additional requirements. The options are:
*+yype-4. Exception For Attribute#*
"<category" needs to be placed at <specific location 1>, but < attribute of category> needs to be placed at <specific location 2>." + An attribute includes a subcategory of the object, the size/weight of the object, specific feature of the object, etc. **ypee-5. Conditional On Space**
"If there less than <N>> objects <priminary specific location>,
+ want <category> to be placed at <priminary specific location>. Else
- I want <category> to be placed at <second choice specific location
4 >."
Now, you need to generate 1 preference as jsons.
Each preference must be in natural language.
Each preference must contain at least one placement requirement for each category of objects: "fruits", "regetables", "drinks", "dairy > product", and "condiments".
You must refer to the demonstrations and output the preferences in this + format 'sjoo
"reasoning": "<You must explain your thought process behind generating 4 this preference.>"
"preference": "<You must write the preference in natural language here>"
53


Reviews:
[{"review_id": "nyaiFNCLVh", "reviewer": "Reviewer_gKEt", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper introduces a three-stage approach called APRICOT for solving personalized organizational tasks with environmental constraints. There are many concerns in the methodology part and some issues in experiments,.", "summary_of_recommendation": "see in the review", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 2, "robotics_focus": 2, "quality_of_limitations": 2, "questions_for_rebuttal": "see in the review.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FregKXvZbT", "reviewer": "Reviewer_SyaR", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This work proposes APRICOT to recover user preferences from demonstrations and active clarification, and utilize inferred preferences in a task while considering environmental constraints. The authors highlight two challenges with preference learning: the inherent ambiguity in inferring preferences from demonstrations, and potentially infeasible preferences. They approach this by reasoning over a closed loop with simulated human feedback, obtained from an LLM, as well as simulated environment feedback, obtained from a planner. The proposed method first employs a VLM to parse visual demonstrations into a textual representation, and uses an LLM to generate a set of candidate preferences based on the textual demonstrations. To disambiguate between different preference hypotheses, APRICOT generates a set of questions using an LLM, then asks the most informative question till the uncertainty collapses. Finally it executes the single remaining optimal plan.", "summary_of_recommendation": "The paper identifies core challenges in preference learning, proposes a formulation to address them, and does a great job at clearly detailing the proposed method. However, the evaluation section, particularly the dataset used, requires more details.", "technical_quality": 4, "clarity_of_presentation": 5, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "- Can the authors elaborate on how the dataset is generated. Is it crowdsourced? Artificially generated using AI? \n- How does the proposed method compares against classical AI methods, and alternatively, why is such a comparison not possible?\n- What are the failure modes of the model, and, intuitively, why can the performance not be near perfect, since the model seems to have access to effectively a perfect human model. More generally, can the authors comment on the second point in the 'concerns' section above?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NEFaZbqklO", "reviewer": "Reviewer_YWXV", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper presents APRICOT, a three-stage approach designed to help home robots perform personalized tasks by balancing user preferences and environmental constraints. APRICOT learns user preferences from visual demonstrations, generates plans that satisfy these preferences, and executes the plans using a real-robot system. The approach leverages LLM-based Bayesian active preference learning to handle ambiguities in user actions and dynamically adapts to environmental constraints. The effectiveness of APRICOT is demonstrated through evaluations on a diverse set of organization tasks and real-world scenarios.", "summary_of_recommendation": "The paper presents an innovative integration of LLM-based Bayesian active preference learning with constraint-aware task planning, addressing significant challenges in home robot task planning. The methodology is detailed and generally clear, providing a robust understanding of the approach. However, the paper could benefit from further elaboration on strategies for resolving ambiguities in user actions, discussions on scalability with increased complexity, and detailed computational requirements for assessing real-time performance. Despite these weaknesses, the contributions and potential impact of the work are substantial, making it a valuable addition to the field.", "technical_quality": 2, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 3, "quality_of_limitations": 2, "questions_for_rebuttal": "• Could you provide more details on the integration process of APRICOT with real-robot systems? Are there specific hardware or software requirements?\n\n• The paper mentions the challenge of inherent ambiguity in interpreting user actions. How does APRICOT specifically address this ambiguity?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: **Summary**

The paper introduces APRICOT, a three-stage framework for enabling home robots to perform personalized tasks while accounting for user preferences and environmental constraints. APRICOT addresses two key challenges: the ambiguity in inferring preferences from user demonstrations and the risk of generating infeasible preferences. The method begins by using a VLM to convert visual demonstrations into text, followed by an LLM that generates candidate preferences. To resolve ambiguities, APRICOT actively asks targeted questions generated by the LLM until the most accurate preference is identified. Finally, it executes the optimal plan that satisfies the clarified preferences. Despite some initial concerns with the methodology and experiments, APRICOT demonstrates effectiveness in various organization tasks through evaluations on real-world robotic systems.

**Review Summary**

*Strengths*:
- The problem of quantifying ambiguity of user preferences is important and relevant to the CoRL community.
- Reviewers thought the approach was interesting, principled, and well described.
- Some of the reviewers found the method novel and a substantial contribution.
- The experiments test the role of each component of the pipeline.
- The paper is well-written in terms of clarity of the method and motivation.

*Weaknesses*:
- Unclear applicability to real-world robotics. Some of the reviewers raised concerns regarding the applicability of the method to the real world. One reviewer made a point about potentially unrealistic assumptions (perfect world model, binary geometric constraints, optical plan can be found, etc.) that over-simplify the problem setting, to the point of reducing the practical novelty and applicability of the approach in more complex and realistic environments. Another reviewer raises issues about the system’s scalability and computational requirements.
- Contested novelty. While some of the reviewers found the integration of the individual components to be novel and significant, others raised the point that the components themselves are not new and simply putting them together is not particularly groundbreaking. The authors should comment in their rebuttal on why integrating these existing components together could be seen as novel and substantive.
- Insufficient baselines. One reviewer pointed out that the baselines used are effectively ablations of the method. Reviewers asked for a comparison with classical AI methods or alternatively a justification for why such a comparison would not be possible. 
- Insufficient discussion. Reviewers asked for more discussion of the experiments and results that the authors should address: how is the data generated, what are the model’s failure modes and why does it struggle with complex preferences, what are strategies for resolving ambiguity, etc. Another point was raised about why the model is not performing better, especially since it appears that the method effectively has access to a perfect human model.
- (minor) Writing could be improved: reviewers have some suggestions for how to improve the grammar, sentence structure, and use of active voice in the paper.

**Outcome and Post-rebuttal**

After many efforts from the authors to address the reviewer comments, from failure case discussion to clearer contributions to applicability to robotics, it seems like all reviewers are in agreement about accepting this paper. Conditioned on including all the changes discussed in the rebuttal phase, I am in favor of accepting this paper.
Decision: Accept