Learning to Plan with Personalized Preferences

## Abstract

Abstract Effective integration of Artificial Intelligence ( AI ) agents into daily life requires them to understand and adapt to individual human preferences, particularly in collaborative roles. Although recent studies on embodied intelligence have advanced significantly, they typically adopt generalized approaches that overlook personal preferences in planning . We address this limitation by developing agents that not only learn preferences from few demonstrations but also learn to adapt their planning strategies based on these preferences. Our research leverages the observation that preferences, though implicitly expressed through minimal demonstrations, can generalize across diverse planning scenarios. To systematically evaluate this hypothesis, we introduce Preference-based Planning ( PbP ) benchmark, an embodied benchmark featuring hundreds of diverse preferences spanning from atomic actions to complex sequences. Our evaluation of State-of-the-Art ( SOTA ) methods reveals that while symbol-based approaches show promise in scalability, significant challenges remain in learning to generate and execute plans that satisfy personalized preferences. We further demonstrate that incorporating learned preferences as intermediate representations in planning significantly improves the agent’s ability to construct personalized plans. These findings establish preferences as a valuable abstraction layer for adaptive planning, opening new directions for research in preference-guided plan generation and execution.

## 1 Introduction

Figure: Figure 1: An example of preference-based planning in a food preparation scenario. When the assistant receives a natural language instruction for a food preparation task, it can follow one of two approaches: (Left, traditional methods) The assistant verifies details with the user at each step through exhaustive communication; or (Right, our personalized approach) it first learns from previous user action sequences to infer explicit preference labels and then generates a personalized plan based on the learned preferences. The planning tree (middle) illustrates how preferences guide the whole decision-making process across multiple dimensions. By learning preferences as a key intermediate representation from minimal human demonstrations, our approach enables AI agents to deliver personalized and adaptable assistance without explicit step-by-step instructions.
Refer to caption: x1.png

The field of embodied Artificial Intelligence (AI) is rapidly advancing, driven by significant progress in foundation models for vision and language (Bommasani et al.,, 2021; Peng et al.,, 2023; Achiam et al.,, 2023; Bai et al.,, 2023). These advances enable AI systems to autonomously collaborate with or assist humans in daily tasks, particularly in domestic settings (Driess et al.,, 2023; Leal et al.,, 2023; Zitkovich et al.,, 2023; Ahn et al.,, 2024). However, recent approaches utilizing natural language instructions (Mu et al.,, 2023; Zitkovich et al.,, 2023; Singh et al.,, 2023) face fundamental limitations in capturing human preferences (Zhu et al.,, 2016). While natural language is our primary means of communication, its inherent ambiguity creates a gap between instructions and intended executions (Yuan et al.,, 2022; Jiang et al.,, 2022; 2021; Yuan et al.,, 2020). For instance, when a user requests help in preparing an apple, the agent needs to understand specific preferences about apple selection, washing requirements, cutting style, and container choice—details that vary significantly across individuals; see also [Figure˜1](https://arxiv.org/html/2502.00858v3#S1.F1) for a graphical illustration.

Preference, central to personalization (Slovic,, 1995), remains inadequately addressed in embodied Artificial Intelligence (AI). Integrating personalized preferences is crucial for tailoring agent actions to individual users, thereby enhancing the effectiveness and satisfaction of embodied assistants (Lee et al.,, 2012; Leyzberg et al.,, 2014). Moreover, preferences guide human-like decision-making and intelligent behavior. Psychological research emphasizes that understanding preferences is vital for interpreting human behaviors (Fawcett and Markson,, 2010) and facilitating social interactions (Gerson et al.,, 2017; Liberman et al.,, 2021), suggesting that preference understanding could enable more grounded planning in embodied assistants.

Learning human preferences in real-world settings presents unique challenges (Peng et al.,, 2024). Humans typically communicate their needs succinctly (Levinson,, 1983), without exhaustive preference details (Lichtenstein and Slovic,, 2006), and many preferences include unconscious or instinctive elements difficult to articulate (Epstein,, 1994; Simonson,, 2008). A more practical approach is to infer preferences from observed human choices and decision-making patterns, as illustrated in [Figure˜1](https://arxiv.org/html/2502.00858v3#S1.F1), where a robot assistant can learn users’ preferences and behavior habits from previous observations.

In this paper, we focus on developing agents capable of learning preferences from human behavior and subsequently planning actions guided by these learned preferences. While previous studies like NeatNet (Kapelyukh and Johns,, 2022) and SAND (Yuan et al.,, 2023) have explored preference-based learning, they are limited to specific tasks (*e.g*., rearrangement) and fail to generalize across different situations. To address this limitation, we introduce Preference-based Planning (PbP), a comprehensive embodied benchmark built upon NVIDIA Omniverse and OmniGibson (Li et al.,, 2023). PbP provides realistic simulation and real-time rendering for thousands of daily activities across 50 scenes, featuring a parameterized vocabulary of 290 diverse preferences. These preferences span multiple levels, from specific action-level preferences (*e.g*., preferred glass type, water temperature) to task sequence-level preferences (*e.g*., task ordering, subtask prioritization).

Given the expensive nature of data collection (Akgun et al.,, 2012) and the few-shot nature of preference acquisition, we frame preference learning as a few-shot learning from demonstration task. In this framework, agents must respond to ambiguous instructions by formulating plans aligned with preferences demonstrated in limited example sequences. Specifically, an agent needs to analyze behavioral data, identify consistent patterns, and extrapolate these patterns to higher-level preference abstractions that can generalize across various tasks (Chao et al.,, 2011). Furthermore, when confronted with new tasks, the agent should leverage these learned preferences to generate adaptive action sequences that align with user preferences while maintaining task efficiency.

With the PbP benchmark developed, we challenge existing learning agents on their ability to learn human preference and subsequently conduct preference-based planning. Our systematic evaluation of SOTA algorithms on PbP reveals that preferences serve as valuable abstractions of human behaviors, and their incorporation as intermediate planning steps significantly enhances agent adaptability. Through extensive experimentation, we demonstrate that symbol-based approaches show promise in scalability, yet significant challenges remain in both preference learning and planning. These challenges stem from the complexity of planning intricate activities and the nuanced nature of learning preferences through perception. Our analysis particularly highlights the difficulties in few-shot preference learning and preference-guided planning, establishing preferences as a crucial abstraction layer between high-level goals and low-level actions. We present this work as a foundation for addressing these challenges in preference-based embodied AI.

## 2 Related Work

### 2.1 Theoretical Foundations of Human Preferences

Preference theory originates from psychological research, where it describes predictable patterns in human behavior that can be modeled mathematically (Kahneman,, 1982). These preferences reflect individual attitudes towards available choices in decision-making (Lichtenstein and Slovic,, 2006) and operate both consciously and unconsciously to shape behavior (Coppin et al.,, 2010). A fundamental principle is that underlying preferences can be inferred from consistent behavioral patterns (Sen,, 1973), enabling systematic analysis of decision-making processes. This framework has extended beyond psychology into economics, where Rational Choice Theory (Scott et al.,, 2000) models decision-making based on rational self-interest (Zey,, 1998). Building on this, Utility Theory provides a mathematical foundation for modeling how preferences relate to attitudes toward rewards and risks (Mongin,, 1997; Aleskerov et al.,, 2007). These theoretical foundations establish preferences as fundamental elements in shaping both individual behavior and broader societal dynamics. In recent years, these preference models have found new applications in artificial intelligence and robotics, particularly in developing human-centric AI assistants capable of understanding and adapting to individual user preferences.

### 2.2 Preference in Embodied Task Planning

The application of preferences in embodied task planning encompasses two distinct approaches. The first focuses on general preference-based planning, where robots leverage commonsense knowledge to execute universally accepted behavioral norms. Object rearrangement exemplifies this approach, with systems organizing items based on common occurrence patterns and spatial relationships (Taniguchi et al.,, 2021; Sarch et al.,, 2022). The second approach emphasizes personalized preferences, where embodied agents align their actions with individual user habits. This includes personalized object placement strategies (Abdo et al.,, 2015; Kapelyukh and Johns,, 2022; Wu et al.,, 2023), preference-aware table setting (Puig et al., 2021a,), and multi-agent coordination where agents maintain individual preferences while achieving optimal coordination (Shu and Tian,, 2019).

Our work extends these approaches by considering preferences across diverse situations and scenes. Beyond spatial arrangements, we address temporal action sequences, state transitions during interactions, and few-shot preference learning. This comprehensive framework enables robust preference modeling and adaptation in real-world scenarios.

Figure: (a) Action level
Refer to caption: x2.png

### 2.3 Embodied Assistants

The development of intelligent embodied assistants has evolved from basic Vision-and-Language Navigation (VLN) tasks (Anderson et al.,, 2018; Chen et al.,, 2019; Thomason et al.,, 2020) to complex interactive scenarios. ALFRED (Shridhar et al.,, 2020) introduced object manipulation, state tracking, and temporal dependencies between instructions, while platforms like Habitat (Savva et al.,, 2019; Puig et al., 2023b,) and AI2-THOR (Kolve et al.,, 2017) emphasize active perception, long-term planning, and interactive learning in realistic environments. Recent research has shifted toward implicit-instruction scenarios, particularly in housekeeping tasks (Kapelyukh and Johns,, 2022; Kant et al.,, 2022; Sarch et al.,, 2022; Wu et al.,, 2023), where robots must reason about object arrangements without explicit directives. Works on proactive assistance (Patel and Chernova,, 2023; Patel et al.,, 2023; Puig et al., 2023a,) further explore anticipating temporal patterns in humans’ daily routines.

Methodologically, recent advances utilize Large Language Models as few-shot planners to generate language-based action sequences from limited demonstrations (Song et al.,, 2023; Driess et al.,, 2023; Ding et al.,, 2023; Zhang et al.,, 2024). Foundation Vision-Language Models have enhanced robotic systems’ perception and reasoning capabilities (Ahn et al.,, 2024; Leal et al.,, 2023; Gu et al.,, 2023; Brohan et al.,, 2022; Zitkovich et al.,, 2023; Xu et al., 2024a,), enabling understanding of complex visual and linguistic inputs in everyday tasks. However, while these foundation models excel at reasoning from text or image information, their ability to learn individual preferences from limited demonstrations and plan adaptively remains an open challenge, particularly in multi-step tasks requiring personalized execution strategies.

## 3 Formulating Preference-based Planning

Tasks in PbP mirror real-world watch-and-help scenarios (Puig et al., 2021b,), where an agent observes a few demonstrations of a user performing tasks that reveal preferences. The agent must then complete similar tasks in different setups while adhering to the demonstrated preferences.

Preference-based planning comprises two key components: few-shot preference learning of user preferences and subsequent planning guided by these learned preferences. Since humans, even infants, can naturally detect others’ preferences from limited decisions (Choi and Luo,, 2023), and collecting extensive personal demonstrations is impractical in daily life, we formulate this as few-shot learning from demonstration.

Given a user with preference $\mathbf{p}$, the agent observes the user performing tasks from a first-person perspective, denoted as $\mathbf{O}$. These observations span multiple demonstrations. Formally, $\mathbf{O}$ contains both state and action observations: $\mathbf{O}=\{(\mathcal{S}_{i},\mathcal{A}_{i},\mathcal{M})_{N}\}$, where $\mathcal{S}_{i}$ denotes the egocentric observation sequence in the $i$-th demonstration, $\mathcal{A}_{i}$ represents the action sequence, and $\mathcal{M}$ optionally provides a bird’s-eye view of the entire scene map.

In the first stage, the objective is to learn the preference representation demonstrated through user actions:

$$ $\mathbf{p}=f(\mathbf{O};\theta_{f}),$ (1) $$

where $\mathbf{p}$ denotes the learned preference representation here. It can either be a hidden representation or an explicit textual label, depending on the task settings.

The learned preference $\mathbf{p}$ should then guide planning when the agent faces different setups with varying objects, room layouts, or entire scenes. Specifically, the agent optimizes:

$$ $\mathcal{L}=\sum_{i=1}\ell(g(s_{i},f(\mathbf{O};\theta_{f});\theta_{g}),a_{i}),$ (2) $$

where $g(\cdot)$ represents a potentially parameterized planning function that maps the current state and preference representation to the next action, and $a_{i}$ denotes the ground-truth action demonstrating the user’s preference at the current stage.

## 4 The Preference-based Planning ( PbP ) Benchmark

Built on NVIDIA’s Omniverse and OmniGibson simulation environment (Li et al.,, 2023), our PbP benchmark enables realistic simulation of thousands of daily activities. It spans 50 distinct scenes and encodes 290 unique preferences, with a comprehensive test set of 5000 instances. Below, we detail the preference structure and test set construction.

### 4.1 Definition of Preferences

We organize preferences in a three-tiered hierarchical structure that captures varying degrees of specificity across tasks. [Figure˜2](https://arxiv.org/html/2502.00858v3#S2.F2) provides an overview of all preferences and their distribution, while [Figure˜3](https://arxiv.org/html/2502.00858v3#S4.F3) illustrates concrete examples of preferences and corresponding agent actions. The 290 preferences are distributed across three levels: 80 sequence-level, 135 option-level, and 75 action-level preferences.

#### Action Level

These bottom-level preferences govern fine-grained execution details within specific sub-tasks, such as water quantity preferences when filling cups or shelf placement choices for books.

#### Option Level

Middle-level preferences encode alternative approaches to sub-tasks. For instance, in “storing-nonperishable-food,” users may prefer cabinet storage versus table placement. These preferences can bind to different objects and may compose multiple action-level preferences.

#### Sequence Level

Top-level preferences define task ordering and prioritization. They capture temporal dependencies between sub-tasks, such as cleaning furniture before rearranging kitchen utensils, followed by dinner preparation upon returning home.

Figure: Figure 3: Example of preferences and their corresponding actions in PbP. At the primitive action level, we demonstrate preferences through basic tasks: (a) cooking using microwave, (b) washing in the sink, and (c) cutting into halves. At the option level, we showcase different approaches to object rearrangement, where users can prefer either (d) grouping objects by their categories (v1) or (e) placing them on the same layer of the fridge (v2). At the sequence level, we illustrate how preferences guide task ordering: (f) shows a user’s preference to have fruits first, followed by specific cleaning tasks.
Refer to caption: x5.png

### 4.2 Constructing PbP Test Set

Our PbP benchmark includes a default test set for systematic model evaluation. Following the formulation in [Section˜3](https://arxiv.org/html/2502.00858v3#S3), we structure PbP tasks as few-shot learning-from-demonstration problems. Each test point comprises several (typically three) unique demonstrations with egocentric observations of action sequences and their corresponding preference labels. As illustrated in [Figure˜4](https://arxiv.org/html/2502.00858v3#S4.F4), a demonstration includes an egocentric video of agent activity, a bird’s-eye-view map tracking agent position, and frame-level action annotations. We also provide third-person view recordings for enhanced visualization. We prioritize the egocentric perspective for two reasons: 1) it offers a clear view with minimal occlusions, and 2) it aligns with human perception, facilitating transfer to real-world data from head-mounted devices.

The test set construction follows a two-stage process. First, we build a reusable and extensible demonstration pool. To generate each demonstration, we randomly assign a preference from our defined primitives to one of 50 OmniGibson scenes, then sample relevant objects within the chosen scene. We generate multi-perspective observations using rule-based planners for high-level planning and predefined scripts for low-level execution (*e.g*., Inverse Kinematics (IK) for grasping, A^⋆ for movement).

Second, we construct test points by sampling preferences and retrieving relevant demonstrations from the pool. To reflect real-world few-shot scenarios, each preference pairs with demonstrations that share the same high-level preference but vary in scene settings or object selections. We also include unrelated demonstrations to prevent sampling bias.

The default test set contains 5,000 test points, drawing from a pool of 15,000 unique recordings. Unless specified otherwise, all experiments use this default set. The benchmark also supports custom test point generation through flexible demonstration sampling, preference definition, and third-person view video creation.

Figure: Figure 4: Example of a demonstration in PbP. The robot in the demonstration is executing the task “Pick Apple from Fridge and place on Table”. Top: A third-person view video provides an overhead perspective of the entire scene. Middle: The bird’s-eye-view map displays the robot’s relative position within the scene. Bottom: The egocentric video captures the robot’s first-person observations during task execution. Text: The per-frame action annotations contain Omniverse object IDs, which ensure each object reference is unique and enable the model to identify specific objects precisely.
Refer to caption: x6.png

### 4.3 Models

Our evaluation focuses primarily on multimodal models that incorporate LLMs and demonstrate strong few-shot learning capabilities. The LLM component serves as a knowledge base that can enhance preference learning through commonsense reasoning. We also include symbol-based LLM models for ablation studies to analyze how different modalities impact PbP performance. Most models evaluated can function in both end-to-end and two-stage pipeline configurations.

#### ViViT

As a baseline, we employ the pure-Transformer-based Video Vision Transformer (ViViT) (Arnab et al.,, 2021), an end-to-end trainable model with proven capabilities in extracting spatial and temporal information from video inputs. Since it lacks a LLM component, ViViT likely serves as a lower bound for commonsense understanding in PbP tasks.

#### LLaVA

Building on more sophisticated architectures, Large Language and Vision Assistant (LLaVA) (Liu et al.,, 2024) represents an end-to-end trainable large multimodal model that integrates vision and text for comprehensive visual-language understanding. We specifically evaluate LLaVA-NeXT, which has been finetuned to excel at zero-shot video understanding tasks.

#### EILEV

For specialized egocentric video processing, we incorporate Efficient In-context Learning on Egocentric Videos (EILEV) (Yu et al.,, 2023), which achieves in-context learning through architectural modifications to a pretrained VLM. Our implementation uses OPT-2.7B (Zhang et al.,, 2022) as the language backbone. The model’s pretraining on Ego4D (Grauman et al.,, 2022) aligns well with PbP’s egocentric perspective.

#### GPT-4V

To benchmark against state-of-the-art visual-language models, we evaluate GPT-4V using the Azure OpenAI API (version “gpt-4-turbo-2024-04-09”). Due to image token limitations, we implement video input subsampling while maintaining temporal coherence.

Beyond multimodal approaches, we also evaluate single-modal models that process only action sequences:

#### DAG-Opt

We approach symbolic reasoning by framing the problem as a DAG-Optimization task that uncovers dependency relations between actions and preferences (Zheng et al.,, 2018). Our implementation uses a score-based NOTEARS model to learn a generalized Structural Equation Model (SEM), following previous few-shot reasoning frameworks (Zhang et al.,, 2021; Xu et al., 2024b,) based on causal dependency structures.

#### LLMs

To assess pure language understanding, we evaluate advanced LLMs like Llama3 (Touvron et al.,, 2023) and GPT-4.1 (Achiam et al.,, 2023) using only action sequence inputs. This approach treats actions as high-level abstractions of egocentric videos, reducing visual complexity while maintaining task semantics. We benchmark Llama3-8B as our baseline against GPT-4.1 as the current state-of-the-art, employing prompt designs informed by the OpenAI Cookbook for optimal few-shot performance. Detailed model architectures and prompt engineering strategies are discussed in [Appendix˜D](https://arxiv.org/html/2502.00858v3#A4).

## 5 Experiments

### 5.1 Experimental Setup

We evaluate preference learning capabilities across two distinct settings: end-to-end and two-stage approaches. In the end-to-end setting, models directly map raw state inputs to action outputs. Leveraging models’ in-context learning abilities, we provide demonstrations alongside current state information as input and evaluate the generated action sequences against ground truth.

The two-stage setting introduces an intermediate step where models first learn to predict explicit preference labels during training. These predicted labels then serve as preference representations for subsequent planning stages. For black-box models, we employ carefully designed prompts rather than fine-tuning approaches.

All demonstration videos maintain consistent technical specifications across models and agents: egocentric perspective, 512 $\times$ 512 resolution, and 8 fps frame rate. Video duration matches the corresponding action sequence length. For LLM inference, we use conservative decoding parameters: temperature of 0.05, top-k of 1, and top-p of 0.05. All experiments run on a single machine with 8 NVIDIA A100 GPUs.

### 5.2 End-to-end Action Preference Learning

We first evaluate model performance in the end-to-end setting, where models generate actions directly from previous demonstrations and current state information. To quantify performance, we use Levenshtein distance to measure discrepancies between generated and ground truth action sequences, treating each individual action as a token.

As shown in [Table˜1](https://arxiv.org/html/2502.00858v3#S5.T1) (the End-to-end row), video-based models produce Levenshtein distances approaching the average ground truth sequence lengths (15.80 at option level, 35.87 at sequence level). These high distances indicate that the models generate predominantly inconsistent action sequences, suggesting a failure to grasp preferences embedded in demonstration videos. While symbol-based models show modest improvements, their performance gains remain limited.

**Table 1: Levenshtein distance between generated and ground truth action sequences. End-to-end represents models directly generating action sequences from demonstration-preference pair examples. Two-stage indicates generation using both demonstrations and previously inferred preference labels based on demonstrations. Second-stage (gt) uses demonstrations alongside ground truth preference labels for sequence generation.**
|  | VIDEO-BASED INPUT |  | SYMBOL-BASED INPUT |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | ViViT | LLaVA-Next | EILEV | GPT-4V |  | Llama3-8B | DeepSeek-R1 | GPT-4.1 |
| Option Level |  |  |  |  |  |  |  |  |
| End-to-end | 15.49$\pm$1.29 | 15.94$\pm$3.41 | 12.88$\pm$2.20 | 15.63$\pm$2.31 |  | 14.74$\pm$3.21 | 8.73 $\pm$3.03 | 7.42$\pm$2.67 |
| Two-stage | - | 12.46$\pm$3.23 | 12.89$\pm$3.74 | 8.37$\pm$2.19 |  | 9.67$\pm$5.16 | 3.19$\pm$2.19 | 2.26$\pm$2.03 |
| Second-stage (gt) | - | 3.28$\pm$5.29 | 11.18$\pm$4.20 | 1.26$\pm$2.55 |  | 8.22$\pm$5.58 | 1.76$\pm$1.89 | 0.15$\pm$2.85 |
| Sequence Level |  |  |  |  |  |  |  |  |
| End-to-end | 34.04$\pm$11.84 | 34.76$\pm$11.25 | 33.10$\pm$12.21 | 33.75$\pm$11.15 |  | 31.79$\pm$7.32 | 28.72$\pm$4.12 | 26.48$\pm$3.25 |
| Two-stage | - | 30.02$\pm$13.54 | 33.03$\pm$13.61 | 27.52$\pm$9.48 |  | 25.46$\pm$5.93 | 18.61$\pm$3.25 | 14.19$\pm$3.01 |
| Second-stage (gt) | - | 18.92$\pm$14.18 | 26.57$\pm$12.21 | 11.36$\pm$8.05 |  | 19.02$\pm$7.10 | 14.10$\pm$3.76 | 10.31$\pm$2.98 |
| Overall |  |  |  |  |  |  |  |  |
| End-to-end | 24.76 | 25.35 | 22.99 | 24.69 |  | 23.26 | 18.72 | 16.95 |
| Two-stage | - | 21.24 | 22.96 | 17.94 |  | 17.56 | 10.90 | 8.22 |
| Second-stage (gt) | - | 11.10 | 18.88 | 6.31 |  | 13.62 | 7.93 | 5.23 |

These findings expose a fundamental limitation in current models: they struggle to extract underlying relationships from perceptual inputs without explicit intermediate guidance. The models appear to learn individual, isolated actions rather than cohesive action patterns that reflect implicit preferences. This significant gap underscores the inherent challenge of performing end-to-end preference learning solely from demonstrations.

### 5.3 Two-Stage Learning-Planning

Given the limitations of end-to-end learning, we implement a two-stage approach to decompose the preference learning problem. The first stage focuses on preference prediction, where we provide models with auxiliary preference token labels and train them to predict hidden preferences explicitly. These preference tokens, as discussed in [Section˜4.1](https://arxiv.org/html/2502.00858v3#S4.SS1), maintain sufficient semantic content for translation into primitive actions.

Results from the first stage ([Table˜2](https://arxiv.org/html/2502.00858v3#S5.T2)) reveal significant performance variations across models. At the option level, GPT-4V achieves superior performance with 48.48% accuracy, demonstrating strong capability in interpreting demonstrated preferences. Among symbol-based models, the stark contrast between DAG-Opt’s limited performance and the improved results from Llama3-8B and GPT-4.1 highlights the advantage of next-token prediction over dependency learning for preference inference. Models with language components consistently show improved preference understanding compared to end-to-end learning.

**Table 2: Preference prediction accuracy in few-shot and ablative settings.**
|  | VIDEO-BASED INPUT |  | SYMBOL-BASED INPUT |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | ViViT | LLaVA-Next | EILEV | GPT-4V |  | DAG-Opt | Llama3-8B | DeepSeek-R1 | GPT-4.1 |
| Few-shot |  |  |  |  |  |  |  |  |  |
| Option Level | 9.38 | 36.87 | 38.33 | 48.48 |  | 10.15 | 72.98 | 86.02 | 88.91 |
| Sequence Level | 4.24 | 24.85 | 32.69 | 37.50 |  | 13.49 | 67.18 | 71.21 | 70.28 |
| Overall | 6.81 | 30.86 | 35.51 | 42.99 |  | 11.82 | 70.08 | 78.62 | 79.60 |
| Ablative |  |  |  |  |  |  |  |  |  |
| Option Level | 9.16 | 15.47 | 4.77 | 29.42 |  | 3.84 | 39.50 | 78.19 | 75.29 |
| Sequence Level | 4.38 | 8.13 | 0.00 | 0.00 |  | 1.28 | 6.25 | 15.29 | 14.11 |
| Overall | 6.77 | 11.8 | 2.38 | 14.71 |  | 2.56 | 22.88 | 46.74 | 44.70 |

The second stage involves generating action sequences based on both demonstrations and predicted preference labels from the first stage, introducing potential error propagation. Results in [Table˜1](https://arxiv.org/html/2502.00858v3#S5.T1) (Two-stage row) and [Figure˜5](https://arxiv.org/html/2502.00858v3#S5.F5) (a)-(b) show significant improvements when models receive explicit preferences. For comprehensive evaluation, we include planning results using ground truth preference labels (Second-stage (gt) row). GPT-4V and GPT-4.1 achieve near-zero Levenshtein distances, indicating almost perfect alignment with ground truth action sequences.

Analysis of both stages reveals distinct challenges across model types. Vision-based models like LLaVA-Next and GPT-4V struggle with preference inference but excel in action planning given preference labels, suggesting difficulty in abstracting preferences from visual input. Symbol-based models perform well in both preference inference and preference-guided planning, yet underperform in end-to-end settings. This indicates that models may lack innate preference-based reasoning capabilities but can effectively plan when preferences are explicitly provided.

To isolate the impact of prior knowledge versus in-context learning, we conduct ablation studies by removing demonstrations and testing preference prediction on isolated test sequences. Results in [Table˜2](https://arxiv.org/html/2502.00858v3#S5.T2) (bottom) show significant performance degradation compared to few-shot learning ([Table˜2](https://arxiv.org/html/2502.00858v3#S5.T2) (top)), particularly at the sequence level. This suggests that while models may encode basic task-specific preferences, they rely heavily on demonstrations to recognize complex preference patterns in varied sequences.

Figure: (a) Option level
Refer to caption: x7.png

### 5.4 Generalization

While human actions may vary across different objects and scenes, underlying preferences often remain consistent. We evaluate the models’ ability to generalize preference learning across varying visual contexts. The original test set inherently tests generalization by randomly sampling scenes and objects when rendering video demonstrations for each preference. To gain additional insights, we conduct complementary experiments with controlled conditions where demonstration and test videos are rendered in identical rooms with the same objects. This controlled setting enables direct performance comparisons under consistent conditions. We evaluate EILEV, LLaVA, and GPT-4.1 series models on this variant of PbP, as these models previously demonstrated strong few-shot reasoning capabilities. Results are summarized in [Table˜3](https://arxiv.org/html/2502.00858v3#S5.T3).

**Table 3: Models’ generalization ability. direct denotes experiments conducted without generalization. gen denotes the original experiments conducted with generalization cases. Also the accuracy of preference prediction.**
|  | LLaVA-Next | EILEV | GPT-4V | DeepSeek-R1 | GPT-4.1 |
| --- | --- | --- | --- | --- | --- |
| Option Level direct | 33.25 | 46.93 | 53.24 | 86.02 | 88.91 |
| Option Level gen | 36.87 | 38.33 | 48.48 | 84.98 | 87.12 |
| Sequence Level direct | 33.12 | 37.53 | 39.42 | 71.21 | 70.28 |
| Sequence Level gen | 24.85 | 32.69 | 37.50 | 70.16 | 68.01 |

Figure: Figure 6: Analysis of test samples in direct and generalization settings. Lines represent distinct scenes, with grid colors indicating different sample statuses.
Refer to caption: x9.png

Symbol-based reasoning (GPT-4.1) demonstrates consistent performance regardless of scene or object variations, while vision-based models show greater sensitivity to scene changes. This distinction stems from the nature of our predefined preferences, which are sufficiently abstract and general to apply across diverse scenes and objects. Vision-based models, however, tend to anchor their few-shot learned preferences to specific visual features of scenes or objects. When these visual elements change, preference recognition accuracy may deteriorate. This contextual dependence remains a persistent challenge for vision-based models, which often overfit to scene-specific features from training videos.

Analysis of test points across direct and gen conditions ([Figure˜6](https://arxiv.org/html/2502.00858v3#S5.F6)) reveals two key findings: (i) Preference learning performance correlates with scene characteristics, with certain scenes proving consistently challenging across both conditions. (ii) While direct cases show better performance overall, failure patterns differ between conditions, particularly for vision-based models. This suggests models rely heavily on visual context consistency–including object arrangement and scene layout–for accurate predictions, indicating potential superficial learning rather than true preference understanding. Symbol-based reasoning maintains robust performance across varied scenes due to the general nature of predefined preferences, whereas vision-based models’ strong dependence on specific visual contexts limits their generalization capability.

### 5.5 Ablations on Demonstration Numbers

We examine the effect of demonstration quantity on model performance through an ablation study ([Figure˜5](https://arxiv.org/html/2502.00858v3#S5.F5) (c)-(d)). Results show that increasing demonstration numbers generally improves preference learning and planning effectiveness. This improvement is most evident in second-stage planning, where models achieve lower sequence distances by more accurately replicating human actions. Models like GPT-4.1, Llama3, and EILEV show consistent performance gains with additional demonstrations. However, we observe that excessive demonstrations (*e.g*., 5-demo cases for GPT-4.1 and EILVE) can sometimes impair first-stage prediction accuracy. Despite these occasional exceptions, the overall trend confirms our intuition: more demonstrations enhance learning and planning performance. These findings highlight the importance of demonstration quantity in developing effective personalized planning systems that align with user preferences.

Figure: (a) First-stage preference learning
Refer to caption: x10.png

## 6 Conclusion

We investigate methods for embodied agents to learn and implement human preferences through behavioral observation and user interaction. We present Preference-based Planning (PbP), a comprehensive embodied benchmark designed to capture the complexity of real-world human preferences. We also develop an evaluation framework to assess models’ preference learning and implementation capabilities. Our findings demonstrate that preferences effectively abstract human behaviors and guide planning processes. While current models still face challenges in preference inference and adaptive planning from limited observations, incorporating preference-based reasoning improves both effectiveness and generalization, particularly in symbol-based systems that represent idealized scenarios. We aim to stimulate further research in this crucial yet understudied domain of developing preference-aware embodied agents.

#### Limitations and Societal Impacts

Our work’s primary limitation stems from its reliance on synthetic data. Despite Omniverse’s high-quality scene rendering, the simulator cannot fully replicate real-world complexity and variability. Furthermore, human-defined preference labels may not completely capture preference subtleties and diversity. We are addressing these limitations by collecting real-world preference demonstrations using head-worn devices, despite associated challenges. Given our focus on private scenarios, we anticipate minimal negative societal impact from this research.

#### Acknowledgment

The authors would like to thank Yuyang Li and Bo Dai for helpful discussions, and NVIDIA for their generous support of GPUs and hardware. This work is supported in part by the National Science and Technology Major Project (2022ZD0114900), the National Natural Science Foundation of China (62376009), the Beijing Nova Program, the State Key Lab of General AI at Peking University, the PKU-BingJi Joint Laboratory for Artificial Intelligence, and the National Comprehensive Experimental Base for Governance of Intelligent Society, Wuhan East Lake High-Tech Development Zone.

## References

- Abdo et al., (2015)
Abdo, N., Stachniss, C., Spinello, L., and Burgard, W. (2015).
Robot, organize my shelves! tidying up objects by predicting user preferences.
In International Conference on Robotics and Automation (ICRA).
- Achiam et al., (2023)
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. (2023).
Gpt-4 technical report.
arXiv preprint arXiv:2303.08774.
- Ahn et al., (2024)
Ahn, M., Dwibedi, D., Finn, C., Arenas, M. G., Gopalakrishnan, K., Hausman, K., Ichter, B., Irpan, A., Joshi, N., Julian, R., et al. (2024).
Autort: Embodied foundation models for large scale orchestration of robotic agents.
arXiv preprint arXiv:2401.12963.
- Akgun et al., (2012)
Akgun, B., Cakmak, M., Jiang, K., and Thomaz, A. L. (2012).
Keyframe-based learning from demonstration: Method and evaluation.
International Journal of Social Robotics, 4:343–355.
- Aleskerov et al., (2007)
Aleskerov, F., Bouyssou, D., and Monjardet, B. (2007).
Utility maximization, choice and preference, volume 16.
Springer Science & Business Media.
- Anderson et al., (2018)
Anderson, P., Wu, Q., Teney, D., Bruce, J., Johnson, M., Sünderhauf, N., Reid, I., Gould, S., and Van Den Hengel, A. (2018).
Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments.
In Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR).
- Arnab et al., (2021)
Arnab, A., Dehghani, M., Heigold, G., Sun, C., Lučić, M., and Schmid, C. (2021).
Vivit: A video vision transformer.
In Proceedings of International Conference on Computer Vision (ICCV).
- Bai et al., (2023)
Bai, J., Bai, S., Yang, S., Wang, S., Tan, S., Wang, P., Lin, J., Zhou, C., and Zhou, J. (2023).
Qwen-vl: A frontier large vision-language model with versatile abilities.
arXiv preprint arXiv:2308.12966.
- Bommasani et al., (2021)
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021).
On the opportunities and risks of foundation models.
arXiv preprint arXiv:2108.07258.
- Brohan et al., (2022)
Brohan, A., Brown, N., Carbajal, J., Chebotar, Y., Dabis, J., Finn, C., Gopalakrishnan, K., Hausman, K., Herzog, A., Hsu, J., et al. (2022).
Rt-1: Robotics transformer for real-world control at scale.
arXiv preprint arXiv:2212.06817.
- Chao et al., (2011)
Chao, C., Cakmak, M., and Thomaz, A. L. (2011).
Towards grounding concepts for transfer in goal learning from demonstration.
In 2011 IEEE International Conference on Development and Learning (ICDL), volume 2, pages 1–6. IEEE.
- Chen et al., (2019)
Chen, H., Suhr, A., Misra, D., Snavely, N., and Artzi, Y. (2019).
Touchdown: Natural language navigation and spatial reasoning in visual street environments.
In Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR).
- Choi and Luo, (2023)
Choi, Y. and Luo, Y. (2023).
Understanding preferences in infancy.
Wiley Interdisciplinary Reviews: Cognitive Science, 14(4):e1643.
- Coppin et al., (2010)
Coppin, G., Delplanque, S., Cayeux, I., Porcherot, C., and Sander, D. (2010).
I’m no longer torn after choice: How explicit choices implicitly shape preferences of odors.
Psychological science, 21(4):489–493.
- Ding et al., (2023)
Ding, Y., Zhang, X., Amiri, S., Cao, N., Yang, H., Kaminski, A., Esselink, C., and Zhang, S. (2023).
Integrating action knowledge and llms for task planning and situation handling in open worlds.
Autonomous Robots, 47(8):981–997.
- Driess et al., (2023)
Driess, D., Xia, F., Sajjadi, M. S., Lynch, C., Chowdhery, A., Ichter, B., Wahid, A., Tompson, J., Vuong, Q., Yu, T., et al. (2023).
Palm-e: An embodied multimodal language model.
In Proceedings of International Conference on Machine Learning (ICML).
- Epstein, (1994)
Epstein, S. (1994).
Integration of the cognitive and the psychodynamic unconscious.
American psychologist, 49(8):709.
- Fawcett and Markson, (2010)
Fawcett, C. A. and Markson, L. (2010).
Children reason about shared preferences.
Developmental psychology, 46(2):299.
- Gebru et al., (2021)
Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., Iii, H. D., and Crawford, K. (2021).
Datasheets for datasets.
Communications of the ACM, 64(12):86–92.
- Gerson et al., (2017)
Gerson, S. A., Bekkering, H., and Hunnius, S. (2017).
Do you do as i do?: Young toddlers prefer and copy toy choices of similarly acting others.
Infancy, 22(1):5–22.
- Grauman et al., (2022)
Grauman, K., Westbury, A., Byrne, E., Chavis, Z., Furnari, A., Girdhar, R., Hamburger, J., Jiang, H., Liu, M., Liu, X., et al. (2022).
Ego4d: Around the world in 3,000 hours of egocentric video.
In Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR).
- Gu et al., (2023)
Gu, J., Kirmani, S., Wohlhart, P., Lu, Y., Arenas, M. G., Rao, K., Yu, W., Fu, C., Gopalakrishnan, K., Xu, Z., et al. (2023).
Rt-trajectory: Robotic task generalization via hindsight trajectory sketches.
arXiv preprint arXiv:2311.01977.
- Jiang et al., (2022)
Jiang, K., Dahmani, A., Stacy, S., Jiang, B., Rossano, F., Zhu, Y., and Gao, T. (2022).
What is the point? a theory of mind model of relevance.
In Annual Meeting of the Cognitive Science Society (CogSci).
- Jiang et al., (2021)
Jiang, K., Stacy, S., Wei, C., Chan, A., Rossano, F., Zhu, Y., and Gao, T. (2021).
Individual vs. joint perception: a pragmatic model of pointing as communicative smithian helping.
In Annual Meeting of the Cognitive Science Society (CogSci).
- Kahneman, (1982)
Kahneman, D. (1982).
The psychology of preferences.
Scientific American.
- Kant et al., (2022)
Kant, Y., Ramachandran, A., Yenamandra, S., Gilitschenski, I., Batra, D., Szot, A., and Agrawal, H. (2022).
Housekeep: Tidying virtual households using commonsense reasoning.
In Proceedings of European Conference on Computer Vision (ECCV).
- Kapelyukh and Johns, (2022)
Kapelyukh, I. and Johns, E. (2022).
My house, my rules: Learning tidying preferences with graph neural networks.
In Conference on Robot Learning (CoRL).
- Kolve et al., (2017)
Kolve, E., Mottaghi, R., Han, W., VanderBilt, E., Weihs, L., Herrasti, A., Deitke, M., Ehsani, K., Gordon, D., Zhu, Y., et al. (2017).
Ai2-thor: An interactive 3d environment for visual ai.
arXiv preprint arXiv:1712.05474.
- Leal et al., (2023)
Leal, I., Choromanski, K., Jain, D., Dubey, A., Varley, J., Ryoo, M., Lu, Y., Liu, F., Sindhwani, V., Vuong, Q., et al. (2023).
Sara-rt: Scaling up robotics transformers with self-adaptive robust attention.
arXiv preprint arXiv:2312.01990.
- Lee et al., (2012)
Lee, M. K., Forlizzi, J., Kiesler, S., Rybski, P., Antanitis, J., and Savetsila, S. (2012).
Personalization in hri: A longitudinal field experiment.
In ACM/IEEE International Conference on Human-Robot Interaction (HRI).
- Levinson, (1983)
Levinson, S. C. (1983).
Pragmatics.
Cambridge UP.
- Leyzberg et al., (2014)
Leyzberg, D., Spaulding, S., and Scassellati, B. (2014).
Personalizing robot tutors to individuals’ learning differences.
In ACM/IEEE International Conference on Human-Robot Interaction (HRI).
- Li et al., (2023)
Li, C., Zhang, R., Wong, J., Gokmen, C., Srivastava, S., Martín-Martín, R., Wang, C., Levine, G., Lingelbach, M., Sun, J., et al. (2023).
Behavior-1k: A benchmark for embodied ai with 1,000 everyday activities and realistic simulation.
In Conference on Robot Learning (CoRL).
- Liberman et al., (2021)
Liberman, Z., Kinzler, K. D., and Woodward, A. L. (2021).
Origins of homophily: Infants expect people with shared preferences to affiliate.
Cognition, 212:104695.
- Lichtenstein and Slovic, (2006)
Lichtenstein, S. and Slovic, P. (2006).
The construction of preference: An overview.
The construction of preference, 1:1–40.
- Liu et al., (2024)
Liu, H., Li, C., Li, Y., and Lee, Y. J. (2024).
Improved baselines with visual instruction tuning.
In Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR).
- Mongin, (1997)
Mongin, P. (1997).
Expected utility theory.
Handbook of Economic Methodology, pages 342–350.
- Mu et al., (2023)
Mu, Y., Zhang, Q., Hu, M., Wang, W., Ding, M., Jin, J., Wang, B., Dai, J., Qiao, Y., and Luo, P. (2023).
Embodiedgpt: Vision-language pre-training via embodied chain of thought.
In Proceedings of Advances in Neural Information Processing Systems (NeurIPS).
- Patel and Chernova, (2023)
Patel, M. and Chernova, S. (2023).
Proactive robot assistance via spatio-temporal object modeling.
In Conference on Robot Learning (CoRL).
- Patel et al., (2023)
Patel, M., Prakash, A. G., and Chernova, S. (2023).
Predicting routine object usage for proactive robot assistance.
In Conference on Robot Learning (CoRL).
- Peng et al., (2024)
Peng, Y., Han, J., Zhang, Z., Fan, L., Liu, T., Qi, S., Feng, X., Ma, Y., Wang, Y., and Zhu, S.-C. (2024).
The tong test: Evaluating artificial general intelligence through dynamic embodied physical and social interactions.
Engineering, 34:12–22.
- Peng et al., (2023)
Peng, Z., Wang, W., Dong, L., Hao, Y., Huang, S., Ma, S., and Wei, F. (2023).
Kosmos-2: Grounding multimodal large language models to the world.
arXiv preprint arXiv:2306.14824.
- (43)
Puig, X., Shu, T., Li, S., Wang, Z., Liao, Y.-H., Tenenbaum, J. B., Fidler, S., and Torralba, A. (2021a).
Watch-and-help: A challenge for social perception and human-ai collaboration.
In Proceedings of International Conference on Learning Representations (ICLR).
- (44)
Puig, X., Shu, T., Li, S., Wang, Z., Liao, Y.-H., Tenenbaum, J. B., Fidler, S., and Torralba, A. (2021b).
Watch-and-help: A challenge for social perception and human-ai collaboration.
In Proceedings of International Conference on Learning Representations (ICLR).
- (45)
Puig, X., Shu, T., Tenenbaum, J. B., and Torralba, A. (2023a).
Nopa: Neurally-guided online probabilistic assistance for building socially intelligent home assistants.
In International Conference on Robotics and Automation (ICRA).
- (46)
Puig, X., Undersander, E., Szot, A., Cote, M. D., Yang, T.-Y., Partsey, R., Desai, R., Clegg, A., Hlavac, M., Min, S. Y., et al. (2023b).
Habitat 3.0: A co-habitat for humans, avatars, and robots.
In Proceedings of International Conference on Learning Representations (ICLR).
- Sarch et al., (2022)
Sarch, G., Fang, Z., Harley, A. W., Schydlo, P., Tarr, M. J., Gupta, S., and Fragkiadaki, K. (2022).
Tidee: Tidying up novel rooms using visuo-semantic commonsense priors.
In Proceedings of European Conference on Computer Vision (ECCV).
- Savva et al., (2019)
Savva, M., Kadian, A., Maksymets, O., Zhao, Y., Wijmans, E., Jain, B., Straub, J., Liu, J., Koltun, V., Malik, J., et al. (2019).
Habitat: A platform for embodied ai research.
In Proceedings of International Conference on Computer Vision (ICCV).
- Scott et al., (2000)
Scott, J. et al. (2000).
Rational choice theory.
Understanding contemporary society: Theories of the present, 129:126–138.
- Sen, (1973)
Sen, A. (1973).
Behaviour and the concept of preference.
Economica, 40(159):241–259.
- Shridhar et al., (2020)
Shridhar, M., Thomason, J., Gordon, D., Bisk, Y., Han, W., Mottaghi, R., Zettlemoyer, L., and Fox, D. (2020).
Alfred: A benchmark for interpreting grounded instructions for everyday tasks.
In Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR).
- Shu and Tian, (2019)
Shu, T. and Tian, Y. (2019).
Mˆ 3rl: Mind-aware multi-agent management reinforcement learning.
In International Conference on Learning Representations.
- Simonson, (2008)
Simonson, I. (2008).
Will i like a “medium” pillow? another look at constructed and inherent preferences.
Journal of Consumer Psychology, 18(3):155–169.
- Singh et al., (2023)
Singh, I., Blukis, V., Mousavian, A., Goyal, A., Xu, D., Tremblay, J., Fox, D., Thomason, J., and Garg, A. (2023).
Progprompt: Generating situated robot task plans using large language models.
In International Conference on Robotics and Automation (ICRA).
- Slovic, (1995)
Slovic, P. (1995).
The construction of preference.
American Psychologist, 50(5):364.
- Song et al., (2023)
Song, C. H., Wu, J., Washington, C., Sadler, B. M., Chao, W.-L., and Su, Y. (2023).
Llm-planner: Few-shot grounded planning for embodied agents with large language models.
In Proceedings of International Conference on Computer Vision (ICCV).
- Sucan et al., (2012)
Sucan, I. A., Moll, M., and Kavraki, L. E. (2012).
The open motion planning library.
IEEE Robotics & Automation Magazine, 19(4):72–82.
- Taniguchi et al., (2021)
Taniguchi, A., Isobe, S., El Hafi, L., Hagiwara, Y., and Taniguchi, T. (2021).
Autonomous planning based on spatial concepts to tidy up home environments with service robots.
Advanced Robotics, 35(8):471–489.
- Thomason et al., (2020)
Thomason, J., Murray, M., Cakmak, M., and Zettlemoyer, L. (2020).
Vision-and-dialog navigation.
In Conference on Robot Learning (CoRL).
- Touvron et al., (2023)
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023).
Llama: Open and efficient foundation language models.
arXiv preprint arXiv:2302.13971.
- Wu et al., (2023)
Wu, J., Antonova, R., Kan, A., Lepert, M., Zeng, A., Song, S., Bohg, J., Rusinkiewicz, S., and Funkhouser, T. (2023).
Tidybot: Personalized robot assistance with large language models.
Autonomous Robots, 47(8):1087–1102.
- (62)
Xu, M., Jiang, G., Liang, W., Zhang, C., and Zhu, Y. (2024a).
Active reasoning in an open-world environment.
Advances in Neural Information Processing Systems, 36.
- (63)
Xu, M., Jiang, G., Liang, W., Zhang, C., and Zhu, Y. (2024b).
Interactive visual reasoning under uncertainty.
Advances in Neural Information Processing Systems, 36.
- Yu et al., (2023)
Yu, K. P., Zhang, Z., Hu, F., and Chai, J. (2023).
Efficient in-context learning in vision-language models for egocentric videos.
arXiv preprint arXiv:2311.17041.
- Yuan et al., (2022)
Yuan, L., Gao, X., Zheng, Z., Edmonds, M., Wu, Y. N., Rossano, F., Lu, H., Zhu, Y., and Zhu, S.-C. (2022).
In situ bidirectional human-robot value alignment.
Science Robotics, 7(68).
- Yuan et al., (2020)
Yuan, T., Liu, H., Fan, L., Zheng, Z., Gao, T., Zhu, Y., and Zhu, S.-C. (2020).
Joint inference of states, robot knowledge, and human (false-)beliefs.
In International Conference on Robotics and Automation (ICRA).
- Yuan et al., (2023)
Yuan, Y., Wang, H., Ding, J., Jin, D., and Li, Y. (2023).
Learning to simulate daily activities via modeling dynamic human needs.
In Proceedings of the ACM Web Conference.
- Zey, (1998)
Zey, M. (1998).
Rational choice theory and organizational theory: A critique.
Sage.
- Zhang et al., (2021)
Zhang, C., Jia, B., Edmonds, M., Zhu, S.-C., and Zhu, Y. (2021).
Acre: Abstract causal reasoning beyond covariation.
In Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR).
- Zhang et al., (2024)
Zhang, H., Du, W., Shan, J., Zhou, Q., Du, Y., Tenenbaum, J. B., Shu, T., and Gan, C. (2024).
zhang2023building.
In Proceedings of International Conference on Learning Representations (ICLR).
- Zhang et al., (2022)
Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., et al. (2022).
Opt: Open pre-trained transformer language models.
arXiv preprint arXiv:2205.01068.
- Zheng et al., (2018)
Zheng, X., Aragam, B., Ravikumar, P. K., and Xing, E. P. (2018).
Dags with no tears: Continuous optimization for structure learning.
In Proceedings of Advances in Neural Information Processing Systems (NeurIPS).
- Zhu et al., (2016)
Zhu, Y., Jiang, C., Zhao, Y., Terzopoulos, D., and Zhu, S.-C. (2016).
Inferring forces and learning human utilities from videos.
In Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR).
- Zitkovich et al., (2023)
Zitkovich, B., Yu, T., Xu, S., Xu, P., Xiao, T., Xia, F., Wu, J., Wohlhart, P., Welker, S., Wahid, A., et al. (2023).
Rt-2: Vision-language-action models transfer web knowledge to robotic control.
In Conference on Robot Learning (CoRL).

## Appendix A Dataset Card

We follow the datasheet proposed in Gebru et al., (2021) for documenting our proposed PbP:

- 1.
Motivation
(a)
For what purpose was the dataset created?
The benchmark was created to evaluate existing learning agents on their ability to understand and adapt to various human preferences. Specifically, it aims to test the agents’ proficiency in few-shot learning from demonstrations, where they must respond to ambiguous task instructions and formulate adaptive task plans based on limited examples of user preferences. The benchmark is designed to highlight the challenges and gaps in current AI systems’ capabilities in planning activities and abstracting human preferences, ultimately driving advancements towards developing more intelligent and personalized embodied agents.
(b)
Who created the dataset and on behalf of which entity?
N/A.
(c)
Who funded the creation of the dataset?
N/A.
(d)
Any other Comments?
None.
- 2.
Composition
(a)
What do the instances that comprise the dataset represent?
Each instance contains an egocentric video of an agent’s activity, its bird’s-eye-view map of the position of the agent, and a frame-level textual annotation of the current action, as shown in [Figure˜4](https://arxiv.org/html/2502.00858v3#S4.F4). Additionally, we provide a rendered third-person view of the entire process.
(b)
How many instances are there in total?
15000.
(c)
Does the dataset contain all possible instances or is it a sample (not necessarily random) of instances from a larger set?
No. The dataset contains a set of demonstrations rendered within the simulator, The users can render more diverse instances if they want. We have provided the rendering instructions.
(d)
What data does each instance consist of?
The instances that comprise the benchmark represent various types of human preferences applied to different tasks within a realistic embodied scene. Each instance is designed to challenge the learning agents to understand and adapt to these preferences based on a few demonstration examples, reflecting the diverse and hierarchical nature of user preferences in real-world scenarios. See above for data details.
(e)
Is there a label or target associated with each instance?
Yes.
(f)
Is any information missing from individual instances?
No.
(g)
Are relationships between individual instances made explicit?
Yes.
(h)
Are there recommended data splits?
No.
(i)
Are there any errors, sources of noise, or redundancies in the dataset?
No.
(j)
Is the dataset self-contained, or does it link to or otherwise rely on external resources (*e.g*., websites, tweets, other datasets)?
Self-contained.
(k)
Does the dataset contain data that might be considered confidential (*e.g*., data that is protected by legal privilege or by doctor-patient confidentiality, data that includes the content of individuals’ non-public communications)?
No.
(l)
Does the dataset contain data that, if viewed directly, might be offensive, insulting, threatening, or might otherwise cause anxiety?
No.
(m)
Does the dataset relate to people?
No.
(n)
Does the dataset identify any subpopulations (*e.g*., by age, gender)?
No.
(o)
Is it possible to identify individuals (*i.e*., one or more natural persons), either directly or indirectly (*i.e*., in combination with other data) from the dataset?
No.
(p)
Does the dataset contain data that might be considered sensitive in any way (*e.g*., data that reveals racial or ethnic origins, sexual orientations, religious beliefs, political opinions or union memberships, or locations; financial or health data; biometric or genetic data; forms of government identification, such as social security numbers; criminal history)?
No.
(q)
Any other comments?
None.
- 3.
Collection Process
(a)
How was the data associated with each instance acquired?
We render PbP using NVIDIA’s Omniverse and OmniGibson simulation environment (Li et al.,, 2023).
(b)
What mechanisms or procedures were used to collect the data (*e.g*., hardware apparatus or sensor, manual human curation, software program, software API)?
The data for each instance in the benchmark was acquired by sampling preferences from a predefined set and constructing tasks paired with a few demonstrations that shared high-level preferences but differed in specific objects and scenes. Each sampled preference was randomly assigned to one of the 50 scenes provided by OmniGibson, with relevant objects sampled within the scene. Egocentric observation and action sequences of an embodied agent were generated as the agent performed tasks guided by a rule-based planner using planning primitives like inverse kinematics for grasping and the A* algorithm for movement.
(c)
If the dataset is a sample from a larger set, what was the sampling strategy (*e.g*., deterministic, probabilistic with specific sampling probabilities)?
N/A.
(d)
Who was involved in the data collection process (*e.g*., students, crowdworkers, contractors) and how were they compensated (*e.g*., how much were crowdworkers paid)?
N/A.
(e)
Over what timeframe was the data collected?
N/A.
(f)
Were any ethical review processes conducted (*e.g*., by an institutional review board)?
The dataset raises no ethical concerns.
(g)
Does the dataset relate to people?
No.
(h)
Did you collect the data from the individuals in question directly, or obtain it via third parties or other sources (*e.g*., websites)?
N/A.
(i)
Were the individuals in question notified about the data collection?
N/A.
(j)
Did the individuals in question consent to the collection and use of their data?
N/A.
(k)
If consent was obtained, were the consenting individuals provided with a mechanism to revoke their consent in the future or for certain uses?
N/A.
(l)
Has an analysis of the potential impact of the dataset and its use on data subjects (*e.g*., a data protection impact analysis) been conducted?
Yes.
(m)
Any other comments?
None.
- 4.
Preprocessing, Cleaning and Labeling
(a)
Was any preprocessing/cleaning/labeling of the data done (*e.g*., discretization or bucketing, tokenization, part-of-speech tagging, SIFT feature extraction, removal of instances, processing of missing values)?
N/A.
(b)
Was the “raw” data saved in addition to the preprocessed/cleaned/labeled data (*e.g*., to support unanticipated future uses)?
N/A.
(c)
Is the software used to preprocess/clean/label the instances available?
N/A.
(d)
Any other comments?
None.
- 5.
Uses
(a)
Has the dataset been used for any tasks already?
No, the dataset is newly proposed by us.
(b)
Is there a repository that links to any or all papers or systems that use the dataset?
No, the dataset is new.
(c)
What (other) tasks could the dataset be used for?
This dataset could be used for research topics like embodied AI and human-computer interaction.
(d)
Is there anything about the composition of the dataset or the way it was collected and preprocessed/cleaned/labeled that might impact future uses?
N/A.
(e)
Are there tasks for which the dataset should not be used?
N/A.
(f)
Any other comments?
None.
- 6.
Distribution
(a)
Will the dataset be distributed to third parties outside of the entity (*e.g*., company, institution, organization) on behalf of which the dataset was created?
No before it is made public.
(b)
How will the dataset be distributed (*e.g*., tarball on website, API, GitHub)?
On our project website upon acceptance.
(c)
When will the dataset be distributed?
Upon acceptance.
(d)
Will the dataset be distributed under a copyright or other intellectual property (IP) license, and/or under applicable terms of use (ToU)?
Under CC BY-NC (^1^11[https://creativecommons.org/licenses/by-nc/4.0/](https://creativecommons.org/licenses/by-nc/4.0/)) license.
(e)
Have any third parties imposed IP-based or other restrictions on the data associated with the instances?
No.
(f)
Do any export controls or other regulatory restrictions apply to the dataset or to individual instances?
No.
(g)
Any other comments?
None.
- 7.
Maintenance
(a)
Who is supporting/hosting/maintaining the dataset?
The authors.
(b)
How can the owner/curator/manager of the dataset be contacted (*e.g*., email address)?
N/A.
(c)
Is there an erratum?
Future erratum will be released through the website.
(d)
Will the dataset be updated (*e.g*., to correct labeling errors, add new instances, delete instances’)?
Yes.
(e)
If the dataset relates to people, are there applicable limits on the retention of the data associated with the instances (*e.g*., were individuals in question told that their data would be retained for a fixed period of time and then deleted)?
N/A. The dataset does not relate to people.
(f)
Will older versions of the dataset continue to be supported/hosted/maintained?
Yes.
(g)
If others want to extend/augment/build on/contribute to the dataset, is there a mechanism for them to do so?
Yes. We will release the source code as well as a licence on our project website after acceptance.
(h)
Any other comments?
None.

## Appendix B Dataset Statistics

The length of the simulations in the dataset ranges from 1 to 5 minutes, depending on the tasks recorded. And the videos are recorded at 30 fps.

### B.1 Preferenes

See [Table˜A1](https://arxiv.org/html/2502.00858v3#A2.T1) for the preference statistics in PbP.

**Table A1: Dataset Statistics in PbP.**
|  | Action_Level | Option_Level | Sequence_Level |
| --- | --- | --- | --- |
| Preference Num | 75 | 135 | 80 |
| Video Num | 5000 | 5000 | 5000 |
| Sub-task Num | 1 | 2-3 | 2-3 |

### B.2 Actions

See [Table˜A2](https://arxiv.org/html/2502.00858v3#A2.T2) for the action statistics in PbP. We implement 17 action primitives in PbP to assist with model planning and dataset rendering. These action primitives have parameters that simplify tasks and are considered the lowest-level actions. Each sub-task contains 8 to 20 such lowest-level actions. Generally, most of these actions consist of two parts: the robot movement part and the arm (gripper) execution part. For robot movement, we use the A* algorithm to find paths and avoid collisions. We build a connection map during scene initialization for navigation, taking the robot’s width into consideration. For the arm (gripper) execution, we primarily use the IK algorithm to compute arm movements. However, since IK cannot handle complex tasks, such as picking objects from the fridge, we also leverage the Open Motion Planning Library (OMPL) planner (Sucan et al.,, 2012) with forward planning to assist in planning the arm positions.

### B.3 More Dataset Details and Discussion

#### Dataset production

The process of producing data is mainly explained in [Section˜4.2](https://arxiv.org/html/2502.00858v3#S4.SS2). In summary, we follow the order of “sample preference - sample scene - sample objects to be manipulated - generate actions guided by a rule-based planner”.

#### Length and FPS of the simulations

The length of the simulations ranges from 1 to 5 minutes, depending on the tasks recorded. The videos are recorded at 30 fps.

#### Actions contained in each simulation

The number of actions in simulations varies among different preference levels. There is 1 subtask for action-level, 2-3 subtasks for option-level, and 2-3 subtasks for sequence-level preferences. Each subtask contains 8-20 actions.

#### Scenes and rooms

Each scene contains various types of rooms. The main differences between scenes are the type, number, and layout of both rooms and furniture. Additionally, each room may contain different objects and have unique layouts. Details of the scenes and rooms can be found in Omnigibson’s official documentation (https://behavior.stanford.edu/omnigibson/), as we directly adopt these scenes from the open-sourced project.

#### 290 preference types

Considering that preferences in household activities are not only multi-dimensional but also hierarchical, we first define a hierarchy of preferences from the perspective of how things happen in a life scenario, that is, from each specific action to a sub-task consisting of several actions, and then to the sequence combining these sub-tasks. The next step is to expand each level with typical tasks and actions. The detailed definition of the 290 preferences can be found in [Section˜4.1](https://arxiv.org/html/2502.00858v3#S4.SS1).

#### The egocentric view

Collecting both egocentric observations and third-person views is feasible in PbP or similar environments built on simulators like iGibson. However, in real-world scenarios, it is generally easier to gather egocentric observations of human daily activities, as these can be efficiently captured through wearable devices. Additionally, there are numerous egocentric-view datasets available, such as Ego4D(Grauman et al.,, 2022), which further facilitate this approach. While third-person views can provide a different perspective, they often encounter issues such as occlusion. Although research based on third-person views is essential for applications involving real robots, focusing on egocentric views in the current work allows for a more straightforward exploration of preference learning and planning. Nevertheless, third-person view data can be obtained by integrating additional cameras, as outlined in our provided code.

#### Action ground truth

In experiments involving vision input, we do not explicitly provide the action sequence of the user. In the symbolic-based experiment, we provide the action sequence to reduce the perception cost to concentrate more effectively on the inference and planning aspects of the study.

**Table A2: Action Primitives in PbP.**
| Action List | Explanation |
| --- | --- |
| Move_to_[] | Move to a specified location, or a specified room, or a specified object |
| Rotate_to_[] | Rotate to a specified orientation or a specified object |
| Pick_[] | Pick up an object using the gripper, *e.g*., “Pick_apple” |
| Place_[] | Place an object at a location, *e.g*., “Place_apple_on_table” |
| Fill_[]_with_[] | Fill a container with a substance, *e.g*., “Fill_glass_with_water” |
| Pour_[] | Pour a substance from a container, *e.g*., “Pour_milk” |
| Open_[] | Open an object, *e.g*., “Open_door” |
| Close_[] | Close an object, *e.g*., “Close_fridge” |
| Cut_[] | Cut an object, *e.g*., “Cut_carrot” |
| Cook_[] | Cook an item, *e.g*., “Cook_pasta” |
| Wash_[] | Wash an object, *e.g*., “Wash_dishes” |
| Clean_[] | Clean a surface or object, *e.g*., “Clean_counter” |
| Cover_[] | Cover an object, *e.g*., “Cover_bowl” |
| Uncover_[] | Uncover an object, *e.g*., “Uncover_bowl” |
| Toggle_on_[] | Turn on a device, *e.g*., “Toggle_on_light” |
| Toggle_off_[] | Turn off a device, *e.g*., “Toggle_off_stove” |
| Wait_[] | Wait some time |

## Appendix C Experiment Details

### C.1 Case Study

We also provide a case with preference Put fruit on the bed in the following table. We present a simplified version of the demonstrations, where all video outputs have been translated into symbol-based action sequences for ease of understanding. Video-based models such as LLaVA-Next and GPT-4V struggle with comprehending preferences and tend to replicate certain action patterns from the video demonstration, such as “move to” and “pick up”. Llama3 demonstrates a partial understanding and execution of the preference. It correctly moves to each fruit (grape, banana), picks them up, and places them on the bed. However, it also interacts with the pencil and places it on the bed, which is not required by the preference. Ideally, the pencil should be placed on the table, similar to the pen. On the other hand, GPT-4(Symbol) accurately interacts with the grape and banana by moving to each fruit, picking it up, and placing it on the bed. This demonstrates a better understanding and execution of the preference compared to the other models.

**Table A3: Case Study with preference Put fruit on the bed.**
| Case | ID | Scene | Actions |
| --- | --- | --- | --- |
| Context | Demo 1 | Beechwood | move to orange, pick up orange, move to bed with orange, place orange on bed, move to apple, pick up apple, move to bed with apple, place apple on bed, move to dvd, pick up dvd, move to table with dvd, place dvd on table. |
|  | Demo 2 | Rs | move to banana, pick up banana, move to bed with banana, place banana on bed, move to club sandwich, pick up club sandwich, move to countertop with club sandwich, place club sandwich on countertop, move to orange, pick up orange, move to countertop with orange, move to bed with orange, place orange on bed. |
|  | Demo 3 | Merom | move to pen, pick up pen, move to table with pen, place pen, move to strawberry, pick up strawberry, move with to bed with strawberry, place strawberry on bed, move to textbook, pick up textbook, move with textbook, move to bookshelf with textbook, place textbook in bookshelf. |
| Test Scene |  | Beechwood | grape, pencil, banana |
| Answer | LLaVA-Next | Beechwood | move to grape, move to bed with grape, move to pencil, pick up pencil, move to banana, pick up banana |
|  | GPT-4V | Beechwood | move to grape, pick up grape, move to bed with grape, place grape on bed |
|  | Llama3 | Beechwood | move to grape, pick up grape, move to bed with grape, place grape on bed, move to pencil, pick up pencil, move to bed with pencil, place pencil on bed, move to banana, pick up banana, move to bed with banana, place banana on bed |
|  | GPT-4(Symbol) | Beechwood | move to grape, pick up grape, move to bed with grape, place grape on bed, move to banana, pick up banana, move to bed with banana, place banana on bed |

## Appendix D Baseline Details

### D.1 ViViT

Inspired by Vision Transformer, ViViT extracts spatio-temporal tokens from the input video and outputs video classification labels for classification. We adopt the ViViT implementation from the official GitHub repo [https://github.com/google-research/scenic](https://github.com/google-research/scenic).

Specifically, we utilize a ViViT with an image size of 224 and a patch size of 16. We extract 2 frames per second from the input video and pad them with the last frame. The Transformer architecture with 3 attention heads operates on features of hidden size of 192 and depth of 4. Each attention head operates on a dimension of 64. We train our model for 30 epochs with a learning rate 3e-5. For the few-shot setting, we concatenate the demo videos temporally.

### D.2 LLaVA

Following the official implementation of LLaVA from [https://github.com/LLaVA-VL/LLaVA-NeXT](https://github.com/LLaVA-VL/LLaVA-NeXT), we test the LLaVA-NeXT-Video-7B-DPO model which is designed for video understanding. Specifically, we run the model following the default inference settings, with vicuna_v1 as the prompt mode, a sample frame number of 32, and a spatial pooling stride of 2. The textual prompts are as follows(^2^22For the textual prompts, we aim to maintain consistency across all LLMs, although some baselines may have additional requirements for the input format. The prompt design is mainly motivated by OpenAI Cookbook [git@github.com:openai/openai-cookbook.git](git@github.com:openai/openai-cookbook.git). We omitted the prompt tuning process, as we found that minor changes in the prompt were unlikely to significantly impact the results. Conversely, selecting the proper demonstrations in the few-shot examples has a much greater influence on the results.):

“Stage One / Preference Prediction”You are a robot assistant that can help summarize the host’s preference.All possible preferences are: {ALL POSSIBLE PREFERENCES}Now there are some prevous video demos:[VIDEO_DEMO_1] The preference is [PREFERENCE_1][VIDEO_DEMO_2] The preference is [PREFERENCE_2][VIDEO_DEMO_3] The preference is [PREFERENCE_3]Now, please summarize the preference from the last video: [TEST_CASE]Quesiton: What’s the user’s preference? Choose from the preference listed before:“Stage Two / Planning”You are a robot assistant. Please view the demos and help generate action sequence.All possible preferences are: {ALL POSSIBLE ACTIONS}Now there are some prevous video demos:[VIDEO_DEMO_1][VIDEO_DEMO_2][VIDEO_DEMO_3]Now you are in the scene with [SCENE DESCRIPTIONS]. Your action sequence is:

### D.3 EILEV

Following the official implementation from [https://github.com/yukw777/EILEV.git](https://github.com/yukw777/EILEV.git), we test the EILEV model in PbP. There are two reasons we chose EILEV among other VLMs as one of our baselines: 1) EILEV elicits in-context learning through a series of architectural modifications and a unique training process, 2) EILEV is trained using ego-centric data, which is compatible with PbP’s input. The textual prompts are as follows. Since EILEV requires the input of the videos and texts to follow a certain pattern for better in-context learning, there are some small modifications to the prompt:

“Stage One / Preference Prediction”You are a robot assistant that can help summarize the host’s preference.All possible preferences are: {ALL POSSIBLE PREFERENCES}Quesiton: What’s the user’s preference? Choose from the preference listed before:Now there are some prevous video demos:[VIDEO_DEMO_1] The preference is [PREFERENCE_1][VIDEO_DEMO_2] The preference is [PREFERENCE_2][VIDEO_DEMO_3] The preference is [PREFERENCE_3][TEST_CASE]“Stage Two / Planning”You are a robot assistant. Please view the demos and help generate action sequence.All possible preferences are: {ALL POSSIBLE ACTIONS}Now there are some prevous video demos:[VIDEO_DEMO_1][VIDEO_DEMO_2][VIDEO_DEMO_3]Now you are in the scene with [SCENE DESCRIPTIONS]. Your action sequence is:

### D.4 GPT-4V

We run our GPT-4 model through the AzureOpenAI API using the GPT version “gpt-4-turbo-2024-04-09”. The API has a limit of 10 images per request. Consequently, for the zero-shot setting, we resample each input video to 8 frames of size 224. For the few-shot setting, where we need to input 3 extra video demonstrations, we concatenate 4 images into a frame, thereby obtaining 4 videos in 8 frames, maintaining the same frame number as the previous setting. We test the model with a temperature of 0.05. The textual prompts are as follows:

“Stage One / Preference Prediction”You are a robot assistant that can help summarize the host’s preference.All possible preferences are: {ALL POSSIBLE PREFERENCES}Now there are some prevous video demos:[VIDEO_DEMO_1] The preference is [PREFERENCE_1][VIDEO_DEMO_2] The preference is [PREFERENCE_2][VIDEO_DEMO_3] The preference is [PREFERENCE_3]Now, please summarize the preference from the last video: [TEST_CASE]Quesiton: What’s the user’s preference? Choose from the preference listed before:“Stage Two / Planning”You are a robot assistant. Please view the demos and help generate action sequence.All possible preferences are: {ALL POSSIBLE ACTIONS}Now there are some prevous video demos:[VIDEO_DEMO_1][VIDEO_DEMO_2][VIDEO_DEMO_3]Now you are in the scene with [SCENE DESCRIPTIONS]. Your action sequence is:

### D.5 DAG-Opt

We implement the DAG-Opt baseline following [https://github.com/xunzheng/notears.git](https://github.com/xunzheng/notears.git). Specifically, we implement a nonlinear NOTEARS using MLP in evaluation.

### D.6 Llama3-8B

We test the Llama3 series model with the official scripts from [https://github.com/meta-llama/llama3](https://github.com/meta-llama/llama3). Specifically, we test the 8B instruction-tuned variant “Meta-Llama-3-8B-Instruct” on PbP. We test the model with a temperature of 0.05. The textual prompts are as follows:

“Stage One / Preference Prediction”You are a robot assistant that can help summarize the host’s preference.Please read the following text file and summarize the user’s preference.All possible preferences are: {ALL POSSIBLE PREFERENCES}[TEXT_ANNOTATION_1] The preference is [PREFERENCE_1][TEXT_ANNOTATION_2] The preference is [PREFERENCE_2][TEXT_ANNOTATION_3] The preference is [PREFERENCE_3]Now, please summarize the preference from the last tet file: [TEST_CASE]Quesiton: What’s the user’s preference? Choose from the preference listed before:“Stage Two / Planning”You are a robot assistant. Please read the following text files and help generate action sequence.All possible preferences are: {ALL POSSIBLE ACTIONS}Now there are some prevous video demos:[TEXT_ANNOTATION_1] (action sequence)[TEXT_ANNOTATION_2] (action sequence)[TEXT_ANNOTATION_3] (action sequence)Now you are in the scene with [SCENE DESCRIPTIONS]. Your action sequence is:

### D.7 GPT-4.1

We use GPT-4.1 from OpenAI with a temperature of 0.05. The textual prompts are as follows:

“Stage One / Preference Prediction”You are a robot assistant that can help summarize the host’s preference.Please read the following text file and summarize the user’s preference.All possible preferences are: {ALL POSSIBLE PREFERENCES}[TEXT_ANNOTATION_1] The preference is [PREFERENCE_1][TEXT_ANNOTATION_2] The preference is [PREFERENCE_2][TEXT_ANNOTATION_3] The preference is [PREFERENCE_3]Now, please summarize the preference from the last tet file: [TEST_CASE]Quesiton: What’s the user’s preference? Choose from the preference listed before:“Stage Two / Planning”You are a robot assistant. Please read the following text files and help generate action sequence.All possible preferences are: {ALL POSSIBLE ACTIONS}Now there are some prevous video demos:[TEXT_ANNOTATION_1] (action sequence)[TEXT_ANNOTATION_2] (action sequence)[TEXT_ANNOTATION_3] (action sequence)Now you are in the scene with [SCENE DESCRIPTIONS]. Your action sequence is:

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 2
presentation: 1
questions: Q1 When this benchmark provides the data for capturing the personalized preference,
how does it handle a particular individual? The dataset is a collection of actions/options/sequences from multiple people/individuals.

Q2 How was the in-context learning with demonstration performed over different models and different modalities?

Q3 Why the proposed approach can handle the personalzied preference?
soundness: 2
strengths: A summary of the overall data statistics will give a strength due to the large number of collections.
The three different levels of data categorization are also interesting, and this could be a good dataset for RL.
The paper also evaluated vision/language models.
summary: The motivation is to let the LLM-based agent handle the preferences of individual users and suggest a response that reflects their preferences.
The main task is to learn preference representation with action observations and use the learned preference representation to perform planning in different situations, illustrated in Figure 1.
Figure 2 shows the organization of the data for the preferences collected at three levels, inspired by hierarchical reinforcement learning.
From the description in the figures, they reflect cooking domains.
Section 4 presents the benchmark, with a test set for evaluation. 
The methods under evaluation are few-shot learning from demonstrations, typically three demonstrations. (It is not clear how this was actually done.)
weaknesses: Sections 1, 2, and 3 don't clearly show what the task is.
It discusses high-level concepts and relevant literature, but it doesn't explain how the benchmark problems are defined/formulated.
It is not clear whether the benchmark is dedicated to the vision-language model or the vision/language only models.
Figure 4 shows the demonstration example with a sequence of figures. This is still confusing to understand what is the input and what is expected to be evalauted.

---

confidence: 2
contribution: 3
initial_rating: 4
presentation: 3
questions: It would be interesting to know how well the approach generalizes beyond the 290 predefined preferences in PBP. In real life, people often have unique habits or styles that weren’t anticipated in the benchmark. Could the agent compose known primitives to infer new ones, or would it need retraining from scratch? Some discussion on how open-ended the preference understanding could be would add depth.

Another open question is real-world deployment. Has the team tried running this on real video demos or physical robots? For example, could it learn how a user likes their tea made from a few wearable-camera clips? Even a small real-world test would help validate how well models trained in simulation hold up when facing noisy perception and less-structured behavior.

The current setup represents preferences as discrete labels, which works well for evaluation but might limit nuance. A learned continuous embedding, encoding each demonstration into a vector the planner can use, could be more flexible and scalable. If the authors explored this, or have thoughts on whether it might outperform classification-based methods, it would be valuable to include.

Finally, there’s the question of efficiency. The best results come from huge models like GPT-4.1 and DeepSeek, which aren’t practical for most applications. Could smaller or specialized models trained directly on PBP data learn preferences more efficiently? Integrating a preference module into a lighter policy network might be one way forward. Insights here could guide future work toward making preference-based planning more accessible and scalable.
soundness: 3
strengths: The paper tackles the interesting challenge of teaching embodied AI to learn personalized user preferences. It introduces Preference-Based Planning (PBP), the first large-scale benchmark designed for this purpose. With hundreds of diverse, hierarchical preferences in realistic simulated environments, PBP provides a strong foundation for future research on personalization in planning.

The authors thoroughly test a wide range of models including vision-language systems (ViViT, LLaVA-Next, GPT-4V) to symbolic planners (Llama, DeepSeek), comparing end-to-end and two-stage setups. The results clearly show that explicitly modeling preferences leads to much better plans, confirming the paper’s core hypothesis. This broad, balanced evaluation gives the findings strong credibility.

Beyond accuracy metrics, the analysis explains why two-stage learning works better: it provides a clear preference cue that guides coherent planning, while end-to-end models fail to infer this latent structure. The paper also discusses overfitting issues in vision-only models and the effects of demonstration count, revealing nuanced insights into model behavior.

The writing is good and well-organized, easy to follow, with effective visuals and detailed methodology. The authors commit to open-sourcing the benchmark and code, which has potential reproducibility and long-term community impact.
summary: This paper introduces Preference-Based Planning (PBP), a large simulated benchmark for testing whether AI agents can learn and follow individual human preferences. Built in NVIDIA Omniverse, it includes 50 scenes and 290 preference types, each shown through a few demo videos.

Agents must infer a user’s preference from demonstrations and then plan actions in new scenarios that match that style. The authors compare end-to-end and two-stage approaches, finding that models perform poorly when planning directly from videos but improve dramatically when given or predicting an explicit preference label. Overall, the paper states that PBP shows that learning preferences is the key bottleneck for personalized, human-aware planning in embodied AI.
weaknesses: One limitation of the work is unclear how well the learned preferences would hold up in messy, real-world conditions where demonstrations are imperfect and preferences don’t fall into neat categories. Even a small-scale user study or robot test discussion would help show that the impact and sim-to-real gap beyond the virtual setup.

The paper also doesn’t introduce a new learning algo, it mostly evaluates existing models like GPT-4, ViViT, and LLaVA within a reasonable two-stage pipeline. The approach makes sense for the problem, but it’s conceptually straightforward, and the novelty lies more in the benchmark and analysis than in new methodology. Some readers might find the contribution more diagnostic than inventive. However, the thorough evaluation across so many baselines gives the work substantial value as a foundation for future research.

In terms of performance, the two-stage design clearly helps, but a big gap remains between using predicted preferences and the true (ground-truth) ones. This shows that current models still struggle to infer user preferences reliably. Moreover, the best results come from massive models like GPT-4.1 and DeepSeek-671B, which are computationally expensive and impractical for deployment. Smaller models such as ViViT or an 8B-parameter Llama perform much worse, raising concerns about scalability and accessibility. It would have been helpful to see a discussion or experiment on how to make preference learning more efficient for lightweight.

While PBP covers an impressive range of preference types, they’re all fixed and labeled. Real preferences are often fuzzier, context-dependent, evolving, and sometimes contradictory. By treating preference inference as a classification problem, the benchmark simplifies the challenge into something more like supervised learning. That’s fine for a first step, but it doesn’t capture the full richness of human personalization. The authors acknowledge this, and it’s worth emphasizing: future work will need to move beyond pre-enumerated labels toward modeling preferences that emerge, shift, and combine dynamically.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: - Would a hybrid approach be possible—for example, performing preference sampling with a rule-based model while delegating action planning to a symbolic model?
- The paper lacks details on how the preference representation model is trained. Could the authors clarify what data and architectures were used for this process?
- During preference prediction, the candidates are explicitly enumerated. Would it be possible to design an open-vocabulary setup where a large language model (LLM) directly infers possible actions and then selects the most preferred one?
soundness: 3
strengths: - Proposes a simple yet intuitive framework that enables agents to infer user preferences and generate corresponding actions from few-shot demonstrations.
- Categorizes preferences into a three-tier hierarchical structure (action, option, sequence), capturing different levels of specificity across tasks.
summary: This paper introduces the Preference-based Planning (PBP) benchmark, designed to evaluate the ability of embodied AI agents to learn and plan based on individual user preferences. Built on NVIDIA Omniverse and OmniGibson, PBP includes 290 diverse preferences across 50 scenes and three abstraction levels—action, option, and sequence—allowing agents to learn personalized behaviors from few-shot demonstrations. The benchmark highlights that learning preferences as intermediate representations significantly enhances adaptability and planning efficiency. Experimental results show that current multimodal and symbol-based models struggle to infer and generalize user preferences, especially from visual demonstrations.
weaknesses: - The definition of Levenshtein distance is not commonly understood; a clear explanation or brief formal description would help readers unfamiliar with the metric.
- The rule-based and scripted demonstrations lack human variability, causing agents to overfit to artificial planning patterns rather than genuine preference behavior.
- Using Levenshtein distance as the main metric penalizes valid alternative plans and fails to reflect real task success or user satisfaction.
- The comparison is unfair because symbolic models receive ground-truth action inputs, while vision-based models must infer preferences from raw egocentric videos.

---

confidence: 3
contribution: 2
initial_rating: 4
presentation: 3
questions: See above - how do you justify the use of Levenshtein distance for such open-ended problems? Or, are the preferences and demos designed to only have one or a few solutions?
soundness: 2
strengths: ### Quality 
- Motivation and background in cognitive literature is useful and more thoughtful than most AI papers in this regard 
- Examples are intuitive and interesting
- Preference examples work. While demonstration construction method doesn't seem to imply a natural distribution, I buy that this benchmark is more about specifically testing ability to reason about high-level latent preferences, not necessarily naturalistic preferences. The former is an important step on the way to the later that I agree needs to be tested in itself.
- Experiment setup makes sense - VLMs as main tests and LLMs as ablations do cover the embodied AI aspect of this benchmark.

### Clarity
- Paper is well-written
- First few figures are well-designed

### Originality and significance 
This is an important problem that I'm excited to see progress on. I appreciate that this paper attempts to benchmark complex personalization, and I also appreciate the difficulty of even setting this problem. Given that a lot of personalization work is closed-source, product-oriented, and doesn't encourage methods beyond RAG, I consider it original and significant to see a research-oriented benchmark toward this end.
summary: This paper presents a benchmark for testing planning methods' ability to adapt to personalized preferences based on prior demonstrations. For example, the benchmark presents a request of a person asking for food to be prepared when they come home, alongside demonstrations from that person's prior interactions which contain information such as the person's food preferences, eating location preferences, etc., all of which might inform the agent on how to do the task at hand. The result is adaptation to the human that reduces need for clarification questions. 

The paper motivates itself with clear examples of preferences and analysis of the distinction between field-standard preferences as in RLHF, language instructions, and here the preferences latent in demonstration information. It goes through significant amounts of cognitive theory, then sets up the benchmark itself: a dataset of action level, option level, and sequence level preferences, plus trajectories relevant to the objects in the preferences, generated synthetically. 

Finally, the paper presents experiments using various VLMs and LLM-based ablations. For action-level preferences, it evaluates primarily through the use of Levenshtein distance between the generated trajectory and the ground-truth trajectory. The distances are high, which the paper attributes to the models' failure to understand overall concepts underlying action sequences and instead output individual matching actions. In general, VLMs fail to infer preferences from demonstrations, and while LLMs succeed when preferences are explicitly given, they still fail in end-to-end scenarios where more abstract reasoning is needed.
weaknesses: ### Quality
- Though the preference generating method is okay despite not being naturalistic, I don't understand the choice of evaluation metric. Using Levenshtein distance suggests that ground truth should be clearly recoverable, despite these seeming to be fairly open-ended problems with many possible solutions at least at the token level. Examples and justification would help here. 
- There are many useful experiments, but the results are presented confusingly - figures with many statistics, followed by general analysis. It would help to have more claims-driven structure with figures and examples that justify each claim; other results can be moved to appendix. 
With justification of the evaluation metric and better structure, I think this paper could be significantly improved. 

### Clarity 
- Tables and data about results are dense and difficult to follow 
- The paper provides explanations for poor performance, e.g. poor symbolic reasoning, but those aren't entailed by the results - instead they are some of many possible explanations. Examples that showcase the lack of abstract reasoning, for example, would help build intuition for why the authors feel that this fully characterizes the errors. 

### Originality and significance
More attention could be paid to efforts such as reward modeling from video or inverse RL (not standard RLHF-type reward modeling), which attempt to learn preference landscapes from data.