COOPERA: Continual Open-Ended Human-Robot Assistance
Authors: Chenyang Ma
Kai Lu
Ruta Desai
Xavier Puig
Andrew Markham
Niki Trigoni
Venue: NeurIPS 2025 spotlight ()
Keywords: Human-Robot Collaboration
Human Simulation
Social Embodied Agent
Primary area: applications
TL;DR: We introduce COOPERA, a novel HRC framework for continual, open-ended collaboration with humans who exhibit individual traits across long horizons.

Abstract: To understand and collaborate with humans, robots must account for individual human traits, habits, and activities over time. However, most robotic assistants lack these abilities, as they primarily focus on predefined tasks in structured environments and lack a human model to learn from. This work introduces COOPERA, a novel framework for COntinual, OPen-Ended human-Robot Assistance, where simulated humans, driven by psychological traits and long-term intentions, interact with robots in complex environments. By integrating continuous human feedback, our framework, for the first time, enables the study of long-term, open-ended human-robot collaboration (HRC) in different collaborative tasks across various time-scales. Within COOPERA, we introduce a benchmark and an approach to personalize the robot's collaborative actions by learning human traits and context-dependent intents. Experiments validate the extent to which our simulated humans reflect realistic human behaviors and demonstrate the value of inferring and personalizing to human intents for open-ended and long-term HRC.

COOPERA: Continual Open-Ended
Human-Robot Assistance
Chenyang Ma' Kai Lu'
Ruta Desait
Xavier Puig
Andrew Markham If
Niki Trigoni
University of Oxford
Abstract
To understand and collaborate with humans, robots must account for individual human traits, habits, and activities over time. However, most robotic assistants lack these abilities, as they primarily focus predefined tasks in structured environ- ments and lack a human model learn from. This work introduces COOPERA, a novel framework for COntinual. OPen- Ended human- simulated humans, driven by psychological traits and long-term intentions, in- teract with robots in complex environments. By integrating continuous human feedback, our framework, for the first time, enables study long-term, open- ended human-robot collaboration (HRC) in different collaborative tasks across various time-scales. Within COOPERA, we introduce a benchmark and an ap- proach to personalize the robot's collaborative actions by learning human traits and context-dependent intents. Experiments validate the extent to which our simulated humans reflect realistic human behaviors and demonstrate the value of inferring and personalizing to human intents for open-ended and long-tew termeermm HRC.
Assistance, where
Introduction
A long-standing goal in robotics is to develop agents that can effectively assist humans in their daily lives by adapting to their preferences and habits. order to this, a robot agent must be able to not only learn to interact in environments with humans in given moment, but also reason about the human across long periods of time, adapting its behavior to provide better assistance. For example, such an agent should be able fetch cup of coffee while also understanding that someone may prefer it cooler in the morning but stronger in the afternoon, heating up water accordingly.
Over recent years, several works have made significant advances in developing agents that can assist humans in household tasks [52 50 75 47), using simulation environments to study human-robot collaboration (HRC) in safe and scalable manner. However, most of these works focus episodic settings, where a robot evaluated over a set short collaboration scenarios with tasks specified in advance. These settings are very different from real-world scenarios, where humans have preferences and long-term goals that guide their behaviors, needing different types of assistance at different times.
To advance robot agents that can assist and adapt to humans, we propose COOPERA, novel framework for COntinual, OPen-Ended human-Roo Robott Assistance in complex household environments (Fig. 1. At its core, COOPERA features a human model with preferences that supports long-term interactions, feedback mechanism, and benchmarks and metrics evaluate if robots can assist humans in long-term tasks and reason about their preferences effectively.
To model realistic humans, we simulate humans using habits, retrieved environment information, and intention history, enabling behaviors that exhibit three characteristies. 1) Dynamic intention-driven: Humans act based on intentions that vary over time
LLM with detailed human traits and
1 Equal advising.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Inter action
Human Traits
for highg
aal Koas
Other Day insracionn
(lonk
Day L. pm Inteaction
Continual Collaboration
Figure 1: Continual human-robot collaboration for open-ended tasks over multiple days. Our framework COOPERA entails an approach to simulate traits-driven humans with long-term, whole-day behaviors within rohot simulation platform, enabling the first study of long-term, open-ended human-robot collahoration. We also introduce a benchmark and method for the robot to personalize collaboration in such continual, open-ended settings by learning human traits and context-dependent intents over time.
(e.g., setting the dinner table 6 pm, then watching TV at 7 pm). 2) Open-ended and environment- conditioned: Rather than following predefined tasks, humans generate spontaneous intentions based on the environment, available objects, and time of day. 3) Traits-driven: Psychological traits and habits shape human behavior, resulting in diverse routines even within similar environments (e.g.. one person starts their day reading, while another prefers cleaning).
As the human interacts in the environment, we need a way to provide feedback to the robot so can improve over time. We structure our framework into two stages which happen each day of interaction. At the beginning of the day, the robot observes and collaborates with the human, assisting in inferred tasks. At the day's 's end, the human communicates with the robot and provides feedback to help improve the robot's 's collaboration success rate for subsequent days.
COOPERA presents unique challenges that are often overlooked in existing HRC benchmarks. First, robot agents need to reason not only about the environment state but also a given human's behavior for effective assistance. Rather than learning behavior that assists all possible humans, they need to adapt to each person's preferences and traits. Second, humans in our framework perform different tasks depending on the time of the day or the day of the week (e.g., they may only do exercise once). Thus, effective agents need to reason about time as a cue for how to best assist the human.
To explore this challenging framework, we provide a benchmark and propose a method that tracks human preference profiles and uses VLMs and classifiers predict and score human goals based on time, observed behavior, and profile, suggesting actions to achieve these intentions. This enables the robot to learn and mimic human behavior by capturing the underlying correlations between human traits, temporal dependencies, and their corresponding intentions and tasks. We compare our method against several baselines, evaluating the robot's collaboration performance over multiple days in different collaborative tasks and across diverse humans and scenes. Furthermore, we conduct extensive experiments to assess how well our simulated humans reflect real human behavior, particularly their ability to exhibit distinct, trait-driven patterns aligned with human profiles.
In summary, our main contributions are threefold:
We present COOPERA. novel HRC framework for continual, open-ended collaboration with humans who exhibit individual traits across long horizons.
We develop method to simulate humans with long-term behavior models driven by individ- ual traits and habits.
Within this framework, we introduce a benchmark and an approach that enables increasingly adaptive and personalized collaboration with humans over multiple days.
2 Related Work
Human- Robot Collaboration. Prior HRC work has largely focused on controlled lab settings 19
10 44 [59] where collaborative tasks are shared by both the human and the robot or narrowly defined. More recent research has expanded to complex household environments, requiring robots to infer human intentions from single demonstration [50 20 64 or in an online fashion [5I]]. Subsequent
works explore human intention inference using data from images 39) or simplified environments (e.g.,
2D worlds) 55 55 70 68), progressing to simulated real-world environments [20] and leveraging recent advances in VIMss. However, these approaches typically rely on predefined, closed-form and often ignore realistic human
representations of human intentions and tasks [4 32 2

---

Continual, open-ended human- behavior. Furthermore, collaboration is usually limited to fixed episodes with predefined task set. In contrast, our work considers open-ended and continual HRC, where humans spontaneously propose their actions based on environmental factors, and the collaboration persists across days.
17 60 assume that environmental changes are solely driven by a single robot [52]. Due to the challenges of real-human experiments (e.g., safety, scalability, cost), recent research has integrated deformable humans with plausible motion and appearance into robot simulation platforms 49 [2 enabling the study of safe and scalable HRC. However, these simulated humans focus only on motion feasibility, lacking the complexity and variability of real human behavior. Another research direction simulates humans with psychological traits and social interactions 776 43 46 26 [64), but remains language-bart and does not involve environmental interaction. In contrast, we simulate humans driven by psychological traits and habits, whose behavior long-term and capable of interacting with their environment.
Human Simulation. Most embodied Al works
LLMs for Human Task Inference. One line of research treats human intentions as direct inputs and investigates how LLMS can interpret open-ended natural language instructions to generate structured 63 (38). These works use techniques such as 3D scene graphs [56] g semantically ground high-level goals and decompose them into actionable subgoals, or incorporate
robot plans [25] 24 23 73
human feedback (58 33 9 to quantify uncertainty and enable skill acquisition through interaction.
In contrast, COOPERA takes a step further by aiming to let LLMs/VLMs infer personalized task
plans, adapting to specific human traits and habits rather than general commonsense knowledge.
3 COOPERA: Continual, Open-Ended HRC Framework
Our goal is to enable the study of continual HRC in open-ended tasks. To that end, we investigate how robotic agent can become more effective in assisting humans by learning from their behavior. Central to COOPERA are LLM--powrtt that the robot can reason for effective collaboration, and human feedback mechanism for improving collaboration over time. We first outline our framework and problem setup, detailing the collaboration settings we explore (Fig. 2 Then, we describe our approach of simulating humans driven by traits with long-term behaviors (Fig. 3}. Finally. we propose a method to tackle our framework (Fig.
simulated humans driven by traits and long-term intentions
3.1 Overview
In order to investigate HRC in safe and repro- ducible manner, we consider a simulated human agent that interacts in a 3D household environ- ment to achieve a set of high-level goals. These goals vary throughout the day and are driven by human traits and habits, as well as by the activ- ities the human has done before. The robot's goal is to assist the human in those tasks, with- out receiving explicit commands about the goal they should help with, or information about the human's traits. Both human and robot have full knowledge of the environment. Each day is rep- resented as 12 onur intervals covering the time from 9 am to 9 pm (the rest is treated as being asleep). At the beginning of each hour, the human proposes high-level intention (e.g.. leisure) and decomposes it into a sequence of tasks and executes them in the environment (c.g., watch TV on the sofa). As the human interacts in the environment, the robot has to infer the human's goals and provide assistance. At the end of each day, the human provides feedback on the robot's help, which is then used to improve the robot's collaboration success in subsequent days.
Continual (Multiple-Day) Open-Endee Enndn HRC Framework
observes
First Task Demo
Human
Model
Assistive Tasks
predicts
Human
Profiie
After Each Day
Opomization Alg.
Feedback
Figure 2: COOPERA: robot collaboration framework. The I.I.M-poered human proposes whole-day intentions and tasks, exe- cuted in the environment. As the robot observes the human actions, it predicts set of tasks to assist them. After each day. the human provides feedback to the robot, cnabling the robot to improve for subsequent days.
Problem Setup. We define two types of collaboration with increasing difficulty and openness. Collaboration type is an open-ended variant of the Watch-and-Heet challenge [50], where one intention (e.g., set up dinner table) is decomposed into undk-and-plcce tasks (i.e., picking an object and placing it on static object). For each intention, the robot is given vidco of the human
3

---

Time
Traits Hebits, Conversations Psychometnc Dara
Rooma in Household Scene
Time, Intention Hatory
Oboet-RRoom Proposed Mapping
Motion Proposed Datauet Motion
Intention
Memory Retrieval
(rosency. relevancet
Search
Search
Profile Extension
Intention Proposal
Tasks Proposal
Human Nlgnmmet Dependence
Execution
Reflect Usage
Thusk lest cer siee
Iecertient Erranine
Figure 3: Human Simulation Pipeline. We seed the human- LLM with an extended profile. At each time of day. the human proposes an intention and decomposes it into tasks, aligning with profile traits and temporal dependence on intention/task history. LLM inputs are optimized with Memory Retrieval and Search. and robustness is enhanced via two rounds of Reflexion. This pipeline generates continuous, whole-day intentions and tasks executed in the environment with expressive whole-body motion. See Appendices and for details.
performing the first task and its textual description, and must infer and assist with the remaining tasks based objects available in the scene. Collaboration type 2 is more challenging and moves beyond pick-and-place each intention (e.g., morning hygiene) is decomposed into tasks involving free-form human motion while interacting with static objects (e.g., the human brushes teeth the mirror). Unlike type 1, the robot is unconstrained by the scene and may propose any object it deems helpful (e.g., the robot offers toothbrush). It receives only the first task video with no textual guidance.
Evaluation Settings. We define four progressively challenging settings. 1) Same human, same scene: The robot collaborates with the same human in the same scene over 5 consecutive days (5 days, 1 scene). 2) Same human, different scenes: The robot collaborates with the same human across different scenes, with a new scene each day (5 days. 5 scenes). 3) Different humans, same scene: The robot collaborates with different humans in the same scene, rotating among Human 1, 2, and 3, each for one day, repeating this cycle three times in the same scene (9 days, scene). 4) Different humans, different scenes: The robot collaborates with different humans across multiple scenes, rotating through Human 1, 2, and 3 in the first scene, then repeating this sequence in the second and third scenes (9 days, 3 scenes). In 3) & 4), we explore if knowledge gained from interacting with different humans improves future collaboration, despite fewer interaction days per human.
3.2 Simulating Humans
We aim to model humans who interact in the environment over long periods of time, act driven by their goals, preferences, and context, and who can react and provide feedback as a robot assists them. To achieve this, we propose a hierarchical model that combines LLMs and 3D human motion to simulate long-term, realistic human behaviors in indoor environments. First, our model generates description of human traits describing their preferences and habits. Based on these traits, the environment, and the history of human actions, the model then generates a sequence of tasks for the human to perform. For every task. and interactions, providing a realistic demonstration of cach task. Fig. 3 shows an overview of our design. Next, we describe in more detail each of the human simulation components.
the environment information to generate human motions
Generating Human Traits. We use LLMs to generate personality traits that determine the human long-tern behaviors. For this, we sample conversations from the Synthetic Human Dataset [26]. containing dialogues between different humans, and prompt an LLM to generate description of the human based on the conversation, inferring attributes such as their job, preferences or common activities. Inspired by [67 77 48
we also prompt the LLM to generate a vector measuring Big-
5 human personality traits 118 (openness, conscientiousness, extroversion, agreeableness, and neuroticism), allowing us to measure the diversity across generated humans and how well the robot agents can infer the human's 's personality from their interactions.
Whole- Intentions and Tasks. Given human traits, we generate long-term human behaviors, with tasks featuring temporal dependences within day and diversity across days. Temporal Depen- dence: Given 3D environment information, we use an LLM to propose intentions for different times of day (e.g., 9 am: clean the living room). Next, we prompt the LLM to decompose the intentions into a sequence of inter-dependent tasks (e.g., dust the area around the white board, clean the counter). The LLM also receives the human's 's intention and task history from previous hours, and is explicitly prompted to consider their inter-dependency. Varying Distribution: While humans with specific traits follow general routines, their daily behavior varies daily (i.c., Monday 9 am for cleaning, Tuesday 9

---

S Model
Human
Inforred foonnt on can help with
Interned Huuman
Trals for Assteaance
LUM CLS
LLM CLS
vy aut teull our
VLM E
Pulco sertiu pisdlovos vour plant.
Hkmmar percaner
Taxin
Time
Fouetion crubre ilea Accions
Inferee Insestians
Iirentin/aaa
email po imopin decamens
Feodback
Aiai fish Day
Interee
Reetiied Himrrry
LUM CLS
(eld panci do sodeing vos
Ristuned
Rrazen
Matti-Dy Collutoratio Hillory
LLS4 Casos
Figure 4: Our approach for human assistance. We decouple robot task inference into intention and task infer- ence. By chaining VLM and classifier, the robot selects tasks aligned with the human's 's traits and temporal context. It maintains human profile inferred from collaboration history, which, combined with feedback, optimizes the robot-VLMM via prompting and the classifiers via supervised learning. See Appendices D for details.
am for exercise). To model this, we reset the intention and task history at the start each new day, setting high temperature for the human-LLM to encourage diversity across days.
Expressive Whole-Body Motion. We simulate human agents physically using expressive 3D whole-body motions during task execution [1 42] by chaining motion sequences for each task. For teck-and-place tasks, the sequence includes walking, reaching and picking, walking again, then reaching and placing. For tasks involving free-form motion (e.g., sitting sofa), the human-LLM describes a free-form human motion that matches each task, using examples from our human motion dataset. The resulting sequence combines walking with the selected free-form motion.
Optimizing Long-Contexs Inputs. Our progressive prompting chain provides the human-LLM with substantial information at each stage, especially during the task proposal stage, where the 3D environment may contain hundreds of objects and the motion dataset includes thousands of data points. Additionally, as the day progresses, the intention and task history grows long (e.g., from 9 am to 9 pm, 13 intention sentences and dozens of task descriptions accumulate). Since LLMs struggle with long-context inputs 30 35 69], we introduce Search and Memory Retrieval mechanisms. Search: Given a query text and a list of texts, we return the top-K most relevant items based on semantic similarity. Memory Retrieval: We use recency and relevance scores to retrieve the top-K memories. Recency decays over time with decay factor from the current time, and relevance is calculated by semantic similarity, similar to search. The final retrieval score is the product of both [46].
Self-Corrections. Given the complexity of our progressive prompting process and the LLM responses, even state-of-the-art models can make mistakes. Therefore, during the most complex task proposal stage, we perform two rounds of Reflexion [6] 74 72 to identify and correct errors related to human traits, temporal dependencies, and object use within the 3D environment.
3.3 Instantiating COOPERA with an Assistive Agent
To study COOPERA, we propose an approach (Fig. 4 for continual HRC, enabling the robot to learn correlations between human intentions, tasks, traits, and temporal dependencies at each time of day.
At any given time, human's 's intentions/tasks can be viewed meta--nnentions/meetaa taakk, encom- passing range of possible options due to the diversity of human behavior across days. Our solution decouples task inference into two stages: first inferring intentions, then identifying specific tasks. We capture the correct sets by chaining VL.M to imagine multiple possible intentions/tasks and classifiers to score and filter them. Given observation (frames uniformly extracted from a video V V= == .. fN]) of the human's 's first task, the robot- generates intention superset. For each positively classified intention by the intention classifier, the robot-VLM infers a set of possible tasks, forming task superset. The task classifier then identifies the tasks most suitable for collaboration.
We optimize the robot-VLM through prompting and the binary classifiers via supervised learning. Using human feedback from the end-of-day discussion, the robot keeps tracks of human profile by prompting Lobot-VLM to infer and summarize the human's traits, habits, and psychometric data. This human profile, along with the retrieved history of intentions and tasks, is incorporated into the robot- VLM prompts and provided as input to the classifiers in the subsequent times and days. The robot- LLM and classifiers are optimized per day. Please see Fig. for the input data format.
5

---

Table 1: Evaluation of 1) human classification, 2) simulated human diversity, 3) human traits-psycchmmeriiss
coherence, 4) temporal dependence, and 5) user studies.
Classification (Ace f)
Coherence (R f)
Temporal Dependence
User Studies (Ace 1)
Diversity (SD 1)
intention
task
aligned
mismatched
Acc 1
F1
MCQ
Matching
0.995
0.830
0.939
0.342
-0.497
0.789
0.790
0.764
0.712
4
Experiments and Analysis
Within COOPERA, we first examine 1) if the central component, the simulated human model, reflects real human behavior and to what extent. 2) We then introduce the benchmark setup (baselines, evaluation metrics) and explore if our proposed approach leads to more personalized robot assistance over multiple days compared to baselines. 3) Subsequently, we analyze the real-worke applicability of our framework. 4) Finally, we evaluate the effectiveness of each module through ablation studies.
4.1 Framework Implementation
Environment and Scene. We use Habitat 3.0 [2 as the robot simulation platform and HSSD 28 as the 3D environment, which includes 18,656 static objects across diverse scenes in style and size. Since the original HSSD includes only static objects, we develop a systematic approach to create dynamic scenes by making small objects from specific categories (e.g.. decor, kitchenware) movable. We also sample 20 dynamic objects from the YCB Datasel (7) and place them in contextually appropriate locations (e.g., mug on a bedside table) using Habitat's s built-in tools. Dynamic scenes are initialized at the start of each episode. Across days, Habitat tracks object locations as the human and robot interact with the environment, allowing them maintain updated environment knowledge. We select 5 scenes with varying of rooms (4-11), I), static objects
and dynamic objects (33-94). All scenes provide enough space for the human and Fetch robot [15] to navigate. Please see Appendix B for more details.
Human Dataset. For modeling unique humans, use the SPC: Synthetic-Personee Chat Dataset (26). a fully synthetic dataset that includes hundreds of short user profiles along with their conversations and compute psychometric data (details in Appendix Cp. We use Motion-X 31 and AMASS [41 as the human motion dataset. We generate 10 human profiles.
Training and Inference. We use open-source models for interpretability and benchmarking value. For simulating humans, we use Llama-3. 1-BB [I3] with temperature 0.7. For search and memory retrieval, we use MiniLM-L6-v2 [66] with decay factor X 0.95, retrieving the top 3 intentions and top tasks. For the assistive agent, we use Llama-3.1 2- IB [ as the robot-VLM. Classifiers are finetuned on Mistral-7B 7B-Instruct--O.. 277 using LoRA 22 in instructional format to output binary yes/no. We train on 3 NVIDIA A10 GPUs (24GB RAM). Please see Appendix D for more details.
4.2 Analysis of Human Simulation
Distinct Simulated Humans. We examine simulated humans with distinct Big-5 traits exhibit srehiine-identfiablee features. Each of 10 humans is placed in 5 scenes, living 20 days per scene. We aggregate daily intentions and tasks into one data point per human. Two way BaRT-large- uncased classifiers [II) are finetuned- -oeeedd--ee-dee for intentions (10 epochs), one for tasks (20 epochs) with train-test split 0.8:0.2, learning rate 5e-6, and tested on an unseen scene. As shown in Table. [ task classification is harder than intention classification, as intentions align more with human traits, but tasks (e.g., drinking water) may correspond to multiple intentions (e.g., leisure, exercise).
Diverse Simulated Humans. We assess diversity by standard deviation (SD) of Big-5 traits (1-5 scale) across 10 simulated humans. We compute per-trait SD and take average. From Table. ! the high SD exceeds the typical 0.7-0.9 range in real-world distributions [22] validating diversity.
Human Traits and Psychometrics Coherence. our main approach, the robot- -LM infers Big-5 traits throughout the day based on the human's 's intention and task history. Using the final scores at the end of collaboration, we assess coherence with ground truth via Pearson correlation 48 4. and introduce one-step mismatch for comparison. The significant drop in correlation for mismatched pairs (Table. T confirms alignment between inferred traits and psychometric data, demonstrating the LLM's ability interpret human psychology from behavior.

---

Habits
Intentions
/
Frue imaginatio perci with arriigge uncorwattin emall lighly
Flydd
Reltor beit ang to pookose
expartnon and coprosa mryee entract aperanntry paster
111
4 pm Imag aod anoyee iig seston Boligg
Cosling scivty in knowe.
renovesioo 1. aguesderras 2.5.
Gyen Dinner
Figure 5: Qualitative examples of full-day intentions and tasks proposed by human with specific human traits and psychometric data.
Temporal Dependence in Human Behavior. We study how current-hour intentions depend on
prior hours via a nexi-sentence prediction task: given three earlier intentions (e.g., 9-11 am), the 12 pm intention is used as the positive example, with other-time intentions as negatives. We evaluate on 10 simulated humans, each living for 20 days in 5 scenes. BERT-large-uncase [TT] is trained for 20 epochs (learning rate 5e-6). Results in Table. confirm temporal dependence.
User Studies. We conduct two user studies (25 participants each) to assess: 1) Whether real humans can identify the same simulated human across days and scenes, and 2) Whether real humans distinguish simulated humans with varying traits and Big-5 scores. For 1), we sample full- day intentions and tasks of 10 simulated humans across 2 days and scenes (4 samples/human), then construct 10 multiple-choice questions showing human profile and three behavior options (1 correct, 2 distractors). For 2), we sample full-day behaviors from 10 simulated humans with distinct traits, and ask participants to match trait descriptions to the corresponding full-day intentions and actions. Results in Table. show higher accuracy in identifying the same simulated human than in distinguishing between different simulated humans. This discrepancy likely arises because multiple- choice tasks provide explicit answer options, reducing ambiguity, whereas trait-based matching requires deeper reasoning about measonality-behaviie relationships, making it more challenging.
Alignment with Real-Human Behavior. We study how simulated human intentions align with real-human intentions. We recruit six participants who provide personality traits and psychometric data, and record their daily intentions over five days. Using these traits, we prompt the LLM to generate simulated intentions for the same time span. To assess aligmment, we aggregate both sets into single paragraphs (removing time formatting like "9am: ... to avoid inflated structural similarity) and compute semantic similarity using SBERT (all- mpnet-base-v2) [57] and OpenAl embeddings ((ext-embedding-3 3-mmlll) [ We compare against: 1) prompting without human profile (generic) and 2) mismatched L.L.-human intention pairs (mismatched). From Table both baselines yield moderate similarity (~-055,, as sentence encoders assign partial similarity to structurally similar content. contrast, aligned pairs achieve much higher scores, indicating strong alignment between simulated and real human intentions. SBERT slightly outperforms OpenAl embeddings, likely due to its sentence-level training objective.
Table 2:
Semantic alignment between simulated and real-humaa human intentions.
Generic
Mismatched
Main
SBERT 1
0.554
0.523
OpenAl Emmh. 1
0.537
0.543
0.772
Qualitative Results. We present examples of full-day intentions and tasks proposed by a human with specific human traits and psychometric data in Fig. 5
4.3 Analysis of Continual, Open-Ended HRC
Since COOPERA involves long-term, open-ended tasks that requires the robot reasoning over human traits and temporal context, we construct baselines using standard LLM/VLM- approaches adapted for task inference. These baselines reflect commonly used paradigms in open-world robot
planning 88 29).
Baselines. 1) Direct Prompting: The robot proposes a single intention from visual input and decom- poses it into tasks. The robot- VLM is optimized solely via prompting with retrieved intention/task history. 2) Direct Finetuning: The robot brain is finetuned to directly output a single intention and decompose it into tasks. 3) Oracle: The robot is given the ground-truth human intention and decomposes into tasks. 4) Random: Intention and task classifiers are removed; all proposed intentions and tasks are accepted without validation. 5) Intention Agnostic: The robot directly
7

---

predicts and filters tasks without first inferring intentions. 6) Human & Context Agnostic: The classifiers do not learn the correlation between human traits and intentions/tasks or the temporal dependence between previous and current intentions/tasks. They only learn the relationship between the current time and the intentions/tasks.
Evaluation Metrics. We assess the assistive agent's performance using Fl-based success rate across three methods, ensuring a comprehensive evaluation from simulation to real-world perspectives, incorporating prior HRC approaches [5] [.. 1) Predicate-based: Tasks are executed and evaluated by predicate functions with success based on object class matches rather than instance matches, following Watch-and and--ell- [50]. 2) LLM-based: Given the ground truth human intention and predicted tasks, the human- -LLM judges whether each task fulfills the intention (binary yes/no), and Fl is computed over these labels. 3) Human verification: Same as 2) but evaluated by real human users. In our main method, the robot- VI.M generates task superset, which is filtered by task classifier assigning yes/no labels used for F1 computation. For baselines without a task classifier (e.g., Direct Prompting). all predicted tasks are treated as positive. Please see Appendices and and for details on predicate construction and LLM evaluation.
Setup. We follow the evaluation settings in Section Setting evaluates humans across 2 scenes; Setting 2 evaluates 10 humans; Setting 3 spans 3 distinct scenes; and Setting 4 spans 9 humans.
Analysis of Assistive Performance. We analyze two aspects: 1) Within-day doprovement-does the robot's collab- oration success increase throughout the day by learning temporal dependencies between human intentions and actions? 2) Across-day improvement-doess col- laboration become more successful and personalized over multiple days, using dnd-of-day feedback? From Fig. (a), our method achieves the highest within- day improvement. In contrast, prompt- ing, random, and oracle exhibit little to no improvement, or even decline. We hypothesize that these methods do not benefit from learning human intentions, which are highly correlated with human traits and temporal context. This finding aligns with our human classification experiment in Section 4.2 From Fig. 6 (b), our method shows the strongest improvement across days, second to oracle. The minimal gain in prompting and finetuning highlights the challenge of varying human behavior across days, as these methods tend to establish 1-to-1 mapping between time and human intentions/tasks. Prompting relies heavily on collaboration history, while finetuning prioritizes the highest probability training data, limiting adaptability to varying human behaviors.
6.11 E.2
013
0024
Changein
Prompong
hireturing
AGruman
Rancort
Figure 6: Evaluation of changes in robot success rate (predicate-based): (a) within a single day and (b) across multiple days. See Appendix for a detailed breakdown of the results and additional analysis.
Out-of-Domain Generalization. We study 1) Scene generalization: can the robot personalize collaboration with human in an unseen scene after interacting in other scenes? 2) Human generalization: can the robot collaborate effectively with new human after training with others? For 1), we use models finetuned from setting on four scenes and evaluate on fifth, unseen scene with the same human. The baseline is the robot's s average performance during its initial interaction in the unseen scene, without finetuning on the previous scenes, averaged over 10 humans. For 2), we use models finetuned from setting 3 on three humans and evaluate on collaboration with a fourth unseen human. The baseline is the robot's unadapted performance when first interacting with the new human, without finetuning on the previous three, averaged over new humans. Result in Table. show that generalizing new human is much harder than to new scene. This is likely due to greater variability in human behaviors. While the robot can learn shared patterns across humans, effective collaboration with new human requires adaptation to fine-grained, person-specific traits.
Table 3: Generalization perfor- mance. We report the average success rate (predicate-based based).
Baseline
Finetuned
Scene
0.269
0.465
Human
0.258
0.343
Qualitative Results We show how the robot improves collaboration within a day by inferring more correct tasks for assistance, along with visualization of HRC specific time in Fig. 7
8

---

Humain proposse intentionn and
Roboot inders intersion bask. Humaan udges colaboration
Lissura lifin room
Enjoy tea
Al the bomann wants agg
robor irtrr intertion is leuure.
relle ng paga as am on the precius
The furnan appeoves the
12 pm--s Lundh kathen
Road cookbook
seap
3 pm
Meditate
bodroam
The sumsnn the The socs
fuon the Uitlen for relsationn
Than Furaan enjays
pm Enjoy raloingg
cony pillow
everiing
Slo drink
Video game with gamepad
9 pm --e Reflect create
Offer pen
bocroom
Blanket comfort
pur the the Godds
cendie
The TAYEOS spore the
dencossu
poum
Human
Robot
Figure 7: Qualitative examples successful human-robot collaboration within one day. The red column displays human intentions, while the blue column shows the robot's 's correctly inferred tasks for assistance.
4.4 Analysis of World Applicability
The ultimate goal of COOPERA is develop robot agents that assist real humans by adapting their preferences over long-term. Yet, fully real-world experiments pose ethical, safety, and cost issues (requiring real human and physical robot interact in a household over multiple days). To validate applicability of COOPERA under real-world conditions, we take three complementary approaches. Please see Appendix E for detailed results, qualitative examples, and additional analysis.
Human Verification. We validate if predicate- based and LLM- -asedd evaluations align with hu- man verifications by sampling one episode per setting from our main method. From Table. 4 the low L1 indicates strong correlation among framework's
Table 4: Correlation between predicates, LLM, and human evaluations (rows 1-2): LI U, averaged over collaboration types. Offline real-human and human- in-the-loop collaboration (row 3-4): predicate- success rate (SR f), averaged over the final day.
all three metrics, supporting real-world applicability. This justifies our use of VLMs/LLMs for predicting human actions in both baselines and main method, as their reason- ing closely resembles human decision-making. Also, in open-ended settings with large state spaces, LLMs/VLMs serve as effective reasoning modules due to their generalization abilities [ 29).
Setting
Setting 2
Setting 3
Setting
Prodicate va. Real-Human (1.1
0.091
0.091
0.085
0.120
LLM vs. Real Human (L) 1)
0,077
0.080
0,77
0.075
Iluman
Humon-in-tho-Leey
Collaborating with Offline Real Humans. Real humans exhibit greater behavioral dynamics and emergent decisions due temporary factors (e.g., plans, mood, weather). We recruit six participants who provide personality traits and psychometric data, and record their daily intentions over five days. An LLM decomposes these into tasks in HSSD scenes, where the robot collaborates across all settings. Results in Table A show performance comparable to simulated humans (Table. row 4), despite increased dynamics.
Six real humans replace the LLM and collaborate with the assistive agent. They are shown retrieved object and motion sets and select which to interact with based on their intention. For ease of implementation and usability, participants input responses as text rather than using keyboard to control the simulated agent. Results in Table indicate that real human collaborators do not make the task more difficult and can, in some cases, lead to higher success rates compared to offline simulated humans.
Human-in-the-LLopp.
4.5 Ablation Studies
Human Simulation. 1) Removing human profile extension: To explore whether our method yields the most distinct simulated humans, we remove simulated conversations and profile extension, prompting LLM only with the original short trait paragraph. We evaluate by finetuning and measuring

---

Table 5: Ablation study human simulation. The effects of removing profile extension on human clas- sification and using single-shot intention proposal on temporal dependence.
Table 6: Ablation study assistive agent: success rate (predicate-based) f averaged over the last day.
Setting
Setting 2
Setting
Setting
No Traits
0.481
0.443
0.239
0.206
Removiig Profile Extension
Intention Proposal
No Context
0
0.414
0.408
0,299
insetiin cls
look cl.
Removed
0,950
0.800
0,251
0,740
Changing Backhone
0.487
0.362
0,310
Ous
0:995
0.830
0,789
0,790
Ours (main)
0.505
0.465
0.439
0,344
classifiers accuracy (Section 4.2) 2) Single-shot human intention proposal: We test if our pipeline
design maximizes temporal dependence in human behavior. Instead of proposing intentions hour by hour with access to intention/task history and using Reflexion, remove these and generate all intentions at once. We evaluate via next-intention prediction (Section 4.2). Results in Table. [ confirm the effectiveness of the profile extension module and the overall pipeline design.
Assistive Agent. 1) Removing human traits inference. We examine the importance of learning human traits by removing robot's inference of human traits or Big-5 scores from past intentions and tasks, preventing classifiers from learning their correlation. 2) Removing temporal context learning: We assess the impact of learning temporal dependence between human intentions and tasks by preventing classifiers from using past intentions/tasks when predicting the current one. 3) Changing the robot brain backbone: We replace the robot- VLM from 3.2ma-3.2-1BB [3] to LLaVA-I.6-Mistral
(34.. Using different VLMs for human and robot reduces alignment and tests robustness. From Table 6 removing trait inference significantly reduces success rate in settings 3 and 4 involving multiple humans, as the robot struggles to distinguish them. The learning of time-based context benefits all settings. Despite smaller model size, the robot still achieves reasonable success.
5 Conclusion
We introduce COOPERA, a framework for continual, open-ended HRC. We propose a human model to generate long-term human behaviors driven by personality traits, a benchmark, and a method to assist humans under by predicting their long-term intentions. Our framework opens exciting directions for future work, such as using communication better infer human traits or build agents that can perform proactive assistance (e.g. arranging house before the start of the day based on preferences). We hope that this work can promote future research on building agents that can work over long time horizons and adapt to human preferences.
Limitations. Despite compelling HRC performance, COOPERA currently focuses single-human settings, leaving multi-human collaboration for future work. While evaluations are primarily con- ducted in simulation, validate sim-to-real transfer through real-human routines and interactions. Our method uses skill primitives compatible with standard robot platforms (e.g., Fetch), making it readily transferable real hardware.
10

---

References
[1] Open AI. New embedding models and api updates. 2024.
httpp:://penai.coo cotiinaii.oom/inndaiee
[2] Stefano V. Albrecht and Peter Stone. Autonomous agents modelling other agents: A compre- hensive survey and open problems. Artificial Intelligence, pages 66-95, 2018.
[3] Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian D. Reid, Stephen Gould, and Anton van den Hengel. Vision-and-lange Sainn-and-language navigation: Interpreting disually-groundee navigation instructions in real environments. In Conference on Computer Vision and Puttern Recognition, pages 3674-3683, 2018.
[4] Danny Azucar, Davide Marengo, and Michele Settanni. Predicting the big 5 personality traits from digital footprints social media: A meta-analysis. Personality and Individual Differences, pages 150-159, 2018.
[5] Chris L. Baker, Julian Jara-Eutinger, Rebecca Saxe, and Joshua B. Tenenbaum. Rational quantitative attribution of beliefs, desires and percepts in human mentalizing. Nature Human Behaviour, 2017.
[6] Dhruv Batra, Aaron Gokaslan, Aniruddha Kembhavi, Oleksandr Maksymets, Roozbeh Mottaghi, Manolis Savva, Alexander Toshev, and Erik Wijmans. Objectnav revisited: On evaluation of embodied agents navigating objects. arXiv preprini arXiv:2006. 2020.
[7] Berk Çalli, Arjun Singh, Aaron Walsman, Siddhartha S. Srinivasa, Pieter Abbeel, and Aaron M. Dollar. The YCB object and model set: Towards common benchmarks for manipulation research. In International Conference Advanced Robotics, pages 510-517. IEEE, 2015.
[8] Matthew Chang, Gunjan Chhablani, Alexander Clegg, Mikael Dallaire Cote, Ruta Desai, Michal Hlavac. Vladimir Karashchuk, Jacob Krantz, Roozbeh Mottaghi, Priyam Parashar, Siddharth Patki, Ishita Prasad, Xavier Puig, Akshara Rai, Ram Ramrakhya, Daniel Tran, Joanne Truong, John M. Turner, Eric Undersander, and Tsung- Yen Yang. Partnr: A benchmark for planning and reasoning in embodied multi-age ailti- tasks. In International Conference on Learning Representations, 2025.
[9] Jinghong Chen, Guangyu Yang, Weizhe Lin, Jingbiao Mei, and Bill Byrne On extending direct preference optimization to accommodate ties. arXiv preprint arXiv:240..14331 2024
[10] Kerstin Dautenhahn. Socially intelligent robots: dimensions of human-robot interaction. Philo- sophical Transactions of the Royal Society of London. Series B, Biological Sciences, pages 679-704, 2007.
[11] Jacob Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Conference of the North American Chapter of the Association for Computational Linguistics, pages
2019.
[12] Yuqing Du, Stas Tiomkin, Emre Kiciman, Daniel Polani, Pieter Abbeel, and Anca D. Dragan. Ave: Assistance via empowerment. In Advances in Neural Information Processing Systems, 2020.
[13] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurélien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Rozière, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Graeme Nail, Grégoire Mialon, Guan Pang. Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arricta Ibarra,
11

---

Isabel M. Kloumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, and et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 2024.
[14] Alan Fern. Sriraam Natarajan, Kshitij Judah, and Prasad Tadepalli. A decision-theoretic model of assistance. Journal of Artificial Intelligence Research, 50:71-104 2014.
[15] Fetch Robotics. Fetch, 2020.
[16]
Chuang Gan, Jeremy Schwartz, Seth Alter, Damian Mrowca, Martin Schrimpf, James Traer, Julian De Freitas, Jonas Kubilius, Abhishek Bhandwaldar, Nick Haber. Megumi Sano, Kuno Kim, Elias Wang. Michael Lingelbach, Aidan Curtis, Kevin T. Feigelis, Daniel Bear, Dan Gutfireund, David D. Cox, Antonio Torralba, James J. DiCarlo, Josh Tenenbaum, Josh H. McDermott, and Dan Yamins. Threedworld: A platform for interactive multi-modal physical simulation. In Neural Information Processing Systems Track on Datasets and Benchmarks, 2021.
[17] Théophile Gervet, Soumith Chintala, Dhruv Batra, Jitendra Malik, and Devendra Singh Chaplot. Navigating to objects in the real world. Science Robotics, 2023.
[18] LR R Goldberg. An alternative "description of personality": the big-five factor structure. Journal of Personality and Social Psychology, pages 1216-122 1229, December 1990.
[19] Michael A. Goodrich and Alan C. Schultz. Human-robot interaction: A survey. Foundations and Trends in Human CoomannComppter Interaction, pages 203-275, 2007.
[20] Moritz A. Graule and Volkan Isler. GG-LLM: geometrically grounding large language models for zero-shot human activity forecasting in human-aware task planning. In International Conference on Robotics and Automation, pages 568-574. IEEE, 2024.
[21] Chuan Guo, Shihao Zou, Xinxin Zuo, Sen Wang, Wei Ji, Xingyu Li, and Li Cheng. Generating diverse and natural 3d human motions from text. In Conference on Computer Vision and Pattern Recognition, pages 51 5152-5151. IEEE, 2022.
[22] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.
[23] Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. In International Conference Machine Learning, pages 9118-9147, 2022.
[24] Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, Pierre Sermanet, Tomas Jackson, Noah Brown, Linda Luu, Sergey Levine, Karol Hausman, and Brian Ichter. Inner monologue: Embodied reasoning through planning with language models. In Conference on Robot Learning, pages 1769-1782, 2022.
[25] Brian Ichter, Anthony Brohan, Yevgen Chebotar, Chelsea Finn, Karol Hausman, Alexander Herzog, Daniel Ho, Julian Ibarz, Alex Irpan, Eric Jang, Ryan Julian, Dmitry Kalashnikov, Sergey Levine, Yao Lu, Carolina Parada, Kanishka Rao, Pierre Sermanet, Alexander Toshev, Vincent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Mengyuan Yan, Noah Brown, Michael Ahn, Omar Cortes, Nicolas Sievers, Clayton Tan. Sichun Xu, Diego Reyes, Jarck Rettinghouse, Jornell Quiambao, Peter Pastor, Linda Luu, Kuang-Huet Lee, Yuheng Kuang, Sally Jesmonth, Nikhil J. Joshi, Kyle Jeffrey,, Rosario Jauregui Ruano, Jasmine Hsu, Keerthana Gopalakrishnan, Byron David, Andy Zeng, and Chuyuan Kelly Fu. Do as can, not as say: Grounding language in robotic affordances. In Conference on Robot Learning, pages 287-318, 2022.
[26] Pegah Jandaghi, XiangHai Sheng, Xinyi Bai, Jay Pujara, and Hakim Sidahmed. Faithful persona-based conversational dataset generation with large language models. In Findings of the Association for Computational Linguistics, pages 15245-15270, 2024.
12

---

[27] Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Manie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang. Timothée Lacroix, and William El Sayed. Mistral 7b, arXiv preprint arXiv:23 2023.
[28] Mukul Khanna, Yongsen Mao, Hanxiao Jiang, Sanjay Haresh, Brennan Shacklett, Dhruv Batra, Alexander Clegg, Eric Undersander, Angel X. Chang, and Manolis Savva. Habitat synthetic scenes dataset (HSSD-200): an analysis of 3d scene scale and realism tradeoffs for objectgoal navigation. Conference on Computer Vision and Pattern Recognition, pages (6384-16393, IEEE, 2024.
[29] Manling Li, Shiyu Zhao, Qineng Wang. Kangrui Wang, Yu Zhou, Sanjana Srivastava, Cem Gokmen, Tony Lee, Erran Li, Ruohan Zhang, Weiyu Liu, Perey Liang. Li Fei-Fei, Jiayuan Mao, and Jiajun Wu. Embodied agent interface: Benchmarking Ilms for embodied decision making. In Advances in Neural Information Processing Systems, 2024.
[30] Tianle Li, Ge Zhang, Quy Due Do, Xiang Yue, and Wenhu Chen. Long-conterd Ilms struggle with long in-context learning, arXiv preprint arXiv:2404.02060, 2024.
[31] Jing Lin, Ailing Zeng, Shunlin Lu, Yuanhao Cai, Ruimao Zhang, Haoqian Wang, and Lei Zhang. Motion-x: large-scale 3d expressive whole-body human motion dataset. In Advances in Neural Information Processing Systems, 2023.
[32] Chang Liu, Jessica B. Hamrick, Jaime F. Fisac, Anca D. Dragan, J. Karl Hedrick, S. Shankar Sastry, and Thomas Griffiths. Goal inference improves objective and perceived performance in human-robot collaboration. In International Conference on Autonomous Agents & Multiagent Systems, pages 940-948. ACM, 2016.
[33] Haokun Liu, Yaonan Zhu, Kenji Kato, Atsushi Tsukahara, Izumi Kondo, Tadayoshi Aoyama, and Yasuhisa Hasegawa. Enhancing the llm-based robot manipulation through human-robot collaboration. IEEE Robotics and Automation Letters, pages 6904-6911, 2024.
[34] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuiigg In Conference on Computer Vision and Puttern Recognition, pages 26286- 26296. IEEE, 2024.
[35] Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, pages 157-173, 2024.
[36] Yuchen Liu, Luigi Palmieri, Sebastian Koch, Ilche Georgievski, and Marco Aiello. DELTA: decomposed efficient long-tern -erm-termme robot task planning using large language models, arXiv preprins arXiv:2404.03275. 2024.
[37] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019.
[38] Kai Lu, Chenyang Ma, Chiori Hori, and Diego Romeres. Kitchenvla: Iterative vision-languaee corrections for robotic execution of human tasks. In International Conference on Robotics and Automation, Workshop on Safe Vision- LangugLa Models for Robotics, 2025.
[39] Chenyang Ma, Kai Lu, Ta- Ying Cheng, Niki Trigoni, and Andrew Markham. Spatialpin: Enhancing spatial reasoning capabilities of vision-language models through prompting and interacting 3d priors. Proceedings of the Conference on Neural Information Processing Systems, 2024.
[40] Chenyang Ma, Xinchi Qiu, Daniel Beutel, and Nicholas Lane. Gradient-less federated gradient boosting tree with learnable learning rates. In Proceedings of the 3rd Workshop on Machine Learning and Systems, pages 56-63, 2023.
13

---

[41] Naureen Mahmood, Nima Ghorbani, Nikolaus F. Troje, Gerard Black. AMASS: archive of motion capture as surface shapes. In 2019 IEEE/CVE International Conference Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 November 2, 2019, pages 5441-5450. IEFE, 2019.
and Michael J.
[42] Aymen Mir, Xavier Puig. Angjoo Kanazawa, and Gerard Pons-Moll. Generating continual human motion in diverse 3d scenes. In International Conference on 3D Vision, pages 903-913. IEEE, 2024.
[43] Xinyi Mou, Jingcong Liang, Jiayu Lin, Xinnong Zhang, Xiawei Liu, Shiyue Yang, Rong Ye, Lei Chen, Haoyu Kuang, Xuanjing Huang, and Zhongyu Wei. Agentsense: Benchmarking social intelligence of language agents through interactive scenarios, 2024.
[44] Stefanos Nikolaidis, Ramya Ramakrishnan, Keren Gu, and Julie A. Shah. Efficient model learning from joint-action demonstrations for human-robot collaborative tasks. In International Conference on SumannRRbol Interaction, pages 189-196. ACM, 2015.
[45]
OpenPsychometrrics The big five personality test, n.d.
[46] Joon Sung Park, Joseph C. O'Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. Generative agents: Interactive simulacra of human behavior. In Symposium User Interface Software and Technology, pages 2:1-2:22. ACM, 2023.
[47] Claudia Pérez-D' Arpino, Can Liu, Patrick Goebel, Roberto Martín-Martin and Silvio Savarese. Robot navigation in constrained pedestrian environments using reinforcement learning. In 202/ IEEE International Conference on Robotics and Automation (ICRA), pages 1 1140-1146. IEEE, 2021.
[48] Heinrich Peters and Sandra C. Matz. Large language models can infer psychological dispositions of social media users. arXiv preprint arXiv:2309.08631 2023.
[49] Xavier Puig, Kevin Ra, Marko Boben, Jiaman Li, Tingwu Wang, Sanja Fidler, and Antonio Torralba. Virtualhome: Simulating household activities via programs. In Conference on Computer Vision and Pattern Recognition, pages 8494 8502, 2018.
[50] Xavier Puig, Tianmin Shu, Shuang Li, Zilin Wang, Yuan-Hong Liao, Joshua B. Tenenbaum, Sanja Fidler, and Antonio Torralba. Watch-and-helpp A challenge for social perception and human-ai collaboration. In International Conference on Learning Representations, 2021.
[51] Xavier Puig, Tianmin Shu, Joshua B. Tenenbaum, and Antonio Torralba. NOPA: neurally- guided online probabilistic assistance for building socially intelligent home assistants. In International Conference on Robotics and Automation, pages 7628-7667 7634. IEEE, 2023.
[52] Xavier Puig, Eric Undersander, Andrew Szot, Mikael Dallaire Cote, Tsung- Yen Yang, Rus- lan Partsey, Ruta Desai, Alexander Clegg, Michal Hlavac, So Yeon Min, Vladimir Vondrus, Théophile Gervet, Vincent-Picce Berges, John M. Turner, Oleksandr Maksymets, Zsolt Kira, Mrinal Kalakrishnan, Jitendra Malik, Devendra Singh Chaplot, Unnat Jain, Dhruv Batra, Ak- shara Rai, and Roozbeh Mottaghi. Habitat 3.0: A co-habitat for humans, avatars, and robots. In International Conference on Learning Representation, 2024.
[53] Xinchi Qiu, Heng Pan, Wanru Zhao, Chenyang Ma. Pedro Porto Buarque de Gusmao, and Nicholas Donald Lane. Efficient vertical federated learning with secure aggregation. In Federated Learning Systems (FLSys) Workshop@ @ MLSys 2023, 2023.
[54] Xinchi Qiu, Heng Pan, Wanru Zhao, Chenyang Ma, Pedro PB Gusmao, and Nicholas D Lane. vfedsec: Efficient secure aggregation for vertical federated learning via secure layer. arXiv preprint crXiv:2305.1679.0 2023.
[55] Neil C. Rabinowitz, Frank Perbet, H. Francis Song, Chiyuan Zhang, S. M. Ali Eslami, and Matthew M. Botvinick. Machine theory of mind. In International Conference on Machine Learning, pages 4215-4224, 2018.
14

---

[56] Krishan Rana, Jesse Haviland, Sourav Garg, Jad Abou- Chakra, Ian D. Reid, and Niko Sünder- hauf. Sayplan: Grounding large language models using 3d scene graphs for scalable robot task planning. In Conference on Robot Learning, pages 23-72, 2023.
[57] Nils Reimers and Iryna Gurevych. Sentence- bert: Sentence embeddings using siamese bert- networks. In International Joint Conference on Natural Language Processing, pages 3980-3990, 2019.
[58] Allen Z. Ren, Anushri Dixit, Alexandra Bodrova, Sumeet Singh, Stephen Tu, Noah Brown, Peng Xu, Leila Takayama, Fei Xia, Jake Varley, Zhenjia Xu, Dorsa Sadigh, Andy Zeng, and Anirudha Majumdar. Robots that ask for help: Uncertainty alignment for large language model planners. In Conference on Robot Learning, pages 061-682,220 2023.
[59] Leonel Dario Rozo, Sylvain Calinon, Darwin G. Caldwell, Pablo Jiménez, and Carme Torras. Learning physical collaborative robot behaviors from human demonstrations. IEEE Transactions on Robotics, pages 513-527, 2016.
[60] Dhruv Shah, Ajay Sridhar, Nitish Dashora, Kyle Stachowicz, Kevin Black, Noriaki Hirose, and Sergey Levine. Vint: A foundation model for visual navigation. In Conference on Robot Learning, pages 71 711-733, 2023.
[61] Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Re- flexion: language agents with verbal reinforcement learning. In Advances in Neural Information Processing Systems, 2023.
[62] Christopher Soto and Oliver P John. The next big five inventory (BFI-2): assessing a hierarchical model with 15 facets to enhance bandwidth, fidelity, and predictive power. Journal of Personality and Social Psychology, pages 117-143, 2017.
Developing and
[63] Andrew Szot, Max Schwarzer, Harsh Agrawal, Bogdan Mazoure, Rin Metcalf, Walter Talbott, Natalie Mackraz, R. Devon Hjelm, and Alexander T. Toshev. Large language models as gener- alizable policies for embodied tasks. In International Conference on Learning Representations, 2024.
[64] Yanming Wan, Yue Wu, Yiping Wang, Jiayuan Mao, and Natasha Jaques. Infer human's intentions before following natural language instructions. arXiv preprint arXiv:2409. 18073, 2024.
[65] Chen Wang, Claudia Pérez-D' Arpino, Danfei Xu, Li Fei-Fei, Karen Liu, and Silvio Savarese. Co-gail: Learning diverse strategies for human-robot collaboration. In Conference on Robot Learning, pages 1799-1200. PMLR, 2022.
[66] Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. Minilm: Deep self-attention distillation for task task-agnostic compression of pre-trained transformers. In Advances in Neural Information Processing Systems, 2020.
[67] Xuewei Wang, Weiyan Shi, Richard Kim, Yoojung Oh, Sijia Yang, Jingwen Zhang, and Zhou Yu. Persuasion for good: Towards personalized persuasive dialogue system for social good. In Conference of the Association for Computational Linguistics, pages 5635-5649, 2019.
[68] Sarah A. Wu, Rose F. Wang, James A. Evans, Joshua B. Tenenbaum, David C. Parkes, and Max
Too many cooks: Bayesian inference for coordinating multi-agent collaboration. Topics in Cognitive Science, pages 414-432, 2021.
[69] Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, Madian Khabsa, Han Fang, Yashar Mehdad, Sharan Narang, Kshitiz Malik, Angela Fan, Shruti Bhosale, Sergey Edunov, Mike Lewis, Sinong Wang, and Hao Ma. Effective long-context scaling of foundation models. In Conference of the North American Chapter of the Association for Computational Linguistics, pages 4643-4663, 2024.
[70] Fengyu Yang and Chenyang Ma. Sparse and complete latent organization for geospatial semantic segmentation. In Conference on Computer Vision and Puttern Recognition, pages 1809-1818. 2022.
15

---

[71] Fengyu Yang, Chenyang Ma, Jiacheng Zhang, Jing Zhu, Wenzhen Yuan, and Andrew Owens. Touch and go: Learning from collenee vision and touch. Advances in Neural Informa- lion Processing Systems, pages 8081-8103, 2022.
[72] Guangyu Yang, Jinghong Chen, Weizhe Lin, and Bill Byme. Direct preference optimization for neural machine translation with minimum Bayes risk decoding. Conference of the North American Chapter of the Association for Computational Linguistics, pages 391-398, 2024.
[73] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R. Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations, 2023.
[74] Mert Yüksckgönül, Federico Bianchi, Joseph Boen, Sheng Liu, Zhi Huang, Carlos Guestrin, and James Zou. Textgrad: Automatic "differentiation" via text. arXiv preprint arXiv:2406007496,
2024.
[75] Hongxin Zhang, Weihua Du, Jiaming Shan, Qinhong Zhou, Yilun Du, Joshua B Tenenbaum, Tianmin Shu, and Chuang Gan. Building cooperative embodied agents modularly with large language models. arXiv preprint (rxiv:2307.02485, 2023.
[76] Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. DIALOGPT Large-scale generative pre-training for conversational response generation. In Annual Meeting of the Association for Computational Linguistics, pages 270-278,11 2020.
[77] Xuhui Zhou, Hao Zhu, Leena Mathur, Ruohong Zhang, Haofei Yu. Zhengyang Qi, Louis- Philippe Morency, Yonatan Bisk, Daniel Fried, Graham Neubig, and Maarten Sap. SOTOPIA: interactive evaluation for social intelligence language agents. In International Conference on Learning Representations, 2024.
16

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the
paper's contributions and scope?
Answer: [Yes]
Justification:
We conducted extensive empirical experiments to demonstrate it.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discussed limitations the Conclusion.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
17

---

Justification:
We did not have theoretical assumptions.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We have thoroughly discussed implementations details in the main paper and Appendix.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
18

---

Answer: [No]
Justification: We did not provide code with the submission because of internal regulations within the authors' organizations but will release after acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co ec/
public/guides/ htwps///oooeeuubsssssoooooooo
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https http://ooSSuuiiss..eoeeee Stt.SSabmiiiioooiiiiii.oooi for more details.
//nips.c//pp wwwppbblccgguiicuceccccccoeooope
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We discussed these in the Experiments and Appendix with lot of details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification:
In our experiments, large time aand computation cost is needed to do this.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
19

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We discussed in the Experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http:////aaaar...cppppaaatttaaaa
Answer:
[Yes]
Justification:
We followed the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our work does not have broader social impacts.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
20

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
No such concern.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All properly cited.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
21

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: All well documented.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We do not have large-scale crowdsourcing experiments.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This work does not have such.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
22

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification:
We explained in great details in framework and experiment sections.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy fattps://neurd/ http::///eerrppsscc///oeeeeeerer co// for what should or should not be described.
23

---

Appendix for COOPERA
A
Overview
This Appendix includes: 1) more details about how construct dynamic HSSD scenes and statistics, 2) additional details of how we build, train, and evaluate the simulated human and the assistive agent, 3) additional HRC results and analysis, including both quantitative metries and qualitative examples, along with comparisons to human verification of our main method and baselines across collaboration types and settings, and 4) prompt details.
B Additional Details of Scenes
B.1 Dynamic Habitat Synthetic Scenes Dataset Construction
scenes dynamic by checking they are supported by any structure and their object super categories. For objects that do not have support, classify them as follows:
We make certain objects in the HSSD 40 53 54
dynamic categories = [trashcan, decor, dining ware, plant, electronics, animate object, apparel, liquid container, kitchen ware, tray, bathroom accessory, gym equipment, toy, wearable]
static_ cattiorietegeesssseeeeeeeeeeeeee = [storage furniture, support furniture, seating furniture, floor covering, lighting, sleeping furniture, bathroom fixtures, mirror, large kitchen appliance, large appliance, kitchen bathroom fixture, vehicle, heating cooling, medium kitchen appliance, display, arch, curtain, small kitchen appliance]
B.2 Scene Statistics
Descr
Bestnwan
Merroy
Plant
Bedreem
Static (56.1%) Dynamic (43.9%)
Support
Storage
Display
Fumiitree
Ploor Cover
Bamiturr
sealing
Others
Figure 8: Distribution object categories within constructed dynamic HSSD scenes.
Our selected scenes feature varying number of rooms (4-11), static objects (51 51--400, and dynamic objects (33-94) We present the distribution of 18 representative object categories (out of 32) in Fig. 8 The large number and diversity of objects and categories enable humans propose wide range of open-ended tasks.
B.3 Scene Summarization and Visualizations
We summarize 3D environment information each scene text-based dictionary, which used as input to the simulated humans. Specifically, we extract the bounding boxes of rooms and map each object to its corresponding room, forming an object-room mapping the format of object_IID ID: [object_nanp] name, room]. For example:
mapping = ( ('Nemo Kepler Pendant, Black': [125, 'corridor'], 'Aqua Vive stortdoucheset Kila met kraan': : [123, 'main bathroom'], "Uttermost Marlow Chandelier': [118. "bathroom of bedroom 1'], "Eisa Pendant': [1 14," "main bedroom'), 'CAR SUV": [12, ]garage'], 'Nolan Upholstered King Bed': [109, "main bedroom"). ...)
We show visualizations of the five scenes used in COOPERA in Fig g

---

Figure 9: HSSD Scenes used in COOPERA.
C Additional Details of Human Simulation
C.1 Psychometric Data Computation
Since SPC dataset lacks psychometric data, we derive Big-5 OCEAN scores by prompting the LLM I) directly infer the scores [48] and 2) complete the Big-5 personality [8 45 and compute scores based on the formula. We then take majority vote across five inference trials, using bins of 0.5 on a scale of 1-5.
C.2
3D Motion
For human simulation, free-form motion data formatted in SMPL-XX enabling detailed control over whole-body motions, including facial expressions and finger articulation. To integrate this data into the human simulation pipeline of Habitat 3.0 [52] we remap the format of Motion-X [3]] data, as illustrated below.
global root orientation: SMPL-X[:. :3]
body: SMPL-X[: 3:3+63]
finger articulation: S.MPL-X[: 66:66+90]
yaw pose: SMPL-X[:. 66+90:66++3]
face expression: SMPL--X[ 159:159+50]
global body position: SMPLL-X[: 209:209+100]
global body position: SMPL-X[:- 309:309+3]
body shape: SMPL-X[;, 312:]
C.3 Feedback
Please refer to Section for details on our designed feedback mechanism for robot's predicted assistive tasks. We emphasize that while structure the feedback binary answers paired with reasoning, can be easily adapted for other learning algorithm by modifying the prompt.
C.4 Additional Qualitative Examples
rumner, and Broadwayy enchusias I'm
radickog at collupe I'm consoraae about loam and suptioenng the world of heartecre, caro. Purk music enather passione paay anoand and movel.
pussion for collectig intresiing things and my roach love decovering hidden gorss and traauu is most
aboue in embork on eoc tio chapter in my it, moving to Fordand
Cregon pursuss
diverrs.
smeopeched pleae. whhnn f'm watching Gaanee
Ein
and
ispenness 4.5. consoiensowann endoucness 4.1. edrovesioon 3.7, agresstlersoc 4.2.
Dopenness: 4T. corstertioutten hess? 34, vedrinasizni: 4.2, 'agivsddiiinaa 244. neurose sm 3.8)
Teperneso: 3.3, lonsaie mosseeettoo.. sress. 2.7 tetroees zen XX0, beteestiareec" 87. neurotoomi 2.5)
reurcticm: 2.0)
Figure 10: Additional examples of simulated human profiles psychometric data.
present additional examples of simulated human profiles along with their corresponding psycho- metric data in Fig. [0
25

---

D Additional Details of Building and Evaluating an Assistive Agent
D.1 Navigation and Movement
To navigate the scene, the agent uses classical motion planning for path calculation, with a stopping threshold determined by the object's Axis- Axis-Aligned Bounding Box (AABB). For the Fetch robot's arm movements, inverse kinematics (IK) is applied to calculate the end effector (EE) position based on the 3D location of the target object. Robot actions are implemented using primitives from Habitat's task_acti registry, combining navigation and EE control to form a full sock-and-place pipeline: 1) MoveEEAction moves the EE via Cartesian steps with IK; 2) PickObj PickObjIdAction IdAction grasps objects using& snap mechanism within distance threshold; 3) PlaceObjIdActe desnap; and 4) ResetEEAction resets the EE to default pose.
releases objects via
D.2
Training
Task videos from Habitat 3.0 [22] are resized to 1024 x 768 and input to Llama-3.2 1BB 1B (our
robot-VLM) using Llama's 's default settings. Classifiers are finetuned on -inrrll----ntttuctt-...... 27 using LoRA (22) (rank 8, dropout 0.2, alpha 16; targets: q. k, v, o) in an instructional format to output binary yes/no. We train for 5 epochs using AdamW (37] (Ir 1e-5, weight decay 0.01), with batch size 1 and gradient accumulation of 4 steps, across 3 NVIDIA A10 GPUs (24GB RAM).
For training the robot's intention and task binary classifiers via instructional finetuning of an L.M,, we structure the input data the following format:
### Instruction:
Considering the human's profile, traits, tenporal dependence on past behaviore, and the current time, deternine if it is likely unlikely that this bunan will: Reepond with 'Yee' 'No",
### Input:
Human Profile.
Big Traita.
Previous Relevant Intentions.
Previous Relevant
Current Time.
### Response:
D.3 Predicate Construction
Collaboration Type
Time: 6 pm
Task 1: Place the fotuuon on the "NITCHEV AD ARTIANN MIEER
Oloood kitchen clear cakg
Internion: Erjcy playful and magraative dinner precar alion kishien oxper monting roc
RED for setup
Task Place the "mustard bonte" on the "Edelwebo desk, astrlenie nite" in the kong room.
ON/ccod tob
now
'Edehve desk, astilehite" in living
ONjjitchen bomms table)
rcom.
Collaboration Type
Time: 9
Task 1: Start with a wam--u perfarming yoga streches near the MMrrraa Skppower Sote" to prepare the bott for more interne suendses
NEEDiyog mat)
Intertions Ungage
morning workout routine in the livig room
Task 2: Perform squrts with dumbiells is neer the Werdowr Art a Morning
N:tDjjumbbellk
Loke View i" for strengh traniin.
Task 3: Transition carcio by daina jumping jadiss nowr the "Wandoww Art
NEHDjjump ropa)
the Path' elevase heart rate
Task Ingage in core exeroees by co ng planks near the "APPLE MMc 5K
NCtDieserriee matl
27" strengthen aedominal musies
down a sterthing routine near "Marina Sipcover musies and prevent nurr..
NEEDOowwD
Figure 11:
Examples of predicate construction.
For collaboration type 1, we map objects to Habitat 3.0 and the YCB dataset's predefined semantic categories, allowing evaluation via exact match. For collaboration type 2, where the robot offers objects from magic box. compute semantic similarity between the robot's predicted tasks/objects for assistance and the human's 's desired objects, using a threshold of 0.6. Fig. iilluttratee examples of predicate construction.
26

---

E Additional Experiments and Details
E.1 Results Breakdown and Analysis
In Fig. 12 we show detailed predicate-based and LLM-based evaluations in both collaboration types across all collaboration settings. Overall, LLM-based evaluations yield higher scores than predicate-based ones, particularly in settings & 4, which are more challenging. This suggests that predicate-basse -ased evaluations are more rigorous, requiring exact matches for collaboration type I and high semantic similarity for collaboration type 2. We highlight several key findings and analyze the performance of each baselines in detail below.
Temporal Fluctuation Patterns Across Days. In Fig. 12 we observe a consistent temporal pattern across days: performance tends to dip around midday and recover toward the end of the day, and improve further on the following day. We attribute this to two primary factors. First, it reflects how human routines are structured, both in reality and in our simulation. Humans generally engage in more predictable activities in the morning and evening (e.g., hygiene, eating, relaxing), which are easier for the robot to infer and assist with. In contrast, midday behavior tends to be more diverse and more strongly influenced by individual traits. Since our human simulation pipeline samples intentions and tasks based on both traits and time, this increases the diversity of midday actions, making inference and aligmment more difficult for the robot and resulting in a temporary performance drop. Performance recovers as routine behaviors re-emerge in the evening the following morning. Importantly, despite these fluctuations, overall performance improves across days. Second, the extent of this fluctuation increases with task difficulty, as defined in our framework (Section [31]]. In Setting (same human, same scene), the robot benefits from consistent exposure to both the user and environment, resulting in relatively small midday dips. In Setting 2 (same human, different scenes), unfamiliar object layouts introduce grounding challenges that increase midday variability. In Setting 3 (different humans, same scene), rotating between users interrupts personalization, making intention interpretation more difficult-paticulkaee Finally, Setting (different humans, different scenes) combines both challenges, resulting in the largest fluctuations as the robot must generalize across both human and spatial contexts.
around midday when behavior is less routine.
Performance Analysis: Main Method vs. Baselines. We evaluate method against six baselines: 1) Direct Prompting. 2) Direct Fine-tuning, 3) Oracle, 4) Random, 5) Intention Agnostic, and 6) Human & Context Agnostic. Fig. 12 shows that our method achieves both the highest adaptation trend over time and the highest final success rate (second only to Oracle) by explicitly modeling the correlation between human intentions/tasks, traits, and temporal dependencies. As the day progresses, the robot accumulates interaction history and adapts its behavior to the human preferences. 1) Direct Prompting shows minimal improvement as it depends on retrieved interaction history without learning the human preferences. 2) Direct Finetuning performs slightly better but still struggles due to its rigid mapping from inputs to intentions/tasks, making it biased toward frequent training examples and overlooking the varying distribution of human behavior. 3) Oracle serves as an upper-bound performance reference by receiving ground-truth human intentions. While this provides a significant advantage, its performance is static by design. This is because Oracle bypasses the core challenge of accurately mapping visual observations of humans performing detailed tasks to high-leve intentions, fundamental problem in human-robot collaboration, even in closed-set scenarios. Moreover, since the correct intention is provided upfront, Oracle has no need to learn temporal dependencies between human intentions (e.g., an intense workout at 11 am typically leads to lunch preparation at 12 pm). 4) Random degrades over time due to the absence of learning or validation. 5) Intention Agnostic shows moderate improvement but lags behind our method, it skips intention inference and thus loses contextual information, which is problematic since temporal dependencies between intentions are typically stronger than between tasks (validated in Section 4.2 6) Human & Context Agnostic captures only superficial time-tase task correlations and ignores human traits or temporal context, leading to weak day gains.
E.2 Human Verification Breakdown and Analysis
For human verification, we present additional results across three baselines and our main method, analyzing how human evaluations align with predicate-based (Table. 7 and M--M-based (Table. g assessments in a detailed breakdown. Due to cost constraints, we use subset of data (one episode per setting), resulting in 32 episodes across the four methods. We recruit eight human evaluators, each
27

---

Table 7: Breakdown of correlation between predicates and human evaluations. We report the L1 4.
Collaboration
Collaboration 2
Setting
Setting
Setting
Setting
Setting
Setting
Setting
Setting
Rundom
0.070
0.134
0.067
0.065
0.073
0.079
0.088
0.112
Oracle
0.072
0.098
0.084
0.065
0.086
0.102
0.104
0.108
Human & Context Agnostic
0.062
0.121
0.066
0.071
0.073
0.109
0.080
0.100
Main
0.092
0.096
0.094
0.087
0.090
0.085
0.076
0.153
Average
0.074
0.112
0.078
0.072
0.080
0.094
0.087
0.118
Table Breakdown of correlation between LLM and human evaluations. report
4.
Collaboration
Collaboration
Setting
Setting
Setting 3
Setting
Setting
Setting
Setting
Setting
Random
0.031
0.054
0.066
0.044
0.068
0.091
0.110
Oracle
0.103
0.096
0.087
0.043
0.071
0.069
0.091
0.094
Hurnun & Context Agnostic
0.045
0.058
0,076
0.071
0.084
0.089
0,079
0.108
Main
0.063
0.075
0.081
0.073
0.090
0.085
0.072
0.076
Averuge
0.061
0.071
0.078
0.058
0.078
0.084
0.082
0.097
assessing four randomly assigned episodes. The results show that the gap between predicate- evaluation and human verification larger than that between LLM-based evaluation and human verification. This is because LLMs/VLMs reason and make judgments for assistance in a way that aligns more closely with human preferences and peciisinnmmainng whereas predicate-based evaluation relies on strict matching criteria that may not fully capture nuanced human reasoning.
E.3 Qualitative Examples of Offline Real Human
We present qualitative examples of offline real humans in Fig. 13 Compared simulated humans (Fig. 5), real humans exhibit greater behavioral dynamics and emergent decisions due to temporary factors (e.g., plans, mood, weather).
E,4 Analysis of Human-in-the-LLopp Collaboration
Intuitively, one might expect lower performance when collaborating with real humans due to their inherent variability and spontaneity. However, results in Table innicatee that real human collaborators do not make the task more difficult and can, in some cases, lead to higher success rates compared to identify likely explanation.
offline simulated humans. Through discussions with participants, Unlike the offline setting where participants record their routines throughout an entire day (12 one- hour intervals covering the time from am pm), the HITL experiments take place in short, controlled session. As result, participants do not act spontaneously hour-by-hour but instead recall and follow their typical weekly routines from memory, which tend be more consistent. This reduced behavioral variability allows the robot to personalize more quickly. and further validates the behavioral diversity modeled in our simulated humans.
28

---

Setting 1
Setting
Setting 3
Setting
(a) Collaboration Type
(b) Collaboration Type
Prompting
Finetuning
Ours
AGHuman
AGincent
Oracle
Random
Figure 12: Breakdown results of HRC. We compare main method with the baselines days.
Traits Characters
Intentions (Day
Intentions (Day 3)
Wate not antireode
Summary
Ganee ensare
Studeet
comsute uammoc The paeae ciout
Nlw steronsnooonn someeee to in doorgmnoo
pm made sare
samthine
Afer waking
30-mii then
me the dhence so heay opporn words has become ne oie of fevoite
to my descre
descaree seramned
Contrnust ad studing sork treak
Exrred proprtin for cxning cutiing ingrsfiene
restto mudeeng
IBG-5 Spore
Pregured
ahrewark
pelk batterseng
play
80m
compone parse
Figure 13: Qualitative examples of full-day intentions recorded by real human with specific human traits and psychometric data. Red boxes highlight emergent behaviors due temporary factors (e.g.. plans, mood, weather).
29

---

Prompt Details of the Simulated Human
We show exact prompts for LLMs in simulated human behavior.
Human Profile Summary and Extension.
Input:
1. Human 1 profile.
2. Human profile.
3. Conversation between Human and Human 2.
Instruction:
Summarize and reasonably expand Human I's profile into a first-person ceff-introduction based on their initial profile and past conversations with other humans. Provide a detailed description covering, if presented, the person's job, hobbies, daily activities, food preferences, social life, physical activity, entertainment preferences, travel habits, personal values, goals and aspirations, stress and coping mechanisms, technological use, cultural interests, health and wellness, community involvement, education, financial habits, and personal style.
Human Psychometric Data Computation (LLM Inference).
Input:
1. Human profile.
Instruction:
Infer Big Five personality traits (scale 1-5, float) based on the provided human profile. Write in the following format: ("openness': a, 'conscientiousness:: b, "extroversion': c, 'agreeablenes'':
d, 'neuroticism': e}
Human Psychometric Data Computation (Big-5 Personality Test).
Input:
1. Human profile.
Instruction:
Based on the provided human profile, complete the Big-5 personality test. In the questions below, for each statement 1-50 mark how much you agree with the scale 1-5, where I=disagree, 2=slightly disagree, 3=neutral, 4=slightly agree and 5=agree.
Questions:
1. Am the life of the party.
2. Feel little concern for others.
......
50. Am full of ideas.
Human Intention Proposal.
Input:
1. Current time.
2. A list of rooms in the house (ignore small spaces like closets).
3. Your Big Five scores (scale 1-5) and human profile.
4. Most relevant human intentions proposed at previous times (ignore if emmpy-thti means it's
the first intention of the day).
5. Most relevant human tasks proposed at previous times.ids (ignore if empty- -tis means it's
the first task of the day).
You are a human living in the house. Propose your intention at the current time.
Instructions:
1. Intention must align with your Big scores, reflect all aspects the profile, and be diverse
yet reasonable based on the house layout and available objects.
30

---

2. Intention must be high-level and either human- (e.g., hygiene, sport, leisure) centric (e.g., clean, organize, set-up). Do not mention specific objects.
3. Intention must have temporal dependence but be non-repetitive with the previous intentions and tasks.
4. Intention should be within the house.
5. All objects are rigid and cannot deform, disassemble, or transform.
Write in the following format. Do not output anything else:
Time: XXx am/pm (c.g., 9 am)
Intention: basic descriptions.
Reason masson_human. detailed descriptions of why it follows your Big 5 scores and profile.
Reason intentions: detailed descriptions of why has temporal dependence with the previous, relevant intentions [list of time].
Reason Reason_tasks: detailed descriptions of why it has temporal dependence with the previous, relevant tasks at [list of time.id].
Human Task Proposal.
Input:
1. The proposed intention at current time.
2. dict mapping rigid, static objects their IDs and rooms.
3. Your Big Five scores (scale 1-5).
4. Most relevant human intentions proposed at previous times (ignore if empty-this means it's the first intention of the day).
5. Most relevant human tasks proposed at previous times.ids (ignore empty- means it's the first task of the day).
You are a human living in the house.
Instructions:
1. Break down the intention into 5 tasks for collaboration with robot.
2. Task types:
Type 1: Creative, reasonable free-form human motion interacting approaching a fixed, static object (static objects cannot be moved) with an object in hand provided by the robot (e.g.. sit on sofa with TV remote control in hand, wipe table with tissue in hand, squat with dumbbell in hand near rug).
3. For interacting with fixed, static objects, only objects from the given static object dict. For objects in hand, a robot will provide them.
4. Both interacting and inhand objects must be specified (cannot be none).
5. Tasks should be continuous and logical, and align with your Big scores and profile.
6. Tasks must have temporal dependence with the intentions and tasks at previous times.
7. Free-form motion should be diverse. Examples: sampled motion list. Feel free to propose others.
8. All objects are rigid and cannot deform, disassemble, or transform.
Write in the following format. Do not output anything else:
Time:
xxx am/pm
Intention:
basic descriptions.
Tasks:
I. Thought: detailed descriptions of the task. Reason human: why it aligns with your Big 5 scores and profile. Reason deason_intentions how it depends previous, relevant intentions at [list of time]. Reason_ tasks: how it depends previous, relevant tasks at [list of time.id]. Act: [type: anter_obj_iie id: real int, inter_obl _bj___oj_____r_e name: xxx, inhand _obj name: yyy, motion: free-form motion] ....
Jeason_human:
Human Reflection (Profile).
Input:
1. The proposed intention at current time.
2. A dict mapping rigid, static objects their IDs and rooms.
31

---

3. Your Big Five scores and human profile.
4. Most relevant human intentions proposed at previous times (if empty, ignore it-this means
it's the first intention of the day).
5. Most relevant human tasks proposed at previous times.ids (if empty, ignore t--tis means it's the first intention of the day).
Your task is to check if the temporal dependence and human profile are strictly followed in each task, and revise make better if necessary.
Instructions:
1. Tasks should be continuous and logical, and align with your Big 5 scores and profile.
2. Tasks must have temporal dependence with the previous intentions and tasks, with detailed explanation mentioning previous intentions and tasks explicitly.
3. For interacting with fixed, static objects, only objects from the given static object dict. For objects in hand, a robot will provide them.
Write in the following format. not output anything else:
Time: XXX am/pm
Intention: basic descriptions.
Reflect Each Task:
1. no mistake or change made.
2.
Revised Tasks:
1. Thought: detailed descriptions of the task. Reason_ human: why it aligns with your Big 5 scores and profile. Reason intentions: of time]. Reason tasks: how it depends previous, relevant tasks [list of time.id]. Act: [type: 1, inter_obj_id: real int, inter_obj_nant name: xxx, inhand_obj _ame yyy, motion: free-form motion]
reason_intentions: how it depends on previous, relevant intentions at [list
2.
Human Reflection (3D Info).
Input:
1. The proposed intention at current time.
2. dict mapping rigid, static objects their IDs and rooms.
Your task is check if the instructions strictly followed in each task, and revise to make better if necessary.
Instructions:
1. Break down the intention into 5 tasks for collaboration with robot.
2. Task types:
TTppe 1: Creative, reasonable free-form human motion interacting or approaching a fixed, static
object (static objects cannot be moved) with an object in hand provided by the robot (c.g., sit on sofa with TV remote control in hand, wipe table with tissue in hand, squat with dumbbell in hand near rug).
3. For interacting with fixed, static objects, only objects from the given static object dict (exact name). For objects hand, a robot will provide them.
4 Both interacting and inhand objects must be specified. Importantly, they cannot be none.
5. Free-form motion should be diverse. Examples: compled_motion list. Feel free to propose others.
6. All objects are rigid and cannot deform, disassemble, or transform.
Write in the following format. Do not output anything else:
Time: XxX am/pm
Intention: basic descriptions.
Reflect Each Task:
I. no mistake or change made.
2.
Revised Tasks:
32

---

1. Thought: detailed descriptions of the task. Reason_human: why it aligns with your Big 5 scores and profile. Reason intenntion:nn how it depends on previous, relevant intentions at [list of time]. Reason_task how it depends previous, relevant tasks at [list of time.id]. Act: [type: 1, knter_ooj__tt real int, inter_ _bj__ob_____ name: xxx, inhand obj name: yyy, motion: free-form motion]
2.
Feedback.
Input:
1. Human intention current time.
2. Current human tasks.
3. Robot-inferred tasks enhance comfort with offered objects (if any).
You the human. Decide if the robot's assistance align with your needs.
Instructions:
The robot's task doesn't t need to be exact match but should be relevant in purpose, context, object categories. Use common reasoning to decide if helps meet your needs.
I. Assess if each robot task supports the human tasks and intention.
2. Consider each robot thought and object individually against the human tasks. Approve it if
meets any one of the human tasks; sequence does not matter.
3. Be fair in your judgment-ave avoid being too generous too harsh.
4. Respond with yes/no for each, followed by an explanation. Ensure items are in a list.
Write in the following format. Do output anything clse:
Tasks: [yes, no, ...]
Reasons tasks:
1.
G Prompt Details of the Assistive Agent
show exact prompts for VLMs building the assistive agent.
Intention Discovery.
Input:
1. Sequence of images showing human motion from your and human's perspectives.
2. Current time.
3. Inferred Big Five personality scores (ignore if empty-this means it's your first collaboration
with this human).
4. Inferred human profile (ignore empty- -ths means it's your first collaboration with this human).
5. Most relevant human intentions discovered at previous times (ignore if empty this means it's the first intention of the day).
6. Most relevant human tasks discovered at previous times.ids (ignore if empty-this means it's the first task of the day).
You are a robot assisting human. Identify 5 possible human intentions based on the current time and visual observations.
Instructions:
I. Map the observed human motion to 5 possible high-level intentions at the current time (without mentioning the specific motion).
2. Intention must align with human Big scores and reflect all aspects of the profile, and be
diverse yet reasonable based the house layout and available objects.
3. Intention must be high-level and either human-centric (e.g., hygiene, sport, leisure) or room- centric (e.g., clean, organize, set-up). Do not mention specific objects.
33

---

4. Intention must have temporal dependence but be non-repetitive with the intentions and tasks at previous times in the input.
Write in the following format. Do output anything else: Time: xxx am/pm
Intention I: basic descriptions.
Reason human: detailed descriptions of why follows the Big scores and profile.
Reason intentions: detailed descriptions of why it has temporal dependence with the previous, relevant intentions [list of time].
Reason tasks: detailed descriptions of why it has temporal dependence with the previous, relevant tasks [list of time.id].
Reason_ vis: detailed descriptions with respect to the visual cues.
Task Discovery.
Input:
1. Human intention at current time.
2. A dict mapping rigid, static fumituues to their IDs and rooms.
3. Inferred Big Five personality scores (ignore if empty- this means it's your first collaboration with this human).
4. Most relevant human intentions discovered at previous times (ignore if empty-thir this means it's the first intention of the day).
5. Most relevant human tasks discovered at previous times.ids (ignore if empte- this means it's the first task of the day).
You are robot assisting human.
Instructions:
1. Break down the intention into 5 tasks.
2. Task type: For each human task, provide one small, handable object from a magical box.
Furnitures in the dict for room understanding and cannot be used.
3. Tasks should be continuous and logical, and align with your Big scores and profile.
4. Tasks must have temporal dependence with the intentions and tasks at previous times.
5. All objects are rigid and cannot deform, disassemble, or transform.
Write in the following format. Do not output anything else: Time: xxx am/pm
Intention:
basic descriptions.
Tasks:
1. Thought: detailed descriptions of the task. Reason Lamioon_hman:: why it alignes with your Big 5 scores and profile. Reason drason_intentions how it depends previous, relevant intentions at [list of time]. Reason_tasteeeeeeee how depends on previous, relevant tasks at [list of time.id). Act:
[obj name: xxx]
2.
Traits Inference.
Input:
1. Human intentions at previous times (ignore if empty this means it's your first inference).
2. Human tasks at previous times.ids.
3. Human profile (ignore if empty- this means it's your first inference).
Task: Mimic this human by:
1. Inferring Big Five personality traits (scale 1-5, float) based on the provided intentions and task.
2. Summarizing the human profile (i.e., preferences/habits)) based on the intentions and tasks within three sentences. Revise the existing human profile if necessary.
Write in the following format. Do not output anything else: Scores: f'openness": a, "conscientiousness" "extroversion": c, 'agreeableness': d, 'neuroti- cism': e}
34

---

Profile:
Reasons_ocean: ocean: explain each ocean. Reasons_profiles explain the profile.
35


Reviews:
[{"review_id": "BUf4xwsqw0", "reviewer": "Reviewer_kYPv", "summary": "This paper presents a novel framework for human-robot interaction (HRI) with language models as simulated human agents. The authors performed standard evaluation on both the framework itself and a curated robotic VLM agent. This novel task opens up new challenges and opportunities in embodied research.", "questions": "1. Why is a fixed daily routine selected instead of leaving room for interaction?\n2. What is the performance of the baselines after sufficient days? If the generated data has limited days / episodes, maybe try repeating the overall routine as a cycle.\n3. From the provided Figure 12 in Appendix, I see clear patterns between each day, i.e., increase->decrease within one day and increase even more on the second day. Is it coming from the task setting or the agent paradigm?\n\nI will consider increasing my score if the authors provide additional details to address my concerns.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\nThe idea of simulating human behavior through the profile -> intent -> task -> execution pipeline is quite interesting. To me, it is even worthier of discussion than the overall framework. Nonetheless, the framework provides a novel perspective of HRI and opens up possibilities for large scale data collection and training.\n\nWeaknesses:\n1. The human agent follows a pre-set routine of actions, but normally a human would at least adapt to what the robot is doing, etc. clearing the path for the robot.\n2. The experiments focus on the success rate increase over time, which is kind of a test-time learning setting. I would expect more analysis on the 'final success rate', the success rate after sufficiently days / episodes so that in theory the agent's intention is deducible from its actions.", "was_revised": true, "final_justification": "The authors addressed my concerns properly. The unusual results of human-in-the-loop and increased HRI raises new points worth of discussion which I'm looking to see in the final version should the chair decided to accept this paper. That apart, I still recommend this paper but certain drawbacks including the soundness of the task itself and the quality of the dataset refrains me from giving a score of 5.", "rebuttal": "Thank you for your insightful and constructive comments. We appreciate your recognition of the novelty and contribution of our framework, and your interest in our human simulation pipeline. This recognition is also kindly shared by other reviewers (Zf7z, irvc, PjNb). Following your suggestions, we conduct **additional experiments** to strengthen the analysis. For all HRI-related experiments, we report results from our main method under the more challenging collaboration type 2 (one trial per setting) due to time constraints. Below, we address each of your concerns.\n\n---\n\n**Q1:** Why is a fixed daily routine selected for the simulated human agent instead of enabling interaction to the robot?\n\n**A1:**\nThank you for pointing this out! We believe “fixed vs. dynamic” routines can be interpreted in two ways. The first is if a human with specific traits shows varied daily behavior while following a general routine (e.g., Monday 9am for cleaning, Tuesday 9am for exercise). We achieve this by setting a high LLM temperature (line 171). The second concerns if the human adjusts plans based on external factors such as mood, weather, or the robot’s behavior.\n\nWhile our framework shows robustness to temporal factors by offline real human collaboration (Section 4.4, Table 3), we did not explicitly simulate human adaptation to robot behavior. Enabling reactive human behavior requires a cognitively richer agent and defining clear criteria for when and how adaptation occurs. This introduces additional complexity, turning the setup into a multi-agent framework. Our current goal is to develop an assistive robot that can improve across long-horizon, and we view simulating mutual adaptation as promising future work.\n\nYet, we fully agree it is valuable to explore human plan adaptation in response to robot behavior. We conduct two **experiments** to study this extension.\n\n1) Increased human-robot interaction (Increased HRI): The human’s hourly intention/task is influenced by the robot’s action in the previous hour. For example, if the robot fails to clean the living room, the human may skip yoga to finish cleaning. We modify the human prompt to include robot's success/failure. We also ask the human to provide hourly feedback with reasoning to the robot (e.g., I disapprove this task because...). This helps the robot understand its mistakes and adjust its next action accordingly in the next hour.\n\n2) Human-in-the-Loop (HITL): Three real humans replace the LLM and collaborate under setting 2. Users propose intentions based on their traits and adjust to robot predictions in real time. Users are shown retrieved object/motion sets and choose which to interact with and use. For simplicity and due to time constraints, users input responses as text instead of using keyboard to control the simulated agent.\n\n|Method|D1|D2|D3|D4|D5|\n|-|:-:|:-:|:-:|:-:|:-:|\n|Increased HRI|0.224|0.243|0.323|0.401|0.415|\n|Main|0.236|0.250|0.342|0.406|0.431|\n\n|Method|D1|D2|D3|D4|D5|\n|-|:-:|:-:|:-:|:-:|:-:|\n|HITL|0.279|0.340|0.411|0.393|0.457|\n|Main|0.290|0.324|0.371|0.476|0.440|\n\nThe results are interesting. For Increased HRI, we expect higher or comparable performance due to frequent feedback but observe a slight drop. We find that the robot often repeats the previous intention to “make amends” when the human disapproves, while the simulated human sticks to its traits and only notes the error in feedback. For HITL, we expect lower performance due to real human variability, yet results are higher. After discussing with participants, we find they tend to recall and follow their mid-week routines, which are relatively stable. This consistency leads to lower variability than our simulated humans, allowing the robot to personalize more quickly and validating the behavioral diversity in our simulated humans.\n\n---\n\n**Q2:** What is the performance of the baseline(s) after sufficient days (in contrast to test-time learning setting)? \n\n**A2:**\nGood question! To clarify, our framework includes two layers of test-time learning. The first layer operates within a day by prompting the robot-VLM with retrieved interaction history and the inferred human profile. The second layer operates across days by finetuning the robot-LLM classifier at the end of each day using data labeled from human feedback over previous days. For HRI at day t = n, the robot trains on data from days 0 to n–1, resulting in progressively more personalized behavior.\n\nWe choose this test-time learning setup for two reasons. First, within-day adaptation improves both success rate throughout a day and final success rate, as in Fig. 6a and Table 5. Second, across-day adaptation reflects a realistic deployment scenario: when a user purchases a home assistant or cleaning robot, they should be able to deploy it immediately, and the robot should improve over time through daily interactions. This contrasts with prior HRI setups that rely on collecting all data upfront and training the model only once [1, 2].\n\nYet, we fully agree it is valuable to study the train-all-at-once  setting. We conduct two **additional experiments**. (1) We extend the benchmark by doubling the collaboration duration to observe continued adaptation trends. (2) we remove across-day learning and train the robot once after the final day, then evaluate its performance on all days.\n\n|Method|D1|D2|D3|D4|D5|D6|D7|D8|D9|D10|\n|-|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|Main (test-time)|0.236|0.250|0.342|0.406|0.431|0.425|0.455|0.473|0.478|0.470|\n|Train-All-at-Once |0.509|0.472|0.474|0.458|0.516|0.421|0.487|0.467|0.491|0.446|\n\nFor the test-time setting, performance stabilizes around day 8. While our benchmark uses 5-day collaboration, our main interest lies in the personalization trend, and longer HRC is easily supported. By day 8, the robot has learned most human preferences shaped by traits and routines, forming reliable alignment. Given the 7B model with LoRA finetuning, further gains become marginal without more capable models or finer-grained signals like multimodal cues or reasoning chains.\n\nFor the train-all-at-once case, daily performance is slightly higher or on par with the final days of test-time learning. This is because the robot remains unoptimized across all days, leading the simulated humans to disapprove more tasks. These disapprovals generate more negative samples for training. However, this comes at the cost of lacking any further personalization after the initial collaboration on day 1.\n\n---\n\n**Q3:** In Fig. 12, there is a pattern across days (increase → decrease within a day, higher increase on second day). Is this pattern due to your task setting or the learning paradigm of the agent?\n\n**A3:**\nThis is a great question! From Fig. 12, we observe performances tend to decrease around midday and recover toward the end of the day. We believe this pattern is due to two reasons.\n\nFirst, it reflects how human routines are structured both in reality and in our simulation. Generally, humans tend to engage in more predictable activities in the morning and evening (e.g., hygiene, eating, relaxing). These behaviors are easier for the robot to predict and assist. In contrast, midday behavior tends to be more diverse and strongly aligned with human traits. Since our human simulation pipeline samples intentions/tasks based on both traits and time, this increases the diversity of midday intentions/tasks, making inference and alignment harder for the robot. As a result, performance temporarily drops. The performance recovers as human routines re-stabilize in the evening or next morning. Importantly, despite these fluctuations, performance improves overall across days.\n\nSecond, from Fig. 12, this fluctuation is amplified in more challenging task settings defined in our framework (Section 3.1). In Setting 1 (same human, same scene), the robot benefits from consistent exposure to the same human and environment, so midday dips are relatively small. In Setting 2 (same human, different scenes), unfamiliar object layouts introduce grounding challenges that increase midday variability. In Setting 3 (different humans, same scene), rotating between humans interrupts personalization, making intention interpretation harder, especially around midday when behavior is less routine. Finally, Setting 4 (different humans, different scenes) combines both factors, resulting in the largest fluctuations as the robot must generalize across both human and spatial contexts.\n\nTo further validate this pattern, we conduct two **additional experiments** by replacing the robot-VLM with a stronger model (GPT-4o) and a weaker one (Qwen2.5-VL-3B-Instruct). To quantify temporal fluctuation independently of overall performance trend, we compute the Coefficient of Variation of Differences (CVD), defined as the standard deviation of hour-to-hour differences normalized by the mean performance (in %).\n\n**Success Rate**\n|Method|D1|D2|D3|D4|D5|\n|-|:-:|:-:|:-:|:-:|:-:|\n|Qwen|0.196|0.218|0.295|0.364|0.373|\n|GPT|0.313|0.325|0.424|0.483|0.526|\n|Main|0.236|0.250|0.342|0.406|0.431|\n\n**CVD**\n|Method|D1|D2|D3|D4|D5|\n|-|:-:|:-:|:-:|:-:|:-:|\n|Qwen|8.71|7.70|7.18|7.41|5.23|\n|GPT|6.28|5.91|4.20|4.32|2.35|\n|Main|8.46|7.06|7.49|6.21|3.77|\n\nAs expected, stronger models produce better performance [3]. Notably, (1) the performance gap between GPT and llama is greater than between llama and Qwen. (2) while all VLMs exhibit fluctuation, stronger models like GPT show lower CVD. We attribute this to stronger models not only achieve better visual grounding for intention prediction but also generate more coherent intention-task proposals across time by better understanding prior interaction history.\n\n---\n\n[1] Watch-and-help: A challenge for social perception and human-ai collaboration. ICLR 2021.\n\n[2] Partnr: A benchmark for planning and reasoning in embodied multi-agent tasks. ICLR 2025.\n\n[3] Embodied Agent Interface: Benchmarking LLMs for Embodied Decision Making. NeurIPS 2024."}, {"review_id": "hC1z3O0ALO", "reviewer": "Reviewer_Zf7z", "summary": "The paper introduces COOPERA, a framework designed for continual, open-ended human-robot collaboration. It simulates human behaviors across multiple days using LLMs, driven by psychological traits and long-term intentions. A robot assistant agent leverages a VLM and another LLM to infer human intentions, classify tasks, and offer collaborative actions. Evaluations are conducted within simulated environments, assessing the robot’s ability to personalize assistance by learning human traits and adapting to context-dependent intents.", "questions": "1. In Figure 6b, intent-agnostic and human-agnostic models show comparable learning progress to the full proposed approach. Does this indicate that the system learns general collaboration strategies rather than specific personalized intent inference?\n2. Clarify how feedback was provided to the robot at the end of each day. Was it binary (e.g., correct/incorrect predictions), detailed action-level feedback, or summary-level feedback?\n3. How exactly were actions presented to the robot? Did the robot rely purely on visual observations, or was textual context also provided? Given the complexity of certain tasks described (e.g., brainstorming), clarification on inference challenges from visual data alone is warranted.\n4. Clearly define the criteria for determining success in collaborative tasks. Were these predefined success conditions, user-defined outcomes, or evaluated through additional automated metrics?", "limitations": "While the authors acknowledge the difficulty in validating simulated human behaviors, explicitly addressing the risk of \"echo chamber\" effects due to reliance on LLM-generated data would strengthen the paper's transparency.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\nThe work addresses a clearly motivated and significant problem and it provides thorough documentation of experimental settings, parameters, and model specifics. The analysis conducted evaluates multiple dimensions of the proposed approach, offering insights into the effectiveness and limitations of the system. The integration and application of foundation models in HRC tasks demonstrates thoughtful methodological design, leveraging existing technology to advance the field.\n\nWeaknesses: \nThe evaluation heavily depends on synthetic human simulations, with limited validation against genuine human behaviors, raising questions about real-world applicability. The methods for assessing the realism of simulated humans are restricted primarily to classifier-based distinguishability and basic user studies, lacking more comprehensive measures of realism or plausibility. Additionally, the human studies narrowly focus on personality type identification and task identification, neglecting the broader context of expected or realistic behavior. There is a notable dependence on foundation models throughout the entire pipeline, which raises concerns regarding the genuine methodological novelty. The dense structuring of the content limits the nuanced discussion of results and findings and the paper appears to insufficiently explore or discuss potential system failures, both in terms of human simulation realism and robot assistance effectiveness. \n\nOther comments:\nFigure 1 appears not to be referenced in the text. Including explicit references or discussing it in relation to the framework's overview would enhance clarity.\n\nGiven the complexity and ambitious scope of the proposed framework, the content seems more suitable for a journal article where the authors can extensively discuss each component with greater depth. Alternatively, the paper might benefit from clearly delineating the presentation into separate components: one part thoroughly evaluating the COOPERA framework's capability to simulate human behavior realistically, and another part focused on the robot assistance approach. As currently structured, it is challenging to fully trust the assistance methodology without first establishing comprehensive confidence in the fidelity and realism of the underlying COOPERA simulation framework.\n\nQuality: Technically solid, though reliance on synthetic data without substantial real-world validation limits completeness.\nClarity: Generally clear but overly dense; improved organization or division into separate works would greatly enhance readability.\nSignificance: Addresses an important, challenging HRC problem with potential high impact, but practical significance uncertain due to limited realism validation.\nOriginality: Novel integration of existing foundation models in a comprehensive continual collaboration framework, clearly differentiating from prior methods.", "was_revised": true, "final_justification": "The authors' detailed rebuttal and additional experiments addressed concerns regarding evaluation clarity and robustness across foundation models. While the realism and validity of synthetic human simulations remain partially unresolved, I acknowledge that these limitations are inherently challenging. Structural clarity and scope also remain ambitious and could benefit from further refinement. The framework and assistance methodology could benefit from further restructuring, potentially as separate papers or an expanded journal article. Given the improvements, my recommendation moved from Borderline Reject (3) to Borderline Accept (4).", "rebuttal": "Thank you for your insightful and constructive comments. We appreciate your recognition of our framework’s contribution, careful integration of foundation models, and completeness of our evaluation. This recognition is also kindly shared by other reviewers (kYPv, irvc, PjNb). Following your suggestions, we added **new experiments** to strengthen our analysis. For HRI experiments, we report results under the more challenging collaboration type 2 (one trial per setting) due to time constraints. Below, we address each of your concerns.\n\n---\n\n**Q1:** The evaluation relies heavily on synthetic human simulations, with limited validation against real human behavior.\n\n**A1**: Thank you for pointing this out! COOPERA relies on synthetic human simulations due to ethical, safety, and practical constraints. Our goal is to study long-term, open-ended HRC, which better mirrors real-world use, unlike most HRC benchmarks focused on episodic, closed-set tasks [1–3]. To support long-horizon HRC, we need humans who can continuously collaborate with robots and provide feedback. However, running full real-world studies with physical robots in household over multiple days is costly and raises ethical and safety concerns. As an initial step, we build a scalable simulation pipeline enabling personalized, long-term HRC entirely in simulation (supported by Reviewer kYPv, irvc).\n\nTo assess the realism of our simulated human model, we present extensive analysis in Section 4.2 (also noted by Reviewer irvc), including classifier-based evaluations (distinguishability, diversity, psychometric coherence, temporal dependence) and two user studies. However, we fully agree more validation is necessary.\n\nTo address your concern, we conduct a **new experiment**: six users provide traits and daily intentions over 5 days. Using the same traits, we prompt the LLM to simulate intentions for the same period. Both sets are aggregated into single paragraphs (removing time formatting like “9am:” to avoid inflated similarity), and semantic similarity is computed using SBERT (all-mpnet-base-v2) and OpenAI embeddings (text-embedding-3-small). We compare against three baselines: (1) prompting without human profile (generic), (2) mismatched LLM-human intention pairs, and (3) generating all intentions at once without hour-level grounding (as in Table 4 ablation).\n\nMethod|SBERT|OpenAI Embed\n-|:-:|:-:\nGeneric|0.554|0.537\nMisalign|0.523|0.543\nAll-Day Intent|0.769|0.746\nMain|0.810|0.772\n\nFrom the results, Generic and Misaligned yield moderate similarity ($\\sim$0.5), since sentence encoders naturally assign partial similarity to structurally similar content. Our main method achieves notably higher scores, indicating closer alignment with human-specific intent. Sentence-BERT yields slightly higher similarity than OpenAI embeddings, likely due to its objective being explicitly tuned for sentence-level semantic comparisons.\n\n---\n\n**Q2:** Notable dependence on foundation models throughout the pipeline.\n\n**A2:** Great point! Both our human simulation pipeline and benchmark methods rely on foundation models. As COOPERA extends closed-set tasks [1, 3] to open-ended ones, foundation models are a natural choice for agent reasoning and decision-making [3], a trend also seen in open-ended embodied AI systems [4].\n\nYet, we fully agree on concerns with heavy reliance on LLMs, especially regarding pipeline brittleness/robustness and prompt over-engineering.\n\nTo address the concerns, we run three **new experiments**: 1) We replace the robot-VLM with a stronger model (GPT-4o) and 2) a weaker one (Qwen2.5-VL-3B-Instruct). 3) Test prompt sensitivity by swapping our handwritten prompts with DSPy [5], an open-source framework for prompt automation using only high-level instructions and I/O examples.\n\nMethod|Day 1|Day 2|Day 3|Day 4|Day 5\n-|:-:|:-:|:-:|:-:|:-:\nQwen|0.196|0.218|0.295|0.364|0.373\nGPT|0.313|0.325|0.424|0.483|0.526\nDSPy|0.222|0.262|0.355|0.387|0.439\nMain|0.236|0.250|0.342|0.406|0.431\n\nAs expected, stronger models yield better performance [3]. Notably, smaller VLMs like Qwen still achieve reasonable performance by day 5 and show a clear personalization trend, demonstrating the robustness of our framework across model scales. We also observe a 3% gain with DSPy, compared to its reported result (5%) in expert domains. This suggests our handwritten prompts are well-structured but not over-engineered.\n\n---\n\n**Q3:** The use of synthetic data without substantial real-world validation raises concerns about completeness and the risk of “echo chamber” effects. Real-world applicability remains unclear.\n\n**A3:** Thank you for bringing this up! The ultimate goal of COOPERA is to develop robot agents that assist real humans by adapting to their preferences over time. To validate real-world applicability (Section 4.4), we conducted human verification and collaborated with offline real humans. While our experiments are in simulation, the robot’s embodiment and actions match the real Fetch robot’s default settings in Habitat [7], enabling seamless transfer of generated action sequences to the real robot.\n\nHowever, we fully agree more validation of real-world applicability is necessary. To address this, we include a **new experiment**. Three real humans replace the LLM and collaborate under setting 2. Users propose intentions based on their traits and adjust to robot predictions in real time. Users are shown retrieved object/motion sets and choose which to interact with and use. For simplicity and due to time constraints, users input responses as text instead of using keyboard to control the simulated agent.\n\nMethod|Day 1|Day 2|Day 3|Day 4|Day 5\n-|:-:|:-:|:-:|:-:|:-:\nHITL|0.279|0.340|0.411|0.393|0.457\nMain|0.290|0.324|0.371|0.476|0.440\n\nThe results are interesting. We expect lower performance due to real human variability, yet results are higher. After discussing with participants, we find they tend to recall and follow their mid-week routines, which are relatively stable. This consistency leads to lower variability than our simulated humans, allowing the robot to personalize more quickly and validating the behavioral diversity in our simulated humans.\n\n---\n\n**Q4 & 5:** Clarification on how success is defined and how feedback is provided to the robot.\n\n**A4 & 5:** Thank you for pointing this out! Evaluation clarity is crucial and we will revise Section 4.3. We compute F1-based success using three methods, consistent with prior HRC benchmarks [1–3]. The robot’s LLM classifier outputs a yes/no-labeled task list, and F1 is computed using the following GT labels:\n\n1) Predicate-based: Tasks are executed and evaluated by predicate functions (Appendix D.3), with class-based object matching following Watch-and-Help [1].\n\n2) LLM-based: Simulated human judges robot tasks and gives binary feedback on if each task fullfills its intention (Appendix C.3 and F).\n\n3) Human verification: Same as (2) but real humans provide the binary feedback.\n\nFor more details, please see our reply to Reviewer PjNb Q4 due to word limit.\n\n---\n\n**Q6:** In Fig. 6b, intent-agnostic and human-agnostic baselines show comparable learning progress to full proposed approach.\n\n**A6:**\nGreat question. Among six baselines (Section 4.3), Intention Agnostic is closest to our full method. It removes intention inference so tasks are predicted without modeling intention. Human & Context Agnostic learns only time-to-task mappings, ignoring traits and temporal cues.\n\nIn Fig. 6b, Intention Agnostic and Human & Context Agnostic reach $\\sim$0.4 and $\\sim$0.35 F1, while our full method achieves $\\sim$0.5. Though the gap seems small due to averaging across all settings, Appendix Fig. 12 reveals: (1) Our method significantly outperforms both in harder settings (3 & 4, different humans), highlighting the value of intention modeling and personalization. (2) In easier settings (1 & 2, same human), both baselines perform similarly since trait learning matters less. (3) In multi-human settings (3 & 4), Intention Agnostic surpasses Human & Context Agnostic, showing while intention modeling helps, learning human-specific behavior is crucial for generalization.\n\n---\n\n**Q7:** How are robot actions defined, and what input modalities support complex task inference?\n\n**A7:**\nRobot actions are implemented as primitives from Habitat’s task\\_action registry, combining navigation EE control: (1) MoveEEAction moves the EE via Cartesian steps with IK; (2) PickObjIdAction grasps objects by snap within a threshold; (3) PlaceObjIdAction releases via desnap; (4) ResetEEAction resets the EE to default. Navigation uses classical planning with AABB thresholds. These form a full pick-and-place pipeline. We will release all code.\n\nFor intention discovery, the robot-VLM receives five 1024×768 RGB frames of the human’s first task in collab type 2. In type 1, we also provide texts (Section 3.1). Intention-to-task decomposition is text-based. For complex intentions (e.g., “meditate”), we let VLM reason over visual cues, prior actions, and traits. For complex tasks (e.g., “brainstorm”), we use the VLM’s proposed task superset, filtered by our LLM-based classifier, to capture valid actions.\n\n---\n\n**Q8:** Paper writing. The structure limits the nuanced discussion of results and findings. Fig. 1 not referenced.\n\n**A8:** Thanks for pointing out! For revision, we will: (1) move selected implementations from Sections 3.2 and 4.2 to Appendix; (2) expand discussion of findings, including new results addressing Q1–Q3 and Q6; and (3) clarify how success rate and feedback are defined, either in the main text or via Appendix references. We will also explicitly describe Fig. 1 in intro.\n\n---\n[1] Watch-and-help. https://arxiv.org/abs/2010.09890\n\n[2] Partnr. https://arxiv.org/abs/2411.00081\n\n[3] NOPA. https://arxiv.org/abs/2301.05223\n\n[4] Embodied Agent Interface. https://arxiv.org/abs/2410.07166\n\n[5] DSPy. https://arxiv.org/abs/2310.03714\n\n[6] Habitat 2.0. https://arxiv.org/abs/2106.14405"}, {"review_id": "PH4hiG22ON", "reviewer": "Reviewer_irvc", "summary": "This paper introduces COOPERA, a novel framework for continual, open-ended human-robot collaboration in simulated household environments. The core innovation lies in simulating psychologically grounded human agents using LLMs that generate temporally coherent, traits-driven behaviors across multiple days. The robot assistant learns from these interactions to infer human preferences and adapt its actions accordingly. The authors propose a benchmark, a method for human trait and task inference, and conduct thorough experiments on within-day and across-day adaptation, scene and human generalization, as well as human studies validating the realism of their simulations.", "questions": "1 All experiments use relatively small or mid-sized models (e.g., LLaMA-3-11B, Mistral-7B). Is there a particular computational or latency constraint driving this choice? Do the authors expect significantly better assistance behavior from larger models, or do smaller models already saturate performance given the task complexity?\n\n2 How sensitive is the pipeline to the choice of LLM/VLM backbones? The paper includes one ablation replacing the robot’s VLM, but further experiments could highlight how brittle (or robust) the system is to different base models or prompting strategies.\n\n3 While the study focuses on offline interactions and day-level adaptation, do the authors envision a version of COOPERA where real-time, in-episode adaptation (e.g., using continual feedback during a task) can be integrated? If so, what components would need to be modified?\n\n4 Given the richness of the simulated human profiles and long-term interaction, are there any qualitative or longitudinal metrics that could better reflect improvement in human-robot support or perceived usefulness of the robot?", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\nThe proposed setting fills a gap in HRC research by addressing long-horizon personalization and temporal reasoning. Before LLMs are widely studied, the cost of obtaining human data was high and it is useful now to have LLM included to simulate the data.\n\nThe paper provides diverse experiments, including simulation, user studies, and offline real-human tasks. The use of predicate-based, LLM-based, and human evaluations strengthens the validity of the findings.\n\nThe simulation of long-term human behavior driven by LLMs with embedded personality traits is original and adds realism to the evaluation framework.\n\n\nWeaknesses:\n\nWhile the study uses a reasonably sized model (e.g., Llama-3.2-11B), it remains unclear whether scaling up LLMs/VLMs (e.g., GPT-4, Gemini, Claude) would significantly enhance robot performance of the proposed methods or the baselines. The authors mention computational constraints but do not explore the trade-offs or potential gains. More experiments could validate if the performance gap between the proposed methods and baselines would become bigger or smaller with more advanced LLMs.\n\nThe method uses specific LLM prompts and retrieval heuristics; a deeper analysis of robustness across LLM backbones or prompt configurations would strengthen generalizability claims.", "was_revised": true, "final_justification": "The issues are resolved and I updated my scores to accept.", "rebuttal": "Thank you for your insightful and constructive comments. We appreciate your recognition of the originality and the contribution of our framework, the novelty of our human simulation pipeline, and the comprehensiveness of our experimental design. This recognition is also kindly shared by other reviewers (Reviewer kYPv, Reviewer Zf7z, Reviewer PjNb). In response to your suggestions, we conduct **additional experiments** to strengthen our analysis. For all HRI-related experiments, we report results from our main method under the more challenging collaboration type 2 (one trial per setting) due to time constraints. Below, we address each of your concerns.\n\n---\n\n**Q1:** How sensitive is the COOPERA pipeline to the choice of LLM/VLM backbone and how would models of different sizes impact the performance? What is the reason behind the current choice?\n\n**A1:** \nWe choose LLaMA-3.2-11B as the robot-VLM and Mistral-7B-Instruct-v0.2 as the robot-LLM classifier for two reasons. First, both are open-source under research-friendly licenses, offering strong performance, transparency, and reproducibility for benchmarking. In contrast, proprietary models like GPT require costly API access, are frequently updated, and make results harder to reproduce. Second, 7B–11B models strike a practical balance between performance and computational efficiency [1, 2]. They support single-GPU inference (24GB RAM) and LoRA finetuning with just 2–3 GPUs, making them accessible to academic labs and the broader research community.\n\nYet, we fully agree it is important to evaluate the framework’s robustness to different VLM backbones. We conduct two **additional experiments** by replacing the robot-VLM with a stronger model (GPT-4o) and a weaker one (Qwen2.5-VL-3B-Instruct).\n\n|Method|Day 1|Day 2|Day 3|Day 4|Day 5|\n |-|:-:|:-:|:-:|:-:|:-:|\n|Qwen|0.196|0.218|0.295|0.364|0.373|\n|GPT|0.313|0.325|0.424|0.483|0.526|\n |Main|0.236|0.250|0.342|0.406|0.431|\n\nAs expected, stronger models produce better performance, aligning with findings in Embodied Agent Interface [3], a benchmark that tests the impact of LLM capacity on LLM-powered embodied AI systems. Notably, the performance gap between GPT-4o and LLaMA is larger than that between LLaMA and Qwen. However, smaller VLMs like Qwen still achieve reasonable performance at the end of day 5 and show a clear personalization trend. This demonstrates the robustness of our framework across varying model capacities.\n\n---\n\n**Q2:** The method uses specific LLM prompts and parameters. Deeper analysis is needed.\n\n**A2:** \nThank you for pointing this out! We agree that prompt design and parameter tuning are important considerations for any LLM-based system. To maintain full control and interpretability of our framework, we handwrote our prompts. However, we also acknowledge there are open-source tools that automate prompt optimization. To ensure our performance is not the result of over-engineered prompts, we conduct an **additional experiment** by integrating DSPy [4, 5], an open-source framework for self-composing and self-improving prompts, into our assistive robot agent. DSPy allows prompt tuning using only high-level instructions and input-output examples. We evaluate our system with DSPy-generated prompts.\n\n|Method|Day 1|Day 2|Day 3|Day 4|Day 5|\n|-|:-:|:-:|:-:|:-:|:-:|\n|DSPy|0.222|0.262|0.355|0.387|0.439|\n|Main|0.236|0.250|0.342|0.406|0.431|\n\nWe observe a slight performance gain using DSPy. DSPy leverages the OPRO algorithm [8], and its evaluation reports an average gain of 5% in expert domains. In our case, we see a 3% improvement, which we find reasonable. It suggests that our hand-written prompts are already well-designed, though not overly engineered.\n\n---\n\n**Q3:** Can COOPERA be extended to support real-time, in-episode adaptation (e.g., continual feedback during a task)? What components of the current system would need to be modified to support this?\n\n**A3:** \nThis is a great question! COOPERA currently focuses on day-level adaptation, which we believe better reflects real-world usage—robot is optimized overnight while humans rest. However, enabling real-time adaptation through hour-level human/between-task feedback and developing corresponding online learning algorithms is a promising direction and a fundamental challenge for any LLM-based agent.\n\nTo conduct a preliminary study for this extension, we conduct one **additional experiment**.\n\nCOOPERA with hour-based communication: The human’s hourly intention/task is influenced by the robot’s action in the previous hour. For example, if the robot fails to clean the living room, the human may skip yoga to finish cleaning. We modify the human prompt to include robot's success/failure. We also ask the human to provide hourly feedback with reasoning to the robot (e.g., I disapprove this task because...). This helps the robot understand its mistakes and adjust its next action accordingly in the next hour.\n\n|Method|Day 1|Day 2|Day 3|Day 4|Day 5|\n|-|:-:|:-:|:-:|:-:|:-:|\n|Hour-Based|0.224|0.243|0.323|0.401|0.415|\n|Main|0.236|0.250|0.342|0.406|0.431|\n\nThe results are interesting. We expect higher or comparable performance due to frequent feedback but observe a slight drop. We find that the robot often repeats the previous intention to “make amends” when the human disapproves, while the simulated human sticks to its traits and only notes the error in feedback. Extending COOPERA to support more frequent feedback and developing new assistive strategies are promising directions for future work.\n\n---\n\n**Q4:** Given the long-term nature of the simulation and human profiles, are there qualitative or longitudinal metrics that could better capture: 1) The robot’s improvement over time? 2) The perceived usefulness or support quality from the human’s perspective?\n\n**A4:** \nThank you for the suggestion! Our current evaluation focuses on quantitative performance (predicate-based, LLM-based, and human-verified success rates), following conventions in existing HRC benchmarks [6, 7]. We fully agree that qualitative and longitudinal metrics can better capture perceived usefulness and long-term improvement.\n\nWe propose two directions for future work:\n\n1) Subset-Based Evaluation for Intention Alignment Over Time: At each hour, the robot infers the human intention and proposes a task set. Our current LLM-based evaluation only gives binary approval/disapproval to each task in the set. We can extend this by selecting subsets of the robot’s proposed tasks and asking the simulated human whether each subset fulfills their intention. Ideally, some subsets will be approved, others rejected. By recursively forming and evaluating the subsets, we can determine which combinations of tasks are most critical for intention fulfillment. We can then assign a final score to the original full set by summing the scores of approved subsets (+1 each) and disapproved subsets (-1 each).\n\n2) Human Preference Between Successive Robot Policies: For HRC at day t = n, we can compare a robot finetuned on data from t = 0 to n-1 versus t = 0 to n-2, then ask the simulated human to express a preference. This offers insight into how alignment improves across days.\n\n---\n\n**References:**\n\n[1] Grattafiori, Aaron, et al. The Llama 3 Herd of Models. ArXiv, 2024.\n\n[2] Kaplan, Jared, et al. Scaling Laws for Neural Language Models. ArXiv, 2020.\n\n[3] Li, Manling, et al. Embodied Agent Interface: Benchmarking LLMs for Embodied Decision Making. NeurIPS, 2024.\n\n[4] Khattab, Omar, et al. DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines. ArXiv, 2023.\n\n[5]  Xian, Jasper, et al. Prompts as Auto-Optimized Training Hyperparameters: Training Best-in-Class IR Models from Scratch with 10 Gold Labels. ArXiv, 2024.\n\n[6] Puig, Xavier, et al. Watch-and-help: A challenge for social perception and human-ai collaboration. ICLR, 2021.\n\n[7] Chang, Matthew, et al. Partnr: A benchmark for planning and reasoning in embodied multi-agent tasks. ICLR, 2025.\n\n[8] Yang, Chengrun, et al. Large language models as optimizers. ICLR, 2024."}, {"review_id": "6hUAIiW86S", "reviewer": "Reviewer_PjNb", "summary": "This paper introduces COOPERA, a framework for continual, open-ended human-robot interaction with simulated humans driven by psychological traits and long-term intentions. The paper provides a new benchmark for such tasks, and presents a method to adapt to human behaviors by predicting their intentions.", "questions": "1. While the proposed method gains the best improvement over time, it seems that it does not converge to the one with ground truth intentions, as in Fig. 6. Will it help if you simulate for more days?\n2. What is the baseline method used for comparison in Table 2?\n3. How do you define success for each task? Any visualization/video on how the trained robot agent actually performs during testing?", "limitations": "N/A", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength:\n1. The long-term human-robot interaction problem is important and the benchmark provided along the paper can be useful.\n2. Ablation experiment for the proposed method is comprehensive\n\nWeakness:\n1. The discussion on the benchmark problem is not sufficient, given it is a newly introduced and not a widely acknowledged testbed.\n2. The comparison with baseline methods needs more discussions.", "was_revised": true, "final_justification": "My concerns are addressed and I raised my score in support of this paper.", "rebuttal": "Thank you for your insightful and constructive comments. We appreciate your recognition of the contribution of our framework and the introduced benchmark, as well as the design of our ablation study. This recognition is also kindly shared by other reviewers (Reviewer kYPv, Reviewer Zf7z, Reviewer irvc). In response to your suggestions, we conducted **additional experiment** to strengthen our analysis. For all HRI-related experiments, we report results from our main method under the more challenging collaboration type 2 (one trial per setting) due to time constraints. Below, we address each of your concerns.\n\n---\n\n**Q1:** The discussion on the benchmark problem is not sufficient, given it is a newly introduced testbed.\n\n**A1:** \nThank you for the valuable suggestions! We fully agree that providing additional explanations to Section 3.1 is essential, given that COOPERA is a newly introduced testbed. Central to COOPERA is our proposed human model and the continuous feedback mechanism operating at a daily scale. With these components, COOPERA differs from previous HRC benchmarks [1, 2] in two aspects:\n\n1) Existing benchmarks typically focus on short-term, episodic HRC tasks that are closed-set and predefined. In contrast, our human model generates open-ended, environment-driven, and dynamic human activities that vary with time, enabling COOPERA to support long-horizon collaboration tasks spanning multiple days. \n\n2) The daily feedback mechanism allows robots to continuously optimize their behavior over extended periods, leading to increasingly personalized interactions tailored to individual human traits and preferences.\n\nWe will explicitly add these points to Section 3.1.\n\n---\n\n**Q2:** The comparison with baseline methods needs more discussions.\n\n**A2:** \nThank you for highlighting this! We strongly agree that additional detailed analysis on the performance of our method vs. the baselines are needed. For paper writing, we will include more analysis in Section 4.3 and move some of the implementation details in Section 4.1 to Appendix. \n\nBelow we provide an in-depth discussion of why each method achieves distinct results. We evaluate COOPERA against six baselines: (1) Direct Prompting, (2) Direct Finetuning, (3) GT\\_Intention, (4) Random, (5) Intention Agnostic, and (6) Human & Context Agnostic.\n\nFor within-day improvement (Fig. 6a), our method achieves the highest performance gain by explicitly modeling the correlation between human intentions/tasks, traits, and temporal dependencies. As the day progresses, the robot accumulates interaction history and adapts its behavior to the human preferences. (1) Direct Prompting shows minimal improvement as it depends on retrieved interaction history without learning the human preferences. (2) Direct Finetuning performs slightly better but still struggles due to its rigid mapping from inputs to intentions/tasks, making it biased toward frequent training examples and overlooking the varying distribution of human behavior. (3) GT\\_Intention, despite having ground-truth intentions, shows limited gains because receiving the correct intention upfront removes the need to model temporal context or adapt over time. Its performance is static by design and it serves as the upper bound method. (4) Random degrades over time due to the absence of validation. (5) Intention Agnostic shows moderate improvement but lags behind our method, as it skips intention inference and thus loses contextual information, which is problematic since temporal dependencies between intentions are typically stronger than between tasks (validated in Section 4.2). (6) Human & Context Agnostic captures only superficial time-task correlations and ignores human traits or temporal context, leading to weak within-day gains.\n\nAcross-day improvement (Fig. 6b) further highlights the importance of learning the correlation between human intentions, human profile, and temporal dependencies. Our method achieves strong across-day gains—ranking second only to (3) GT\\_Intention, which receives the correct daily intention and thus bypasses the most challenging aspect of the problem. (1) Direct Prompting, (2) Direct Finetuning, and (4) Random exhibit minimal across-day improvement for the same reasons they underperform within a day: lack of learning from feedback, rigid input-behavior mappings, or absence of validation. (5) Intention Agnostic and (6) Human & Context Agnostic show some improvement, but their performance remains significantly lower than our main approach, as they exclude critical components such as explicit intention modeling and the integration of human profile and temporal context.\n\n---\n\n**Q3:** What is the baseline method used in Table 2?\n\n**A3:** \nThank you for raising this point! We will add more explanations regarding the baselines for this experiment for better clarity. Table 2 evaluates our framework’s out-of-domain generalization (Section 4.3, line 319) in two scenarios: (1) scene generalization and (2) human generalization.\n\nIn the scene generalization scenario, we first train the robot to collaborate with the same human across four distinct scenes (as defined in Setting 2: same human, different scenes). We then evaluate its ability to generalize to a fifth, unseen scene with the same human. The baseline here is the robot’s average performance during the initial interaction in the unseen scene without any finetuning on the previous four scenes, evaluated across 10 different humans.\n\nIn the human generalization scenario, we train the robot to collaborate with three different humans in the same environment (Setting 3: different humans, same scene). We then assess its generalization to a fourth, previously unseen human. The baseline in this case is the robot’s initial, unadapted performance when interacting with the new human, without finetuning on the previous three, averaged across 5 new humans.\n\nNotably, our method outperforms the baseline in both cases, indicating that the robot learns transferable representations of human traits and environments. This shows that the system avoids overfitting and generalizes well to new humans and scenes.\n\n---\n\n**Q4:** How do you define “success” for each task?\n\n**A4:** \nThank you for pointing this out! We agree that evaluation clarity is important and will revise Section 4.3 (line 289) accordingly. In COOPERA, we assess F1-based success rates using three complementary methods, consistent with existing HRC benchmarks [1, 2, 3].\n\nIn our framework, the robot-VLM predicts a superset of tasks to assist the human. An LLM-based classifier then filters this list, assigning \"yes\" or \"no\" to each task. For baselines that lack this component (e.g., Direct Prompting or Fine-tuning), we assume all predicted tasks are labeled \"yes\". Please see Section 3.3 for details.\n\nWe measure success using three criteria:\n\n1) Predicate-based success: Tasks are executed in the simulator and evaluated via predefined predicate functions (Appendix D.3, line 714). Matching is based on object classes, following Watch-and-Help [1], and F1 is computed over these outcomes.\n\n2) LLM-based success: The simulated human (human-LLM) is shown the robot’s predicted tasks and the ground-truth intention, and gives binary (\"yes/no\") feedback on whether each task fulfills the intention (please see Appendix C.3 and F for detailed feedback machnism and exact prompt). F1 is computed from this feedback.\n\n3) Human verification: Real humans replace the simulated human in (2) and provide judgments. This ensures our evaluation extends from simulation to real-world validation by human evaluators.\n\n---\n\n**Q5:** Is there any visualization or video demonstration showing how the trained robot agent performs during testing?\n\n**A5:** \nYes! For the predicate-based evaluation, the robot directly executes tasks in the simulated environment. We also provide functions for the human agent to perform actions collaboratively when the robot’s assistance is approved. This enables us to create visualizations and demonstration videos illustrating the human-robot collaboration process, as in Fig. 7 of our paper. Due to this year's conference policy restricting attachment of supplementary PDF during rebuttal, we are unable to include these visuals here. However, we will add additional visualization examples to the Appendix and our project website after the rebuttal period. Furthermore, we will release all code related to our framework.\n\n---\n\n**Q6:** While the proposed method gains the best improvement over time, it seems that it does not converge to the one with ground truth intentions, as in Fig. 6. \n\n**A6:**\nThis is a great question! We conduct one **additional experiment**. We extend the current setup by doubling the collaboration duration to observe continued adaptation trends.\n\n|Method|D1|D2|D3|D4|D5|D6|D7|D8|D9|D10|\n|-|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|Main|0.236|0.250|0.342|0.406|0.431|0.425|0.455|0.473|0.478|0.470|\n\nFrom the results, we observe that performance stabilizes around day 8. While our benchmark focuses on collaboration up to day 5, our main interest lies in the personalization trend. Simulating longer HRC is an easy extension within our framework. We believe this stabilization reflects a natural saturation point. By day 8, most human preferences driven by traits and routines are learned, allowing the robot to form a reliable alignment. Given the capacity of our 7B model and LoRA-based finetuning, further gains become marginal without introducing more capable models or finer-grained learning signals such as multimodal cues or reasoning chains.\n\n---\n\n**References:**\n\n[1] Watch-and-help: A challenge for social perception and human-ai collaboration. ICLR 2021.\n\n[2] Partnr: A benchmark for planning and reasoning in embodied multi-agent tasks. ICLR 2025.\n\n[3] NOPA: Neurally-guided Online Probabilistic Assistance for Building Socially Intelligent Home Assistants. ICRA 2023."}]

Author rebuttal:

Meta-review: This paper introduces COOPERA, an Human-Robot Collaboration framework designed to facilitate open-ended collaboration with humans over long horizons, specifically considering the traits demonstrated by (simulated) humans.  The reviewers appreciated both the framework and the simulation of human behavior.  Ultimately, all reviewers agree that the paper should be accepted — the authors’ rebuttal caused most of the reviewers to raise their scores.
Decision: Accept (spotlight)