Title: COOPERA: Continual Open-Ended Human-Robot Assistance
Venue: NeurIPS 2025
Accepted: yes
Keywords: Human-Robot Collaboration

Human Simulation

Social Embodied Agent
Abstract: To understand and collaborate with humans, robots must account for individual human traits, habits, and activities over time. However, most robotic assistants lack these abilities, as they primarily focus on predefined tasks in structured environments and lack a human model to learn from. This work introduces COOPERA, a novel framework for COntinual, OPen-Ended human-Robot Assistance, where simulated humans, driven by psychological traits and long-term intentions, interact with robots in complex environments. By integrating continuous human feedback, our framework, for the first time, enables the study of long-term, open-ended human-robot collaboration (HRC) in different collaborative tasks across various time-scales. Within COOPERA, we introduce a benchmark and an approach to personalize the robot's collaborative actions by learning human traits and context-dependent intents. Experiments validate the extent to which our simulated humans reflect realistic human behaviors and demonstrate the value of inferring and personalizing to human intents for open-ended and long-term HRC.


Peer reviews:
comments: SUMMARY:
This paper introduces COOPERA, a framework for continual, open-ended human-robot interaction with simulated humans driven by psychological traits and long-term intentions. The paper provides a new benchmark for such tasks, and presents a method to adapt to human behaviors by predicting their intentions.

STRENGTHS AND WEAKNESSES:
Strength:
1. The long-term human-robot interaction problem is important and the benchmark provided along the paper can be useful.
2. Ablation experiment for the proposed method is comprehensive

Weakness:
1. The discussion on the benchmark problem is not sufficient, given it is a newly introduced and not a widely acknowledged testbed.
2. The comparison with baseline methods needs more discussions.

QUESTIONS:
1. While the proposed method gains the best improvement over time, it seems that it does not converge to the one with ground truth intentions, as in Fig. 6. Will it help if you simulate for more days?
2. What is the baseline method used for comparison in Table 2?
3. How do you define success for each task? Any visualization/video on how the trained robot agent actually performs during testing?

LIMITATIONS:
N/A
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-29

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces COOPERA, a novel framework for continual, open-ended human-robot collaboration in simulated household environments. The core innovation lies in simulating psychologically grounded human agents using LLMs that generate temporally coherent, traits-driven behaviors across multiple days. The robot assistant learns from these interactions to infer human preferences and adapt its actions accordingly. The authors propose a benchmark, a method for human trait and task inference, and conduct thorough experiments on within-day and across-day adaptation, scene and human generalization, as well as human studies validating the realism of their simulations.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nThe proposed setting fills a gap in HRC research by addressing long-horizon personalization and temporal reasoning. Before LLMs are widely studied, the cost of obtaining human data was high and it is useful now to have LLM included to simulate the data.\n\nThe paper provides diverse experiments, including simulation, user studies, and offline real-human tasks. The use of predicate-based, LLM-based, and human evaluations strengthens the validity of the findings.\n\nThe simulation of long-term human behavior driven by LLMs with embedded personality traits is original and adds realism to the evaluation framework.\n\n\nWeaknesses:\n\nWhile the study uses a reasonably sized model (e.g., Llama-3.2-11B), it remains unclear whether scaling up LLMs/VLMs (e.g., GPT-4, Gemini, Claude) would significantly enhance robot performance of the proposed methods or the baselines. The authors mention computational constraints but do not explore the trade-offs or potential gains. More experiments could validate if the performance gap between the proposed methods and baselines would become bigger or smaller with more advanced LLMs.\n\nThe method uses specific LLM prompts and retrieval heuristics; a deeper analysis of robustness across LLM backbones or prompt configurations would strengthen generalizability claims.\n\nQUESTIONS:\n1 All experiments use relatively small or mid-sized models (e.g., LLaMA-3-11B, Mistral-7B). Is there a particular computational or latency constraint driving this choice? Do the authors expect significantly better assistance behavior from larger models, or do smaller models already saturate performance given the task complexity?\n\n2 How sensitive is the pipeline to the choice of LLM/VLM backbones? The paper includes one ablation replacing the robot’s VLM, but further experiments could highlight how brittle (or robust) the system is to different base models or prompting strategies.\n\n3 While the study focuses on offline interactions and day-level adaptation, do the authors envision a version of COOPERA where real-time, in-episode adaptation (e.g., using continual feedback during a task) can be integrated? If so, what components would need to be modified?\n\n4 Given the richness of the simulated human profiles and long-term interaction, are there any qualitative or longitudinal metrics that could better reflect improvement in human-robot support or perceived usefulness of the robot?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces COOPERA, a framework designed for continual, open-ended human-robot collaboration. It simulates human behaviors across multiple days using LLMs, driven by psychological traits and long-term intentions. A robot assistant agent leverages a VLM and another LLM to infer human intentions, classify tasks, and offer collaborative actions. Evaluations are conducted within simulated environments, assessing the robot’s ability to personalize assistance by learning human traits and adapting to context-dependent intents.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\nThe work addresses a clearly motivated and significant problem and it provides thorough documentation of experimental settings, parameters, and model specifics. The analysis conducted evaluates multiple dimensions of the proposed approach, offering insights into the effectiveness and limitations of the system. The integration and application of foundation models in HRC tasks demonstrates thoughtful methodological design, leveraging existing technology to advance the field.\n\nWeaknesses: \nThe evaluation heavily depends on synthetic human simulations, with limited validation against genuine human behaviors, raising questions about real-world applicability. The methods for assessing the realism of simulated humans are restricted primarily to classifier-based distinguishability and basic user studies, lacking more comprehensive measures of realism or plausibility. Additionally, the human studies narrowly focus on personality type identification and task identification, neglecting the broader context of expected or realistic behavior. There is a notable dependence on foundation models throughout the entire pipeline, which raises concerns regarding the genuine methodological novelty. The dense structuring of the content limits the nuanced discussion of results and findings and the paper appears to insufficiently explore or discuss potential system failures, both in terms of human simulation realism and robot assistance effectiveness. \n\nOther comments:\nFigure 1 appears not to be referenced in the text. Including explicit references or discussing it in relation to the framework's overview would enhance clarity.\n\nGiven the complexity and ambitious scope of the proposed framework, the content seems more suitable for a journal article where the authors can extensively discuss each component with greater depth. Alternatively, the paper might benefit from clearly delineating the presentation into separate components: one part thoroughly evaluating the COOPERA framework's capability to simulate human behavior realistically, and another part focused on the robot assistance approach. As currently structured, it is challenging to fully trust the assistance methodology without first establishing comprehensive confidence in the fidelity and realism of the underlying COOPERA simulation framework.\n\nQuality: Technically solid, though reliance on synthetic data without substantial real-world validation limits completeness.\nClarity: Generally clear but overly dense; improved organization or division into separate works would greatly enhance readability.\nSignificance: Addresses an important, challenging HRC problem with potential high impact, but practical significance uncertain due to limited realism validation.\nOriginality: Novel integration of existing foundation models in a comprehensive continual collaboration framework, clearly differentiating from prior methods.\n\nQUESTIONS:\n1. In Figure 6b, intent-agnostic and human-agnostic models show comparable learning progress to the full proposed approach. Does this indicate that the system learns general collaboration strategies rather than specific personalized intent inference?\n2. Clarify how feedback was provided to the robot at the end of each day. Was it binary (e.g., correct/incorrect predictions), detailed action-level feedback, or summary-level feedback?\n3. How exactly were actions presented to the robot? Did the robot rely purely on visual observations, or was textual context also provided? Given the complexity of certain tasks described (e.g., brainstorming), clarification on inference challenges from visual data alone is warranted.\n4. Clearly define the criteria for determining success in collaborative tasks. Were these predefined success conditions, user-defined outcomes, or evaluated through additional automated metrics?\n\nLIMITATIONS:\nWhile the authors acknowledge the difficulty in validating simulated human behaviors, explicitly addressing the risk of \"echo chamber\" effects due to reliance on LLM-generated data would strengthen the paper's transparency.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

comments: SUMMARY:
This paper presents a novel framework for human-robot interaction (HRI) with language models as simulated human agents. The authors performed standard evaluation on both the framework itself and a curated robotic VLM agent. This novel task opens up new challenges and opportunities in embodied research.

STRENGTHS AND WEAKNESSES:
Strengths:
The idea of simulating human behavior through the profile -> intent -> task -> execution pipeline is quite interesting. To me, it is even worthier of discussion than the overall framework. Nonetheless, the framework provides a novel perspective of HRI and opens up possibilities for large scale data collection and training.

Weaknesses:
1. The human agent follows a pre-set routine of actions, but normally a human would at least adapt to what the robot is doing, etc. clearing the path for the robot.
2. The experiments focus on the success rate increase over time, which is kind of a test-time learning setting. I would expect more analysis on the 'final success rate', the success rate after sufficiently days / episodes so that in theory the agent's intention is deducible from its actions.

QUESTIONS:
1. Why is a fixed daily routine selected instead of leaving room for interaction?
2. What is the performance of the baselines after sufficient days? If the generated data has limited days / episodes, maybe try repeating the overall routine as a cycle.
3. From the provided Figure 12 in Appendix, I see clear patterns between each day, i.e., increase->decrease within one day and increase even more on the second day. Is it coming from the task setting or the agent paradigm?

I will consider increasing my score if the authors provide additional details to address my concerns.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 2
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-02