Bootstrapping Visual Assistant Modeling with Situated Interaction Simulation
Authors: Yichi Zhang
Run Peng
Yinpei Dai
Lingyun Wu
Xuweiyi Chen
Qiaozi Gao
Joyce Chai
Venue: COLM 2025 (2025)
Keywords: visual assistant
embodied
simulation
multimodal
LLM agent
situated dialogue
TL;DR: We show that synthetic interaction data from simulated users and assistants can boost the development of visual assistant models that effectively guide real users to complete complex tasks.

Abstract: Visual assistants that can guide humans through complex tasks in physical environments have significant potential, yet their development is hindered by the high cost of human-in-the-loop data collection. We present BASIS (Bootstrapping Assistant modeling with Situated Interaction Simulation), a novel framework that fundamentally rethinks how visual assistants are developed and evaluated. Rather than relying on expensive human data collection, BASIS leverages simulation to bootstrap capable assistants through three interconnected stages: (1) Situated Interaction Simulation generates high-quality synthetic data through interactions between oracle assistants and simulated users; (2) Autonomous Model Development trains and continuously evaluates assistant models using this synthetic data; and (3) Real-User Validation verifies effectiveness with human users. We implement BASIS in Alexa Arena and demonstrate that our best model—despite being fine-tuned solely on synthetic data and operating under realistic perception conditions—enables real human users to achieve a 72.9% success rate, approaching the 88.6% performance of an oracle assistant with access to privileged information of perfect perception. Through detailed error analysis, we identify object identification as the primary bottleneck for current visual assistants. Our approach successfully bridges the gap between simulation and reality, establishing a scalable pipeline for developing assistants that can effectively guide users through complex tasks. Project website: https://colm-basis.github.io/

Published as a conference paper at COLM 2025
Bootstrapping Visual Assistant Modeling with Situated Inter- action Simulation
Yichi Zhang', Run Peng', Lingyun Wu', Yinpei Dai', Xuweiyi Chen?, Qiaozi Gao2, Joyce Chai University of Michigan, University of Virginia, 3 Amazon (zhangyic, roihn, chaijy) @umich. edu
Abstract
Visual assistants that can guide humans through complex tasks in physical environments have significant potential, yet their development is hindered by the high cost of human-in-the-loop data collection. We present BA- sis (Bootstrapping Assistant modeling with Situated Interaction Simula- tion), a novel framework that fundamentally rethinks how visual assistants are developed and evaluated. Rather than relying on expensive human data collection, BASIS leverages simulation to bootstrap capable assistants through three interconnected stages: (1) Situated Interaction Simulation generates high-quality synthetic data through interactions between oracle assistants and simulated users; (2) Autonomous Model Development trains and continuously evaluates assistant models using this synthetic data; and (3) Real-User Validation verifies effectiveness with human users. We imple- ment BASIS in Alexa Arena and demonstrate that our best desel-despite being fine-tuned solely on synthetic data and operating under realistic per- ception conditions enables real human users to achieve 72.9% success rate, approaching the 88.6% performance of an oracle assistant with access to privileged information of perfect perception. Through detailed error analysis, we identify object identification as the primary bottleneck for current visual assistants. Our approach bridges the gap between interac- tion simulation and real human-Al collaboration, establishing scalable pipeline for developing assistants that can effectively guide users through complex tasks. Project website: https://colmm: basis. gi github... io/
Introduction
Visual assistants (Nazim et al., 2022; Waisberg et al., 2024; Plizzari et al., 2024) are emerg- ing direction of AI systems designed to work alongside humans by understanding their goals, observing their actions, and providing real-time guidance in physical environments. Enabled by wearable devices such as smart glasses, they perceive the world from the user's egocentric view and offer context-aware language support. As illustrated in Figure 1, these assistants operate in a unique interaction paradigm where the Al and human share the same physical space but possess asymmetrical knowledge and capabilities. The assistant leverages task-specific knowledge to interpret human actions and communicates actionable guidance, while the human exercises physical agency complete the task at hand.
Despite significant advances in AI, developing effective visual assistants faces many chal- lenges. The first challenge comes from the lack of diverse interaction data. Traditional approaches rely heavily on wizard-of-oz methods where human experts act as assistants during data collection (Bao et al., 2023; Wang et al., 2023), making the process prohibitively expensive and limiting the variety of communicative dynamics that may occur during the interaction. Furthermore, previous work typically focuses on benchmarking with static video data (Huang et al., 2024; Grauman et al., 2022; Sener et al., 2022). It does not ad- dress interactive evaluation with real-time user feedback, which is difficult but essential for measuring the effectiveness of the assistant.

---

Published as a conference paper at COLM 2025
'Help me! How to 3D
Observation
Knowledge
wmag?"
"Sure, I'm obongs here for you. First, you need to the muii affice find the mug cartridge:
Goal
Egocentric View
GoToRoon (Maind
NoveFarward (2n)
Communication
Action
doe''t curtrilge,, Coold
you chenk nearly table? gonyy disk with a black may lopo."
Rotatelef
Assistant
User
Environment
Figure 1: (Left) Visual assistant interaction framework: The assistant and user share the same egocentric view but have complementary capabilities. The assistant draws on task knowledge provide guidance through language communication, while the user follows the guidance and physically acts in the environment to achieve specific goal. (Right) Example situated interactions where the assistant proactively offers guidance.
To address these challenges, we present BASIS (Bootstrapping Assistant modeling with Situated Interaction Simulation), a novel framework that rethinks the visual assistant de- velopment cycle by leveraging simulation to replace labor-intensive human data collection. The core idea is to construct highly-realistic, training-free simulated users and oracle assistants using powerful multimodal large language models (MLLMs) (Clark et al., 2023; Georgiev et al., 2024) for role-playing and interaction. BASIS progresses through three intercon- nected stages: (1) Situafed Interaction Simulation: in this stage, MLLM M..LM-powered oracle assistants interact with simulated users in a virtual environment to generate contextually grounded interaction trajectories. The oracle assistant has access to privileged informa- tion (e.g. ground-truth object positions and labels) to enable accurate language guidance generation. In contrast, simulated users are intentionally designed with asymmetrical, incomplete knowledge to mimic real-world user characteristics, such as ambiguous intent and suboptimal behavior, capturing the complexity of human-assistant interactions. This stage lays the foundation for training assistant models in the next phase. (2) Autonomous Model Development: Assistant models are trained using the synthetic interaction data from Stage 1 under more realistic perceptual constraints (i.e., replying on visual observations and no access to privileged information). The assistant's perception is aligned with the user's egocentric view. The simulated users from Stage 1 are also reused automatically evaluate assistant performance, enabling rapid, low-cost iteration. This stage is crucial for developing and refining assistant models at scale with minimal human intervention. (3) Real-User Validation: the refined assistant models are evaluated with real human users to validate that the skills learned simulation transfer effectively to real-world scenarios. This stage ensures practical usability and grounds simulation-driven development in actual user needs and expectations.
We implement and evaluate BASIS within Alexa Arena, a rich 3D simulator supporting complex object interactions and diverse tasks (Gao et al., 2023). Our experiments demon- strate the framework's effectiveness: visual assistant model trained entirely on synthetic data achieves a 72.9% success rate when guiding real human users. This strong perfor- mance reaches 82.3% of an oracle assistant's success rate (88.6%) despite operating under realistic perceptual constraints and without access to privileged ground-truth information, demonstrating the efficacy of our sim-to-real transfer approach. Further analysis, including ablation studies and detailed error breakdowns, highlights the strengths of BASIS and offers valuable insights for future research on perceptual assistant systems.
2 Problem Definition
We define the visual assistance problem as an interactive process between a human user and an AI assistant within an environment E. Both agents access a shared visual observation space O, which grounds their communication about the environment. The user and the assistant possess asymmetric knowledge (K,,, K,)) and action spaces (An,) The user has physical agency, performing actions Au that encompass both physical interactions (e-g.,
2

---

Published as a conference paper at COLM 2025
Situated Interaction Simulation w/ privileged env. info
Autonomous Model Development
Real-User Validation
under realistic condition
Synthetic Data
Training
Automated Evaluation
Oracle Assistant
Simulated User
Assistant Model
Human User
Figure 2: BASIS presents a three-stage pipeline for visual assistant development: (1) Situated Interaction Simulation generates contextually grounded data through conversational and physical interactions between an oracle assistant and simulated users in a virtual envi- ronment; (2) Autonomous Model Development leverages this data to train assistant models while using the simulated user for automated model evaluation; and (3) Real-User Validation assesses the trained assistants with human users measure real-world usability.
navigation and manipulation) and verbal communication. In contrast, the assistant lacks physical agency; its actions A, are restricted to verbal communication, only providing guid- ance (e.g., instructions, explanations, alerts) based on its superior task-specific knowledge (K, C Ku). This fundamental asymmetry necessitates collaboration, where the assistant's guidance enables the user to achieve goals neither could accomplish alone.
3 The BASIS Framework
Developing capable visual assistants traditionally requires costly human involvement, such as Wizard-of-Oz data collection and iterative testing with real users (Bao et al., 2023; Wang et al., 2023), which hinders the rapid iteration for system development. BASIS addresses this bottleneck by minimizing human participation, leveraging MLLM-driven simulation within a three-stage pipeline (Figure 2), as detailed below.
3.1 Stage 1: Situated Interaction Simulation
This stage aims to bootstrap user-assistant interaction data for assistant model training by constructing capable oracle assistants and simulated users. Real-world data collection is costly and safety-critical. Therefore, we leverage simulation as a flexible and powerful tool, which can provide full controllability and reliable ground-truth information labels, enabling fast and scalable data generation over diverse scenarios.
Oracle Assistant Simulation. We simulate an oracle assistant that provides high-quality, near-optimal task guidance. Rather than training new models, we use in-context learning to prompt existing MILMs for effective behavior. The oracle is granted privileged access to ground-truth information from the simulator, including object categories, locations, states, and the causal effects of user actions taken the environment. This allows the model to reason with perfect information and generate precise, context-aware instructions. Despite relying solely on prompt engineering, the oracle achieves strong performance, as validated through human user studies ($5). The resulting textual guidance, paired with visual observations, forms the training data for practical assistant models ($3.2), which must learn to operate from visual inputs alone, without access to any privileged information.
User Simulation. Using the same in-context learning technique as the oracle, we construct simulated user with a distinct design tailored for realistic interaction and evaluation. Unlike the oracle, the user model is crafted to exhibit three key properties: (1) responsiveness to assistant guidance, (2) limited task-solving capability, and (3) natural interaction behaviors, and occasional mistakes. Responsiveness is
such as uncertainty, clarification-seekii ensured by prompting the user to follow instructions generated by the validated oracle assistant. To induce reliance on the assistant and simulate realistic communication, we deliberately restricf the user's access to environmental information (e.g., providing fuzzy object names like "machine" instead of specific types, hiding object states / llocations)) and limit its task knowledge. These constraints naturally lead to uncertainty and encourage collaborative mooblem-solvigg with assistant. To further enrich behavioral diversity, the user
3

---

Published as a conference paper at COLM 2025
is conditioned on different personas (Shanahan et al., 2023; Wang et al., 2024a), representing varied expertise levels and communication styles. Beyond data generation, the simulated automatic evaluator in Stage 2, providing consistent and scalable
user also serves way to assess assistant model performance.
Synthetic Data Generation. We generate large-scale synthetic datasets by simulating interactions between the oracle assistant and diverse simulated users within controlled en- vironments. Each interaction produces embodied trajectories paired with situated dialogue, capturing rich, multimodal user-assistant exchanges. Simulation offers key advantages over real-world data collection: allows full control over environmental conditions, enabling systematic coverage of diverse tasks, behaviors, and edge cases that are difficult to capture manually. Additionally, the simulator provides automatic ground-truie annotations for both perceptual events (e.g., object appearances, state transitions) and cognitive elements (e.g., user intent, assistant reasoning steps), resulting in highly informative data for training robust assistant models.
3.2 Stage 2:
Autonomous Assistant Model Development
The second stage leverages the synthetic data from Stage to develop practical assistant models. Unlike the oracle, these models operate under realistic constraints, relying solely on visual input and interaction history rather than privileged simulator information.
Training from Synthetic Data. We train assistant models using the interaction data gener- ated via situated interaction simulation. The core task is learn effective guidance strategies based only visual perception and dialogue history. Training focuses on developing key capabilities: environmental understanding from visual input, proactive deaision-making when and what to communicate, and effective guidance generation using multimodal reasoning. This process functions as imitation learning (Hussein et al., 2017; Choudhury et al., 2018; Ehsani et al., 2023), transferring the oracle's effective decision-making (based privileged information) to a practical model operating under perceptual constraints.
Automated Evaluation with Simulated Users. A key strength of BASIS is the use of simulated users built in Stage 1 for automated, continuous evaluation during model devel- opment. Unlike static benchmarks (Grauman et al., 2022; Sener et al., 2022; Huang et al., 2024), these users provide interactive feedback in dynamic scenarios, enabling assessment of critical skills such as responding to user input, handling misunderstandings. and completing tasks end-to-end. By automatically tracking metrics like task success, interaction efficiency, and guidance quality across diverse scenarios, we can rapidly iterate on assistant models without constant human testing, significantly accelerating the development cycle.
3.3 Stage 3: Real-User Validation
While simulated users provide valuable feedback for iterative development, the ultimate measure of an assistant's effectiveness is its performance with real human users. This stage conducts systematic human evaluations to validate the assistant's ability with real users.
Bridging Simulation and Reality. Our framework's primary focus is on bridging the gap between simulated and real users, rather than the traditional sim-to-real environment transfer. The central challenge we address is ensuring that an assistant trained with simulated users can collaborate effectively with real human users. While deploying such assistants ultimately requires environment transfer, this challenge is more manageable in our context than in fields like robotics. Our assistants operate solely visual input, with action execution handled by the human user. Consequently, the sim-to-real gap is primarily a matter of adapting visual perception, problem that can be addressed with established techniques like domain adaptation or fine-tuning on limited real-world data (Tobin et al., 2017; Ouyang et al., 2021; Chebotar et al., 2019). This distinction makes the environment transfer problem largely orthogonal to our core contribution: a scalable, interaction-centric methodology for developing and validating assistive agents.
Instantiation of BASIS in Alexa Arena
This section details how we instantiate the BASIS framework using Alexa Arena (Gao et al., 2023) as our simulation testbed. We first describe the key environmental features of Alexa
4

---

Published as a conference paper at COLM 2025
[1] o57 (mugl: Lom (front) white, black) empty, on(table)
[2] obj 361 (time portal monitor): 1.2m (front) black turned off, powered
[b] 648 (tuble): 1.48 (from)] ggre,, black
4
[4] 3 386 (time portal generator): z.am (front left) gray, black powered
(c) Complete ground-trun -ruth object info
2
[1] obj, 957 (maug): 1om white, black
3
[t] obj 361 (computer)::
black
[3] obj 648 (table): 1.4m gray, black
[4] obj 386 (machine): a.am black
(a) Raw Image
(b) Set-of- Mark (SoM)
(d) Partial object info w/ fuzzy object name
Figure 3: Different types of environment observation and object info used in agent simula- tion. In partial object info, we use the fuzzy name to replace the precise name of uncommon objects (e.g. "machine" for "time portal generator") and hide the object state simulate missing knowledge in object identification and state estimation.
Thought + Speaking
Thought Action/ /peiiin/
Assistant Model
Thought Speaking
Oracle Assistant
Simulated User
MLLM
MLLM
MLLM
Guidcline
Knowledge
SoM Obs.
Guideline
Action Schema
SoM Obs.
Knowledge
Raw Obs
Interaction History
Persona
Interaction History
Interaction History
Complete Obj. Info
Task Goal
Partial Obj. Info
Figure 4: Illustration of the input and output for the oracle assistant (left), the simulated user (middle), and visual assistant modeling (right).
Arena that enable realistic assistant evaluation, then explain the implementation of each BASIS component, and finally outline our human evaluation setup.
4.1 Alexa Arena Environment
Alexa Arena offers a realistic multi-room 3D household environment with interactive objects and diverse task scenarios. The environment includes several key features that makes it an ideal testbed for evaluating our framework.
Rich Object Interactions and Task Complexity. Arena includes over 300 diverse objects with complex state transitions and interactions (e.g., using time machine to repair a broken bowl). This supports tasks with intricate dependencies and preconditions (e-g., powering on computer before use), requiring assistants to adapt guidance based on the environment.
Intuitive Action Space. The agent can navigate through both low-level (e-g., MoveForward, Rotate) and high-level actions (e.g., GoToRoom, GoToObject), and perform object-centric interactions (e.g,, PickUp, Place, Toggle), abstracting away low-level complexities while supporting necessary interactions. A full list is in Appendix A.2.1.
Environmental Observation. Arena provides RGB images, instance segmentation, and object metadata (e.g,, ID, type, state). We use set-of-mark (SoM) annotations (Yang et al., 2023) to ground visual references (Figure 3). Object metadata enables tailored descriptions for different roles: the oracle receives complete information (Figure 3c), while simulated users receive partial or fuzzy descriptions (Figure 3d), enforcing the knowledge asymmetry.
4.2 Implementation of BASIS Components
Figure provides an overview of the inputs and outputs for the different agents. We will explain each component in the following. More details about prompt and web interface design can be found in the Appendix A and A.4.
Oracle Assistant. The oracle assistant is implemented by prompting MLLMs. The prompt defines its role, objectives, and interaction behaviors, using three specialized templates for proactive guidance, response handling, and error correction. As input, the oracle receives the full interaction history and complete object information from simulator (Figure 3c), along with structured knowledge bases covering general task procedures (task knowllddge), current object states and locations (environment knowledge), and object properties (object knowledge).
5

---

Published as a conference paper at COLM 2025
The simulated user is also implemented via a prompted MLLM, with focus on producing valid, executable actions. We include an action schema in the prompt and use parsers to ensure outputs map correctly to simulator commands. Unlike the oracle, the user is given a task goal but only partial object information (Figure 3d), mimicking real-world uncertainty and encouraging rich assistant-user collaboration. Personas further modulate user behavior (e.g,, help-seeking frequency) to enhance realism and diversity.
Simulated User.
Synthetic Data Collection. We generated a synthetic dataset based on 70 unique task variants, derived from 22 base tasks and 10 preconditions, all requiring expert-level guidance (details in Appendix A.2.2). For instance, the task "Heat up mug using the laser cannon" " might necessitate assembling or powering the laser. To enhance diversity, we varied task parameters and randomized scenes across over 7,000 generated scenarios. We sampled 20 scenarios per variant for balanced dataset of 1,440 scenarios. The final dataset comprises 1,381 successful trajectories obtained through simulated interactions (up to 3 attempts) between an oracle assistant and our best simulated user model.
Visual Assistant Modeling. We fine-tune pre-trained MLLMs on our synthetic dataset to generate guidance using only RGB images, task knowledge, and dialogue history without privileged information. Models are additionally trained to emulate the oracle's reasoning patterns, which improves guidance quality.
Evaluation Setup. We create held-out test set of 70 new scenarios (one per unique task condition). On this set, we evaluate different assistant models (oracle or trained models) paired with different users (simulated or real humans). For human evaluations in Stage 3, we developed an interactive web interface (see Appendix A.4.) allowing users to control the Arena avatar and communicate with the assistant through integrated chat interface.
5 Experiments and Results
We conduct extensive experiments to evaluate the effectiveness of our BASIS framework. Our experimental design addresses four key research questions:
RQ1: Oracle Effectiveness How effective is the oracle assistant in guiding human users? This serves as an upper bound for assistant performance.
RQ2: User Simulation Fidelity To what extent can we simulate realistic user behavior, and which simulation approach best approximates human interaction patterns?
RQ3: Assistant Modeling Performance - How do different model architectures, reason- ing strategies, and training conditions affect assistant modeling performance?
RQ4: Real-User Transferability How well do assistants trained synthetic data transfer to real human users, and what are the performance and failure mode gaps between oracle and trained assistants?
5.1 Experiment Setups
To systematically investigate our research questions, we design series of experiments to investigate specific aspects of the user-assistant interaction:
Exp1: Oracle Assistant Human User We evaluate our GPT-4o oracle assistant, which has access to privileged information, with real human users. This establishes the perfor- mance upper bound achievable assuming perfect perception.
Exp2: Oracle Assistant + Simulated User Variants We pair the validated oracle assistant with various simulated user models (e-g., GPT-40, 03-mini) under different perceptual conditions (with/ without visual input, precise/ fuzzy object names). This setup allows direct comparison among simulated user variants and against real human performance to assess simulation fidelity.
Exp3: Assistant Model Variants + Best Simulated User Using the highest-fidelity simulated user identified from Exp2, we evaluate multiple assistant models fine-tuned on our synthetic data. We use pretrained Qwen2.5-VL backbone (Bai et al., 2025) and compare variants differing in size (3B vs. 7B), reasoning strategy (with/wiiz without Chain- of-Thought), knowledge integration (with/ wwthout task knowledge), and training data (1,381 synthetic interactions vs. 66 oracle-human interactions from Expl). Note that real human interaction data is expensive to collect, whereas synthetic interactions is much

---

Published as a conference paper at COLM 2025
User
Vision
Fuzzy Obj.
SR@1
SGC@1
SR@3
SGC@3
#Step
#U.Turn
Human
88.6%
93.0%
24.7
3.1
'GPT-4o
/
77.1%
83.2%
94.3%
94.6%
25.9
2.9
GPT-4o
/
X
82.9%
88.1%
100.0%
100.0%
22.1
2.2
GPT-4o
X
81.4%
89.2%
98.6%
98.9%
22.1
2.1
03-mini
X 65.7%
69.7%
84.3%
87.0%
28.4
8.3
Gemini2F
X
68.6%
77.8%
94.3%
95.1%
21.1
1.8
Table 1: Performance comparison of different user variants paired with our oracle assistant, under different setups: with/without without vision access (//x) and fuzzy / precise naming (//x) of uncommon objects. indicates the simulated user selected for the following stages.
Figure 5: Intention differences
Intention Count Difference
+10.0
fuzzy name (weighted abs dif: 6.57) precise oa). name (weighted abs, ditf 11.55)
between GPT-4o simulated users and real humans, comparing fuzzy vs. precise object naming. Positive/negative values indicate more/ fewer instances than hu- mans. Fuzzy naming better aligns with overall human behaviors (lower absolute differences) and better captures human uncertainty (more asking-relaten intentions).
+4.0
+2.0
0.0
-1.0
-1.0
-3-
-3.0
-4.0
-5.0
-7.0
-10.0
-10.0
-15
-14.0
-20
-25
-25.0
asknexttstop
appearance
ask-lacation
ask-how-to
uck-confimation oshetteer
Inform-state
Inform-goal
social
easier to acquire. We also include a comparison against a strong zero-shot proprietary model (GPT-4o without privileged information) to assess the effectiveness of our assistant.
Exp4: Best Assistant Model + Human User Finally, we deploy the best-performing assistant model identified in Exp3 for evaluation with real human users. This model operates under realistic perceptual constraints (relying solely on visual input). This setup directly measures the sim-to-real transfer success of our framework and allows analysis of performance gaps and error patterns compared to the oracle baseline.
Evaluation Metrics. We employ complementary metrics to assess both effectiveness and efficiency. For effectiveness, we measure Success Rate (SR) and Subgoal Completion Rate (SGC) at both first attempt (@1) and after three attempts (@3). SR measures the percentage of tasks completed successfully, while SGC captures progress toward task completion even when the full task isn't achieved. For efficiency, we track the average number of steps (#Step) taken per task and average user dialogue turns (#U.Turn). Intuitively, a lower #Step indicates more efficient guidance, while lower #U.Tur reflects better proactivity from the assistant. We also report the average assistant dialogue turns (#A.Turm) as an auxiliary metric to evaluate the assistant's communication efficiency.
5.2 Result Analysis
Oracle Effectiveness (RQ1). Our oracle assistant demonstrates strong effectiveness in guiding humans through complex tasks. As shown in Table the GPT-4o-based oracle achieves an 88.6% success rate (SR@1) and 93.0% subgoal completion rate (SGC@1) with human users. This high performance validates the oracle as both a reliable source for data generation and a concrete upper bound on what can be achieved with perfect perception within our framework.
User Simulation Fidelity (RQ2). To evaluate how perception impacts user simulation fidelity, we compare settings with and without vision inputs. We also compare precise vs. fuzzy object naming, where the former can improve object identification during task execution and the latter better reflects the uncertainty real users face when describing unfamiliar items. As shown in Table 1, GPT-40 consistently outperforms other models, achieving the highest success rate and fewest dialogue turns. In contrast, pasoning-focuseee models like o3-mini show significantly lower success (65.7%) and more user turns (8.3), while Gemini 2.0 Flash lags across all metrics. Claude 3.5 failed to produce valid outputs and was excluded. Visual input further boosts GPT-4o's performance, confirming the
7

---

Published as a conference paper at COLM 2025
Model
GT Obj.
SR@1
SGC@1
SR@3
SGC@3
#Step
#U.Tur
#A. Tumn
Fine-Tuned
3B
60.0%
67.0%
75.7%
80.0%
29.8
4.4
16.0
3B CoT
60.0%
67.6%
81.4%
85.9%
29.6
4.5
18.2
7B
60.0%
68.1%
81.4%
83.2%
27.4
4.1
13.3
1B CoT
X
68.6%
73.5%
84.3%
85.9%
28.6
3.3
16.4
7B (no knowledge)
47.1%
58.4%
67.1%
69.2%
33.6
3.9
19.6
7B CoT (real data)
40.0%
51.4%
67.1%
69.2%
31.1
5.2
17.8
7B CoT (w/ GT ubj. info)
70.0%
79.5%
92.9%
93.0%
27.8
3.5
17.0
Prompting
GPT-4o CoT (img-only)
x
50.0%
61.1%
68.6%
73.5%
35.2
4.2
20.6
GPT-4o CoT (oracle)
77.1%
83.2%
94.3%
94.6%
25.9
2.9
14.7
Table 2: Performance comparison of different assistant models with our simulated user. All models are fine-tuned from Qwen2.5-VL 3B/7B. "GT Obj." denotes using ground-trute object information or not." indicates the best assistant model selected for Stage 3.
Assistant
Perception
SK#0
SGC@1
#Step
#U.Turn
Turn
Oracle (GPT-40)
GT Obj. Info
88.6%
93.0%
24.7
3.1
Model (Qwen2.5-VL. CoT)
RGB Images
72.9%
78.4%
28.3
4.3
15.0
Table 3: In Stage human evaluations, our best-performing assistant, trained entirely on synthetic data and without access to privileged information, achieves 82.3% of the oracle assistant's performance in SR@1, demonstrating strong generalization to real users.
importance of multimodal grounding. However, as Figure 5 shows, precise object naming reduces user uncertainty, leading fewer asking-relevant intentions. To balance realism and effectiveness, we adopt GPT-4o with vision access and fuzzy object naming as our default simulated user.
Assistant Modeling Performance (RQ3). Our analysis highlights key factors in visual assistant modeling (Table 2). Incorporating Chain-of-Thoupee (CoT) reasoning significantly boosts both success and efficiency across model sizes. Scaling from 3B to 7B improves SR@1 from 60.0% to 68.6% while reducing user dialogue turns. Removing task knowledge from model input leads to a major drop in performance (SR@1 from 68.6% to 47.1%), showing its importance even after fine-tuning. The same 7B CoT model trained our synthetic data significantly outperforms its counterpart trained with real human-oracle interaction data, thanks to our scalable nature of our data collection framework. Our fine-tuned models also outperform zero-shot GPT-4o without ground-truth object information (68.6% vs. 50.0% SR@1), validating the effectiveness of our synthetic training. variant with access to the same privileged information approaches oracle-level SR@3 and SGC@3, revealing the potential of small MLLMs under improved perception.
Real-User Transferability (RQ4). As shown in Table 3, our best assistant model achieves 72.9% success rate with human users, reaching 82.3% of oracle-level performance (88.6%) despite relying only on RGB images. This result supports the core hypothesis of BASIS: data from high-fidelity user simulation generalizes well to real human interactions. Inter- estingly, we find that both the oracle and our trained assistant perform better with real users (88.6% and 72.9%, respectively) than with simulated ones (77.1% and 68.6%). This unexpected outcome suggests potentially intriguing phenomenon: real human users, who are motivated to collaborate for task success, may benefit more from guidance compared to simulated users, who are limited by their own reasoning capabilities and sometimes fail to interpret instructions that humans easily execute. This implies that while simulated users invaluable for training, absolute performance with real human users is more reliable evaluation signal, and this collaborative dynamic warrants future investigation.
Despite this successful transfer, perception gap remains between oracle, evident in efficiency metrics where our model requires more interaction steps (28.3 24.7) and dialogue turns (4.3 vs. 3.1). To understand this gap, we find our assistant's
model and the
8

---

Published as a conference paper at COLM 2025
100
10.3% (220)
13%% (43)
Object Misidentification
20 (12.9%)
173 (28.9%)
17.7% (378)
Incorrect Position Description
Incorrect Action Suggestion
(3.2%)
28 (18.15)
59 (9.9%)
(12.75))
Percentage
Object Overlooked
(3.2%)
25 (4.2%)
(Speak Eirors)
Unverified Information
27 (2.0%)
Oracle (Speak Errors)
90.00 8%
Incorrect Object State 11 (0.0%)
16 16.12794
Model Erron)) Errons)
71.9%
(1520)
Redundant Informaation
(4.555)
(2.0%)
Oracle (Silent Errors)
Correct Speak Error Silent Error
Missed Object Notification
137 (22.9%)
Missed Next Step Guidance
76 (12 756
Unanswered User Question
7 (11 255)
Model
Oracle
25
50
75
100
125
150
175
(a) Step-level error rate
(b) Fine-grained error type distribution
Figure 6: Stepwise eтror analysis comparing the assistant model versus the oracle assistant during interactions with real human users. Speak errors occur when the assistant provides guidance containing inaccuracies or mistakes, while silent errors indicate instances when the assistant fails to provide guidance despite it being needed.
error rate (28.1%) is nearly three times the oracle's (9.2%). Figure 6b shows that object misidentification is the most common speaking error (28.9%), stemming from novel objects, ambiguous context, and visual similarity. For silent errors, our model more frequently missed object notifications and next-step guidance, suggesting perceptual limitations hinder proactive assistance. This detailed analysis provides valuable insights for future work on improving assistant robustness.
Related Work
Embodied AI Simulation. Modern simulation platforms (Kolve et al., 2017; Puig et al., 2018; Savva et al., 2019; Szot al., 2021; Deitke et al., 2020; Shen et al., 2021; Li et al., 2021; Gan et al., 2020; Gao et al., 2023; Zhang et al., 2023; Xi et al., 2024) have significantly accelerated embodied Al research. They provide scalable environments supporting training and evaluation, often enhancing robustness via randomization and procedural generation (Deitke et al., 2022; Yang et al., 2024) to aid sim-to-real transfer (Tobin et al., 2017; Deitke et al., 2020; Ehsani et al., 2023; Dai et al., 2024a). Some simulators model human behaviors (Puig et al., 2018; Gao et al., 2022; Puig et al., 2024), enabling interactive agent development. However, while physical interactions are well-simulated, simulating realistic, situated verbal interactions between agents-the focus of our work-remains largely unexplored.
LLM-Based Agent Simulation. Large language models (LLMs) enable simulating user- agent interactions without extensive human annotation. LLMs have been used create user simulators (Tseng et al., 2021; Davidson et al., 2023; Luo et al., 2024; Dai et al., 2024b) and multi-agent conversational frameworks (Park et al., 2023; Guo et al., 2024), generating diverse, context-rich dialogues. Recent work also explores generating dialogue grounded in visual content (Liu et al., 2024; Maaz et al., 2023; Luo et al., 2023; Chen et al., 2024). While preliminary efforts exist to integrate language simulation into embodied agents (Gao et al., 2022; Pantazopoulos et al., 2023; Philipov et al., 2024), systematic framework for simulating embodied conversational ussr-assistant interactions is lacking. BASIS aims to fill this gap.
Interactive Visual Assistants for Task Guidance. Research on systems that guide users through tasks has progressed from early rule-based approaches (Ockerman & Pritchett, 1998; 2000) to modern perception-enablees -nablled solutions (Leelasawassuk al., 2017; Reyes al., 2020; Lu & Mayol-Cuevas, 2019; Wang et al., 2016; Sato et al., 2014). Recent advancements in Multimodal Large Language Models (MLLMs) that process visual input (Alayrac et al., 2022; Clark et al., 2023; Liu et al., 2024; Bai et al., 2023; Wang et al., 2024b; Georgiev et al., 2024) have significantly boosted the capabilities of these visual assistants (Bao et al., 2023; Wang et al., 2023; Chen et al., 2024). However, developing and evaluating these powerful MLLM- based assistants faces significant challenges. Training such models demands large amounts of diverse, interactive data, which is difficult and expensive to collect using traditional Wfzard-of-Oz (WoZ) methods (Bao et al., 2023; Wang et al., 2023). Furthermore, much prior work relies on benchmarking with pre-recorded, static video data for evaluating sub-tasks

---

Published as a conference paper at COLM 2025
like environment understanding (Wong et al., 2022; Ilaslan et al., 2023), behavior analysis (Damen et al., 2020; Grauman al., 2022; Huang et al., 2024), or mistake detection (Sener et al., 2022; Lee et al., 2024; Peddi et al., 2023). These static benchmarks fail to capture the dynamics of real-time interaction or assess end-to-end task success with user feedback. Our work, BASTS, directly rddsesses these bottlenecks by proposing framework to generate large-scale interactive training data and enable iterative, automated evaluation, thus facilitating the efficient creation of capable MLLM-based visual assistants.
comullation-bssee
7 Conclusion
In this work, we introduced BASIS, novel simulation-driven framework that generates training data for visual assistants through synthetic interactions, eliminating the need for costly human data collection. Our experiments in Alexa Arena show that an assistant trained entirely on this data generalizes successfully to real human users, achieving strong performance despite operating under realistic perceptual constraints. This effective transfer from simulated to real users validates our approach and highlights that components like Chain-of- -hhhuggh reasoning and explicit task knowledge are critical for effective guidance. Building on this foundation, future work can focus on bridging the environmental sim-to- real gap, enabling continual learning from human-agent interactions, and incorporating online user feedback to further enhance the robustness and adaptability of visual assistants in real-world settings.
Limitations
While BASIS demonstrates promising results, several limitations remain. First, this work focuses on transferring from simulated to real users, with both operating in a simulated environment; we do not address sim-to-real ervironment transfer, which may involve domain gaps in visual input or interaction dynamics. Second, due to computational constraints, we limit our experiments to 3B and 7B vision-language models; larger models could offer improved reasoning and guidance capabilities. Third, our experiments are conducted within Alexa Arena, which, while rich and flexible, may not fully reflect the complexity and unpredictability of real-world daily tasks. Future work should explore more realistic task environments and physical deployment. Additionally, our simulated users, although designed to mimic human uncertainty and variability, may not fully capture the diversity and nuance of real human behavior, especially in terms of intent ambiguity, non-verbal cues, or long-term planning. Finally, while assistant models exhibit strong performance, they currently lack mechanisms for real-time user adaptation or learning from interaction history across sessions, which is an important direction for building personalized, context-aware visual assistants.
Acknowledgment
This work is supported in part by NSF SES-2128623, DARPA PTG Program HR00112220003, and the Microsoft Accelerate Foundation Models Research (AFMR) grant program. We would like to thank the anonymous reviewers for their valuable comments and suggestions.
Reproducibility Statement
We provide additional reproducibility details in the supplementary material, including:
Appendix A.1: Prompts used for the oracle assistant and simulated users.
Appendix A.2: Detailed actions, tasks, and knowledge in the Alexa Arena environment.
Appendix A.3: Implementation details covering LLM configurations, fine-tuning, and evaluations.
Appendix A.4: The web interface used for human evaluation.
Appendix A.5: Qualitative examples.
10

---

Published as a conference paper at COLM 2025
Ethics Statement
The institution's Institutional Review Board (IRB) approved this project as exempt from ongoing review (IRB ID: HUM00234647). The data collection process between researchers and participants adhered to standard ethical practices. All participants reviewed and signed informed consent forms, which can be provided upon request.
Consent Statement. You are invited to participate in a research study that intends to improve generative AI agents that can follow instructions specified by their human partners to complete tasks. If you agree to be part of the research study, you will be asked to interact with the AI agents to accomplish set of tasks. Some typical tasks include: (1). instructing AI models to navigate in 2D environment or text world; (2). instructing AT models to create images or paragraphs on daily topic; (3). asking AT models to generate instructions for daily tasks. The study will last approximately an hour. The interaction history, i.e., only the text/images generated by Al models and the subjects' language instructions, feedback, and numerical evaluations, will be recorded in a datafile. The data collected in this study will be analyzed and used for research purposes. No personally identifiable information will be stored in the datafile.
Potential Harm. The simulated environment and the tasks assigned to participants were designed and strictly controlled by the research team. This ensured that the potential for safety concerns was minimized, allowing participants to engage with the study with minimal risk. Data collection involved only non-personal information, adhering to standard ethical practices and was used exclusively for research purposes. We ensured confidentiality and privacy, and the data will not be published publicly. Please refer to Appendix A.3.3 for implementation details of our human study.
References
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. Advances in neural information processing systems, 15523366--3736, 2022.
Jinze Bai, Shuai Bai, Shusheng Yang. Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint ar Xiv:2300 21i::2308.12966, 2023.
Shuai Bai, Kegin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-v1 technical report. ar Xiv preprint ar arXit:2502.13923, 2025.
Yuwei Bao, Keunwoo Peter Yu, Yichi Zhang, Shane Storks, Itamar Bar-Yossef, Alexander De La Iglesia, Megan Su, Xiao Lin Zheng, and Joyce Chai. Can foundation models watch, talk and guide you step by step make a cake? ar preprint xrXiv:2311.00738, 2023.
Yevgen Chebotar, Ankur Handa, Viktor Makoviychuk, Miles Macklin, Jan Issac, Nathan Ratliff, and Dieter Fox. Closing the sim-to-real loop: Adapting simulation randomization with real world experience. In 2019 International Conference on Robotics and Automation (ICRA), pp. 8973-8979. IEEE, 2019.
Joya Chen, Zhaoyang Lv, Shiwei Wu, Kevin Qinghong Lin, Chenan Song, Difei Gao, Jia-Wei Liu, Ziteng Gao, Dongxing Mao, and Mike Zheng Shou. Videollm-online: Online video large language model for streaming video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18407-18418 2024.
Sanjiban Choudhury, Mohak Bhardwaj, Sankalp Arora, Ashish Kapoor, Gireeja Ranade, Sebastian Scherer, and Debadeepta Dey. Data-driven planning via imitation learning. The International Journal of Robotics Research, 37(13-14);1632-1672 2018.
11

---

Published as a conference paper at COLM 2025
Aidan Clark, Alex Paino, Jacob Menick, and et al. Hello gpt-4o. httpss//opemai.co///neeeeehllllt gpt-4o/, 2023.
Yinpei Dai, Jayjun Lee, Nima Fazeli, and Joyce Chai. Racer: Rich language-guidde failure recovery policies for imitation learning. ar Xiv preprint ar 4417: 2024a.
Yinpei Dai, Run Peng, Sikai Li, and Joyce Chai. Think, act, and ask: Open-world interactive personalized robot navigation. In 2024 TEFF International Conference on Robolics and Automation (ICRA), pp. 3296-339600 IEEE, 2024b.
Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, al. The epic-kitchens dataset: Collection, challenges and baselines. JEEE Transactions on Pattern Analysis and Machine Intelligence, 43(11)::125-47)) 4141, 2020.
Tri Dao. FlashAttentionn- Faster attention with better parallelism and work partitioning. In International Conference on Learning Representations (ICLR), 2024.
Sam Davidson, Salvatore Romeo, Raphael Shu, James Gung, Arshit Gupta, Saab Mansour, and Yi Zhang. User simulation with large language models for evaluating task-oriented dialogue. ar Xiv preprint ar Xiv::20091133 13233333,,,,, 2023.
Matt Deitke, Winson Han, Alvaro Herrasti, Aniruddha Kembhavi, Eric Kolve, Roozbeh Mottaghi, Jordi Salvador, Dustin Schwenk, Eli VanderBilt, Matthew Wallingford, et al. Robothor: An open simulation-to-rest embodied ai platform. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3164-3174, 2020.
Matt Deitke, Eli VanderBilt, Alvaro Herrasti, Luca Weihs, Kiana Ehsani, Jordi Salvador, Winson Han, Eric Kolve, Aniruddha Kembhavi, and Roozbeh Mottaghi. Procthor: Large- scale embodied ai using procedural generation. Advances in Neural Information Processing Systems, 35:5982-5999, 2022.
Kiana Ehsani, Tanmay Gupta, Rose Hendrix, Jordi Salvador, Luca Weihs, Kuo-Hao Zeng, Kunal Pratap Singh, Yejin Kim, Winson Han, Alvaro Herrasti, et al. Spoc: Imitating shortest paths in simulation enables effective navigation and manipulation in the real world. Xiv preprint ar Xix:2212.002966 2023.
Chuang Gan, Jeremy Schwartz, Seth Alter, Damian Mrowca, Martin Schrimpf, James Traer, Julian De Freitas, Jonas Kubilius, Abhishek Bhandwaldar, Nick Haber, et al. Three- dworld: A platform for interactive multi-modal physical simulation. arvivv preprint arXiv:2007.04954, 2020.
Qiaozi Gao, Govind Thattai, Suhaila Shakiah, Xiaofeng Gao, Shreyas Pansare, Vasu Sharma, Gaurav Sukhatme, Hangjie Shi, Bofei Yang, Desheng Zhang, et al. Alexa arena: A user- centric interactive platform for embodied ai. Advances in Neural Information Processing Systems,,6:1191000 36:1170--19999 2023.
Xiaofeng Gao, Qiaozi Gao, Ran Gong, Kaixiang Lin, Govind Thattai, and S Sukhatme. Dialfred: Dialogue-enabled agents for embodied instruction following. IEEE Robotics and Automation Letters, 744:10044-10055 2022.
Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understand- ing across millions of tokens of context. arXiv preprint ar arXiv:2403.05530 2024.
Kristen Grauman, Andrew Westbury, Eugene Byrne Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 18995-19012, 2022.
Taicheng Guo, Xiuying Chen, Yaqi Wang, Ruidi Chang, Shichao Pei, Nitesh V Chawla, Olaf Wiest, and Xiangliang Zhang. Large language model based multi-agents: survey of progress and challenges. In Proceedings of the Thirty-Thire International Joint Conference on Artificial Intelligence, pp. 8048-800 8057, 2024.
12

---

Published as a conference paper at COLM 2025
Pin-Lun Hsu, Yun Dai, Vignesh Kothapalli, Qingquan Song, Shao Tang, Siyu Zhu, Steven Shimizu, Shivam Sahni, Haowen Ning, and Yanning Chen. Liger kemel: Efficient triton kernels for Ilm training. arXiv preprint ar Xf:::2240011988, 2024. URL https::/arxii..om abs/2410. 111/2411.109988 10989.
Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.
Yifei Huang, Guo Chen, Jilan Xu, Mingfang Zhang, Lijin Yang, Baoqi Pei, Hongjie Zhang, Lu Dong, Yali Wang, Limin Wang, et al. Egoexolearn: A dataset for bridging asynchronous ego-and exo-centric view of procedural activities in real world. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22072-220868 2024.
Ahmed Hussein, Mohamed Medhat Gaber, Eyad Elyan, and Chrisina Jayne. Imitation learning: A survey of learning methods. ACM Computing Surveys (CSUR), 50(2):1-35, 2017.
Muhammet Ilaslan, Chenan Song, Joya Chen, Difei Gao, Weixian Lei, Qianli Xu, Joo Lim, and Mike Shou. Gazevqa: video question answering dataset for multiview eye-gaze task-oriented collaborations. In Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, pp. 10462-10479, 2023.
Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Matt Deitke, Kiana Ehsani, Daniel Gordon, Yuke Zhu, et al. Ai2-thor: An interactive 3d environment for visual ai. Xiv preprint ar X1i::1712.05474, 2017.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Shih-Po Lee, Zijia Lu, Zekun Zhang, Minh Hoai, and Ehsan Elhamifar. Error detection in egocentric procedural task videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. (8655-18666, 2024.
Teesid Leelasawassuk, Dima Damen, and Walterio Mayol-Cuevas. Automated capture and delivery of assistive task guidance with an eyewear computer: the glaciar system. In Proceedings of the 8th Augmented Human International Conference, pp. 1-9, 2017.
Chengshu Li, Fei Xia, Roberto Martin-Martin, Michael Lingelbach, Sanjana Srivastava, Bokui Shen, Kent Vainio, Cem Gokmen, Gokul Dharan, Tanish Jain, et al. igibson 2.0: Object-centric simulation for robot learning of everyday household tasks. arXiv preprint ar arXiv:2108.00322, 2021.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024.
Yao Lu and Walterio Mayol-Cuevas. Higs: Hand interaction guidance system. In 2019 IEEE international symposium mixed and augmented reality adjunct (ISMAR-Adfunct), pp. 376-381. IEEE, 2019.
Ruipu Luo, Ziwang Zhao, Min Yang, Junwei Dong, Da Li, Pengcheng Lu, Tao Wang, Linmei Hu, Minghui Qiu, and Zhongyu Wei. Valley: Video assistant with large language model enhanced ability. ar Xiv preprint ar arXiv:2306.07207, 2023.
Xiang Luo, Zhiwen Tang, Jin Wang, and Xuejie Zhang. Duetsim: Building user sim- ulator with dual large language models for task-oriented dialogues. arXiv preprint ar arXi::2405.13322 2024.
Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video- chatgpt: Towards detailed video understanding via large vision and language models. ar Xiv preprint arXit:2306.05422 2023.
13

---

Published as a conference paper at COLM 2025
Simra Nazim, Saba Firdous, Swaroop Ramaswamy Pillai, and Vinod Kumar Shukla. Smart glasses: a visual assistant for the blind. In 2022 International Mobile and Embedded Technology Conference (MECON), Pp. 621-626 IEEE, 2022.
Jennifer Ockerman and Amy Pritchett. A review and reappraisal of task guidance: Aiding workers in procedure following. International Journal of Cognitive Ergonomics, 4(3):191-212, 2000.
Jennifer J Ockerman and Amy R Pritchett. Preliminary investigation of wearable computers for task guidance in aircraft inspection. In Digest of Papers. Second International Symposium on Wearable Computers (Cat. No. 98EX215), pp. 33-40. IEEE, 1998.
Hao Ouyang, Zifan Shi, Chenyang Lei, Ka Lung Law, and Qifeng Chen. Neural camera sim- ulators. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 7700-7709, 2021.
Georgios Pantazopoulos, Malvina Nikandrou, Amit Parekh, Bhathiya Hemanthage, Arash Eshghi, Ioannis Konstas, Verena Rieser, Oliver Lemon, and Alessandro Suglia. Multitask multimodal prompted training for interactive embodied task completion. In Proceedings of the 2023 Conference on Empirical Methods Natural Language Processing, pp. 768-789, 2023.
Joon Sung Park, Joseph O'Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th annual acm symposium on user interface software and techmology, pp- 1-22, 2023.
Rohith Peddi, Shivvrat Arya, Bharath Challa, Likhitha Pallapothula, Akshay Vyas, Jikai Wang, Qifan Zhang, Vasundhara Komaragiri, Eric Ragan, Nicholas Ruozzi, et al. Cap- taincook4d: A dataset for understanding errors in procedural activities. arXiv preprint arXiv:2312.14555 2023.
Daniel Philipov, Vardhan Dongre, Gokhan Tur, and Dilek Hakkani-Tür. Simulating user agents for embodied conversational-ai. arXiv preprint ar arXiv:2410.23533 2024.
Chiara Plizzari, Gabriele Goletto, Antonino Furnari, Siddhant Bansal, Francesco Ragusa, Giovanni Maria Farinella, Dima Damen, and Tatiana Tommasi. An outlook into the future of egocentric vision. International Journal of Computer Vision, 132((1))488004928 2024.
Xavier Puig, Kevin Ra, Marko Boben, Jiaman Li, Tingwu Wang, Sanja Fidler, and Antonio Torralba. Virtualhome: Simulating household activities via programs. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8494-8502, 2018.
Xavier Puig, Eric Undersander, Andrew Szot, Mikael Dallaire Cote, Tsung-Yen Yang, Ruslan Partsey, Ruta Desai, Alexander Clegg, Michal Hlavac, So Yeon Min, et al. Habitat 3.0: A co-habitat for humans, avatars, and robots. In The Twelfth International Conference on Learning Representations, 2024.
Arvin Christopher C Reyes, Neil Patrick A Del Gallego, and Jordan Aiko P Deja. Mixed reality guidance system for motherboard assembly using tangible augmented reality. In Proceedings of the 2020 4th International Conference Virtual and Augmented Reality Simulations, pp. 1-6, 2020.
Ayaka Sato, Keita Watanabe, and Jun Rekimoto. Mimicook: a cooking assistant system with situated guidance. In Proceedings of the 8th international conference on tangible, embedded and embodied interaction, pp. 121-124, 2014.
Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, et al. Habitat: A platform for embodied ai research. In Proceedings of the IEEE/CVE international conference on computer vision, pp. 1339-99344 2019.
14

---

Published as a conference paper at COLM 2025
Fadime Sener, Dibyadip Chatterjee, Daniel Shelepov, Kun He, Dipika Singhania, Robert Wang, and Angela Yao. Assembly101: A large-scale multi-view video dataset for un- derstanding procedural activities. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 21096-21 2022.
Murray Shanahan, Kyle McDonell, and Laria Reynolds. Role play with large language models. Nature, 623((987):493-44) 4994499, 2023.
Bokui Shen, Fei Xia, Chengshu Li, Roberto Martin-Martin, Linxi Fan, Guanzhi Wang, Claudia Pérez-D' Verez-D'Arpino, Shyamal Buch, Sanjana Srivastava, Lyne Ichapmi, et al. igibson 1.0: A simulation environment for interactive tasks in large realistic scenes. In 2021 [EEE/RS] International Conference Intelligent Robots and Systems (IROS), pp. IEEE, 2021.
Andrew Szot, Alex Clegg, Eric Undersander, Erik Wijmans, Yili Zhao, John Turner, Noah Maestre, Mustafa Mukadam, Devendra Chaplot, Oleksandr Maksymets, Aaron Gokaslan, Vladimir Vondrus, Sameer Dharur, Franziska Meier, Wojciech Galuba, Angel Chang, Zsolt Kira, Vladlen Koltun, Jitendra Malik, Manolis Savva, and Dhruv Batra. Habitat 2.0: Training home assistants to rearrange their habitat. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojeiech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In 2017 LEEE/RS] international conference on intelligent robots and systems (IROS), pp. 23-30. IEFE, 2017.
Bo-Hsiang Tseng, Yinpei Dai, Florian Kreyssig, and Bill Byrne. Transferable dialogue systems and user simulators. In Proceedings of the 59th Anmual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 152-166, 2021.
Ethan Waisberg, Joshua Ong, Mouayad Masalkhi, Nasif Zaman, Prithul Sarker, Andrew G Lee, and Alireza Tavakkoli. Meta smart glasses-large language models and the future for assistive glasses for individuals with vision impairments. Eye, 38(6) 1038, 2024.
Noah Wang, Zy Peng, Haoran Que, Jiaheng Liu, Wangchunshu Zhou, Yuhan Wu, Hongcheng Guo, Ruitong Gan, Zehao Ni, Jian Yang, et al. Rolellm: Benchmarking, eliciting, and enhancing role-playing abilities of large language models. In Findings of the Association for Computational Linguistics ACL 2024, pp. 14743-14777, 2024a.
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arviv preprint arXiv:2409.12 101.299.99111, 2024b.
Xin Wang, Taein Kwon, Mahdi Rad, Bowen Pan, Ishani Chakraborty, Sean Andrist, Dan Bohus, Ashley Feniello, Bugra Tekin, Felipe Vieira Frujeri, et al. Holoassist: an egocentric human interaction dataset for interactive ai assistants in the real world. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 20270-20281, 2023.
Xuan Wang, SK Ong, and Andrew Yeh-Ching Nee. Multi-modal augmente--rellity assembly guidance based on bare-hand interface. Advanced Engineering Informatics, 30(3):406-47) 2016.
Benita Wong, Joya Chen, You Wu, Stan Weixian Lei, Dongxing Mao, Difei Gao, and Mike Zheng Shou. Assistq: Affordance-centric question-driven task completion for egocentric assistant. In European Conference on Computer Vision, pp. 485-501. Springer, 2022.
Jiajun Xi, Yinong He, Jianing Yang, Yinpei Dai, and Joyce Chai. Teaching embodied reinforce- ment learning agents: Informativeness and diversity of language use. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Pp. 4097-4114, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/ v1 20004..mmlp-main.255 URL https: /aac:://cllaana www.ahooooooyyycoooooooooooooooe org 2224. cemnl-mmain 237/
15

---

Published as a conference paper at COLM 2025
Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. Set- of-mark prompting unleashes extraordinary visual grounding in gpt-4v. arXiv preprint (rXiv:2310.11441 2023.
Yue Yang, Fan-Yun Sun, Luca Weihs, Eli VanderBilt, Alvaro Herrasti, Winson Han, Jiajun Wu, Nick Haber, Ranjay Krishna, Lingjie Liu, et al. Holodeck: Language guided generation of 3d embodied ai environments. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, Pp. 16227-16237, 2024.
Yichi Zhang, Jianing Yang, Keunwoo Yu, Yinpei Dai, Shane Storks, Yuwei Bao, Jiayi Pan, Nikhil Devraj, Ziqiao Ma, and Joyce Chai. Seagull: An embodied agent for instruction following through situated dialog. Alexa Prize SimBot Challenge Proceedings, 2023.
Yaowei Zheng, Richong Zhang, Junhao Zhang, YeYanhan YeYanhan, and Zheyan Luo. Llamafactory: Unified efficient fine-tuning of 100+ language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), pp. 400-410, 2024.
A Appendix
A.1 Prompts for Oracle Assistant and Simulated User Implementation
We provide the prompts used for prompting the MLLM to implement the oracle assistant and simulated user agents. Each agent operates using a combination of static system prompt and dynamic interaction prompt. The system prompt defines the agent's role, operational rules, capabilitics, expected output format, and representative output examples. The interaction prompt, updated at each step, provides the situational context, primarily the current visual observation with object information (Figure 3) and the ongoing interaction history (List 2).
For the oracle assistant, we employ three distinct system prompts tailored to different scenarios: one for proactive decision-making and guidance generation (List 2), another handles responses to user questions (similar to the first but omitting the decision maaisig section), and third manages mistake correction after user action fails (List 3). The corresponding interaction prompt for the oracle (List 4) integrates relevant task knowledge, current environmental observations, and the interaction history.
The simulated user is prompted similarly, with its system prompt (List 5) specifying its persona, available action schema, and general behavioral guidelines. Its interaction prompt (List 6) contains the assigned task goal along with the interaction context accessible to the user (history and partial observations). We tested two primary personas (List 7): 'Alice' (less talkative) and 'Bob' (more talkative). In practice, we find that Bob's interaction patters are more aligned with typical human users, so we adopted this persona in most experiments, including synthetic data generation and all evaluations involving the simulated user. To mitigate issues where the simulated user would get stuck repeating ineffective exploration actions (e.g,, rotation back and forth), we introduced simple heuristic: if three consecutive rotation or movement actions occur without task progress, a search_note tip is added to its prompt, encouraging change in strategy. This proved effective in enhancing the simulated user's 's exploration capabilities.
Assistant: "How can help today?' [User Action 1] say: "I need to find lever for the fusebox. Where is it?" table in the robotics lab. Let's there to find it."
Assistant: 'The lever
[User Action 2] go to the robotics lab
Assistant: "The lever you
right front of you table. You it now."
[User Action 3] pick up lever_139 (lever)
Assistant: "Good job picking up
Now, let's
room install
fusebox.
[user Action 4] to the break roon
[User Action 5] turn right 98 degrees
[User Action 6] turn right 98 degrees
Assistant: "The fusebox should be an the wall neartyy Look around for a white netal box with a yellow lightning warning sign."
[User Action 7] turn right 98 degrees
8] go to Fusebox_628 (fuse box)
16

---

Published as a conference paper at COLM 2025
Assistant: "You're in of
Open place inside to precondition not net [User Action 9] Attempted to pick m15ebox_626 (fuse box) but FAILED: Invalid action: invalid target
[Your Last Message] Assistant: "The fusebox cannot be picked it fixed the wall. Please open the fusebox instead to ettach the lever."
[Last User Action] open Fusebox_620 (fuse box)
[ourrent User Statel In break roon at (-20.4, 18.1) facing north
holding: lever_138
Listing 1: Example interaction history from the assistant side. Note: The simulated user's history post-processed to omit specific object type details from object names and IDs.
helping complete You observe their perspective, respond to questions, and provide timely guidance.
environment from their
*# Context
Interaction history
User observation with highlighted objects (calored outlines index numbers)
Object metadata in the format: [index] object_id (object_type): distance in meters (relative position)
Knowledge about the environnent and objects
General Information
Rooms: break roon; main office; manager office; warehouse; reception area; robotics lab; quantum lab
User Actions:
Navigation: GotoRoon, GotoObject, Move, Rotate, Looküp, LookDown
Interaction: Pickup, PlaceTo, Open, Close, Toggle, PourTo, Break, ThrowTo
Cross-roon Navigation Rules:
Simply tell users which room go
NEVER detailed directions about how
specific
Interaction Notes:
understand basic toggle operations Direct users to use "Toggle" on target objects if they specifically ask how
Toggle controls machines, buttons, levers,
Knowledge
Familiar with common objects (tables, bowls, etc.)
Need guidance about unconnan objects' appearance, Function, and usage
Objects:
Object types and states (turned on/off, open/closed, etc.) are provided in the object netadata, interacts with the object knowledge guide user. Distinguish between nachines and their monitors (e.g. enbiggenator vs. embiggenator nonitor)
Object states updated when
** Guidelines
**# Environnent and Task Awareness
1. Base all reasoning strictly on provided context. Do not assume environment details until observed. of object before seeing unless explicitly stated your
knowledge.
example, If the user asks about the location of object and successfully finds it following guidance, you should ask "what da you want to do with it?" instead of suggesting next steps.
Only use provided knowledge far guidance,
guessing
"how change color of object", you should your knowledge guide them the color changer, but not to guess which color they want to change to.
If the
#d# Comunication Decisions
Respond when:
User asks questions provide simple, concise, conversational instructions
proactive when:
The task goal is unclear. Ask the user about the goal. Never guess the task goal. user completed nentioned goal.
follow-up goal. User misidentifies objects or goes to wrong places provide correction and guidance
Target object appears current view notify user
User makes progress (find target etc) offer encouragenent
Inform the user when there is state change (e.g. a nachine is turned on/off, a lever is pulled)
problen observed
prevents the from completing
(e.g., power, nonitor infected, etc)
inform the user and suggest solution
Active gravity flipper visible (state: "turned an") warn user keep away explain it off
Remain silent when:
User perforning expected actions based previous guidance
Your last nessage already contains the information you would repeat
VaN Rules for Responses
friendly, assistant-like, conversational
Reference objects by visual features (color/size/shape), never nention object I0/index.
Rely only on observed environnent state, not knowledge-based assunptions.
Keep responses brief (under words preferred, naximum 58 words)
Break longer responses into multiple nessages (e.g. will guide you step by step. First, let's ...")
your previous nessages, especially object descriptions Use relative positioning (near, far, front, left/right) based on user perspective. Only mention directions for objects you can
repeating information
17

---

Published as a conference paper at COLM 2025
Apologize suggest alternatives your guidance
failed actions.
If hints suggest checking location and the object isn't found, direct users to similar locations in the sane roon.
Renind users that task details appear in the top-left corner if they ask.
NEVER suggest checking on the floor, under tables, or behind objects.
Example Responses
(examples)
Output Fornat
<Think>
Think comprehensively about the task, observation, and history make reasonable decision.
</think>
<Speak>
"(your response)"
N/A (for keeping silent)
</Speals>
output strictly follow guidelines examples above.
Listing 2: Oracle assistant system prompt for proactive guidance generation.
helping users complete
user's fails, you explain why suggest
alternatives.
Conmon Failures and Solutions:
Hovenent Interaction:
"Move forward" fails due to obstacles. Suggest alternative directions to nove. interactions
"out range", advise noving closer target object (e.8- go object)
When the user trind to interact with object that you do not see its 1D in the current view, suggest that the object is not there and to look for it in other places
Object Manipulation:
failed "pick actions:
If holding enother object:
Suggest putting it down on nearby surface (e.g. table, countertop, etc) first container (inactessible):
open container first objects are too large to be picked if enbiggenated. Suggest to turn off the erbiggenator first.
object
For failed "place to" actions:
receptacle full: Recomend alternative locations (tables, shelves, counters)
If not holding anything: Remind the user that they do not have anything to place, and ask what they want to
For containers:
Open/close
on already open/closed
non-openable objects
Device Operations:
containers and devices:
Toggle actions only work on toggleable devices
machines require object inside before toggling
Sone nachines (freeze
laser, gravity flipper, enbiggenator, time portal generator) camnot be directly instead.
toggled. their nonitors contral
Toggling the laser monitor may fail due to various reasons, including laser not assenbled, nonitor infected, unpowered, Reason based on the situation and knowledge to suggest solutions.
"Object Knowledge" below for
details about how to interact those
objects
Toggle fails when computer is infected by a virus
cannat toggled directly. Toggle
inside instead.
Objects night be inaccessible in the robatics lab due to box blocking the way. If so, suggest taggling the robotic arm to renove the box.
Important Notes:
when judging object interactions breakable itens. Before making the suggestion, exanine the interaction history carefully to understand the user's intent. following the rules blindly.
Adjust suggestion based on user's
Be consisent with the dialogue history
Keep responses brief (under words preferred, maximum 50 words)
Example responses:
"Your
full. Please "The object is out of range. Please nove closer to the object.."
object your first."
abject already open.
open again."
"The object is not breakable. You can't break it." "Oops, looks like computer infected by a virus. need to disinfect it first before using it. Let's "You do not have anything to place. What you trying da here?"
"Looks like the object cannot be placed here. Sorry about that! Let's try to find another empty table to put object."
Output
18

---

Published as a conference paper at COLM 2025
<Think>
step-by-step reasoning about the failure rules above, object knowledge and common sense. </Think>
<Speak>
"..."
</Speak>>
Listing 3: Oracle assistant system prompt for mistake correction.
Knowledge
(knowledge)
Context
*aCurrent Observationee
See image. Object metadata ([index] object_id (objact__type): distance (relative position) colors states): (object_list)
Note:
Examine carefully at the object metadata to understand the objacts, including their types, states, and knowledge provide *naccuratenn guidance. Do not mention the object ID or index in your response. Use natural reference.
relative positions
Associate
**Interactions**
(history)
Important Notes:
Speak the user asking question, any condition proactive guidance (e.a., target appears, makes mistakes, deviates from expected actions, good progress, danger presents, state changes, etc) according to the guicelines.
Keep silent when need to DO NOT repeat what you have said before, especially in your last In <Think>, conprehensively analyze the task, current observation, and history step-by-step, until reaching a reasonable decision. Ensure your response strictly follow ANALL*a the guidelines and exarples, especially the environment and task awareness, communication decisions, and rules for responses.
Listing 4: Oracle assistant interaction prompt.
You will play the role of (name), research intern working in futuristic laboratory, Your goal is complete tasks with help from an assistant.
** tharacter Personality (persona)
You observe the world through first-person view
Interactable abjects have colored outlines and index numbers
object metadata fornat: [index] <object_id> (sabject_type>): distance colors
Available roons: brea,_roon, nain_office, manager_office, warehouse, reception_ares, robotics_lab, quantun_led
#d Actions
### Communication
Speak
Talk naturally to assistant
### Movement
nroor_name' Navigate reans
GotoObject robject_id>> à Approach object to interact
heverorward/HooBBaaeeer <1,
RotateRight/Rottteeeet <90, 45>
Looklp/Lookkown <30>
**# Interaction
Pickup cobject_ids
Requires
PlaceTo <object_is>
Place held sten into recoptacle/machime
Open/Close <object_id>
Toggle <object_id>
For buttons, nechines, levers
PourTo <object_id>
Pour contents of held iten
<object_io>
held Throw held iten
ThrowTo sebject_id>
#* Response Fornat
<Think>
Reason comprehensively about the current situation to decide what to </Think>
<Action:
action_type type action_argunent
19

---

Published as a conference paper at COLM 2025
</Action>
Guidelines
V44 Think Like [name)
<Think> simulate (nane)'s decision based their persanality
2. Think step-by-step comprehnsiveel:
Sumariize your progress
recent actions
If there is assistant nessage, use it to help you nake decision.
However, you should not blindly follow assistant's nessage.
Analyze whether that aligns with task goal. If goal, or ask for clarification.
Inform the assistant about task
Carefully examine the current observation, especially the object netadata.
Re-examine all the previous context, and the guidelines, to nake decision. Ensure that your action is guidelines,
compliant
reasonable under the React naturally explore when needed, ask for help when uncertain
Avoid repeating actions (except rotation to complete full viex). nothing happens after action, consider ask assistant for help.
Understand objects based an the visual inage, and the correspanding object netadata associated with each indexed object.
Use object__id (not index) for specify targets
Follow physical logic (e.g., open containers before accessing contents)
Must within 2.8 neters
GotoObject before interacting Can interact while holding itens
Goto@bject when target visible its is known from history 5. Use Gotofoon for room-to-roon navigation
actions only roon exploration
8. Can hold only one item at time
Use PlaceTo for inserting items into machines and pressing buttons
10. Do not Looklp/Down twice
11. Do not repeat go to the same object twice
atterpt perform dangerous actions nentioned the assistant
Strategy
Analyze visible objects carefully using netadata and appearance Start search by rotating around before noving. Rotate effectively (98 degrees, same direction) to cover the whole first
3. If not see the target, try to nove to the other side of the roon move forward when there is an open space ahead
4. Do not attempt to nove forward when blocked by walls or objects place (e-g- tables) places
target is not Machine nonitors in quentun 1ab and robotics lab often close to their corresponding nachines, and vice versa. Try rotating to find then.
Try
### Natural Speech
speaking 2. Use natural references (names, descriptions, locations)
mention ebject
unfaniliar objects (e.g., Don 't ask about common objects (e.g., "What does plate look like?")
questions
color changer like?")
Ask task-relevant questions about what to do next, or where to find something
occasionally give short confirnation acknovledgement "Cot it")
assistant provides information ge.g., "Ok",
nention room nanes (without "_")
Examples
Example (ask for help)
<Think>
to peel an apple. in roon now, see any apple here. Besides, what tools are relevant to this task. Maybe should ask the assistant for help. </thinke>
<Action>
"My task
apple. Please help"
</Action>
Example 2 (look for an object)
<Think>
My task is to turn off KON-9eea, According to the assistant, XON-9008 is nachine with a red light on it, already arrived the left 9e degrees. There are serveral machines and computers in my current view: nuchine 325 computer_327 (enbiggenator nonitar), nachine_328 does not say anything at the nonent. Maybe should rotate one nore time to check other side of the roon. </Think>
located
robotics
retated lab started search, (color changer),
maker), nane
XON-9820.
assistant
<Action>
RotateLeft
20

---

Published as a conference paper at COLM 2025
</Action
Listing 5: Simulated user system prompt
Vd Persona (persona)
Vi Task Goal: (goal)
Interaction Context (action_history)
(curr_state)
gussistant_nessaee
Current Observation: see the inage. Object netatita (Lindex] cobject_id> (sobject_type)): distance neters colars (object_list)
Note: Objects not listed are either not visible, too far to see clearly.
Important Notes:
Strictly follow all the rules and guidelines above, and correctly fornat your response. In <Think>, comprehensively analyze the task, step-by-step, until reaching reasonable decision. Ensure following the iuidelines and examples. Strategy" very carefully if you looking for object.
abservation and assistant nessage, and history
Exanine "Action Rules" and Talk appropriately according the persona. (search_note)
Listing 6: Simulated user interaction prompt.
Alice: Efficiency-driven
Start with describing the task to the assistant and ask for help
the appearance uncommon, daily used objects
Do not ask the appearance of objects that daily used (e.g. bowl, trophy etc.)
Responsive to the assistant's instructions
Use very short sentences
asking question confirmation Communicates efficiently with short sentences Responsive to the assistant's question
Acknowledge with "ok' 'got it' occasionally (when receiving instructions)
Ask 'wheee is ... 7' assistant mentions seeing something but you don't see
wwich one?' if there is miltiple chaices
Ask "hyy
if the assistant mentions an object that does not seens relevant provide
'how ...
assistant
details
Ask 'what is
'what does
look like?' if the object is asuncommente and the assistant does nat
drovide details
Listing 7: Example of user personas.
A.2 Additional Details on Alexa Arena
We provide more details about supported actions, tasks and knoweldge descriptions for the Alexa Arena environment.
A.2.1 Actions
The action space of the Alexa Arena environment consists of 16 actions. Among them, there are 6 low-level navigation actions: MoveForward, MoveBackward, RotateLeft, RotateRight, LookUp, LookDown, where an argument supported together to specify how far to move or how much to rotate, etc. There are also 2 high-level navigation actions: GoToRoom that directly go to one of the rooms (Break Room, Robotics Lab, Quantum Lab, Main Office, Manager's 's Office, Reception, and Warehouse), and GoToObject that navigates to a specific object in the current view of the agent. The agent can perform 8 manipulation actions: PickUp, Place, Open, Close, Toggle, Break, Pour, and Throw, which also needs to select target object from the current view as an argument. Actions only take effect when certain preconditions met, such as the agent being close enough to the target object or the target
21

---

Published as a conference paper at COLM 2025
object being in specific state. After action is executed in the environment, the agent can receive a descriptive feedback message that indicates the result of the action, indicating whether the action successful or not, and why it failed.
A.2.2 Tasks
Alexa Arena incorporates fantastical objects with diverse state changes that enable rich, complex interactions. These objects support 14 different properties (e-g,, pickupable, open- able, breakable, toggleable, heatable) and include unique features like color changers that modify object colors, time machines that rewind object states to repair broken items, and em- bigenators that enlarge normalize objects. Since users may not know what these objects look like or how to use them, it is essential for the assistant to provide guidance about their appearance and functionalities. Additionally, the environment introduces safety-critical scenarios where irreversible actions-such as approaching an activated gravity flipper-can harm the user avatar and end the task. These dangers require proactive warnings to ensure user safety and test the effectiveness of assistance.
The environment further supports tasks with complex dependencies and diversified pre- conditions, creating intricate challenges. For example, changing an object's color involves locating the object, finding the color changer, and activating it, but can be complicated by factors like enlarged objects that cannot be picked up unpowered rooms where devices remain inactive. Such conditions necessitate that assistants adjust their guidance dynami- cally based on real-time environmenta observations. In total, we implement 22 tasks with 10 preconditions, yielding unique task variants, with each task potentially supporting multiple parameters, such as the target object to heat the color to change object to.
Moreover, the environment is programmatically configurable, enabling the creation of numerous tasks with varying preconditions, constraints, and differences in room layout, object location, and initial states. This flexibility supports the generation of diverse training scenarios and controlled evaluation conditions, detailed in List
assemblee_fssbbr Birtie__fuusoo___Baaeeooooo Breeakooon_ssippl
1. assemble Laaaa__eseee_CCorrroaaeee...
break___Bol__1__siipl...
3. Bow__oo
5. Baaok___oofeeeu__ossssc.c
18. oealbo__DDesFaan_eeee-------
Martt__ denbbgggenn
12. Nrrt__Narrhhuse__andii
13. dart__.arrhouse__cimi..
14.
15. dneabiggen_DDrtt_poom
deetbiggen__Deekaan.eeee.......
17. Satprfect_ccmpuuee___MMllll
Gravity__simpil
eabiigen__BannnaBuneii
19. harpe__Coooteee__nttto________ Bro_eeen___dddinng
28. 1ta.ge__Deedaaa...ooeee.._1.1_..
21. embiggen__Radio.ev
22. feei_dinoo_Apple__sime
feed__dino__.andyaarr..
24. feed _ar__diio____r_aa____ de__bignng
26. www.trroeryy_etnn_ddiiiffffeee.
27. tiee
28. fire_frreeerry_Hoo___rreee blocker
29. Sire_freezera__Hoon__stttt
rre_ffreeeeray_tocco____. power
31. treezzeray_toccool.. Caael___ce________ renoee blocker
32.
[etrervwy_ooccl____CCnatteeeeea @1__disinfect_coppuer
33. Aire_freezrryy_ooccol__ htw.cccooo_..ooeeoooo._...._....
34. Air__laser_.one__assetttt
35. Laaser__Noeedddiiifeeet.....
36. Fire_laser_NNom__paaw
fire_Laser__Nene_reeeae
38. Fire_laser__Mnn__siipp
39. Bire_laser_to_heat__ae
22

---

Published as a conference paper at COLM 2025
Fire_laser_tu paser__to_hede wwt.hett..nSSoaaeeee...cc
41. fire_laser_to_hoa
42. Lllaerrtohhatt_cooffeegggggeee
43. Fire_laser_ tottto_hht_______
fire__asrr_t__het____P hoo.toohttt...oooyeeeeee
44. wtl.oopptee__.eeppptoooooooooppe
45. peteel_ccopurrr__eeeppptnnnnnntt
46. infeet__comuter___.Me V.rnuuter__VMMMttorrroeeee hwt.://ttoo...rooooy..taaannoooa
Soppcct_copputer___MMe wtt..e/nnnnttrravvii..oo.o. proe
48 C__tooppuuurr__MMMttttto denrr
43. Landing..LLser_iip_____
50. Bake_cearrot__Bnnna..3
51 make_carrot__CanSddaleee Crushed__1__
52 galke_caroot_lonn_Pentttttttt ber.tttrrr____dddenniieeeee
53 aakee..arre__PPPiieer.rtrrrteeee
power_ poerr__raaRoon_assmeii
55. power Sioer__Smallffffiee__siip
56 print friunt__ffgurre__poer
57 pr print Btwpt______ue___________ Fii________e__sipppl
58 pr int dembeeee
58 print__in_aa htt____amr___aadding
52 print... lever_..eerrrrreee.... [es_t__lever..rrrrfffmm..aa
print_ Lia_n___nu__reeooe__ueee
62. [tmevee_lockkr___ab____10
63. remove
64 reppir_brokem_Coffcchhhhee sippl
65. trannsionn_crroot_.nnttt1
power
67 Transforn_carrot__Flpp
unmake_ sarekkecoffeee..e Soreree__oooeeeee__eeeeeee power
69. cateeeecooeeee_.oooeeeeee sippo
Listing 8: 70 scenarios we used for testing. Each scenario is composed of the task name, task
parameters and the task precondition.
A.2.3 Knowledge
We provide three types of knowledge that empower the assistant to offer effective guidance:
Task Knowledge:
This encompasses the general steps required for task completion. For example, guidance on how to repair object using the time machine might be: "1. Place the broken object into the time machine; 2. Close the door and toggle the time machine."
Environment Knowledge: This covers episode-specific details, such object locations and states. For instance: "The mug is on the green table in the main office."
Object Knowledge: This includes information about functionality, usage, appearance, and typical locations of task-relevant, uncommon objects. See List 9 for illustrative examples.
Printer]
Function: 30 print objects from cartridges.
Usage: Insert the appropriate cartridge into the printer and toggle the printer on.
where: Robotics lab.
Acpearance: grey nachine with square, light blue printing bed and black PC case the side.
[Laser Cannon]
Function: Fires laser bean heat up objects.
Usage: Laser be assenbled with the control panel fire. check, the laser and see whether the control panel is visible. If not, find and place the control panal on the laser. To fire the laser, toggle the lab door
nonitor the robotics lab.
objects, they should placed the wall shelf by the robotics
Robotics
Appearance: big machine with grey base, an orange nozzle and power cable.
[Laser Monitor]
Function: Operate the laser to fire laser bean. Toggle the firing.
fire
monitor Infected, renove first before
Robotics
Appearance:
red computer with the label 'LASER CANON' top.
[Tine Machine]
Function: Convert objects to their previaus state. Known uses: Fix broken objects. 2. Recover transformed
their original objects.
Usage: Place the object inside the time nothine and toggle it on. Need to open the door to before placing to it. door toggling
open
pick reverted object.
23

---

Published as a conference paper at COLM 2025
Where: Break counter top.
Appeorance: square nachine with red frane and a silver controller on the top.
[Gravity Flipper]
Function: Generates an anti-gravity field to nake objects Float.
Usage; when turned objects placed the gravity pad will float. [WARNING] Keep away from the gravity flipper when it is on to avoid gotting hurt, Hust turn it off to access the objects on it.
Where: Quantuu
Appearance: large, octagonal, futuristic table with glowing blue grid pattern on the surface.
[Gravity Flipper Monitor]
Function: Controls the gravity pad to enable or disable its anti anti-gravity effect.
Usage: Toggle the monitor to the gravity pad off, If the nonitor is infected, need the virus first before toggling.
Quantun
Appearance:
computer moni tor with green base and 'Gravity Flipper' label top.
[Carrot Maker]
Function: Transforns objects into corrats.
Usage: Place the object the nachine and toggle it transforn the objact into carrot.
Where: Quantun lab.
Aspearance: tall. grey machine Looks vented panels extending ceiling, carrot" sign.
Listing 9: Examples of object knowledge.
A.3 Experiment Implementation Details
A.3.1 Model Configuration, Fine-tuning and Deployment
A.3.2 Model Configuration, Fine-tuning and Deployment
We utilize the Azure OpenAl services for our GPT models. For GPT-40, we employ the GPT-44-20241122 version with the temperature set to 0.7 and top-p to 0.95 in all experiments. For fine-tuning our assistant model, we adopt a supervised fine-tuning (SFT) approach. The training data is formatted in image-text style where each instance consists of the current visual observation and dialogue history as input. The model is trained to generate either Chain-of-Thought followed by action sequence or direct action prediction as output, which is then parsed into an executable command. To implement this, we employ LoRA (Hu et al., 2022) with a rank of 8, training for 2 epochs with batch size of 32 and a learning rate of le-4 using a cosine decay schedule. Fine-tuning is conducted using LLaMA-Factory (Zheng al., 2024) and accelerated with FlashAttention-22 (Dao, 2024) and the Liger Kernel (Hsu et al., 2024), taking approximately 3 hours on 4 A100 GPUs (80GB). For inference, we deploy the model using vLLM (Kwon al., 2023).
A.3.3 Human Evaluation Setup
We recruited 28 human subjects with no prior experience of the Alexa Arena environment to participate in the evaluation of our assistant models. Prior to the experiment, each participant signed a consent form (available upon request). We organized 70 task scenarios from our test set into 14 groups, each consisting of five distinct tasks that do not overlap in goals or object usage. Each participant was assigned to one group and paired with either the oracle assistant or the trained assistant model, without being informed which one they were interacting with. In total, each of the 14 groups was tested by pair of participants one per assistant type. Each session typically lasted around 45 minutes, and participants received $20 Amazon gift card as compensation.
At the start of the experiment, participants were introduced to the study environment through detailed tutorial covering the environment, task setup, and interface operation (see Appendix A.4.1). Following the tutorial, they completed five sequential sessions. In each session, participants were given high-level task goal without detailed instructions, requiring them to communicate with the assistant to achieve the goal. Once a task was completed, the web interface automatically advanced to the next task until all tasks in the group were finished. Participants had the option to give up at any time if they felt stuck or if the assistant's guidance was ineffective. Additionally, maximum step limit of 50 was
24

---

Published as a conference paper at COLM 2025
enforced to prevent tasks from taking too long, which was also applied in simulated user experiments (see Table 1) for fair comparison.
A.3.4 User Intention Analysis
We perform a user intention analysis to understand the types of questions and statements made by users during the human study (Table 5). We define nine distinct user intentions based on manul review of the collected utterances. The following list outlines these intentions along with representative examples:
inform-goal: Statements declaring objectives.
"I want to fire the laser cannon."
"I need to freeze the cake."
"My task is to break the bowl."
inform-state: Statements providing status information.
"I have picked up."
"I only see a jar and a computer."
"I didn't see the mug printer cartridge."
"It won't let me place the cake it."
ask-object-ooction: Questions about where objects are located.
"Where is the mug?"
"Where is the carrot maker machine?"
eakkobjecc-appearance Questions about what objects look like.
"What does look like?"
"What is carrot maker?"
"Describe the shape of the cartridge."
ask-how-to: Questions about how to perform specific actions.
"How to fire the laser cannon?"
"How to 3D print mug?"
"How do feed the dinosaur?"
ask-next-step: Questions about what to do next.
"What's next?"
"What should do next?"
"Then what?"
"Now?"
ask-confirmation: Questions seeking confirmation.
"Is this the correct machine?"
"Do you see it?"
"Is it still in view?"
"Is it finished?"
ask-other: Other questions that do not fit the above categories.
"How should go there?"
"Is it on my left or right?"
"Why do we need to transform the carrot?"
"Whyy do you believe the computer is in this room?"
social-exchanger Social exchanges or acknowledgments.
-"Hi."
"Okie."
For the annotation process, first employed GPT-4o to pre-annotate user utterances cording to the schema above. We then manually reviewed and corrected any misannotations, which accounted for less than 1% of the total labels.
25

---

Published as a conference paper at COLM 2025
A.3.5 Step-level Assistant Error Analysis
We perform the step-level error analysis on each assistant turn to gain fine-grained understanidng of the error made by the assistants, to better understand the typical errors and performance gap between the oracle and the trained model. During the annotation process, we annotate both Speak Errors where the assistant's guidance is wrong, and Silent Errors where the assistant fails provide a guidance when it is necessary. We further categorize the errors into 10 types, for Speak Errors and 3 for Silent Errors, as explained below:
Speak Errors
Object Misidentification: Incorrectly identifying one object as another (e.g., referring to Object A as "Object B") claiming the presence of an object that is not visible.
Incorrect Action Suggestion: Recommending next step or action that does not align with the current task.
Incorrect Position Description: Providing an inaccurate location for an object (e.g., stating it is the user's left when it is actually on the right).
Object Overlooked: Failing to acknowledge a visible object, instead instructing the user to search for it or asserting that it is not in view.
Unverified Information: Presenting details without confirmation (e.g., discussing task specifics when the assigned task is unknown, or asserting that a computer is powered without verification).
Incorrect Object State: Misidentifying an object's status (e.g,, indicating that powered-off computer is turned on).
Redundant Information: Repeating correct information multiple times in short period, resulting in unnecessary duplication.
Silent Errors
novel object first appears in view. This error is counted once per object each time it re-enters the scene.
Missed Object Notification: Failing to inform the user when
Missed Next-Step Guidance: Not providing the necessary guidance on the next action or on relevant details of a novel object when the user is unaware of them.
Unanswered User Question: Not responding to user's 's question at all, leaving it completely unaddressed.
We manually annotated the all the 140 sessions collected from the human study, resulting in 3806 turns in total. The web interface used for annotation is shown in Appendix A.4.2.
A.4
Web Interface
This section provides an illustration of the web user interface (UI) designed for human study and error analysis.
A.4.1 User Interface Tutorial
The user interface serves as the primary platform for participants to engage in our user study. It allows users to observe the environment, perform physical actions, and communicate with the assistant through an integreated chatbox. We show the tutorial that is provided to the users below for an overview of the interface functionalities.
A.4.2 Assistant Interface
The Assistant Interface (Figure 13) is monitoring page displaying the interaction from the assistant's viewpoint. It includes: Task Selection for choosing tasks within a study group; Screen Recording to capture user activity; Interaction History showing the dia- logue between user and assistant; and Knowledge Display providing access to the task, environment, and object knowledge used by the assistant to provide effective guidance.
26

---

Published as a conference paper at COLM 2025
DEESION
f
OFFER SCORE: 66,000
Room Room
Vaiin Office io poweret
Holding htem
Figure 7: Game Video Introduction (Tutorial 1/6): Your goal is to complete the task assigned to you with the help of an AI assistant. You can move around, interact with objects, and talk to the assistant. The assistant shares the same view as you do, and will provide you with instructions and guidance throughout the task. Mission Goals: Displays the goal of the current task. This only visible to you but not the assistant. Map & Current Room: Shows your position in the world and the current room you are in. Score can be ignored for now. Current Room Status: Indicates the operational status the room, such as whether it is powered empty you are not holding an item. Note: You can only hold one item at a time, so you need to place the current item down before picking up new one.
not. Holding Item: Displays the object you are holding. This area will be
Arena
Nanigation
Oatabbure
Maripdatton
Open
Cless
Ensas
TAX
OFFICE
ScoRe: $8,000
Assitant te yerssey
Bend mossage...
Figure 8: Action Control Panel (Tutorial 2/6): The Action Control panel lets you move around and interact with objects. Use Movement Actions for navigation, and Object Manipulation for actions like picking up or placing items, toggling machines and computers, and opening/closing doors. Hover over the question mark icon for details of each action. Go to Object: Quickly navigate to specific objects in view. Go to Room: Directly move to a specific room. Note: Go to Object/ /ooom/oommmm is much more efficient than moving around manually, so it is recommended to use them whenever possible.
A.4.3 Annotation Interface
We develop an annotation interface for step-level error analysis of assistant performance. Annotators can choose from list of pre-defined labels to annotate each action made by the assistant, including both "speak errors" that there is something wrong with the guidance, and "silent errors" where the assistant keeps silent when there is a need of speaking, If an action has error, then "correct" label should be selected. The knowledge and the full
27

---

Published as a conference paper at COLM 2025
Arena
Annotated Observation
Nasigation
LuGP
VD In
LOOP
Maniputation
Hullup
Toggle
AER
20
13
Evant History
Great
meoiic lco to attah the control pard lear cannen
Do temes
Ase stant
you' ontorel treak
noon inaeedt of the retictis lib.
locor
counta
CDcountar
Tt counto
sobertt
13 cideer
14 lowde
message
Figure 9: Show Observation (Tutorial 3/6): A pop-up window displaying all the interactable objects in the current view will show up after clicking the Show Observation button. It can help you better understand the objects and plan your actions. Objects that are not common in daily life are displayed with their base category (e.g. "machine") instead of their specific type (e.g., what type of machine it is). The assistant may help you identify them if they are relevant to the current task.
Arena
Glee
Show Obsarsstion
Navigation
Outufizees
OufeDrgeet
REALS
Manipulation
Pree
Three
Splu
BREAKROOM
Event History
Place the cancot iin time back the hammmac
Astist
Thoplle time ne on
Invatic acticn preconaition not inald target
Bandi
Figure 10: Event History Intro (Tutorial 4/6): The Event History panel shows: (1) Chat history between you and the assistant; (2) Actions you have taken; (3) Action execution feed- back when an action has failed (starting with [!]); (4) System messages (in grey) indicating the start and end of a task. You can use the input field at the bottom to send a message to the assistant.
interaction history with the generated thought from the assistant is displayed to help the annotator understand what's going on.
28

---

Published as a conference paper at COLM 2025
Arena
Omm
Octobues
Oulideact
Martpulation
Broch
N
DELWA
RECAPIION SCORE: $7.670
cin
Asdirtant
frns lets check thes and taae
Ture rypi dngrees
Send meesage.
Figure 11: Blocking Page Intro (Tutorial 5/6): After you select an action or send message, blocking overlay will be shown while the system processes your request, which normally takes few seconds. You can only continue after the overlay disappears.
Arena
Chr
Stow Oiverveie
Nawigation
Ortsffee
HI GOALS
LOWP
Break
Monpulation
There
ROBOTICS LAB
Event
SCORE: #9.970
Assistant
How can helo you toda??
hext baok dtont seceeds
messagee..
Figure 12: Give Up or Task Completion (Tutorial 6/6): The task will end if (1) you suc- cessfully complete the task; (2) you click the Give Up button; (3) max steps (50 actions) is reached. Note: the assistant is not perfect, so normal that the assistant's 's instructions are not always correct. You may choose to give up a task at any time, when you feel getting stuck and the assistant is not helpful. After a task is finished, the system will automatically advance you to the next task. You will work total of 5 tasks in this session.
We built an Annotation Interface for step-level error analysis of the assistant. Annotators label each assistant action using pre-defined categories, identifying "speak errors" (incorrect guidance), "silent errors" (missed necessary interventions), or marking actions as "correct". The interface shows the full interaction history including actions and the assistant's thought process, and relevant knowledge to support annotation decisions.
29

---

Published as a conference paper at COLM 2025
Arena
Interaction History
Tash strkel
Adsesamt
Faeh diooess
SCORE: 95,578
Card unssagge...
MIW
KEEP SLLENCE
Knowledge
tour Oot
Homalh
togale crner -iaar panta,
Figure 13: Assistant interface for task monitoring and debugging.
Arena
Learch Sume
Scarding
Bun
Oisorvaiion
actton
webe: qoe goicad anpeecnii pibzaoo top Garan
Wiinnn puacton
Redurataa tata
show in oetioca the peeiriad
DELWAN IS MAKING
Manager's Office la Prwernt
It coc but von but cas de cos inr
Masse ruttl
aad
History
neu une Mous unde dher
lngllt
trrst dban pavn can offec
Audieatt
aftee bostre locve
offer to pude the aer Shos bnaenell
the appoars
Ausistat
Figure 14: Annotation interface for assistant error anntation.
A.5 Qualitative Examples
In this section, we present two examples of dosistant-user interactions, highlighting key events that demonstrate the assistant's decision-making
30

---

Published as a conference paper at COLM 2025
A.5.1 Example Interaction Between Oracle Assistant and Human User
Task Setting
Step
Step 2
Tost: Brnst
Eet eechiry theelhy RONGE NNWW
How salee the took uniesoen?
help you todmy?
break the plnte, youfll need lob to retrieve it Use 'GotoRoonn to head quaetum
Tosk knowledger Complctey/
broak
bo niwr to brack ccect
piate?
GotallocmmCuurtumm Lob)
Neew aiveys shoret eeth
Object tnowlldge Compleiny
Step
Step
Eustlis Enarge und seen chartss
The hommer on the emélggann enéiigennttr reams find mochinee with a round bose and cal mountee top. know
ethiinc If the embigg enoter or nomnerr becomes vilille
Alrrectvvl erecn in seeot nor
orggir
yeass Guend lit
cuert provide further osoitonce
Precondition happened! The hommer too lorge hold
Quanium Lob
Guorium
Right
Step
Step 12
Step
CAnken por lisal abon stralooo Frantt the hommer
Schrndd anse I0p
to your right Look for the pirt mon noorbay loboled Embiiggenetor or inci sinw
Fow
great jobr you now have the lets head manager's office
use actercent to go theme.
YAN Roimn
Right ap*
GonoRoom (Manoger Offices
In Quaniam
Quanlam
Office
Figure 15: Illustration of the asymmetric task setting and key events during the interaction, where the oracle assistant provides assistance for the user to normalize a hammer using the embiggenator and use it to break plate.
Figure 15 shows an interaction where the user is tasked with breaking plate, but is initially unaware of the necessary steps or object locations. The oracle assistant possesses complete task and object knowledge and provides step-by-step guidance:
The assistant initiates by asking, "How can I help you today?" When the user re- sponds, "How do I break the plate?" the assistant confirms the task and begins reasoning.
Step 1:
Step 2: Concluding that hammer is required, the assistant directs the to the Quantum Lab to retrieve it.
Step 3: In the Quantum Lab, the assistant describes the embbiggenator-a tool unfamiliar to the user-by noting its appearance (a grey machine with a round base and ceiling- mounted top), helping the user identify it.
Step 4: The assistant then intentionally remains silent to avoid redundant guidance, monitoring the user's progress.
Step 6: Observing the user near the embbiggenator and hammer, the assistant notices that the hammar has been embiggenated and thus too large for the user to pick up. The assistant then instructs the user toggle the embbiggenator resize the hammer before picking it up.
Step 12 - 15: After several additional steps, the user retrieves the hammer, is praised by the assistant, and subsequently guided the manager's office to break the plate, successfully completing the task.
Overall, the example illustrates how the oracle assistant effectively bridges the user's knowledge gap through adaptive, contextual, and timely guidance.
A.6 Example Interaction Between Trained Assistant Model and Human User
Figure 16 illustrates an interaction with our trained model assistant where we show two common errors observed in our experiments (also seen with oracle assistants):
Step 5: The user seeks robotic arm to remove obstacle. Although the robotic arm is clearly visible on the left, the assistant overlooks it and advises the user to search the area. This typical perception error leads to inefficient navigation and additional steps.
31

---

Published as a conference paper at COLM 2025
Step 8: After the obstacle is cleared, the assistant misidentifies the object as the freeze ray monitor. Despite this error, the user relies on previously learned visual cues to correctly locate the actual freeze ray monitor. This demonstrates how human memory and situational reasoning capability can compensate for the assistant's perceptual limitations.
32

---

Published as a conference paper at COLM 2025
Setting
Step
Step
Tral: for the trous noy Oijject knowledge unienown
How can help today?
freeze freeze monthrr the robotics lot. heed there
Taals unknown?
Taok knswdedger Comp itreg/
how do fire the fresow ray?
"GutoRoonn
Guantum
Querium Lab:
(Rebotics (etboioom
GatoRoom
Lob))
kespwledge: Compleney
(irece soy)
Step3 3
Step
trosting
The
Ecooticcion
fie the freene myy teggle the blue froit
insccess bee due to obstacts Let's toggle rebotic first activate the robonic crm
glowing bis aocartt.
Turn Right 90*
Step
Step
Step 11
The réboiic cmm vnible moving around to locate rebotize Togglee (Mtociinn)
The
ray marthor leteled Fieede RuY front you teeee ray.
stepis
Turn Left 90*
In Robotics
Lab
Figure 16: Illustration interactions between the trained model assistant and a human user where the task is to fire the freeze ray. Two assistant errors are highlighted in Steps 5 and 8.
33


Reviews:
[{"review_id": "ExirmWYDJ9", "reviewer": "Reviewer_PSAt", "summary": "This paper proposes BASIS, a three‑stage simulation pipeline that bootstraps visual assistants entirely from multimodal LLM‑driven oracle/user interactions, achieving 72.9 % real‑user success in Alexa Arena and 82 % of an oracle with privileged perception. Object identification emerges as the key failure mode.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This work reduces human effort and interactions like in Wizard-of-Oz data collection.\n\n2. The sim-to-real transfer is successful with 72.9 % SR@1\n\n3. Comprehensive ablations show Chain‑of‑Thought and task knowledge benefits", "reasons_to_reject": "1. Real human training data (66 dialogues) can be limited, which raises some doubts about whether human variability may be under‑represented.\n\n2. Experiments in Table 2 and Table 3 only involve Qwen and GPT-4o models. More open-sourced models could be experimented on for a better demonstration of the generalizability.\n\n3. There might be experimental baselines missing regarding prior visual‑assistant systems, which could lead to comparisons restricted to in‑domain LLM variants.", "questions_to_authors": "1. Given 28.9 % misidentification (Figure 6b), have you tried integrating external vision backbones or active perception to mitigate this bottleneck?\n\n2. Since only 66 oracle‑human traces are collected (line 251), do you plan to collect larger‑scale human data to cover varied communication styles?\n\n3. How would BASIS handle real camera noise and egocentric motion in the physical world (lines 161‑165)?", "extra_scores": {}}, {"review_id": "BnK2edt22K", "reviewer": "Reviewer_3Fih", "summary": "This paper presents a framework for visual assistant development, consisting of three stages: situated interaction simulation, autonomous model development, and real-user validation. The authors implemented the framework by prompting and fine-tuning MLLMs using a synthetic dataset generated based on 70 unique task variants derived from 22 base tasks and 10 preconditions. The total number of successful trajectories in the dataset is 1381. On real-user validation, the fine-tuned model (Qwen2.5-VL 7B CoT) achieved a 72.9% success rate compared to the GPT-4 oracle of 88.6%.", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The simulation framework is vital for fine-tuning MLLM for the complex tasks of visual assistants. \n- The experiment results and detailed error analysis are thoroughly conducted.\n- Qualitative examples to show the interaction between the assistant and the user.", "reasons_to_reject": "- The presented framework is limited to a visual assistant with specific settings. Therefore, it is unclear how to generalize the framework for other problems.\n- Lack of an ablation study to show the effectiveness of the Qwen2.5-VL model trained on synthetic data compared to the base model without training\n- Without a comparison between synthetic and real data. It is challenging to assess the utility of the synthetically generated data.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jAk6xqxPKa", "reviewer": "Reviewer_DrY1", "summary": "This paper proposes a framework named BASIS(Bootstrapping Assistant modeling with Situated Interaction Simulation) to develop visual assistants. It does not rely on expensive human data collection, but rather leverages simulation to bootstrap capable assistants in three stages. The experiments on Alexa Arena demonstrate the effectiveness of the method, and authors leverage detailed error analysis to highlights object current primary failure mode", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper addresses an interesting topic of developing visual assistants and it is easy to follow.  It introduces a novel framework (BASIS) that bypasses the high cost of human-in-the-loop data collection by using LLM-powered simulation to generate interactive training data and conduct automatic evaluations, addressing bottleneck in developing visual assistants.\n2. Experiments across oracle-human, model-simulated user, and real-user settings, and includes fine-grained error analyses that identify key challenges\n3. The experiments derives interesting some insights for future research.", "reasons_to_reject": "1. Although this paper focuses on a novel and interesting topic, the methods (i.e., pipeline) are widely used in other agentic or reasoning-intensive tasks, where (M)LLMs interact with the environment, collect data and trains themselves[1,2,3 ... ]. So, I think it makes technical contributions limited.\n2. The Alexa Arena only includes 70 tasks. It is unclear whether the fine-tuned model really generalizes well to broader scenarios. No sim-to-real environmental transfer scenarios is explored, which weakens the claim that the assistant is broadly deployable in real-world settings.\n3. Although the authors conduct real-user validation, human evaluation appears limited in scope and scale. Furthermore, comparisons to other SOTA visual assistant systems or baselines (beyond the internal oracle) are minimal, making it harder to assess the absolute performance improvements.\n4. It is also unclear whether \"88.6% performance of an oracle assistant\" is good enough for the real-world application. In other words, why 88.6% of an oracle assistant can be viewed as effective? In addition, why the performance of oracle assistant can be viewed as \"*an upper bound for assistant performance\",* it is merely the SOTA performance of closed-source MLLM?\n\n[1] ReAct Meets ActRe: When Language Agents Enjoy Training Data Autonomy\n\n[2] OS-Genesis: Automating GUI Agent Trajectory Construction via Reverse Task Synthesis\n\n[3] NNetscape navigator: complex demonstrations for web agents without a demonstrator", "questions_to_authors": "1. How to implement the training of the assistant model? by SFT?\n\nI will discuss with authors further during the rebuttal.", "extra_scores": {}}, {"review_id": "sfsOb5T9wQ", "reviewer": "Reviewer_EikH", "summary": "The paper proposes a novel training strategy for visual assistants, that is, assistants that have access to visual and audio perception are meant to help users perform physical tasks (for example in smart glasses).\nThe training strategy consists of three phases:\n1. construction of simulated data using a simulated environment, a simulated user and an oracle assistant with access to privileged information about the environment (location, position, properties about the objects)\n2. given the simulated dialog data, training of the assistant model under the actual environment constraints (visual and language perception only); the model is tuned using the simulated user as a source of evaluation data\n3. evaluation of the trained model with real human users, in the simulated environment.\n\nEach step is thoroughly evaluated, and the method is shown to outperform various state of the art approaches, without relying on annotated data.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Previous methods to train visual assistants rely on human interactions, which are incomplete and very expensive to acquire. The proposed method instead only relies on synthetic data, thus it can scale to larger datasets at a significantly lower costs.\nUsing synthetic rather than annotated data is also necessary to scale the complexity of the environment action space, thus allowing more refined simulated environment that more closely match the true purpose of visual assistant (helping in the physical world).\n\nThe authors also perform extensive evaluation of each step of the proposed pipeline, showing strong results and giving confidence of the effectiveness of the method.", "reasons_to_reject": "While the experiments are generally comprehensive, there are a few areas that could potentially be improved:\n1. The goal of Exp2 should be to select the model that more closely simulates a human user. Yet, the comparison given in Table 1 is on task success rate among simulated users, not for each simulated user against a human user on the type of utterances produced. The comparison against human data in Fig. 5 is only for two models, not all 5 in Table 1. It is unclear why a _more effective_ user simulator is also a _more faithful_ user simulator, because the simulator can take shortcuts.\n2. I am uncertain that Exp4 actually fulfills the goal of RQ4, which is to show that the proposed method has better transfer ability from simulated to real _users_ and not from simulated to real _environments_. Exp4 compares a model trained (so to speak) with oracle environment information to one trained with limited information, both in interactions with human users. That's an environment transfer, not a user transfer. To evaluate that the model actually transfers, one would at least compare and analyze the performance of the same model against a simulated and a real user (ie Table 2 line 4 vs Table 3 line 2), but even better it would be to compare against a reasonable amount of human data.", "questions_to_authors": "1. The chechmark convention in Table 1 for precise/fuzzy is extremely confusing, and it took me a long time to understand that precise is the X and fuzzy is the checkmark: they are reversed in the caption, and not clarified in the main body. I would suggest changing that table.\n\n2. SOM annotation, which is provided to the oracle assistant, is not provided to the trained assistant at step 2. Yet, it is reasonable that even in a real-world physical setting, segmentation would be applied prior to passing the image to the assistant, so it would be plausible that the SOM is available. What's the rationale for omitting and relying purely on RGB data?\n3. Similar question to 1, but for depth data: I understand that pretrained MLLMs are not trained with depth data, so it does not make sense as an input for step 1, but a real-world application could well have depth or point cloud data. Would this method be able to simulate this information in order to have a more robust assistant?", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper addresses the compelling and significant challenge of collecting human-in-the-loop data for visual assistants via simulation. The experimental setup is comprehensive, covering oracle-human, model-simulated user, and real-user scenarios, and the subsequent analysis provides valuable insights.,

The work would be further strengthened by incorporating additional experimental evaluations, particularly through expanded real-user studies.
Decision: Accept