Autonomous Workflow for Multimodal Fine-Grained Training Assistants Towards Mixed Reality (2024)

Abstract: Autonomous artificial intelligence (AI) agents have emerged as promising protocols for automatically understanding the language-based environment, particularly with the exponential development of large language models (LLMs). However, a fine-grained, comprehensive understanding of multimodal environments remains under-explored. This work designs an autonomous workflow tailored for integrating AI agents seamlessly into mixed reality (MR) applications for fine-grained training. We present a demonstration of a multimodal fine-grained training assistant for LEGO brick assembly in a pilot MR environment. Specifically, we design a cerebral language agent that integrates LLMs with memory, planning, and interaction with MR tools and a vision-language agent, enabling agents to decide their actions based on past experiences. Furthermore, we introduce LEGOMRTA, a multimodal fine-grained assembly dialogue dataset synthesized automatically in the workflow served by a commercial LLM. This dataset comprises multimodal instruction manuals, conversations, MR responses, and vision question answering. Last, we present several prevailing open-resource LLMs as benchmarks, assessing their performance with and without fine-tuning on the proposed dataset. We anticipate that the broader impact of this workflow will advance the development of smarter assistants for seamless user interaction in MR environments, fostering research in both AI and HCI communities.

heading: 1 Introduction
text: The advent of “Industry 4.0”, centered on the concept of smart manufacturing, presents a landscape with both opportunities and challenges for enhancing production efficiency (Goel and Gupta, 2020;
∗Corresponding author. 1https://www.youtube.com/watch?v=KkZKL3aKMJs
(a) Industrial Car Assembly.
(b) LEGO Brick Assembly. We illustrate several use cases in the demo of BrickDream.1
Figure 1: Examples of fine-grained assembly in MR systems.
Bécue et al., 2021; Jan et al., 2023). Training assistance for automating and accelerating industrial assembly is in huge demand across various manufacturing applications, such as furniture manufacturing (You et al., 2022), industrial product assembly (Funk et al., 2017), and car assembly (Bellalouna et al., 2020).
Mixed reality (MR), encompassing both virtual reality (VR) and augmented reality (AR), spans a spectrum from fully real environments to “matrixlike” virtual environments, showing promise for industrial manufacturing assembly tasks (Gavish et al., 2015; Stender et al., 2021; Butaslac et al., 2022). These multimodal, interactive, user-centric environments provide a solution for trainees who experience significant cognitive workload for training (Hou and Wang, 2013; Botto et al., 2020; Dalim et al., 2020). However, the assistance of a senior
4051
person as a trainer is required, either in person or remotely (Fidalgo et al., 2023).
To advance intelligent virtual assistants, traditional work leverages natural language processing (NLP) techniques (Li and Yang, 2021; Li et al., 2021, 2022; Colabianchi et al., 2023) and reinforcement learning (Sloan et al., 2022) to promote human-machine interactions. Large language models (LLMs), as the new era of prevalent NLP techniques, have been observed to elicit diverse interaction patterns across tasks, demonstrating their versatility and feasibility (Mahmood et al., 2023). However, (i) tailoring assistant services by grounding interactions; and (ii) understanding users’ situated multimodal contexts remain challenging and under-explored (Dong et al., 2023).
To this end, we introduce an autonomous workflow (see Figure 2) tailored for integrating AI agents into MR applications for fine-grained training. We present a demonstration of a multimodal fine-grained training assistant within a toy MR application for LEGO brick assembly. Specifically, we design a cerebral language agent that integrates the LLM with memory, planning, and interaction with MR serving functional tools and a visionlanguage agent, enabling agents to decide their actions based on experiences. Then, we introduce LEGO-MRTA, a multimodal fine-grained assembly dataset synthesized automatically by a commercial LLM. This dataset comprises 65 multimodal instruction manuals, 1,423 conversations with visionlanguage pairs, serving usages of 18 functional
tools in an MR environment. Additionally, several prevailing open-resource LLMs are presented as benchmarks, assessing their performance with and without fine-tuning on the proposed dataset. Furthermore, we anticipate that the broader impact of this workflow will advance the development of smarter assistants for seamless user interaction in MR environments, fostering research in both AI and HCI communities.
We summarize our contributions as follows: • We design a workflow, which integrates au-
tonomous AI agents for fine-grained assembly assistance in an MR demonstration. • We create a multimodal manual-grounded finegrained assembly conversation dataset in the MR context. • We assess several open-resource LLMs as benchmarks, evaluating their performance with and without fine-tuning on the proposed dataset.

heading: 2 Related Work
text: We summarize previous research concerning multimodal datasets and virtual dialogue assistants within the realm of MR.

heading: 2.1 Multimodal Datasets towards MR
text: Traditional multimodal datasets focus on the interactions with sensor data (Patrik et al., 2018) between human-human or human-robot, and only a few of them provide small-scale task-oriented dialogues, such as OFAI-MMTD (Schreitter and Krenn, 2016) and Kontogiorgos et al. (2018), and
Chinese Whispers (Dimosthenis et al., 2020). ScanScribe (Zhu et al., 2023b) releases a 3D scene-text pairs dataset for 3D vision and text alignment learning. HoloAssistant (Wang et al., 2023c) provides a dataset containing 350 unique instructor-performer pairs with AR metadata to perceive, reason, and interact in the physical world. However, the conversations are not publicly available.
Recent studies have concentrated on multimodal datasets with conversations. MDC (Narayan-Chen et al., 2019) presents a collection of 509 humanhuman conversations in the Minecraft VR games. CerealBar (Suhr et al., 2019) creates 1,202 humanto-human conversations that map user instructions to system actions in a situated VR game environment. CVDN (Thomason et al., 2020) collects 2,050 human-robot conversations on Amazon Mechanical Turk for improving parsing and perception for natural language commands. Teach (Padmakumar et al., 2022) builds over 3,000 human–human, interactive dialogues to complete household tasks in the simulation.
Different from those aforementioned datasets, LEGO-MRTA gathers 1,423 synthetic natural conversations between trainers and trainees. Unlike robotic commands, the length of utterances is relatively longer. Furthermore, these conversations are generated by grounding both on an instruction manual and responses from an MR, ensuring that the simulated conversations closely resemble natural human language. We compare the statistics of the above datasets in Table 1.

heading: 2.2 Virtual Dialogue Assistants for MR
text: Conventional efforts focus on creating virtual assistants for human-machine interactions using NLP techniques (Li and Yang, 2021; Li et al., 2021, 2022; Colabianchi et al., 2023) and reinforcement learning (Sloan et al., 2022). LLMs, representing the forefront of contemporary NLP techniques, hold tremendous promise for advancing towards the next generation of intelligent assistants (Naveed et al., 2023). The recent remarkable achievements
of LLMs have spurred a growing interest in utilizing them to address a great variety of complex tasks (Zhang et al., 2023), with particular attention being drawn to LLM-augmented autonomous agents (Yao et al., 2022; Huang et al., 2022; Shinn et al., 2023; Madaan et al., 2023).
Autonomous agents expand the capabilities of LLMs into sequential action execution, demonstrating their proficiency in interacting with environments and addressing complex tasks through data collection (Wang et al., 2023b; Liu et al., 2023). A crucial aspect of this advancement relies on the capacity of LLMs to generate and interpret images, enabling them to access visual content and provide inputs, thereby integrating with MR environments (Oyanagi et al., 2023; Wei et al., 2024). Regarding skill training, autonomous agents and LLMs can create immersive learning experiences that blend virtual and physical environments. For instance, students can utilize them to explore workflows and concepts in a more interactive and engaging manner (Gong et al., 2023; Li et al., 2024). In the context of MR serving as a sandbox (Li et al., 2023b) for LLMs and autonomous agents, the relationship is mutually beneficial. MR offers a secure, adaptable, and regulated setting for training models (Naihin et al., 2023). LLM-powered autonomous agents together with MR hold the potential to revolutionize our interaction with the digital world (Xu et al., 2023).
The convergence of LLMs, autonomous agents, and MR presents both excitement and challenges. As MR training experiences become more realistic and personalized, they demand larger amounts of data, encompassing detailed information about trainees’ behaviors, preferences, and interactions. Ensuring the availability and reusability of this data poses a significant challenge. Overall, our workflow’s ultimate goal is to enhance MR training experiences by facilitating more natural language interactions, generating precise 3D models of real-world objects (Li et al., 2023a), and fostering dynamic and interactive experiences. While
challenges remain (Xi et al., 2023; Ayache et al., 2023), the potential of this powerful technological fusion offers numerous exciting possibilities that could revolutionize personalization in virtual experiences. This entails the development of dedicated workflows and datasets.

heading: 3 Fine-Grained Training Workflow
text: In this section, we describe the proposed workflow (See Figure 2) that advances AI agents towards MR guided fine-grained training.

heading: 3.1 Definition of Fine-Grained Training
text: In the context of fine-grained training, we anticipate the ability to (i) accurately follow professional training instructions documented in an instruction manual; and (ii) be sensitive to detailed visual information, ultimately for complex industrial assembly tasks, as illustrated in Figure 1 (a).
We define the following two roles during a training session: • User: A human trainee who aims to acquire ex-
pertise and will work on fine-grained assembly tasks through interaction with the MR environment. • Assistant: A virtual AI agent who will be able to assist the trainees in training and respond to their inquiries. It offers support with (i) a conversation agent that replies to trainees’ requests and provides guidance grounded in the instruction manual; (ii) an interface for users to interact with the MR environment; and (iii) a vision-language agent that understands and transmits users’ visual context to language.

heading: 3.2 Autonomous AI Agent
text: We design the autonomous AI agent with a chain of two agents, namely (i) a cerebral language agent that serves to reply to trainees’ requests, provide guidance, interact with MR and the vision-language agent; and (ii) a vision-language agent that understands and transmits users’ visual context to language, which is then utilized by the cerebral language agent for planning.

heading: 3.2.1 Cerebral Language Agent
text: Inspired by the concept of LLM-powered autonomous agents (Wang et al., 2023b), we develop a cerebral language agent that incorporates an LLM with memory, planning, and functional tools that can interact with MR application, thereby enabling agents to make decisions regarding their actions
based on past experiences. It can handle multimodal inputs, such as instruction manuals, historical conversations, and metadata within MR environments, and subsequently generate actions (i.e., responses or API calls for the MR application). The scope of responsibility of the agent is defined in a system prompt (See P2, Table 5, Appendix A) Notably, it is able to alleviate the challenges (See §1): (i) it tailors assistant services by seamlessly interaction with MR applications to discover the business needs gradually; (ii) it interacts with a vision-language agent (See §3.2.2), which facilitates the capability of understanding the multimodal context in MR environments.

heading: 3.2.2 Vision-Language Agent
text: The vision-language agent’s mission is to bridge the gap between understanding visual context and language, enabling effective utilization by the cerebral language agent (See §3.2.1) to conduct comprehensive planning for global optimization. Its core is the vision-language model (VLM), which is a task-driven large model that transmits vision input into language output needed by specific tasks.
In the context of LEGO assembly training, we observe two distinct patterns in LEGO instruction manuals (See an example in Figure 3) and define the following two tasks: (T1) Object detection. Given an image or a sequence of images as input, the objective is to predict the position of an object requested in a query and generate output in the format of “<Object> <Xleft> <Ytop> <Xright> <Ybottom>”. For example, during assembly step 2, the AI trainer might direct the trainee, “Please gather the earth blue pair of legs and the silver metallic upper part of the body.” In response, the trainee may ask, “Is this the one?” The vision-language agent is tasked with recognizing the object the trainee is referring to. (T2) Assembly state detection. Given an image or a sequence of images as input, the objective is to identify if the current assembly state matches the reference state provided in the instruction manual. For example, during assembly step 3, the vision-language agent is responsible for assisting the user’s request, such as “Am I assembling them correctly?”

heading: 3.3 Pilot MR Application Design
text: We design an MR application as a pilot to show intuitive demonstration. First, we utilized a commercial LLM to generate candidate user requirements using the prompt (See P1, Table 5, Appendix A)
as input. Then, we brainstormed and discussed the generated user requirements within a group of researchers and developers and finalized 7 user requirements (See Table 6, Appendix A) and 18 serving functional tools (See Table 2). We develop a standard application programming interfaces (APIs) to enable seamless interactions between functions in the MR application and AI agents.

heading: 4 Dataset Creation
text: In this section, we introduce how to use the proposed workflow (See §3) to create a multimodal dialogue dataset in the MR environment.

heading: 4.1 Instruction Manual Crawling
text: We crawled 65 multimodal instruction manuals for fine-grained training from the LEGO official website2. A manual provides illustrated images and
2https://legoaudioinstructions.com/ instructions
textual instructions on how to use, operate, assemble, and install a LEGO brick set. The key sections of an instruction manual include: (i) a summary that describes the general information, such as topics and candidate parts for assembly. It is followed by (ii) a sequence of multimodal step instructions. Each step contains a set of textual instructions and an illustration by image. Key functional phrases such as theme entities are highlighted in textual instructions. Here we show an example of a LEGO instruction manual in Figure 3.

heading: 4.2 Tool Response Generation
text: First, we use the crawled instruction manuals and the well-designed prompt template to produce prompts as the LLM input to generate user functional requirements and decide the serving functional tools. Then, we randomly choose up to 6 tools for each conversation session and record the simulated responses generated from templates.

heading: 4.3 VLM-Based QA Construction
text: First, we use the step instruction to construct a query, containing a special token (“[detection]”) for the object detection task and a single instruction in a step. Second, we employ a query and the aligned image as inputs for MiniGPT-v2 (Chen et al., 2023; Zhu et al., 2023a), generating inference output as an answer of the query in the format of “<Object> <Xleft> <Ytop> <Xright> <Ybottom>”. Last, we iterate through all instruction steps in a conversation session, repeating the above two steps to construct vision question answering (VQA) pairs.

heading: 4.4 Multimodal Context-Aware Conversation Generation
text: We generate conversations grounded on both the instruction manual and simulated tool responses using a commercial LLM. First, we reconstruct full instruction manuals with a summary and 10 step instructions because the average number of steps per manual is 215.3, which is quite long. This may limit the input tokens of an LLM and potentially distract the LLM with less grounding capability. Second, we instantiate the designed prompt template (See P3, Table 5, Appendix A) with the chunked instruction manuals. Last, we utilize a commercial LLM as the core of the proposed workflow to generate the conversations. Specifically, the system prompt informs the language agent about its responsibilities. The query prompt is used for each round of requests to generate a conversation.
The historical rounds of requests are tracked by memory.

heading: 4.5 Dataset Statistics
text: We report the statistics of instruction manuals (See Table 3) and conversations (See Table 1).
We obtain 65 instruction manuals as grounding to lead a commercial LLM to generate 1,423 human-human natural conversations between trainers and trainees. Each instruction manual can make 21.9 conversations on average. Theoretically, the amount of conversations can be enlarged by multiple times of requests. However, we focus on showcasing how to create meaningful datasets automatically. We construct 26,405 context-response pairs from generated conversations and VQA pairs as data samples. The average length is 107 tokens for the context and 145 tokens for the response utterance. We utilize 21.10k samples for fine-tuning open-resource LLMs to enhance the instruction-
following capability and evaluate their performance on 5.25k test samples 3. Compared with existing datasets, LEGO-MRTA ensures that the simulated conversations closely resemble natural human language due to the design of the simulation method.

heading: 5 Experimental Setup

heading: 5.1 LLM Benchmarks
text: We consider several prevailing 7B open-source decoder-only LLMs as benchmarks, considering privacy concerns associated with fine-grained training in manufacturing. • BLOOM (Le Scao et al., 2022) is pretrained on
the multilingual ROOTS corpus, offering multilingual capabilities for various NLP tasks. • Falcon-instruct (Almazrouei et al., 2023) is pretrained on a large corpus of RefinedWeb data and fine-tuned on mixed chat and instruct datasets. • Llama2-Chat (Touvron et al., 2023) is a pretrained and fine-tuned generative text model optimized specifically for dialogue tasks, ensuring high-quality conversational responses. • Vicuna1.5 (Zheng et al., 2023) is a chat assistant derived by fine-tuning Llama 2 on user-shared conversations collected from ShareGPT. • OpenChat3.5 (Wang et al., 2023a) is a chat model fine-tuned with the C-RLFT strategy on mixed-quality data, achieving performance com-
3https://huggingface.co/datasets/voxreality/ vox_arta_lego_v2
parable to larger models like ChatGPT. • XVERSE4 is a versatile model supporting 8k
context length, ideal for longer multi-round dialogues, knowledge question-answering, and summarization tasks, trained on a diverse dataset of 2.6 trillion tokens. • BlueLM-Chat5 is a large-scale language model optimized for chat tasks, offering improved context understanding. • Qwen-Chat (Bai et al., 2023) is a chat model that fine-tunes the pretrained Qwen model using human alignment techniques. • Mistral-Instruct (Jiang et al., 2023) is a finetuned version of the Mistral-7B-v0.1, specifically tailored for instruction-based tasks using publicly available conversation datasets.

heading: 5.2 Evaluation Metrics
text: We evaluate the performance in terms of both overlap (BLUE-n, ROUGE-n) and informativeness (ToolACC, ThemeACC): • BLUE-n measures precision, which measures
the ratio of n-grams in the generated responses that match those in the reference responses. We consider n = 4. • ROUGE-n measures recall, which calculates the ratio of n-grams in the reference responses that are captured by the generated responses. Here we consider n = 1, 2, L and L denotes the number of longest common subsequences. • ToolACC is defined as the ratio of correctly men-
4https://github.com/xverse-ai/XVERSE-7B/blob/ main/README_EN.md
5https://huggingface.co/vivo-ai/ BlueLM-7B-Chat
tioned entities by the generated responses, compared to the reference response, from a list of serving tools. • ThemeACC is defined as the ratio of correctly mentioned entities compared to the reference response, from a list of theme entities obtained from the instruction manual.

heading: 5.3 Implementation Details
text: The implementation of the workflow is based on LangChain.6 The model “gpt-3.5-turbo-16k-0613” is used as the commercial LLM for generating data, e.g., conversations, user requirements, and serving functions. The MiniGPT4-v27 is used as the VLM to detect the object, followed by simple rules to generate vision-language pairs.
We employ LoRA to conduct parameter-efficient fine-tuning of 7B open-source LLMs on the proposed dataset using the framework proposed by Hiyouga (2023). Specifically, the maximum sequence length is set to 1024 and the learning rate is 5e-05. The model is trained for 3 epochs with a per-device batch size of 4, and accumulated gradients every 4 steps. A cosine learning rate scheduler is employed, with a maximum gradient norm of 1.0. We log results every 5 steps and save model checkpoints every 100 steps. Warm-up steps are set to 0. LoRA is used with a rank of 8 and a dropout rate of 0.1 for regularization. All experiments are run on NVIDIA A100 SXM4 40GB GPUs.
An example of generated conversation
19
20

heading: 6 Outcomes

heading: 6.1 Evaluation on Benchmark LLMs
text: Table 4 shows the performance on 9 prevailing open-source LLMs, without and with fine-tuning on the LEGO-MRTA dataset.
First, after fine-tuning, the performance of all models gets dramatically improved in terms of all metrics, except for the results that are underlined. This demonstrates the feasibility and effectiveness of tailoring LLMs for fine-grained training in MR environments. In addition, this shows the proposed LEGO-MRTA dataset contains distinct characteristics that have not been captured by existing publicly available datasets.
Second, there exists a trade-off between overlap and informativeness evaluation. For example, for metrics concerning overlap, Qwen consistently achieves the highest scores; while its informativeness is inferior to best models, i.e., ToolACC is 6.83% lower than that of XVERSE, ThemeACC is 10.73% lower than that of OpenChat3.5.
Third, the choice of backbone LLMs inherently impacts the performance of both overlap and informativeness. We compute the standard deviation for each metric over all models to gauge the variability in performance scores: BLEU-4 (19.60), ROUGE-
6https://python.langchain.com/docs/get_ started/introduction
7https://github.com/Vision-CAIR/MiniGPT-4

heading: 1 (17.79), ROUGE-2 (16.02), ROUGE-L (20.64),
text: ToolACC (19.85), and ThemeACC (19.19).

heading: 6.2 Case Study
text: Figure 4 shows an example to intuitively verify the feasibility of generating conversation based on an instruction manual. As highlighted in the trainer’s utterance, “Place the upper body on the legs”, this accurately conveys the instruction from the manual in a human-like manner. The generated conversation is feasible at instruction-following capability.
Figure 5 illustrates an example demonstrating the feasibility of constructing queries based on instructions from a manual to accurately request positions in a multimodal context. We transferred the generated position and highlighted a frame with markers. We observed that the prediction was relatively accurate. Additionally, another aspect we observe that needs improvement in the future is the redundant output from the VLM.

heading: 7 Discussion of Broader Impact
text: The research presented in this paper offers a fully new environment to advance how workers are trained and get help by using MR technologies. By integrating AI assistants into MR environments, workers can tackle complex tasks more effectively. This innovation not only enhances worker productivity but also reduces training costs for companies, as it eliminates the need for expert instructors to be physically present for employee training sessions.

heading: 8 Conclusion
text: In this work, we introduce an autonomous workflow to develop smarter multimodal fine-grained training assistants in MR environments. Specifically, we have designed a cerebral language agent that integrates LLM with memory, planning, and interaction with MR tools, along with a visionlanguage agent. This integration enables agents to make decisions based on past experiences, thereby
addressing the challenge of tailoring assistant services through grounded interactions. We have designed a vision-language agent to better understand users’ situated multimodal contexts. Notably, we have created a dataset for fine-grained training in MR. We have compared the performance of openresource LLMs before and after fine-tuning using this dataset. We aim to facilitate the development of smarter assistants for seamless user interaction, fostering research in AI and HCI communities.
Reproducibility
We release resources including the source code and dataset at https://github.com/Jiahuan-Pei/ AutonomousDialogAgent4AugmentedReality.
Limitations
The generation of user requirements and the dataset relies solely on the simulation process. This workflow serves as a fast solution to verify the concept of an LLM agent aiding in a specific use case, such as a LEGO assembly assistant. However, we acknowledge that the study of user requirements are valuable and needed to build up user-centric AI agents and MR applications. Besides, the demonstration codes do not optimize LLM and VLM simultaneously, potentially leading to suboptimal outcomes. We have only assessed LLMs as benchmarks. However, we have not conducted separate assessments of the influence on the vision-language agent and user experience in MR. We plan to explore these aspects in future work.
Ethics Statement
We realize that there are risks in developing a large language model for users, so it is necessary to pay attention to the ethical issues. Therefore, we use the open-resourced LLMs as benchmarks and consider user-centric points: A user will first be provided with an explanation of what will be happening during their MR training experience. Users will then be provided with relevant consent forms to sign, and after signing they will be fitted with AR glasses and the training scenario will begin. After launching the application, the user will be greeted by the virtual assistant and prompted to confirm they would like to begin training. After confirming, the user will then be asked by the virtual assistant which difficulty level they would like to be trained on.

heading: Acknowledgements
text: This work was supported through the Horizon Europe research and innovation programme, under grant agreement No 101070521 (VOXReality). This work used the Dutch national e-infrastructure with the support of the SURF cooperative grant (No. EINF-7705). Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or Directorate-General for Communications Networks, Content and Technology. Neither the European Union nor the granting authority can be held responsible for them.

heading: A Appendix
text: A.1 Prompt Templates
A.2 Generated User Requirements
A.3 Qualitative Analysis of the Dataset
The collective results as seen in the figure 6 illustrates that our simulated model has practical assistance capabilities as summarized below:
Realistic simulation. LEGO is a well-known block building concept. The dataset simulates various real-world scenarios encountered during LEGO assembly tasks. By replicating factors such as piece variability, environmental conditions, and assembly constraints, the dataset provides a realistic training environment for machine learning models. This realism enhances the model’s ability to generalize to unseen situations, ensuring reliable performance in diverse assembly settings.
Diversity in task difficulty. From simple structures to intricate designs, the dataset exposes the model to diverse assembly scenarios, enabling it to learn robust representations of LEGO building principles. This diversity fosters adaptability in the model, empowering it to tackle simple to difficult or probably novel assembly tasks with confidence and efficiency.
Transfer learning to other tasks. The dataset is structured to facilitate transfer learning, allowing knowledge and representations learned from one assembly task to be applied to related tasks or domains. By leveraging pre-trained models or features learned from similar assembly tasks, machine learning models can bootstrap their learning process on new assembly tasks. This transfer learning capability accelerates model adaptation to new environments and tasks, reducing the need for extensive retraining and improving overall training efficiency.
A.3.1 Instruction Tokens (Figure 6a) Analyzing the provided instruction tokens, we can derive several factors that contribute to the usability and effectiveness of our simulated dataset for XR training and assembly training: • Clear instructional guidance. Tokens like “Put,”
“Find,” “Collect,” and “Open” provide clear and concise instructions for performing various assembly tasks. These instructions guide users by the assembly process step-by-step, ensuring clarity and direction in the training environment. • Spatial orientation and manipulation. Tokens such as “front,” “right,” “left,” “horizontally,” and “vertically” offer spatial orientation cues, helping users understand the spatial relationships between LEGO components and how to manipulate them during assembly. This spatial awareness enhances users’ ability to accurately position and align LEGO pieces. • Feedback and assistance. Tokens like “help” and “response” indicate provisions for feedback and assistance within the training environment. Offering assistance and feedback helps users troubleshoot issues, learn from mistakes, and improve their assembly skills over time, enhancing the learning experience. • Multimodal learning: The inclusion of tokens like “Audio Instructions” suggests the incorporation of multimodal learning techniques within the training environment. Integrating audio instructions alongside visual cues enhances usability by catering to different learning styles and preferences, making the training experience more accessible and engaging for users. • Adaptive learning. Tokens such as “current” and “previous” imply a dynamic learning environment where users can track their progress and revisit previous steps if needed. Adaptive learning features enhance usability by allowing users
to learn at their own pace, review concepts as needed, and progress through the training material in a structured manner.
• Interactive learning environment.: The presence of tokens like “conversations” and “trainer” indicates an interactive learning environment
where users can engage in dialogue and receive guidance from trainers or virtual assistants. Interactivity enhances usability by promoting engagement, collaboration, and active participation in the learning process, leading to more effective skill acquisition and retention.
The instruction tokens in our simulated dataset indicate clear guidance, and spatial orientation cues within an interactive learning environment.
A.3.2 Theme Entities (Figure 6b)
Based on the theme entities provided, we analyze their relevance to the learning process: • Part identification. Tokens such as “plate 1x2,”
“plate 2x4,” and “brick 1x1” provide specific identifiers for different LEGO parts commonly used in assembly tasks. By including a variety of part identifiers, the dataset facilitates part recognition and identification, enabling the model to learn the characteristics and properties of each component.
• Spatial orientation and configuration. Tokens like “head,” “upper body,” and “feet” suggest the inclusion of assembly instructions related to spatial orientation and configuration of LEGO structures. Understanding the spatial arrangement of components is essential for accurate assembly, and these tokens help the model grasp the hierarchical structure of assemblies and the placement of parts within them. • Assembly techniques. Tokens such as “can be tricky” and “structure upside down“ hint at the inclusion of assembly techniques and strategies within the dataset. Learning various assembly techniques is crucial for efficiently building complex structures, and these tokens provide guidance on overcoming challenges and optimizing assembly processes. • Component variations. Tokens like “round plate 1x1,” “flat tile 2x4,” and “grille tile 1x2“ introduce variations of standard LEGO compo-
nents, reflecting the diversity of parts encountered in real-world assembly scenarios. By including a range of component variations, the dataset contains different types of parts and adapts to varying assembly requirements. • Accessory identification. Tokens such as “pair of legs,” “tire,” and “hair” denote accessory pieces commonly used in LEGO constructions, adding realism and complexity to assembly tasks. Recognizing and incorporating accessory pieces is essential for creating realistic and detailed models, and these tokens help the model understand the role of accessories in assembly. • Quality control and correctness. Tokens like “correct sticker” and “correct bag” emphasize the importance of quality control and correctness in assembly tasks. Ensuring that the correct parts are used in the right context is essential for achieving accurate and high-quality assemblies, and these tokens highlight the need for attention to detail and accuracy in the assembly process. • Structural components. Tokens such as “shaft,” “structure upside down,” and “roof tile” suggest the inclusion of structural components and building techniques within the dataset. Understanding the role of structural components and mastering advanced building techniques is critical for creating stable and aesthetically pleasing assemblies, and these tokens provide guidance on constructing sturdy and well-balanced structures. The theme entities included in our simulated dataset provide a realistic representation of the assembly tasks by encompassing part identification, spatial orientation, assembly techniques, component variations, accessory recognition, quality control, and structural components, the dataset contains the knowledge and skills necessary to effectively assemble LEGO structures in MR.
A.3.3 Conversation Tokens (Figure 6c) We can infer several aspects that contribute to the usability and effectiveness of our simulated dataset for conversations during assembly training: • Role identification. The presence of “Trainer”
and “Trainee” tokens indicates a clear distinction between the roles of the instructor guiding the training session and the learner receiving instructions. This role identification fosters clarity and structure in the conversation, ensuring effective communication between trainer and trainee. • Instructional guidance. Tokens such as “step,” “plate,” “use,” and “find” suggest the provision
of instructional guidance within the conversation. The trainer entity likely provides step-by-step instructions and prompts to the trainee, guiding them through the assembly process and facilitating learning in a structured manner. • Interactive dialogue. The conversation tokens include interactive dialogue cues such as “Let’s,” “Yes, let’s,” and “Thank you!” These cues foster engagement and collaboration between the trainer and trainee entities, creating a supportive and interactive learning atmosphere conducive to effective learning and skill development. • Feedback and encouragement. Tokens like “Great!” and “Alright” suggest the inclusion of positive feedback and encouragement within the conversation. Positive reinforcement enhances motivation and engagement, encouraging active participation and fostering a positive learning experience for the trainee. • Error handling and assistance. The presence of tokens like “check,” “help,” and “completed” indicates provisions for error handling and assistance within the conversation. The trainer entity likely offers guidance and support to the trainee in identifying and correcting errors, ensuring a constructive learning process and facilitating skill development. • Spatial orientation and task management. Tokens such as “right,” “left,” “back,” and “steps” provide spatial orientation cues and references to assembly tasks. This spatial orientation facilitates effective communication of assembly instructions and task management between the trainer and trainee entities, ensuring accurate placement and alignment of LEGO components during assembly. The conversation tokens provide instructional guidance, facilitate interactive dialogue, offer feedback and encouragement, handle errors, and provide spatial orientation cues for task management.
A.3.4 Vision-language Tokens (Figure 6d) Analyzing the provided tokens from the vision language model, we can identify several factors contributing to its usability and effectiveness: • Object recognition. Tokens such as “plate,”
“brick,” “tile,” and “knob” represent common LEGO elements that users encounter during assembly tasks. By including these tokens, the dataset enables the vision language model to recognize and identify various LEGO components accurately, facilitating object recognition and un-
derstanding in XR training environments. • Color detection. Tokens like “bright,” “grey,”
“white,” and “blue” provide color descriptors for different LEGO pieces. Incorporating color information allows the vision language model to detect and differentiate between LEGO components based on their color, enhancing the model’s ability to interpret and analyze assembly scenes accurately. • Shape recognition. Tokens such as “round,” “flat,” “roof,” and “connector” describe the shapes and configurations of LEGO elements. By including shape descriptors, the dataset enables the vision language model to recognize and classify different types of LEGO pieces based on their shapes, facilitating shape recognition and classification in XR training environments. • Size specification. Tokens like “1x2,” “2x2,” and “1x1” specify the sizes and dimensions of LEGO elements. Incorporating size information allows the vision language model to understand the scale and proportions of LEGO components within assembly scenes, aiding in size estimation and spatial reasoning during XR training tasks. • Material and texture. Tokens such as “smooth,” “nougat,” and “transparent” describe the materials and textures of LEGO elements. Including material and texture descriptors enables the vision language model to identify and distinguish between different surface finishes and textures, enhancing its ability to recognize and characterize LEGO components accurately. • Part relationships. Tokens like “side,” “top,” and “bottom” provide spatial relationship cues between LEGO elements. By including part relationship descriptors, the dataset enables the vision language model to understand the spatial arrangement and orientation of LEGO components within assembly scenes, facilitating the interpretation of complex assembly structures and configurations. • Visual context understanding. Tokens such as “image” and “shows” suggest the inclusion of visual context information within the dataset. Providing visual context cues enables the vision language model to interpret and analyze assembly scenes holistically, incorporating visual information to enhance its understanding of the surrounding environment and improve object recognition accuracy.
Our simulated dataset successfully provides ob-
ject recognition, color detection, shape recognition, size specification, material and texture characterization, part relationships, and visual context understanding. Altogether, these tokens contribute to the usability and effectiveness of the training environment by providing clear guidance, realistic representation of components and challenges, interactive dialogue, and enhanced vision understanding. These elements collectively enhance the learning experience and skill development in XR assembly tasks.
A.3.5 Called Tools (Figure 7)
As shown in Figure 7, we plot the distribution of the number of tools invoked in the generated conversations. The most frequently called and essential functional tools are those related to process control: “NextStep” (57.02%), “StartAssemble” (4.58%), “CheckStepStatusVR” (4.28%), “GoToStep” (2.44%), “GetRemainingStep” (1.90%), “GetCurrentStep” (1.78%), “1.31%”. This indicates that users prioritize adherence to the assembly procedure during the fine-grained assembly task. Functional tools related to user interactions are also significant, for example, “HighlightCorrectComponents” (4.64%).
A.4 Engineering Details in the Workflow

heading: Future Work
text: We plan to explore these aspects in future work