SIMMC-VR: A Task-oriented Multimodal Dialog Dataset with Situated and Immersive VR Streams
Year: 2023

Abstract:
Building an AI assistant that can seamlessly converse and instruct humans, in a user-centric situated scenario, requires several essential abilities: (1) spatial and temporal understanding of the situated and real-time user scenes, (2) capability of grounding the actively perceived visuals of users to conversation contexts, and (3) conversational reasoning over past utterances to perform just-in-time assistance. However, we currently lack a large-scale benchmark that captures user↔assistant interactions with all of the aforementioned features. To this end, we propose SIMMC-VR, an extension of the SIMMC 2.0 dataset to a video-grounded taskoriented dialog dataset that captures real-world AI-assisted user scenarios in VR. We propose a novel data collection paradigm that involves (1) generating object-centric multimodal dialog flows with egocentric visual streams and visually-grounded templates, and (2) manually paraphrasing the simulated dialogs for naturalness and diversity while preserving multimodal dependencies. To measure meaningful progress in the field, we propose four tasks to address the new challenges in SIMMC-VR, which require complex spatial-temporal dialog reasoning in active egocentric scenes. We benchmark the proposed tasks with strong multimodal models, and highlight the key capabilities that current models lack for future research directions.

heading: 1 Introduction
text: With the growing popularity of smart glasses, studies on visually grounded conversational agents have gained significant interest. For instance, SIMMC 2.0 (Kottur et al., 2021) introduces an imagegrounded, task-oriented dialog (TOD) dataset where an assistant agent co-observes the user’s egocentric viewpoint to aid with user requests. Many follow-up works (Huang et al., 2021a; Lee et al., 2022; Chiyah-Garcia et al., 2022) focus on challenges around dialog-image grounding, such as
∗Work done during an internship at Meta Reality Labs.
visual coreference resolution (e.g. ‘the yellow dress behind the rack’) of a static image.
However, several technical gaps still remain in applying prior work to build a real-world, situated multimodal assistant (Figure 1). For instance, a typical multimodal user-assistant scenario (with a video capturing capability) would include (1) spatial and temporal language references as grounding contexts (‘the shirt I saw earlier when I entered the store’), (2) actively perceived egocentric motions as part of conversation contexts (“No – turn around the other way"), (3) references to conversational memories from past sessions (‘the one I
6273
bought earlier’, the ‘black coat’ in Figure 1 being retroactively mentioned by both the assistant and the user), etc. While these scenarios are perceived as the expected capabilities of a next-generation multimodal assistant, our survey of datasets (Sec. 5) highlights that due to the static and constrained nature of the datasets’ grounding context, they lack sufficiently complex interactions.
To this end, we present SIMMC-VR, a videogrounded task-oriented dialog dataset comprising 4K user↔assistant task-oriented dialogs (95.3K utterances) grounded on diverse photorealistic VR video streams (4.8M frames). For data collection, we propose a novel two-stage approach with: (1) a multimodal interaction simulator that generates egocentric VR streams grounded on object-centric multimodal dialog flows, and (2) a manual paraphrasing step for naturalness and diversity while preserving multimodal dependencies between visual scenes and their grounding language. Our pipeline allows for flexible and cost-effective data collection, easily extendable to simulate any other domains given the availability of 3D virtual assets.
To measure progress towards real-world applicability, we propose four SIMMC-VR tasks that address new challenges in complex spatio-temporal dialog reasoning. We then extend state-of-the-art multimodal models to the SIMMC-VR tasks and discuss the limitations of current models.
Our contributions are as follows: (1) we present SIMMC-VR, a video-grounded task-oriented dialog dataset (95K utterances over 4.8M frames) targeted towards real-world applications for an assistant on smart glasses. (2) We propose the tasks with complex spatio-temporal conversational dependencies, and benchmark them by extending the state-ofthe-art multimodal models. (3) Our data collection platform allows creation of a similar dataset in any target domains.

heading: 2 SIMMC-VR Dataset
text: SIMMC-VR is actively multimodal, where each data instance is a video from a user’s egocentric viewpoint recording all interactions within a virtual shopping environment, densely paired with dialog utterances and essential attributes. Each taskoriented dialog mimics real-world shopping scenarios where the assistant’s goal is to help the user make purchases and navigate through the environment. In each instance, the user walks around a virtual shop while the assistant provides product
information or recommendations; as well as help the user locate and navigate to products of interest.
Dataset Collection Strategy. Multimodal or embodied dialogs (Das et al., 2017a; Padmakumar et al., 2022) are often constructed via a two-player game where participants interact with the environment and converse with each other (i.e. in a Wizard of Oz (WOZ) (Mrkšić et al., 2017; Budzianowski et al., 2018a) role-playing fashion). However, it can be overly challenging to require annotators to role-play as the AI assistant in our complex and quite cluttered VR shop environments (>100 products). Furthermore, to match the potential retroactive reasoning shopping scenarios (e.g. concerning products priorly seen/mentioned), it could add much mental burden for annotators to memorize object attributes and their locations while composing authentic long dialogue interactions. Lastly, in conjunction with the aforementioned difficulties, it is rather unscalable and inextensible to manually annotate all the required labels (dialog acts, coreferences) cross-referencing complex moving scenes for a task-oriented dialog dataset.
We therefore collect the dataset through two phases: (1) simulating multimodal dialog flows with templated utterances – thereby programmatically generating fine-grained-scene-grounded annotations and systematically ensuring the diversity of the conversations, and (2) manual paraphrasing, which ensures the naturalness of utterances with a significantly less annotation overhead (Rastogi et al., 2020; Shah et al., 2018).

heading: 2.1 Multimodal Dialog Generation
text: Our pipeline for multimodal dialog generation simulates plausible and natural multimodal interactions in a virtual environment (Figure 2), The process is as follows: (1) Decide a meta-agenda based on object attributes and traversal routes. (2) Sample specific objects that fulfill the decided agenda as the object-centric flow. (3) Perform the user traversal path planning and video recording using the sampled objects as starting/ending points. (4) Synthesize the corresponding utterances via prewritten templates and the multimodal contexts. (5) Manually paraphrase the templated utterances.
We categorize a full dialog instance (generated through the previously described steps) into two phases: (a) static phase where the user mostly focuses on a specific viewpoint (with a small amount of randomness in movement or eye-gaze) when conversing with the assistant (Section 2.1.2), and (b) active phase, where the user navigates to another spot within the environment, at will or following assistant instructions, containing larger movements and actions (Section 2.1.1). The two phases interleave each other, creating a realistic shopping scenario (e.g. user walks into a shop, stopping by a few products, and wanders to other ones).
Virtual Environment. Following SIMMC 2.0, we use the same set of photorealistic VR shopping environments in Unity (Unity, 2020), where a set of seed scenes with pre-arranged digital assets (e.g. shirts, dresses for fashion domain and sofas, tables for furniture domain) are programmatically re-arranged into randomized larger sets of scenes.
Table 6 lists the asset (product item) categories used for constructing the SIMMC-VR dataset for both fashion and furniture domains.

heading: 2.1.1 Active Scene Simulation
text: Figure 3ab illustrates the process of simulating visual observations of a user traversal, where a path planning is performed (connecting the start and end user position/orientation) in the environment, and the trajectories are rendered into egocentric videos.
Path Planning. Ideally, the navigational guidance should minimize the overall traversal distance (to a target spot), while taking the smoothness of movements into consideration. Given a start and end position in the extracted environment layout, we perform an A∗search to plan a trajectory simulating a user’s traversal within a shop. Additionally, we modify the standard A∗algorithm to minimize the amount of turning for smoother and more natural user movements1, with random noises added to naturally jitter the planned path. We then augment the output path with rotation angles computed to account for the user orientation during the traversal. At each viewpoint on the planned path, a Unity camera snapshot is taken, and the traversal video is rendered by combining all the snapshots. Referential Objects. Once the intended usertraversal video is planned and recorded, we define key action points, using the start/end viewpoints of user movements (i.e. displacement or turning actions). Inspired by the natural communication behavior, where we often refer to certain landmarks when giving navigational guidance, we derive a set of referential objects from objects placed across these viewpoints (e.g. “Turn left when you see the red shirt."). Figure 3a illustrates the referential object sampling strategy: (1) Compute the cosine similarity between an egocentric viewpoint (3D) vector (gaze point at the center of yellow dotted lines) and a look-at vector to each of the objects within the scene – a higher similarity implies that it is closer to the eye-gaze line of sight, hence more probable to be referenced during conversations. (2) Augment the previously derived rankings with other plausible features such as stronger color contrast with neighboring objects. (3) Lastly, transform these rankings into sampling probabilities (via a Softmax) to sample object(s) for reference. Scene Graphs & Disambiguation. When referring to an item in a cluttered environment, its surroundings often serve as good candidates to disambiguate items that may share similar attributes (often useful when users under-specify items). In light of this, for each object within the same scene, we build a local scene-graph to include the closest three objects to its left, right, top, bottom (four main directions). An object can then be referred to with its neighbors when further clarification is needed (e.g. "Not that one, I mean the white hat below the red coat."). Scene Metadata. To facilitate templated utterances for paraphrasing (Section 2.1.2) and to formulate a modeling task with visual labels (Section 3), we compute 2D bounding boxes for all 3D assets in a particular viewpoint, where each object is crossreferenced across every frame. As the dense bounding box computation in a 3D environment is timeconsuming (repeated for thousands of frames per di1A∗’s distance minimization may lead to excessive turns.
alog), we expedite this process via an approximate reconstruction. Specifically, we record the camera position and orientation for each video frame, and provide the mesh data for each asset and a function to reconstruct 2D bounding boxes on-the-fly.

heading: 2.1.2 Dialog Simulation
text: In real-life shopping experiences, customers typically explore a shop with certain product attributes of interest in mind (e.g. clothing colors, types), thus shopping experiences are often objectcentric (Yinyin, 2011). Inspired by this, we program several (extendable) object-centric flows that focus on certain objects within an environment to mimic how a user may wander (self-motivated or guided) around from one product to another. Dialog Flows. To have full control over the diversity of dialog flows, and to encourage certain patterns of flows to emerge for more interesting userAI conversations, we propose an object-centric generation pipeline. Specifically, to generate an objectcentric flow, we (1) define a meta-agenda, a sequence of meta-goals2 defined by certain object attributes that simulate a complete shopping experience (e.g. a customer looking for certain types or colors of clothing, or asking for a complementary item to match a previously purchased one) and (2) for each meta-goal, sample an object according to a planned traversal route (e.g. short or long travel distance, traveling back to a previously observed item) and a user-position/orientation to look
2We cap the max sequence length at 3, i.e. 3 meta-goals.
at the object (where the path planning can perform on).3 The meta-agenda is either human-written or programmatically generated, and diversified while ensuring a balanced distribution of scenarios. The traversal route is engineered to ensure user’s navigation/orientation changes are necessary and natural.
For each of the sampled-objects, a goal generator will sample a high-level dialog goal to define the theme of a few turns of utterances (e.g. COMPARE→ user requesting product comparisons). The user simulator then utilized both the sampled objects and goals to generate corresponding NLU labels following a probability distribution, consisting of user intents (e.g. INFORM:GET), request slots (e.g. color, brand) and object references. The assistant simulator then resolves the user requests, leveraging the multimodal context and the simulation API (e.g. for info lookup).4
3Each flow is uniquely defined by the sampled objectsequence. We over-sample totally >1K object-centric flows evenly across 27 programmed meta-agenda (Figure 3c).
4In contrast, SIMMC 2.0 plans a dialog only by randomly sampling a sequence of abstract goals (e.g. BROWSE → GET_INFO→ ...), often resulting in unrealistic scenarios.
Meta-Agenda. Table 2 lists the candidates that can be programmed into the meta-agenda. For alternative and complementary item mappings, we consider: (1) Relations in ConceptNet 5.0 (Speer et al., 2017) such as distinct_terms (jacket is distinct_to coat), similar_terms and/or related_terms (e.g. sofa is related_to end-table). And (2) Manual inspections and annotations, where we ask internal members to annotate the alternative and complementary items to a particular one of interest, and refine the annotated list with majority vote (e.g. hat is complementary to both shirt and dress as they can go in pairs, and coat is alternative to jacket as they share similar functionalities and thus can complement each other).
For the positions agenda, we pre-define a distance threshold to denote far or close depending on the environment room layout (differ in fashion and furniture domains). For the come_back_to_X program, we engineer that the user will traverse back to an item that is previously seen and indicated with interests, to simulate relevant shopping experiences in the real-world.
Templated Utterances. Grounded by the multimodal context, we pre-define a few utterance templates each associated with a specific dialog act, leaving the specific object-related information (e.g. object ids, modifiers, pronouns) as placeholders that are filled-in according to the visuals. This allows us to easily sample an utterance template that is suitable for a particular situation and the associated user or AI intention, determined by the dialog act. We list a few exemplar utterances and their paraphrases, and highlight the placeholders in Table 3. Notice that the local object scene-graphs (Section 2.1.1) are also useful for generating diverse reference expressions for the same object (second role of the Assistant examples in Table 3).
Manual Paraphrase. Next, we ask human annotators to paraphrase the templated utterances to better match the real-world natural language distribution. We design an interface that dynamically displays a multimodal scene that features either a still image (static dialog phase) or a user egocentric video (active dialog phase). When clicking on a specific turn of a dialog, the corresponding visual input is shown in the display panel to help annotators navigate through the entire dialog flow. We ask the annotators to pay attention to detailed and sophisticated spatial-temporal relations of objects and encourage writing interesting shopping
experiences. The paraphrases are collected from more than 20 different linguistic experts for diverse language patterns/usages.
Once manual paraphrases are collected, we perform text-to-speech synthesis (TTS) on the utterances, and synchronize the speech with the relevant motion renders for improved naturalness, making the rendered user shopping videos more realistic (and comprehensive). We use an open-sourced tool, Coqui TTS (Coqui.ai, 2022) to generate the spoken speech from the paraphrased utterances. This also helps computing the natural duration of each utterance when spoken so that we can interpolate certain number of video frames (under a fixed frame-rate) to fit such utterance would span.
An exemplary dialog is shown in Appendix. A.
Dialog Dataset Structures. Similar to other existing task-oriented dialog systems (Eric et al., 2019; Rastogi et al., 2020; Moon et al., 2020), each turn of SIMMC-VR’s dialog data consists of NLU (and NLG) intent and slot labels (e.g. "How do their prices compare?" → REQUEST:COMPARE, slots: price, objects: [1, 4]), as well as object references (a unique object ID across the same room environment) like SIMMC 2.0. In SIMMC-VR, due to the newly introduced active dialog phase and the richer dialog scenarios (object-centric flows), the list of intents is expanded as compared to SIMMC 2.0 (see Section 2.2 and Appendix. A.2).

heading: 2.2 SIMMC-VR Dataset Analysis
text: Table 4 shows the essential dataset statistics. In total, SIMMC-VR contains 4K dialogs with the corresponding videos (equating to 95.3K utterances).
Videos. We set the frame per second (fps) as 10.0, which roughly leads to an average of 1.2K frames per video (∼2 minutes length). On average there are 24.6 visible objects in the key video frames.
Dialog Acts & Flows. Each algorithmically generated flow, i.e. the meta-agenda-induced objectcentric flow (Section 2.1.2), is capped to have at most 5 different dialogs with randomly sampled dialog goals and intents. The average number of utterances is 23.4, significantly larger than that in SIMMC 2.0 (10.4). Its length distribution over different turns is shown in Figure 4a. SIMMC-VR extends SIMMC 2.0’s annotation to a set of 5 dialog acts (e.g. INFORM, REQUEST) and 17 activities (e.g. REFINE, DIRECTION_TURN). Figure 4b shows their frequency breakdown and the complete lists are in Appendix. A.2. A visualization of dia-
log transition is shown in Figure 5 to illustrate the diversity and patterns of our generated dialog flows. Figure 4c plots the coreference distances according to how many utterances separate the mentions.

heading: 2.3 Novel Challenges to SIMMC 2.0
text: SIMMC 2.0 shares the general goal of achieving multimodal task-oriented dialog systems for fu-
ture real-world and VR applications. However, the active and rich multimodal contexts of SIMMCVR introduce the following new challenges: (1) Anchoring egocentric videos as visual contexts, SIMMC-VR requires the spatial and the additional temporal multimodal reasoning, posing new categorical patterns of object coreferences and associated user/assistant utterances. (2) The novel dialog simulation pipeline allows for more diverse and realistic interactions (e.g. navigation and localization scenarios) with a number of transitory dialog actions and viewpoints, many of which have not been studied in the previous datasets. This results in the higher degree of complexities in conversational tasks – for instance, the coreference resolution task gets significantly harder with a much larger number of objects mentioned in a dialog (13.2 vs. 4.7 in 2.0), and with the increased average utterance counts (23.4 vs. 10.4 in 2.0). (3) SIMMC-VR requires that a perception model maintains object correspondences across their variations from dif-
ferent angles and disjoint viewpoints over time, to ensure the correctness of their resolution. While this requirement poses a practical challenge for a real-world application, a robust solution has not been explored especially for its use in the context of the multimodal dialog management.

heading: 3 SIMMC-VR Task Formulation
text: The SIMMC-VR is created to help AI models cope with realistic shopping scenarios and assist human users in real-world applications in AR/VR. To investigate the (multimodal) conversational and assistive abilities of current AI systems in this immersive and situated environment, we propose four main bechmarking tasks leveraging the created dataset. Several tasks inherit from SIMMC 2.0 with additional challenges brought by the nature of active user scenes and expanded dataset annotations.

heading: 3.1 Multimodal Dialog State Tracking
text: Following SIMMC 2.0, in SIMMC-VR we retain the multimodal dialog state tracking (MM-DST) task, which aims at inferring structured information for understanding and planning out dialog policies/actions, with dialog utterances and/or multimodal contexts given. Each DST is required to resolve both the dialog intents (as a dialog act) and the user request slots, which is mainly evaluated by the F1 scores of the predicted slots and intents.

heading: 3.2 Multimodal Coreference Resolution
text: It is crucial for an assistant to be able to recognize objects that a user is referencing, either within the current visual context, or any previously mentioned items.Therefore, for each environment, a canonical ID is uniquely assigned to each object as the target for multimodal coreference (MM-Coref) resolutions, where the mentions can be resolved by both the dialog context (e.g. "Add the shirt I liked to the cart.") and the multimodal context (e.g. "How does the red shirt next to the jeans com-
pared to the one before?"). Following SIMMC 2.0, we allow the models to take ground-truth bounding boxes as inputs to bypass the needs for perfect visual detectors. The evaluation metric is the F1 scores for the predicted object IDs. Note that as the multimodal contexts are videos, the models are implicitly conditioned to identify the frames that likely contain the target objects, leading to comprehensive multimodal spatial-temporal reasoning. Additionally, while there are no explicit textual coreference annotations, the models are still implicitly required to perform textual coreference resolution for those utterances mentioning the same objects from prior dialogue turn(s).

heading: 3.3 Failure-Mode Prediction
text: SIMMC-VR features user failure-modes that simulate users accidentally failing to correctly follow the assistant guidance. In this task, given a dialog snippet (consisting of utterances in the active phase) and the video frames surrounding it, we ask the model to predict whether the current user actions correctly follow the instructions or not (i.e. binary classification evaluated by F1 scores). The task is highly multimodal as the model needs to understand the sophisticated active grounding of the visual and dialog contexts. During the training time, we pre-sample the same amount of negative samples to make the labels balanced.

heading: 3.4 Dialog Response Generation
text: This task requires a trained dialog agent to generate the assistant responses (measured in BLEU4 (Papineni et al., 2002)), given user utterances as well as the resolved multimodal information (belief states and referred canonical object IDs). Note that even though the aforementioned information is given as ground-truths, the generation still needs to conform to natural language responses that do not contain flattened DSTs or object IDs (e.g. INFORM:COMPARE, (OBJ_ID:
5,9)→ "The white and blue shirts differ by ...").

heading: 4 Modeling & Experimental Analysis
text: In this section, we introduce the investigated baseline models to perform a preliminary benchmarking of the proposed dataset, where we hope to inspire more sophisticated and tailored modeling efforts from the community for future research. Dataset Split. For the empirical modeling analysis and performance benchmarking, we randomly split the dataset into 3 sets: train (70%), dev (5%), and test (25%) sets, while ensuring both domains (fashion and furniture) have the same split distributions. Baselines. To benchmark the dataset, we adopt: (a) MM-DST Model is a 12-layered multi-task GPT-2 model (Radford et al., 2019; Kottur et al., 2021) trained with joint supervision signals from MM-Coref, MM-DST, and response generation tasks, inspired by causal language modeling approach to dialog systems (Peng et al., 2020; Hosseini-Asl et al., 2020). The inputs to the model include both the dialog context (utterances) and the multimodal contexts flattened as structurally formatted text strings, where the outputs are the predicted DST labels. This baseline has two versions: one uses the ground-truth multimodal contexts provided from the scene generator (hence a soft oracle) to simulate the outputs from a robust object detector or from a controlled VR environment, whereas the other has to infer visual descriptors from raw videos, simulating real-world scenarios. (b) Adapted-VIOLET Model is a multimodal video-language model based on VIOLET (Fu et al.), adapted to fit our task structure (Figure 6). Due to computational limitations, we randomly sub-sample 10 − 15 video frames during train-
ing (while ensuring a proportion of these frames contain objects of ground-truth coreferences), and sweep through the entire video for test-time inference with a fixed window-size. In addition to the frame-level whole image feature, we feed the dense object descriptor features extracted in each groundtruth bounding boxes (assuming a perfect object detector) to the model for the MM-Coref task.5
All baseline models are trained for ten epochs, and the best model on the dev set is used for test.

heading: 4.1 Experimental Results
text: Table 5 summarizes the model performance and the probabilistic guess performance (proportional to training label distributions) for each sub-task. Main Results. The baselines show strong overall performances especially in the DST task. The MM-Coref is understandably a very challenging task (resolving tens of items over moving frames), as evidenced in the relatively low scores – suggesting areas for future research. It is worth noting that without the ground truth multimodal contexts for assistant turns, the MM-DST model performs close to zero, indicating that the created dataset does not leak unintended artifacts for the object mentions (that language-only models can easily exploit without visual contexts). For the failure mode prediction, we prepare a test-set that focuses on the active scene utterances, where the random guess roughly equates to the amount of the failure probabilities (30%). We expect the future modeling efforts can better perceive discrepancies between the visual behaviors and the instructed guidance. Effects of Temporal Grounding. We break down the MM-Coref performance by identifying coref utterances with temporal dependencies. With the Adapted-VIOLET model, we get an F1 of 10.5 for
5Here to simplify the task, our dataset can also be approached without assuming any perfect vision modules.
utterances without temporal dependencies, and a significantly lower 2.81 for the others – suggesting the difficulty in encoding long-standing contexts. Comparison with 2.0. We also include the MMDST model performance for the SIMMC 2.0 dataset as a reference, to signify the new challenges that SIMMC-VR brings with the active VR-streams and the complex multimodal dialog flows.

heading: 5 Related Work
text: The proposed work addresses unique requirements for a task-oriented assistant on smart glasses, making it a first-of-its-kind – while complementing other related works within multimodal NLP.
SIMMC (Moon et al., 2020; Kottur et al., 2021) is a class of research areas that the proposed work builds upon, which addresses using virtual environments to simulate a co-observing multimodal dialog agent. Moving away from the sanitized and static scenes that they concern for the limited use cases, SIMMC-VR introduces several additional challenges as summarized in Section 2.3.
Several models (Kung et al., 2021; Senese et al., 2021; Lee and Han, 2021; Huang et al., 2021b) are proposed for the SIMMC benchmark tasks – primarily focusing on grounding dialogs on visual objects from a single image. Taking inspirations from these works, we extend the models to accommodate temporal dependencies within frames.
Multimodal Dialog Datasets. Many of the existing literature in multimodal dialogs (Das et al., 2017b; Hori et al., 2018; Kottur et al., 2019; de Vries et al., 2017, 2018; Le et al., 2021) typically assume asymmetric visual information between two observers, i.e. questioner and answerer, where conversational goals are limited to reducing information asymmetry (similar to VQA). In contrast, we study task-oriented dialog scenarios – an assistant co-observes the same scene as a user does, thus focusing on serving user requests to achieve functional goals (e.g. giving recommendations).
The embodied AI dialog systems (Gao et al., 2022; Padmakumar et al., 2022), on the other hand, study the scenarios where a human participant teaches an AI agent a set of skills or gives navigational directions – hence posing an opposite role to an AI agent. While it is an important area to study, its distribution of utterance patterns is completely different and therefore not applicable for our target domain – building a situated AI assistant.
Egocentric Video Datasets. With the popularity
of wearable devices, several datasets (Grauman et al., 2022; Lv et al., 2022; Damen et al., 2021) are released to study the unique properties of egocentric videos. Our work also features similar visual properties, while adding conversational layers that showcase an assistant use case of such videos. Task-Oriented Dialog Systems (Henderson et al., 2014; Rastogi et al., 2019; Budzianowski et al., 2018b; Eric et al., 2019) have long been studied to support various assistant scenarios (e.g. booking hotels). Our work takes its roots in this line of work – focusing on predicting user belief states and dialog acts to achieve functional goals – and extends it to a unique multimodal setting.
A popular thread in the task-oriented dialog system modeling is to fine-tune end-to-end causal LLMs (Hosseini-Asl et al., 2020; Peng et al., 2020; Chao and Lane, 2019; Gao et al., 2019; Crook et al., 2021). We extend this line of work and propose a multimodal extension to account for visual inputs.

heading: 6 Conclusions
text: We present SIMMC-VR, a situated and interactive dialog dataset that features immersive VR streams as multimodal contexts, simulating realistic shopping scenarios along with user-assistant dialog interactions. The dataset consists of 4K user-egocentric videos paired with densely annotated dialog utterances. We build a novel meta-agenda generator for automatically synthesizing rich interactive dialogs grounded on active and diverse visual scenes, paraphrased manually for more natural speech. We propose four sub-tasks on SIMMC-VR which aims at inspiring future dialogue modeling endeavors on high-fidelity egocentric (user POV) environments; where the baseline performance highlights many challenges the dataset brings forth towards actualizing the real-world-ready VR/AR assistant. With rich annotations it provides, SIMMC-VR can as well expand beyond the proposed tasks to spur relevant future research, which includes (but not limited to): (1) augmented with speech-like spoken utterance interventions to enrich the naturalness of the dialogues, and (2) environments and room layouts beyond ones used under the scope of this paper.

heading: Acknowledgments
text: Many thanks to Renato Sanchez and Becka Silvert for their help on coordinating the paraphrases collection; the linguists from the Appen tool company for the high-quality paraphrases; and the anonymous reviewers for their constructive feedback.

heading: 7 Limitations
text: We hereby discuss the current limitations of our work: (1) The SIMMC-VR dataset, similar to the SIMMC 2.0 version, focuses on shopping scenarios (clothing and furniture purchasing domains), one of the most common everyday activities that virtual reality could enable users to do from anywhere, anytime. We have not tested whether the models would generalize to domains outside of the shopping experiences, thus we cannot speak to the transferability of our results to environments with very different visual properties than what our virtual environments provide. (2) In this dataset, we hand-design several possible dialog acts that we assume are common for human buyers, as well as their associated scenarios. This may not exhaust all the possible interactions a shopper can do with the assistant. However, we emphasize that the coverage should be sufficient for common shopping experiences. Additionally, although most of our proposed subtasks should be modeling generic user-assistant multimodal dialogue interaction and thus could be transferred well to other domains, the (our) domain specific MM-DST may not generalize as much. Nevertheless, they should still be transferable to similar (shopping) environments. (3) The audio of the SIMMC-VR videos are generated by automatic TTS, which may fall short to represent the natural human speech. However, we do not foresee this causing problems for multimodal dialog modeling, which this work mostly focuses on.

heading: 8 Ethics and Broader Impacts
text: We hereby acknowledge that all of the co-authors of this work are aware of the provided ACM Code of Ethics and honor the code of conduct. This work is mainly about collecting a multimodal taskoriented dialog dataset with primary applications in actualizing a virtual assistant in the AR/VR world. The following gives the aspects of both our ethical considerations and the potential impact to the community.
Dataset. While most parts of our created dataset are automatable, our main human annotation efforts lie in the paraphrasing phase of our templated synthetically constructed dialog utterances. We ask in total 20 workers that possess linguistic expertise to paraphrase our templated utterances with carefully designed guidance and examples. We encourage the diversity where we do not pose any limits on the background of the paraphrasor as long
as English proficiency and linguistic domain expertise is possessed.
The main annotation task is conducted via the Appen6 provided interface, where we ensure that all the personal information of the workers involved (e.g., usernames, emails, urls, demographic information, etc.) is discarded in our dataset. The designed virtual environment scenes are not intended to have any bias towards any communities, where we aim at constructing generic domain and diverse scenes.
Overall, we ensure our pay per task is well above the annotator’s local minimum wage (approximately $30-35 USD / Hour). In this work, we primarily consider English speaking regions for setting up the initial benchmark, though our dataset can be easily extended to contain multilingual annotations for learning virtual AI assistants that are capale of different languages. This research has been reviewed by the IRB board and granted the status of an IRB exempt. Techniques. We benchmark the constructed dataset with modern strong large-scale pretrained language and multimodal models with our own designs to adapt them to suit our formulated tasks. Due to the nature of our dataset (assistant AI that focuses on the needs of the human users and the surrounding environments), as well as the proposed main challenges this dataset feature (i.e. mainly focusing on resolving multimodal coreferences, tracking dialog states, and generating useful assistive responses to human users), we do not anticipate production of harmful outputs, especially towards vulnerable populations, after training models on our SIMMC-VR dataset/tasks.

heading: A Details of the Dataset
text: A.1 Digital Assets
Table 6 lists the asset categories used for constructing the SIMMC-VR dataset.
Inventories & Scenes. As the base environment inherits from SIMMC 2.0, there are around 290 and 110 digital assets for fashion and furniture items. 7 seed fashion scenes are manually created for fashion and 1 seed scene for furniture, with each seed scene rearranged 20 times (Section 2.1) to create (randomized) diverse shopping environments. We do so by randomly swapping an asset from either the same or semantically similar asset category (Table 6) to retain spatial appropriateness (e.g. avoiding collisions or over-sized to a container) of the replacement object.
The user traversal video is then planned and recorded in these environments. The number of visible distinct objects in key video frames of static dialog phase is 24.6 (Table 4), which implies quite rich multimodal contents are presented in the recorded videos.
A.2 Dialog Goals & Acts
Table 7 lists all the sub-goals during the high-level agenda for planning the overall dialog flows, with their detailed explanations.
Table 8 lists all the used dialog acts and activities. Recall that a full dialog act is composed by the act term and the activity term, e.g. REQUEST:COMPARE. Most of the activities are self-explainable. Navigational activities are separated to DIRECTION_STRAIGHT and DIRECTION_TURN, etc., to make the utterance template sampling more straightforward (as we most likely will use different ways to speak about moving straight as opposed to making turns.) The PRE_CONDITION and POST_CONDITION activities are for querying referential objects during key action points, with the former querying the objects before the action (e.g. "Turn right when you see a pair of blue jeans."), and the latter during the ending of the action (e.g. "Turn right and then you should see a pair of blue jeans."). SPATIAL_RELATIONS are for referring objects within a local object-scene-graph when sampling corresponding utterance templates, while REMEDIAL_* activities indicate that the current assistant speech is trying to get the user back on the correct track (towards an item(s) of interest).
A.3 Other Details
Failure Modes for Dialog Simulation. In reality, the user may not always perfectly follow an instruction. We model such behaviors in SIMMCVR by (with 30% probability) deliberately failing an instructed action during the path planning stage (e.g. making a wrong turn, moving further from a desired spot). We achieve this by randomly selecting one (or a few) key action points along a proper traversal path and record the opposite actions in the user video. The correct action path will then be used to derive the original instructed utterance (where the user fails to follow) and we additionally perform remedial path planning to guide the user back to the right track. Bootstrapping from Sparse Scenes. While most of the dialogs from SIMMC 2.0 have a single static image, the dataset contains a small proportion of dialogs with two randomly sampled (sparse) scenes as multimodal contexts. We propose to recover and re-purpose these sparse scenes to add onto our dataset by connecting the two scenes with a newly collected active phase navigating the user from one scene to another, with augmented conversations along the traversal. This step essentially adds diversity and depth to conversations in our dataset. For the static phase (the original two snapshots of SIMMC 2.0), we animate the scenes with user’s eye-gaze movements combined with a small amount of local wandering movements to appear more natural. We use ∼1.4K static phase conversations from SIMMC 2.0, ensuring that at least one turn of user↔assistant conversation exists in the second scene snapshot. TTS Utterances. To make the rendered user shopping videos more realistic (and comprehensive), we also perform an automatic text-to-speech synthesis (TTS) on each user and assistant utterance using an open-sourced tool, Coqui TTS (Coqui.ai, 2022). The TTS helps compute the natural duration of each utterance when spoken, which is then used to calculate the number of video frames (under a fixed frame-rate) an utterance would span. The random eye-gaze movements mentioned in Section 2.1 extend the sparsely rendered scene snapshots to the continuous video frames, synchronized with the aforementioned speech.
A.4 Data Examples
Figure 7 shows a sample sub-sampled video frames for both the fashion and furniture domains, for
qualitative purpose.
Figure 8 shows full-scale example of one of the data instance in SIMMC-VR – with some navigational utterances from the assistant shortened for brevity.

heading: B Details of Modeling
text: B.1 General Modeling
The respective author-released pretrained weights for both models (GPT-2 and VIOLET) are used for model initializations.
As described in Section 4, we use VIOLET (Fu et al.) due to the model’s architectural simplicity and convenience to adapt to our task (as well
as its remarkable performances on various videolanguage tasks). The multi-framed vision transformer stream of video encoder is suitable for the SIMMC-VR task, where we further engineer it to be able to take on dense object descriptors and dialog structures. For MM-DST, MM-Coref, and Failure Mode Prediction tasks, we mainly adopt the original VIOLET’s BERT module as its language encoder, while for response generation, we replace the BERT with GPT-2 and train the model from scratch directly on our dataset (with the visual streams initialized from pretrained weights).
B.2 Hyper-Parameters Table 9a and Table 9b report the hyper-parameters used in this work for model training and their search bounds, respectively. We simply perform a manual search trials.
B.3 Implementation Details & Hardware The implementations of the transformer-based models are extended from the HuggingFace7 code base (Wolf et al., 2020) and other cited authors’ released code-bases. Our entire code-base is implemented in PyTorch.8 All the models in this work are trained on a single Nvidia A100 GPU9 on a Ubuntu 20.04.2 operating system.
7https://github.com/huggingface/transformers 8https://pytorch.org/ 9https://www.nvidia.com/en-us/data-center/a100/
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Section 7.
3 A2. Did you discuss any potential risks of your work? Section 8.
3 A3. Do the abstract and introduction summarize the paper’s main claims? Abstract and Section 1.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 2 and 3.
3 B1. Did you cite the creators of artifacts you used? Section 4.
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Section 4.
3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Section 2 and 3 and 4.
3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Section 8.
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Section 2 and Appendices.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 2.
C 3 Did you run computational experiments? Section 4.
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Appendix Section B.1.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Section 4 and Appendix Section B.
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 4 and Appendix Section B.
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 4 and Appendix Section B.
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Section 2 and Section 8.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Not applicable. Left blank.
3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Section 8.
3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Section 8.
3 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Section 8.
3 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Section 8.

heading: Future Work
text: We propose four sub-tasks on SIMMC-VR which aims at inspiring future dialogue modeling endeavors on high-fidelity egocentric (user POV) environments; where the baseline performance highlights many challenges the dataset brings forth towards actualizing the real-world-ready VR/AR assistant. With rich annotations it provides, SIMMC-VR can as well expand beyond the proposed tasks to spur relevant future research, which includes (but not limited to): (1) augmented with speech-like spoken utterance interventions to enrich the naturalness of the dialogues, and (2) environments and room layouts beyond ones used under the scope of this paper