Title: SIMMC 2.0: A Task-oriented Dialog Dataset for Immersive Multimodal Conversations
Benchmark: SIMMC 2.0
Summary: SIMMC 2.0 introduces a dataset for task-oriented dialogue systems in immersive multimodal shopping contexts, specifically fashion and furniture. It presents 11k user-assistant dialogues grounded in realistic VR scenes, aiming to support the development of robust multimodal virtual assistants
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: grounding, user interaction, reasoning, nlp
Phenomenon defined by authors: Yes
Phenomenon definition: The ability of virtual assistants to handle task-oriented dialogues grounded in multimodal contexts, such as co-observed VR environments and complex visual scenes.
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: The task involves an agent assisting a user in a shopping scenario (fashion or furniture) through natural language dialogue grounded in a shared multimodal context (photo-realistic VR scenes). The agent needs to understand user utterances, track dialogue state, resolve references to objects in the scene, and generate appropriate responses.
Task item definition: A single item in the dataset appears to represent a turn within a dialogue, consisting of a user utterance, the corresponding assistant response (for training/evaluation), and the multimodal context (scene snapshot) relevant to that turn, along with associated annotations like dialogue acts, object references, and belief states.
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Expert-crafted task examples (e.g. hand-written examples), Crowd-sourced task examples (e.g. Prolific-created tasks), Modified from another benchmark (e.g. translation into another language), Procedurally-generated task examples (e.g. Creating instances from a template), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task ecology: Partial real task (e.g. answering medical questions collected from real people), Representative task (e.g. answering medical licensing exam questions)
Task ecology detail: Dialogs were generated through simulation and then paraphrased by human annotators.
Dataset size: 15% of 11,244 dialogues ≈ 1,687 dialogues.
Dataset size notes: Train: 65% ≈ 7,309 dialogues. Validation: 5% ≈ 562 dialogues. Dev-test: 15% ≈ 1,687 dialogues
Dataset metadata available: Yes
Dataset metadata detail: domain (fashion or furniture), object IDs, 2D bounding boxes of objects in images, an index to additional catalogue metadata (such as price, available sizes, colour, and pattern), dialogue annotations including NLU/NLG intents, slots, and object references linked to scene objects
Sampling method: Random sample (creators defined a task space and sampled from it), Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test, Train, Validation
Response format: Multiple choice, Free response (e.g. summary paragraph, executable code), Structured response (e.g. valid JSON, API call alone)
Metric definition: Exact Match (accuracy, F1, precision, recall), n-gram (BLEU, ROUGE, chrF)
Metric definition detail: The primary metrics vary by task: Binary classification accuracy for Multimodal Disambiguation; Coref Precision/Recall/F1 for MM-Coref; Intent Accuracy and Slot Precision/Recall/F1 for MM-DST; BLEU for Response Generation (generation task) and Accuracy@k, mean reciprocal rank, mean rank for Response Generation (retrieval task).
Metric aggregation: Simple Mean
Metric statistics: simple mean/sum, mean and variance for accuracy and BLEU
Metric subscores: Yes
Metric subscores detail: Scores are provided individually for each of the four benchmark tasks: Multimodal Disambiguation, MM-Coref, DST, and Response Generation. For DST, separate scores are reported for Intent and Slot performance.
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Industry
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: The benchmark is itself realistic
Human baseline: No
Authors discuss validity: Yes
Author validity detail: SIMMC 2.0 addresses the shortcomings of SIMMC 1.0 by incorporating more complex and realistic contexts (multimodal context, number of objects, and partially observed objects, suggesting that these factors make the benchmark more challenging and closer to real-world scenarios). They also show that their baseline model achieves significantly lower performance on MM-Coref compared to the best model on SIMMC 1.0 to show that SIMMC 2.0 presents new challenges.
Taxonomy: User Interaction / 
New taxonomy: General Capability / User Interaction