Title: MIntRec2.0: A Large-scale Benchmark Dataset for Multimodal Intent Recognition and Out-of-scope Detection in Conversations
Benchmark: MIntRec2.0
Summary: MIntRec2.0 is a multimodal dataset (image, text, and audio) to assess intent recognition. Scenes are sourced from TV shows with corresponding subtitles, and models must match one of 30 defined intent classes.
Contribution: The authors spend the bulk of their attention on benchmarking their own multimodal fusion approach and conduct only a small comparative experiment between ChatGPT and humans, taking ChatGPT's score as representative of LLMs.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: multimodal intent recognition
Phenomenon defined by authors: Yes
Phenomenon definition: "Understanding human intentions in multimodal scenarios... perceiving user tones, expressions, and body language" (1)
Definition scope: Comprehensive
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Contested
Task definition: Models are given a scene from a TV show, partitioned into dialogue steps, and must classify the intent of the utterance into one of 30 intent classes. "Out-of-scope" is an available class for utterances not expressing intent.
Task item definition: A sequence of dialogue turns, with corresponding image, text, and audio, including an annotation for who is speaking. The correct intent class might be "Taunt," "Criticize," "Care," ...
Task source: Produced media (TV sitcom scenes)
Task ecology: Representative task (e.g. answering medical licensing exam questions)
Dataset size: 3,230
Dataset size detail: Dialogue samples from 1,245 dialogues total
Dataset size notes: 9,989 train; 1,821 val
Dataset metadata available: Yes
Dataset metadata detail: in-scope vs. out-of-scope
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible)
Split reviewed: Test, Train, Validation
Response format: Multiple choice
Response format detail: Multi-class classification with 30 possibilities is a kind of multiple choice...
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric statistics: simple mean/sum
Metric subscores: Yes
Metric subscores detail: in-scope vs. out-of-scope accuracy
Metric access: Outputs alone
Few-shot evaluation: Yes
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: No
Results realism: No
Human baseline: Yes
Authors discuss validity: Yes
Author validity detail: "The intent taxonomies are highly applicable across various domains, offering con- siderable promise for real-world applications (Further discussions can be found in Appendix H)" (5)
Taxonomy: VQA / Understanding
New taxonomy: General Capability / Visual Understanding