MIntRec2.0: A Large-scale Benchmark Dataset for Multimodal Intent Recognition and Out-of-scope Detection in Conversations
Authors: Hanlei Zhang
Xin Wang
Hua Xu
Qianrui Zhou
Kai Gao
Jianhua Su
jinyue Zhao
Wenrui Li
Yanting Chen
Venue: ICLR 2024 poster (2024)
Keywords: intent recognition
multimodal dataset
out-of-scope detection
multi-turn conversations
benchmark framework
benchmark evaluation
Primary area: datasets and benchmarks
TL;DR: MIntRec2.0 is the first large-scale dataset for multimodal intent recognition and out-of-scope detection in multi-party conversations.

Abstract: Multimodal intent recognition poses significant challenges, requiring the incorporation of non-verbal modalities from real-world contexts to enhance the comprehension of human intentions. However, most existing multimodal intent benchmark datasets are limited in scale and suffer from difficulties in handling out-of-scope samples that arise in multi-turn conversational interactions. In this paper, we introduce MIntRec2.0, a large-scale benchmark dataset for multimodal intent recognition in multi-party conversations. It contains 1,245 high-quality dialogues with 15,040 samples, each annotated within a new intent taxonomy of 30 fine-grained classes, across text, video, and audio modalities. In addition to more than 9,300 in-scope samples, it also includes over 5,700 out-of-scope samples appearing in multi-turn contexts, which naturally occur in real-world open scenarios, enhancing its practical applicability. Furthermore, we provide comprehensive information on the speakers in each utterance, enriching its utility for multi-party conversational research. We establish a general framework supporting the organization of single-turn and multi-turn dialogue data, modality feature extraction, multimodal fusion, as well as in-scope classification and out-of-scope detection. Evaluation benchmarks are built using classic multimodal fusion methods, ChatGPT, and human evaluators. While existing methods incorporating nonverbal information yield improvements, effectively leveraging context information and detecting out-of-scope samples remains a substantial challenge. Notably, powerful large language models exhibit a significant performance gap compared to humans, highlighting the limitations of machine learning methods in the advanced cognitive intent understanding task. We believe that MIntRec2.0 will serve as a valuable resource, providing a pioneering foundation for research in human-machine conversational interactions, and significantly facilitating related applications.
The full dataset and codes are available for use at https://github.com/thuiar/MIntRec2.0.

Published as a conference paper at ICLR 2024
MINTREC2.0: A LARGE-SCALE BENCHMARK DATASET FOR MULTIMODAL INTENT RECOGNITION AND OUT-OF-SCOPE DETECTION IN CONVERSATIONS
Hanlei Zhang Xin Wang
Hua Xu
Qianrui Zhou' Kai Gao
Jianhua Su1
Jinyue Zhao
Wenrui Li"
Yanting Chen'
Tsinghua University' Hebei University of Science and Technology²
Samton (Jiangxi) Technology Development Co.,Ltd, Nanchang 330036 ,China²
ABSTRACT
Multimodal intent recognition poses significant challenges, requiring the incorpo- ration of non-verbal modalities from real-world contexts to enhance the compre- hension human intentions. However, most existing multimodal intent bench- mark datasets are limited in scale and suffer from difficulties in handling out- of-scope samples that arise in multi-turn conversational interactions. this pa- per, we introduce MIntRec2.0, large-scale benchmark dataset for multimodal intent recognition in multi-party conversations. It contains 1,245 high-quality di- alogues with 15,040 samples, each annotated within 30 fine-grat fine-grained classes, across text, video, and audio modalities. In addition to more than 9,300 in-scope samples, it also includes over 5,700 out-of-scope sam- ples appearing in multi-turn contexts, which naturally occur in real-world open scenarios, enhancing its practical applicability. Furthermore, we provide com- prehensive information on the speakers in each utterance, enriching its utility for multi-party conversational research. We establish a general framework support- ing the organization of single-turn and multi-turn dialogue data, modality fea- ture extraction, multimodal fusion, as well as in-scope classification and out-of- scope detection. Evaluation benchmarks are built using classic multimodal fu- sion methods, ChatGPT, and human evaluators. While existing methods incor- porating nonverbal information yield improvements, effectively leveraging con- text information and detecting out-of-scope samples remains a substantial chal- lenge. Notably, powerful large language models exhibit significant perfor- mance gap compared to humans, highlighting the limitations of machine learn- ing methods in the advanced cognitive intent understanding task. We believe that MIntRec2.0 will serve as a valuable resource, providing a pioneering foundation for research in human-machine conversational interactions, and significantly fa- cilitating related applications. The full dataset and codes are available for use at https: https://githus: cwm/thuaeeeccocoooooo http/hhhiia//tttee..////
intent taxonomy of
INTRODUCTION
Understanding human intentions in multimodal scenarios holds significant research importance and has broad applications, such as human-computer interaction (Xu, 2019). intelligent transportation system (Kaffash et al. (021). and medical diagnosis (Tíwari et al. 2022 Moon et al.) 2022). For instance, perceiving user tones, expressions, and body language enables better capture of user needs in intelligent customer systems. This also leads to more personalized, efficient, and natural interac- tions (Luo et al. (022).. While there emerge numerous multimodal language datasets in recent years, particularly multimodal sentiment analysis and emotion recognition (Li al.) 2019 Chudasamal ct al. 2022 Hu et al.] 2022b), few datasets provide high-quality multimodal intent resources, which significantly hampers related research. Zhang et al. (2022a) pioneered this area by formulating intent taxonomies multimodal conversational scenarios and providing 2,224 annotated utterances with
"Equal contribution. *Corresponding authors.

---

Published as a conference paper at ICLR 2024
Ioneat
Entiede
CLS
Teant
Koon DE riloue
Figure I: An example from the MIntRec2.0 dataset. More examples are provided in the Appendix A
text, video, and audio information. However, it has three major limitations: First, its scale is rela- tively small compared to other multimodal datasets (Zadeh et al. 2018b Poria et al. 2019), leading to potential overfitting and impacting the generalization ability. Second, it only includes utterances from single-turn dialogues, neglecting context and multi-party information. Third, it fails sider out-of-scope utterances, which commonly occur in dialogue systems (Larson et al. 2019) and crucial for improving system robustness.
To address these issues, we propose MIntRec2.0, a large-scale multimodal multi-party benchmark dataset that comprises 1,245 high-quality dialogues, totaling 12.3 hours. representative sample is depicted in Figure] I The construction of this dataset involves four main steps. Initially, raw videos from three TV series are collected and segmented into utterance-level portions based on timestamps. These segments are then manually grouped into dialogues in aligmment with the conversational scenes and events. Subsequently, each utterance is annotated with speaker identity information to leverage specific contextual information. Following this, we propose new intent taxonomy incorporating 30 fine-grained intent classes. An OOS tag is also added to identify utterances that do not belong any known classes, a phenomenon commonly occurred in real-world, open-ended scenarios. Lastly, six experienced workers annotate each piece of data using text, video, and audio information. The final dataset contains 9,304 in-scope and 5,736 out-of-scope samples.
We develop a general framework for multimodal intent recognition and out-of-scope detection within single-turn and multi-turn conversations. First, data imputs are organized at both utterance and dialogue levels, where the latter retrieves all the context information corresponding to the speaker in the current dialogue turn. Secondly, extract text, video, and audio features for each utter- ance. For multi-turn dialogues, context information is concatenated to the utterance in the current turn using special token as separator. Third, we perform multimodal fusion on the extracted features. Specifically, we employ two strong multimodal fusion methods (Tsai et al. 2019 Rah- man et al. (020)) to leverage nonverbal information by capturing cross-modal interactions. In the training stage, in addition to the multimodal fusion loss, cross-entropy loss is applied under the su- pervision of hard and soft targets for learning in-scope and out-of-scope data, respectively. During inference, threshold-basee based method (Shu et 2017) is adopted to both identify high-confidence in- scope and detect low-confidence out-of-scope samples. Experimental results demonstrate that using multimodal information effectively improve in-scope intent recognition accuracy and enhance out-of-scope detection robustness. Furthermore, we evaluate ChatGPT and human performance un- der a challenging setting with few-shot samples as prior knowledge. The results reveal significant performance gap of over 30% absolute scores between large language models (LLMs) and humans. Humans achieve the state-of-the-ar benchmark performance of 71 % accuracy with merely 7% of the training data, indicating this dataset is extremely challenging for existing machine learning methods.
Contributions. (1) This paper presents MIntRec2.0, the first large-scale multimodal multi- party conversational intent dataset. This dataset provides detailed annotations for both intent and speaker identity for each utterance within multimodal contexts and enables out-of-scope detection in open- world scenarios. (2) We establish a universal framework for in-scope classification and out-of-scope detection, applicable to both single-turn and multi-turn conversations, and introduce strong bench- mark baselines. (3) Extensive experiments demonstrate the effectiveness of leveraging multimodal information in intent recognition. However, considerable opportunities for enbancement persist in existing methods when compared with human performance, highlighting the challenges inherent in high-level cognitive intent recognition tasks and underscoring the value of this dataset in advancing
2

---

Published as a conference paper at ICLR 2024
Comparison of the MIntRec2.0 dataset with previous intent datasets. #I and #U represent the number of intent classes and utterances. Conv. denotes the conversational nature of the dataset. OOS and Multi-Party indicate the inclusion of out-of-scope examples and multiple speakers per dialogue, respectively. T, V, and A represent text, video, and audio information.
Table I:
Soctes
Can. Type
Mulii-Ptty
AT1
Turetal
2010
17
6,311
Single-tum
Snips
Lowcse
2018
Single Single-um
CLINCC100
Lapes ata
2019
150
23,700
x
MDID
Kmit
2010
1,209
Intenianeu
x
K
Mineite
Zhang st
20
2,234
Single-um
MieiRez200
15,040
Mulli-ttrn
/ /
related research. This dataset will be released under the CC BY-NC-SA 4.0 license, and codes will be publicly available as open source. A portion of the data accessible in supplementary materials.
RELATED WORK
This section provides a brief overview of the existing literature in benchmark datasets, multimodal fusion methods, and multimodal multi-turn conversations. Further related works focusing on video understanding and intent analysis are detailed in Appendix
Benchmark Datasets. Intent recognition is a substantial task in natural language processing (NLP) and is supported by numerous of benchmark datasets. These datasets can be broadly categorized into two branches. The first branch originates from task-oriented dialogues and includes datasets like ATIS (Tür et al., 2010), SNIPS (Coucke al.) 2018), CLINC150 (Larson al., 2019). BANK- ING77 (Casanueva et al. 2020). Notably, CLINC150 incorporates out-of-scope data to test system robustness. SIMMC 2.0 (Kottur et al.) (021) is multimodal dataset focusing on the shopping domain, but it lacks intent annotations. The second branch stems from open-endes dialogues. rep- resented by multi-turn dialogue datasets such as DailyDialog (Li et al. (017)) and SWBD (Godfirey et al. [192). However, these datasets primarily offer dialogue acts and may not be well-suited for applications requiring specific intent classes. In recent years, there has been a growing interest in multimodal language datasets for both single-turn (Zadeh al. 2016 2018b Yu al. 2020) and multi-turn dialogues (Busso et al. 2008 Poria et al. 2019). EMOTyDA (Saa et al. (020 is another large-scale multimodal dataset for multi-turn dialogues, but it only includes coarse gaase-grained dialogue acts. Some studies have also explored visual or multimodal intents using image modality (Ta et al., 2021 Kruk et al., 2019). MIntRec (Zhang et al.) (022a) stands as the first multimodal intent recog- nition dataset for open-ended dialogues. MIntRec2.0 significantly expands in scale from 2,224 to 15,040 utterances and is designed to handle both out-of-scope utterances and multi-turn dialogues. A comparison between MIntRec2.0 and other benchmark intent datasets is presented in Table
Multi-modal Fusion Methods. Multimodal fusion presents prosperous development in multimodal language understanding. Early methods aim to learn cross-modal relations and single-modal prop- erties (Fukui et al.) 2016 Zadeh al. 2017 2018 8a Hazarika et al. 2020 or efficient multimodal representations (Liu et al. 2018). MulT Tssi et al. 2019) designs an effective crossmodal attention module learn adaptations across different modalities. MAG-BERT (Rahman al., 2020) inte- grates nonverbal information into pre-trained language models using a multimodal adaptation gate. MBT (Nagrani al. (022I) restricts cross- modal information flow through tight fusion bottlenecks, facilitating the connection of relevant inputs in each modality. Very recently, TCL-MAP (Zhou) jtt al. 2024) leverages prompt learning to provide high-quality supervised signals for multimodal representation learning. We also explore (MSA), such as Self-MM (Yu et al. 2021) and MMIM (Han et al., (022)) However, these methods rely on specific sentiment properties (e.g., polarity) that are not applicable to our task.
methods in multimodal sentiment analysis
Multimodal Multi-turn Conversations, Leveraging multimodal information is a hot topic in multi- turn conversations (Ghosal et al. 2019 Majumder et al. 2019: Ghosal et al., (220a).. For instance, DialogueRNN (Majumder et al.
2019) uses GRU networks to track important temporal informa-
3

---

Published as a conference paper at ICLR 2024
Table 2: Expanded intent classes in the MIntRec2.0 dataset with brief interpretations.
Enteat Cateaerias
Interpertations
Doaftt
Counry sere rifomne uncetiitty regarilngg scranee something in spesiloring with expresionn disheiiion
Expess
cospedaation
Refoe
Stow new ligg os rejecions teg, csing regaire worhh to dedise as offer reqeest.
Wan
Alas potesiaal canacn tilo signdie saan sacino
ueat
Haghtighe ieporaane craignnonee oenetinn ie speting
Auk be eperions
Regaer ethens ware er dhoart en purtorar moreri alcing for otwer perpuctrecs
Cielien
Wilkder aoorus the
werehing it.. w.l.aiinng
withon obkin
Explis
Providi as claburae accust in dariicatic ica. wiss explandary canal wouh bassach
Texke
Offer somoome pariciuuee in adiriy or even aking simore to jois cctsites lik cu)) detberning
Orgasis
cakking
fetac).
tion, including the history of speaker states and global states. MM-DFN (Hu et al. 2022a) proposes a graph-based dynamic fusion module to reduce historical redundancy while tracking the history of speaker states. Another approach is to construct multimodal fusion networks to integrate con- textual information between different modalities, such as M2FNet (Chudasama al., 2022) and MMGCN (Hu et al. 202T). However, modeling temporal contextual information with multimodal fusion representations does not yield good results (see Appendix C). Therefore, we propose a simple baseline that concatenates the context information of inputs before multimodal fusion.
THE MINTREC2.0 DATASET
Data Sources & Dialogue Division. First, we collect raw videos from three different TV series: Superstore, The Big Bang Theory, and Friends on YouTube and obtain subtitles from OpenSubtitles. We ensure the selected videos do offend user privacy and do not contain malicious content (Appendix D). We split them into continuous video segments according to the timestamps in the transcripts and extract corresponding audio segments. Then, we organize them into a series of dialogues for multi-turn dialogue intent analysis. Specifically, we manually annotate the starting and ending indices video segments for each dialogue and distinguish different dialogues based on whether they are in the same scene and episode, as suggested in (Poria et al. 2019). Besides, we establish a baseline to estimate the utterance boundary each segmented dialogue (Appendix
Table 3: Data statistics. de- notes the total number.
edx
data sources
intents classes
R7%
dialogues
1,245
utterances
15,040
scope utternnces
out-of-scope utternnees
5,736
words in utterances
118:477
unique words uilernnees
9.524
Joerage langth unerances
7.9
Muximum length of utterances
45
Averuge video clip duration
3.0
Figure 2: Ic-scope and out-of- Figure 3: Distribution of scope in- scope data distribution.
Maximum video clip duration
tents in the MIntRec2.0 dataset.
Speaker Information. In multi-turn conversations, we can leverage context information help analyze the intent conveyed by the speaker in each dialogue turn. However, context information may involve multiple speakers (c.g., there are a total of 51.5% dialogues with more than two speakers). As using context information of speakers is helpful for intent analysis (Ghosal et al. (020b), we aim to differentiate different speakers in each dialogue and annotate the identities of the speakers. Specifically, we perform annotation of 21, 7, and 6 main characters in Superstore, The Big Bang Theory, and Friends, respectively, which account for 90.4% of the data. The remaining data include other characters with fewer appearances (Refer to Appendix F for statistics of different characters).
Expanded Intent Classes. In this work, utilize the established intent taxonomy from the MIntRec dataset (Zhang et al.) (022a). However, as the dataset primarily focuses on discrete single- turn conversations, and the existing 20 intent classes are insufficient for capturing the diverse range
4

---

Published as a conference paper at ICLR 2024
of intents in continuous multi-turn conversations. To address this issue, conduct comprehen- sive analysis of the divided dialogues and collect 10 additional coarse- coarse-grained intent classes (i.c., express emotions attitudes and achieve goals). Specifically, add doubt, acknowledge, refuse, warn, emphasize to the former category, and ask for opinions, confirm, explain, invite, plan to the latter. Interpretations of both the expanded and existing intent categories can be found in Table 2 and Appendix [ respectively. Notably, these newly introduced classes account for 37.3% of the utterances in our dataset, highlighting their significance in intent understanding. The intent taxonomies are highly applicable across various domains, offering con- siderable promise for real- applications (Further discussions can be found in Appendix H).
intent tags for the two
Out-of-scope Utterances. As intents usually reside within particular contextual events (Schröder er 2014), there inevitably exist some utterances that fall outside the predefined intent categories in continuous conversational interactions, as suggested in (Larson et al. 2019). There are two common types of such utterances. First, there are statements that primarily convey factual information, such as cratement-non-oping of dialogue act covers significant proportion of ufterances in multi-turn conversations, it provides limited contribution to understanding specific and applicable intents. Second, due to the diverse and uncertain nature of human intentions, the predefined intent classes cannot cover all possible intentions in an open-wont open-world environment (Zhang et al.) 2023b), and there may exist utterances falling under open intent classes. Given the ambiguous boundary in determining specific out-of-scope utterances, we adopt a similar manner as in (Larson et al. 2019) and define them as those that do not belong to any of the existing intent classes. Taking these utterances into account in multi-turn conversations brings us closer to real-world scenarios and presents many practical applications.
defined in the 42 dialogue acts (Godfreyy et al. 1992). While this type
Annotation Process. Six college students proficient in English are employed to perform multimodal label annotation. They are provided with a comprehensive guidebook detailing intent interpretations and application scenarios and are only permitted to begin after achieving high accuracy on seed examples. The annotators are evenly divided into two groups and assigned to annotate half of the data simultaneously. To facilitate their work, a user-friendly annotation platform with unified database has been developed (Appendix D Each worker is tasked with analyzing the speaker's intention in video segment by combining text, video, and audio information. They are then required to choose from a set of 30 known intent tags, as well as an OOS tag. The final label for each utterance determined through majority voting, with at least two out of three votes required to reach a consensus. We operate under the assumption that each utterance has a single intent, and the rationale for not opting for multi-intent labeling is elaborated Appendix [ To mitigate potential issues, utterances that receive three different votes are excluded from our dataset.
Annotation Results. We have successfully collected 1,245 high-quality dialogues to create the MIntRec2.0 dataset. This dataset consists of 9,304 in-scope and 5,736 out-of-scope utterances with multimodal labels. The statistics of the dataset presented in Table 3 To assess annotation relia- bility, we calculate the Fleiss's ' kappa statistics for each of our six annotators to measure interrater reliability. The Fleiss's kappa scores range from 0.66 to 0.70, averaging 0.69. This indicates level of substantial agreement, as defined in (McHugh 2012). 2). The distribution of the dataset across three different data sources is illustrated in Figure 2 Superstore, The Big Bang Theory, and Friends contribute 53%, 22%, and 25% of the dataset, respectively. Each data source contains between 54.5% and 67.9% of in-scope utterances. The intent distribution of in-scope utterances depicted in Figure 3 demonstrating a common long-tailed distribution similar to real-world scenarios. As expected, some intents such as inform, explain, doubt, and complain are more prevalent in daily life, while others like warn, refuse,, emphasize, and flaunt tend to occur less in specific occasions and scenes. To ensure adequate training, each intent class contains more than 90 samples.
4 BENCHMARK FRAMEWORK
This section presents a general benchmark framework, illustrated in Figure 4 It includes data orga- nization, multimodal feature extraction, multimodal fusion, training, and evaluation.
Data Organization. In the case of single-turn dialogues, we utilize the pre-segmented utterance- level samples. Each individual utterance represents a complete turn of dialogue and includes corre-
5

---

Published as a conference paper at ICLR 2024
Data Oeganiztiinn
Madinndal Fritare Eetretiion
Matiiaadi Fasken
tharps
Infortnce
OOS
Figure 4: Overview of the benchmark framework for the MIntRec2.0 dataset.
sponding text, video, and audio information of one speaker. For multi-tum dialogues, we employ well-divided dialogues as described in Section 3 In particular, the utterances within each dialogue are arranged chronologically based on the order in which the speakers take their turn. To further leverage the context of the respective speaker. we attribute the corresponding speaker identity infor- mation to each utterance, as suggested in (Poria et al. 2019).
Text Feature Extraction. We select the pre-trained BERT (Devlin et al. 2018) language model as powerful backbone for processing the text modality, which has demonstrated strong performance when fine-tuned on our dataset. For each text utterance s, we first tokenize it in the required format, where L" is
i.c., [CLS], $1. Sn, [SEP], and then obtain the token embeddings ET E R IT"xD" the sequence length, and DJ the feature dimension.
Video Feature Extraction. Video features are extracted at the frame-level, as suggested in (Yu iet al. 2020, Zadeh et al. 2018b). Since video frames often contain multiple individuals, we begin by identifying regions of interest (Rols) for the speakers, using a sequence of automated procedures.
This involves scene detection, object detection (Ren al.) 2015). face detection (Zhang et al.) 2017),
face tracking, and audio visual active speaker detection (Tao et al.] 2021). as described in (Zhang al. 2022a). This process can generate more than 1,000K high-quality keyframes with speaker bounding boxes in approximately days. Next, we use these annotated Rols and employ the instance segmentation method, Mask R-CNN (He et al. 2017). pre-trained on the COCO (Lin et al. 2014) dataset to extract visual features. We utilize the well-initialized Swin Transformer (Liu et al. 2021). pre-trained on the ImageNet- IK (Deng et al. 2009) dataset, as the backbone due to its superior vision task performance. We use it to extract feature maps of each keyframe and apply RoLAlign (He et al., 2017) to convert them into fixed sizes using annotated Rols. Finally, applying average pooling to these feature maps yields the overall Rol feature embeddings EV E RRL* xD"
Audio Feature Extraction. To process the audio modality, we first use the librosa toolkit (MeFee al. 2015) to load the audio waveform data with a sampling rate of 16,000 Hz. Then, we employ WavLM (Chen et al., 2022). a speech pre-trained model to extract audio feature representations. Due to its masked speech prediction and denoising pre- pre-training strategy, has shown remarkable performance in wide range of speech tasks, outperforming other powerful speech pre-trained mod- els such as wav2vec 2.0 (Baevski et al., 2020) and HuBERT (Hsu et al., (021). Notably, excels in speaker verification and speech separation tasks, which is suitable for conversational scenarios involving multiple speakers. By utilizing WavLM, we acquire audio embeddings E E R x
Incorporating Context Information. In single-turn dialogues, we can directly extract embeddings for text, video, and audio modalities, as mentioned previously. However, in multi-turn dialogues, it is substantial to consider the context information of different modalities to gain better understanding of the conversation. To address this, we utilize the context information based on different speakers, as suggested in (Majumder et al. 2019, Ghosal et al. 2019). Specifically, for the utterance in the current turn, we first obtain the speaker identity information and then retrieval all the content from the previous dialogue turns corresponding to this speaker, which serves as the context information.
Next, we employ simple and effective method to leverage the context information by concatenating with the utterance in the current turn. Taking the context information from one turn of utterance example, for the text modality, the first sequence comprises all the token embeddings in the

---

Published as a conference paper at ICLR 2024
E sSErj The second sequence comprises the context infor- mation. We remove the first token [CLS] and concatenate the remaining embeddings with the first E2 sequence: PICLS)
current turm: E [CLS):
Ej
Tas Tan
E
E
[SEP]- Besides, we include segment embeddings to aid in understanding the relationships between current and contextual utterances. The segment embed- dings for the first and second sequences are encoded as zero and one vectors, respectively, with the same length as the token embeddings. For nonverbal modalities, we insert a one-dimensional zero vector between the feature embeddings of the two sequences to distinguish them. If additional con- text information is available, such as more contextual utterances, we append each of them to the end of the latest context utterance using the same operation as the second sequence.
E 20 E
E Laa
[SP!]
After extracting multimodal features, our goal is to utilize multimodal fusion techniques to capture cross-modal interactions and exploit complementary information from differ- ent modalities to further enhance intent recognition capability. Specifically, we use E EV and EA as inputs and feed them into a multimodal fusion network F to obtain multimodal representa- tions F(E", ,E E In this work, adopt two strong multimodal fusion methods, namely MAG-BERT (Rahman et al. 2020) and MulT' (Tsai et al. 2019) as baselines.
Multimodal Fusion.
Training. Following multimodal fusion, we employ the multimodal representations z for training. For in-scope samples z {zi 1y: E } we perform classification on z using the cross entropy loss LcE, where N is the number of training samples, and y is the set of K known intent labels. For out-of-scope samples nut {z, ly/ & V we apply the outlier exposure (OE) (Hendrycks al,, 2018) loss, denoted as Log, to distinguish them from the in-scope samples and enhance the model's robustness and its generalization ability for out-of-scope samples. Specifically, we use a uniform distribution over the K known classes as soft targets. The definitions for losses are as follows:
1 LCE N
Σ log
exp(o(zi")u) K Σn exp(o(z)))) fOR = N
log
exp(o(z(ut))
K
exp(0(zqut)mm)
where o(.) is the classifier with a linear layer. The training loss Lraain LCE + Loe + Frusion where Crusion is the loss specified in different multimodal fusion methods. Besides, we also conduct experiments by training a (K+1)-way classifier with out-of-scope samples grouped as the (K+1) thh class, resulting in significant decrease in the performance of in-scope classification (Appendix K
Inference. During inference, our goal is to both identify in-scope classes and detect out-of-scope samples. To accomplish this, we employ a threshold-basee open world classification method in NLP called DOC (Shu et al.) 2017), which performs well in our experiments. This method rejects low- confidence samples, assigning statistical thresholds to each known class. For each sample z; the predicted probability of the htth class is given by p(k|z.)) probabilities from each class of the training samples to calculate the corresponding class threshold òj. Specifically, we fit them to one half of the Gaussian distribution with μ and calculate the standard deviations σμ using two symmetric halves of the probabilities. The class threshold is then given by & max(0.55 1 adx)), where a usually works well. test sample detected as oft-of-scope if p(k z,) ) < 5g. vk E y. Otherwise, it is considered as an in-scope sample and is assigned the predicted class with the maximum probability, denoted as y/ = argmax p p(k(z)). 22)).
Sigmoid (((z))) We use the output
5 EXPERIMENTS
Implementation Details. We partition our dataset into training, validation, and testing sets, main- taining an approximate ratio of 7:1:2 for both dialogues and utterances. (Further details are provided in Appendix [L. For the text modality, we utilize BERTLARGE as a powerful backbone consisting of 24 transformer layers implemented in the Huggingface transformers library (Wolf et al. 2020). to extract features with the dimension D of 1024. For the video modality, we employ well-traines checkpoints of Mask R-CNN from the MMDetection toolbox (Chen et al. 2019) to extract features with the dimension DV of 256. For the audio modality, we use the pre- tranned model WavLM, im-
plemented in Wolf et al. 2020) to extract features with the dimension D4 of 768. In single-turn dia-
7

---

Published as a conference paper at ICLR 2024
Table 4: Benchmark baseline results on the MIntRec2.0 dataset.
In-scape Classification
Insscage Omn-ff-aaaee Clossiltation
Tmin
Methods
ACC
WP
F1-15
ACC
F1-005
58.01
43.37
MAG-BERT
55.17
57.78
55.10
60.55
59.08
59.98
46.48
44.80
34,05
46.08
W/o.0S
A (MAG BERT)
3,577
2:317
3.797
1.287
1,67
L.131
3,111
1.561
3,631
3.121
MulT
60.66
59.55
45.65
46.14
(Mull)
2.521
2,557
2.467
1.367
1,271
2:281
2,901
8.171
2.461
TEXT
52,08
51.57
52.11
59.99
58.63
58.65
45.83
55.61
61,54
46.34
54.84
60.12
56.20
w: 0OS
A (MAG BERT)
1,561
0.271
1.681
0,13
0.49
0,181
1.69
0.591
0,931
1,661
MulT A (MulT)
52.72
56,45
$2.56
60.18
58.82
$9.38
46.88
$6.00
61.66
47.35
0.641
1.881
0.457
0.207
0.731
1,05
0.391
0.121
1.017
Context TEXT
53.61
54,46
54.10
59.04
$8.69
$9.07
46.42
56. 12
63.56
46,98
Ceatest MAG-RERT
53.89
55.72
54.21
59.84
59.41
60 22
46
56.70 70
62,52
47.25
OOS
(Contest BERT)
1,261
0.721
0.321
0,081
1.14
Context MuTT
$3.96
54.91
54.15
59.33
60.04
46.45
56.07
62.93
46,96
((Comeett MolT)
0.351
0.451
0.051
0.447
0.647
0.771
0.031
0.05]
0.63)
0.00
logues, we apply zero-padding with a maximum sequence length 50, 180, and 400 for text, video, and audio features, respectively. The number of training epochs is set to 40, and the training batch size is set to 16 for all baselines. We employ Adam (Loshchilov & Hutter (019) for optimization, implement our approach using PyTorch 1.13.1, and conduct experiments on Tesla V100-SXM2- 32GB GPUs. For all experiments, we report the results averaged over five runs, using random seeds ranging from 0 to (Additional hyper-parameters details available in Appendix M).
Benchmark Baselines. As text is the predominant modality in conversational multimodal intent recognition (Zhang et al.) 2022a), we establish a robust baseline by fine-tuning BERTLARGE, com- paring its performance with two multimodal fusion methods: MAG-BERT and MulT. We evaluate these methods in both single-turn and multi-turn conversations, focusing on in-scope classification and out-of-scope detection. For single-turn conversations, we use only in-scope utterances for train- ing. The oft-of-scope utterances are included in the testing set and treated separate class, following (Lin & Xu 2019) Zhang et al., 2023b). For multi-turn conversations, we consider both in-scope and out-of scope samples at the dhalogue-level during training, and all the baselines uti- lize the context information as described in section We conduct additional baselines related to dialogue intent classification in NL.P and out-of-distribution detection across different sources in Appendices N and O respectively. Besides, we test the performance of ChatGPT on our dataset using both zero-shot and few-shot settings. In the zero-shot setting, ChatGPT provided with the prompts of the label sets (e.g., 30 intent labels and one OOS) and an introduction to the task. In the few-shot setting, we use 10 dialogues with 227 utterances that cover all intent classes as the learning data (Details of the utilized prompts can be found in Appendix PP. Finally, we invite ten evaluators to assess human performance. Each worker assigned an equal portion of the testing set, ensuring they have not seen the data before. They receive the same background information as that provided to ChatGPT to ensure a fair comparison. Additionally, we provide them with more prior knowl- edge, consisting of 100 dialogues and 997 utterances, explore human potential in addressing this complex problem. We average the predictions from all evaluators to obtain the final score.
Evaluation Metries. To evaluate the in-scope classification performance, we adopt six commonly used metrics: Fl-score (F1), Precision (P), Recall (R), Accuracy (ACC), Weighted Fl (WF1), and Weighted Precision (WP). To evaluate out-of-scope detection performance, we utilize four metrics commonly employed in open intent classification (Shu et al., 2017 Zhang et al. 2023b): Accuracy, Macro Fl-score over all classes, In-scope classes (FI-IS), and the Out-ol-scope class (F1-OOS).
Results. The performance of benchmark baselines on the MIntRec2.0 dataset is presented in Table 4 this table, A denotes the improvements achieved by multimodal fusion meth- ods compared to the text baseline using the current evaluation metric. For single-turn dia- logues, we conduct experiments on two settings: training without out-of-scope samples (w / OOS) and with out-of-scope samples (w OOS). It is evident that when only in-scope utterances are available, all multimodal fusion methods significantly outperform the text baseline. MAG- BERT and MulT demonstrate 1~4% increase in scores across all metries. Moreover, we ob-
8

---

Published as a conference paper at ICLR 2024
serve that multimodal fusion methods show larger proportion of improvement of over 2% in- crease in almost all settings when involving out-of-scope samples. This suggests that mod- eling cross-modal interactions and utilizing complementary information not only enhances in- scope identification but also remarkably improves the model robustness of out-of-scope detection. After using out-of-scope data for training, we find that all baselines may suffer a slight decrease in some in- scope evaluation metrics but gain significant improve- ments in out-of-scope detection with an increase of over 30% in F1-OOS scores. Though incorporating mul- timodal information brings improvements on all met- rics, they show less increase compared with the former setting, indicating the challenges of effectively utiliz- ing multimodal information on out-of-scope data. For multi-turn dialogues, multimodal fusion methods yield
Table Performance of ChatGPT and humans the MIntRec2.0 dataset.
In scoecppe
ln-scope Out scope
Methods
WFI
WP
ACC
F1-00S
F1
MAG-BERI-10
11.58
15:34
50.57
3.75
ChaGPPT-
35.27
37.10
48.22
27.68
21.21
28.34
ChaGPP--1
34.53
36.39
49.27
29.72
27.85
28.41
Humans- 10
64.34
67.82
72,00
60.43
62.83
57.83
Humans
71.03
75.63
81.83
71.86
75.41
69.49
improvements in all metrics for in-scope classification compared with the text baseline. However, it shows minor improvements or even decrease when testing on a mixture of in-scope and out-of-scope data. This also indicates that there remain sub- stantial opportunities explore the potential of multimodal information in conversational contexts.
ChatGPT v.s. Humans. Finally, we present the performance of ChatGPT and humans in Ta- ble [ As humans typically excel at learning from few-shot samples and quickly grasping new concepts (Lake al.] 2015), we apply a challenging setting with only 10 dialogues of 227 utter- ances. Multimodal fusion baselines, such as MAG- BAG-BERT-10, struggle significantly in this setting, easily overfitting and falling into trivial solutions, such as predicting the most frequent in-scope or out-of-scope class, due to the challenges posed by imbalanced and few-shot training samples. In contrast, ChatGPT demonstrates much better performance even without prior knowledge of la- beled data (ChatGPT-O), which shows its strong language understanding and reasoning capabilities, comprehending complex textual semantics and understanding human intentions (Bang et al. 2023). Besides, ChatGPT shows overall improvements with 1~-6% score increase across most metrics with only 10 dialogues for training (ChatGPT-10). This suggests that ChatGPT can learn from prior knowledge and enhance intent recognition capability. Notably, it achieves a significant 6% im- provement in FI-OOS, highlighting its improved out-of-scope detection robustness. However, when humans are provided with the same prior knowledge of 10 dialogues (Humans- 10), they achieve an increase of over 30% in scores across almost all metrics compared to ChatGPT. This demonstrates that humans can effectively leverage limited multimodal information to understand high-level inten- tions and discern between known and unknown boundaries, highlighting the significant limitations of existing Al methods this challenging task, To further explore human potential, we observe their performance with additional knowledge of 100 dialogues of 997 utterances (Humans- 100). Com- pared with Humans-10 10, they achieve over a 10% improvement in almost all metrics and achieve the otate-of-the-art benchmark performance. This also underscores the advantages of humans in mastering this complex task by leveraging multimodal information.
6 CONCLUSIONS
This paper presents MIntRec2.0, a pioneering dataset for multimodal intent recognition, encompass- ing 1,245 dialogues and 15,040 multimodal utterances. This marks MIntRec2.0 the first large- scale dataset in this domain. The dataset includes annotations for speaker identity and introduces comprehensive taxonomy of 30 intent classes, spanning 9.304 in-scope utterances. To evaluate model robustness, 5,736 out-of-scope utterances are also annotated. We propose a general frame- work for organizing data, extracting multimodal features, and performing multimodal fusion for in-scope classification and out-of-scope detection in both single-turn and multi-turn conversations. Extensive experiments reveal the substantial potential of using multimodal information and uncover significant opportunities for improvement in effectively utilizing out-of-scope data and context in- formation. Moreover, even with strong LLM such as ChatGPT, using text-only modality remains challenging in scenarios with limited prior knowledge, highlighting the importance and challenge of using multimodal information compared to human performance. The limitations and potential negative societal impacts of this work are discussed in Appendix Q

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This work was supported by the National Natural Science Foundation of China (Grant No. 62173195), the National Science and Technology Major Project towards the new generation of broadband wireless mobile communication networks of Jiangxi Province (03 and 5G Major Project of Jiangxi Province) (Grant No. 20232ABC03402), High-level Scientific and Technological Inno- vation Talents "Double Hundred Plan" of Nanchang City 2022 (Grant No. Hongke Zi (2022) 321-16), and supported by Hebei Natural Science Foundation (Grant No. F2022208006). We would like to thank Jiayan Teng, Zhaochen Yang, Shaojie Zhao, and Hao Li for their efforts during dataset construction.
REFERENCES
Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A frame- work for self-supervised learning of speech representations. In Proceedings of the Advances in Neural Information Processing Systems, volume 33, pp. 12449-1222 12460, 2020.
Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai. Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, et al. A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity. arXiv preprint arXiv:2302. 02220402,, 2023.
Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for video understanding? In Proceedings of the International Conference on Machine Learning. volume 2, pp. 4, 2021.
Hervé Bredin and Antoine Laurent. End-to-end speaker segmentation for overlap-aware resegmen- tation. In Proceedings of the 22nd Annual Conference of the International Speech Communication Association, pp. 31131111 3111-3115, 2021.
Hervé Bredin, Ruiqing Yin, Juan Manuel Coria, Gregory Gelly, Pavel Korshunov, Marvin Lavechin, Diego Fustes, Hadrien Titeux, Wassim Bouaziz, and Marie-Philippe Gill. Pyannote.audie Neural building blocks for speaker diarization. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 7124-7128 2020.
Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jean- nette N Chang, Sungbok Lee, and Shrikanth S Narayanan. Iemocap: Interactive emotional dyadic motion capture database. Language resources and evaluation, 42(4):335-35) 2008
Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. In proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp- 6299-6308, 2017.
Inigo Casanueva, Tadas Temcinas, Daniela Gerz, Matthew Henderson, and Ivan Vulic. Efficient intent detection with dual sentence encoders. Proceedings of the 2nd Workshop on Natural Lan- guage Processing for Conversational AI, 2020.
Kai Chen, Jiaqi Wang. Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, et al. Mmdetection: Open mmlab detection toolbox and benchmark. arXiv preprint arXiv: 1901.00055715 2019.
Sanyuan Chen, Chengyi Wang. Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, et al. Wavlm: Large-scale self-supervised pre-training for full stack speech processing. IEEE Journal of Selected Topics in Signal Processing, 16(6):1505- 1518, 2022.
Zifeng Cheng, Zhiwei Jiang, Yafeng Yin, Cong Wang, and Qing Gu. Learning to classify open intent via soft labeling and manifold mixup. IEEE/ACM Transactions Audio, Speech, and Language Processing, 30:635-664 6433,666,222 2022.
Vishal Chudasama, Purbayan Kar, Ashish Gudmalwar, Nirmesh Shah, Pankaj Wasnik, and Naoyuki Onoe. M2fnet: multi-modal fusion network for emotion recognition in conversation. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1662- 2022.
10

---

Published as a conference paper at ICLR 2024
Alice Coucke, Alaa Saade, Adrien Ball, Théodore Bluche, Alexandre Caulier, David Leroy, Clément Doumouro, Thibault Gisselbrecht, Francesco Caltagirone, Thibaut Lavril, et al. Snips voice plat- form: embedded spoken language understanding system for private-by-desigee voice interfaces. arXiv preprint arviv: 1805. 10190, 2018.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 248-255, 2009.
Jacob Devlin, Ming-Wey Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 1rXiv:1810.04800 04805, 2018.
Jeffrey Donahue, Lisa Anne Hendricks, Sergio Guadarrama, Mareus Rohrbach, Subhashini Venu- gopalan, Kate Saenko, and Trevor Darrell. Long-term recurrent convolutional networks for visual recognition and description. In Proceedings of the IEEE conference on computer vision and pat- tern recognition, pp. 2625-2634, 2015.
Christoph Feichtenhofer, Axel Pinz, and Andrew Zisserman. Convolutional two-stream network fusion for video action recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1933-1941, 2016.
Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recogmition. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 6202-6211, 2019. 2019.
Akira Fukui, Dong Huk Park, Daylen Yang, Anna Rohrbach, Trevor Darrell, and Marcus Rohrbach. Multimodal compact bilinear pooling for visual question answering and visual grounding. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, 2016.
Deepanway Ghosal, Navonil Majumder, Soujanya Poria, Niyati Chhaya, and Alexander Gelbukh. Dialoguegen: A graph convolutional neural network for emotion recognition in conversation. In Proceedings the the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natural Language Processing, pp. 154-164, 2019.
Deepanway Ghosal, Navonil Majumder, Alexander Gelbukh, Rada Mihalcea, and Soujanya Poria. Cosmie: Commonsense knowledge for emotion identification conversations. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 2470-2481, 2020,
Deepanway Ghosal, Navonil Majumder, Rada Mihalcea, and Soujanya Poria. Utterance-level dia- logue understanding: An empirical study. arXiv preprint arXiv:2009.1333 13902, 2020b.
John J Godfrey, Edward C Holliman, and Jane McDaniel. Switchboard: Telephone speech corpus for research and development. In Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing, pp. 517-520, 1992.
Wei Han, Hui Chen, and Soujanya Poria. Improving multimodal fusion with hierarchical mutual information maximization for multimodal sentiment analysis. In Proceedings the 2021 Con- ference on Empirical Methods in Natural Language Processing, pp. 9192, 2021.
Kensho Hara, Hirokatsu Kataoka, and Yutaka Satoh. Can spatiotemporal 3d cnns retrace the history of 2d enns and imagenet? In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 6546-6555, 2018
Devamanyu Hazarika, Roger Zimmermann, and Soujanya Poria. Misa: Modality-invariant and- specific representations for multimodal sentiment analysis. In Proceedings of the 28th ACM In- ternational Conference on Multimedia, pp. 1122-1131, 2020.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference computer vision, pp. 2961-2969, 2017.
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of distribbtin examples in neural networks. In Proceedings of the 5th International Conference on Learning Representations, 2017.
11

---

Published as a conference paper at ICLR 2024
Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. In Proceedings of the International Conference on Learning Representations, 2018.
Yutai Hou, Yongkui Lai, Yushan Wu, Wanxiang Che, and Ting Liu. Few-shot learning for multi- label intent detection. In Proceedings of the AAAI Conference on Artificial Intelligence, vol- ume 35, pp. 13036-13044 2021.
Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:3451-3460 2021.
Dou Hu, Xiaolong Hou, Lingwei Wei, Lianxin Jiang, and Yang Mo. Mm-dfn: Multimodal dynamic fusion network for emotion recognition in conversations. In ICASSP 2022-2022 IEEE Interna- tional Conference on Acoustics, Speech and Signal Processing, pp. 7037-7041. IEEE, 2022a.
Guimin Hu, Ting-En En Lin, Yi Zhao, Guangming Lu, Yuchuan Wu, and Yongbin Li. Unimse: To- wards unified multimodal sentiment analysis and emotion recognition. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 7837-7851, 2022b. 2022b.
Jingwen Hu, Yuchen Liu, Jinming Zhao, and Qin Jin. Mmgen: Multimodal fusion via deep graph convolution network for emotion recognition in conversation. In Proceedings of the 59th An- nual Meeting of the Association for Computational Linguistics and the I I1th International Joint Conference on Natural Language Processing, pp. 5666-5675, 2021.
Menglin Jia, Zuxuan Wu, Austin Reiter, Claire Cardie, Serge Belongie, and Ser-Nam Lim. Intenton- omy: a dataset and study towards human intent understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12986- 12996, 2021.
Sepideh Kaffash, An Truong Nguyen, and Joe Zhu. Big data algorithms and applications in in- telligent transportation system: review and bibliometric analysis. International Journal of Production Economics, 231:107868, 2021.
Satwik Kottur, Seungwhan Moon, Alborz Geramifard, and Babak Damavandi. SIMMC 2.0: A task-oriented dialog dataset for immersive multimodal conversations. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 4903 4912, 2021.
Julia Kruk, Jonah Lubin, Karan Sikka, Xiao Lin, Dan Jurafsky, and Ajay Divakaran. Integrating text and image: Determining multimodal document intent in Instagram posts. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Interna- tional Joint Conference on Natural Language Processing. pp. 4622 0622,463,210 2019.
Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. Human-level concept learning through probabilistic program induction. Science, 350(6266): 1332-13333112233 2015.
Stefan Larson, Anish Mahendran, Joseph J Peper, Christopher Clarke, Andrew Lee, Parker Hill, Jonathan K Kummerfeld, Kevin Leach, Michael A Laurenzano, Lingjia Tang, al. An eval- uation dataset for intent classification and out-of-scope prediction. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pp. 1311-1316, 2019.
Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. Dailydialog: A manually labelled multi-turn dialogue dataset. In Proceedings of the Eighth International Joint Conference on Natural Language Processing, pp. 986-995, 2017.
Yinfeng Li, Chen Gao, Xiaoyi Du, Huazhou Wei, Hengliang Luo, Depeng Jin, and Yong Li. Au- tomatically discovering user consumption intents in meituan. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 3259-3269, 2022.
Yuanchao Li, Carlos Toshinori Ishi, Koji Inoue, Shizuka Nakamura, and Tatsuya Kawahara. Ex- pressing reactive emotion based on multimodal emotion recognition for natural conversation in human-robot interaction. Advanced Robotics, 33300): 2019.
12

---

Published as a conference paper at ICLR 2024
Shiyu Liang Yixuan Li, and R. Srikant. Enhancing the reliability of outoof-diitribution image detection in neural networks. In Proceedings of the International Conference on Learning Repre- sentations, 2018.
Ting- En Lin and Hua Xu. Deep unknown intent detection with margin loss. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 5491-5496, 2019.
Ting-En En Lin, Hua Xu, and Hanlei Zhang. Discovering new intents via constrained deep adaptive clustering with cluster refinement. In Proceedings of the AAAI Conference on Artificial Intelli- gence, pp. 3360-8367,2022
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Proceedings of the European conference on computer vision, pp. 740-755. Springer, 2014.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 10012-10022, 2021.
Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu. Video swin trans- former. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3202-3211, 2022.
Zhun Liu, Ying Shen, Varun Bharadhwaj Lakshninarasimhan, Paul Pu Liang, AmirAli Bagher Zadeh, and Louis-Philippe Morency. Efficient low-rank multimodal fusion with modality-specific factors. In Proceedings the 56th Annual Meeting of the Association for Computational Linguis- tics, pp. 2247-2256, 2018
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In Proceedings of the International Conference on Learning Representations, 2019.
Bei Luo, Raymond YK Lau, Chunping Li, and Yain-Whar Si. A critical review of state-of-the-ard chatbot designs and applications. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 12(1),e1434, 2022.
Navonil Majumder, Soujanya Poria, Devamanyu Hazarika, Rada Mihalcea, Alexander Gelbukh, and Erik Cambria. Dialoguernn: An attentive rnn for emotion detection in conversations. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pp.
2019.
Brian McFee, Colin Raffel, Dawen Liang, Daniel P Ellis, Matt McVicar, Eric Battenberg, and Oriol Nieto. librosa: Audio and music signal analysis in python. Proceedings of the 14th python in science conference, pp. 18-25, 2015.
Mary L McHugh. Interrater reliability: the kappa statistic. Biochemia medica, 22(3):276-282, 2012.
Jong Hak Moon, Hyungyung Lee, Woncheol Shin, Young-Hak Kim, and Edward Choi. Multi-modal understanding and generation for medical images and text via vision-language pre-training. IEEE Biomed. Health Informatics, 26(12):6070 00800000000000000000000000000000 2022.
Arsha Nagrani, Shan Yang, Anurag Arnab, Aren Jansen, Cordelia Schmid, and Chen Sun. Atten- tion bottlenecks for multimodal fusion. In Proceedings of the Advances in Neural Information Processing Systems, 2021.
Bolin Ni, Houwen Peng, Minghao Chen, Songyang Zhang, Gaofeng Meng, Jianlong Fu, Shiming Xiang, and Haibin Ling. Expanding Language-image pretrained models for general video recogni- tion. In Proceedings of the European Conference on Computer Vision, pp. 1-18. Springer, 2022.
Soujanya Poria, Devamanyu Hazarika, Navonil Majumder, Gautam Naik, Erik Cambria, and Rada Mihalcea. MELD: A multimodal multi-part calli--prrt dataset for emotion recognition in conversations. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pp. 527- 536, 2019.
13

---

Published as a conference paper at ICLR 2024
Libo Qin, Tianbao Xie, Wanxiang Che, and Ting Liu. A survey spoken language understanding: Recent advances and new frontiers. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, pp. 4577-4584, 2021.
Wasifur Rahman, Md Kamrul Hasan, Sangwu Lee, AmirAli Bagher Zadeh, Chengfeng Mao, Louis- Philippe Morency, and Ehsan Hoque Integrating multimodal information in large pretrained transformers. In Proceedings of the 58th Anmual Meeting of the Association for Computational Linguistics, pp. 2369, 2020.
Shaoqing Ren, Kaiming He, Ross B Girshick, and Jian Sun. Faster r-enn: Towards real-time object detection with region proposal networks. In Proceedings of the Advances in neural information processing systems, 2015.
Tulika Saha, Aditya Patra, Sriparna Saha, and Pushpak Bhattacharyya. Towards emotion-aided multi-modal dialogue act classification. Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pp. 4361-4372, 2020.
Tobias Schröder, Terrence Stewart, and Paul Thagard. Intention, emotion, and action: A neural theory based on semantic pointers. Cognitive science, 38(5):851-880) 2014.
Lei Shu, Hu Xu, and Bing Liu. Doc: Deep open classification of text documents. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 2911-2916, 2017.
Janienke Sturm, Olga Houben-van Herwijnen, Anke Eyck, and Jacques M. B. Terken. Influencing social dynamics meetings through a peripheral display. In Proceedings of the 9th International Conference on Multimodal Interfaces, pp. 263-270, 2007.
Ruijie Tao, Zexu Pan, Rohan Kumar Das, Xinyuan Qian, Mike Zheng Shou, and Haizhou Li. Is someone speaking? exploring long-term temporal features for audio-visual active speaker detec- tion. In Proceedings of the 29th ACM International Conference on Multimedia, pp. 3927-3935, 2021.
Abhisek Tiwari, Manisimha Manthena, Sriparna Saha, Pushpak Bhattacharyya, Minakshi Dhar, and Sarbajeet Tiwari. Dr. can see: Towards a multi-modal disease diagnosis virtual assistant. In Proceedings of the 31st ACM international conference on information & knowledge management, pp. 1935 1935-1944, 2022.
Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, Zico Kolter, Louis-Philippe Morency, and Ruslan Salakhutdinov. Multimodal transformer for unaligned multimodal language sequences. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp- 6558-6569, 6569, 2019.
Gökhan Tür, Dilek Z. Hakkani-Tür, and Larry P. Heck. What is left to be understood in atis? 2010 IEEE Spoken Language Technology Workshop, pp. 19-24, 2010.
Limin Wang, Yu Qiao, and Xiaoou Tang. Action recognition with trajectory-pooke deep- convolutional descriptors. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4305-4314, 2015.
Limin Wang, Yuanjun Xiong, Zhe Wang. Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool. Temporal segment networks: Towards good practices for deep action recognition. In Proceedings of the European conference on computer vision, pp. 20-36. Springer, 2016.
Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7794-7803, 2018a.
Yu Wang, Yilin Shen, and Hongxia Jin. A bi-model based mn semantic frame parsing model for intent detection and slot filling. In Proceedings the 2018 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pp- 309-314, 2018b.
14

---

Published as a conference paper at ICLR 2024
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Transformers: State-of-the art natural language processing. In Proceedings of the 2020 conference empirical methods in natural language processing: system demonstrations, pp. 38 2020.
Saining Xie, Chen Sun, Jonathan Huang, Zhuowen Tu, and Kevin Murphy, Rethinking spatiotem- poral feature learning: Speed-accuracy trade-offs in video classification. In Proceedings of the European conference on computer vision, pp. 305-321, 2018.
Puyang Xu and Ruhi Sarikaya. Exploiting shared information for multi-intent natural language sentence classification. In Proceedings of the 14th Annual Conference of the International Speech Communication Association, pp. 3785-3789, 2013.
Wei Xu. Toward human-centered ai: perspective from human-computer interaction. interactions, 26(4):42-46, 2019.
Guangfeng Yan, Lu Fan, Qimai Li, Han Liu, Xiaotong Zhang, Xiao-Ming Wu, and Albert Y. S. Lam. Unknown intent detection using gaussian mixture model with an application to zero-shot intent classification. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. pp. 1050-1060. Association for Computational Linguistics, 2020.
Wenmeng Yu, Hua Xu, Fanyang Meng, Yilin Zhu, Yixiao Ma, Jiele Wu, Jiyun Zou, and Kaicheng Yang. CH-SIMS: A Chinese multimodal sentiment analysis dataset with fine fine-grained annotation of modality. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 3718-3727, 2020.
Wenmeng Yu, Hua Xu, Ziqi Yuan, and Jiele Wu. Learning modality-specifie representations with self-supervised multi-task learning for multimodal sentiment analysis. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pp. 10790-10797, 2021.
Amir Zadeh, Rowan Zellers, Eli Pincus, and Louis-Philippe Morency. Mosi: multimodal cor- pus of sentiment intensity and subjectivity analysis in online opinion videos. arXiv preprint 1606v:1606.06259 2016.
Amir Zadeh, Minghai Chen, Soujanya Poria, Erik Cambria, and Louis-Philipee Morency. Tensor fusion network for multimodal sentiment analysis. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 1103-1114, 2017.
Amir Zadeh, Paul Pu Liang, Navonil Mazumder, Soujanya Poria, Erik Cambria, and I Louis-Philipp Morency. Memory fusion network for multi-view sequential learning. In Proceedings of the AAAI Conference Artificial Intelligence, 2018a.
AmirAli Bagher Zadeh, Paul Pu Liang, Soujanya Poria, Erik Cambria, and Louis-Philiphe Morency. Multimodal language analysis in the wild: Cmu-mosei dataset and interpretable dynamic fusion graph. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguis- tics, pp. 2236-2246, 2018b.
Chenwei Zhang, Yaliang Li, Nan Du, Wei Fan, and S Yu Philip. Joint slot filling and intent detection via capsule neural networks. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 5259-5267, 2019.
Hanlei Zhang, Xiaoteng Li, Hua Xu, Panpan Zhang, Kang Zhao, and Kai Gao. TEXTOIR: an integrated and visualized platform for text open intent recognition. In Proceedings of the Joint Conference the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pp. 167-174, 2021a.
Hanlei Zhang, Hua Xu, and Ting-En En Lin. Deep open intent classification with adaptive decision boundary. Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp- 14374-14382, 2021b.
Hanlei Zhang. Hua Xu, Ting-En En Lin, and Rui Lyu. Discovering new intents with deep aligned clustering. Proceedings of the AAAI Conference on Artificial Intelligence, (4315-)14366-44377 May 2021c.
15

---

Published as a conference paper at ICLR 2024
Hanlei Zhang. Hua Xu, Xin Wang, Qianrui Zhou, Shaojie Zhao, and Jiayan Teng. Mintrec: A new dataset for multimodal intent recognition. Proceedings of the 30th ACM International Conference Multimedia, pp. 1688-1697, 2022a.
Hanlei Zhang, Hua Xu, Xin Wang, Fei Long, and Kai Gao. clustering framework for unsuper- vised and semi-supervised new intent discovery. IEEE Transactions Knowledge and Data Engineering. 2023a.
Hanlei Zhang. Hua Xu, Shaojie Zhao, and Qianrui Zhou. Learning discriminative representations and decision boundaries for open intent detection. IEEF/ACM Transactions on Audio, Speech, and Language Processing, 31:1611-1625 1-1623, 2023b.
Shifeng Zhang, Xiangyu Zhu, Zhen Lei, Hailin Shi, Xiaobo Wang, and Stan Z. Li. S3fd: Single shot scale-invariant face detector. In Proceedings of the IEEE International Conference on Computer Vision, Oct 2017.
Yuwei Zhang, Haode Zhang, Li-Ming Zhan, Xiao-Ming Wu, and Albert Lam. New intent discovery with pre-training and contrastive learning. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, pp. 256-269, 2022b.
Qianrui Zhou, Hua Xu, Hao Li, Hanlei Zhang, Xiaohan Zhang, Yifan Wang, and Kai Gao. Token- level contrastive learning with modality-aware prompting for multimodal intent recognition. In Proceedings of the 38th AAAI Conference on Artificial Intelligence, 2024.
Yunhua Zhou, Peiju Liu, and Xipeng Qiu. Knn-contrastive learning for out-of-domain intent clas- sification. In Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics, pp. 5129-5141, 2022.
Yunhua Zhou, Guofeng Quan, and Xipeng Qiu. A probabilistic framework for discovering new intents. Proceedings of the 61 Annual Meeting of the Association for Computational Linguis- fics, pp. 3771-3784 2023. 2023.
A SAMPLE SELECTION WITHIN THE MINTREC2.0 DATASET
Figure 5
iillustrates diverse selection of samples from our MIntRec2.0 dataset to showcase repre- sentative examples. The selected samples cover all 30 intent categories and the OOS label.
B ADDITIONAL RELATED WORK
Video Understanding. As a significant research field within computer vision, video understanding involves the extraction of valuable information from video content. Numerous methods have been developed to handle spatial and temporal data in videos, including the Two-Stream method, which comprises TDD (Wang et al. 2015). LRCN (Donahue al. 2015), Fusion (Feichtenhofer al.) 2016), and TSN (Wang et al. 2016). This methodology integrates secondary path to learn a video's temporal information by training convolutional neural network on the optical flow stream. How- ever, these methods require extensive computation and storage capacity due to the pre-computation of optical flow.
To address this, researchers introduce 3D convolutional neural networks (3D CNNs) such as I3D (Carreira & Zisserman (017). R3D (Hara al. 2018), S3D (Xie et al., 2018). Non-local (Wang ct al. 2018a). and SlowFast (Feichtenhoter et al. 2019). More recently, self attentive mechanisms like TimeSformer (Bertasius et 2021 and Video Swin Transformer (Liu et al., 2022) are demon- strating exceptional performance in image and video tasks. TimeSformer encodes video frames into a sequence of two-dimensional images, employing temporal self-attention to understand temporal relationships, while Video Swin Transformer partitions the input video into two-dimensionall spa- tial and one-dimensional temporal patches, applying self-attention and cross-attention to manage long-disse distanee temporal dependencies. X-CLIP (Ni et al., 2022),, CLIP-based method, has achieved state-of-the-al performance in video understanding by processing video content through matching video frames with text data.
16

---

Published as a conference paper at ICLR 2024
While these techniques show proficiency in action recognition, they encounter difficulties when at- tempting to understand fine-grained intentions with high-level semantics and require considerable computational resources. For instance, X- demonstrates subpar performance demands a substantial amount of GPU memory, underscoring the need to incorporate other modal- ities such language and acousties in multimodal intent recognition tasks. Consequently, we have established baselines using multimodal fusion methods in this work.
task and
Intent Analysis. Intent analysis is an important research area in spoken language understand- ing (Qin et al., 202T). It plays a pivotal role in task-oriented dialogue systems, enabling the recog- nition of user queries' intentions alongside the slot filling task (Wang al.] 2018b Zhang et al. 2019). However, early research usually focus on the closed-world classification problem. lacking the capability to handle out-of-scope utterances encountered in real-world scenarios (Zhang et al., 2021a). To address this challenge. [in & Xu (2019) first explore this task by employing margin loss to detect unknown intent. /Zhang et al. (2021b) learn adaptive decision boundaries for each known class, thereby further reducing the open space risk. Yan et al. (2020) use Gaussian mixture models to tackle this problem and extends the task to zero-shot intent detection. Cheng et al. (2022) construct out-of-scope samples using manifold mixup technologies and employed soft labels for representa- tion learning. Zhou et al. (202)) enhance intent representations to balance both empirical and open space risks with the aid of contrastive learning in the K-nearest neighbors space.
In practical applications, out-of-scope utterances may contain multiple fine-grained intent classes, making the discovery of potential new intent classes highly valuable for industry applications, such as dialogue and user-modeling systems (Lin et al., 22200; Li et al., 2022). Lin et al. (2020) formulate this task in a semi-supervised manner, with limited labeled data for known intents and vast amount of unlabeled data for both known and new intents. To address this task. [in et al. (2020); ZZhng al.) (202bb) identify group-level known and new intent clusters by learning from both strong and weak pairwise supervised signals. Zhang et al. (2021c); Zhou et al. (2023) employ centroid-bate based alignment strategies to generate high-quality and specific pseudo-lahels for self-supervised learn- ing. However, these methods perform poorly in purely unsupervised scenarios. However, these methods have shown limited success in purely unsupervised scenarios, Zhang et al. (2023a) propose groundbreaking approach in unsupervised new intent discovery utilizes unsupervised pre-training with strongly augmented data, followed by effective clustering. This method leverages historical centroid information for initialization and employs cluster assignments learn discriminative rep- resentations at both the instance and cluster levels, marking a significant advancement over previous ofte-offthheart methods.
C PERFORMANCE OF DIALOGUERNN
Table 6: Results of DialogueRNN on the MIntRec2.0 dataset.
In In-scope Classitication
In In-ecope + Out of scope Classificaiion
Setting
F1
R
ACC
WFI
WP
F1-15
ACC
F1-00S
Fl
K+1
0.67
0.58
3.34
10.7
2.15
1.77
0.36
16.65
34.82
1.47
Outlier Exposure
2.75
3.74
3.89
3.23
5.29
221
11.10
23,67
To leverage context information, existing methods typically use multimodal fusion representations to directly model the temporal information of contexts. However, we find this approach be in- effective for our task. Specifically, we select DialogueRNN (Majumder et al. 2019), a method specifically designed for multimodal emotion detection in conversations, for evaluation. We con- duct experiments under two settings: K+1 and Outlier Exposure. The former treats the out-of-scope class as the (K (Kh class and trains using both K intent classes and one out-of-scope class, while the latter employs the outlier exposure loss on out-of-scope data during training.
As illustrated in Table DialogueRNN demonstrates significantly low performance across all met- rics. Furthermore, we observe that it tends to fall into trivial solutions, predominantly predicting most utterances as the out-of-scope class. This observation suggests that leveraging temporal in- formation with fused multimodal representations remains considerable challenge. Consequently, we adopt a simple method to leverage context information by concatenating the context information from the inputs of each modality.
17

---

Published as a conference paper at ICLR 2024
Lesa
Powerky
Faer's
Fanari
Los dide
Tear?
Erglee
Oppse
Thaak
Iufaee
Wy dat Jen
Fateciyy
Bos
orde
Look whill graniod sem
Bait Butoory
Torc
Figure Samples of the MIntRec2.0 dataset.
18

---

Published as a conference paper at ICLR 2024
D DATA PRIVACY AND CONTENT CONSIDERATIONS
Our dataset is meticulously curated and consists exclusively character names and dialogues sourced from television shows, ensuring infringement the privacy or disclosure of personal in- formation pertaining to real individuals. We have rigorously reviewed the content to maintain a high standard of decorum, assiduously avoiding any material that could be construed as offensive. Our focus remains strictly confined to the dialogues and interactions, all contextualized within the nar- rative framework of the respective shows, allowing for comprehensive understanding of character dynamics without compromising ethical standards.
E UTTERANCE BOUNDARY ESTIMATION
To further validate the accuracy of these boundaries, we conduct additional experiments using metric known Speaker Boundary Error Rate (SBER), commonly employed in speech diarization tasks (Sturm et al. 2007 This metric quantifies the difference between predicted and reference speaker boundaries, with a lower SBER indicating better performance and serving as proxy for sentence boundary accuracy. We utilize an end-to-end method implemented with pyannote (Bredin) jet al. 2020; Bredin & Laurent 2021], a pre-trained speaker change detection model, to predict speaker IDs, starting times, and durations for each utterance within dialogue segment. These predictions are then compared to the ground truth.
The results show an average SBER of 0.59 across all dialogues, suggesting considerable room for improvement in automatic sentence boundary segmentation. We believe this approach offers a rea- sonable method for evaluating utterance boundary performance.
F STATISTICS OF CHARACTERS
To further analyze the character distribution in each of the three data sources (i.e., Superstore, Friends, The Big Bang Theory) within our dataset, we present the proportions of characters from these sources in Figure Figure 7 and Figure X
In Superstore, seven main characters and 21 recurring characters are observed. can be noted that the seven main characters represent significant proportion of nearly 80%, distributed uniformly. Friends have six main characters who constitute about 85% of the data, also distributed uniformly. The Big Bang Theory has seven main characters, while their distribution imbalanced, a property we preserve due to the distinctive nature of each speaker. It is worth noting that there are other characters involved in the conversations, contributing 9.3%, 14.4%, and 5.9% respectively in each of the three TV series. These characters are also differentiated within each dialogue in our experiments.
G INTENT TAXONOMIES DEFINED IN THE MINTREC DATASET
The MintRec dataset /Zhang et al. 2022a) introduces hierarchical intent taxonomy, including two coarse-graines and 20 fine- ane-graimed intent categories. The two searse-grained classes include Express Emotions or Artitudes and Achieve Goals. Based on these, it further includes 11 and 9 fine-grained classes for them, respectively. In particular, Express Emotions or Attinudes contains complain. praise, apologize, thank, criticize, care, agree, oppose, taunt, flaunt, and joke. Achieve Goals contains inform, advise, arrange, introduce, confort, leave, prevent, greet, and ask for help. The interpretations of these categories are shown in Table referring to (Zhang et al. 2022a).
H APPLICATION OF INTENT LABELS
Our intent labels can be generalized to many domains, including intelligent customer service, health- care, mental health therapy, hazard detection, virtual assistants, and personalized recommendation systems. For instance:
complain, criticize, confort: These labels are instrumental in identifying potential mental health concerns in patients and can be pivotal in healthcare settings.
19

---

Published as a conference paper at ICLR 2024
Superstore's characters
Cheyenne
5.7%
14.2%
Mynte
9.%
Corrit
Justie
0.5%
Janel
1.3%
Jary
Sanar
2.8
JulT
4,3%
Emily
Marcss
3.2%
Nioo
9.2%
-Kely
2.45
Jonsh
16.1%
Others 9.3%
Figure 6: Proportions of characters from the TV series of Superstore.
Friends's characters
Chardler
Phoebe
12.2%
Moxica
14.5%
Jocy
13.5%
Ross 15.8%
Otbers
14.4%
Figure Proportions of characters from the TV series of Friends.
The Big Bang Theory's characters
Penny 11.5%
Howard
6.0%
Sheldon
22.7%
Any
31.5%
Barsulete
Rajes 4,0%
Others 9%
Figure 8: Proportions of characters from the TV series of The Big Bang Theory.
20

---

Published as a conference paper at ICLR 2024
warn, prevent, OOS: These labels can be employed effectively in systems designed for hazard detection.
ask for help. inform: These labels are particularly suited for customer service platforms. praise, complain, agree: These labels can be harnessed in personalized recommendation engines.
the majority of these intent labels: These labels are ideal for virtual robots designed to interact naturally with users.
Table 7: Intent taxonomics of the MIntRec dataset with brief interpretations.
Intent Categories
Interpretations
Complain
Express dissatisfintion
something g. saying encounters
expression helpless mation).
Praise
Express admiration for
something (e.g. saying with appreciative expression).
Apologize
Express regret for doing something wrong g. saying words apology such
Express gratitude in word or deed for the convenience or kindness given or edfered by others (e.g. saying words
Thank
appreciation such thank you).
Express
Critcizee
Point out and emphasize someone's 's mistakes (e.g.. yelling out someone' 's problems).
emotions
Care
Concern about somsone or be cursess about something (e.g. wocrying about someone's 's health).
stlitues
Agrer
Have the same sltitade about something is g. saying allirmative worth such as yeah and yes).
Oppose
an inconsistent
about something (e g. saying nogative words express disagreement)
Taunl
ridicule g. complimentng Bous about onesell gain admiration, umyy praise (e.g. stying something complimentary about oneself aruganlly).
Use metaphors exegaeratioss
with negative expression).
Hauns
loke
Say something provoke laughter (eg. saying something funny and exaggersted with cheerful expression).
Inform
Tell somcone to make them awan of something (e.g. broudasting somothing with a microphonc).
Advise
Offer suggesions for considertioon (e.g. saying words that make suggestions)
Arninge
Plin or arganize something (e requesting someone what they should do feemally)
Communicate
sopuintance another recommend something (e.g. describing ideniily person the proparties an ohjoet).
Achieve
Intradace
goals
Alleviste pain with encoragement compussion g. deseribing something hopeful).
Leave
Get away from somewwiere (e.g. soying where while turning around or getting up).
Prevent
Make someone unable do something (e stop somoone from doing something with a hand).
Greet
Express munsal kindness recognition during the encounter (eg.. waving someone and saying hello).
Ask for help
Request somcone help (e.g. asking someone to deal with the trouble).
I MULTIMODAL INTENT ANNOTATION PLATFORM
We have developed an efficient platform featuring unified database for multimodal label annota- tion, aiming to facilitate seamless interaction between annotators and the diverse set of multimodal data. The interface of this platform is depicted in Figure g This user-friendly interface allows anno- tators to access transcripts and associated videos from the dialogues and data sources easily, thereby ensuring accurate and consistent annotations. Annotators simply need to select one label from the 30 intent classes and an out-of-scope (OOS) tag by clicking a button. This intuitive design mini- mizes the learning curve for annotators and accelerates the annotation process. Once annotation complete, the selected labels are automatically recorded in the database for statistical analysis.
This systematic approach ensures the reliability and consistency of the annotated data, which is crucial for training robust and high-performing models. The platform not only aids in the efficient collection of annotated data but also serves as a valuable tool for exploring and understanding the intricate relationships between different modalities and intents.
J SINGLE-INTENT ASSUMPTION
In real-world scenarios, it is possible for multiple intents coexist among the 30 pre-defined classes in a single utterance. In this work, we obey the single-intent assumption due to the following two reasons:
Single vs. Multi-Intent Datasets: Most existing single-turn intent datasets in NLP, such SNIPS, CLINC, and BANKING, focus on single-intention labeling. This is also true for multi-tum dialogue datasets like SWBD (Godfrey et al.) 1992) and DailyDialog (Li et al. 2017), which generally assume single dialogue act label at the utterance level. Therefore,
21

---

Published as a conference paper at ICLR 2024
Florr
Cipas
Clact
Coject
View
Add endrenidgge
Nakis
Figlet
Unuee
Figure 9; The interface of the annotation platform.
while multiple intentions could theoretically exist in an utterance, the prevailing practice is to identify a primary intent for the sake of clarity and brevity.
Applicability Real-World Scenarios: We have examined multi-intent datasets like Standford_LU (Hou et al. 2021) and (Xu & Sarikayal 2013). These datasets often in- clude action and slot labels (e.g., find music movie, request address route), which more suited for task-oriented dialogue systems. Such labeling is generally not applicable in real-world multimodal scenarios, as suggested in (Zhang
2022a).
To verify assumption, we conduct additional multi-intent annotation on the testing set. Six annotators are asked identify up to three probable intents for each utterance. The results are shown in Table
Table 8: Statistics of multiple intents in one utterance.
Express emotions or attitudes
Classes
complain, praise, apologize, thank, criticize, care, agree, warn
Number
9,5, 1, 8. I,
Classes
oppose, taunt, flaunt, joke. doubt, ac acknowledge. refuse, emphasize
Number
7, 2, 14, 3. 1,8
Classes
inform, advise, arrange, introduce, comfort, leave, prevent
Achieve
Number
5. I, 2, 4,
goals
Classes
greet, ask for help, ask for opinions, confirm, explain, invite, plan
Number
1, I,2
The results show that only 136 out of 3,230 utterances (4.2%) have second most probable intent, and none have a third. This suggests that multi-intent scenarios are relatively rare, reinforcing the adequacy of our single-intent taxonomy. In summary, our findings align with those of most existing benchmark intent datasets, indicating that our intent taxonomy is both general and distinguishable enough for real-world applications.
K (K+1)-WAY CLASSIFICATION PERFORMANCE
We also investigate another prevalent method, the (K+1)-way classification, to utilize the out-of- scope samples during training. In other words, train both the K known classes and one
22

---

Published as a conference paper at ICLR 2024
out-of-scope class. The results of this approach are displayed in Table noticeable decrease of approximately 10% in in-s classification performance across numerous metrics (c.g.. Fl-score, recall, accuracy, weighted F1) is observed, compared to the results obtained with outlier exposure (OE) as depicted Table 4 in the paper. Although there are slight improvements in F1-0OS (2% score increase) for out-of-scope detection in most methods, these methods still underperform when recognizing known classes and in overall performance. Therefore, we opt for outlier exposure as more effective technique deal with out-of-scope samples and adopt this approach in our work.
Table K+1 classification results on the MIntRec2.0 dataset.
Tn-scope Classification
In-scope Out-of-sopp Classiication
Mechods
R
ACC
WFI
F1-15
ACC
F1-008
37.42
MAG BEKE
40.68
53.34
36.57
43.75
48.95
63.14
38.87
55.76
64.41
39.70
MulT
39.48
54.96
34.90
42.47
45.04
64.17
38.26
56.33
65.48
39.14
Context TEXT
40.33
50.45
36.97
43.72
47.50
59.18
38.21
54.65
63.79
Context MAG BERT
43.14
53.20
39.34
47.09
$1.70
62.53
40.87
55.65
64.04
41.62
Context MUlT
34.72
31.54
35.30
68.88
42.59
L DATA SPLITS
We partition our dataset into training, validation, and testing sets at an approximate ratio 7:1:1 for both utterances and dialogues. Detailed statistics for each set, encompassing both in-scope and out-of-scope data, are presented Table
I0
Table 10: Data splits of the MIntRec2.0 dataset. # denotes the number.
Item
Dialogues
# Utterances
# In-scope Utterances
# Out-of-scope Utterances
Total
1,245
15,040
9,304
5,736
Training
871
9,989
6,165
3,824
Validation
125
1,821
1.106
715
Testing
249
3,230
2,033
1.197
M HYPER-PARAMER CONFIGURATIONS
The comprehensive configurations of hyper- parameters used our experiments are presented in Table I1
Table 12
Table 13 Table 14 Table 15, and Table [6
Table 11: The hyperparameters of the TEXT baseline single-turn conversations.
Setting
hyperparameters
value
Setting hyperparameters
value
eval monitor: train size:
accuracy
eval monitor: train batch
accuracy
16
16
eval batch size:
8
eval batch size:
8
test batch size:
8
test batch size:
wait patieece:
8
wait patience: num num-train taamm-rrineepppee epochs:
8
o OOS
num mummrrin_epae spochs..
40
W OOS
40
parmupp-roportine
0.1
warmup proportion:
0.1
Ir: weight decay:
2c-5
Ir:
lc-5
0.1
weight decay:
0.1
N DIALOGUE INTENT CLASSIFICATION IN NLP
We have conducted experiments to benchmark our dataset with two sfate-of-the-ar algorithms in
open intent detection for NLP: DA-ADB (Zhang al. 20235) and KNNCL. (Zhou et al. (022) with
23

---

Published as a conference paper at ICLR 2024
Table 12: The hyperparameters of the MAG-BERT baseline in single-turn conversations.
Setting
hyperparameters
value
Setting
hyperparameters
value
need aligned::
True
need aligned::
True
eval monitor:
accuracy
eval monitor:
accuracy
train batch size:
16
train
16
eval_batch.st bitch
8
eval botchhhthsstee test test.batch size: wait patience:
8
rest_batch_size
8
wait patience::
8
num train epocks:
40
mum train epochs:
40
w o/o OOS
W OOS
beta shift:
0.005
heta shift: dropout prob: warmup proportion:
0.005
dropout prob:
0.5
0.5
warmup proppupproporioon
0.1
0.1
Ir:
5e-6
Ir: metgned.method::
5e-6
aligned method:
ctc
method:
ctc
weight decay:
0.03
weight
0.1
Table 13: The hyperparameters of the MulT baseline in single-turn conversations.
Setting
hyperparameters
value
Setting
hyperparameters
value
padding mode: podding Joc: need aligned: evul monitor:
zero
padding mode;
zero
end
padding Joc. need aligned: eval monitor:
end
False
False
accuracy
accuracy
train train_bacch_se batch_size: eval size:
16
train brachh batch_size: eval batch size:
16
test batch size:
8
test batch size:
wait pattience: num_trann_ppohtr
8
wait patience: num-srain_ce -eocchs.
40
40
dst dst_feature_dine nheads:
80
dst featuee feature_dims: dims:
80
nheads:
n levels:
8
levels: atin dropout: atin dropoud ati drrpout
altn dropout: attn dropout v:
0.0
0.0
/ OOS
W OOS
0.1
0.1
attn dropoott
relu relu dropout:
0.3
relu dropout:
0.3
embed_dropont dropout:
0.0
embed dropout:
0.0
dropout:
0.0
dropout:
0.0
output ddopput:
0.2
output dropout:
0.0
text dropout:
text text_ dropout:
grad clip:
0.5
grad clip:
0.5
attn mask: conlld ld kernel size I:
True
attn mask:
True
5
comldd kernel size
conv kemel
conv kernl
convl kernel Ir:
cowv Id kerne
5c-6
Ir:
5c-6
the open-source TEXTOIR platform (Zhang et al. 2021a). Consistent with the original settings of these algorithms, they are trained on in-scope samples and tested on both in-scope and out-of-scope samples. The results are shown in Table T7
The results show that even state-of-the-ee methods for open intent detection generally underperform compared the BERTLARGE text classifier across most metrics. However, they do excel iden- tifying out-of-scope utterances, typically achieving higher F1-OOS scores. Notably, KNNCL also scores higher accuracy.
O OUT-OF-DISTRIBUTION DETECTION ACROSS DIFFERENT SOURCES
We also explore the model performance in an out-of-distribution (OOD) setting across different sources. To address this have conducted experiments where data from source as
24

---

Published as a conference paper at ICLR 2024
Table 14: The hyperparameters of the TEXT baseline in multi-turn conversations.
hyperparameters
value
eval monitor:
accuracy
train
eval batch size: 2
test best_batchsizee
wait patiiene::
num frain toantrrraoolllll epochs:
warmup proportion:
0.1
Ir:
1c-5
weight decay:
0.1
train bruihbatch_size
Table 15: The hyperparameters of the MAG-BERT baseline in multi-turn conversations.
hyprraarameters
value
neednei:
True
eval_monitor
accuracy
train hatih_hach_stee
select batch size:
eval batch size:
test batch size:
wait pattiene:::::
num train epochs:
context
0.5
beta_shift
0.05
dropout.pron
0.05
warmup proportion:
0.01
Ir:
aligned method:
convld
weight decay:
0.1
the in-distribution dataset for training, validation, and testing. We then use data from the other two sources exclusively for OOD testing, in accordance with (Hendrycks & Gimpel 2017 Liang et al. 2018). For evaluation, we utilize a comprehensive set of metrics: AUROC (Area Under the Receiver Operating Characteristic Curve), AUPR-In (Area Under the Precision-Resd Recall Curve for in-distribution detection), AUPR-Out (Area Under the Precision-Recct
Curve for OOD detection), FPR-95 (False Positive Rate 95% True Positive Rate), and EER (Equal Error Rate). Higher scores are preferable for the first three metrics, while lower scores are desirable for the last two.
As shown in Table 18 the results indicate that MAG-BERT shows lower performance on OOD detection compared with the text baseline on most metrics. Both text and multimodal fusion methods achieve very low performance on OOD detection metrics, highlighting the substantial challenges presented by this setting. This opens up intriguing avenue for future research in OOD detection under these conditions.
P CHATGPT PROMPTS
We provide prompts for both zero-shot (ChatGPTT-O) and few-shot (ChatGPT-10)) settings of Chat- GPT. The detailed prompts are as follows:
Here is a set of given intent labels: [Acknowledge, Advise, Agree, Apologise, Arrange, Ask for help, Asking for opinions, Care, Comfort, Complain, Confirm, Criticize, Doubt, Emphasize, Explain, Flaunt, Greet, Inform, Introduce, Invite, Joke, Leave, Oppose, Plan, Praise, Prevent, Refuse, Taunt, Thank, Warn, OOS]. Additionally, OOS represents an unknown intent that does not belong to the known set of intents. Next, I will provide you with collection of dialogs: utterances. The collection contains multiple utterances presented in sequential order, and they can be considered contextualized conversations. When considering each sample and taking into account
ChatGPT-O Prompts:
25

---

Published as a conference paper at ICLR 2024
Table 16:
The hyperparamelers of the MulT bascline in multi-turn conversations.
hyperparameters
value
padding mode:
zer
padding loc:
end
need aligned:
False
evalmonnton
accuracy
train batch_aiessizee
welect_hatch_ste
eval batch size:
2
test batch size:
2
wait patience:
context length:
num Iruin -pucchs
40
dst feature dims:
80
nheads:
4
nevvlll: attn dropout:
attn drtmouroouut.v
atin dropout at
0.1
relu dropout:
0.2
embed dropout:
res dropout:
output dropout:
text dropout:
0.4
grad clip:
0.5
attn mask: convld kernel sizeel
True
convld kernel vr:
cowvld Id kernel Ir:
5c-6
Table 17: Performance of open intent detection on the MIntRec2.0 dataset.
In scope Classification
Of Classification
Methods
F1
ACC
WFI
WP
F1-1S
ACC
F1-OOS
F1
TEXT
51.60
55.47
51.31
59.30
58.01
58.85
43.37
43.24
30.40
42.96
DA-ADB
46.16
51.28
46.08
57.44
54.96
55.66
39.60
39.18
36.17
39.49
KNNCL
50.64
51.19
50.71
56.54
56.27
56.39
35.58
48.58
55.77
36.23
Table 18: OOD detection performance across different sources.
ID source
OOD source(s)
Methods
AUROC
AUPR-In AUPR-Out FPR95
EER
TEXT
51.33
21.75
80.25
93.47
49.43
Superstore
Bigbang Friends
MAG-BERT
50.96
21.28
80.14
93.74
49.21
TEXT
51.33
21.75
80.25
93.47
49.43
Bigbang
Superstore & Friends
MAG-BERT
50.96
21.28
80.14
93.74
49.21
TEXT
55.97
26.17
80.56
91.22
45.40
Friends
Bighang Superstore
MAG-BER BERT
51.01
25.62
79.81
92.57
45.90
its contextual information, please select an appropriate label from the intent label set (emphasis: you only choose intent labels from the given set of intent labels). If there set, assign the label of the sample as OOS. Please provide the output in the following format: Serial number and original text of the sample: Intent label. Apart from that, do not output anything else.
suitable labels in the
ChatGPT-10 Prompts: Here is a list of multiple multi multi-turn conversations. Each dictionary in the list represents conversation paragraph, where each key-value pair represents intent example
26

---

Published as a conference paper at ICLR 2024
a key and its corresponding label as value. Next time I will enter my request, please only reply "received". This is a list of given intent labels: [ Acknowledge, Advise, Agree, Apologise, Arrange, Ask for help, Asking for opinions, Care, Comfort, Complain, Confirm, Criticize, Doubt, Emphasize, Explain, Flaunt, Greet, Inform, Introduce, Invite, Joke, Leave, Oppose, Plan, Praise, Prevent, Refuse, Taunt, Thank, Warn, 0OS), where OOS represents unknown intent that is not intended otherwise. Now, you need to learn from the conversations that you were given in the last Q&A, and then I'll provide you with dialog that contains utterances in and these utterances are given in order and can be considered as contextual. Now, for each utterance that requires you to use the knowledge you gained from the given conversations, select label as output from the given list of labels: for the following given dialog, in this format: Original sample: Intent labels output.
Q LIMITATIONS AND POTENTIAL NEGATIVE SOCIETAL IMPACTS
This study presents several limitations that warrant acknowledgment. First, deploying this system in real-world settings necessitates collecting personal data, including facial expressions, voice, and text, thereby raising critical privacy concerns requiring meticulous attention. Second, the issue of liability remains ambiguous, especially in sensitive applications such medical diagnosis, should the technology produce erroneous results. Third, our training dataset may lack comprehen- sive representation across diverse cultural backgrounds, potentially resulting in misunderstandings or the perpetuation of stereotypes. Lastly, substantial opportunities exist for enhancing the system's 's performance, particularly in effectively utilizing context information and out-of-scope sample data and incorporating non-verbal modalities.
Limitations:
Potential Negative Societal Impacts:
While our work contributes valuable advancements in the field of multimodal intent recognition, it also has the potential to introduce negative societal impacts.
Firstly, there is the potential for misuse of our dataset if it becomes publicly available under an open- source license. Such misuse could include unauthorized commercial applications or other nefarious purposes that could result in harm. To mitigate this, we strongly urge users to adhere strictly to the licensing terms associated with this dataset.
Secondly, as Al systems like ours become increasingly sophisticated and prevalent, there is the risk of oovo-vrrlitt vver-reliance on these technologies. This could lead to a decline in certain human skills, especially those related to understanding and interpreting conversational cues. As researchers and developers, we must continue to balance the advancement of AI with the preservation and enhancement of human capabilities.
Thirdly, the baseline system might be used with malicious intent. While any technology can be used for both beneficial and harmful purposes, our system is designed detect out-of-scope (OOS) categories, which could be exploited to identify harmful or malicious intents. By integrating robust OOS detection, our system can flag conversations or utterances that deviate from predefined, accept- able intents. This feature could act as first line of defense against technology misuse, as it can be tailored detect and flag potentially harmful conversation intents.
Furthermore, establishing a benchmark in this field can have numerous positive societal impacts, such enhancing cuman-ccomptee interactions, aiding mental health assessments, and improving customer service automation. We believe the ethical deployment of this technology largely hinges on implementation safeguards and the specific contexts which it is used.
27


Reviews:
[{"review_id": "AaWbp2R9Nv", "reviewer": "Reviewer_Un9C", "summary": "This paper introduces the \"MIntRec2.0\" dataset, which offers a significant advancement in the field of multimodal intent recognition, addressing critical gaps in existing benchmarks by including multi-turn conversations, out-of-scope (OOS) utterances, and multi-party interactions. The dataset, derived from three TV series, encompasses 12.3 hours of dialogue across 1,245 conversations, totaling 15,000 annotated utterances. The expanded intent classes and inclusion of OOS labels mark a notable progression from previous datasets, aiming to bridge the gap between current benchmarks and real-world conversational scenarios.", "questions": "1. Why is the improvement from using non-verbal multimodality not more pronounced? What are the key challenges in leveraging these modalities to enhance intent recognition accuracy?\n2. Considering the complexity of intent in human interactions, what are the major influencing factors that the dataset and framework account for, and how are ambiguous or contradictory multimodal signals handled?\n3. What are the significant differences between datasets focused on human-computer interactions and the proposed human-human interaction dataset in terms of multimodal intent classification tasks?\n4. How do factors like laugh tracks or audience reactions in the TV series-based dataset influence the intent recognition process, and how do these sources differ from real-world multi-turn conversations?\n5. Were annotators asked about which modalities were most valuable in understanding conversational intent? Such insights could help compare human perception with multimodal intent recognition systems.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Dataset: The dataset's scale and inclusion of multi-party dialogues with both in-scope and OOS samples enhance its realism and applicability in human-computer interaction research.\n- Approach: The framework efficiently handles multimodal data, and the detailed annotation, including speaker information, enriches its utility for diverse conversational research.\n- Experiment: The comparison of human performance with state-of-the-art multimodal approaches in the dataset highlights the existing gap and provides a challenging benchmark for future research.", "weaknesses": "- The dataset's sourcing from only three TV series restricts the diversity of scenes and topics. This limitation might not fully represent the vast array of real-world conversational contexts.\n- The unclear performance improvement in some metrics for multimodal fusion needs a more explicit explanation. This clarification will support the dataset's effectiveness in demonstrating the advantages of multimodal approaches.\n- Additional experiments could better demonstrate the dataset's effectiveness, particularly in the nuances of multimodal intent recognition.\n- The lack of detailed discussion on the incorporation and impact of multi-modal information, especially regarding out-of-scope data, is a notable omission. More detailed analyses or case studies would illuminate the challenges and benefits of using multi-modal data.\n- The comparison between ChatGPT and human evaluations could benefit from more detail, such as the consistency of dialogue samples across experiments and how results might vary across different intent classes.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wNODXTFOEd", "reviewer": "Reviewer_SFq3", "summary": "Multimodal intent recognition is very important to natural human-computer interactions and has gained more attention in recent years. The authors released a new version of MIntRec, named MIntRec 2.0, which contains more categories and considers out-of-scope scenarios. It will support more explorations in this field.\nThere are also some limitations of this work, the authors should add more experiments to support the effectiveness of this dataset and improve the writing to make the paper more clear.", "questions": "1. What do you think are the reasons why the improvement of using non-verbal multimodality is not obvious?\n2. The intention of a speaker in interactions is very difficult and complex, so what do you think are the influencing factors?\n3. What do you think is the difference between a human-computer interaction dataset and the proposed human-human interaction dataset for the multimodal intent classification task?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This work builds a larger multimodal intent recognition dataset under interaction scences with 30 categories of fine-grained intent annotations and some out-of-scope samples, which is closer to the real-world scenarios.", "weaknesses": "1. The dataset is only from three different TV series, which limits the diversity of scenes and topics.\n2. The multimodal fusion performance is not obvious in some metrics, the authors should explain the results more clearly, which can support the effectiveness of the multimodal intention dataset.\n3. There are also some mirror errors, such as: \n1) A representative sample is depicted in Figure 5.  -> Figure 1\n2) Interpretations of both the expanded and existing intent categories can be found in Table 7 and Appendix G, respectively. -> Table 2", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Pb6J58Vjd7", "reviewer": "Reviewer_kGfg", "summary": "This paper proposes MIntRec2.0, a large-scale multimodal multi-party benchmark dataset that comprises 1,245 high-quality dialogues, totaling 12.3 hours, which is interesting and valuable for multimodal training and evaluation. The proposed dataset serves as a valuable\nresource, providing a pioneering foundation for research in human-machine conversational interactions, and significantly facilitating related applications.", "questions": "None", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The proposed dataset is interesting and valuable for research in human-machine conversational interactions. The motivation is clear, the authors proposed three limitation including single-turn utterances, scales and out-of-scope utterances. The overall structure is well organised. In addition to more than 9,300 in-scope samples, it also includes over 5,700 out-of-scope samples appearing in multi-turn contexts, which naturally occur in real-world open scenarios, enhancing its practical applicability. Furthermore, they provide comprehensive information on the speakers in each utterance, enriching its utility for multi-party conversational research. This paper is a good dataset and resource paper.", "weaknesses": "no obvious flaws. The figures should be expanded.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "N5mlwvsj5i", "reviewer": "Reviewer_ju3D", "summary": "The authors identify two main gaps in current benchmarks for assessing multimodal intent recognition systems: multi-turn conversational interactions in the real world contain out-of-scope utterances (not relevant to the intent detection taxonomy), and the existence of multiple parties/agents in dialogs.\nTo address this gap, this paper proposes the MIntRec2.0 dataset with 1.2K conversations containing 15K annotated utterances harvested from three TV series totalling ~12 hours of dialogue.\nThe authors use strong LLM baselines (ChatGPT), human evaluation, and existing methods to populate the benchmark evaluation results and identify challenges in the dataset that are not addressed by existing models.", "questions": "Q1: Given the TV series (comedy), how were things like laugh tracks or other \"audience signals\" of e.g. humor taken into account? What are the primary differences from the authors' perspective between these data sources and multi-turn conversations in the real world that intent detection systems would mainly work with?\n\nQ2: Were annotators queried about what modalities of information were most valuable to their understanding of the conversational intent? This would be helpful information to gauge the differences between how humans and MMI systems work.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "S1: The authors give detailed human performance results and identify the gap between SOTA multimodal approaches and humans. Furthermore, this indicates the benchmark is fairly difficult (not simple common sense) considering human performance of 71% with ~7% of training data.\n\nS2: Comparison of resources in Table 1 is clear and convincing; in particular this dataset seems to be the first to include multi-party dialogs, and one of the only datasets with OOS labels. The expanded intent classes for the coarse-grained \"Express Emotions\" and \"Achieve Goals\" existing intents make sense and are sufficiently distinct from one another to add value to the taxonomy.", "weaknesses": "W1: Would have liked to see more discussion on the effect of incorporating multi-modal information aside from mentions of numbers and \"indicating the challenge of using multi-modal information on out-of-scope data\". A case study or deeper slice of results would be illuminating here, or even a deeper analysis in the main paper of what primarily constitutes OOS.\n\nW2: More context should be provided on why 1-4% increases on metrics are considered significant in this case (is it statistically significant or is there some other meaning here e.g. for real life use cases?)\n\nW3: Some clarity in the ChatGPT vs. Humans evaluation would be helpful - were the 10 dialogues of 227 utterances fixed across experiments, or were other few-shot training samples of 10 dialogs with different intent class balances attempted / were metrics aggregated? It would be helpful to picture whether the metric improvements are conditioned on specific intent classes.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)