InMu-Net: Advancing Multi-modal Intent Detection via Information Bottleneck and Multi-sensory Processing

Authors: zhihong  zhu
xuxin  cheng
zhaorun  chen
yuyan  chen
yunyan  zhang
xian  wu
yefeng  zheng
bowen  xing
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-09 00:39:23

Abstract: Multi-modal intent detection (MID) aims to comprehend users' intentions through diverse modalities, which has received widespread attention in dialogue systems. Despite the promising advancements in complex fusion mechanisms or architecture designs, challenges remain due to: (1) various noise and redundancy in both visual and audio modalities and (2) long-tailed distributions of intent categories. In this paper, to tackle the above two issues, we propose InMu-Net, a simple yet effective framework for MID from the $\textbf{In}$formation bottleneck and $\textbf{Mu}$lti-sensory processing perspective. Our contributions lie in three aspects. First, we devise a $\textit{denoising bottleneck module}$ to filter out the intent-irrelevant information in the fused feature; Second, we introduce a $\textit{saliency preservation loss}$ to prevent the dropping of intent-relevant information; Ultimately, $\textit{kurtosis regulation}$ is introduced to maintain representation smoothness during the filtering process, mitigating the adverse impact of the long tail distribution. Comprehensive experiments on two MID benchmark datasets demonstrate the effectiveness of InMu-Net and its vital components. Impressively, a series of analyses reveal our denoising potential and robustness in low-resource, modality corruption, cross-architecture and cross-task scenarios.

Main paper: 
 introduction: 
 Intent detection (ID) aims to ascertain the objectives of users conveyed through their utterances, which serves as a crucial component of task-oriented dialogue systems [31, 37] . Prior studies have extensively researched ID and validated the significance of textual modality [26, 59] . However, beyond textual utterances, facial expressions and audio signals are also informative as they are often complementary and interact synergistically [50] . Therefore, multimodal intent detection (MID) has attracted increasing research attention, which is more practical in real-world scenarios.To effectively leverage the information from various modalities, numerous methods have been proposed for MID. Therein, Saha I've never been so tired. Figure 1 : An example of redundancy in the visual modality in MIntRec [60] dataset. As indicated, the utterance is only relevant to the woman highlighted in gray, and the information like the distracting background is intent-irrelevant. et al. [35] , Zhang et al. [60] pioneered MID area by introducing trimodal ID benchmarks with text, visual and audio information; Zhou et al. [61] proposed a token-level contrastive learning method with modality-aware prompting to facilitate modality fusion; Huang et al. [14] introduced a shallow-to-deep Transformer-based framework with ChatGPT-based data augmentation strategy to align different modality features, obtaining state-of-the-art (SOTA) results.Despite promising advancements achieved, we discover that existing MID models still suffer from two main issues:(1) Noise and redundancy in visual and audio modalities. As depicted in Figure 1 , there exists plenty of redundancy and noise in the visual modality, which can impede the performance of multimodal fusion. To this end, several studies proposed methods like gating mechanisms [21, 54] to filter noise and redundancy. However, an underexplored aspect is the potential of these fusion gates to filter vital information within the filtered modalities inadvertently.(2) Long-tailed distributions of intent categories. Existing MID benchmarks [35, 60] exhibit a pronounced long-tailed distribution of intent categories as shown in Figure 2 , where a few classes, i.e., head classes, contain a major number of samples while the remaining classes, i.e., tail classes, have only a small number of samples. A straightforward remedy is to rebalance the training dataset through weighted sampling. However, this is a suboptimal strategy that may be detrimental to the accuracy of the head classes [13, 49] .In this paper, we propose a new framework termed InMu-Net to address the existing issues jointly, drawing inspirations from Information bottleneck [9, 48, 52] and Multi-sensory processing [28, 43] . Through three vital components, InMu-Net advances towards its denoising and redundancy reduction capabilities, alongside bolstering robustness across multiple categories. Specifically, ❶ we design a denoising bottleneck module to effectively reduce intentirrelevant feature redundancy. ❷ we present a saliency preservation loss, which provides explicit supervision to maximize the intentrelevant information in the fused feature. ❸ we perform kurtosis regulation on both unimodal and multi-modal representations. In this manner, InMu-Net can diminish sensitivity towards tail intents, thus mitigating the adverse effects of long-tailed distribution.  
preliminaries 2.1 task description: 
 Formally, given a tri-modal input comprising text, visual and audio modalities, the multi-modal intent detection (MID) task can be conceptualized as a classification task that determines the intent label for the tri-modal input, which is expressed as follows:EQUATIONwhere 𝑓 (•) represents the MID model; 𝑋 𝑡 , 𝑋 𝑣 , 𝑋 𝑎 denote the text, visual and audio input, respectively; 𝑦 ∈ 𝑌 = {𝑦 1 , 𝑦 2 , . . . , 𝑦 𝐾 } is the intent label associated with one of the 𝐾 predefined intents.To begin, we first encode the multi-modal sequential input 𝑋 𝑚 (where 𝑚 ∈ {𝑡, 𝑣, 𝑎}) into unit-length representations 𝑓 𝑚 . Specifically, we employ separate modality-specific Transformers [45] to capture the features of distinct modalities as follows:EQUATIONin which 𝜃 𝑚 represents the parameters of the Transformer 𝑚 .Mutual information [17] is a measure of the amount of information shared between random variables. Formally, it quantifies the statistical dependency of two random variables 𝑋 and 𝑌 :EQUATIONwhere 𝑝 (𝑋, 𝑌 ) denotes the joint probability distribution between 𝑋 and 𝑌 , while 𝑝 (𝑋 ) and 𝑝 (𝑌 ) are their marginals. Kurtosis [6] is a statistical measure utilized to categorize the tail behavior of distributions. It is sensitive to rare events and is employed for analyzing distributions with "fatter tails". For univariate variables, kurtosis is quantified as the standardized fourth moment:EQUATION)It is commonly used to assess deviations from normalcy. Building upon this, Mardia [25] extended kurtosis to a multivariate context:EQUATIONwhere 𝑋 denotes a multi-dimensional random vector with 𝜇, representing the mean and covariance matrix of 𝑋 , respectively. 
methodology: 
 In this section, we detail our proposed framework (InMu-Net) for MID, the architecture of which is depicted in Figure 3 . Concretely, we start by concatenating the features extracted from each modality ( §2.2) to create an initial fused feature 𝑓 , which retains comprehensive information from each modality. Then, We present the underlying motivations for our proposed supervision ( §3.1). After that, we apply a denoising bottleneck module ( §3.2) to perform information distillation, culminating in the refined feature f . Notably, we incorporate a saliency preservation loss ( §3.3) to ensure no critical intent-relevant information is discarded. Additionally, we introduce a kurtosis regulation loss ( §3.4) inspired by neural synergy [5] to represent unimodal and multimodal features more effectively against long-tail distribution.As discussed above, we aim to eliminate redundancies and noise in the visual and audio modalities within MID. To this end, we resort to information bottleneck (IB) [24] , which can find the concise and compressed representation of the input. By applying the IB principle, the model can learn to filter out noisy and redundant information that might otherwise interfere with prediction accuracy. Mathematically, consider the encoded fused feature f , derived from the direct fused feature 𝑓 . Our objective is for f to encapsulate only intent-relevant information while eliminating intent-irrelevant redundancies. Drawing on the principles of mutual information [9] in IB, the information between f and 𝑓 can be quantified as follows:EQUATIONThis expression can be further expanded by introducing the target variable 𝑦 and applying the chain rule as follows:EQUATIONwhich distinguishes the intent-irrelevant information I (𝑓 ; f |𝑦) from the intent-relevant information I ( f ; 𝑦). Thus, the goals of IB are to ❶ minimize the I (𝑓 ; f |𝑦) while ❷ maximize the I ( f ; 𝑦):EQUATIONin which 𝛾 is a scalar that determines the weight of the intentrelevant information constraint I ( f ; 𝑦) during optimization. However, conducting a min-max game as formulated in Eq. ( 8) is challenging [27, 30] due to the well-documented difficulty in estimating mutual information in high-dimensional spaces [29] . To address this issue, we introduce a denoising bottleneck to achieve goal ❶ and saliency preservation to achieve goal ❷. Additionally, we implement kurtosis regulation to ensure smooth representations across both unimodal and multimodal contexts, thereby mitigating the negative effects associated with tailed intents.Considering three encoded features 𝑓 {𝑡,𝑣,𝑎} from distinct modalities ( §2.2) in MID, we first generate an initial fused feature 𝑓 = [𝑓 𝑡 , 𝑓 𝑣 , 𝑓 𝑎 ] ∈ R 𝑛 , where [•, •] denotes concatenation operation. Subsequently, we suggest automatic denoising at the feature level. To elaborate, the denoising bottleneck module comprises two linear projection layers along with dropout and ReLU activation functions:EQUATIONwhere 𝑧 is of dimension 𝑝 < 𝑛, while the final feature representation f retains 𝑛 dimensions. The re-projection of 𝑧 back to the same dimensions as 𝑓 serves two purposes: ❶ It facilitates feature-level supervision, enabling the learning of a more intent-relevant feature f as discussed in the subsequent subsection. ❷ Empirically, we observed that the feature-level supervision proves more effective when f and the initial feature 𝑓 are aligned.Remark 1. There are several applications of the IB within the multimodal community. [11] explored the application of IB at different stages of fusion, while [52] implemented IB at the sequence level. Our method offers several advantages: ❶ simplicity in fusion approach; ❷ flexibility in handling variable-length sequence inputs. Additionally, direct comparisons are provided in the experiments (cf. Table 8 ).The denoising bottleneck constrains information flow across modalities to filter out redundancy and noise. However, it might inadvertently lead to the loss of vital information as well [11] . Inspired by the success of mutual information in applications within the computer vision community [8, 22, 41] , we introduce saliency preservation, which is designed to explicitly maximize intent-relevant information I ( f ; 𝑦) in Eq.( 8), ensuring that the process of information reduction does not compromise the quality of crucial information.Theorem 1. To elucidate, maximizing I ( f ; 𝑦) can be interpreted as minimizing the difference in the mutual information between the original and the denoised features for 𝑦, formally expressed as:EQUATIONwhere 𝐻 (𝑦|𝑓 ) is defined as the conditional entropy:EQUATIONBuilding on this theoretical foundation, we further validate the effectiveness of f through the following corollary: [18] between the predicted distributions of the fused feature 𝑓 and the denoised f equals to 0, then f is sufficient for 𝑦 as well i.e.,Corollary 1. If the KL-divergenceEQUATION)where 𝑝 (𝑦|𝑓 ), 𝑝 (𝑦| f ) represent the predicted distributions under 𝑓 and f , respectively; and 𝐷 𝐾𝐿 (•) denotes the KL-divergence.To operationalize this theory, we finally frame our saliency preservation loss as follows:EQUATION)In multi-sensory neural processing, around 20% of the neurons account for 80% of the information propagation in cortical circuits [28] . Several researches by [7, 42] demonstrated that multi-modal computation tends to concentrate in such local cortical clusters and found significantly lower kurtosis in such clusters. Besides, low kurtosis helps alleviate sensitivity to rare events, as discussed in §2.4. These all suggest that individual unimodal and multi-modal representations in MID should exhibit low levels of kurtosis. To maintain this characteristic, we regulate the multivariate kurtosis by plugging in standard estimators for the mean and covariates:EQUATION)where 𝑁 denotes the number of features, 𝑓 𝑗 represents samples from features, including both unimodal features like 𝑓 {𝑡,𝑣,𝑎} and the direct fused feature 𝑓 . f denotes the empirical mean feature f= 𝑛 𝑗 =1 𝑓 𝑗 𝑁and 𝑆 signifies the empirical covariance matrix:EQUATION)Since f is derived from 𝑓 , we do not perform regularization on f . Note that the covariance matrix is computed via a decaying moving average over a window of multiple batches to produce smoother estimates before the inversion operation.Eventually, the overall loss function in our InMu-Net is:EQUATIONwhere L 𝑓 and L f are respective losses to supervise the direct fused feature and denoised feature, and L 𝑚𝑜𝑑𝑎𝑙𝑖𝑡 𝑦 supervises core textual modality encoder since textual information dominates across modalities [15, 40] . 𝛼 and 𝛽 are trade-off hyper-parameters. 𝑦 ′ 𝑓 and 𝑦 ′ f are the classifier results from the direct fused feature and denoised feature, respectively. Note that the first three losses are directly supervised by 𝑦 and serve as the foundational supervision of the overall framework in the form of cross-entropy losses.During inference, the denoised feature f is employed to determine the ultimate intent. Consequently, InMu-Net serves as an augmentation during the training phase, with only the denoising bottleneck module invoked during inference, given that simple Linear and Dropout contribute minimally to inference latency. 
experiments 4.1 datasets and metrics: 
 We conduct experiments on two benchmarks to evaluate the proposed InMu-Net: ❶ MIntRec [60], 1 Following previous works, we employ accuracy (ACC), weighted F1-score (wF1), weighted precision (wP), and recall (R) as evaluation metrics to assess the proposed InMu-Net framework. To account for category imbalances, the wF1 and wP metrics are calculated as weighted averages, with weights corresponding to the sample counts in each category. Unless specified otherwise, higher values indicate better performance across all metrics in this work.For a fair comparison, we follow Huang et al. [14] , Zhang et al. [60] to adopt bert-base-uncased [16] and wav2vec2-base-960h [1] from Huggingface Library [51] to extract text and audio features and Faster R-CNN [33] from Torchvision Library to extract visual features. AdamW [23] is utilized as the optimizer with a learning rate searched from [1𝑒 -6 , 3𝑒 -5 ]. The batch size is set as 16 for training and 8 for validation/testing. For hyper-parameter 𝛼 and 𝛽, we test them in the range from 0.2 to 1.0 on the validation set and 2 https://github.com/thuiar/TCL-MAP choose the best-performing one to the test set, respectively. Paired t-test is performed to test the significance of performance improvement with a default significance level of 0.05. All experiments are conducted on one single NVIDIA GeForce RTX 3090. The results reported in all experiments are averages of 5 random runs.We compare InMu-Net with a series of competitive MID baselines, including: MAG-BERT [32] , MulT [44] , MISA [12] , TCL-MAP [61] and SDIF-DA [14] . The main results on two benchmarks are reported in Table 1 , from which we have the following observations:❶ Our proposed InMu-Net consistently outperforms all baselines on both MIntRec and MELD-DA datasets and achieves a new SOTA performance. Specifically, on MIntRec dataset, it overpasses the previous SOTA model SDIF-DA by 2.91% and 2.75% on ACC and wF1, respectively; on MELD-DA dataset, it overpasses TCL-MAP by 3.29% and 3.13% on ACC and wF1, respectively. This verifies the effectiveness of InMu-Net in the MID task. Furthermore, the significance tests of InMu-Net over the baseline models show that our InMu-Net significantly outperforms the baseline models (the results of 𝑝-value on all evaluation metrics are less than 0.05).❷ Notably, the gains on MELD-DA are more pronounced. We suppose the reason is that MELD-DA is more challenging, whose data redundancy problem is more serious, involving complex scenes 16).Setting L 𝑓 DB L f L 𝑠𝑎𝑙𝑖𝑒𝑛𝑐𝑦 L 𝑚𝑜𝑑𝑎𝑙𝑖𝑡 𝑦 L 𝑘𝑢𝑟𝑡𝑜𝑠𝑖𝑠 MIntRec MELDand overlapping characters. The proposed InMu-Net excels in denoising and preserving vital information, coupled with enhanced robustness via kurtosis regulation, resulting in superior performance over strong baseline models TCL-MAP and SDIF-DA. ❸ To clearly demonstrate how our method addresses the longtail distribution issue, we selected the top five and bottom five intents based on their occurrence frequencies in the training set. These are referred to as the "Common" and "Long-tail" subsets, respectively, and their F1 scores are presented in Table 2 . We find our InMu-Net achieves superior performance in 8 out of the 10 intent categories, particularly showing notable improvements in the "Long-tail" intents. Thanks to the kurtosis regularization, InMu-Net can adaptively mitigate the adverse effects of the long-tail distribution, achieving robust performance.We perform a set of ablation studies to understand the necessity of the different designs and strategies in the proposed InMu-Net. From the results in Table 3 , we can obtain the following takeways: Denoising Bottlneck. The denoising bottleneck module coupled with saliency preservation loss is first removed to not perform information filtering. By comparing the row setting (b) to setting (h) in Table 3 , we can conclude that the decreased performance implies that the proposed denoising bottleneck actually contributes to eliminating redundancy and noise within the modalities. Saliency Preservation. There is a consistent performance degradation on both datasets (comparing setting (c) to setting (h)) when the denoising bottleneck lacks saliency preservation supervision. A plausible deduction is that the supervision mitigates the loss of valuable information due to intent-irrelevant feature filtering. Kurtosis Regualion. ❶ Here, we remove the kurtosis regulation to verify its effectiveness (comparing setting (d) to setting (h)). ThePerplexity (↓) TCL-MAP [61] 2.55 SDIF-DA [14] 2.48 InMu-Net 1.82 poor results show that appropriate regulating multi-modal features can boost model performance. ❷ Furthermore, when comparing the data in Table 2 without the L 𝑘𝑢𝑟𝑡𝑜𝑠𝑖𝑠 term (w/o L 𝑘𝑢𝑟𝑡𝑜𝑠𝑖𝑠 ), a noticeable decline in model performance is observed. This further demonstrates that the proposed kurtosis regularization can effectively alleviate the sensitivity to tail intents and mitigate long-tail distribution issues, thereby enabling more robust predictions. Centre Modality. As mentioned in §3.5, results by text-centric supervision tend to perform better as low information intensity and high redundancy in other modalities. Thus, we evaluate results based on no, audio and visual modality supervision (setting (e), (f) and (g)). We observe an obvious decline in performance when audio, visual, or no modality is used as the central supervision.Since Eq.( 16) encompasses multiple loss components, we delved into the influence of the core elements as shown in Figure 4 . Specifically, we select the best-performing model based on the validation set and then evaluate it on the test set: ❶ The parameter 𝛼 indicates the importance of L 𝑠𝑎𝑙𝑖𝑒𝑛𝑐𝑦 . We evaluate the scale range setting 𝛼 ∈ [0.2, 1.0] as shown in Figure 4 (a). We find that accuracy is improved and saturated with 0.8 and 1.0 on MIntRec and MELD-DA, respectively. Thus, we set 𝛼 = 0.8 for MIntRec and 1.0 for MELD-DA in practice. ❷ The parameter 𝛽 signifies the extent of involvement of L 𝑘𝑢𝑟𝑡𝑜𝑠𝑖𝑠 . Our observations indicate a relative insensitivity to parameter selection, as the incorporation of kurtosis regulation generally yields gains for MID, albeit to varying degrees. As a result, we adopt 𝛽 = 1.0 to achieve the best performance.Since the denoising module plays an important role in the proposed approach, we provide more insight analysis about it. Specifically, we evaluated the perplexity for the prediction of the golden label across two state-of-the-art baselines (i.e., TCL-MAP and SDIF-DA) and our proposed InMu-Net on the MIntRec test set. From the results in Table 4 , we observe that both TCL-MAP and SDIF-DA exhibit perplexity around 2.5, whereas our method demonstrates a significant reduction (-0.7). This compellingly illustrates the effectiveness of our denoising bottleneck module in accurately predicting intents.To investigate the effect and robustness of InMu-Net in low-resource scenarios, we conduct experiments using different limited training sizes following Huang et al. [14] . From the results in Figure 5 , we observe that InMu-Net consistently outperforms SOTA baselines, especially when the resource is quite limited (10%). This indicates that InMu-Net can sustain denoising capabilities within resourceconstrained scenarios and distill intent-relevant information from multi-modal representations, thus boosting performance.We further assess the model performance by removing one modality at a time. From the results in Table 5 , we find that ❶ the trimodal combination yields the highest performance, indicating that our InMu-Net can learn complementary information from different modalities. ❷ the performance drops sharply when the textual modality is removed. We attribute this to the fact that textual modality has higher information density compared to redundant audio and visual modalities. This underscores two critical insights: First, eliminating noise and redundancy to enhance the information density of visual and audio modalities is crucial during fusion. Second, text-central fusion results may help boost performance in MID.Analysis of Cross-architecture Scenario. To evaluate the generalizability of our proposed InMu-Net, we conduct preliminary experiments on two representative baselines MuIT and SDIF-DA.To be specific, we retain the baseline loss function for a fair comparison, integrating only our three vital components and corresponding supervisions on the fused feature. The results depicted in Latency/Inference Time per Sample Speedup TCL-MAP [61] 25.8ms 1.0x SDIF-DA [14] 25.4ms 1.0x InMu-Net 25.5ms 1.0x As the proposed method is applicable to real-world applications, we conducted a preliminary latency comparison among the two most advanced baseline models and our method, with the results reported in Table 7 . We find that our model's inference speed surpasses that of TCL-MAP and maintains a speedup on par with current SOTA methods. This can be attributed to the fact that our method functions as an augmentation during the training phase, with only the denoising bottleneck module being activated during inference. Given that simple Linear and Dropout layers contribute minimally to the inference latency, this ensures efficient performance.To qualitatively demonstrate how our proposed InMu-Net filters noise and redundancy while capturing precise intent-relevant information, we provide GradCAM-CAM [36] visualizations of InMu-Net and SOTA baseline SDIF-DA. From Figure 6 case (a), it can be seen that while both models SDIF-DA and InMu-Net successfully identify the critical term "gift" in the visual modality. However, the capture range of SDIF-DA is broader and more diffuse, including irrelevant background details. In contrast, our proposed InMu-Net focuses narrowly and precisely on the gift on the Baseline Ours 
related work: 
 Multi-modal Intent Detection. Multi-modal intent detection (MID) is a significant task for understanding human language in task-oriented dialogue systems [2] [3] [4] [62] [63] [64] . Compared to text-only intent detection, which ascertains the objectives of users conveyed through their utterances, MID integrates facial expressions and audio signals to fully leverage the complementary and interactive information provided by diverse modalities. A series of models [14, 35, 60, 61] have been proposed and made promising progress. Therein, Zhou et al. [61] introduced a token-level contrastive learning coupled with modality-aware prompting to improve modality fusion. Concurrently, Huang et al. [14] developed a Transformerbased [46, 47] framework that progresses from shallow to deep interactions, complemented by ChatGPT-based data augmentation techniques to align features across modalities. Our work do not focus on intricate fusion mechanisms [39] or architectural intricacies, our approach emphasizes the perspective of multi-modal information and data distribution. Information Bottleneck. The InfoMax proposed by Linsker [20] seeks to maximize the mutual information between feature and model output. Along this way, there have been many works that explore optimal ways for mutual information estimation. Han et al. [11] built up a hierarchical mutual information maximization guided model for multi-modal sentiment analysis. Wu et al. [52] focused on video-based sentiment analysis and performed contrastive learning to achieve mutual information maximization. Despite these advancements, these theories in MID remain under-studied.In contrast to existing works, our InMu-Net devises a denoising bottleneck and a saliency preservation loss to precisely filter intentirrelevant information and keep intent-relevant information in an adaptive manner, which obviates the reliance on heuristic or greedy feature selection methods [19, 52] and coarse-grained filtering mechanism [53, 55] . Furthermore, the proposed feature-level denoising offers advantages in handling variable-length inputs. Multi-sensory Processing. In multi-sensory processing resarch area, different sensory modalities are processed individually and then combined in various multimodal convergence zones, including cortical and subcortical regions [10] . Several research such as Faber et al. [7] , Timme et al. [42] demonstrated that multi-modal computation tends to concentrate in such local cortical clusters and found significantly lower kurtosis in such clusters.In this work, we resort to multivariate kurtosis [34] to alleviate the long-tail distribution problem in MID. The most straightforward remedy to this problem is to rebalance the training dataset through weighted sampling. However, it is a suboptimal strategy that may be detrimental to the accuracy of the head classes [49] . Innovatively, we treat the fused multi-modal feature as cortical clusters, controlling peaking to reduce sensitivity to tailed intents. 
conclusion: 
 In this paper, we proposed InMu-Net, a new framework from the information bottleneck and multi-sensory processing perspectives, to tackle modality redundancy and long-tailed distribution of labels in MID jointly. InMu-Net maximizes the intent-relevant information in fused multi-modal features by the proposed denosing bottleneck and minimizes the intent-irrelevant information by the proposed saliency preservation loss. Moreover, kurtosis regulation is introduced to reduce the negative impact of long-tail distributions. Extensive experiments and analyses on two MID benchmarks show the superiority of our proposed framework. 
 
 Appendix: https://github.com/thuiar/MIntRec/tree/main 


Decision: **Conclusion: Accept as an Oral Paper**

The paper presents InMu-Net, a framework designed to address challenges in multi-modal intent detection (MID), such as noise, redundancy in different modalities, and long-tailed distributions of intent categories. The reviewers have provided positive feedback on several fronts, including the well-founded motivation, extensive experimentation, and innovative approach. Based on these merits, I recommend this submission for acceptance as an Oral Paper.

**Strengths:**

1. **Motivation and Theoretical Support:** The paper is well-motivated with clear theoretical support, making a compelling case for its proposed methods. The framework's components, such as the denoising bottleneck and kurtosis regulation, are theoretically sound and convincingly argued.
  
2. **Comprehensive Experiments:** Extensive evaluations across multiple benchmarks demonstrate the effectiveness and robustness of InMu-Net. The model performs well in various conditions, including low-resource scenarios and modality corruption tests, highlighting its practical applicability.

3. **Novelty in Handling Long-Tailed Distributions:** The introduction of kurtosis regulation to address the issue of long-tailed distributions is innovative. This component significantly enhances the model's performance on underrepresented categories, which is a notable contribution to the field.

**Weaknesses:**

1. **Details of Joint Probability Distribution Calculation:** The paper does not provide sufficient details on how the joint probability distribution between the 1D vector and intent label is calculated. This omission leaves a gap in understanding the full implementation of the proposed method.

2. **Lack of Novelty in Denoising Bottleneck Design:** The design of the denoising bottleneck is seen as a straightforward application of a denoising autoencoder at the feature level. This could be perceived as lacking novelty in comparison to existing methods.

3. **Limited Visualization and Appendix Details:** Only one visualization result is presented, and the appendix lacks detailed analysis of the cross-task scenario. More visual comparisons and additional information in the appendix would strengthen the paper's empirical evidence and overall presentation.

In summary, despite some areas needing further clarification and enhancement, the paper's strengths in motivation, comprehensive experimentation, and novel approach to handling long-tailed distributions make it a valuable contribution to the field of multi-modal intent detection. Therefore, it merits acceptance as an Oral Paper for ACM MM 2024.

Reviews: review: paper_summary: paper_summary: This paper aims to address the multi-modal intent detection task. It proposes InMu-Net to tackle the challenges that noise and redundancy in different modalities and long-tailed distributions of intent categories. A denoising bottleneck is constructed to distill information for mutual information estimation and maximize the intent-relevant information between the modal-fused feature and target intent label. Besides, a kurtosis regulation loss is designed to alleviate the negativity of tailed intents. Extensive experiments demonstrate the effectiveness of the InMul-Net in different benchmarks and its ability of denoising and robustness.
strength_weakness: strength_weakness: 1. The motivation of this paper is well-founded and clear, and the theoretical support of the model is sufficient and convincing.
2. The experiments are sufficient and have demonstrated the effectiveness of the proposed method.
limitations: limitations: 1. As  is a 1D vector and  is a intent label, how to calculate their joint probability distribution is not mentioned.
2. The detailed formula of loss  and  is not mentioned.
paper_summary: paper_summary: The paper introduces InMu-Net, a framework for multi-modal intent detection (MID). The authors propose a denoising bottleneck module to filter out irrelevant information, a saliency preservation loss to ensure relevant information is retained, and kurtosis regulation mitigate issues from long-tail distributions. The results show that InMu-Net outperforms state-of-the-art models on two MID benchmark datasets.
strength_weakness: strength_weakness: 1.	The paper provides an extensive evaluation of the model across multiple benchmarks and conditions, including low-resource scenarios and modality corruption tests. 

2.	The introduction of kurtosis regulation to handle long-tail distributions is novel. This component helps the model perform better on underrepresented long-tail subset.
limitations: limitations: 1.	The design of denoising bottleneck is not novel, which can be seen as a simple denoising autoencoder on the feature level.

2.	There are two hyperparameters $\alpha$ and $\beta$ for the saliency loss and Kurtosis loss. However, the weights for other loss components are set to 1. The reason for these setups is not clearly discussed.
paper_summary: paper_summary: This paper propose InMu-Net, a novel framework for Multi-modal intent detection that tackles modality redundancy and long-tailed distribution of labels in MID using information bottleneck and multi-sensory processing perspectives.
strength_weakness: strength_weakness: This paper proposes a novel framework for Multi-modal intent detection that introduces a denoising bottleneck module, saliency preservation loss and kurtosis regulation method.

The authors conduct comprehensive experiments on MID benchmark datasets, and the results reveal the potential and robustness of InMu-Net.
limitations: limitations: •	In Table 2, the results of “Flaunt”, ”Opposes” and “Joke” in the “Long-tail” subsets have huge improvement, but the improvement of “Agree” and “Ask for help” is  relatively low and equivalent to the improvement of “Praise” and “Apologies” in “Common” subsets. This makes it less convincing that InMu-Net mitigate the adverse effects of the long-tail distribution since it may come from the improvement of the model's detection ability for certain scenarios rather than being more adaptable to long tailed distributions.

•	Only one visualization result is provided, it is recommended to present more visualization results for comparison.

•	Appendix for more details of analysis of cross-task scenario(L799) is not provided.