Neuro-Inspired Information-Theoretic Hierarchical Perception for Multimodal Learning
Authors: Xiongye Xiao
Gengshuo Liu
Gaurav Gupta
Defu Cao
Shixuan Li
Yaxing Li
Tianqing Fang
Mingxi Cheng
Paul Bogdan
Venue: ICLR 2024 poster (2024)
Keywords: multimodal learning
information bottleneck
sentiment analysis
Primary area: representation learning for computer vision, audio, language, and other modalities

Abstract: Integrating and processing information from various sources or modalities are critical for obtaining a comprehensive and accurate perception of the real world in autonomous systems and cyber-physical systems. Drawing inspiration from neuroscience, we develop the Information-Theoretic Hierarchical Perception (ITHP) model, which utilizes the concept of information bottleneck. Different from most traditional fusion models that incorporate all modalities identically in neural networks, our model designates a prime modality and regards the remaining modalities as detectors in the information pathway, serving to distill the flow of information. Our proposed perception model focuses on constructing an effective and compact information flow by achieving a balance between the minimization of mutual information between the latent state and the input modal state, and the maximization of mutual information between the latent states and the remaining modal states. This approach leads to compact latent state representations that retain relevant information while minimizing redundancy, thereby substantially enhancing the performance of multimodal representation learning. Experimental evaluations on the MUStARD, CMU-MOSI, and CMU-MOSEI datasets demonstrate that our model consistently distills crucial information in multimodal learning scenarios, outperforming state-of-the-art benchmarks. Remarkably, on the CMU-MOSI dataset, ITHP surpasses human-level performance in the multimodal sentiment binary classification task across all evaluation metrics (i.e., Binary Accuracy, F1 Score, Mean Absolute Error, and Pearson Correlation).

Published as a conference paper at ICLR 2024
NEURO-INSPIRED INFORMATION-THEORER HIERAR- CHICAL PERCEPTION FOR MULTIMODAL LEARNING
Xiongye Xiao Gengshuo Liu", Gaurav Gupta', Defu Cao', Shixuan Li', Yaxing Li',
Tianqing Fang', Mingxi Cheng' & Paul Bogdan
University of Southern California, Los Angeles, CA 90089, USA Hong Kong University of Science and Technology. Hong Kong, China
ABSTRACT
Integrating and processing information from various sources or modalities are critical for obtaining comprehensive and accurate perception of the real world in autonomous systems and cyber-physical systems. Drawing inspiration from neu- roscience, we develop the Informatioon-hhoretti Hierarchical Perception (ITHP) model, which utilizes the concept of information bottleneck. Different from most traditional fusion models that incorporate all modalities identically in neural net- works, our model designates a prime modality and regards the remaining modalities as detectors in the information pathway, serving to distill the flow of information. Our proposed perception model focuses on constructing an effective and compact information flow by achieving a balance between the minimization mutual infor- mation between the latent state and the input modal state, and the maximization mutual information between the latent states and the remaining modal states. This approach leads to compact latent state representations that retain relevant information while minimizing redundancy, thereby substantially enhancing the performance of multimodal representation learning. Experimental evaluations on the MUStARDD, CMU-MOSI, and CMU-MOSEI datasets demonstrate that our model consistently distills crucial information in multimodal learning scenar- ios, outperforming state-of-the- benchmarks. Remarkably, on the CMU-MOSI dataset, ITHP surpasses human-leve performance in the multimodal sentiment binary classification task across all evaluation metries (i.e., Binary Accuracy, F1 Score, Mean Absolute Error, and Pearson Correlation).
INTRODUCTION
Perception represents the two-staged process of, first (i), understanding, mining, identifying critical and essential cues, and translating multimodal noisy sensing (observations) into world model. Second (ii). defining the premises and foundations for causal reasoning, cognition, and decision- making in uncertain settings and environments. Consequently, major focus in neuroscience refers to elucidating how the brain succeeds in quickly and efficiently mining and integrating information from diverse external sources and internal cognition to facilitate robust reasoning and decision- making (Tononi et al. 1998). Promising breakthroughs in understanding this complex process have emerged from exciting advancements in neuroscience research. Through a method of stepwise functional connectivity analysis (Sepulcre et al.) 2), the way the human brain processes multimodal data has been partially elucidated through recent research. The research outlined in Pascual-Leene & Hamilton 2001);; Meunier et al. (2009); Zhang et al. (2019); Raut et al. (2020) suggests that information from different modalities forms connections in a specific order within the brain. This indicates that the brain processes multimodal information in hicrarchical manner, allowing for the differentiation between primary and secondary sources of information. Further research by (Mesulam 1998] Lavenex & Amaral 2000) elucidates that synaptic connections between different areas are reciprocal. This reciprocity enables information from various modalities reciprocally exert feedback, thereby influencing early processing outcomes. Such sequential
Equaal Contribution.
'taiuttiinn.nrrreee 'Corresponding xi xiooneeee@us edu and pbogdan@usc.edd.

---

Published as a conference paper at ICLR 2024
and hierarchical processing approach allows the brain to start by processing information in a certain modality, gradually linking and processing information in other modalities, and finally analyzing and integrating information effectively in coordinated manner.
The brain selectively attends to relevant cues from each modality, extracts meaningful features, and combines them to form more comprehensive and coherent representation of the multimodal stimulus. To exemplify this concept, let us consider sareasm, which is often expressed through multiple cues simultaneously, such straight-looking face, a change in tone, or an emphasis word (Castro et al. 2019). In the context of human sarcasm detection, researchers have observed significant variation in sensitivity towards different modalities among individuals. Some are more attuned to auditory cues such as tone of voice. while others are more responsive to visual cues like facial expressions. This sensitivity establishes a primary source of information unique to each individual, creating personalized perception of sarcasm. Simultaneously, other modalities aren't 't disregarded; instead, they reciprocally contribute, serving as supportive factors in the judgment process. In fact, researchers employ various experimental techniques, such as behavioral studies, neuroimaging (e.g., fMRI, Glover 201T PET. Price 2012), and neurophysiological recordings, to investigate the temporal dynamics and neural mechanisms underlying multimodal processing in the human brain. The insights gained from these areas of neuroscience have been instrumental in guiding our novel approach to information integration within the scope of multimodal learning.
In multimodal learning, integrating different information sources is crucial for a thorough understand- ing of the surrounding phenomena the environment. Although highly beneficial for wide variety of autonomous systems and cyber-physice systems (CPS) Xue & Bogdan (017), effectively under- standing the complex interrelations among modalities is difficult due to their high-dimensionality and intricate correlations Yin et al. 2023). To tackle this challenge, advanced modeling techniques are needed to fuse information from different modalities, enabling the discovery and utilization of rich interactions among them. Existing works have employed various approaches to fuse information from different modalities, treating each modality separately and combining them in different ways CCastro et al.) 2019 Rahman et al.) 2020 However, it has been seen that these fusion approaches deviate from the information--roccssinng mechanism of neural synapses in the human brain. The above neuroscience research inspired to devise novel fusion method that constructs a hierarchical information perception model. Unlike conventional approaches that directly combine various modal information simultaneously. our method treats the prime modality as the input. We introduce novel biomimetic model, referred to as Cnermation-Thoorett Hierarchical Perception (ITHP), for effectively integrating and compacting multimodal information. Our objective is to distill the most valuable information from multimodal data, while also minimizing data dimensions and redundant information. Drawing inspiration from the information bottleneck (IB) principle proposed by Tishby et al. (Tishby et al. 2000), we propose a novel strategy involving the design of hierarchical latent states. This latent state serves to compress one modal state while retaining as much relevant informa- tion as possible about other modal states. By adopting the IB principle, we aim to develop compact representations that encapsulate the essential characteristics of multimodal data.
1.1 OUR CONTRIBUTIONS
In this paper, we propose the novel Thforration-Theoretii Hierarchical Perception (ITHP) model, which employs the IB principle construct compact and informative latent states (information flow) for downstream tasks. The primary contributions of this research can be summarized as follows:
We provide a novel insight into the processing of multimodal data by proposing a mecha- nism that designates the prime modality as the sole input, while linking other modal information using the IB principle. This approach offers a unique perspective on multimodal data fusion, reflecting a more
petoooogicall--iisiieee model of information processing.
We design ITHP model on top of recent neural network architectures, enhancing its accessi- bility and compatibility with most existing multimodal learning solutions.
Remarkably, our ITHP-DeBERTa framework outperforms human-level benchmarks in multimodal sentiment analysis tasks across multiple evaluation metrics. The model yields an 88.7% binary accuracy, 88.6% Fl score. 0.643 MAE, and 0.852 Pearson correlation,
DeBERTa is an advanced variant of the BERT model for natural language embedding.
2

---

Published as a conference paper at ICLR 2024
outstripping the human-level benchmarks of 85.7% binary accuracy, 87.5% Fl score, 0.710 MAE, and 0.820 Pearson correlation.
1.2 BACKGROUND AND RELATED WORK
Information bottleneck. Information bottleneck (IB) has emerged as a significant concept in the field of information theory and machine learning, providing a novel framework for understanding the trade-off between compression and prediction in information flow. The IB principle formulates the problem of finding a compact representation of one given state while preserving its predictive power with respect to another state. This powerful approach has found numerous applications in areas such as speech recognition (Hecht & Tishby, 2005), document classification (Slonim & Tishby) 2000). and gene expression (Friedman et al., 2001). Moreover, researchers have successfully applied the IB principle to deep learning architectures (Shwartz-Ziv & Tishby 2017] Tishby & Zaslavsky 2015). shedding light on the role of information flow and generalization in these networks. this paper, we employ the principles of IB to construct effective information flow by designing hierarchical perception architecture, paving the way for enhanced performance in downstream tasks.
Multimodal learning. Multimodal learning has been particularly instrumental in understanding language across various modalities text, vision, and acoustic and has been applied extensively to tasks such sentiment analysis (Poria et al., (0088), emotion recognition (Zadch et al.) (018d). personality traits recognition (Park et al. 2014). and chart understanding (Zhou et al. 2023). Work in this domain has explored a rragge of techniques for processing and fusing multimodal data. These include tensor fusion methods (Liu et al. 2018 Zadeh et al. 2011) 2016). attention-base cross-modal 2019). and multimodal neural architectures (Pham) 2019 Hazarika et al.) 2018), among others. Such explorations have given rise to advanced models like Tensor Fusion Networks (Zadeh et al. 201 8a). Multi-attention Recurrent Network (Zadeh et al. 2018b). Memory Fusion Networks (Zadeh et al.) [201 8a). Recurrent Memory Fusion Network (Liang et al. 2018), and Multimodal Transformers (Tsai et al. 2019). These models utilize Transformers hierarchical fusion mechanisms, attention mechanisms, and separate modality-specin
interaction methods (Lv al. [022 T Tsai
et al.
to selectively process information from multiple input channels. Despite significant strides in this domain, challenges persist, particularly when dealing with unaligned or limited and noisy multimodal data occurring in multi-agent systems navigating in uncertain unstructed environments. To surmount these obstacles, researchers have started exploring data augmentation methods and information- theoretic approaches to enrich multimodal representations and boost overall system performance.
2 MULTIMODAL LEARNING BASED ON INFORMATION BOTTLENECK
In this section, we introduce a novel method, Hierarchical Perception (ITHP), for integrating and com- pressing multimodal noisy data streams. Tak- ing inspiration from neuroscience research, our approach models the fusion and distilla- tion of the information from multimodal in- formation sources as hierarchical structure, with designated prime modality serving as the input. To link the input information with information from other modalities, we apply
called information-Theorett
X
X1
X₂
B
B
Y
Figure Constructing two latent states, Bo and Bi, fa- cilitates the transfer of pertinent information among three modal states Xo. X1, and X2.
the principle of Information Bottleneck (IB), ensuring an optimized compact representation of rel- evant information between modalities. Here, we begin by presenting a concise overview of the IB principle, followed by the construction of the hierarchical structure for multimodal information fusion. typical example in multimodal information integration problems is the 3-modalities problem. We use this case as practical illustration of the efficacy of our proposed model. To provide detailed explanation, we initially formulate an optimization problem that delineates the specific question. Following this, we leverage the concept of the information bottleneck to build the structure of the ITHP model, which is then trained through the application of neural networks. Importantly, the ITHP model possesses the flexibility to be expanded handle multimodal context encompassing N
3

---

Published as a conference paper at ICLR 2024
modalities. The exact model for this extension, along with a step-by-step derivation process, can be found in Appendix [ for reference.
Information Bottleneck (IB) approach. The IB approach compresses state X into latent state B, while maintaining as much relevant information as possible about another state of interest Y. The latent state B operates minimize the remaining information of the compression task and to maximize the relevant information. Provided that the relevance of one state to another is often measurable by an cormmttion-thoreete metric (i.e., mutual information), the trade-off can be written as the following variational problem:
min P(
(I)
The trade-off between the tasks mentioned above is controlled by the parameter B. In our model of ITHP, the latent states, solving the minimization problem ([). encodes the most informative part from the input states about the output states. Building upon this concept, we extend this one level of IB structure to create a hierarchy of bottlenecks, enabling the compact and accurate fusion of the most relevant information from multimodal data.
2.1 PROBLEM FORMULATION
From an intuitive perspective, fusing the multimodal information in compact form is an optimization between separating the irrelevant or redundant information and preserving/xxtracting the most relevant information. The multimodal information integration problem for 3 modalities is shown in Fig. [ In the information flow, we can think that Xo. X, and X₂ are three random variables that are not independent. We assume that the redundancy or richness of the information contained in the three modal states Xo. X1, and X2 is decreasing. Therefore, the system can start with the information of Xu and gradually integrate the relevant information of X1 and X2 according to the hierarchical structure. In this configuration, we aim to determine the latent state By that efficiently compresses the information of Xo while retaining the relevant information of X1. Moreover, Bu acts as a pathway for conveying this information to B, Next, B₁ quantifies the meaningful information derived from Bo, representing the second-level latent state that retains the relevant information of X2. Instead of directly fusing information based on the original dx,, -dimensional data, we get the final latent state with a dimension of dB, as the integration of the most relevant parts of multimodal information, which is then used for accurate prediction of Y.
The trade-off between compactly representing the primal modal states and preserving relevant information is captured by minimizing the mutual information while adhering to specific information processing constraints. Formally, we formulate this an optimization problem:
I I(x;;Xa) I(Bo;
14
min ((
(Xx; Bp)
subject to
1(Bo B1)
VA
62
VA
C31
where 62, E3 > O. Our goal is to create hidden states, like Bo and B1. that condense the data from the primary modal state while preserving as much pertinent information from other modal states possible. Specifically, we hope that the constructed hidden state Bp covers much relevant information in X₁ as possible, so (Xoo constructed to retains much relevant information in X2 possible, with [Xo X2) E3 lower bound. Additionally, we want to minimize I (Bo B1) to ensure that we can further compress the information of Bg into B1.
e is a lower bound of I(Bo; X,). Similarly, B, is
2.2 INFORMATION-TIIORRTT IHERARCHICAL PERCEPTION (ITHP)
This section takes the 3-modalities information integration problem as an example to illustrate ITHP as shown in Fig. 2 It introduces hierarchical structure of IB levels combined with multimodal representation learning task. Inspired by the deep variational IB approach proposed in Lee & van der Schaar (022), we formulate the loss function incorporating the IB principle.
4

---

Published as a conference paper at ICLR 2024
Encoder 1 4a, (B))
Mu Σ
Encoder 9n, (B,1Ba)
B₁
Predictor
Detector Po, (g)Bg)
Detector Pr, (Kg101)
B
X
X
N,
Bg
N
1(X,5 B
I(B;; X.)
Figure 2: An illustration of the proposed model architecture. In each encoder, we have two MLP layers: the initial layer extracts feature vectors from input states, while second layer generates parameters for the latent Gaussian distribution. The Venn diagrams illustrate the information constraint from the optimization problem 2.
To address the optimization problem 2 we construct a Lagrangian function, incorporating the balancing parameters 3, X, and y to impose information processing constraints. Consequently, we solve the problem by minimizing the following function, which serves as the overall system loss function for training the network:
T(Xo; Bo) - BB; X1) + (T( B1) - (B1
(3)
We derive the loss function Eqn. (3) based on the IB principle expressed Eqn. (1) as two- level structure of the Information Bottleneck. By minimizing the mutual information 1(Xo: Bp) and [Bn;; B1), and maximizing the mutual information T(B;; X and [(B;; X2), based on the optimization setup in Eqn. 2 and given balancing parameters, ensure that this loss function is minimized during the neural network training process.
The ITHP model dealing with 3-modalities problems contains two levels of information bottleneck structures. We define Bu as latent state that compactly represents the information of the input Xo while capturing the relevant information contained in X1- The latent state Bu is obtained through stochastic encoding with Neural Network, denoted as qa, (Bol Xo). aiming to minimize the mutual information between Xo and Bg- Given the latent state representation Bo obtained from the encoding Neural Network, the outputting MI.P plays the role of capturing the relevant information of X through 4% (X, Similarly, we have the latent state B1 stochastic encoding of Bg. achieved through go, (B, Bo), and an output predictor to X₂ through qv, (1)) Throughout the rest of the paper, we denote the true and deterministic distribution as p and the random probability distribution fitted by the neural network q. Here, Gg and 0 denote the parameters of encoding neural networks while v and 1 denote the parameters the output predicting neural networks. The following loss functions are defined based on the IB principle:
CIBo (XpiX1) (Xoi Bb) 31 (Boi X1)
Exoxp(Xx KL (40, Xp) [lg (Bo))
- B LE Bo Bolx 11, XEx [log qvo (X₂ [o)],
(By; X2) 1 (Bo: B1 71 (Bi; X2) LBB,
E E B.) KL (99, (B, Bo) 11q (B1))
(5)
EB, BB. BB. (log 9oh (X2[
where B,y > 0 are balancing coefficients chosen to reflect the trade-off between two information quantities. Here, KL (p (X) (X)) is the Kullback-Leile distributions (X) and q (X). Detailed derivations of the Eqns. (4) and (5) are presented in Appendix
(KL) divergence between the two
C.I. while the extension from the 3-modalities problem to N -oodliities has been elaborated in Appendix C
5

---

Published as a conference paper at ICLR 2024
2.3 TRAINING METHOD OF ITHP
We train the consecutive two-level hierarchy ITHP with an overall loss function labeled with X and X2. To minimize the overall loss, introduce the hyper-parameter À >> 0 that balances the trade-off between the losses of the two-level hierarchy:
Co Saveralt
X2) = Cloooo (Xo:X1) A- Ll (Bo: X2).
(6)
In the context of multimodal learning, we apply the final fused latent state Bx+1 the processing of representation learning tasks. Specifically, the final loss function combines the two-level hierarchy loss with rask-related loss function:
2
L
Lo overatt (2) + Ltask-reiated (B1,Y) Y)
3 +
The Eqn. introduces a new parameter a 2 0 to weigh the loss function of multimodal information integration and the loss function of processing downstream tasks. The pseudo-code of the ITHP algorithm is provided in Appendix D
3 EXPERIMENTS
In this section, we evaluate our proposed Information Theoretic Hierarchical Perception (ITHP) model on three popular multimodal datasets: the Multimodal Sarcasm Detection Dataset (MUSIARD: Castro et al. 2 the Multimodal Opinion-level Sentiment Intensity dataset (MOSI: Zadeh et al. 2016), and the Multimodal Opinion Sentiment and Emotion Intensity (CMU-MOSEI; Zadeh et al. 2018d). Detailed descriptions of the datasets can be found in Appendix F We demonstrate that the ITHP model is capable of constructing efficient and compact information flows. the sentiment analysis task, show that the TTHP-DeBRRTa framework outperforms human-level benchmarks across multiple evaluation metrics, including binary accuracy, Fl score, MAE, and Pearson correlation. All experiments were conducted on Nvidia A100 40GB GPUs. Detailed information regarding the model architecture and training parameters used in each experiment can be found in Appendix
3.1 SARCASM DETECTION
In the sarcasm detection task, ensure fair comparison, we utilize the same embedding data as provided in Castro et al. (2019). The embedding data was produced through multiple methods. Textual utterance features were derived from sentence-level embeddings using a pre-trained BERT model (Devlin et al. 2018), which provides representation of size d, = 768. For audio features, the Librosa library (McFee et al. 2018a) was utilized to extract low-level features such as MFCCs, Mel spectrogram, spectral centroid, and their deltas (McFee et al. 2018b), which are concatenated together to compose a da = 283 dimensional joint representation. For video features, the frames in the utterance processed using pool5 layer of an ImageNet (Deng et al.) 2009) pre-trained
ResNet- 152 model (He et al. 2016), which provides a representation of size d,, 2048.
In addition providing the multimodal embedding data, Castro al. (019) also conducted experi- ments using their own fine-tuned multimodal sarcasm detection model (referred to as "MSDM" in this paper). These experiments demonstrated that the integration of multimodal information can enhance the automatic classification of sarcasm. The results of their sarcasm detection task, as reported in Table [ (MSDM), serve as benchmark for comparing the performance of our proposed model. The classification results for sarcasm detection, obtained with different combinations of modalities (Text- T, Audio A, Video V), provide a basis for the comparative analysis conducted in our study.
Varying combinations of modalities: To test for the unimodal data, we initially established two-layer neural network to serve as a predictor for the downstream task. The binary classification results are evaluated by weighted precision, recall, and F-score across both sarcastic and non- sarcastic classes. The evaluation is performed using a 5-fold cross-validation approach to ensure robustness and reliability. When evaluating each individual modality using the predictor, the results are comparable to those the MSDM, with metrics hovering around 65%. This finding indicates that the embedding data from each modality contains relevant information for sarcasm detection. However, depending solely on a single modality appears to restrict the capacity for more accurate predictions. Considering the binary classification results across the three modalities and taking into account the size of the

---

Published as a conference paper at ICLR 2024
Information
X
Flow
Video
Feature
Zack: don't know, Somchody bigger and... Yeah, sure. you?
X1
By
Extraction
Text
X2
B1
Sarcasm
Audio
Figure 3: A schematic representation of our proposed ITHP and its information flow. The diagram illustrates the process of feature extraction from multimodal embedding data including video frames, text, and audio patterns. These modalities pass through "Feature Extraction" phase, where they are embedded to get modal states Xu, X1, and X2. The derived states are then processed to construct latent states Bg and B1. This processing includes reciprocal information exchange between X1 and Bu, as well as between B1 and X2. The resulting information from this process then used to make determination about the presence of sarcasm.
Modalities
Algorithm
Metrics
T-A
V-T
V-A
V-T-A
Precision
65.1
65.9
68.1
66.6
72.0
66.2
71.9
MSDM
Recall
64.6
64.6
67.4
66.2
71.6
65.7
71.4
F-Score
64.6
64.6
67.4
66.2
71.6
65.7
71.5
Precision 65.0
65.1
67.6
69.7
72.1
70.5
75.3
ITHP
Recall
64.8
64.9
67.5
69.1
71.7
70.3
75.2
F-Score 64.7
64.8
67.5
69.1
71.6
70.3
75.2
Table 1: Results of sarcasm detection on the MUStARD dataset. The table presents the weighted Precision, Recall, and F-Score across both sarcastic and non non-sarcastic classes, averaged across five folds. Underlined values indicate the best results for each row, while bold data represents the optimal results overall. The modalities are denoted as follows: T represents text, A stands for audio, and V signifies video.
embedding features for each modality (d, 2048, the prime modality and construct the consecutive modality states of V(Xo) T(X,) A(X2). During the evaluation of two-modal combinations, MSDM utilizes concatenated data to represent the combination, whereas we employ the single-stage information theoretical perception (TTP) approach, which involves only one latent state, denoted as Bo-
768, da 283), we designate V as
For the two-modal learning, there are three different combinations T(Xo) - A(X1), V(X))-((XX i and ((x)) - A(i)). In the case of T(Xo) A(Xi) and V(Xo) T(Xi), both MSDM and ITHP improved performance compared using single modality, indicating the advantages of leveraging multimodal information. However, is worth noting that the result of the combination V(X.) A(Xi) ) is lower than the result V(X)) from MSDM, suggesting that the model struggled to effectively extract meaningful information by combining the embedding features from video and audio. In contrast, ITP successfully extracts valuable information from these two modalities. resulting in higher metric values. For the combination V(Xo) T(X,) - A(X2), when considering the metrics of weighted precision, recall, and F-score, compared to the best-performing unimodal learning, MSDM's improvements are 5.58%. 5.93%, and 6.08%, respectively; whereas our ITHP model shows the improvements of 11.39% 11 11.41% % and 11.41% respectively. These results indicate that our ITHP model succeeds to construct the effective information flow among the multimodality states for the sarcasm detection task. Compared to the best-performing 2-modal learning. MSDM shows lower performance in the 3-modal learning setting, while our ITHP model demonstrates improvements 4.44% in precision, 4.88% in the recall, and 5 5.03% F-score, highlighting the advantages of the hierarchical architecture employed in ITHP model as shown in Fig. 3
Varying Lagrange multiplier: To conduct comprehensive analysis, we explore the mutual informa- tion among these modalities. In the IB method, the Lagrange multiplier controls the rrae-offf between two objectives: (i) maximizing the mutual information between the compressed representation B and the relevant variable Y to ensure that the pertinent information about Y is retained in B, and (ii)
7

---

Published as a conference paper at ICLR 2024
minimizing the mutual information between the compressed representation B and the original data X to eliminate redundant details in X.
A high value of the Lagrange multiplier encourages the model to retain more relevant information about Y. In our ITHP model learning for modalities, we have two Lagrange multipliers / and y to regulate the balance between the mutual information of the latent states and multimodal states. Here, we adjust the values of the Lagrange multipliers to observe their influence on the overall performance of the ITHP model for the sarcasm detection task, and the results are shown in Fig. When and 7 = 2, the model shows the worst performance with a weighted precision of 0.619 and recall of 0.607, which indicates that the model camnot retain the relevant information from the other modalities while losing the information from the input state with a low Lagrange multiplier, leading to even worse performance than the unimodal learning.
The best performance with precision of 0.753 and recall of 0.752 comes when 3 32 and 8. The region around the point is also the brightest 3 x square with all values above 04, which indicates optimal balance to construct the most effective information flow among the varying values of 3 and 7. It is worth noting that the upper right triangular part is brighter than the lower left triangular part, which shows that the model benefits more from a higher B than higher 7. suggests that retaining more relevant information from Text (X₂) is more important for sarcasm detection, which is also verified by the best performance of the combi- nation V(Xo) T(X1) among all the two modality combinations.
Precision
Recall
16
32
0.65
16
32 64
16
Figure 4: Weighted precision and recall for the binary classification task under varying Lagrange multipli- ers. The graph shows the impact of varying the I Lagrange multipliers and 7). For each plot, the Red (Orange) color denotes highest (lowest) value, respectively.
3.2 SENTIMENT ANALYSIS
Sentiment Analysis (SA) has undergone sub- stantial advancements with the introduction of multimodal datasets, incorporating text, images, and videos Gandhi et al. (2022). Our study concentrates on two pivotal datasets: the Mul- timodal Corpus of Sentiment Intensity (MOSI) and Multimodal Opinion Sentiment and Emo- tion Intensity (MOSEI). Notably, the MOSI dataset employs a unique feature extraction pro- compared to sareasm detection, with bedding feature sizes of d, = 768, d₂ = 74, and d, 47 for text, audio, and video, re- spectively. We hypothesize that the text em- bedding features T' hold the most substantial information, followed by A. leading to the formulation of consecutive modality states of T(X)) A(X1) ((X2) for this task.
CMU-MOSI
Task Metric
BA
F11
MAET
Corrt
BERT
Self-MM;
84.0
84.4
0.713
0.798
MMIM,
84.1
84.0
0.700
0.800
MAG,
84.2
84.1
0.712
0.796
Self-MMa
55.1
53.5
1.44
0.158
MMIM
85.8
85.9
0.649
0.829
MAGa
86.1
86.0
0.690
0.831
UniMSE
85.9
85.8
0.691
0.809
MIB
85.3
85.3
0.711
0.798
BBEN
84.3
84.3
0.776
0.755
ITHP
88.7
88.6
0.643
0.852
Human
85.7
87.5
0.710
0.820
Table Model performance on CMU-MOSI dataset. The table presents the performance other BERT-based (indicated with subscript "b") and DeBERTa-based (in- dicated with subscript "d") pre-trained models, Models developed
In evaluating the versatility of our approach across diverse multimodal language datasets, experiments are also conducted on the CMU- MOSEI dataset (Zadeh al.) 2018d) 8d). The CMU-MOSEI dataset, while resembling MOSI in annotating utterances with sentiment intensity, places greater emphasis positive sentiments. Additionally, it provides annotations for nine discrete and three continuous emotions, expanding its applicability for various affective computing
are highlighted in bold, and optimal sults are underlined.
tasks Das & Singh (2023): [Ling et al. (2023).
8

---

Published as a conference paper at ICLR 2024
Our analysis encompasses a range of prominent models, including Self-MM (Yu et al. 2021). (Han et al. 2021b), MAG (Rahman et al., 2020), MIB (Mai et al., (022). BBFN Han et al.
MMIM
2021a).
and UniMSE (Hu et al. 2022). Please refer to Appendix B for details.
Delving into framework integration, this research incorporates ITHP with pre-trained DeBERTa models, optimizing textual data processed by DeBERTa's 's Embedding and Encoder layers, unlike the MAG model which utilizes embedding data from BERT's middle layer. A significant development introduced is ITHP-DeBERTS integrating the Decoding-enhaneed BERT with Disentangled Attention (DeBERTa: He et al., 2020; 2021). This advanced model enhances BERT and RoBERTa (Liu et al. 2019 through disentangled attention and a sophisticated mask decoder.
The results on the CMU-MOSI dataset are reported in Table 2 Additionally, Table provides reference results for more DeBERTa-basee baseline models. As shown in the results, our ITHP model outperforms all the SOTA models in both BERT and DeBERTa incorporation settings. It is noteworthy that significant strides per- formance were observed with the integration of DeBERTa with the ITHP model, surpassing even human levels. This integration with De- BERTa resulted a 3.5% increase in Binary
4 in Appendix F.]
CMU-MOSEI
Task Metric
BA
F11
MAE
Corrt
Self-MMa
85.0
85.0
0.529
0.767
MMIM,
86.0
86.0
0.526
0.772
Classification Accuracy (BA), 1.3% improve- ment in F1 Score (F1), a reduction of 9.4% in Mean Absolute Error (MAE), and a 3.9% increase in Pearson Correlation (Corr). To ex- plore the applicability of our approach var- ious datasets, we perform experiments on the extensive CMU-MOSEI dataset. The results in Table 3 demenstrate that our ITHP model consistently surpasses the SOTA, thereby con- firming its robustness and generalizability. In contrast, it is worth noting that Self-MM itself heavily relies on the feature extraction process performed by BERT, resulting in a significant degradation of DeBERTa-based Self-MM on both CMU-MOSI and CMU-MOSEI datasets, which restricts its applicability in certain scenarios.
MAG,
84.8
84.7
0.543
0.755
Self-MMa
65.3
65.4
0.813
0.208
MMIMA
85.2
85.4
0.568
0.799
MAGa
85.8
85.9
0.636
0,800
ITHP
87.3
87.4
0.564
0.813
Table Model performance on CMU-MOSEI dataset. Models developed by us are highlighted in bold, and optimal results are underlined.
CONCLUSION
Drawing inspiration from neurological models information processing, we build the links between different modalities using the information bottleneck (IB) method. By leveraging the IB principle, our proposed ITHP constructs compact and informative latent states for information flow. The hierarchical architecture of ITHP enables incremental distillation of useful information with applications in perception of autonomous systems and cyher-physical systems. The empirical results emphasize the potential value of the advanced pre-trained large language models and the innovative data fusion techniques in multimodal learning tasks. When combined with DeBERTa, our proposed ITHP model the first work, based on our knowledge, outperform human-level benchmarks on all evaluation metrics (i.e., BA, FI, MAE, Corr).
5 LIMITATIONS AND FUTURE WORK
A potential challenge of our model is its reliance on preset order of modalities. We've illustrated this aspect with experiments showcasing various modalities' orders Appendix Table II in Appendix Typically, prior knowledge is utilized to select the primary modality and rank others. Without any knowledge, addressing this challenge by adjusting the Lagrange multipliers or iterating through all possible modality orders can be very time-consuming, Another potential challenge emerges when the primary modality lacks sufficient information independently, necessitating complementary information from other modalities for optimized performance. Nonetheless, we believe that the integration of deep learning architecture into the hierarchical perception can mitigate this issue. For more detailed discussion, please refer Appendix H Future work will focus on addressing the identified challenges by exploring how to utilize deep learning frameworks to achieve efficient neuro-inspired modality ordering.
F.5

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENT
The authors acknowledge the support by the U.S. Army Research Office (ARO) under Grant No. 191-NF-33-1-0111 the National Science Foundation (NSF) under the Career Award CPS-1453860, CCF-1837131, MCB-1936775, CNS-1932661 and the NSF award No. 2243 104 under the Center for Complex Particle Systems (COMPASS), the Defense Advanced Research Projects Agency (DARPA) Young Faculty Award and DARPA Director Award under Grant Number N66001-17-1-4044, an Intel faculty award and Northrop Grumman grant. The authors are deeply grateful to Nian Liu, Ruicheng Yao, and Xin Ren from Tuyou Travel AI for their efforts in promoting industry integration and providing both computing support and helpful feedback. The views, opinions, and/or findings in this article are those of the authors and should not be interpreted as official views or policies of the Department of Defense or the National Science Foundation.
10

---

Published as a conference paper at ICLR 2024
REFERENCES
Yitao Cai, Huiyu Cai, and Xiaojun Wan. Multi-mode sarcasm detection Twitter with hierarchical fusion model. In Auna Korhonen, David Traum, and Lluís Marquez (eds.), Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 2506 2506-2515. Florence, Italy, July 2019a, Association org/
for Computational Linguistics. doi: 10. 18613/11111111111 1239, URL P19-1239
Yitao Cai, Huiyu Cai, and Xiaojun Wan. In Proceedings of the 57th annual meeting of the association for computational linguistics, pp. 2506-2515, 2019b.
sarcasm detection in twitter with hierarchical fusion model.
Santiago Castro, Devamanyu Hazarika, Verónica Pérez-Rosas, Roger Zimmermann. Rada Mihalcea, and Soujanya Poria. Towards multimodal sarcasm detection (an _bvviousl__. perfect paper). arXiv preprint arXiv:190.001815, 2019.
Ringki Das and Thoudam Doren Singh. Multimodal sentiment analysis: A survey of methods, trends and challenges. ACM Computing Surveys, 2023.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEFE conference on computer vision and pattern recognition, pp. 248-255 leee, 2009.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training deep bidirectional transformers for language understanding. arXiv preprint 1rXiv:1810.04805, 2018.
Nir Friedman, Ori Mosenzon, Noam Slonim, and Naftali Tishby. Multivariate information bottleneck. In Proceedings of the Seventeenth conference Uncertainty in artificial intelligence, pp. 152-161. Morgan Kaufmann Publishers Inc., 2001.
Ankita Gandhi, Kinjal Adhvaryu, Soujanya Poria, Erik Cambria, and Amir Hussain. Multimodal sentiment analysis: A systematic review of history, datasets, multimodal fusion methods, applications, challenges and future directions. Information Fusion, 2022.
Gary H Glover. Overview of functional magnetic resonance imaging. Neurosurgery Clinics, 22(2):133-139, 2011.
Wei Han, Hui Chen, Alexander Gelbukh, Amir Zadch, Louis phiispphilippe Morency, and Soujanya Poria. Bi- bimodal modality fusion for correlation-contron controlled multimodal sentiment analysis. Proceedings of the 2021 International Conference on Multimodal Interaction, pp. 6-15, 2021a.
Wei Han, Hui Chen, and Soujanya Poria. Improving multimodal fusion with hierarchical mutual information maximization for multimodal sentiment analysis. arXiv preprint arXiv:2109.0040 2021b.
Devamanyu Hazarika, Soujanya Poria, Amir Zadeh, Erik Cambria, Louis-Philiyee Morency, and Roger Zimmer- mann. Conversational memory network for emotion recognition dyadic dialogue videos. In Proceedings of the conference. Association for Computational Linguistics. North American Chapter. Meeting, volume 2018, pp. 2122. NIHI Public Access, 2018.
Devamanyu Hazarika, Roger Zimmermann, and Soujanya Poria. Misa: Modality-invarian and-specific repre- sentations for multimodal sentiment analysis. In Proceedings of the 28th ACM International Conference on Multimedia, pp. 122-1131, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Decoding-enhanced bert with disentangled attention. arXiv preprint arXiv:2006.0365.. 2020.
Pengcheng Hc, Jianfeng Gao, and Weizhu Chen. Debertav3: Improving deberta using electra-style pre-training with gradicnt-disentangle embedding sharing, 2021.
Ron M Hecht and Naftali Tishby. Extraction of relevant speech features using the information bottleneck method. In Ninth European Conference on Speech Communication and Technology, 2005.
Guimin Hu, Ting-En En Lin, Yi Zhao, Guangming Lu, Yuchuan Wu, and Yongbin Li. Unimse: Towards unified multimodal sentiment analysis and emotion recognition. arXiv preprint 1rXiv:2211.11255 2022.
11

---

Published as a conference paper at ICLR 2024
Kung-IIsiang Huang, Mingyang Zhou, Hou Pong Chan, Yi R. Fung, Zhenhailong Wang, Lingyu Zhang, Shih-Fu Chang, and Heng Ji. Do Ivims understand charts? analyzing and correcting factual errors chart captioning. 2023.
Kuig-Hsiang Huang, Hou Pong Chan, Yi Fung, Haoyi Qiu, Mingyang Zhou, Shafiq Joty, Shih-Fu Chang, and Heng Ji. From pixels to insights: A survey on automatic chart understanding in the era of large foundation models. 2024.
Pierre Lavenex and David G. Amaral. Hiipocaampll-neocortiiaal interaction: A hierarchy of associativity. Hippocampus, 10(4):400-300, 2000. doi: http:///ww...a.00/////////000000 3.0.CO:2-5.
Changhee Lee and Mihacla van der Schaar. A variational information bottleneck approach to multi-omics data integration, 2021.
Paul Pu Liang. Ziyin Liu, Amir Zadeh, and Louis Chiis-Philippe Morency. Multimodal language analysis with recurrent multistage fusion, 2018.
Paul Pu Liang, Yiwei Lyu, Gumjan Chhablani, Nihal Jain, Zihao Deng, Xingbo Wang. Louis-Phillppe Morency, and Ruslan Salakhuldinov. Multiviz: Towards visualizing and understanding multimodal models. In The Eleventh International Conference on Learning Representations, 2023.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du Mandar Joshi, Dangi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanow. Roberta: A robustly optimized bert pretraining approach, 2019.
Zhun Liu, Ying Shen, Varun Bharadhwaj Lakshminarasimhan, Paul Pu Liang, Amir Zadeh, and Louis- Philippe Morency. Efficient low-rank multimodal fusion with modality-speciinf factors. arXiv preprint 1rXiv:106.000064 2018.
Fengmao Lv, Xiang Chen, Yanyong Huang, Lixin Duan, and Guosheng Lin. Progressive modality reinforcement for human multimodal emotion recognition from unaligned multimodal sequences. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2554-2562, 2021.
Sijie Mai, Ying Zeng, and Haifeng Hu. Multimodal information bottleneck: Learning minimal sufficient. unimodal and multimodal representations. TEEE Transactions on Multimedia, 2022.
Brian McFee, Matt Vicar,r, Stefan Balke, Carl Thomé, Vincent Lostanlen, Colin Raffel, Dana Lee, Oriol Nicto, Eric Battenberg, Dan Ellis, et al. Wzy. Rachel Bittner, Keunwoo Choi, Pius Friesch, Fabian-Rober Stter, Matt Vollrath, Siddhartha Kumar, nehz, Simon Waloschek, Seth, Rimvydas Naktinis, Douglas Repetto, Curtis" Fjord" Hawthorne, CJ Carr, Joo Felipe Santos, JackicWu, Erik, and Adrian Holovaty."lirros//ibn "ibbosa//ibrooa: 0.6, 2, 20.
Brian McFee, Matt Vicarar, Stefan Balke, Carl Thomé, Vincent Lostanlen, Colin Raffel, Dana Lee, Oriol Nieto, Eric Battenberg, Dan Ellis, Ryuichi Yamamoto. Josh Moore, WZY, Rachel Bituner, Keunwoo Choi, Pius Friesch, Fabian-Robert Stöter, Matt Vollrath, Siddhartha Kumar, nehz, Simon Waloschek, Seth, Rimvydas Naktinis, Douglas Repetto, Curtis "Fjord" Hawthorne, CJ Carr, João Felipe Santos, JackieWu, Erik, and Adrian Holovaty. librosa/librosa: 0.6.2, August 2018b. httpttttppe//...ooggggg/.o 5282://.0..81//eoooo. 1342708
M M Mesulam. From sensation cognition. Brain, 21(66)1013-1052 06 1998. ISSN 0006-8950. doi: 10. 1093/brai//11...10113
David Meunier, Renaud Lambiotte, Alex Fornito, Karen Ersche, and Edward T Bullmore. Hierarchical modularity in human brain functional networks. Frontiers neuroinformatics, 3:571, 2009.
Sunghyun Park, Han Suk Shim, Moitreya Chatterjee, Kenji Sagae, and Louis-Philippe Morency. Computational analysis of persuasiveness social multimedia: novel dataset and multimodal prediction approach. In Proceedings of the 16th International Conference on Multimodal Interaction, pp. 50-57, 2014.
Alvaro Pascual-Leone and Roy Hamilton. The metamodal organization of the brain. Progress in brain research. 134:427-44 445, 2001.
Hai Pham, Paul Pu Liang, Thomas Manzini, Louis-Philippe Morency, and Barnabás Póczos. Found in translation: Learning robust joint representations by cyclic tramslations between modalities. Proceedings of the AAAI Conference Artificial Intelligence, 33001):6822-6999 Jul. 2019. doi: 10.1609/nai.v33110.3330000 33990.330166699. URL /tttss//::// aaai. http:////eeeeeep//////////////e/
Soujanya Poria, Amir Hussain, and Erik Cambria. Multimodal sentiment analysis. Springer, 2018.
12

---

Published as a conference paper at ICLR 2024
Cathy J Price. A review and synthesis of the first 20 years of pet and fmri studies of heard speech, spoken language and reading. Neuroimage, (2(2):816-888 847, 2012.
Wasifur Rahman, Md Kamrul Hasan, Sangwu Lee, Amir Zadeh, Chengfeng Mao, Louis-Philippe Morency, and Ehsan Hoque. Integrating multimodal information in large pretrained transformers. In Proceedings of the conference. Association for Computational Linguisties. Meeting. volume 2020, pp. 2359. NIH Public Access, 2020.
Ryan V Raut, Abraham Z Snyder, and Marcus E Raichle. Hierarchical dynamics macroscopic organizing principle the human brain. Proceedings of the National Academy of Sciences, 1177344)20890-20897, 2020.
Jorge Sepulcre, Mert R. Sabuncu, Thomas B. Yeo, Hesheng Liu, and Keith A. Johnson. Stepwise connectivity of the modal cortex reveals the multimodal organization of the human brain. Journal of Neuroscience, 32(31): 10661 2012. ISSN 0270-6474. doi: (0.1523/NEUROSCI.ON
12.2012.
Ravid Shwartz-Ziv and Naftali Tishby. Opening the black box of deep neural networks via information. arXiv preprint 1rXiv:1703.00SS 2017.
Noam Slonim and Naftali Tishby. Document clustering using word clusters via the information bottleneck method. In Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval, pp. 208-20110 205- ACM, 2000.
Yijun Tian, Chuxu Zhang, Zhichun Guo, Yihong Ma, Ronald Metoyer, and Nitesh V Chawla. Recipe2vec: Multi-modal recipe representation learning with graph neural networks. In IICAL, 2022.
Yijun Tian, Yikun Han, Xiusi Chen, Wei Wang, and Nitesh Chawla. Tinyllm: Learning small student from multiple large language models. arXiv preprint a1Xiv:240.006616 2024.
Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle. In Information Theory Workshop (ITW), pp. 1-5. IEEE, 2015.
Naftali Tishby, Fernando C Pereira, and William Bialek. The information bottleneck method. arXiv preprint physics/0004057, 2000.
Giulio Tononi, Gerald M Edelman, and Olaf Sporns. Complexity and coherency: integrating information in the brain. Trends in cognitive sciences, 2(12):4444484 1998.
Yao-Hung Hubert Tsai, Shaojic Bai, Paul Pu Liang. J Zico Kolter, Louis-Philippe Morency, and Ruslan Salakhutdinov. Multimodal transformer for unaligned multimodal language sequences. In Proceedings of the conference. Association for Computational Linguistics. Meeting, volume 2019, pp. 6558. NIH Public Access, 2019.
Yansen Wang, Ying Shen, Zhun Liu, Paul Pu Liang, Amir Zadeh, and Louis-Philippe Morency. Words can shift: Dynamically adjusting word representations using nonverbal behaviors. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 7216-7223, 2019.
Changsong Wen, Guoli Jia, and Jufeng Yang. Dip: Dual incongruity perceiving network for sarcasm detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2540 2550, 2023.
Xiongye Xiao, Defu Cao, Ruochen Yang. Gaurav Gupta, Gengshuo Liu. Chenzhong Yin. Radu Balan, and Paul Bogdan. Coupled multiwavelet operator learning for coupled differential equations. In The Eleventh International Conference on Learning Representations, 2022.
Yuankun Xue and Paul Bogdan. Constructing compact causal mathematical models for complex dynamics. In Proceedings of the 8th International Conference Syber-Physical Systems, pp. 97 107, 2017.
Zihui Xue and Radu Marculescu. Dynamic multimodal fusion. In Proceedings of the IEEE/VV Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 2575-2584, June 2023.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. Xlnet: Gen- eralized autoregressive pretraining for language understanding. Advances in neural information processing systems, 32, 2019.
Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. A survey on multimodal large language models. arXiv preprint ar Xrx::23006.1 13549, 2023.
13

---

Published as a conference paper at ICLR 2024
Tianshu Yu, Haoyu Gao, Ting-En Lin, Min Yang. Yuchuan Wu, Wentao Ma, Chao Wang. Fei Huang, and Yongbin Li. Speech-text dialog pre-training for spoken dialog understanding with explicit cross- alignment, 2023.
Wenmeng Yu,
Ziqi Yuan, and Jiele Wu. Learning modality-speeiifi representations with self supervsse multi-task learning for multimodal sentiment analysis. Proceedings the AAAI conference on artificial intelligence, (5(12):0770-10799) 2021.
Amir Zadeh, Rowan Zellers, Eli Pincus, and Louis Phill-ph Morency. Mosi: multimodal corpus of sentiment intensity and subjectivity analysis in online opinion videos. arXiv preprint arXiv:
2016.
Amir Zadeh, Minghai Chen, Soujanya Poria, Erik Cambria, and Louis-Philippe Moreney, Tensor fusion network for multimodal sentiment analysis, 2017. MfXiv:1707.07250
Amir Zadeh, Paul Pu Liang, Navonil Mazumder, Soujanya Poria, Erik Cambria, and Louis Memory fusion network for multi-view sequential learning, 2018a.
Louis-Philippe Philippe Morency.
Amir Zadeh, Paul Pu Liang, Soujanya Poria, Prateck Vij, Erik Cambria, and Louis- PhiliispPPpppp Morency, Multi- attention recurrent network for human communication comprehension. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1), Apr. 2018b. doi: 10. v3291. 12214.10024 URL https://ojs.aaa
org/ ndex. php/ AAA.AAAAaaaataaaaa clevieew/ 12024
Amir Zadeh, Paul Pu Liang, Soujanya Poria, Pratcek Vij, Erik Cambria, and Louis-Philippe Moreney. Multi- attention recurrent network for human communication comprehension, 2018c. arXiv:1802.000223
AmirAli Bagher Zadeh, Paul Pu Liang. Soujanya Poria, Erik Cambria, and Louis-Phillipe Morency. Multimodal language analysis in the wild: Cmu-mosci dataset and interpretahle dynamic fussinn graph. In Proceedings of 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp-
2d.
Shu Zhang, Qinglin Dong, Wei Zhang, Heng Huang, Dajiang Zhu, and Tianming Liu. Discovering hicrarchical common brain networks via multimodal deep belief network. Medical image analysis, 54:238-252, 2019.
Mingyang Zhou, Yi R Fung, Long Chen, Christopher Thomas, Heng Ji, and Shih Chang. Enhanced chart understanding in vision and language task via cross-modal pre-training on plot table pairs. In 61st Annual Meeting of the Association for Computational Linguistics, ACL 2023, pp. 1314-1326, Association for Computational Linguistics (ACL), 2023.
14

---

Published as a conference paper at ICLR 2024
A
CODE AVAILABILITY
Our codebase can be found in http::/gthtu.ccmm//ssuuxaa///// Our code uses assets from MAG-BERT (Rahman al., (020). BERT (Devlin al. 2018)
Apache License 2.0]. and DeBERTa (He et al.
2020 2021)
https://attho.comm/mmiccos////ss MIT License].
B RELATED WORK
Multimodal sentiment analysis has been a prominent research area, focusing on the integration of verbal and nonverbal information across different modalities such as text, acoustics, and vision. Prior studies in this field have primarily concentrated on representation learning and multimodal fusion approaches. For representation learning, Wang et al. (Wang al., 2019) proposed recurrent attended variation embedding network to learn the shifted word representations, while Hazarika et al. and modality-specifie representations. In terms
(Hazarika et al., 2020) introduced modality-invarinn of multimodal fusion, models can be categorized into early fusion and late fusion methods. Early fusion models incorporate delicate attention mechanisms for cross-modal fusion, as demonstrated by Zadeh et al. (Zadeh et al., 2018a) who developed memory fusion network for cross-view interactions, and Tsai et al. (Tsai et al. 2019) who proposed cross-modal transformers to enhance a target modality through cross-modal attention. Late fusion methods, on the other hand, prioritize intra-modal representation learning before inter-modal fusion. For instance, Zadeh et al. (Zadeh) et al. 2017) utilized a tensor fusion network to compute the outer product between unimodal representations, while Liu et al. (Liu et al. 2018) introduced a low-rank multimodal fusion method to reduce computational complexity.
In recent years, the fusion methods, combined with advanced Transformer-based large language models such as BERT (Devlin et al. 2018) and XLNet (Yang et 2019), have achieved remark- able progress in multimodal sentiment analysis. These methods have reached near-human level performance on the CMU-MOSI dataset in the task of multimodal sentiment binary classification (Rahman et al. 2020). Innovations such as Self-MM (Yu et al. (202)) I) have introduced self-supervised strategies that refine unimodal label learning, while MMIM (Han et al., 2021b) utilizes mutual information to enhance sentiment discernment from multimodal data. The MAG approach (Rahman al. 2020) integrates affective gaps between modalities, and MIB (Mai al.) (022) focuses on isolating essential data features, filtering out the extraneous for purer multimodal fusion. BBFN (Han) ct al.) 2021a) reimagines the fusion process by merging and differentiating between modal inputs. The current SOTA model presents novel insights, UniMSE (Hu et al. 2022) innovatively integrates multimodal sentiment analysis and emotion recognition in conversations into a unified framework. enhancing prediction accuracy by leveraging the synergies and complementary aspects of sentiments and emotions. The SPECTRAL ( Yu et al. 2023) is pioneering in spoken dialog understanding by pre-training on speech and text, featuring unique temporal position prediction task for precise speech-text alignment.TTi The DyпMM (Xue & Marculescu 2023) creates the possibility of reducing computing resource consumption. By introducing dynamic multi-modal fusion, which fuses input data from multiple modalities adaptively, leading to reduced computation, improved representation power, and enhanced robustness.
Additionally, the latest work in the field of sarcasm detection has expanded our research horizons. The HFM (Cai et al. 2019a) utilizes early fusion and representation fusion techniques to refine feature representations for each modality. It integrates information from various modalities to better utilize the available data and enhance sarcasm detection performance. The DIP et al.) 2023) creatively adopts dual incongruity perception at factual and affective levels, enabling effective detection of sarcasm in multi-modal data. Aside from multimodal sentiment analysis, numerous applications
involve the integration of multiple modal inputs (Tian et al. 2022 2024; Huang et al.) 2023 2024
Xiao et al. 2022), making the management of modal interactions a significant area of ongoing research interest.
15

---

Published as a conference paper at ICLR 2024
C DERIVATION OF THE VARIATIONAL IB LOSS FOR ITHP
C.I 3-MODALITIES SCENARIO
In the realm of multimodal learning, which usually involves three useful modalities, we outline loss function for 3-modalities setting in our study. This scenario incorporates three consecutive modalities, Xu. X1, and X2, these three modalities are ranked according to the richness of main information. In this section, we derive the formula of the two-level information bottleneck hierarchy the three-modal problem detail, as shown in Eqn. and Eqn. (5)
Equation represents the loss of the input-level Information Bottleneck as (Xo Bo X1) in the system, while the Eqn. (5) represents general expression the subsequent-level Information Bottleneck as (Bo- B1 X2), as shown in Fig. ( In this section, first show the detailed derivation of the loss function 4 and then give the derivation of Eqn. (5) in a concise manner. Firstly, the loss function of the first input-level Information Bottleneck can be given
CTB. (Xo; X1) (Xo: Bo) 3T (Bo; X1),
where > trading-off parameter chosen to balance between the two mutual information quantities. Here, I (Xo; Bo) and / (Bo; X1) can be derived using variational approximations, i.e., qe, (o) and qvo Bo), as follows:
(Xo: Bo) = / dXydBop (Xo, Bo) log
p (Xo: Bu)
p (Xo) .p (Bo)
p (Bo Xo)
dXadBoo (Xo, Bo) log
(Bu)
- - d.Bopl (Bo) log p (Bo)
dXudBop (Xo, Bo) log p Bol (BolXo)
(9)
d'udBBo (Xo, Bo) log qo. (BolXo) - / dBop (Bo) log a (Bo)
dXndBoo (Xo) q6, (BolXo) log 98 (BolXo)
y (Bo)
E X(xo(()) [K (go, Xoo |1g (Bo))],
where qo, (o) is used as the variational approximation for p (Bo (Xo) and g (Bb) N (0,
the probability distribution over the latent states based on the assumption.
/ dBodXip (X1, Bo) log
p (X1, Bo)
(Bo; X1)
p (X1) (Bo)
/ dBddXip (X1, Bo) log
p [X1] (Bo)
p (X1)
Tvo (X1 (B)
dBndXip (X1, Bo) log
(10)
p (X1)
/ dBndXip (X,, Bb) log qvv (X, dXip((i)) log (X1)
/ dBodXip (X1, Bo) log qvo (x) + (X1),
where H (X1) is the entropy of Xi, which is determined only by the distribution of X, itself, so this can be ignored in the loss function. Since we cannot derive the distribution of p (Bo)Xi) directly, we have to introduce the modal Xo with the relationship that Bp is independent of X1 given Xo
f dXop (Xo, X1, X1, Bo) dXop (Xo) (XX])XX)) (Bo) (11)
(X1, Bp) =
16

---

Published as a conference paper at ICLR 2024
X
X
X
X A-1
X x+2
X XN-1
X
Bp
B
B
B
By-1
Y
Figure 5: N latent states constructed for extracting and transferring the relevant information of of N + modal states. The modal states are represented by Xo. Xi..., Xx-1. Xx. This order is determined by the richness of the amount of information contained in the modalities. The latent states Bo. B1. Bi.... By -1 represent the pathway for transferring the relevant information among X. Through the hierarchical structure, the most relevant information from Xo to Xx gradually retained and distilled.
Therefore, (Bo; X1) can be given by
T (Bo; X1) = / (X)dBelXip(Xo) (X1(X)) (p llgg Aslo (X1)
(12)
(o Batoop/Boxx,BxxxppeCe (xox)(Xo) [log 44o (Xi]Bo)],
where yvo (oo are used the variational approximation for p (Bo Xo). Then have the derivation of the Eqn. (4) as
L
(1) I (Xo; Bo) BI (Bo;
E Exo=p(Xx) KL (40, (Bol llg (Bo))
(13)
B E Bo Bo No Xo [log Ivo (X, (X1
And we can easily get the derivation of the Eqn. 5 in similar steps above. The loss function of the subsequent-level Information Bottleneck shown as:
01.01 X2) = (Bo: B1) (B1: LBB
(14)
The results of / (Bo; B,) and / (B1; X2) are shown
(Bo. B1)
T (Bp; Bτ)
d.BdBBpp (Bo, B1) log
p (BB)) (Bt)
9o, BO)
(15)
=
d.BudBip (Bo) q0, (B, Bb) log
q (Bi)
-E -B [KL (go, (Bi Bo) 11g (B)))).
dB, dB,dX2p(X2 Bi) log (X2, Bi)
I (B1:
(X2) (BB)
(16)
dB dBBdX2p (X2, B1) log av. (X2 Bi) + H1 (X2)
=EB,<p(B, Ba) Bo) [log gv (X2l B1)]
Then we have the derivation of the Eqn. as
01. (Bo; X2) = I (Bo: Bi) 71 (B1;X2)
~ EB Ba) K L (go, (B, Bo) 11q (Bi))
(17)
- EB, (B (B BBo [log qoh (X2
C.2 GENERAL SCENARIO FOR N MODALITIES
In what follows, we present the loss function for scenarios involving more than modalities, thus, a framework for more general multimodal learning case. Let's consider a multi-modalities learning system with N informative modalities, as shown Fig. 5 According to the difference information richness in N modalities, according to information richness as X,, X,, ...... _x - Xxv Therefore, we start integrating the multimodal information from the modal state Xo with the highest information richness. The input
sort the information of these N modalities, and ranked from high to low
17

---

Published as a conference paper at ICLR 2024
level Information Bottleneck loss can be derived from Eqn. (4). which is the same as 3- 3-modalities scenario. The loss function is shown as:
20.400 LBo (xu;X1] = I (Xo: Bu) BI (Bu; X1)
= E (Xx) LL (40, (BolXo) lla (Bo))
(18)
- B .. Bon-p(Bl|Xo)) Xxx [log Tvo [o]]-
Then, the subsequent level IB loss can be given as:
0011. LB8-11
(Bxi Xx+2)
(Bxi BE+1) - Tkd BB++1: Xx+2)
= EB, upl B&) KL (go.41 (Bx+ [Bx) 11a (B++1)
(19)
Y E mm (
But IBB)) ) ~B( [log ges+1 (x+))B
(k 0,
N 2)
Then, we train the overall network using N modal states. It is worth mentioning that in our model (ITHP), only Xo with the highest information richness used as the model input, and X........ .N X act detectors in the information flow. To minimize the overall loss, introduce a series of Lagrange multipliers Ag 0 that balance the trade-off between the losses the input-level IB and all subsequent IB levels in the whole hierarchical structure. Therefore, the overall loss function mentioned Eqn. can be generalized to the form with N modal states as:
N-2
c overalli (N):x)) = B (X): +
Σ As k=
1111k 1Bb44
(Br; Xx42).
(20)
With the overall loss derived here, we can further get the expression of the loss function considering the downstream task, so the Eqn. (7) can be generalized to the form with N modal states as:
C.v. (Xo, NN) ) + Lrask- related B). (21)
B + E 7
D ALGORITHM: PSEUDO-CODE FOR TRAINING ITHP
D. 3-MODALITIES SCENARIO
We provide the pseudo-code for ITHP in 3-modalities problem (please refer to Algorithm [ which we used in our experiments including both sarcasm detection and sentiment analysis. As mentioned in C we have (X, X1 X2) as the three modal states. Here, we present the algorithm for training ITHP 3-modalities scenario.
D.2 GENERAL SCENARIO FOR MODALITIES
We additionally provide the pseudo-code and the training method for ITHP general form involving N modalities (please refer to Algorithm 2 This broader algorithmic representation extends from the 3-modalities example delineated in
E MODEL ARCHITECTURE AND HYPERPARAMETERS
In this section, we provide a detailed description our model architecture and the hyperparameter selection in different tasks for reproducibility purposes. For the ITHP, depicted in Figure 2 to get the latent state's 's distribution encoder, where the first layer extracts the feature vectors of input Xu and the second layer outputs parameters for a latent Gaussian distribution a mean vector and the logarithm of the variance vector. By sampling from this distribution, obtain latent state that represents the input state further used
Bg, we directly use two fully connected linear layers to build the
18

---

Published as a conference paper at ICLR 2024
Algorithm 1 Pseudo-code for training ITHP in 3-modalities scenario
Input:
Dataset D ( X2), Y); Coefficient: a, 9, A; Learning rate ry; Mini-batch size
nomb; Assumed distribution of latent states {Bo, B1 = & r N (0, 1);
Output: ITHP parameters ([0x) k=0,1 (1hx k=o,)= Task-related prediction y;
Initialize the parameters in the Neural Networks (f0) k f (=
repeat
Sample a mini-batch: {(Xp, XT, X2), Y" 1")
18mb
~
for
1, 2,
7lmb do (X0)
Mo: Σn <- S0p
2p f- Mg + 20
X f- fv (28)
11, 21 <- f0, (2g)
21 - M1 + En
X. 5 fw (ZP)
end for
Xo, X1, X2 are categorical data then
LiB,
800 KL (N (mB. 27) || (o, 1))+
Dmmb
Σ XI [C] log
[X] [C]
Tomb
0 KL (N (pf mn) || (0, 1))+ Mmmb
Dn
2C=1 X3 [C] log
C
Xx [C]
else if Xo, X1. X2 are continuous data then
00.00 LBB
KL (N (p6. 23) | (0, 1)) +
Ix=l
X1 X1"
2
Tlmb
01.4/ LB,
KL (N (MT, ΣT) I (0, 1)) +Y
mmb
X2 X3
Rind
n=1
end if
overalll
LBo
No + X LBi
Liask rrelttd
B4g Boveraal + Amm
2 8.v
Damm (2f y
(Ok, 1
(0k, v/) πl Vikks ke vn related. (k 0,
until convergence
to (I) feed into the multilayer perceptron (MLP) layer to predict X1; (2) feed into the next hicrarchical layer for another modality information. Note that this is serial structure, and the remaining layers share the same design. The predictor is chosen specifically by the downstream tasks. For the task of sarcasm detection, elect use MLP with two layers the predictor. For the sentiment analysis task, the model acoustic and visual modalities are pre-extracted from CMU-MOSI (Zadeh et al.) (016)). The textual features are extracted by pre-trained DeBERTa (He et al.) 2020 (021). The textual features serve as input to our ITHP module while the pre-extracted acoustic and visual act as the detectors. The distilled representations that capture the information from multimodalities are generated from the ITHP module. Additionally, we incorporate residual connections for enhanced information flow. Finally, I-layer Transformer encoder and several fully-connected layers are designed as predictor to make sentiment predictions.
trained in an end-to-ena manner. The raw features of
For the task of sarcasm detection, unless otherwise specified, we set the hyperparameters as follows: 32.7 = 8, X = 1. We perform a 5-fold validation, and for each experiment, we train the ITHP model for 200 epochs using an Adam optimizer with learning rate of 10-3, For the task of sentiment analysis, unless otherwise specified, we set the hyperparameters as follows: = 8, 32, A 1. We run each experiment for 40 epochs using an Adam optimizer with learning rate of 10-5 We repeat each experiment times to calculate the mean value of the metrics. All experiments done on an Nvidia A100 40GB GPUs.
F DATASETS DESCRIPTION AND ADDITIONAL RESULTS
By integrating DeBERTa into the highly successful fusion models of recent years, we establish new baselines for DeBERTa-based models on both the CMU-MOSI and CMU-MOSEI datasets. Our ITHP-DeBERTa model achieves state-of-the-art (SOTA) performance both datasets. In
19

---

Published as a conference paper at ICLR 2024
Algorithm 2 Pseudo-code for training ITHP in general scenario
Input: Dataset D = {(Xo, XI........ Xx), Y}; Coefficient: a, 3, (7H ) k= (Ax )
N-2
N-2, Learning
rate 17, Mini-batch size nmmb Assumed distribution of latent states: : {Bx k=0
N (0,
Output: ITHP parameters (40x (40x)m NN- {th ) Task-related prediction Ø;
Initialize the parameters in the Neural Networks (10x) 1 {thk ) repeat
Sample mini- {(Xp XT.
X Y " Mmm D
for n ......... 7omb do
44. -6 f0g (X6')
2n - Hi + E 20
X1 t- fuu (2i)
for k: = 0, 1. ............NN N 2 do
M Σn 2x11 t f0++1 (2%)
*k+1 - 1k+11 + 8 2k+1
X forns (2
end for end for
if Xo. X1
XN are categorical data then
L KL (N (16. Σg) II (0,
1 Enmi (-Ecss X [C]- log (X: 101)
for k 0, 1,
N 2o
L KL (W (wh+1: 211 (1, +
Ee X [C] log (xt+2 (01)
7k
Rmd
end for else if Xo, X1,
X are continuous data then
(L (mD. 28) | (0, + Amhh Enm XT - X²)) 2
1Ba
for h: 0, ......... N do
1241,0011 KT. (N (1 E 1 (0, 1)) + 1
1 Mim (x 8
end for end if
Loverall
No + Ek=o L
L18++1
96 11:2811
Lrask-related
1-
1Beveraal + a Aunh Exmi (2%+1 yn)²
8+2 0m 76
(0k, v/x) - 2/)) - 0,0 U' Stuck---tteee related. (k: = 0, 1, N 2)
v/h)
until convergence
the of Self-MM,, utilize the raw data and codebase provided in the self-mm repository Ihttps//github.comm/iee com/tuia//SSS/IMMMM MIT License] and modify the feature extraction method to employ DeBERTa, replacing the original requirement for BERT. However, we note that Self-MM itself heavily relies on the feature extraction process performed by BERT, resulting in a significant degradation of DeBERTa-bas based Self-MM, which restricts the applicability of Self-MM in certain scenarios.
F.1 CMU-MOSI
The MOSI dataset comprises 2,199 video clips from YouTube movie reviews, annotated with sentiment intensity -3 to +3 scale. This multimodal dataset integrates transcriptions, visual gestures, and audio-visual features, facilitating comprehensive sentiment analysis research. Its unique structuring allows nuanced sentiment interpretation opinion level Gandhi al. (2022).
20

---

Published as a conference paper at ICLR 2024
Task Metric
BAT
FI1
MAE
Corr
BERT
TFN
Zadeh et al.
2017
74.8
74.1
0.955
0.649
MARN
Zadeh ct al.
2018c
77.7
77.9
0.938
0.691
MFN
Zadeh et al.
201 8a
78.2
78.1
0.911
0.699
RMFN
Liang al.
2018
79.6
78.9
0.878
0.712
LMF Jiu et al. 201
79.1
77.3
0.899
0.701
MulT Tsai al. 2019
81.5
80.6
0,861
0.711
Self-MM; Yuu et al. 2021
84.0
84.4
0.713
0.798
MMIM, (Han 2021b)
84.1
84.0
0.700
0.800
MAG; Rahman et al.
2020
84.2
84.1
0.712
0.796
DeBERTa
MAGa
86.1
86.0
0.690
0.831
ITHP
88.7
88.6
0.643
0.852
Human
85.7
87.5
0.710
0.820
Table 4: Model performance on CMU-MOSI. The table presents the performance of different BERT-based pre-trained models. Models developed by are highlighted in bold, and optimal results are underlined. ITHPa consistently outperforms other models. The metries used include Binary Accuracy (BA), F1 Score (F1), Mcan Absolute Error (MAB), and Pearson Correlation (Corr).
Task Metric
BA
F11
MAE/
Corrt
BERT
Self-MM, Han et al.
202 lb
85.0
85.0
0.529
0.767
MMIM, Han et al. 202
86.0
86.0
0.526
0.772
MAGg
HHa et al.
2021b
84.8
84.7
0.543
0.755
DeBERTa
Self-MMa
65.3
65.4
0.813
0.208
MMIM,
85.2
85.4
0,568
0.799
MAGa
85.8
85,9
0.636
0.800
ITHP
87.3
87.4
0.564
0.813
Table Model performance on CMU-MOSEI dataset. Models developed by us are highlighted in bold, and optimal results are underlined. The metries used include Binary Accuracy (BA), Fl Score (FI), Mean Absolute Error (MAE), and Pcarson Correlation (Corr).
F.2 CMU-MOSEI
The CMU Multimodal Opinion Sentiment and Emotion Intensity (CMU-MOSE)) dataset stands as the most extensive dataset available for multimodal sentiment analysis and emotion recognition. It encompasses over 23,500 sentence utterance videos derived from more than 1000 YouTube speakers, ensuring a diverse and gender-balanced collection. The sentence utterances, sourced from variety of topics and monologue videos, are randomly selected, transcribed, and appropriately punctuated. This dataset represents expanded version of MOSI, which contained 3,228 videos and 23,453 utterances, offering a richer resource for research and development in sentiment analysis and emotion recognition.
21

---

Published as a conference paper at ICLR 2024
F.3 MUSTARD
MUStARD is a dataset constructed to support the exploration of automated sarcasm detection. Comprised of 690 annotated video utferances from various TV shows like Friends and The Big Bang Theory, it offers a diverse range of sarcastic and non-sarcastic instances. The utterances, spanning video, audio, and text modalities, are each associated with speaker identifiers and contextual information.
We benchmarked against another state-of-the-arr (SOTA) Hierarchical Fusion Model (HFM) (Cai et al.] (019b), which employs task-oriented hierarchical fusion approach. This approach integrates early fusion, representation fusion, and modality fusion techniques augment the understanding and representation of individual modalities. To maintain consistency in our comparative framework, we integrated HFM's modality fusion module into our experimental setup. The subsequent tests on dataset yielded results that facilitate direct performance comparison between HFM and our proposed model.
Model
Precision
Recall
F-Score
MSDM
71.9
71.4
71.5
HFM
69.4
68.8
68.9
ITHP
75.3
75.2
75.2
Table 6: Results of sarcasm detection the MUStARD dataset. This table displays the weighted Precision, Recall, and F-Score for the V-T-A (video, text, audio) modality, averaged across five folds. Bold figures represent the highest scores achieved.
F.4 VARYING HYPERPARAMETERS
The sizes of the latent states (Bo and B1) have an impact on the performance of the model. The small size may not capture sufficient information, whereas large size may introduce redundancy and hinder the goal of compactness. To investigate the impact of different sizes, we vary the dimensions of the latent states and evaluate the model's performance. The results are depicted in Fig. 6
In Table 6 we present the comparative results of ITHP, MSDM. and HFM for sarcasm detection the MUStARD dataset. For the HFM model, we adopted the Modality Fusion structure mentioned in its original publication to facilitate multimodal integration. As stated in the HFM paper, the modalities they employed were image, attribute, and text modalities. In our implementation, these were substituted with video, audio, and text modalities, corresponding the MUStARD dataset's specifications. Regarding parameter settings, we retained all original configurations of the HFM model, making adjustments only the input channel numbers of certain layers to align with the input data requirements of the MUStARD dataset.
The results indicate that when the sizes of By and B1 are both to the model exhibits the poorest performance, suggesting that this size may be insufficient for effectively capturing and conveying the information. On the other hand, when B, size is set to 128 and B₁ size set to 64, we observe optimal performance with a weighted precision of 0.754. This finding suggests that latent space dimension of 64 is adequate for representing the multimodal information, while the dimensions of the embedding features of the modalities d, 2048, d, 768, and da 283.
The specific values of the weighted precision and recall shown in Fig. are provided Table 7 and T respectively. Similarly, the specific values of the weighted precision and recall shown in Fig. 6 are presented in Table and Table 10 respectively.
22

---

Published as a conference paper at ICLR 2024
Precision
Recall
0.75
8
16
-0.70
Bo size
32
64
0..5
128
256
16
64 128 256 B1 size
32
8
16
32
64 128 256
0.60
B1
size
Figure 6: Weighted precision and recall for the binary classification task under varying sizes of latent states. The graph shows the impact of varying the sizes latent states (Bo and B1). For each figure, the Red (Orange) color denotes the highest (lowest) value, respectively.
Precision
2
4
8
16
32
64
7
0.619
0.632
0.702
0.735
0.742
0.739
4
0.621
0.655
0.715
0.742
0.749
0.744
8
0.637
0.658
0.734
0.745
0.753
0.746
16
0.693
0.686
0.736
0.746
0.749
0.740
32
0.705
0.714
0.734
0.747
0.746
0.748
64
0.710
0.728
0.729
0.743
0.747
0.741
Table 7: Weighted precision for the binary classification task under varying Lagrange multipliers. The table shows the corresponding values of weighted precision Lagrange multipliers (3 and 7). The optimal result underlined.
Recall
2
4
8
16
32
64
2
0.607
0.627
0.696
0.737
0.742
0.736
4
0.619
0.650
0.706
0.740
0.747
0.742
8
0.622
0.649
0.736
0.742
0.752
0.742
0.681
0.672
0.726
0.745
0.747
0.735
32
0.692
0.698
0.722
0.745
0.741
0.748
64
0.703
0.721
0.723
0.740
0.745
0.738
Table 8: Weighted recall for the binary classification task under varying Lagrange multipliers. The table shows the corresponding values of weighted recall for different combinations of Lagrange multipliers (B and 7). The optimal result underlined.
23

---

Published as a conference paper at ICLR 2024
Precision
B1 size
8
16
32
64
128
256
Bo size
8
0.617
16
0.637
0.654
32
0.659
0.692
0.719
64
0.667
0.714
0.738
0.740
128
0.683
0.721
0.747
0.754
0.728
236
0.714
0.727
0.731
0.734
0.745
0.739
Table 9: Weighted precision for the binary classification task under varying sizes of latent states Bo and B1. The table shows the corresponding values of weighted precision for different combinations Bo and B optimal result underlined.
Recall
B1 size
8
16
32
64
128
256
Bo size
8
0.605
16
0.624
0.631
32
0.643
0.678
0.711
64
0.649
0.701
0.725
0.733
128
0.678
0.714
0.745
0.749
0.725
236
0.711
0.723
0.730
0.732
0.740
0.735
Table 10: Weighted recall for the binary classification task under varying sizes of latent states Bo and B1. The table shows the corresponding values of weighted recall for different combinations of Bo and By. The optimal result is underlined.
F.5 VARYING ORDERS OF MODALITIES
Our research includes varying the order of modalities to assess the impact on our model's performance. The detailed results are catalogued in Table [I] which shows the effects of different modality sequences on weighted precision, recall, and F-score.
The results reveals that the modality sequence significantly affects performance metries. Specifically, positioning video the primary modality (V-T-A) yielded the highest scores, with all metrics around 75%. In contrast, sequences starting with audio (A-T-+V) registered the lowest, with scores near 71%. This evidence underscores the importance of strategic modality ordering, which can be guided by domain expertise to enhance model efficiency for specific tasks.
Order of Modalities
Metrics
V-T-A
V-A-T
T+V=A
T-A-V
A+V-T
A-T-V
Precision
75.3
73.7
73.0
72.0
71.4
70.8
Recall
75.2
73.5
72.8
71.8
71.2
70.6
F-Score
75.2
73.4
72.9
71.7
71.5
70.6
Table 11: Varying Orders of Modalities. The table showcases the impact of changing the order of modalities the weighted Precision, Recall, and F-Score across both sarcastic and non-sarcastic classes, averaged across folds. Underlined values highlight best results for each metric. modalities are denoted as: T' text. for audio, and for video.
24

---

Published as a conference paper at ICLR 2024
G ASSESSMENT OF INFERENCE LATENCY
Our study included comparative analysis of inference latency, utilizing a simplified multi-layer perceptron (MLP) as benchmark. The MLP head, consisting of 64 neurons, served as the predictor within our ITHP model framework. In this setup, we concatenated modalities M0, MI, and M2 and subjected them to the MLP head. The resulting inference latency was approximately 0.00221 milliscondss (ms) per sample. By comparison, the complete ITHP model, which features a more complex hierarchical structure, exhibited an inference latency of 0.0120 ms per sample.
is important note that the detectors within our ITHP model remain inactive during the inference phase, which contributes to the model's efficiency. Despite the inherent latency increment attributable to the hierarchical design, the ITHP model demonstrates commendable performance efficiency. The observed slight increase in latency is testament to the model's capability to balance the demands of complex multimodal learning tasks with the necessity for swift processing times.
H LEARNING COMPLEMENTARY INFORMATION FROM SECONDARY MODALITIES
Here, we provide additional elaboration on the potential limitations that may arise when the primary modality lacks sufficient information. It's essential to highlight that our selection criteria for determin- ing the primary modality have been designed to encapsulate the most pertinent information. However, it's equally vital to acknowledge that depending solely on the primary modality might not always suffice, necessitating the inclusion of information from secondary modalities. By harnessing the ca- pabilities of our deep learning framework, model has a certain ability to learn the complementary information from auxiliary modalities. To achieve a harmonious representation, we 've incorporated specific terms in the loss function. Specifically, taking the 3-modalities learning problem in Eqn. (3) as an example, term L, - 31 Bo; ()) is tasked with supervising the knowledge transfer from XI Bo, while term L2 = - (B X2) responsible for supervising the knowledge dissemination from X2 to B1. Through this framework, the ITHP model is primed to discern and incorporate complementary information from non-input modalities. Our model, therefore, endeavors to strike a balance between mutual information. irrelevant data, and task-essential information, ensuring an optimized flow of information.
I METRICS FOR EVALUATION
We employ a variety of metrics for evaluating the performance models in binary classification: weighted precision for different tasks. These metries include weighted precision, weighted recall, weighted F score, Binary Accuracy (BA), F1-Score (F1), Mean Absolute Error (MAE), and Pearson Correlation (Corr). We consider the following notations: TP (True Positive); TN (True Negative); FP (False Positive); and FN (False Negative).
1.1 SARCASM DETECTION
To ensure fair comparison with the model performance outlined in (Castro et al. 2019). we have utilized the same evaluation metrics: weighted precision, weighted recall, and weighted F-score, following the guidelines provided by (Castro et al.) 2019). The weighted average precision, recall, and F-score are determined by incorporating the support (number of true instances for each label) as weights. The support for class is denoted as S, Here are the calculations for these metrics:
Weighted Precision P.: measures the proportion of correct positive predictions, adjusting for class prevalence. The formula is given as:
P. En (S. precision,)
(22)
>m S;
where precision,
TPP (TP:+FP.)
25

---

Sample Entropy (SampEn) is a method traditionally used for time-series data to quantify their complexity by evaluating the predictability of patterns within the data. When adapting SampEn for Published as a conference paper at ICLR 2024
Weighted Recall Rw: This metric estimates the model'ss ability to identify all positive instances, accounting for the frequency of each class. computed as:
Di= (S, * recall;)
Rv
(23)
Limi St
where recall, (TP:+FN:)"
Weighted F-score Fw:: This balances precision and recall, weighting based on the true instances for each label. The formula for this metric is:
Fw 2i=1 (S, $ Fj)
(24)
D S:
where F, 2- Alppecision,erecally (preciston; trooall;))
In these formulas, TP; represents the true positives for class i, FP; represents the false positives for class i, FN; represents the false negatives for class i, and S; represents the number of true instances for each label (class i).
1.2 SENTIMENT ANALYSIS
We adopt the well-acknowledged metrics in sentiment analysis, specifically, Binary Accuracy (BA), F1-Score (FI), Mean Absolute Error (MAE), and Pearson Correlation (Corr), to evaluate our model's performance and contrast it with the benchmark models on CMU-MOSI and CMU-MOSEI datasets. The formulas for these metrics are as follows:
Binary Accuracy (BA): It's straightforward measure calculated as the number of correct predictions divided by the total number of predictions. Here's 's the formula:
TP + TN
BA
(25)
TP + TN FP FN F1-Score (FI): The Score a metric that seeks to strike balance between Precision and Recall. Here's the formula:
(Precision * Recall)
=1= 2
(26)
(Precision + Recall)
where Precision
TP TP4FF and Recall TP4FN
TP
Mean Absolute Error (MAE): The MAE quantifies the average magnitude of errors in a set of
predictions, disregarding their direction. Here's 's the formula:
- 1ú V4
MAE(Ú. v)
(27)
n
where ys the true label, is is the predictive value and the total number of predictions.
Pearson Correlation (Corr): Pearson Correlation (Corr) assesses the linear association between predicted probabilities and true labels. Here's the formula:
Corr(z, (ar y) = Σ (x:
2)
(1t 9)
(28)
Et (2: Er
(yi
4)²
where refers to predicted probabilities for the positive class, and yi is the true label. The ä and the means of the predicted probabilities and actual labels, respectively.
J ALGORITHMS FOR RANKING MODALITIES
J.1 ADAPTING SAMPLE ENTROPY (SAMPEN) FOR DATASETS
26

---

Published as a conference paper at ICLR 2024
feature datasets, especially those resembling time-series data in structure, the same principles are used to analyze the distribution and regularity of features within individual samples in multidimensional feature space.
For a dataset represented as a set of feature vectors, where each vector can be considered as sequence, the information richness can be quantified by measuring the regularity and unpredictability of these sequences. SampEn in this context measures the likelihood that subsequences of features within given vector that are close (within certain tolerance) remain close when one additional dimension considered. This approach evaluates the diversity and distribution of the patterns within each individual sample, reflecting the intrinsic information richness of the feature data.
The SampEn of a dataset is calculated as follows:
1. Choose an embedding dimension 77 2 and tolerance r standard deviation of the data.
0.2 X
2. For each sample in the dataset, form vectors U. for 1,..., D - m where D is the number of features.
3. Define (r) as the count of times when vectors U and U:41 within the same sample are within of each other.
4. Define A (r) similarly for m 1.
5. SampEn is given by:
SampEn(m, r, -- in (r)(r)
Bim
Algorithm 3 Ranking Modalities Based on SampEn
function SAMPLEENTROPY
m, N <- number of samples in data
8mh +0
A +
for each sample in data do
for i <- 1 to number of features do if norm(sampled 8Th th + i < number of features m - and norm (samplllel|i i+ m+ + 1] sample[i 1 then
+ m] sample m + 1]) < then
m 2))
A + Am +1
end if end if end for end for return "oog(A"//""" end function
> SampEn value
function RANEMODALTTIESSmoallttt m,r)
M Modaitty List + [] for each modality in modalities do
entropy + SAMPLEENTROPY (moddaltt, m.r)
Append (modality, entropy) to LodalttyList end for
Sort ModalityTist based on entropy in descending order
return ModalityTise
& List of modalities ranked by entropy
end function
27

---

Published as a conference paper at ICLR 2024
J.2 RANKING MODALITIES USING SUBMODULAR FUNCTIONS
Given a set of N different modalities M {m1, m2,.... my) }. the goal to find subset of modalities S S M that maximizes the predictive performance in given downstream task.
Submodular Function Definition: Let f 2 - I& submodular function that maps set of modalities to real number indicating their performance in predictive task. For any A C B C M and æ E B, the function f satisfies:
f(AU (2)) f(4) 2 f(BU(=)) - f(B)
Greedy Algorithm:
The greedy algorithm starts with an empty set and iteratively adds modality that maximizes the function f(S):
S St U (arg max f(S, (x)))
ztS;
The greedy algorithm provides an approximation guarantee of (1 - !) for maximizing submodular function, where is the base of the natural logarithm.
Performance Guarantee:
Assuming function 'evaluatePer Per that assesses the performance of given set of modalities.
Algorithm 4 Greedy Algorithm for Ranking Modalities
function DDEEDDSSAMOOULLASSRtttnnddtdtt horp:ttteeeeeeeeuuuateoeoorraam
S empty set Modality +- emppy list while [S| < |modalities] do best.mmproeement +- -00 best.M odality f- null
for each modality in modalities do
new Per formance improvement + newPer forrformmcce evaluate Per formance if improvemend best then
Pewpluaateeefffrromcccecce (modality))
bestI messlmppruemmnt +- improvement
odality M end if end for if best is null then break end if t SU odality)
best
modality
M ModalityList.appene (bstMooall
end while
return Modality List
end function
28

---

Published as a conference paper at ICLR 2024
K
DEFAULT NOTATION
Xo
The input modal state with the highest information richness
{X+]R=1
The other modal states for information fusion Latent states constructed for ITHP Task-related labels values
[Bx) KO
Y
dxa/ /d
The dimensions for modal states/ latent states
(-)
Deterministic distribution of random variables
q (-))
Random probability distribution fitted by the neural network
N
Total number modal states
B
Lagrange multiplier for the first level IB
7
Lagrange multipliers for the subsequent level IBs
Coefficients for integrating the first and sebsequent level IBs
Coefficient for balancing the ITHP loss and the task-related loss
Ok
Parameters for the encoding-level Neural Networks
Parameters for the predicting-level Neural Networks
pk
Mean value learned by the encoding neural networks
Eg
Covariance value learned by the encoding neural networks
c
Assumed distribution of latent states
Tomb
Mini-batch size for training the ITHP
Learning rate for training the ITHP
29


Reviews:
[{"review_id": "uzOrvfKXMd", "reviewer": "Reviewer_mWBz", "summary": "This paper proposes ITHP, a brain-inspired hierarchical information perception model that fuses information from different modalities. ITHP employs the information bottleneck framework to extract relevant information from different modalities in a hierarchical/sequential manner. Extensive experiments on MUStARD, CMU-MOSI, and CMU-MOSEI demonstrate the effectiveness of ITHP.", "questions": "- how do you choose the primary modality? And how would different choices of primary modality affect the performance of ITHP?\n- how would ITHP perform in the presence of noise in some of the modalities (the primary or the secondary ones)?\n- how can ITHP accommodate tasks that require combining complementary information from different modalities?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- the paper is clear and well-organized.\n- the proposed ITHP is intuitive and well-grounded in information theory and provides a novel view of information fusing in multi-modal learning.\n- extensive experiments demonstrate the effectiveness of ITHP.", "weaknesses": "- ITHP extracts commonly encoded information from different modalities; however, would this procedure be called \"fusion\" appropriately?\n    - suppose the primary modality is $M_p$ and a secondary modality be $M_1$. ITHP cannot capture information encoded in $M_1$ but not $M_p$. In this sense, ITHP would be an information distillation rather than a fusion method.\n    - when talking about \"fusion,\" one expects to integrate information of **different** types.\n    - in addition, when both modalities contain similar information, if the one in $M_1$ is noisy, would ITHP be affected? Can ITHP correctly identify the noise in $M_1$ and mainly use the information from $M_p$ instead?\n    - as the main idea of ITHP is to distill relevant information from different modalities, the starting point would be important. However, the experiments do not show results with different primary modalities.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XjWuFPM9RZ", "reviewer": "Reviewer_Qqaa", "summary": "- This paper introduces the Information-theoretic Hierarchical Perception (ITHP) model, which aims to integrate and process information from multiple modalities effectively. The authors show inspiring motivation corresponding neuroscience research to a model that designates a prime modality and utilizes the concept of the information bottleneck to construct compact and informative latent states, which enable to balance between preserving relevant information and reducing noise in the latent states.\n\n- This paper tries to justify the validity of design philosophy and the effectiveness of the proposed models experimentally with the performance comparison of two tasks such as (1) sarcasm detection from videos and (2) multimodal sentiment analysis. They seem to show better scores in both tasks. (note that seems not to pursue state-of-the-art performances).", "questions": "See the Weaknesses above and answer, please.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- This paper presents interesting connections between neuroscience and proposed models. It provides good motivations for designing the proposed models.﻿ This integration of neuroscience principles into the design of the models adds a novel perspective to the field of multimodal learning.\n\n- The authors demonstrate the effectiveness with competitive performances of the proposed models in the two tasks.", "weaknesses": "- Even though this paper shows interesting modeling design and experimental results, there is a gap between state-of-the-art methods such as UniMSE and SPECTRA, for example, with respect to performances. What kinds of criteria to choose the model configuration and comparative methods in the paper?\n\n- This paper seems not to present enough information on experimental setting and implementation to achieve reproducibility.\n\n- It lacks justification for the component choices. Which points should be clarified through experiments? Why BERT and DeBERTa are utilized as language models? \n\n\n* References\n\n  - UniMSE : G. Hu, et al., UniMSE: Towards Unified Multimodal Sentiment Analysis and Emotion Recognition, EMNLP 2022\n\n  - SPECTRA : T. Yu, et al., Speech-Text Dialog Pre-training for Spoken Dialog Understanding with Explicit Cross-Modal Alignment, ACL 2023\n\n\n----------------------------------------------------------------------------\nPost-rebuttal\n----------------------------------------------------------------------------\nI read the rebuttal and other reviewers' opinions. \nAt first, the rebuttal addresses most of my major concerns and provides convincing information to resolve mine.\nI'd like to raise my score from 5 to 6.\nThank you for your good contributions to our community!", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cGDdoifyTz", "reviewer": "Reviewer_uxTC", "summary": "The authors develop a new Information-Theoretic Hierarchical Perception (ITHP) model that designates a prime modality and regards the remaining modalities as detectors in the information pathway, serving to distill the flow of information. The primary modality yields the highest degree of information extraction, with subsequent modalities contributing information in a sequentially ordered manner.  To address the challenge of high dimensionality of multimodal data, they construct \"Information bottlenecks.\" These bottlenecks function as compressed latent representations of the data, with each bottleneck responsible for compressing a single modality while retaining  the relevant information of other modalities. \n\nThe method shows impressive performance on the CMU-MOSI dataset surpassing human-level performance for sentiment analysis.", "questions": "The authors have established a direct quantitative association between the embedding size of modality X and its corresponding priority order for the task. However, the embedding size can be contingent upon the model's architectural characteristics. Does changing the model which provides the embedding has any effect on the priority order?  \n\nHow is embedding size related to the amount of context information present in a modality?  \n\nFor experiments in table 1, is there any effect of changing the audio and text priority, when all three of the modalities are present?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The fundamental premise of the research paper is interesting. It employs the concept of hierarchical information flow in a multi-modal context which was shown to be useful for downstream tasks like sentiment analysis and sarcasm detection.  \n\nThe paper is well written and easy to understand.", "weaknesses": "Comparison with other methods in Table 1 for sarcasm detection is insufficient. The authors need to provide comparison results on either additional datasets [3] or other existing methods in literature [4].  \n \n[3] Cai Y, Cai H, Wan X. Multi-modal sarcasm detection in twitter with hierarchical fusion model. InProceedings of the 57th annual meeting of the association for computational linguistics 2019 Jul (pp. 2506-2515). \n \n[4] Wen C, Jia G, Yang J. DIP: Dual Incongruity Perceiving Network for Sarcasm Detection. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition 2023 (pp. 2540-2550). \n\nThe paper is specific to the task sentiment analysis whereas the title and presentation of the paper refers to downstream tasks in multimodal learning as a whole. To support the claim the authors should report the method’s performance on some general tasks like visual question answering.  \n\nThe literature survey needs to be updated. They are several recent methods of multimodal fusion and representation learning: \n \n[1] Xue Z, Marculescu R. Dynamic multimodal fusion. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition 2023 (pp. 2574-2583). \n\n \n\n \n\nMissing table 5 and 10 on page 9 which were referred to in text.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "knTYo4aLf9", "reviewer": "Reviewer_4bfy", "summary": "The paper proposes a neuro-inspired sequential and hierarchical processing model for multi-modality data. The key is to balance the minimization of mutual information between the latent state and the input modal state, and the maximization of mutual information between the latent states and the remaining modal states. The paper presents the theory formulation of the processing steps and evaluate on two multimodality tasks, sarcasm detection and sentiment analysis.", "questions": "1.\tWhy sequential processing is good? Except for inspiration from neural inspirations. This operation introduces the extra problem of the order of modality. \n2.\tDoes the best beta and gamma share across different tasks? Or it is needed to balance for different tasks, making this method quite cumbersome.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1.\tThe paper provided detailed formulation of the perception process and deduction of the loss function.\n2.\tThe paper conducted detailed experiments on the ablation of hyperparameters.", "weaknesses": "1.\tThe inspiration from neuroscience research seems simple, and unnecessary to the main processing pipeline of the main work, making the main work distracting.\n2.\tThe order of the modality in processing the data seems quite important, which highly impacts the performance. For such an importance factor, it is better to provide further theoretical analysis or practical guidance on it.\n3.\tThe paper should include some recent SOTA works, at lease works in 2022, on comparing the sentiment analysis.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yXpNkavRWV", "reviewer": "Reviewer_JgMF", "summary": "This paper proposes an Information-Theoretic Hierarchical Perception (ITHP) model, based on the concept of information bottleneck, for effective multimodal fusion.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ The proposed Information-Theoretic Hierarchical Perception model looks novel to me; This stands in contrast to mainstream multimodal fusion where different modalities are treated equally, and the final prediction is usually acquired by concatenating the features from each modality. The concept of designating a prime modality and using the other modalities to guide the flow of information looks interesting, and makes sense to me --- In many multimodal tasks, modalities do not contribute equally to the final prediction.\n\n+ The experimental results are strong. The analysis about varying Lagrange multiplier in Sec. 3.1 provides good insight.\n\n+ Many design choices (e.g. latent state size, order of modalities) are thoroughly evaluated in the appendix.", "weaknesses": "+ As the authors already point out in Section 5, the proposed approach requires a predefined order of modalities, which could be a problem when there are > 3 modalities and we do not have prior information about the multimodal task.\n\n+ Would the proposed hierarchical architecture introduce some inference latency compared with the standard non-hierarchical approaches (Take Figure 3 as an example, say compared to concatenation of $X_0$, $X_1$ and $X_2$ and then passed to an MLP head)?\n\n+ It seems in the current framework, feature extraction from raw video, audio & text & the proposed ITHP are separate (first feature extraction then use the encoded features in the hierarchical information flow). The temporal nature of these data is not utilized. I wonder if the authors could briefly comment on the potential of extending the ITHP model to handle sequence data (that naturally comes with a temporal order).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)