Rethinking Missing Modality Learning: From a Decoding View
Authors: Tao Jin
Zhou Zhao
Venue: Submitted to ICLR 2023 (2023)
Keywords: multimodal
decoding
tensor decomposition

Abstract: Conventional pipeline of multimodal learning consists of three stages, including encoding, fusion, and decoding. Most existing methods under missing modality condition focus on the first stage and aim to learn the modality invariant representation or reconstruct missing features. However, these methods rely on strong assumptions (i.e., all the pre-defined modalities are available for each input sample during training and the number of modalities is fixed). To solve this problem, we propose a simple yet effective method called Interaction Augmented Prototype Decomposition (IPD) for a more general setting, where the number of modalities is arbitrary and there are various incomplete modality conditions happening in both training and inference phases, even there are unseen testing conditions. Different from the previous methods, we improve the decoding stage. Concretely, IPD jointly learns the common and modality-specific task prototypes. Considering that the number of missing modality conditions scales exponentially with the number of modalities ${\bf O}({\text 2^n})$ and different conditions may have implicit interaction, the low-rank partial prototype decomposition with enough theoretical analysis is employed for modality-specific components to reduce the complexity. The decomposition also can promote unseen generalization with the modality factors of existing conditions. To simulate the low-rank setup, we further constrain the explicit interaction of specific modality conditions by employing disentangled contrastive constraints. Extensive results on the newly-created benchmarks of multiple tasks illustrate the effectiveness of our proposed model. 

Under review as a conference paper at ICLR 2023
RETHINKING MISSING MODALITY LEARNING: FROM A DECODING VIEW
Anonymous authors Paper under double-bling review
ABSTRACT
Conventional pipeline of multimodal learning consists of three stages, including encoding, fusion, and decoding. Most existing methods under missing modality condition focus on the first stage and aim to learn the modality invariant repre- sentation or reconstruct missing features. However, these methods rely on strong assumptions (i.e., all the pre-defined modalities are available for each input sample during training and the number of modalities is fixed). To solve this problem, we propose a simple yet effective method called Interaction Augmented Prototype Decomposition (IPD) for a more general setting, where the number of modalities is arbitrary and there various incomplete modality conditions happening in both training and inference phases, even there are unseen testing conditions. Different from the previous methods, we improve the decoding stage. Concretely, IPD jointly learns the common and modality-speciff task prototypes. Considering that the number of missing modality conditions scales exponentially with the number of modalities ((2") and different conditions may have implicit interaction, the low- rank partial prototype decomposition with enough theoretical analysis employed for modality-specin
components to reduce the complexity. The decomposition also can promote unseen generalization with the modality factors of existing condi- tions. To simulate the low-rank setup, we further constrain the explicit interaction specific modality conditions by employing disentangled contrastive constraints. Extensive results on the newly-created benchmarks of multiple tasks illustrate the effectiveness of our proposed model.
INTRODUCTION
Multimodal learning is recently one of the increasingly popular yet challenging tasks involved in both computer vision and natural language processing Ben- Younes et al. (201)); Do et al. (2019); Gabeur et al. (2000): Liu et al. (2018). The target of multimodal learning is to utilize complementary information contained in multimodal data for improving the performance of various tasks. Many superior approaches on multimodal learning have been well developed in an ideal situation. However, common assumption underlying these approaches is the completeness of modality (i.e., the full modalities are available in both training and testing data). In practice, such an assumption may not always hold in real world due to some overwhelming reasons. For example, some uploaded YouTube videos do not have audio tracks, also, the black screen may occur during the broadcast, leading to the lack of visual modality.
Although a bunch of endeavors are devoted to developing effective methods to cope with the missing modality conditions in the training and inference stages, there is no unified paradigm that takes into account all possible scenarios. For instance, Phha et al. 2019) Zhao et al. (2021) only consider the incompleteness of testing data, however, obtaining lot complete data for training extremely iabor-intensive. Ma et al. (202I) formulates new setting that considers the incompleteness of training data, while there are only two modalities used. As we know, due to the quick developments of feature-extraatton extraction skills, one data sample may have more than two kinds of modality representation.
In this paper, we focus on a more general setting (as showin in Fig. where the number of modalities is arbitrary and there various incomplete modality conditions, systematically study the missing modality problem. We propose a simple yet effective method called Interaction Augmented Prototype Decomposition (IPD) to capture the universality and particularity of different modality conditions

---

Under review as a conference paper at ICLR 2023
(ineluding both complete and incomplete conditions). To the best of our knowledge, we try to study generalizable missing modality learning from the decoding perspective for the first time. Following Zhou et al. (2021 2022), treat the weights of the standard linear classifier task prototypes. Since different modality conditions correspond to different task prototypes, training multiple models separately for all the conditions is time-consuming. IPD jointly learns the common and mosality-specific prototypes. Considering that the number of missing modality conditions scales exponentially with the number (n) of modalities O(2") and different conditions may have implicit interaction, the low-rank partial prototype decomposition with enough theoretical analysis is employed for modality-spor -ppeciii component to reduce the complexity. To simulate the low-rank setup, we further constrain the explicit interaction of specific modality conditions by employing disentangled contrastive constraints. We conduct extensive experiments on the newly-created benchmarks of multiple tasks, the experimental results show that IPD could achieve competitive results compared with the state-of-ther thrr methods of conventional multimodal learning. The main contributions can be summarized as follows:
We propose a novel method called Interaction Augmented Prototype Decomposition (IPD) for generalizable missing modality learning, which jointly learns the common and modality specific prototypes. To the best of our knowledge, it is the first time to improve the decoding stage.
Considering that the number of missing modality conditions scales exponentially with the number (n) of modalities O(2") and different conditions may have implicit interaction, the low-rank partial prototype decomposition with is employed to reduce the complexity. The decomposition also can promote unseen generalization with the modality factors of existing conditions.
We constrain the explicit interaction of specific modality conditions by employing disentangled contrastive constraints.
We conduct low-rank ensemble learning to enhance the performances the conditions with relatively few modalities.
RELATED WORK
2.1 CONVENTIONAL MULTIMODAL LEARNING
Multimodal learning utilizes complementary infor- mation contained in multimodal data to improve the performance of various tasks. The key point of this area is multimodal fusion. Early fusion methods are mainstream and integrate features of different modal- ities before feeding them to the task modules. For example, concatenating different features Zadeh et al.) (2016b) is simple way. Zadch al. (2017) proposes a product operation to allow more interaction among different modalities during the fusion process. Liu et al. (0188) 8 considers the large complexity Zadeh (2017) and utilizes modality-specift factors to achieve efficient low-rank fusion. However, the intra-modal dynamics cannot be effectively captured in the above methods. Liang et al.) (2019) employs low-rank fusion for each time step of multi-view se- quential input. T sai et al. (2019) utilizes Transformer to replace LSTM due to the powerful encoding ca-
M1
M2
Ms
Training
Testing
Figure 1: An example of generalizable miss- ing modality learning, where M1,M, Ms denote three modalities. There are unseen modality combinations in the testing stage.
pacity. Rahman et al. 2020) employs large-scale pre-trained Bert embeddings for textual modeling. Liang et al. (2021) proposes moodaltt-invariian crossmodal attention towards learning crossmodal interactions over innaaiit-invariant space in which the distribution mismatch between different modalities is well bridged. However, these methods do not consider the missing modality problems at all. For this aim, propose IPD solve the problems.
2

---

Under review as a conference paper at ICLR 2023
Visuel
Orth.
Eq.
Cons.
Etn
Encoder
Text
Fusion
Ceau
Decoding
3:1
Audia
100
The framework IPD, where the left part denotes the low-rank partial decomposition and the right part denotes the disentangled contrastive constraints. Different color corresponds to different modalities (red denotes visual modality, blue denotes text modality, green denotes audio modality). aO" "I" denote the states of existence and nonexistence. the left part, the process of Eq. D to Eq. i included. "Orth." denotes orthogonal constraints (Eq. 4 Yellow bars denote A, prototypes.
Figure 2:
that consists of as,. Gray bars of eg and es2 denote the common and prodality-speciien
ex can be calculated with A, and 9 through Eqs. and The decoding process (Eq. [ is developed by correlating x and et. In the right part, we mainly present Eq. of contrastive constraints. "Cons"
denotes contrastive learning (Eq. The gray bars of eta: Ctavs En denote the modality specific prototypes of ta, tav,
2.2 MULTIMODAL LEARNING WITH INCOMPLETENESS
Multimodal learning with incompleteness is another topic of concern. There are also two lines of incomplete multimodal learning. One to process the representation noise, for example, the visual features of several time steps are not available, while all the remaining time steps are normal. Liang et al. (2019) proposes rank constraints to solve this problem. The other to process the missing modality conditions that more likely to be encountered in real-world applications. Also, there exist more researches that study it. Parthasarathy
Sundaram (2020) proposes a strategy to randomly discard the visual input during training the clip frame level mimic real-world missing modality scenarios for audio-visual multimodal emotion recognition, Tran et al. (2017); Zhho al. (021)) proposes Cascaded Residual Autoencoder (CRA) utilize the residual mechanism over the autoencoder structure, which can take the corrupted data and estimate function to well restore the incomplete data. |hha et al. (2099) proposes sequential translation-basel model learn the joint representation between the source modality and multiple target modalities. Although many efforts devoted developing effective methods cope with the missing modality conditions during training and inference, there is no unified paradigm considering all possible scenarios. Thus, we focus on more general setting. Also, inspired by the usage of low-rank decomposition in multi-view interaction Liu et al. (2018): Do al. (2019); Piratla et al. (2020), we introduce it into the task.
3 APPROACH
3.1 PROBLEM FORMULATION
We formulate the problem setting for generalizable missing modality learning in this section. Suppose the number of modalities is n, there various modality scenarios in both training and inference stages. Concretely, about 2n - scenarios are available, since each modality has two states (i.e. O-nonexistence, 1-existence). Under such real-world conditions, there are challenging problems, one is to obtain good performances of existing modality combinations. The other is to generalize to unseen conditions, when the training does not cover all the modality scenarios.
3.2
COMMON-SPECIHER JOINT LEARNING
Following the previous work, we employ different encoding blocks for corresponding tasks. For multimodal sentiment analysis and speaker traits recognition, we adopt the same structure as LMF Liu et al. (2018), where LSTMs are utilized to process the multimodal features. As for multimodal
For convenience, utilize 2" in the following sections.
3

---

Under review as a conference paper at ICLR 2023
video retrieval, borrow the idea from MMT Gabeur et al. (2020), where the multimodal features are encoded by Transformer. To fuse the multimodal features, we use the mainstream tensor- based method, LMF, which considers the fine- tne-grained interaction among different modalities. The training paradigms of multimodal sentiment analysis and speaker traits recognition are similar to the settings of LMF As for the training objective of video retrieval, employ the contrastive cross-entropy loss and transform it to a classification task (i.e., the matched pairs are positive samples)
We mainly focus on classification and regression. Similar Zhou al. (2021). Zhou al. (2022), we treat the linear weights of the final layer as task prototypes of different modality scenarios. There 2" scenarios and the examples (z, y) E Rm denotes the fusion result of multiple modalities, E R denotes the label or regression score) from specific scenario E [2"] are generated as;
y x et,
el
Ee + Exp
where e, RT denotes the common prototypes of different modality combinations, 6s 1 e, R denotes the modality-specific prototypes of [-th modality scenarios, et denotes the complete task prototype for (-th modality combination. Intuitively, the common component contains more high-level semantic information (i.c., the reasoning of performance), and the modality-speciif part contains more low-level detailed information (i.e., the loudness of voice, the movement range). y = + for classification tasks and [-p.p] p] for regression tasks.
3.3 LOW-RANK PARTIAL PROTOTYPE DECOMPOSTION
Based on Eq. there exist superior scenario specific classifiers {eill E [2"]]. one for each modality scenario, such that e = Ec + ca.. Note that all these classifiers share the common component 6 If we are able to find multiple specific classifiers of this form, e, and es, could be extracted from them. Although the prototypes cover all the modality combinations, the complexity (O(2")) would be large when n increases. Further, training the modality-specifet prototypes separately is inefficient with neglecting the interaction among different modality scenarios. Therefore, we reformulate the E, Rmx2" that consists of es; A, B where A, E mRm* / R22 and & denotes the low-rank value. Our idea can be extended from Eq. [
k == x Teo + Σ j=1
B1,jae,) = x Tec A.DT)
where O2, 1 e. E IR",, and as 1 az, for j.q E [k),, # q are modality specific features. The correlation between the as, and task is given by the coefficients 3r E R², varies among multiple modality combinations. Under this setting, there also exists superior modality specific classifier et such that = e. + A,3, E R", where e R" Ag E Rmnk,, 3r, E R are trainable variables. More concretely, the trainable task prototypes can be written as:
E E= + A = [e,,, A. 1,
(3)
where E [e1, 62, ... 62n]
Rmx2", 1 R denotes the all ones vector and B However, in practice, given a general matrix E which be writ-
[81, B2 B2.... B2n] kx2 ten as E = e + A there are multiple ways of decomposing E into this form, ez and Ag33T cannot be uniquely determined by the decomposition alone. Therefore, we conduct constraints the [ec, A,] with orthogonal regularization:
C, [ - [ec, [e]
(4)
where L. denotes the regularization loss. Is41 E ((++1)x(++11) denotes identity matrix. Such regularization term caters the following propositions:
Proposition 1: When e, 1 Span(A.). ).). the decomposition of = e,1 + A has unique solution. Suppose E e.l A, w.1 +W,TT is a rank-(k: matrix, where A, E R
4

---

Under review as a conference paper at ICLR 2023
E 2 We E Rmxk and I 2 all rank-k matrices with & < min(m,2") When 6c (pan(As), We = ee is equal to we 1 Span(Wa).
Proposition 2: If the orthogonal regularization is not satisfied (e. is not orthogonal to Span(A,) the performance of partial modality combinations will be influenced.
3.4 HIERARCHICAL DECOMPOSITION OF A, AND UNSEEN GENERALIZATION
Although the decomposition of A, results in the change of complexity (from m 2n to k 2²), the exponential term 2 still exits. Furthermore, the decomposition of A,9 does not consider the fine-grained interaction among different modality combinations. To solve this problem, we employ the hierarchical decomposition for the high-order tensor 3 E xk Concretely, we divide 3 into series of n-order tensors:
B = [C C² CO,..., C*k
(5)
where C E IR",, we apply CP decomposition Harshman et al. 1970) to each C+;
C#
C Grt C² R
(6)
where @ denotes tensor outer product operation over a set of vectors, R denotes the rank value CP decomposition. Such operations change the complexity of / from exponential to linear. With the approximation, the weights for unseen modality combinations can also been obtained with tensor multiplication. We employ these weights for the evaluation of missing modality generalization.
3.5 DISENTANGLED CONTRASTIVE CONSTRAINTS FOR DECOMPOSITION
The rank value is related to the interaction among different modality combinations to some extent. To further enhance the explicit interaction of specific modality combinations and control the rank value, we employ disentangled contrastive constraints for decomposition. Specifically, we suggest employing the task prototype (i.e. 6x;) of the modality-specific part. Different from the obscure modality gaps that cannot be obtained, the interaction between different modality combinations obvious. For example, suppose that there are totally three modalities (t, a, e) and the possible combinations include {t, ta, tv, av, tav). We can easily find that ta is more related to and tov than e, since ta and v do not have intersection. We express the above statements as:
D(ta, n) 2 D(ta, v), E {t, a, tav)
where D() denotes the correlation function. Following the laws of nature, we conduct disentangled contrastive constraints the modality-specine part. The naturally similar pairs like la and lav should have higher matching scores than negative pairs like to and v.
Le max (0 10.A-h(e
Ctav) A hhCeas h((eta, C..)
(8)
where Cza: Ctavs C. denote the modality prototypes, A is the margin value, h() denotes the normalized inner product operation (cosine similarity). In practice, the number of existing positive and negative pairs also scales exponentially with the number of modalities, therefore, we utilize the maximum suppression sampling scheme to reduce it. Concretely, for a specific input (i.c. eta). we consider the combination (i.e. etae) with most modalities that contain the processed modalities as positive object, the complementary set (i.e. e,)) is treated negative object.
3.6 LOW-RANK ENSEMBLE LEARNING
With the approximation of 3, we employ low-rank ensemble learning as an auxiliary, since the samples with modalities could enhance the training of more modality factors produced by Eq.
5

---

Under review as a conference paper at ICLR 2023
(e.g. sample with modalities ta can be used for training Etas Ca simultaneously). Concretely, data point with n² modalities can be augmented to 2" modality combinations. We calculate the average results of all the augmented modality scenarios an imput sample. Suppose that the encoded features represented as U1, u.,...u E _ where u, denotes the feature of i-th modality. If modality is missing, the original input would replaced with all zeros vector, the encoded features represented 01, 02, R Thus, if the i-th modality is missing, u, = Oz- We concatenate each W and corresponding o to v = [W?u,, Wo'p] Rm*x where W E "R" "x** denotes the linear mapping weights following LMF. The fusion results for all the modality combinations can be denoted as V E Imxx"", where Vi1.ta...... = II=== Vi,t; E Rmm The result O, O, + O, consisting of common O₂ and specific O, parts can be calculated follows:
O₂ sum (III
O
cc /M
R
0, = sum (II
C
U
o A,) /M
where @ denotes element-wise multiplication, sum() denotes summation operation for all the elements of the vector, Cir- 2 .M denotes the ensembled number (i.e. 2" which is related to n²" The detailed derivation is shown the appendix (section A
TRAINING AND INFERENCE
The overall framework is shown Fig. 2 We employ C, f y) denote the prediction loss, where y" denotes the prediction result with LMF and is the ground-truth label, f() can be MAE for sentiment analysis or cross-entropy loss function for retrieval. L. = f(O y) denotes the loss when using ensemble learning. The final optimization objective is L. + Ae + XeCo Azfe.
During inference, we employ E = (2,, ez-l for both existing modality combinations and unseen ones. The unseen task prototypes can be calculated by the trainable modality factors as introduced in Eq. (6. We provide the training and inference details in the appendix (Algs. [and 2
EXPERIMENTS
5.1 DATASET AND METRICS
We evaluate method three challenging tasks, multimodal sentiment analysis, multimodal speaker traits recognition, and multimodal video retrieval. In this section, we provide a brief introduction of the datasets and metrics:
CMU-MOSI Zadeh et al. (2016a): is a collection of 93 opinion videos from YouTube movie reviews. Each video consists of multiple opinion segments (2199 segments in total) and each segment annotated with the score in the range (-3, 3], where 33 and 3 indicate highly negative and positive. We report the metrics of BA (binary accuracy). F1, Corr (Correlation Coefficient), MA (Multi-class accuracy, higher is better), MAE -cean-Absolute Error, lower is better).
al. 2013 : POM is speaker traits recognition dataset made up of 903 movie review videos. Each video has 16 speaker traits. We report the multi-class accuracy of different traits.
MSR-VTT al. (016)): MSR-VTT is composed of 10K YouTubr videos, collected using 257 queries from commercial video search engine. Each video is 10 to 30s long, and is paired with 20 natural sentences describing it. We report the common metries of @ and MdR.
5.2 DATA PREPROCESSING
CMU-MOSI and POM. Each dataset (CMU-MOSI, POM) consists of three modalities, including textual, visual, and audio modalities. For textual features, we employ the pre-trained c00-dimmnsional Glove embeddings Pennington et al. (014). For visual features, utilize Facet [Mottoos] (2017)

---

Under review as a conference paper at ICLR 2023
Table 1: The results on CMU-MOSI.
Existing Combs.
Unseen Combs.
Method
BA
F1
MAE
Corr
MA
BA
F1
MAE
Corr
MA
Random Drop
61.7
61.8
1.594
0.277
15.8
70.6
70.5
1.252
0.482
26.4
TFN Zadeh et al. 2017
62.7
62.2
1.407
0.317
19.8
71.6
71.1
1.090
0.513
28.6
MFN
Zadeh
al. 8a
64.3
64.1
1.329
0.381
24.0
73.2
73.0
1.012
0.577
32.9
LMF
Liu et al.
2018
63.3
63.6
1.276
0.417
22.7
72.2
72.5
1.037
0.613
31.6
T2FN Liang et 2219
64.3
64.1
1.250
0.428
23.0
72.7
72.8
1.055
0.610
31.9
MulTT T sai al. 2019)
64.8
64.7
1.236
0.439
24.5
73.5
73.6
1.015
0.620
33.2
MVAE
Wu et al. (2018)
65.1
65.3
1.283
0.425
25.0
74.9
74.7
0.997
0.608
32.5
MCTN
Pham al. (2019
64.8
65.1
1.244
0.456
24.7
73.7
74.0
1.007
0.607
32.5
MMIN
Zhao et al.
2021
65.4
65.6
1.208
0.445
25.6
74.5
74.0
1.054
0.618
31.9
IPD (Ours)
67.7
68.0
1.142
0.486
27.9
76.6
77.4
0.984
0.625
34.0
Table 2: Ablation study on CMU-MOSI.
Existing Combs.
Unseen Combs,
Method
BA
F1
MАE
Corr
MA
BA
F1
MAE
Corr
MA
w/o. All
63.3
63.6
1,276
0.417
22.7
72.2
72.5
1.037
0.613
31.6
w/o. LR
66.7
67.0
1.180
0.459
26.2
73.8
73.6
0.998
0.617
32.8
w/o. Orth
65.5
65.6
1.207
0.449
24.2
73.6
73.7
1.007
0.622
32.4
w/o. Ens
67.5
67.7
1.161
0.464
26.8
75.2
75.3
0.994
0.627
33.0
w/o. Cs
67.0
66.7
1.195
0.467
26.4
74.5
74.3
0.987
0.623
33.2
IPD (Full)
67.7
68.0
1.142
0.486
27.9
76.6
77.4
0.984
0.625
34.0
indicate 35 facial action units, which records facial muscle movement for representing the basic and advanced emotions. For audio features, we use COVAREP Degottex et al. (2014) acoustic analysis framework. To align the different modalities along the temporal dimension, we perform word alignment with P2FA Yuan & Liberman (2008).
The videos contain abundant multimodal information. Thus, we use multiple pre-trained models for extracting features. Concretely, we utilize seven modality experts: motion, scene, OCR, audio, speech, face, appearance following Gabeur al. (2020).
MSR-VTT.
5.3 MISSING MODALITY SETTING
We evaluate IPD from views. One the performances existing modality combinations, the other is the performances of unseen combinations. Concretely, we divide CMU-MOSI, POM into 23 7 pieces (scenarios). One piece kept for the evaluation of unseen generalization (Since the combinations of fewer modalities can be augmented with those of more modalities, we employ the complete modality combination for unseen evaluations). As for the remaining 6 pieces, We employ 70%, 10%, 20% of the samples for training, validation, and testing for existing modality combinations To be realistic, we randomly generate the ratio pieces (by giving each piece a number from 0 to and employing normalization) for times and report the average results.
For the simulation of a large number of the modality combinations, we conduct vidco retrieval experiments MSR-VTI which contains rich multimodal information. Since the number of modalities is 7, there are 27 combinations, employ 8 pieces (the combinations with more modalities). The remaining pieces are divided in the proportion of previous work Gabeur et al. (2020) (90% for training, 10% for testing). We follow CMU-MOSI randomly generate ratio and report the average results.
127 combinations. For the evaluation of unseen modality
The experimental details of all the tasks shown in the appendix.

---

Under review as a conference paper at ICLR 2023
Table 3: The performances on POM, where we report the multi-class accuracy of four traits, including Credible (Cre), Vivid (Viv), Expertise (Exp), Entertaining (Ent).
Existing Combs.
Unseen Combs.
Method
Cre
Viv
Exp
Ent
Cre
Viv
Exp
Ent
Random Drop
19.6
23.2
19.7
24.5
25.6
27.6
25.6
29.6
TFN
Zadeh et al.
2017
23.2
23.2
19.7
24.5
28.1
27.6
25.6
29.6
MIN
Zadeh et al.
(0188))
19.6
23.6
20.7
26.1
25.6
29.6
26.6
31.0
LMF
Liu ct al.
2018
19.6
23.2
20.7
28.1
25.6
27.6
26.6
33.5
T2FN
Liang ct al. (2019)
23.2
23.2
22.7
28.1
29.6
27.6
26.6
33.5
MulT
Tsai et al. 2019)
23.2
23.6
22.7
28.1
28.1
29.6
27.6
33.5
MVAE
Wu
al. (2018)
23.2
23.6
25.1
26.1
29.6
29.6
27.6
31.0
MCTN
Pham et al. (2019)
23.2
23.2
20.7
24.5
29.6
27.6
25.6
31.5
MMIN
Zhao et al.
(2021)
25.6
27.6
25.1
28.1
30.5
30.5
27.6
33.5
IPD (Ours)
27.1
30.5
27.1
30.5
34.5
36.9
31.0
33.0
5.4 EXPERIMENTS FOR MULTIMODAL SENTIMENT ANALYSIS
Table I presents the overall comparison of IPD and existing methods on CMU-MOSI (both existing and unseen combinations). Note that we mainly compare IPD with the methods that use the same features for fairness. As for the evaluation of existing modality combinations, we could observe that MFN, TFN, LMF, and MulT perform worse than IPD as they pay more attention to the usage of complete modalities, leading to the poor adaptation on the missing modality scenarios. Even MulT utilizes Transformer Vaswani et al. (2017). Besides, IPD achieves the best performances missing-modalle modality learning methods (including MVAE, MCTN, MMIN). Particularly, the performance of IPD increases the "MA" from 25.6 to 27.5 compared the best counterparts. We carefully analyze the observations: (1) The mainstream reconstruction based methods (i.e. MCTN, MMIN) depends the existence of all the modalities obtain the supervised information in the training stage, therefore, when the samples for training are imcomplete, there will be a big drop in performance. (2) The methods about modality invariant representation (i.e. MVAE) are also widely studied, however, since the noise covariance matrix in Eq. I varies across different modality combinations and samples, none of the features have the same distribution. Further, the assumptions for training of VVF also include the availability of all of the pre-defined modalities. As for the evaluation of unseen modality combinations, IPD performs better than all the baseline methods, which demonstrates that IPD has competitive generalization ability. In general, the best performances of IPD attribute to the advanced prototype decomposition which captures modality-shared knowledse ledge and modality knowledge, respectively, well as the disentangled contrastive constraints that enhances the interaction of different modality conditions.
Results:
all the metrics among the existing missing-
Ablation Study: We set some control experiments CMU-MOSL to verify the effectiveness of IPD and the results are shown in Table [ "w/o. All" denotes the model without all the contributions and equals LMF. "w/o. LR" denotes that all the orthogonal task prototypes other contributions (i.e. low-rank approximation, contrastive constraints, augmented ensemble). "w/o Orth" denotes that we just remove the orthogonal constraints for e, and A, from the complete model. "w/o. Ens" denotes that we just remove the low-rank ensemble learning from the complete model. "w/o. Cs" denotes the model without contrastive constraints for the modality specific prototypes. From the Table we could find that "w/o. All" achieves worst results on all the metrics, which is consistent with the objective law, since all the contributions for bassiing-mdaalle results of "w/o. LR" is similar to those of "IPD (Full)", we analyze that the model without low- rank constraints can also learn limited multimodal interaction from scratch. "w/o. Orth" achieves limited improvement based on the baseline method, which fits Proposition 2, the performance of some modality combinations would be influenced. The bad performances of "w/o. Ens" reveal the effectiveness of data augmentation. "w/o. Cs" achieves relatively bad results compared with IPD (Full), since the interaction enhancement among different modality combinations is important to simulate the low-rank condition.
trained separately without
learning are removed. The
We utilize low-rank approximation for E, and B, the rank values kt, R should be considered. We examine the performances of IPD CMU-MOSI (Existing Combs.) with different values of k; R,
8

---

Under review as a conference paper at ICLR 2023
Table 4: Retrieval performances (lext-to-video) on the MSR-V VTT dataset.
Existing Combs.
Unseen Combs.
Method
R@ If
R@
R@ 10T
M
R@ 14
R@5t
R@ 10
MdR.
CE Liu (019
14.9
40.2
52.8
16.4
41.3
54.4
8.7
MMT Gabeur et al. 2020)
18.2
46.0
60.7
20.3
49.1
63.9
6
MVAE Wu et al. (2018)
17.8
46.5
61.0
20.1
48.7
64.2
6
IPD (Ours)
18.6
47.4
62.3
6.5
20.6
50.3
65.8
5.3
Table 5: Paramters of the implementation of task prototypes.
Method
Params. (Prototypes)
Params. (All)
R@5
w/o. All
5.12 x 10a
1.33 10
46.0
w/o. LR
6.56 x 104
1.33 x 10
46.4
IPD (Full)
2.05 X 104
1.33 x 10
47.4
as shown in Fig. 3 relatively small rank values (suppose that & R). When removing the contrastive constraints that can promote the multimodal association, a large rank value is needed for the satisfactory results.
IPD with contrastive constraints could achieve competitive performances based a
We also conduct analysis for the low-rank ensemble learning. As shown in Fig. IPD has poor performances on the single modality without the mechanism, which reveals its effectiveness again.
5.5 EXPERIMENTS FOR SPEAKER TRAITS RECOGNITION
Table the experimental results of different methods on speaker traits recognition dataset POM, where the top half part corresponds to existing combinations and the bottom half part corresponds to unseen combinations, we report the multi-class accuracy of multiple traits) The simialr observation could be found from the table, IPD achieves competitive performances on both existing and unseen modality combinations compared with the baseline methods. Particularly, the performance of IPD increases the average multi-class accuracy from 26.6 to 28.8 (existing combs.) and from 30.5 to 33.8 (unseen combs.) compared to the best counterparts.
5.6 EXPERIMENTS FOR VIDEO RETRIEVAL
To evaluate the generalization for more modalities, report the evaluation results of IPD and the
competing text-video retrieval methods on MSR-VTT (Table 4). Note that MMIN, MCTN, and most of the existing methods (i.e. Ji et al. (2022) ; Lei al. (202 can be used for 7 modalities, thus, we do not compare IPD with them. We provide the metrics from two points of views (i.c. existing and unseen modality combinations). For both settings, IPD outperforms baseline methods in all the metrics. Benefiting from the low-rank prototype decomposition and disentangled contrastive constraints, the modality-shared knowledge and modality-speeific knowledge are efficiently combined for the corresponding tasks. Further, the complexity reduction is more obvious for video retrieval, due to the larger modality number and hidden size. To provide an intuitive complexity reduction, report the parameter number (Table 5) of the task prototypes.
6 CONCLUSION
We propose novel method called Interaction Augmented Prototype Decomposition (IPD) solve the generalizable missing modality problem. Concretely, IPD disentangles the task prototypes into modality-shared part and low-rank modality part. We present principled analysis to provide the rationality of low-rank approximation. Further, to control the rank value, we facilitate the interaction of different modality conditions by employing disentangled contrastive constraints, which complements the decomposition. Extensive results on the newly-created benchmarks of multiple tasks illustrates the superiority of our proposed method.
Due the space limits, provide the results of traits, the complete results shown in the appendix.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Hedi Ben- Younes, Rémi Cadene, Matthieu Cord, and Nicolas Thome. Mutan: Multimodal tucker fusion for visual question answering. In ICCV, 2017.
Gilles Degottex, John Kane, Thomas Drugman, Tuomo Raitio, and Stefan Scherer. Covarep collaborative voice analysis repository for speech technologies. ICASSP, 2014.
Tuong Do, Thanh-Toan Do. Huy Tran, Erman Tjiputra, and Quang D Tran. Compact trilinear interaction for visual question answering. In ICCV, 2019.
Valentin Gabeur, Chen Sun, Karteek Alahari, and Cordelia Schmid. Multi-modal transformer for video retrieval. In ECCV, 2020.
Richard A Harshman al. Foundations of the parafac procedure: Models and conditions for an" explanatory" multimodal factor analysis. 1970.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image
recognition. In CVPR, 2016.
Jie Hu, Li Shen, and Gang Sun. Squeeze-ane aauuezeeand-excitation networks. In CVPR, 2018.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700-4708, 2017.
iMotions. Facial expression analysis. 2017.
Kaixiang Ji, Jiajia Liu, Weixiang Hong, Liheng Zhong, Jian Wang, Jingdong Chen, and Wei Chu Cret: Cross-modal retrieval transformer for efficient text-video retrieval. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp.
2022.
Jie Lei, Linjie Li, Luowei Zhou, Zhe Gan, Tamara L Berg, Mohit Bansal, and Jingjing Liu. Less is more: Clipbert for video-and-lame aaeoo-nn-lanngage learning via sparse sampling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7331-7341, 2021.
Paul Pu Liang, Zhun Liu, Yao-Hung Hubert Tsai, Qibin Zhao, Ruslan Salakhutdinov, and Louis- Philippe Morency. Learning representations from imperfect time series data via tensor rank regularization. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 1569-1576, 2019.
Tao Liang, Guosheng Lin, Lei Feng, Yan Zhang, and Fengmao Lv. Attention is not enough: Mitigating the distribution discrepancy asynchronous multimodal sequence fusion. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp.
2021.
Yang Liu, Samuel Albanie, Arsha Nagrani, and Andrew Zisserman. Use what you have: Video retrieval using representations from collaborative experts. In BMVC, 2019.
Yunze Liu, Qingnan Fan, Shanghang Zhang, Hao Dong, Thomas Funkhouser, and Li Yi. Contrastive multimodal fusion with tupleinfonce. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 754-763, 2021.
Zhun Liu, Ying Shen, Varun Bharadhwaj Lakshminarasimhan, Paul Pu Liang, AmirAli Bagher Zadeh, and Louis-Philipee Morency. Efficient low-rank multimodal fusion with modality factors. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 2247-2256, 2018.
Mengmeng Ma, Jian Ren, Long Zhao, Sergey Tulyakov, Cathy Wu, and Xi Peng. Smil: Multimodal learning with severely missing modality. In AAAI, 2021.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation word representa- tions in vector space. arXiv preprint arXiv: 2013.
10

---

Under review as a conference paper at ICLR 2023
Srinivas Parthasarathy and Shiva Sundaram. Training strategies to handle missing modalities for audio-visual expression recognition. In ICMI, 2020.
Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In EMNLP, 2014.
Verónica Pérez-Rosas, Rada Mihalcea, and Louis-Philippe Morency. Utterance-level multimodal sentiment analysis. In ACI, 2013.
Hai Pham, Paul Pu Liang, Thomas Manzini, Louis-Philippe Morency, and Barnabás Póezos. Found in translation: Learning robust joint representations by cyclic translations between modalities. In AAAI, 2019.
Vihari Piratla, Praneeth Netrapalli, and Sunita Sarawagi. Efficient domain generalization via common- specific low-rank decomposition. In International Conference on Machine Learning, pp. 7728- 7738. PMLR, 2020.
Wasifur Rahman, Md Kamrul Hasan, Sangwu Lee, Amir Zadeh, Chengfeng Mao. Louis-Philipee Morency, and Ehsan Hoque. Integrating multimodal information in large pretrained transformers. In Proceedings of the conference. Association for Computational Linguistics. Meeting, volume 2020, pp. 2359. NIH Public Access, 2020.
Luan Tran, Xiaoming Liu, Jiayu Zhou, and Rong Jin. Missing modalities imputation via cascaded residual autoencoder. In CVPR, 2017.
Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, J Zico Kolter, Louis-Philippe Morency, and Ruslan Salakhutdinov. Multimodal transformer for unaligned multimodal language sequences. In ACL, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In NIPS, 2017.
Weiran Wang, Raman Arora, Karen Livescu, and Jeff Bilmes. On deep multi-view representation learning. In International conference on machine learning, pp. 1083-1092. PMLR, 2015.
Mike Wu, Noah Goodman, and Mike Wu. Multimodal generative models for scalable weakly- supervised learning, Advances in Neural Information Processing Systems, 31, 2018.
Saining Xie, Chen Sun, Jonathan Huang, Zhuowen Tu, and Kevin Murphy, Rethinking spatiotemporal feature learning: Speed-accuracy trade-offs in video classification. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 305-321, 2018.
Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2288-5296, 2016.
Jiahong Yuan and Mark Liberman. Speaker identification on the scotus corpus. Journal of the Acousticul Society of America, 2008
Amir Zadeh, Rowan Zellers, Eli Pincus, and Louis-Philippe Morency. Mosi: multimodal cor- pus of sentiment intensity and subjectivity analysis in online opinion videos. arXiv preprint arXiv:1606.06255, 2016a.
Amir Zadeh, Rowan Zellers, Eli Pincus, and Louis Louis-Philippe Morency. Multimodal sentiment intensity analysis in videos: Facial gestures and verbal messages. IEEE Intelligent Systems, 2016b.
Amir Zadeh, Minghai Chen, Soujanya Poria, Erik Cambria, and Louis Philippe Morency. Tensor fusion network for multimodal sentiment analysis. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 1103-1114, 2017.
Amir Zadeh, Paul Pu Liang, Navonil Mazumder, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. Memory fusion network for multi-view sequential learning. In AAAI, 2018a.
11

---

Under review as a conference paper at ICLR 2023
Without Contrastive Constraints
With Contrastive Constraints
28.5
26.5
26.5
25.5
25.5
23.5
23.5
22.5
22.5
Rank
Figure 3: Ablation study of rank value, suppose that two rank values (k in Eq. and R in Eq. 6 are
equal. Left and right parts denote IPD without and with contrastive constraints, respectively.
With Ensemble Learning
Without Ensemble Learning
37.5
32.5
32.5
27.5
27.5
22.5
22.5
17.5
17.5
12.5
12.5
7.5
Modality Combination
Modality Combination
Figure 4: Ablation study of low-rank ensemble learning, where we report the multi-class accuracy of different modality conditions. Left and right parts denote IPD without and with ensemble learning.
AmirAli Bagher Zadeh, Paul Pu Liang, Soujanya Poria, Erik Cambria, and Louis-Philipee Morency, Multimodal language analysis in the wild: Cmu-mosei dataset and interpretable dynamic fusion graph. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pupers). pp. 2236-2246, 2018b.
Jinming Zhao, Ruichen Li, and Qin Jin. Missing modality imagination network emotion recogni- tion with uncertain missing modalities. In ACL, 2021.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE transactions on pattern analysis and machine intelligence, 2017.
Kaiyang Zhou, Chen Change Loy, and Ziwei Liu. Semi-supervised domain generalization with stochastic stylematch. arXiv preprint 1.Xiv:200600099) 2021.
Tianfei Zhou, Wenguan Wang, Ender Konukoglu, and Luc Van Gool. Rethinking semantic segmen- tation: A prototype view. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2582-2593, 2022.
A PROOF AND DERIVATION
A.1 THE PROOF OF PROPOSITIONS
Proposition 1: When e 1 Span(A,), the decomposition of = c.1T + A has unique solution.
Proof: Suppose E =. + A w.11 + W.rT is rank- (k: 1) matrix, where A, E m R2 xk W. Rmik and r R22 all rank-k matrices with & & min(m, 2"). When
12

---

Under review as a conference paper at ICLR 2023
ec 1. Span(AAg) (Ag), to prove that we ee is equal to we 1 Span(W,). We should deduce the necessary and sufficient conditions.
Sufficient Condition: Suppose that we 1 Span(Wa). Then E We (Ee, w.) 1 3 (A]we)
||welll 1. Since E' is a rank- (I: + 1) matrix, we know that & Span(3) and so it has to be the case that (ee, we) = l| and A_ we = 0. Both of these together imply that ue is the projection of e onto the space orthogonal Ag i.c. we = = PA, ec, where PA, is the projection matrix onto the span of vectors Ag. Since 1 A, we can obtain that w. =
Necessary Condition: Let we Cc. Since E fixed, we could obtain A, w,r", i.c. (p,n(W.) = (A,) Since e, - (A,) we - Sp.n
Proposition If the orthogonal regularization is not satisfied (e, not orthogonal to (A.)) the performance of partial modality combinations will be influenced.
Proof: Suppose the modality -peecific prototypes of two single modalities are 61 and 62 (e1 e2). the fusion representation of input sample (corresponding to e2) is u. If the orthogonal regularization not satisfied (i.e., é1 is not orthogonal to e.). The prediction can be expressed as u en + u e2- In the extreme situation (i.e.,
en), the result is u 62. The value of u is influenced by the nonorthogonality, to some extent.
A.2 THE DERIVATION OF EQ. 9
Note that we omit M for convenience. To prove the calculation of O, follows:
R (EII C,C @ AA) r=1
O, = sum V@ (A, = sum
(10)
for convenience, we adopt the element-wise multiplication. Concretely, V R and A, Rmx2" cannot be operated with matrix (vectorial) multiplication. Therefore, we indirectly calculate "a= R²" d [m], we rewrite the above equations,
the result of (V)a E R² and AA3"
0x (AeddT) ( Aga
( (cri)))
(11)
((8 (us)d) 3 A
((0 (miida)
[c [Ct,... ckI)
1=1
where (v:)a E TR2,, As,d E & We then calculate
8
C as follows:
n ΣI
X
(oi)a))
1 (c))a)
(o)).O
(12)
br
r=
we put the result of Eq.
into Eq. and obatain:
R
R
0, = II
(o))C.rr) Aod (o.)C..))A. sum [EI
& Aad)
(13)
r=1
r=1
According to the conclusion of Eq. follows:
13
Eq.
10
proved. We then prove the the calculation of O, as
O_ sum sum
V o
(e)1))
= sum [
① Cc
(14)
We also adopt element-wise multiplication.
13

---

Under review as a conference paper at ICLR 2023
O. ((8))
(w:)d) 12" Ce,d = sum (II (1)aa 1
o Cead)
(15)
According to the conclusion of Eq.
15 Eq.
is proved.
Algorithm The Training Process of IPD
1: Given: m, k, Å1, A2, λ3. train-data
2: Initialize params 6z. E IRmm, A E R.
3: Initialize C IR² E [n],r [R] E [k] in Eq.
4: Co < |1++1 [ec, A.] [ec, A,]l
> Orthonormality constraint
5: for 11 E dtrain-data do
6: E eel A A.3
D B1 and 3 are calculated based on Eqs. and
7: f Agg.
8: loss += Li (r),)e))) (NC. deCo + Asho
9: end for
10: Optimize loss wrt cc. Ag. C2,
11: Return Ecr As, Ch,
Algorithm 2 The Inference Process of IPD
1: Given: n, m, k, test-data
Trained params & E. A. Rmnk , , R22
is calculated based on Eqs.
and
3: for (x, test-data do
4: ej 6- 6, + A,3
5: (²(x) Net(1r;
6: end for
B MORE EXPERIMENTAL DETAILS
B. THE DETAILED PROCESS OF IPD
Due to the space limitation, we put the algorithmic process (both training and testing) in this section. The mathematic symbols are same as the main paper. In the training process. Oz(z) denotes the output Og of input x, Le (z). ) y; E) denotes the calculated loss based on modality factors E, and (y (I), ex) denotes the calculated loss based modality factor e₁.
In the testing process, we first obtain the modality factor e according to the modality combination index, then, we send the input to the network and obtain y (ar) = Net(z;er).
B.2 THE TRAINING ORJECTIVE OF VIDEO RETRIEVAL
As shown in MMT Gabeur et al. (020), the training objective of video retrieval is triplet contrastive loss, which compares the matching scores between video representation and text representation. To implement
method, we employ new contrastive cross-entropy loss. Concretely, we treat the query text as one of the modalities the video and text modality always exists. If the query text and the video are matched, the classification output is 1. In batch, treat the matched text and video as positive sample and change one side to construct negative samples. Then, the cross-entropy loss can be used train the network.
B.3 THE FEATURE EXTRACTION FOR VIDEO RETRIEVAL
Motion embeddings extracted from S3D Xie et al. 2018 trained on the kinetics dataset. Scene embeddings are extracted with DenseNet- Milee1661ee Huang et al. (2017) trained on the Places365 dataset Zhou et al. (207)) OCR embeddings extracted in three stages. First, the pixel link text detection
14

---

Under review as a conference paper at ICLR 2023
model is used detect the overlaid text. Then, the detected boxes passed through the text recognition model. Finally, each character sequence is encoded using a word2vec embedding. Audio embeddings obtained with a VGGish model, trained the YouTube-8nn 8m dataset. Speech features are extracted using the Google Cloud speech API, to extract word tokens from the audio stream, which are then encoded via pre-trained word2vec embeddings Mikolov et al. (2013). Face features are extracted by ResNet-50 He et al. (2016) trained for face classification on the VGGFace2 dataset. Appearance features are extracted from the final global average pooling layer of SENet-154 Hu et al. (2018) trained on ImageNet.
B.4 EXPERIMENTAL DETAILS FOR MULTIMODAL. SENTIMENT ANALYSIS AND SPEAKER TRAITS RECOGNITION
We follow LMF to implement IPD. All the components of IPD same those of LMF, except for the final classification layer. Specifically, the hidden size of the task prototype is 16, the rank k and R set to 5. À is the batch size is set to 16, the learning rate is 0.001. The hidden size of common feature space is 16. \1, A2, A3 are set 0.1. We develop all the experiments with 5 RTX-3080TI GPUs (10GB).
0.1. The tuning of other parameters is similar LMF. After grid searching,
B.5 EXPERIMENTAL DETAILS FOR VIDEO RETRIEVAL
We follow MMT implement IPD. All the components of IPD are same those of MMT. We add the new contrastive cross-entropy loss introduced above to the module and employ it as an auxiliary loss. Specifically, the hidden size of the task prototype is 512, the rank & and R are set 32. A is set to 0.1. The tuning of other parameters is similar to MMT. Concretely, the batch size is 32, the initial learning rate is set to 5e steps. We train for 50k steps. The hidden size is 512 for all the Transformer structures, the number of heads is and there are 6 stacked attention blocks. X1, X2, Ag are set to 0.1. We develop all the experiments with 5 RTXX308011 GPUs (10GB).
which decays by multiplicative factor 0.95 every 1k optimization
B.6 EXPERIMENTAL DETAILS OF BASELINE METHODS
We simply reproduce the baseline methods by replace the missing-modality features by all-zero vectors and follow these methods for the subsequent processes.
C MORE EXPERIMENTAL RESULTS
C.I ADDITIONAL RESULTS OF POM
The results of other all 16 traits are shown in Table Table We could obtain similar conclusions.
C.2 ADDITIONAL RESULTS THE CASE THAT ALL THE MODALITIES ARE AVAILABLE WHEN TRAINING
We also conduct experiments when all the modalities available during training. In this way, adopt the same train/val/test splits as LMF The validation and test sets are equally divided into 7 pieces of modality combinations. We report the corresponding results of different methods. As shown in the figure, the metric gap between IPD and baseline methods narrows, since MVAE, MCTN, MMIN are original designed for this case.
C.3 ADDITIONAL RESULTS OF COU-M
The results of CMU-MOSEI Zadeh al. 2018b) proposed for the evaluation of multimodal sentiment analysis. It is similar to CMU-MOSI and has 23454 movie review videos. We could obtain similar conclusions based on Table g
shown in Table 9
CMU-MOSEI is also
15

---

Under review as a conference paper at ICLR 2023
Table 6: The performances POM for existing modality combinations, MA(5,7) denotes multi-class
accuracy for (5,7) classes
Con MA7
Pas
Voi
Dom
Cre
Viv
Exp
Ent
Method
MA7
MA7
MA7
MA7
MA7
MA7
MA7
TFN Zadch et al.
(017))
21.7
26.1
20.6
27.1
23.2
23.2
19.7
24.5
MFN
Zadeh al.
2018a)
20.6
25.5
23.2
28.5
19.6
23.6
20.7
26.1
LMF
Liu et al.
20
23.6
26.1
24.5
29.5
19.6
23.2
20.7
28.1
MulT
sai ct al.
(2019)
23.6
27.6
28.6
29.5
23.2
23.6
22.7
28.1
MVAE Wu et al.
(2018)
21.7
20.2
28.6
28.5
23.2
23.6
25.1
26.1
MCTN
Pham
(2019
23.6
27.6
28.6
28.5
23.2
23.2
20.7
24.5
MMIN Zhao (2021)
21.7
27.6
30.5
30.0
25.6
27.6
25.1
28.1
IPD (Ours)
25.6
27.1
34.5
31.5
27.1
30.5
27.1
30.5
Res
Tru
Rel
Out
Tho
Ner
Per
Hum
Method
MA5
MA5
MA5
MA5
MA5
MA5
MA7
MA5
TFN
Zadeh et al.
(0177)
22.7
37.4
36.9
37.4
36.5
35.5
36.0
37.4
MFN
Zadeh et al.
(2018a)
25.6
37.9
38.4
38.9
35.5
35.5
37.4
38.4
LMF
Liu et al. 18)
25.1
38.9
38.9
37.4
36.9
34.0
36.9
37.4
MulT
Tsai al.
(2019)
27.6
39.9
38.9
37.4
37.9
35.5
36.9
37.4
MVAE Wu et al.
(2018)
26.6
37.9
40.4
38.4
37.9
37.4
38.9
39.9
MCTN
Pham et al.
(2019)
27.6
39.9
40.9
40.9
38.4
37.9
38.9
40.4
MMIN
Zhao et al.
(2021)
29.6
39.9
40.4
41.9
38.4
39.4
39.9
41.9
IPD (Ours)
29.6
39.9
41.9
43.8
39.4
39.4
40.9
42.4
Table 7: The performances on POM for unseen combination. MA(5,7) denotes multi-elass accuracy for (5,7) classes.
Con MA7
Pas
Voi
Dom
Cre
Viv
Exp
Ent
Method
MA7
MA7
MA7
MA7
MA7
MA7
MA7
TFN
Zadch et al.
(017)
27.6
32.0
27.1
32.5
28.1
27.6
25.6
29.6
MFN
Zadeh al.
(018a)
26.1
31.5
29.6
34.5
25.6
29.6
26.6
31.0
LMF
Liu al. (2018)
28.6
32.0
30.5
35.5
25.6
27.6
26.6
33.5
MulT T sai
2009
27.6
31.5
34.5
34.5
28.1
29.6
27.6
33.5
MVAE Wu al.
(2018)
27.6
27.1
34.5
34.0
29.6
29.6
27.6
31.0
MCTN
Pham
(2019
28.6
33.5
34.5
34.0
29.6
27.6
25.6
31.5
MMIN Zhao ct al.
2021
27.6
33.5
34.5
35.5
30.5
30.5
27.6
33.5
IPD (Ours)
30.5
33.5
36.9
37.4
34.5
36.9
31.0
33.0
Res
Tru
Rel
Out
Tho
Ner
Per
Hum
Method
MAS
MA5
MAS
MA5
MA5
MAS
MA7
MA5
TFN Zadeh et al.
2017
27.6
41.9
42.9
44.8
40.9
38.4
39.9
42.4
MFN
Zadeh al.
2018a)
29.6
39.9
42.9
45.8
39.9
38.9
41.9
43.8
LMF
Liu al.
28.6
41.4
43.8
44.8
39.4
38.4
40.9
43.8
MulT
Tsai
2019
31.0
43.8
43.8
44.8
40.9
38.9
41.4
42.4
MVAE Wu et al.
(2018)
30.5
43.3
44.3
45.3
40.4
40.9
41.9
44.3
MCTN
Pham
(2019
30.5
43.8
45.3
46.8
40.9
41.4
41.4
44.8
MMIN
Zhao et al.
(021))
31.0
44.8
45.3
47.8
40.4
41.4
40.9
44.3
IPD (Ours)
32.5
43.8
46.8
48.3
41.9
42.4
42.9
44.8
C.4 ADDITIONAL RESULTS OF RGB-D AND XRMB
We also conduct experiments on XRMB and RGB-D. As for XRMB, it has two modalities (273D acoustic inputs and 1 112D articulatory inputs) following Wang al. (2015), thus, the power of IPD not obvious with achieving a similar results to baseline methods. As for RGB-D, due to the imbalanced importance of different modalities (3D point cloud, RGB color and height, following
16

---

Under review as a conference paper at ICLR 2023
Table The results on CMU-MOSI when all the modalities are available when training.
Metrics MAE
Method
BA
F1
Corr
MA
MVAE
Wu et (2018)
74.3
74.5
1.065
0.621
27.8
MCTN
Pham
(2019)
73.9
74.0
1.019
0.601
28.9
MMIN
Zhao al. (2021)
74.8
74.7
0.987
0.625
29.6
IPD (Ours)
75.4
75.7
0.985
0.621
30.2
Table 9:
The results on CMU-MOSEL. We report BA (binary accuracy), F1, Corr (Correlation Coefficient), MA (Multi-class accuracy, higher is better), MAE (Mean-absolute Error, lower is better).
Existing Combs.
Unseen Combs.
Method
BA
F1
MAE
Corr
MA
BA
F1
MAE
Corr
MA
MVAE
Wu et al. (2018)
67.2
67.5
1.050
0.409
36.7
76.1
75.7
0.725
0.498
43.5
MCTN
Pham et al. (2019)
66.9
67.3
1.007
0.440
36.4
74.9
75.0
0.735
0.497
43.5
MMIN
Zhho et al. (0211
67.5
67.8
0.975
0.429
37.3
75.7
75.0
0.782
0.508
42.9
IPD (Ours)
69.8
70.2
0.909
0.470
39.6
77.8
78.4
0,712
0,515
45.0
keep the 3D point cloud fixed available modality. The missing modality setting is adopted to RGB color and height. Since it is hard to model multimodal interaction with two modalities, IPD also performs similar to the baseline methods.
Liu et al. [02
17

---

Under review as a conference paper at ICLR 2023
Table 10: The results on XRMB. We report PER (phone error rate).
Existing Combs.
Unseen Combs.
Method
PER
PER
MVAE Wu et al.
2018
0.274
0.255
MCTN
Pham et al.
(2019)
0.272
0.248
MMIN
Zhao et al. 2021
0.283
0.252
IPD (Ours)
0.267
0.245
Table 11: The results on RGB-D. We report m.P@0.25 for 3D object detection.
Existing Combs.
Unseen Combs.
Method
mAP@0.25
mAP@0.25
MVAE
Wu et al. 2018
43.6
49.2
MCTN
Pham et al. (2019)
44.7
49.5
MMIN
Zhao et al. 2021
45.8
49.8
IPD (Ours)
46.7
50.4
18


Reviews:
[{"review_id": "ivGeaYXe6d", "reviewer": "Reviewer_QUyJ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an Interaction Augmented Prototype Decomposition (IPD) model for missing modality learning.", "summary_of_the_review": "This paper proposes an Interaction Augmented Prototype Decomposition (IPD) model for missing modality learning. Some key points are easy to follow. However, the clarity needs to be improved.", "strength_and_weaknesses": "The idea is interesting. However, the motivation is not clearly given. Besides, the writing can be improved.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Rae1jfm5Cs", "reviewer": "Reviewer_pCS1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the problem of robustness in the face of noisy or missing modalities and proposes a method called Interaction Augmented Prototype Decomposition (IPD) for the setting where the number of modalities is arbitrary and there are various incomplete modality conditions happening in both training and inference phases. Their approach jointly learns the common and modality-specific task factors via low-rank decomposition, which seems to promote unseen generalization in empirical experiments. They show strong results on 3 major multimodal datasets.", "summary_of_the_review": "Some additional comparisons and tradeoff analysis would be good.", "strength_and_weaknesses": "Strengths:\n1. The paper is well motivated and described. The ideas are clear and experiments are on several large multimodal datasets.\n2. The paper is largely clear with clear figures and exposition.\n\nWeaknesses:\n1. There are insufficient comparisons to work in missing modalities, with 'Jinming Zhao, Ruichen Li, and Qin Jin. Missing modality imagination network for emotion recognition with uncertain missing modalities. In ACL, 2021.' being the only one. The authors cite 'Paul Pu Liang, Zhun Liu, Yao-Hung Hubert Tsai, Qibin Zhao, Ruslan Salakhutdinov, and LouisPhilippe Morency. Learning representations from imperfect time series data via tensor rank regularization. In ACL 2019.' but do not compare to it (they also use low rank decomposition) of multimodal representations, and there are also some simple baselines such as including modality dropout during training that should also be compared to.\n2. There should be more comparisons in both performance and complexity - the proposed method does better but certainly uses more computation so the tradeoff should be analyzed.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "LLacwd3H-I", "reviewer": "Reviewer_YKCH", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper under review considers the problem of multimodal learning in the case where one or more of the modalities might be unavailable during either training or testing. The proposed solution works by decomposing representation space into a collection of orthogonal vectors, a “common” component to capture high-level semantic information, and modality-specific components to capture low-level information associated with each modality.", "summary_of_the_review": "I recommend rejection for this paper. The proposed method may work well, but I think the exposition is too difficult to follow for readers to learn much from this work. I suggest that the authors redraft this work with a focus on clarity of method, clarity of data, and clear understanding of why the proposed method works well.", "strength_and_weaknesses": "*Strengths:* Dealing with missing modalities during both training and testing is an important practical problem that deserves study. The proposed method outperforms other recent methods on a benchmark introduced by the authors.\n\n*Weaknesses:* I found this paper very difficult to follow. After several close readings, I am still not confident that I understand the technical details of the proposed method. Specifically, \n\n(1) Are the pairs (x, y) the training data? If so, how is the input x formed?\n\n(2) Is Equation (1) the abstract model of data generation? If so, why is that a reasonable model? If not, then where is Equation (1) coming from?\n\n(3) How does the “maximum suppression” scheme described in Section 3.5 stop Equation (8) from blowing up exponentially?\n\n(4) Is the ensemble learning scheme of Section 3.6 used during training only or also during testing?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "-tPpZCcoTU", "reviewer": "Reviewer_SiTt", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors are interested in the multimodal learning problem. They want to focus on cases where incomplete data (not data from all modalities) are present during training. To do that, they assume that the fused vector x (output of LMF) can be given by the task classification (or regression) labels times e_l, which is the complete task prototype for all possible modality combinations L. Since (x,y) are given, the paper's main focus is how to learn the vector e_l efficiently. Finally, they test their trained models on unseen modality combinations.", "summary_of_the_review": "We believe that it is important that the authors tried to develop a method for incomplete modality information during training. Because it is closer to real scenarios. The proposed method managed to improve the classification/regression performance when compared to other methods. But we believe that the paper can be improved and make more clear its advantages by addressing our comments in the previous sections.\n\n------------------------------------\nPost rebuttal summary:\n\nWe thank the authors for answering all of our questions. \n1) We believe the authors' responses for 2.1, 2.2, and 2.3 should be part of the paper.\n2) For 2.3 and the L1 normalization, the question referred to normalized data. It needs to be clarified from the text on what the L1 is applied.\n3) Figure 2 is more clear now! But it can be further improved; currently, it mainly lists the definitions of each parameter. The authors could reformulate it, so it has a better flow.\n\nAfter reading the other reviews and authors' responses, we kept our recommended score to 6.", "strength_and_weaknesses": "> Strengths\n1.\tThe authors try to develop a method to handle more realistic scenarios where some data modalities are missing.\n2.\tThe authors try to make good use of all available problem information with the regularized losses (disentangled contrastive constraints, ensemble learning). \n3.\tThey reduce the exponential complexity of the problem, which is dependent on the possible modality combinations (parameter l) through low-rank representations and CP decomposition.\n\n> Weaknesses\n1.\tFor the unseen modality combination: We believe that the current experiments do not test the model's performance on unseen modalities. It would make sense if a completely new modality were present or some modality was abnormal or out-of-distribution during testing. The fact that the trained models were exposed to all possible multimodal combinations except the case where all modalities are present seems not to test the unseen case but rather how much the model's performance can be improved when all modalities are available. \n2.\tBased on the previous argument, we would like to ask the authors to compute the performance of their method (and the rest) when all modalities are present to see the impact on the model when we have missing modalities. This will help us assess its performance compared to the other methods. Page 8 \"...methods (i.e. MCTN, MMIN) depends on the existence of all the modalities to obtain the supervised information in the training stage; therefore, when the samples for training are imcomplete, there will be a big drop in performance. \" these experiments could potentially help support this argument. Since the performance of MMIN is very close to IPD, it even, in some cases, outperforms it.\n3.\tEnrichment of related work, most of the referenced publications are from 2017,2018, with only one from 2019 (the most recent one). Also, give more details about previous papers, for example, \" Liang et al. (2019) employs low-rank fusion for each time step of multi-view sequential input. \" and how the proposed method compares to Liang et al. (2019).\n4.\tSome captions of figures are too short. Specifically, Figure 2. Not all diagram variables are explained. Figure 3: similarly, what is k, and what is R?\n5.\tFigures 3: We believe the figures would be more readable if there were a title above the bar graphs \"With, without contrastive constraints.\"\n6.\tFigure 2: Please consider adding more details, like the equation connecting the two decoding parts. How are vectors connected to the architecture, and what are the gray cells? In general, the figure and caption need more work. It is tough to understand what the model is doing and how the different pieces connect to the figure. Even if we read the paper and then look at the figure, it isn't easy to map things from text to pictorial.\n7.\tThroughout the document, there exist claims/sentences that remain unsupported or not fully explained; we would like the authors to go through these sentences and explain what they hold.\na.\tPage 4 \" Furthermore, the decomposition of A\\beta^T does not consider the fine-grained interaction among different modality combinations. \" why is this the case? How does CP decomposition solve this problem? Does it?\nb.\tPages 6: \"We report the metrics of BA (binary accuracy), F1, Corr (Correlation Coefficient), MA (Multi-class accuracy, higher is better), and MAE (Mean-Absolute Error, lower is better). \" \"We report the common metrics of R@K and MdR. \" Can the authors include the definition of these metrics or provide references?\nc.\tPage 7: \"To be realistic, we randomly generate the ratio of 7 pieces (by giving each piece a number from 0 to 1 and employing normalization) \": not clear what normalization is applied and why.\nd.\tPage 8: \" Even MulT utilizes Transformer Vaswani et al. (2017). \" what's the conclusion for this sentence? What are the authors trying to say to the reader?\ne.\tPage 9: \"Note that MMIN, MCTN, and most of the existing methods (i.e. Ji et al. (2022);Lei et al. (2021)) can not be used for 7 modalities, thus, we do not compare IPD with them.\" Please give more details, why not applicable.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject