A Mathematical Framework for Characterizing Dependency Structures of Multimodal Learning
Authors: Weida Wang
Tao Shi
Yaoyuan Liang
Xiangxiang Xu
Fei Ma
Shao-Lun Huang
Lizhong Zheng
Venue: Submitted to ICLR 2023 (2023)
Keywords: Multimodal learning
dependency structures
correlation anaylses
emotion recognition
classification

Abstract: Dependency structures between modalities have been utilized explicitly and implicitly in multimodal learning to enhance classification performance, particularly when the training samples are insufficient. Recent efforts have concentrated on developing suitable dependence structures and applying them in deep neural networks, but the interplay between the training sample size and various structures has not received enough attention. To address this issue, we propose a mathematical framework that can be utilized to characterize conditional dependency structures in analytic ways. It provides an explicit description of the sample size in learning various structures in a non-asymptotic regime. Additionally, it demonstrates how task complexity and a fitness evaluation of conditional dependence structures affect the results. Furthermore, we develop an autonomous updated coefficient algorithm auto-CODES based on the theoretical framework and conduct experiments on multimodal emotion recognition tasks using the MELD and IEMOCAP datasets. The experimental results validate our theory and show the effectiveness of the proposed algorithm.

Under review as a conference paper at ICLR 2023
A MATHEMATICAL FRAMEWORK FOR CHARACTERIZ- ING DEPENDENCY STRUCTURES IN MULTIMODAL
Anonymous authors Paper under double-blind review
ABSTRACT
Dependency structures between modalities have been utilized explicitly and im- plicitly in multimodal learning to enhance classification performance, particularly when the training samples are insufficient. Recent efforts have concentrated on developing suitable dependence structures and applying them in deep neural ner- works, but the interplay between the training sample size and various structures has not received enough attention. To address this issue, we propose a mathematical framework that can be utilized to characterize conditional dependency structures in analytic ways. It provides an explicit description of the sample size in learning various structures in non-asymptotic regime. Additionally, it demonstrates how task complexity and a fitness evaluation of conditional dependence structures affect the results. Furthermore, we develop an autonomous updated coefficient algorithm auto-CODES based on the theoretical framework and conduct experiments on multimodal emotion recognition tasks using the MELD and IEMOCAP datasets. The experimental results validate our theory and show the effectiveness of the proposed algorithm.
INTRODUCTION
Multimodal learning is recently an active research area in machine learning aiming at jointly extracting information and learning knowledge from different categories of data, such as images, audios, and texts (Ngiam al., 2001 Zadeh al.) 2019; Kiela et al., (020). In multimodal learning, a critical issue is to design efficient algorithms to extract features from various modalities, such that the label information can be effectively extracted for classification, especially when the number of training samples is insufficient to learn huge and complex multimodal structure. There have been many kinds of literature addressing this issue with different kinds of algorithms (Baltrušaitis et al., 2018). in which the main research stream focuses on extracting features from several modalities that are both relevant to the label and connected to one another (Gao et al.) 2020: Ma et al., 2020 Summaira et al. (021). The effectiveness of such algorithms could have resulted from the intuition that the labels appear to be the common patterns shared between different modalities in many real multimodal datasets. For multimodal data with such property, designing modality features with higher correlations will implicitly force the algorithm to search for more informative features to the label, and hence can often require fewer training samples to achieve good performance.
From the statistical learning aspects, such benefits can be interpreted as that the modalities and the label follow a conditional dependency structure where modalities are independent of each other once the label is given. Since it is in relatively Now-dimensiona space, it will demand less number of samples to learn a good representation (Varma al., 2019). Thus, there can be several factors affecting the classification performance including (i) the number of labeled training samples, (ii) the fitness of the conditional dependency structures to represent the true one, and (iii) the complexity of the discrimination task.
There are works exploiting appropriate dependency structures to achieve good performances by designing effective networks, fusion approaches (Zadeh et al. 2017 Liu et al. 2018 Nagrani et al.) 2021) and objective functions (Sohn et al. 2014; Sutter et al. 2020 Piergiovanni et al. 2020). However, most existing works focus designing learning algorithms and architectures without theoretical understanding of the training sample size in multimodal problems, which potentially limits the performance, especially for complicated multimodal problems.

---

Under review as a conference paper at ICLR 2023
In this paper, we consider the model to learn a linear combination of two types of estimators (Pier giovanni et al. 2020)) representing the general dependency structure and the considered conditional dependency structure. Then, we propose testing loss function evaluate the performance of the trained model in a non-asymptotie regime. It considers the average performance of the linear combined estimator over certain number of training samples. Note that is parametrized by linear combination coefficient of different types of estimators. Minimizing the testing loss will give us the optimal coefficient which determines the optimal estimator which can lead to more informative classification features. The optimal coefficient can also be used to characterize the conditional dependency structure in the process. The detailed mathematical formulation and interpretations are presented in Section and Section
In particular, we show that the explicit analytical solution of the optimal coefficient inversely proportional to the number of labeled training samples and the fitness measurement of the conditional dependency structure. Also,
proportional to the complexity of the learning task, which we will discuss in Section 2 For instance, when the number of samples is insufficient to learn high- dimensional model, it will be preferable to fit the low-dimensional conditional dependency structure which requires less number of parameters, Mcanwhile, the coefficient for the estimator corresponding to the conditional structure will be large. Therefore, the optimal coefficient essentially indicates the efficient model that one shall choose for predicting the label in the multimodal problem with the number of training samples taken into account. Moreover, our approach essentially provides guidance and theoretical understanding for designing efficient multimodal algorithms utilize the sample size information and different dependency structures.
At last, extend our theoretical results and propose an autonomous updated coefficient on depen- dency structures (auto-CODES) algorithm by exploiting parametric models. It can compute the weights on different dependency structures automatically with the features evolving in deep neural networks. The experiments on the emotion recogmition tasks with the MELD and IEMOCAP datasets validate our theoretical results and show the effectiveness of the algorithm. The main contributions of this paper can be summarized as follows:
We propose a novel theoretical framework for multimodal analyses to characterize the influence of the conditional dependency structure. To the best of our knowledge, it is the first work to give an explicit characterization of the number of training samples toward different dependency structures for multimodal learning in a non-asymptotic regime. Also, it quantifies the task complexity and the fitness of the conditional dependency structure, measured by the >ivdieergence, to the estimation.
We extend the analyses from discrete to continuous data in the real world by exploiting parametric models. Furthermore, we propose an algorithm with the autonomous updated coefficient on different dependency structures (auto-CODES) based on the theoretical analyses.
We evaluate the proposed algorithm auto-CODES on multimodal emotion recognition tasks with the widely used MELD and IEMOCAP datasets. The experimental results validate our theory and show the effectiveness our algorithm.
Due to space limitations, the proofs of theorems are presented in the supplemental material.
PROBLEM FORMULATION AND ANALYSIS
In this section, we consider multimodal scenario where both modalities are discrete random variables. For better illustration, we elaborate the framework in two modalities case. Specifically, we focus the linear combination of two types of estimators. By introducing the testing loss, evaluate the performance of the proposed estimator. Finally, we determine the optimal combining coefficient by minimizing the testing loss and illustrate the aspects that affect the optimal coefficient.
Notation:
First, let random variables X1, X2 and Y denote different modalities and label over finite alphabets X;, Xr and )/ respectively. Then, n sample tuples 'D & {zz(" , .x2 ())) }2_ are generated in an independent, identically distributed (i.id.) manner from the true joint distribution Pxix,r, where Px, X2y (I), (2, y) > 0 for all entries. Specifically, we consider two different
estimators approximate the joint distribution Xx,,, : (i) the empirical joint distribution Px, Xxy,
2

---

Under review as a conference paper at ICLR 2023
and (ii) the empirical Markov-structured distribution ((M) X, which characterize the conditional dependency structure X1
X2.
(x1, y) 1 71 E 1(
x2) y)
(Ia)
p XiXay (21,
22, y) ( (
(lb)
where 1 [[.} denotes the indicator function, Py denotes the marginal empirical distribution of label y
Y and P(M) X, denotes Px, yx, Py. For simplification, we consider the case when the label distribution has been learned wel i..., (y) (y))), for y V.
To begin, we focus on a linearly combined estimator based on two different structures.
Xx,x, _ a)) Px,x,r + ((M)
X2
where the coefficient a E [0, 1] is the parameter to be designed
Note that when becomes zero, the proposed estimator equation 2 will degrade into widely-used unbiased estimator Xx, When a goes to one, it will become the estimator ((M)
X,y representing the conditional dependency structure, i.e. X1 - Y X2. Thus, there has optimal combining coefficient to make the estimator equation [2 the most appropriate estimation for label prediction. In
addition, the optimal coefficient be used characterize the dependency structure and provide theoretical insights for deriving the optimal estimation.
2.1 OPTIMAL COMBINATION COEFFICIENT
We define testing loss function based on the referenced x2 -divergennce to measure the performance of the estimator equation 2 where the referenced divveeeree is defined as follows Definition 1. For discrete random variable Z over finite alphabet Z, and its distributions Pz and Qz, with reference distribution Rz, the referenced x² -dviiergee between them defined as
(Pz(2) Q2(2))²
Xiy (Pz, (2,,Qz)=
Σ #EZ
R2(z)
We denote x2
x7,, (Pz, Qz), which corresponds to the Pearson x² -iveeregen
Based the referenced x2 -diveergence, define the testing loss the average divergence between the estimator (2) and the true joint distribution under the fixed training sample size n.
Definition 2. For estimator Px,X,y with coefficient and the corresponding true distribution Px,X,Y, the testing loss and the optimal coefficient of are defined as
Lrest (a) AE ( (x
arg min (aest aE|0,1]
where the expectation is taken over all i.i.d. samples generated from the true distribution.
Then, have the following characterization of our proposed testing loss over the linearly combined estimator and the optimal combining coefficient a*.
Theorem 3. The testing loss as defined equation can be expressed as
Crest (a) (-c+ n n
Lv X2 x1xxr) a² a -
|D1 1). (5)
n n
'Note that in many real- world datasets, such as MNIST, the label will be uniformly distributed in the training set which makes this assumption reasonable.
"It can be shown that the estimator equation 2 can be naturally derived from optimizing a linearly combined Log-Loss function, where we refer to the supplementary material for detailed discussion.
'Conventionally. such performance is computed by logarithm loss. However, in our setting, will be ill- defined when some 32, y) tuple is missing training samples. By that time, we have Xx,x, (x), 22, v) 0 while Px,x, x.y (111 22 y) O, which would bring the logarithm loss infinite.
3

---

Under review as a conference paper at ICLR 2023
and the optimal coefficient a' to minimize the testing loss equation 4
can be given
1C
o² X
(6)
PlM) x2x,y)
_ +
where
C 12 (122) [22 + an
(7)
V -6 (Pxxxr, P +2 Σ
EXPPIIX
#2.31
21-1/
+ |D] (124) + (122)) - bn.
(8)
On and bn are of the order O,,), which will go to constants when goes to infinity. The proof and the detailed analytical expressions of an and bn are provided in the supplementary material.
By considering the conditional dependency structure and tuning the coefficient a, the improvement can be calculated by the difference in the testing losses.
Corollary 4. The improvement of considering the optimal coefficient a²* can be given as
C
(0) Lrest(a") n
(M) PxXXa)
where parameters C and V are defined in Theorem 3
From the expressions equation the optimal combining coefficient a" is determined by three main factors: (i) the training sample size nı (ii) the fitness the conditional dependency to describe the true one, measured by the >d-divergence x2 (Px, X,y,
((M) x,X,,) and terms in the parameter V; and (iii) the task complexity C, characterized by the number parameters needed to estimate the joint distribution. The last characterization comes from the fact that when the task is to learn all the entries of the true distribution, the number of parameters we need corresponds to the cardinality of the sample space.
Many existing multimodal algorithms (Ma et al., 2020) focus on finding the appropriate dependency structures to approximate the true one, while the number of training samples is not sufficiently addressed. In Theorem 3 we show that the combining coefficient is inversely proportional to the number of training samples and the fitness measure of the conditional dependency the true one. Also, it is proportional to the task complexity measured by the number model parameters.
There two interesting special cases for better understanding of Theorem and Corollary 4 Case 1: When the true dependency structure is Markovian, i.e. X1 Y X2, the optimal coefficient will becomes VCC + V)-1, which is nearly
| |21 -1 The cardinality terms |.X] and [22] are usually large which results in that the a is quite close to 1, representing that the model should be close Markov one | and the improvement relatively large. Case 2: When the number of training samples is relatively small and insufficient to learn a complex model, the optimal coefficient a" would be close to 1, meaning that the model behaves as a "near Markov" one and would improve from considering the conditional dependency structure. Such insights were not well captured in many multimodal algorithms, and our results essentially provide the optimal characterization of the combing coefficient among different dependency structures adjusted by the sample size and the fitness measure.
Additionally, the established expression of a can be interpreted as the optimal varianee trade-off (Duda et al.) [173) of the low-dimensional structure to the estimation. Note that the bias- -ariinne trade-off in testing loss (4) is tuned by the coefficient
(Ca² + Va² + 2Ca + | - + a² (2x P ) (a) n
(10)
variance termis)
bias term
The variance terms will vanish with the increase in the training sample sizes. And the bias term characterizes the cost of utilizing dependency structures. Then, the coefficient a achieves the optimal bias-variance trade-off when the testing loss is minimized.
'Due to the consideration of the limited number of samples and the assumption on the distribution of label Y, it will not be strictly 1.
4

---

Under review as a conference paper at ICLR 2023
3 ALGORITHM WITH AUTONOMOUS UPDATED COEFFICIENT ON DEPENDENCY STRUCTURE
In this section, we build an algorithm with an autonomous updated coefficient on the dependency structures (auto-CODES)
realization of our theoretical framework multimodal learning. Specifically, we first extend our theory from a discrete data domain to a continuous one that can be applied to practical datasets using representations in factorization form. Then, we give the optimal coefficient a expressed by multimodal features and the objective loss function which linearly combines components corresponding to different dependency structures. Finally, we give the proposed auto- Cuto-CCOES algorithm and the discrimination rule using maximum a posterior (MAP).
Throughout this section, we consider a dataset with k modalities XI...., XR. n training samples (of") ..... x2 ),) i 1..... n, and m labels. The conditional dependency structure considered here is that all the modalities independent of each other once the label is given.
3.1 MULTIMODAL REPRESENTATIONS IN FACTORIZATION FORM
To utilize the previous analyzing framework, we introduce a parameterized representation for model- ing the density function of the continuous data. The parameterized model is based on two parts: an early fusion model and an embedding layer which can help us reduce the parameters for classification finite. An early fusion model can be described as the following. First, k modalities will go through different deep neural networks and output & features. Then, they will be concatenated and fully connected with dddimensional output layer to learn a joint representation f. The embedding layer is the topmost layer for linear classifications, with weights corresponding to label is given by ( = !9 (v)....r .............. (y))". For a specific task, the weights the topmost layer with a finite number of parameters, i.e.
................................ g(1),....g()))) can be effectively trained with the joint representation f.
Our framework considers an inference model p(f.a) "|X which is widely used in natural language processing (Levy & Goldberg 2014) and image recognition (Xu & Huang 2020), in the following factorization form
p(f-g) X, (ylri, rK) A (r + (f (#1, mx) (v)))-
(11)
The optimal weights gü and gi, which make the model p(fa)t) Y| fit the training samples, min-
p(t imize the distance between empirical distributions and the estimation Px x. Y X spectively. They can be given as: gā A arg migningo X (PxI....r, Px...., p(f.g0) Y|X
re-
arg ming, x r Pxx. f(f..g) xxx where the fitness is measured by the x diivergeece and the reference distribution R. A Px This allows to apply the previous analyses and y!!i....." Analogous to the linearly combined estimator we consider the linear combination of these inference models
focus on the inference model p(f=a.)
A (1 a) P(f ((1.9))
YIX....x aa a(f.i)
=B(f.9") YIXI...x -
Qrix
(12)
with g* (1 a)gi + agg.
Further, we define the testing loss and the corresponding optimal coefficient a* as
testt (a) A E X (Px,..XY,,P,, (fa) ( a - arg min Lest
((fe)
(13)
aE/o.1]
We have the following characterization.
$ Note that it can be negative in real applications. But we can also use it to make discriminative decisions through maximum a posterior (MAP) rule.
"Noe that when the discriminative model B(f.)) X,...x, fixed, Px...x, X, p(f.g) X, the optimal approxi- mation of the true distribution Px.... X.Y.
5

---

Under review as a conference paper at ICLR 2023
Theorem 5. For given dataset, the optimal a² for testing loss equation 13 can be given as L
a²
where
T C"
Vu
E E
F(.'....,,<?))) flaf.....,,1
Pr(
yev
2) 2iuwnty 2i
p(a)) Xux (2 Fk.8) Px Xxx (el 2tto ()))
A₂ A Σ Px (21... , Ix)f (#1....,,,,) on) (
(f1,...,
Terms C", V and the calculation approach of those terms from the training data are given in the supplementary material.
These terms can be represented by some expectations of features f and and are approximated by the corresponding empirical means. For instance, A can be computed from the data by A f- 1 D f(z) (6) x o (x (1))
3.2 OUR PROPOSED ALGORITHM
First, based on linear estimator equation 12 the objective function can be chosen as the linear combination of two referenced x dd-distaness measuring the gap between the learned distribution with distributions corresponding to different dependency structures, i.e.,
(1 (1-a)xñ (Px Px,
p(f.9) Y|X + ax2 (00) X,y, Xx Px,
p(f-a) yIx:...x.)
(14)
Y|X,
Based on the approach established in Waag et al. 2019 Huang et al. 2017). the objective equation 14 can then be transformed to the following loss function which can be computed by the feature (f.g). g.)
((0) (f.g) (1 Cae ( + (
(15)
(f.p( 1
TT((
(0)
( ()) (r(cov(f) cov(g))
(16)
t=1
(f Σ Pj)
(i5)
1 (1) gg(i) tr(cov(f,) cov(g)) tr(cov(f))cov(())
B2
(17)
(i) (2(0)) g(i) is the where f(x) ....................... (1) is the feature output for i-th sample (a Ir..
embedding for label i, cov(f) - f( ...., 2 F ,0.,, (()) cov(g)
(t)
- - (())))))))))))) (z)()), (J) E,= (2 = - 1, 1...., ................................ m. As for loss Cley (f.g) needs a permutation on samples' modalities within the subset of the same label. We denote the subset of training samples with label j E {1....,k)
k} as Dj = ((x((h)) ........ =x )1 where dy is the number of samples whose label is j in the overall dataset D. It (ij) is chosen from {x, (i.5)
(a(( 1d jint== (1.i)) ( (T(2(()) ( (t.3)).
1,...,k, k, and n, I d., cov(f,) + - Em,, f(z (f.5) (.....((
Then, our algorithm can be organized
iteration of two main optimizations: (i) the optimization of a for given (f. g) by minimizing the testing loss Creat (a) equation 13 and use the expression in Theorem 5 for computation; (ii) the optimization of features (f. g) for given a to minimize the training loss equation 14 by the deep neural network. We summarize it in Algorithm [
With the output features f and g" trained by the algorithm, the classification of newly observed sample (x1) is given by the maximum posterior (MAP) decision rule
(Z11..... 2x) arg max Py|x (y/21,...., 2k)
yEy
arg max Py (1) (f" xx) (v))).
wEy

---

Under review as a conference paper at ICLR 2023
Algorithm 1 An Auto-updated Coefficient on Dependency Structures (auto-CODES) Algorithm
Input:
multimodal data samples {(x) ,...,, Initialize a² = 0
222 222 )))
repeat
(f",g") + arg min
f,g
crain (f.g)
E(0')
a' + arg mejo0
Lest
Ei 9
(a)
until a² converges
(f', .9") t E (f.g)
return f ."""
EXPERIMENTS
In this section, we verify our model and algorithm to answer the following research questions (RQ):
RQI: Can auto-CODES make discrimination well?
RQ2: Can auto-CODES automatically determine an appropriate coefficient a?
RQ3: Is the optimal coefficient o scale the inverse proportion of the training sample size?
Experimental settings. In our experiments, two widely-used multimodal emotion recognition datasets used, MELD (Poria et al. 2018) and IEMOCAP (Busso et al.) (008). MELD contains 13K utterances from 1433 dialogues from the TV series Friends. Each utterance is annotated with three emotion labels, positive, neutral, and negative. We use the audio and textual modalities for our verification. As for IEMOCAP, it contains approximately 12 hours of audiovisual data with six emotion categories: anger, happiness, sadness, neutral, excitement, and frustration. We visual and audio modalities for our verification. In our settings, the sample size plays crucial role. Thus, we randomly select subsets of both datasets with certain levels of sample sizes as our training sets. To preserve the inner structure within dialogues, the random selection is towards the dialogues. The exact sample sizes are listed in Table [ As for the model structure, we use DialogueRNN (Majumder et al.) 2019) as our backbone network for extracting multi-modal features and adopt 2-layer Multi-layer Perceptron (MLP) with RELU (Glorot et al., (011)) activation for extracting one-hot label features. We use accuracy and F1 -core our evaluation metric.
4.1 EMOTION RECOGNITION RESULTS
To answer RQI, we compare auto-CODES with the following methods: (i) CE: Cross entropy loss that is widely used in machine learning classification tasks, (ii) MaskedNLL: a variant of NLL loss to cope with excluding logit value of the padded sample, which is used in (Majumder et al. 2019) 9). (iii) Focal loss(Mukhoti et al. 2020): Focal loss is designed to address the issue of the class imbalance problem, and (iv) Soft-HGR (Wang et al. 2019): Soft-HGR loss learns correlated representation across modalities without hard whitening constraints. All the experiments are conducted on various training sample sizes. The discrimination accuracies along with the Fl-score on emotion recognition tasks are reported and shown in Table and Table
The results demonstrate that our auto-updated method achieves superior performance against existing methods among all settings with different sample sizes. From the dialogue size 10 40, auto- CODES outperforms the second-best method Soft-HGR by the margin of 2.086% (size 20) to 2.63% (size 30) on the Fl Fi-score metric. For the dialogue sizes 40 and 60, auto-CODES achieves absolute improvements by margins of 1.629% and 1.515% over Focal loss. Also, examine the impact of the coefficient a, we conduct experiments with prefixed static a as a comparison with our auto-updated algorithm. Comparing the last two columns in Table D can observe that auto-CODES obtains 1.856% (size 60) to 4.456% (size 10) improvements in terms F1-score over static CODES. These results suggest that: (1) when the training samples outperforms focal loss, Soft- HGR, Cross Entropy loss, and MaskedNIl loss, (2) auto-updating a can improve the results static CODES by large margins.
insufficient, proposed auto-CODES
7

---

Under review as a conference paper at ICLR 2023
Table I: Comparison with other objectives in MELD dataset with different training sample sizes. All reported results are averaged over 10 repeated experiments.
Sample size
Method
Metrie
(Dialogue size)
CE
MaskedNNI
Focal
Soft-HGR
CODES
auto-CODES
accuracy
50.22242 709
50.544+1.380
52.784+2.143
51.74143.221
53.640s
107 (10)
F1-score
38.099+2.290 296
40.147+2.269
19649#11999
44.998+2.227
42.751
1..206=:11179
198 (20)
accuracy
51.218:1:312
51.571=1.255
51.22611.477
53.890&1
53.09112.404
55.179:1.147
F1-score
41.840++
42.912+1.611
42,535+2, 146
47.038+ .957
45.786+ 1.210
49.124=0.902
accuracy
51.272+114667
52.084=0.990
52.655#0.421
54.736 1,576
53.986#1.44
05.943=0.925
302 (30)
Fl Fl-score
42,091
43.351
43.58911.986
48.3584 1.....1.700
46.81612.313
50.98820.712
413 (40)
accurcc
52.475a00.979
52.943&1.159
53.47140.534
55.441 1.362
.62
1..219
57.81620.964
Fl Foooe
43.647=0.764
44.341=1.165
46.135=1. 146
01211++0.0 .27
47.942+1.252
51.538:0.699
accuracy
54.176::1.08
53.58621.000 .042
55.613+0.66 665
55,901=0.937
14.730=1.467
58.467=0.783
(50)
F1 seore
47.397%41.100
46.537=0.934
50.509 700
50.10740.776
18.52.
52.13880.872
(60)
accuracy
55.44810.707
55,44120,941
56.57610.479
56.467+0.792
56.087+1.280
59,847=0.781
F1-score
50.54710.8766
50.263a1.231
51.13820.944
50.981+0.811
50.747.0.812
52.653:0.584
Table 2: The optimal coefficients a* derived by auto-CODES and grid search method on different training sample sizes the MELD dataset.
Sample size n (dialogue size)
grid search (gs)
auto-CODES (auto)
aga
accuracy
Olanto
accuracy
n Ogauto
107 (10)
0.02
52.912
1.164
0.0189 = 0.0011
53.640 1.345
3.74
198 (20)
0,02
55.144 # 1.470
0.0179 # 0.0009
55.179 1.147
3,54
302 (30)
0.01
55.949
1.342
0.0130 0.0016
55.943 0.925
3,93
413 (40)
0.01
57.486 = 1.292
0.0092 # 0.0007
57.816 0.964
3.81
516 (50)
0.01
58.372
0.964
0.0078 0.0008
58.467 0.783
4.02
636 (60)
0.01
59.573 0.745
0.0058 0.0006
59.848 0.781
3.69
4.2 DETERMINATION OF THE OPTIMAL CORFFICIENT
To answer RQ2, we conduct experiments compare our auto-updated coefficient a and the coefficient determined by grid search and manual adjustments conventionally. Instead of only relying on empirical experience with no theoretical assurance, our theory determines the optimal coefficient and updates the learned modalities' features automatically.
During training, our algorithm and grid search method use the same number of training epochs for each a updating iteration circle at different training tuple sizes. In the grid search method, we search 101 values of a ranging from 0 to with step length of 0.01. In our algorithm, we stop the iteration once the difference ratio of the updating alpha is smaller than 0.1. The results are shown in Table 2 suggests that our method can locate an appropriate and even better a than the grid search method.
4.3 OPTIMAL COEFFICIENT WITH THE NUMBER OF TRAINING SAMPLES
To answer RQ3, we examine the optimal a determined by CODES on MELD and IEMOCAP. According to our theory, a" is roughly of the order For different training sample sizes, we conduct 10 repetitive experiments using auto-CODES to locate the optimal coefficient and report their average as the final a". The values of their product n a* for both MELD dataset are demonstrated in Table 2 can be approximately recognized as a fixed number around 3.8. The results for IEMOCAP has given in the appendix. According the column "n Gauto" of Table our theoretical outcome can not only determine a good a but also reveal its relation with the number of training sample sizes. For more experiment results the JEMOCAP dataset, please see Supplementary Material.
8

---

Under review as a conference paper at ICLR 2023
Table 3: Comparison with other objectives in the IEMOCAP dataset with different training sample sizes, All reported results are averaged over 10 repeated experiments.
Sample size
Method
Metric
(Dillogue size)
CE
MaskedNII
Focal
Sof--HGR
CODES..0.01
auto-CODES
446 (10)
uccurncy
42,580 1.791
41.956 1.645
42.890
1,783
42.466
1,980
42.784
1.465
43,023 1.706
Fl-score
36.710 +
35.518 1.480
37.302
1.592
36.398
37,010 1.573
37.415
1.292
accurncy
43,217 + 1,877
44,789 + 1.669
44,962
1,374
44.511
1.655
44.754
1.221
45,478
1.236
(12)
F1-score
37.399 1.642
40.726
0.878
40.519
1,587
39.507
1.132
40.751
1,383
40,910 1.085
accuracy
45.116 1.054
45.533 # 1.577
45,599 = 1,568
44,969
1,513
45.730 = 1.420
46.2.3 1.128
647
F1 score
40.714 1.199
41.227 1.114
41.376
1.247
40.397
1.022
41.786
1.209
42.803 0.976
5 RELATED WORK
In this section, we summarize the related works from two aspects: correlation analysis and the dependency structures in multimodal learning.
Correlation Analyses in Multimodal Learning. Correlation analysis methods can be fusion-based or objective-based. Fused methods design the fusion approaches of different modalities representations, such as tensor fusion network for multimodal sentiment analysis (Zadeh et al. 2017). and factorized multimodal representations (Liu al., 2018 T'ai al.] 2018). Objective- based methods capture modality interactions by using distinct statistical notions. For example, canonical correlation analysis (CCA) approaches like (Karami & Schuurmans 2021) are based Pearson correlation. Trisen-Shannon-Diveeee
(Sutter et al. (020)) Variation of Information (Sohn et al.
2014), and HGR maximal correlation (Wang et al. 2019 Ma et al. 2020 TTogg et al., 2021) have also shown that the statistical design of learning objectives can facilitate the correlation extraction among modalities. The concept of sample efficiency has been discussed in the weighted algorithm TAWT (Chen et al. [202T). The explicit and accurate quantity of training samples for analysis is not appropriately addressed.
Learning Dependency Structures in Multimodal. Learning features in complex manifolds formed by different modalities of data is essential. One approach is to learn the discriminative one such as conditional random fields (CRF) (Lafferty et al. 2001). Another strategy involves learning the generative model using multimodal deep Boltzmann machines (DBMs) (Salakhutdinov & Hinton 2009), or coupled, factorial and multi- hidden Markov models (HMM) method (Nefian et al. 2002 Ghahramani & Jordan 1997] Gurban et al. 2008). However, the theoretical characterization of utilizing the dependency structures is not sufficiently explored.
DISCUSSION
In this paper, we propose a new theoretical framework analytically characterize the explicit and exact relation between the sample size with conditional dependency structures in multimodal learning in non-asymptotic regime. Moreover, we propose a weighted training algorithm, auto-CODES, based on the theoretical framework. It can iteratively update the coefficient on different dependency structures based on the evolving modalities' features. The effectiveness of auto CODEE further corroborated through multimodal emotion recognition experiments on MELD and IEMOCAP datasets with promising results.
Limitations and Future Work. There are two main limitations in our work. First, we proposed a tractable algorithm for one specific type of conditional dependency structure, but we left the generalized tractable approach for all different kinds of dependency structures for future work. Second, even though we have specified the precise shape of the coefficient that features can compute, it is still laborious specify its role in the algorithm: therefore, we intend to improve the computing method for the coefficient in the future. We also intend to integrate our framework with pre-train networks and run further experiments various modalities across various datasets, including MOSEI.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Tadas Baltrušaitis, Chaitanya Ahuja, and Louis- Phiil--iilpppe Morency. Multimodal machine learning: A survey and taxonomy. IEEE transactions on pattern analysis and machine intelligence, 41(2):
423 443, 2018.
Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jean- nette N Chang, Sungbok Lee, and Shrikanth Narayanan. lemocap: Interactive emotional dyadic motion capture database. Language resources and evaluation, 42(4):335-33) -393359,99111 2008.
Shuxiao Chen, Koby Crammer, Hangfeng He, Dan Roth, and Weijie Su. Weighted training for cross-task learning, arXiv preprint TrXiv:2105.14095, 2021.
Richard O Duda, Peter E Hart, and David G Stork. Pattern classification and scene analysis, volume 3. Wiley New York, 1973.
Jing Gao, Peng Li, Zhikui Chen, and Jianing Zhang. A survey on deep learning for multimodal data fusion. Neural Computation, 32(5):829-864) 2020.
Zoubin Ghahramani and Michael Jordan. Factorial hidden markov models. Machine learning, 29 (2):245-22) 273,, 1997.
Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier neural networks. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp. 315-323. JMLR Workshop and Conference Proceedings, 2011.
Mihai Gurban, -han-Philippp Thiran, Thomas Drugman, and Thierry Dutoit. Dynamic modality weighting for multi-strean hmms candioovvisua speech recognition. In Proceedings of the 10th international conference on Multimodal interfaces, pp. 237-240, 2008
Shao-Lun Huang, Anuran Makur, Lizhong Zheng, and Gregory W Wornell. An enformation-theoree approach universal feature selection in high-dimensional inference. In 2017 IEEE International Symposium on Information Thhoory (ISIT), pp. 1336-1340. IEEE, 2017.
Mahdi Karami and Dale Schuurmans. Deep probabilistic canonical correlation analysis. In Proceed- ings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 8055-8063, 2021.
Douwe Kiela, Hamed Firooz, Aravind Mohan, Vedanuj Goswami, Amanpreet Singh, Pratik Ringshia, and Davide Testuggine. The hateful memes challenge: Detecting hate speech in multimodal memes. arXiv preprint 20Xi::2000.04790. 2020.
John Lafferty, Andrew McCallum, and Fernando CN Pereira. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. 2001.
Omer Levy and Yoav Goldberg. Neural word embedding as implicit matrix factorization. Advances in neural information processing systems, 27, 2014.
Zhun Liu, Ying Shen, Varun Bharadhwaj Lakshminarasimhan, Paul Pu Liang, Amir Zadeh, and Louis-Philippe Morency. Efficient low-rank multimodal fusion with modality-specific factors. arXiv preprint arXiv: arXiv:1806.000 (a0::8806000000 2018
Fei Ma, Wei Zhang, Yang Li, Shao-Lun Huang, and Lin Zhang. Learning better representations for audio-visual emotion recognition with common information. Applied Sciences, 10(20):7239, 2020.
Navonil Majumder, Soujanya Poria, Devamanyu Hazarika, Rada Mihalcea, Alexander Gelbukh, and Erik Cambria. Dialoguernn: An attentive rnn for emotion detection in conversations. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pp. 6818-6888 2019.
Jishnu Mukhoti, Viveka Kulharia, Amartya Sanyal, Stuart Golodetz, Philip Torr, and Puneet Dokania. Calibrating deep neural networks using focal loss. Advances in Neural Information Processing Systems, 01111588--52990 2020.
Arsha Nagrani, Shan Yang, Anurag Amab, Aren Jansen, Cordelia Schmid, and Chen Sun. Attention bottlenecks for multimodal fusion. Advances in Neural Information Processing Systems, 34, 2021.
10

---

Under review as a conference paper at ICLR 2023
Ara V Nefian, Luhong Liang, Xiaobo Pi, Liu Xiaoxiang, Crusoe Mao, and Kevin Murphy. A coupled hmm for audio-visual speech recognition. In 2002 IEEE International Conference on Acoustics, Speech, and Signal Processing, volume 2. pp. 11-3013. IEEE, 2002.
Jiquan Ngiam, Aditya Khosla, Mingyu Kim, Juhan Nam, Honglak Lee, and Andrew Y Ng. Multi- modal deep learning. In ICML, 2011.
AJ Piergiovanni, Anelia Angelova, and Michael S Ryoo. Evolving losses for unsupervised video representation learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 133-142, 2020.
Soujanya Poria, Devamanyu Hazarika, Navonil Majumder, Gautam Naik, Erik Cambria, and Rada Mihalcea. Meld: A multimodal multi-party dataset for emotion recognition in conversations. arXiv preprint 0rXiv:/810.025500 2018.
Ruslan Salakhutdinov and Geoffrey Hinton. Deep boltzmann machines. In Artificial intelligence and statistics, pp. 448-455, PMLR, 2009.
Kihyuk Sohn, Wenling Shang, and Honglak Lee. Improved multimodal deep learning with variation of information. Advances neurul information processing systems, 27:2141-2149, 2014.
Jabeen Summaira, Xi Li, Amin Muhammad Shoib, Songyuan Li, and Jabbar Abdul. Recent advances and trends in multimodal deep learning: A review. arXiv preprint arXiv:2105.11097, 2021.
Thomas M Sutter, Imant Daunhawer, and Julia Vogt. Multimodal generative learning utilizing jensen-shammon--diverenee arXiv preprint arXiv:2006.08242, 2020.
Xinyi Tong, Xiangxiang Xu, Shao-Lun Huang, and Lizhong Zheng. A mathematical framework for quantifying transferability in multi-source transfer learning. Advances in Neural Information Processing Systems, 34, 2021.
Yao-Hung Hubert Tsai, Paul Pu Liang, Amir Zadeh, Louis-Philippe Morency, and Ruslan Salakhutdi- nov. Learning factorized multimodal representations. arXiv preprint arXiv: 1806.06/ 76, 2018.
Paroma Varma, Frederic Sala, Ann He, Alexander Ratner, and Christopher Ré. Learning dependency structures for weak supervision models. International Conference on Machine Learning, pp. 6418-6427. PMLR, 2019.
Lichen Wang, Jiaxiang Wu, Shao-Lun Huang, Lizhong Zheng, Xiangxiang Xu, Lin Zhang, and Junzhou Huang. An efficient approach to informative feature extraction from multimodal data. In Proceedings of the AAAI Conference Artificial Intelligence, volume 33, pp. 5281-5288, 2019.
Xiangxiang Xu and Shao-Lun Huang. Maximal correlation regression. IEEE Access, 8:26591-2660 2020.
Amir Zadeh, Minghai Chen, Soujanya Poria, Erik Cambria, and Phuis-Philippe Morency. Tensor fusion network for multimodal sentiment analysis. arXiv preprint arXiv:1707.072550 2017.
Amir Zadeh, Michael Chan, Paul Pu Liang, Edmund Tong, and Louis Chiis-Philippe Morency. Social-iq: A question answering benchmark for artificial social intelligence. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8807-8817, 2019.


Reviews:
[{"review_id": "uAs6JYRycG", "reviewer": "Reviewer_c4xb", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studied the dependency structure between modalities in multimodal learning. Specifically, it intends to understand the interplay between training samples and various dependency structures. A novel theoretical framework is proposed to characterize the training sample size toward different dependency structures.\n", "summary_of_the_review": "The paper needs further improvements to justify the motivation and clearly specify its theoretical and technical contributions.", "strength_and_weaknesses": "Strength :\n+ A theoretical framework is proposed to characterize the optimal combination coefficients between the empirical joint distribution and Markov-structured distribution.\n\n+ The experimental designs are clear and straightforward, with solid results to verify the proposed algorithm. \n\n \nWeaknesses:\n- The proposed research is not well motivated. The introduction section is hard to follow in general. For instance, the introduction starts with the issue of insufficient training samples in multimodal learning. It is unclear how this can logically motivate the proposed optimal combination coefficients study.\n\n- Missing key background information. Eq 2 defines the combination coefficients alpha. However, it is completely unclear why the two distributions are selected and why they can be linearly combined using alpha. This makes the problem definition ungrounded and hard to follow.\n\n- Unclear key technical details. When calculating referenced x^2-divergence, how to obtain the ground-truth distribution P?\n\n- Experimental issues. All experiments are limited to two modalities. It is unclear if the proposed algorithm can handle more than two modalities. \n(Table 2) For the first row (sample size 107), please check the result of  “n・\\alpha.”\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "pwSNJ0umV-", "reviewer": "Reviewer_MdJu", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a mathematical framework to characterize the conditional dependency structures for multimodal learning in an analytic way. Concretely, the authors consider three factors affecting the multimodal classification performance including the number of labeled training samples, the fitness of the conditional dependency structures to represent the true joint distribution, and the complexity of the discrimination task, where the $\\chi ^2-divergence$ is utilized to estimate the fitness of the conditional dependency structures. \nThe authors consider a model to learn the linear combination of the general dependency structure and conditional dependency structure estimators and propose an explicit analytical solution to get the optimal combination coefficient.\nThe authors not only show some theoretical analysis for designing the efficient multimodal algorithm but also provide experiment results to show the effectiveness of this algorithm.", "summary_of_the_review": "This paper proposes a novel mathematical framework to characterize the conditional dependency structures for multimodal learning in an analytic way and provides experiments to prove the theory and algorithm. However, it still needs more solid results to prove the results.", "strength_and_weaknesses": "Strength:\n+ This paper writes well and explains the formulations and analytical results clearly.\n+ The authors first analyze the discrete scenarios and conclude some theoretical understanding for model designing with the formulation of optimal combination coefficient, then extend to the continuous scenarios to propose an autonomous updated coefficient on dependency structures algorithm by exploiting parametric models.\n+ The experiments demonstrate the effectiveness of this algorithm.\n\nWeaknesses:\n- This work suggests models learn the linear combination of general dependency structure and conditional dependency structure estimators, which is a strong assumption, especially in multimodal scenarios, for example, in audio-visual scenarios, whatever the label is, it usually contains high temporal consistency between vision and sound.\n- The experiments only provide small-sale results, it cannot prove the conclusion useful for the large-scale dataset.\n- This paper point out three factors affecting the multimodal classification performance, but the results only demonstrate the results with different training samples sizes, however, in different datasets, the intrinsic sample structures are various, and the estimated $\\chi ^2-divergence$ in different datasets should also be provided to prove the algorithm.\n- The coefficient is estimated with training samples, however, the name of loss to estimate is $L_{test}$, which may be ambiguous.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "kX_qmFSLPl", "reviewer": "Reviewer_kPWv", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a theoretical framework to study the relationship between sample size with conditional dependency structures in multimodal learning. The method's effectiveness is validated on two emotion recognition experiments. ", "summary_of_the_review": "Overall, I think the paper did a great work on the theoretical analysis. However, the weakness is the paper's readability.", "strength_and_weaknesses": "Strengths: The authors did a very comprehensive theoretical analysis, and the proof looks sound.\n\nWeaknesses: The paper is not easy to follow and read. \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "Wtk9UL88Vk", "reviewer": "Reviewer_pfqE", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on multi-modal learning problem, and proposes a mathematical framework to characterize so-called dependency structures.", "summary_of_the_review": "The writing of this paper is very poor, which let me not able to understand the core idea of this paper.", "strength_and_weaknesses": "1.  The writing of this paper is very poor. There are too many typos:\n- 'Learning' is missing in the title;  \n- 'from different categories of data, such as images, audios, and texts' 'categories' should be substituted to 'types';\n- 'There have been many kinds of literature addressing this issue with different kinds of algorithms', but only one reference is cited here;\n- 'training sample size' $\\rightarrow$ 'training dataset size'\n2. Some concepts are not clear:\n- 'design feature' and 'extract feature' are both mentioned in the introduction, but these are two different concepts;\n- ' exploiting appropriate dependency structures' and 'representing the general dependency structure' are both mentioned in the introduction. So the dependency structure can be straightly adopted or need to learn and capture?\n3.  No enough experiments to support the proposed method.\n \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject