Modality Complementariness: Towards Understanding Multi-modal Robustness
Authors: Siting Li
Chenzhuang Du
Yu Huang
Longbo Huang
Hang Zhao
Venue: Submitted to ICLR 2023 (2023)
Keywords: multimodal robustness theory

Abstract: Along with the success of multi-modal learning, the robustness of multi-modal learning is receiving attention due to real-world safety concerns. Multi-modal models are anticipated to be more robust due to the possible redundancy between modalities. However, some empirical results have offered contradictory conclusions. In this paper, we point out an essential factor that causes this discrepancy: The difference in the amount of modality-wise complementary information. We provide an information-theoretical analysis of how the modality complementariness affects the multi-modal robustness. Based on the analysis, we design a metric for quantifying how complementary the modalities are to others and propose an effective pipeline to calculate our metric. Experiments on carefully-designed synthetic data verify our theory. Further, we apply our metric to real-world multi-modal datasets and reveal their property. To our best knowledge, we are the first to identify modality complementariness as an important factor affecting multi-modal robustness.

Under review as a conference paper at ICLR 2023
MODALITY COMPLEMENTARITY: TOWARDS UNDER- STANDING BULTI-MODAL ROBUSTNESS
Anonymous authors
Paper under double-blind review
ABSTRACT
Along with the success of multi-modal learning, the robustness of multi-modal learning is receiving attention due to real-world safery concerns. Multi-modal models are anticipated to be more robust due to the possible redundancy between modalities. However, some empirical results have offered contradictory conclu- sions. In this paper, point out essential factor that causes this discrepancy: The difference in the amount of modality wiie complementary information. We provide cntemmation-thhoretiin affects the multi-modal robustness. Based on the analysis, we design metric for quantifying how complementary the modalities are others and propose effec- tive pipeline to calculate our metric. Experiments on carefully-designee synthetic data verify our theory. Further, we apply our metric real-world multi-modas datasets and reveal their property. To our best knowledge, we are the first to identify modality complementarity as an important factor affecting multi-modal robustness.
analysis of how the modality complementarity
INTRODUCTION
Recently, deep neural networks have proved successful in various areas, such as image recognition (He al., 2015; Krizhevsky et al., 2012), speech recognition (Chorowski et al., 2015) and neural machine translation (Wu et al., 2016). The revolution is also happening in multi-modal research, e.g. RGB-D semantic segmentation (Wang al., 2016), audio-visual learning (Zhao et al., 2018), and visual question answering (Antol et al., 2015). Intuitively, multi-modal models are anticipated to be more robust due to the potential redundancy between modalities. When one of the modalities is corrupted, others can compensate for the loss. This intuition is supported by both psychological studies of the human perception system (Sumby & Pollack, 1954) and deep learning practices (Zhang et al., 2019b; Qian et al., 2021; Wang et al., 2020).
However, some recent studies cast doubt on this belief. From a theoretical perspective, the multi- modal models usually have larger input dimension than uni-modal models, and the increase of input dimensions significantly degrades model robustness (Ford et al., 2019; Simon-Gabriel et al., 2019). From empirical view, some experiments suggest that multi-modal integration may be more vulnerable to attacks or corruptions than uni-modal models (Yu et al., 2020; Tian & Xu, 2021; Ma al., 2022).
What causes this contradiction in multi-modal robustness? We notice that the conclusions above drawn under assorted multi-modal task settings ranging from action classification to question answering, which vary in the presence and type of modality interconnections (Liang et al., 2022). Therefore, question arises naturally:
What aspects of modality interconnection affect the multi-modal robustness?
We hypothesize that the complementarity of modalities plays an essential role. If the complementary part of each modality is negligible, the corruption of one modality would not severely damage the model performance. Otherwise, the multi-modal model could perform even worse than uni-modal model. For the visual question answering task, the two modalities are highly complementary: Only perceiving either the question or the image could not lead to ideal answer (Agrawal et al., 2018). For the action classification task, the RGB and optical flow are less complementary since each of them can suggest roughly correct answer (Feichtenhofer et al., 2016b).

---

Under review as a conference paper at ICLR 2023
To validate the above hypothesis, we first demonstrate the key role of modality complementarity to model robustness through theoretical analysis. Following previous work (Tsai et al., 2020; Sun et al., an information ifformation-thcoretiic framework
2020; Sridharan & Kakade, 2008; Tosh et al., 2021), for multi-modal learning and study how the complementary information affects robustness under missing and noisy modality settings. Based on the analysis, we design novel metric and a practical calculation pipeline built on Mutual Information Neural Estimator (MINE) (Belghazi et al., 2018) to quantify the complementarity of modalities in multi-modal datasets.
With the specially designed metric and pipeline on hand, we verify our theory and the effectiveness of our proposed metric on synthetic data and carefully-designed toy dataset AAV-MNIST. The results are consistent with the model robustness in modality missing, noisy modality. and adversarial attack settings on the datasets we test on. Then we apply our metric to real-world multi-modal datasets to further investigate the modality complementarity in different settings. To our best knowledge we are the first to identify and prove the important role of modality complementarity in multi- modal robustness. Hence, for future research, we recommend that researchers consider the modality complementarity as a control variable for a fairer comparison multi-modal robustness.
The main contributions are highlighted as follows:
We point out the effect of modality complementarity on multi-modal model robustness analysis.
through information-theoree
We propose a dlatasetwisst metric to qualitatively evaluate how complementary the modalities are in each multi-moda dataset, and also design a pipeline for computing the metric in real-world datasets.
We create a synthetic dataset and a toy dataset (AAV-MNIST) to test our metric and pipeline. These datasets cover various complementary situations of different modalities and are used to verify the effectiveness of our pipeline.
We further reveal the modality complementarity and its relationship with model robustness in real-world multi-modal datasets, which could lead to less biased comparison for multi- modal robustness.
RELATED WORK
Multi-modal learning. Various multi-modal learning tasks and models are proposed in recent years (Baltrusaitis et al., 2017; Liang et al., 2021), such as multi-modal reasoning (Yi et al., 2019; Johnson et al., 2016), cross-modal retrieval (Gu et al., 2017; Radford et al., 2021), and cross-modal translation (Ramesh et al., 2021). Among these settings, we mainly focus on the supervised multi- modal classification setting. The theoretical understanding of multi-modaa learning is relatively under-explored, with (Huang et al., 2021) deriving generalization error bounds and (Sun et al., 2020) comparing with the Bayesian posterior classifiers. multi-view learning (Xu et al., 2013). The theory of multi-vien learning has long been studied both theoretically (Zhang et al., 2019a; Tosh et al., 2021) 1) and empirically (Sindhwani et al., 2005; Ding al., 2021; Amini et al., 2009; Tian al., 2019). Earlier work (Kakade & Foster, 2007; Sridharan & Kakade, 2008) proposes the multi-view assumption: Each modality suffices to predict the label. Recently, many multi-view analyses adopted this assumption (Han et al., 2021; Tsai et al., 2020; Lin et al., 2021; Federici et al., 2020; Lin et al., 2022). However, as pointed out by (Huang et al., 2021; 2022), this might not hold in the multi-modal learning setting.
concept close to multi-modal learning is the
Model robustness. Model robustness under data missing (Ramoni & Sebastiani, 2001), random corruption (Hendrycks & Dietterich, 2019), and adversarial attacks (Madry et al., 2017) constantly been concerned in consideration of real-world safety issues. For uni-modal models, several methods are proposed to strengthen model robustness (Papernot al., 2015; Huang et al., 2015; Meng & Chen, 2017). For multi-modal models, some papers regard the use of multi-modality as a way to improve robustness (Zhang et al., 2019b; Qian et al., 2021; Wang et al., 2020), while others continue to improve multi-modal models' robustness by designing new network architectures and fusion methods (Kim & Ghosh, 2019a; Tsai et al., 2018; Yang et al., 2021) and training routines (Eitel et al., 2015; Ma et al., 2021). When dealing with known missing patterns, researchers explore additional ways: data imputation through available modalities or views (Tran et al., 2017; Lin et al., 2021) ), or training different models for different availability of modalities (Yuan et al., 2012). Our analysis
2

---

Under review as a conference paper at ICLR 2023
Info
Info Loss
Multimodal
Missing Modality
Noisy Modality
Y
Figure Illustration of relationships between the inputs and the target of multi-modal task in different cases. X and Z are random variables representing the input of two modalities. Y is the target we would like infer. The Info Loss refers the loss Y -reevvant information provided by inputs, which is caused by missing or corruption of modality Z.
points the upper bound of these methods. Apart from improving robustness, another line of work aims to analyze or estimate the robustness of existing uni-modal methods (Cohen et al., 2019; Carlini al., 2019; Mahmood et al., 2021) and multi-modal methods (Yu al., 2020; Tian & Xu, 2021; Ma et al., 2022; Rosenberg et al., 2021; Li et al., 2020), We in this work analyze one factor of multi-modal model robustness both theoretically and empirically.
Mutual information in deep learning, Mutual information is tightly related to deep learning through multiple ways, including information bottleneck method (Tishby et al.. 2000), analysis of learning methods (Wu & Verdu, 2012; Tsai et al., 2020; Shwartz-Ziv & Tishby, 2017), and new learning methods based on mutual information (Hjelm al., 2018; Bachman et al., 2019; Sun et al., 2020). On the other hand, learning methods help to estimate the amount of mutual information. Representative work includes Mutual Information Neural Estimator (MINE) (Belghazi et al., 2018), CPC (van den Oord et al., 2018), DIM (Hjelm et al., 2018), and DoE estimator (McAllester & Stratos, 2018). We apply the MINE to our calculation pipeline for its simplicity and effectiveness.
THEORETICAL ANALYSIS
In this section, we first build an iiformation-thhorettiaal framework for multi- learning and show the impact of complementary information to model robustness in modality missing and single noisy source cases, which are commonly studied in previous work (Kim Ghosh, 2019a; Tian & Xu, 2021) and widely encountered in practice, e.g., some sensors are broken, prone to noise (e.g. cameras in foggy environments), or expensive to use(e.g. X-ray data for medical analysis). An illustration of these cases is plotted in Figure 1.
3.1 PRELIMINARIES
Notations. We use (A) to represent the entropy of a random variable A, H () for the conditional entropy given another variable B3, / (A; B) for the mutual information between random variable A and B, [(A; BJC) for the conditional mutual information conditioned on random variable C,, and [(A; B; C') for the interaction information (i.e., mutual information of three variables, possibly negative value).
Multi-modal learning formulation. We adopt the formulation for multi-modal learning problem proposed in (Huang et al., 2021) Denote the M- -moodllity input space as X = X2 x X' x Xmr and the target space as V. Each data point (X,, X2...., XM. Y) sampled from an unknown distribution X' x y. Our goal is that, based on the random input variables X1, X2...., XM from M modalities, we would like to infer the target Y. classification tasks, is discrete random variable, while in regression tasks Y is continuous. For instance, considering mudio-visual action recognition (Gao et al., 2019; Feichtenhofer et al., 2016 let X, be the audio part and X₂ be the frames of a video clip, and want to infer the label Y, i.c. what kind of action is performed in the clip. In the subsequent analysis, we will focus on the common case M (Feichtenhofer et al., 201 6a) for simplicity, and denote the two modalities X E X' and Z Z respectively. Notice
3

---

Under review as a conference paper at ICLR 2023
that our analysis and results can be extended cases with more than two modalities the expense of notations.
Complementary information. Now define the complementary information in the following, which is essential through our theoretical analysis.
Definition 1 (complementary information). For input variables X, Z and the target Y, define the
complementary information provided by X, Z as follows
I x I(X:YIZ)
22 = X)
When the target clear from the context, we omit the Y in the subscript.
Mathematically. [X;; Z)) represents the information in the target Y that is accessible for X but predictable for Z. Thus I characterize the unique label information owned by modality X, and similarly for z- Hence rx together with Zz can determine the complementarity of modality X and Z. Clearly, larger I'x and z imply higher complementary information content.
From the standard derivation in information theory, we can obtain the following relation:
T(X, Y) - Iz Ix;
Previous theoretical analyses of multi-view learning Sridharan & Kakade (2008); Xu et al. (2013); Tosh al. (2021) usually adopt the multi-view assumption that each view redundant terms of predicting the target, i.e. I'x and I are both small. However, this assumption does not always hold the multi-modal learning setting Antol et al. (2015). In the following subsections, we will show how the complementary information I'x and I'z affect the model robustness in missing modality and noise settings. Motivated by this theoretical observation, we will propose metric to evaluate the modality complementarity and pipeline for calculation in Section 4.
Bayes error rate. We introduce the Bayes error rate Fukunaga Hummels (1987) to measure the model performance, which is the lowest possible error for any arbitrary classifier predictor from the multiple modalities to infer the target. Formally, given two modalities X and Z, the multi-modal Bayes errors for classification Pe. and regression Pe, are defined follows:
Pe, = Ex
max P(Y = y|z, 2)] WEY
Pe, = E, The Bayes error rate helps us focus on the interconnection among modalities X, Z and target Y in each multi-modal task and omit other factors' effects model robustness, e.g. dataset size, training routines, and network architectures.
[Vv E|y [r, =|)
3.2 MISSING MODALITY
We first consider the missing modality scenario and assume modality Z is missing w.l.o.g.. Then the Bayes error rates for missing modality, denoted as pMiss and pMiss become
co
Ecc
En Px [1 max P(Y y/x:)
DEY
pMiss
= E..yy Px.y [(v E [x)
Now we establish the following theoretical guarantees to quantify differences between the Bayes rate of multi-modal and missing- modality.
Theorem 3.1. For random variables X, Z and discrete random variable Y, we have log
H( 2) log 2
Pe, < 1 exp(-H((Y)))
H(Y X, Z) + Tz log2 pMss < expeeeep(XHXX)) 2) (22)
(3)
log [Y] For continuous random variable Y, if we further assume that Y takes value in [-1, 1), then we have
pMiss Cr
Pe. < Tz
(4)
4

---

Under review as a conference paper at ICLR 2023
Remark 1. The gap between PMiss (best model performance in modality missing setting) and Peo (best model performance in normal setting) reflects the best model robustness against modality missing. For the classification task, when I' = O, i.c. there is complementary information of 7, the information from can be covered by the information from for predicting Y. In this case, the PMiss shares the same lower and upper bound with P...
- so the performance of the best model would not be affected by modality missing. As the I'z increases, the bounds for pMiss rise, while the bounds for Pe. is fixed, indicating that the best model performance drops under modality missing, i.e. the robustness decays. Considering the extreme case when Z large enough, the lower bound of PMiss is greater than the upper bound of Peg. so the missing modality performance is provably worse than normal performance.
Remark 2. For the regression task, the closer PMiss and Pe. are, the robust the best model is. From the result above, the gap between two Bayes optimal predictors is bounded above by the complementary information, hence increased by Izz So the model robustness under modality missing is worsened along with the increase of z-
3.3 SINGLE NOISY MODALITY
The modality corrupted by noise is another situation that we often encounter in practice, e.g., the foggy weather results in noisy RGB images in autonomous driving. In this section, we study the case that one of the modalities has additional noise, which can be easily extended to the case that all modalities are noisy at the expense of notations. Formally, we consider that Gaussian noise N is added to the input modality Z (Zheng et al., 2016; Kim & Ghosh, 2019b). We use Ry (Z) = 7 + N to denote the modality Z after adding Gaussian noise. By (Cover, 1999) we can obtain the following characterization for the mutual information between Z and Ry (2)
Proposition 1. If Z, N E R, assuming that < 2 S pz, N ~ NNO,, σ), and N is independens of Z, then we have
I(Z: RN (Z)) - log(1 Pz
1
(5)
Remark 3. When the noise is heavy, i.e., the a is large, the upper bound of ((2; RN(2), decays, indicating that is harder to recover Z from R, (Z) and thus harder to infer Y from R (Z). When the noise becomes very heavy, I(Z; R,y (Z)) will be near zero and R (Z) is close to pure Gaussian noise, as if the modality Z is missing, which suits our intuition. In this extreme case, we can refer to the analysis in section 3.2.
In this setting, the Bayes error rate for classification denoted as PNo can be written as:
NNo = Ex,2=Px.e max P(Y y/,z, Rx(=))]
VEY
Then we can provide the lower bound for PNo. Theorem 3.2. For random variables X, Y, N, 2 $ P2, N ~ -N(0, a), then
PNo H(Y|X, 2) + I + I Y: R2 (Z)) 1 log(4 4+ 1pz
(6)
log Y
Remark 4. Similar to the analysis modality missing setting, the gap between pÑo (best model performance in noisy modality setting) and Pe. (best model performance in normal setting) reflects the best model robustness against noisy modality. For the classification task, the lower bound of NN increases as Iz or a increases. Since the bounds of Pa, are fixed, the gap between PNo and Pa. becomes larger, and the model robustness under noisy setting is worse. Therefore, if the z is larger, the best predictor become more vulnerable to the added noise.
METRIC
In this section, propose dataset-wise metric based on the complementary information to quantify the modality complementarity. We also bring our metric to practical use by leveraging the existing mutual information estimator, Mutual Information Neural Estimator (MINE) (Belghazi et al., 2018
5

---

Under review as a conference paper at ICLR 2023
Modality 1
Feature Extractor 1
Mutual Information Estimator
Update by L(0)
Pre-trained Network
MLP
To
Feature
Feature Extractor 2
Modality 2
Middle-fusion
Pre-trained Network
Feature
One-hot Label
Figure 2: Pipeline calculate the metric: First extract the features of the input data from two modal- ities by pre-trained models. Then apply the MINE to estimate the mutual information T(Z; Y, X), I(X;Y, Z), or I (X;; Z).
4.1 METRIC DESIGN
From the above analysis, is natural to use rx + I' as the metric since they represent how much complementary information the modalities X and 7 can provide exclusively about the target Y.
However, Fx + Zz z is not enough for comparing among datasets. According to equation 1, the same amount of I'xx I'z could indicate different situations if the total information / [(X, Z; Y)) different. Therefore, to make the metric comparable among datasets, we need to perform normalization by dividing it with [XX, Y), written
F'x +1z
T(X, YYY)
Now, our metric becomes the "proportion" of If + Iz in I(X, Z;Y). When our metric is large, the modalities are more complementary each other and more indispensable for the task. Note that this "proportion"" could be greater than because T(X; Y; Z) = T(X; 7; Y) rx Tz may be negative. This happens when Z (or X) greatly increases the correlation strength between X (or Z) and Y. Without Z (or X), the other modality becomes nearly uncorrelated with the target Y. Hence, when the metric is greater than I, it can still reflect the modality complementarity and reveals more about the interconnection between the modalities and the target.
4.2 CALCULATION
Now we consider how to calculate our metric. F'x and I'z are in the form of conditional mutual information and could not be computed directly. We notice that
Tz T(Z:Y|X) = Y, x) r(z;
Fx ((X;YIZ) (X:YX I(X;2)
So we transform the metric into
Fx + Izz I(X:Y,2) + I(Z;Y, 21(X;2)
T(X, ZY
T(X,2:Y)
Additionally, considering that most real-world datasets roughly satisfy the realizability assumption, ie., there exists function in the hypothesis space that predict Y given X and Z with zero population risk, we could approximate T(X, Z;Y) -Y)= H(Y)= - HY X, Z) with H(Y) because the second term is close zero. HYY is easier compute given the distribution of Y, especially when we focus on the classification task with discrete labels.
For each mutual information term with the form T(A; design two-phase pipeline for computation (See Figure 2):
In the first phase, reduce the dimension of the high- dish-iimensional input A and B to accelerate the computation by pre- are-trained feature extractors. The pre- pre-trained models are shared among the calculation of all three terms.

---

Under review as a conference paper at ICLR 2023
In the second phase, use the extracted features inputs for MINE (Belghazi al., 2018) to compute the mutual information. Specifically, we calculate the value through optimization converging to lower bound of the mutual information. For each iteration, we sample an m-sample batch {(a(9) b )()) from the joint distribution P(A, B) and an m-sample batch ) from the marginal distributions P(B). Denote the estimator network as T' and its parameters f. We evaluate the lower bound L as follows and the moving average of gradients of L(0) for updating the network parameters.
((0)-
1 m m
(a .b()) log(
1
exp(To (TT(( br(()))
m
We adjust the original MINE by adding the following trick: The calculation of I(X; Y, Z) and (2; Y, X) involve the target Y, we concatenate the one-hot encoding label with the extracted feature in a middle-fusion fashion and ensure that the estimator network T could combine the two information sources. For more details, please see the supplementary material.
We believe that modality complementarity is crucial to the analysis of multi-moda robustness. Without controlling this factor, we cannot fairly compare experimental results on various multi- modal datasets, and thus cannot derive universal conclusion on multi-modal robustness. By calculating our metric on multi-modal datasets, we will better understand their difference in modality complementarity, leading to less biased comparisons and conclusions.
EXPERIMENTS
We conduct experiments to verify the validity of our analysis and the effectiveness of our pipeline. We first introduce the training and testing settings and then show the results on the synthetic dataset, Additive AV-MNIST dataset, and real-world datasets. Unless otherwise specified, the missing/noiss/adversaae
robustness mentioned in the following subsections refers to the average accuracy under two sources of missing/noise/ad versaraili attack, divided by the model accuracy in the clean setting. For more detailed settings and results, please see the Appendix B and C.
Training setting. We varies to match the input size. We train the estimator on the training set since in reality we only have access it, and we assume that the validation set is i.i.d. sampled from the same distribution as the training set. For the pre-trained feature extractors, we use Resnet18 (He et al., 2015) and AudioNet (1-D CNN) (Tian & Xu, 2021) ) for Kinetics-Sounds and AVE, LeNet5 (LeCun al., 1998) and an 2-D CNN for AAV-MNIST, and 15snet152 (He et al., 2015) and BERT (Devlin et al., 2019) for Hateful-Meme dataset. For the models tested for robustness, we use late fusion models for AVE, SSnndscs-Souds, and AV-MNIST, and we apply MMBT (Kiela et al., 2019) for Hateful-Meme dataset.
MLP as the estimator. For different datasets, the structure of the MLP
Test setting, We test the model robustness under two settings discussed above: missing modality and single noisy modality. We also explore the model robustness under adversarial attack. For the missing image audio, we substitute them with the average of all inputs in the training set. For the missing text, we use a blank sentence <SOS> <SS>>>>O>>>>>>>>>>>>>>>>>>>>>>>> as the input. Note that the inputs are all scaled to the range [-1, 1] (spectrogram) or [0, 1] (image). For noisy image and audio, we add Gaussian noise N (0, 0.5) to each dimension. For noisy text, we replace each word by a random word with a probability 0.5. For adversarial attack image and audio, we FGSM (Goodfellow et al., 2014) with step size = 0.03. We use the results of missing text for adversarial text.
5.1 SYNTHETIC DATASET
We first test analysis on well-designed synthetic dataset since
adjust the degree of its modality complementarity. Inspired by previous work (Hessel & Lee, 2020) and (Huang et al., 2021). we generate a set of synthetic data (z, 2, y). First, we sample random projection Px Rg
Pz E Rdax and P E dxd from a uniform distribution U( (.5. 0.5). Then we repeat following steps:
We sample 5000 data points with 80/20 train/val split and (d, dr, d2, (50, 200, 100, 0.25).
7

---

Under review as a conference paper at ICLR 2023
12-
-.0
1.2-
-0.2
meseing
out milc
massig
-a" 10000
0.0
1.0-
0070
8-
0.8
Our metric
0.8
0.58
6-
0.8
0.5
0.4
0.34
2-
-0.4
0-
0.0
0.0
0.1
0.2
0.3 0.4
0.5
0.8
-2.0
-1.5
-1.0 -0.5 0.0
Figure 3: Two line plots showing the mean value of estimated metric (blue line) with error bars (standard variances) of three independent repeated experiments and tested robustness (orange line) on the synthetic dataset (left) and AAV-MNIST dataset (right). The x-axis is the parameter used in data generation. For the synthetic dataset, plot a. For AAV-MNIST, As the overlap of two modalities becomes larger, they are less complementary and our metric correspondingly goes down. Meanwhile, the tested robustness increases in all three settings. The variance and trend of each mutual information term can be found in the supplementary material.
plot for unity.
Step 1. Sample x, z E Rd ~ .N 1).
Step 2. Set (1 - a) + oz; Then do projection +- Pz.
Step 3. Normalize x, to unit length; if x I 6, return to the Step 2.
Step 4. Generate the label y: If x- 0, then y = 1; else 0. Return the tuple (Pxx, Pzz,
The a used in data generation controls the overlap between the two modalities X, Z. When a = 0, the modalities are independent and complementary in predicting the label Y. When
they are redundant for prediction. Viewing the synthetic dataset with different a as different datasets, we calculate our metric using the pipeline in section and test the robustness of simple two-layer perceptron neural networks trained on these datasets. The results are shown in the plot 3. In each dataset, our pipeline can estimate the proposed metric and quantify the complementarity of the two modalities. Further, the model robustness decreases as the complementarity increases, which verifies our analysis.
5.2 ADDITIVE AV-MNIST
To show that our pipeline effectively estimate the modality complementarity of more complex datasets, we further design toy dataset named Additive AV-MNIST (AAV-MNIST) adapted from the AV-MNIST dataset (Vielzeuf al., 2018). The modality complementarity can controlled by parameter a in the data generation process. Below, we show how to generate AAV-MNIST dataset from the original AV-MNIST dataset. The following steps are repeated for every image in AV-MNIST:
Step 1. Let æ be the label of Sample ~ NO o) and round 8 to the nearest integer.
Step 2. Set + (x + 5) mod 10. Uniformly sample spectrogram s from spectrograms in AV-MNIST with label y.
Step 3. Calculate the new label i + (x + 9)/2. Round f to the nearest integer. Return the tuple
(i, st).
The AAV-MNIST dataset is an extension of AV-MNIST dataset. When a AAV-MNIST dataset is equivalent to AV-MNIST dataset where each image and its paired spectrogram represent the same number. As a increases, each image become less correlated with its paired spectrogram, they become more complementary for predicting the label t.
We show in the plot that our metric reflects the complementarity of the AAV-MNIST dataset with different a, indicating that our pipeline is effective in more complex settings beyond the synthetic
Fach achieves accuracy > 96% on corresponding validation sets. The neural network structure and more training settings he found the supplementary materials.
8

---

Under review as a conference paper at ICLR 2023
Dataset
Our metric
Clean
Missing
Noisy
Adversarial
AAV-MNIST(O = 2.0)
0.9212
0.6435
0.3368
0.5399
0.1612
Hateful-Meme
0.2403
0.3249
0. 0.1005
0.5171
0.3144
AV-MNIST
0.0490
0.9969
0.5666
0.6478
0,6012
Kinetics-Sounds
0.0455
0.6387
0.5540
0.6098
0.2672
AVE
0,0126
0.7637
0,4838
0.5831
0.3355
Table Our estimated metric and tested robustness of real-world datasets: Kinetics-Sounds, AVE, AV-MNIST, and Hateful-Meme. Since the Hateful-Meme Challenge is binary classification task, we use Fl score for evaluation instead accuracy. We also provide results in clean setting for reference.
dataset. Further, the robustness in the three settings verifies our conclusion that with other conditions unchanged, the more complementary the modalities are, the less robust the best model will be.
5.3 REAL-WORLD DATASETS
Now we apply our pipeline to real-world datasets to investigate their modality complementarity. Our results on AVE (Tian et al., 2018), Kinetics-Sounds (Carreira Zisserman, 2017: Arandjelovic & Zisserman, 2017), and Hateful-Meme dataset (Kiela et al., 2020a) are shown in the table 1. The details of these datasets are described the Appendix B. We also list results on the AV-MNIST dataset and AAV-MNIST (a 2.0) for reference.
The low value in our metric of AVE, relatively little modality complementarity, revealing the heavy redundancy between the two modalities. On the contrary, the modalities the Hateful-Ment dataset are more complementary. This finding suits our intuition: In the Hateful-Meme dataset, altering the paired text of an image probably changes the label (Kiela al., 2020b). Hence, only perceiving the image would not derive the right answer. For the event classification task defined by Kinetics-Souneee or AVE, the audio and frames both lead to rough answer.
and AV-MNIST indicates that they possess
The tested robustness demonstrates how the modality complementarity affects model robustness. The missing case affects AAV-MNIST(o = 2.0) and Hateful-Meme far more than the other three datasets. They also more vulnerable in single source noisy than other datasets. Hence, to compare model robustness among these datasets, we should take modality complementarity into account. For instance, we only compare robustness among datasets with similar degree of modality complementarity, or we can normalize the results by our metric. We show an analysis of the existing measure of modality missing robustness by applying our metric in the Appendix C.3.
Furthermore, the model robustness, especially the adversarial robustness, is also affected by factors other than modality complementarity. For instance, the model adversarial robustness of AVE and Linetics-Sounde dataset is significantly lower than that of AV-MNIST dataset. We conjecture that this is related the number of robust features in each modality of the datasets, which requires future work to confirm.
CONCLUSIONS
In this work, we partly explain the contradiction in previous conclusions multi-modal robustness by pointing out the importance of the modality complementarity through information-theorelt analysis and creffully-dssinned experiments. As reflection of modality interconnection, our proposed metric provides a basis for better understanding various multi-modal datasets/tasks and can be used beyond analyzing multi-moot modal robustness.
REPRODUCIBILITY STATEMENT
We provide the source code and configuration for the key experiments, including instructions on generating data, training the models, and evaluating the robustness. We thoroughly checked the code implementations and empirically verified the effectiveness our method. All proofs are stated the appendix with explanations and underlying assumptions.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Aishwarya Agrawal. Dhruv Batra, Devi Parikh, and Aniruddha Kembhavi. Don't just assume; look and answer: Overcoming priors for visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4971-4980, 2018.
Massih R. Amini, Nicolas Usumier, and Cyril Goutte. Learning from multiple partially observed views application to multilingual text categorization. In Y. Bengio, D. Schuurmans, J. Laf- ferty, C. Williams, and A. Culotta (eds.). Advances in Neural Information Processing Systems, volume 22. Curran Associates, Inc., 2009. URL ht http::ss///pccceeeeeeee
nwwrips........... cc/ pdÉ.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitmick, and Devi Parikh. VQA: visual question answering. CoRR, 1o0s0555500468, 2015. URL http://arxixe org/ abs abs/1505 00468.
Relja Arandjelovic and Andrew Zisserman. Look, listen and learn. CoRR, abs/ /tbs///0550006.. 2017.
URL hwtp::tttttttt./ org/ abs /175. 08168.
Philip Bachman, R. Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. CoRR, abs/1906.00910, 2019. URL /tttp://arxiv org/
abs/1 1 9006906000 00910.
Tadas Baltrusaitis, Chaitanya Ahuja, and Louis-Philippe Morency. Multimodal machine learning: A survey and taxonomy. CoRR, 0bs//705509406 2017. URL http::/arxiv..crg/////// 11705.
09406.
Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and Devon Hjelm. Mutual information neural estimation. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 531-540. PMLR, 10-15 Jul 2018. URL htttps://proceedingg: mlr...ss.......m..
mlr press / v80 /ttp///haiiiiaaa../ea/ html.
Nicholas Carlini, Anish Athalye, Nicolas Papernot, Wieland Brendel, Jonas Rauber, Dimitris Tsipras, Ian Goodfellow, Aleksander Madry, and Alexey Kurakin. On evaluating adversarial robustness. CoRR, abs/1190.006755, 2019. URL http: attaariivvv.oo wwwp:xiwwwv..oo//oootoooooootsep 902. 06705.
João Carreira and Andrew Zisserman. Quo vadis, action recognition? A new model and the kinetics dataset. CORR, abs/1 705.07750 2017. URL htt.tt::/arrivv..rr//assss 705. 07750.
Jan K Chorowski, Dzmitry Bahdanau, Dmitriy Serdyuk, Kyunghyun Cho, and Yoshua Bengio. Attention-based models for speech recognition. Advances in neural information processing systems, 28, 2015.
Jeremy M. Cohen, Elan Rosenfeld. and J. Zico Kolter. Certified adversarial robustness via randomizzd smoothing. CoRR, abs/1902.02918. 2019. URL /tttp://axxiv... wwwpwwwwwwww..aaaaaaaaaaeaeaaeaa http//bss/1/900 02918.
Thomas M Cover. Elements of information theory. John Wiley & Sons, 1999.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Jill Burstein. Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume I (Long and Short Papers), pp. 4171- 4186. Association for Computational Linguistics, 2019. doi: 10.186653/1/111-14422 URL https: /oi..iio////// www.ooo..........o.. 1tt.111188660 11w.1/1661ll/e
Daisy Yi Ding, Balasubramanian Narasimhan, and Robert Tibshirani. Cooperative learning for multi-view analysis, 2021. URL httpp:://axxiv..rggg/////// wrgg/b////1////////////// 12337.
Andreas Eitel, Jost Tobias Springenberg, Luciano Spinello, Martin A. Riedmiller, and Wolfram Burgard. Multimodal deep learning for robust RGB-D object recognition. In 2015 IEEE/RSJ Intelligent Robots and Systems, IROS 2015, Hamburg, Germany, September 28- October 2, 2015, pp. 181-681777778 IEEE, 2015. doi: 10.110//ROS.2215..333444 URL https::////......... /tt.///oooo/////// org / 10. 11 09 TROS. 2015. 7353446.
International Conference
10

---

Under review as a conference paper at ICLR 2023
M. Feder and N. Merhav. Relations between entropy and error probability. IEEE Transactions Information Theory, 40(1):259-286 266, 1994. doi: 10.1109/18.222994
Marco Federici, Anjan Dutta, Patrick Forré, Nate Kushman, and Zeynep Akata. Learning robust
representations via multi view information bottleneck. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa. Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL
ew. net wwt./nettoootnne forum? d=B1xxcyHHFF.
Christoph Feichtenhofer, Axel Pinz, and Andrew Zisserman. Convolutional two-stream network fusion for video action recognition. CoRR, abs// 604.06573, 2016a. abs/ 604.0044 06573.
/
Christoph Feichtenhofer, Axel Pinz, and Andrew Zisserman. Convolutional two-stream network fusion for video action recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1933-1941, 2016b.
Nic Ford, Justin Gilmer, Nicholas Carlini, and Ekin Dogus Cubuk. Adversarial examples natural consequence of test error in noise. CoRR, abs/1901.1055 2019, URL abs/ hts/1901.10511
Keinosuke Fukunaga and Donald M Hummels. Bayes error estimation using parzen and k-nn procedures. IEEE Transactions on Pattern Analysis and Machine Intelligence, (5):634-643, 1987.
Ruohan Gao, Tae-Hyun Oh, Kristen Grauman, and Lorenzo Torresani. Listen to look: Action recognition by previewing audio. CoRR, abs/1912.04487, 2019. URL abs/1912, 04487.
Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples, 2014. URL https: org/ absaabs/a44 /1412.111.. 6572.
Jiuxiang Gu, Jianfei Cai, Shafiq Joty, Li Niu, and Gang Wang. Look, imagine and match: Improving textual-visual cross- retrieval with generative models. CoRR, abs/1711.000 1.06420, 2017. URL http:://axii..ceeeeee org. abs
06420.
Zongbo Han, Changging Zhang, Huazhu Fu, and Joey Tianyi Zhou. Trusted multi-view classification. CoRR, abs/2102.02051, 2021. URL https:tppp.....m...tt pettps://arxise /ttp/aaxvvvv..///////eee/ea.at.a wrw.iiioorrraaaee abs////b///// 22022.12222222 02051.
Kaiming He, Xiangyu Zhang, Shaoging Ren, and Jian Sun. Deep residual learning for image
recognition. CoRR, abs/1512.03385, 2015. URL http: /ttp:///aaivvv//ii httpaaawii..ga..aea../r//.. org/ abs///b//// 11t...11...000 03385.
Dan Hendrycks and Thomas G. Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. CoRR, abs/1903. 12261, 2019. URL httptt:://xxxv.ooggg./ /abs/ 1903. 12261.
Jack Hessel and Lillian Lee. Does my multimodal model learn cross-modal interactions? it's harder to tell than you might think! CoRR, abs/2010.065722 2020. URL htttttp: 1010.06577
abs/
R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representatiions by mutual information estimation and maximization, 2018. URL https: http://awwwv...aaaaaaa/ /1808. 06670.
Ruitong Huang, Bing Xu, Dale Schuurmans, and Csaba Szepesvári. Learning with a strong adversary. CoRR, 0tts/1511.0303.. 2015. URL
org/o//g////.... 1111 03034.
Yu Huang, Chenzhuang Du Zihui Xue, Xuanyao Chen, Hang Zhao, and Longbo Huang. What makes multimodal learning better than single (provably). CoRR, abs/2106.04538, 2021. URL
https::/aaxii org/ alss/ 2105. 04538.
Yu Huang, Junyang Lin, Chang Zhou, Hongxia Yang. and Longbo Huang. Modality competition: What makes joint training of arXiv:2203. /2221, 2022.
network fail deep (proving(provablly... arXiv preprint
11

---

Under review as a conference paper at ICLR 2023
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross B. Girshick. CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning. CoRR, 1b1//661, 2016. URL http:://axxiiii org/ abs 068800
Sham M Kakade and Dean P Foster. Multi-view regression via canonical correlation analysis. In International Conference Computational Learning Theory, pp. 82-96. Springer, 2007.
Douwe Kiela, Suvrat Bhooshan, Hamed Firooz, and Davide Testuggine. Supervised multimodal bitransformers for classifying images and text. CoRR, 0bs//199900299) 2019. URL http:
//arxiv. wtaaxii..rg/baaaa....oaoaa. .w9.909900000 02950.
Douwe Kiela, Hamed Firooz, Aravind Mohan, Vedanuj Goswami, Amanpreet Singh, Pratik Ringshia, and Davide Testuggine. The hateful memes challenge: Detecting hate speech in multimodal memes. CoRR, abs/2005.04727 2020a. URL.
org/ abs 200//00550000
Douwe Kiela, Hamed Firooz, Aravind Mohan, Vedanuj Goswami, Amanpreet Singh, Pratik Ringshia, and Davide Testuggine. The hateful memes challenge: Detecting hate speech in multimodal memes. arXiv preprint arXiv:2005.04770 04790, 2020b.
Taewan Kim and Joydeep Ghosh. On single source robustness in deep fusion models. CoRR, abs/1906.04699 04991, 2019a. URL httt:://axiv..oo org/ abs/190s/ 04691.
Taewan Kim and Joydeep Ghosh. On single source robustness in deep fusion models. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Asso- ciates, Inc., 2019b. URL
neurips cc/ppaper/211//11eee pdf..
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolu- tional neural networks. Advances in neural information processing systems, 25, 2012.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEFE, 86(11) (1))))2728-2224, 1998.
Linjie Li. Zhe Gan, and Jingjing Liu. A closer look at the robustness of vision-and- lann-uugee pre-trained models. CoRR, abs/2012.08673, 2020. URL 08673.
2212..
Paul Pu Liang. Yiwei Lyu, Xiang Fan, Zetian Wu, Yun Cheng, Jason Wu, Leslic Yufan Chen, Peter Wu, Michelle A Lee, Yuke Zhu, et al. Multibench: Multiscale benchmarks for multimodal representation learning. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021.
Paul Pu Liang, Amir Zadch, and Louis-Philippe Morency. Foundations and recent trends mul- timodal machine learning: Principles, challenges, and open questions, 2022. URL https: //arxiv. www.iivorr//bbaee www.////99900 03430.
Yijie Lin, Yuanbiao Gou, Zitao Liu, Boyun Li, Jiancheng Lv, and Xi Peng. COMPLETER: incomplete multi-view clustering via contrastive prediction. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pp. 11174 11183. Computer Vision Foundation / IEEE, 2021. doi: 10.11099CVPR44644..200000 URL https::/openacceeet thecvf. com/ conteenttccp222211/ttm wttpeeolsseeeengeaaaaaootttooeee
ToopLereR_nncomppleeeeeeee Predictiion_cvv__22211 ppwppppprrpppppppr___p html.
Yijie Lin, Yuanbiao Gou, Xiaotian Liu, Jinfeng Bai, Jiancheng Lv, and Xi Peng. Dual contrastive prediction for incomplete multi-view representation learning. IEEE Transactions on Putlern Analysis and Machine Intelligence, pp. 1-14, 2022. doi: 10. 119//PAMM..2222.319733
Mengmeng Ma, Jian Ren, Long Zhao, Sergey Tulyakov, Cathy Wu, and Xi Peng. Smil: Multimodal learning with severely missing modality. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 2302-2310, 2021.
12

---

Under review as a conference paper at ICLR 2023
Mengmeng Ma, Jian Ren, Long Zhao, Davide Testuggine, and Xi Peng. Are multimodal transformers robust to missing modality?, 2022.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks, 2017. URL https://arxiv org/abs/ 1706. 0 06083.
Kaleel Mahmood, Rigel Mahmood, and Marten van Dijk. On the robustness of vision transformers to adversarial examples. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pp. 7818-7827, IEEE, 2021. doi: 10. 10.11999/CV489922200000000 00774.
https://:/ /tttpp:///o...oo/1......... http://...111/1/11/
2021.
David McAllester and Karl Stratos. Formal limitations on the measurement of mutual information. CoRR, abs/181 1111111,,,,,1, 2018. URL
04251.
Dongyu Meng and Hao Chen. Magnet: two-pronged defense against adversarial examples. CoRR, abs/1 000004, 2017. URL http: /ttt:p://axxiv..oo abs /105. 09064.
Nicolas Papernot, Patrick D. McDaniel, Xi Wu, Somesh Jha, and Ananthram Swami. Distillation as defense to adversarial perturbations against deep neural networks. CoRR, abs//5111.4400 2015.
URL http: /ttp/viiiivv..///.///ee////aatvv org/ abs/ 04508.
Kun Qian, Shilin Zhu, Xinyu Zhang, and Li Erran Li. Robust multimodal vehicle detection in foggy weather using complementary lidar and radar signals. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 444-453, June 2021.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. CoRR, abs/2103.00020, 2021. URL https: / httpaaaxvvvvvio// oww.//ii.oroooooooooo abs/ / 2212200 2103..1 00020.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and μla Sutskever. Zero-shot text-to-image generation. CoRR, abs/2102.12092, 2021, URL fttps://arxxt wwwg/////ssa.. alos/ 2102 12092.
Marco Ramoni and Paola Sebastiani. Robust learning with missing data. Machine Learning, 45: 147-170, 11 2001. doi: 10. 10.1023/A:10119987029299
Daniel Rosenberg. Itai Gat, Amir Feder, and Roi Reichart. Are VQA systems rad? measuring robustness to augmented data with focused interventions. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 2: Short Pupers), Virtual Event, August 1-6, 2021, pp. 61-70. Association for Computational Linguistics, 2021. doi: 10.18653/w1/2021..dg/ URL https::t///// doi org/ 10. 11..1.6653/11/100
10.
10.
Ravid Shwartz-Ziv and Naftali Tishby. Opening the black box of deep neural networks via information. CoRR, abs/1703.008 703.00810, 10, 2017. URL http: wrwii.ivvvo.o. www.wiwwgg................oooeee ars//bs/////eoooooee/////////eer 1703 00810.
Carl-Johann Simon-Gabriel, Yann Ollivier, Leon Bottou, Bernhard Schölkopf, and David Lopez- First-order adversarial vulnerability of neural networks and input dimension. In Kamalika Chaud- huri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 5809-5817. PMLR, 09- 15 Jun 2019, URL https: https::/ http://pooceeeeeeeeeeeeee mlr. wwwpppesssvveeee simnn//smooo/eiom http://mmmmmaaiiieiiieeeeeee gabriell9e html.
Vikas Sindhwani, Partha Niyogi, and Mikhail Belkin. A co-regularization approach to semi- supervised learning with multiple views. In Proceedings of ICML workshop on learning with multiple views, volume 2005, pp. 74-79. Citeseer, 2005.
Amanpreet Singh, Vedanuj Goswami, Vivek Natarajan, Yu Jiang, Xinlei Chen, Meet Shah, Mareus Rohrbach, Dhruv Batra, and Devi Parikh. Mmf: A multimodal framework for vision and language research. https: httpt//gttttt.//////////ttt.. thub. com/ facebook rchaanchhmmmm 2020.
13

---

Under review as a conference paper at ICLR 2023
Karthik Sridharan and Sham M. Kakade. An information theoretic framework for multi-view learning. In Rocco A. Servedio and Tong Zhang (eds.). 21st Anmual Conference on Learning Theory COLT 2008, Helsinki, Finland, July 9-12. 2008, pp. 403-414. Omnipress, 2008. URL http:/ / /////otttta (co.t2008... cs. he mellsikki.. iinki...... paperss/p///paa / 94 Sri wwwraa.aaaa.ee pdf.
William H Sumby and Irwin Pollack. Visual contribution to speech intelligibility noise. The journal of the acoustical society of america, 26(2):112-215, 1954.
Xinwei Sun, Yilun Xu, Peng Cao, Yuqing Kong, Lingjing Hu, Shanghang Zhang, and Yizhou Wang. TCGM: an ceoormation-thorettti framework for semi-supervised multi-modality learning. CORR, abs/2007.06793, 2020. URL https: http:ps:///xxx...rr///////////// wwtp:/wwwwww../////////oooo 00770000333
Yapeng Tian and Chenliang Xu. Can audio-visual integration strengthen robustness under multimodal attacks? CoRR, abs/2104.02000, 2021. URL /.l.:nttsss//arx...orr///// 0000//2010000000.... 02000.
Yapeng Tian, Jing Shi, Bochen Li, Zhiyao Duan, and Chenliang Xu. Audio-visual event localization in unconstrained videos. CoRR, abs/1803.08842, 2018. URL http:::aaxx.v..rrgg////////////
1803.08842.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. CoRR,
abs/1906.05849, 2019. URL hwtp::t/t///tttt org/absg////..ooooo...o......... /106. 058.9.
Naftali Tishby, Fernando C. Pereira, and William Bialek. The information bottleneck method, 2000.
URL https: /arpivv///aiivva//a org/ httss//hysccs//e 00044057.
Christopher Tosh, Akshay Krishnamurthy. and Daniel Hsu. Contrastive learning, multi-view re- dundancy, and linear models. In Vitaly Feldman, Katrina Ligett, and Sivan Sabato (eds.), Algorithmic Learning Theory, 16-19 March 2021, Virtual Conference, Worldwide, volume 132 of Proceedings of Machine Learning Research, pp. 1179-1206. PMLR, 2021. URL http:/ http:///poeeeeeeneeeeeee www.mmlrrrsss.....m ww.pprsssvvii. vw1///l1//////////// http//tsh22aa....
Luan Tran, Xiaoming Liu, Jiayu Zhou, and Rong Jin. Missing modalities imputation via cascaded residual autoencoder. In 2017 IEEE Conference Computer Vision and Pattern Recognition doi: 10.11099CVPR.2217.5222
(CVPR), pp. 4971-4980, 2017.
Yao-Hung Hubert Tsai, Paul Liang, Amir Zadeh, Louis-Philippe Morency, and Ruslan Salakhut- dinov. Learning factorized multimodal representations. CoRR, Abs/1806.06176 1666, 2018. URL
http://arxiv. org/ abs//ab/ 111.8....666........11..1..11166 06176.
Yao-Hung Hubert Tsai, Yue Wu, Ruslan Salakhutdinov, and Louis-Philippe Morency. Demystifying framework. CoRR, abs/2006.05576, 2020.
self-supervised learning: An serfoommttion-theoreee
URL wwt.t:tpttto..... http:///xxxvvv.voo abaaaaass//000 2006 05576.
Aäron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. CoRR, 01770007003755 2018. URL hwtpthtt://ee /ttp://aaaivv.//.a//ia/ wttpaatwww..aa.aaaaaooooo.aooott wwwp///g/////ooeeeee..o/eeeeeeee /tbbs/180 /1807.110..... 03748.
Valentin Vielzeuf, Alexis Lechervy, Stéphane Pateux, and Frédéric Jurie. Centralnet: a multilayer
approach for multimodal fusion. CoRR, abs/1808.072755 2018. URL http://arxiv org/
abs/1808. 07275.
Jinghua Wang, Zhenhua Wang, Dacheng Tao, Simon See, and Gang Wang. Learning common and specific features for rgb-d semantic segmentation with deconvolutional networks. In European Conference on Computer Vision, pp. 664-679. Springer, 2016.
Shaojie Wang, Tong Wu, and Yevgeniy Vorobeychik. Towards robust sensor fusion in visual perception. CoRR, abs/2006. 13192, 2020. URL ht : / /arxivvv.v.///////v///a///avvav/ org/ abs/20066 13192.
Yihong Wu and Sergio Verdu. Functional properties of minimum mean-square error and mutual information. IEEE Transactions Information Theory, 1889) 2012. doi: 10.1 10.1
TIT.2011.217495,,
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc Le. Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, al. Google's neural machine translation sys- tem: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08/1 160.008144, 2016.
14

---

Under review as a conference paper at ICLR 2023
Chang Xu, Dacheng Tao, and Chao Xu. A survey multi-view learning. CoRR, abs/1304.5634, 2013. URL http:/ wawpiv.iiii.v///ii//eee//aaaaava www.eeiwi...raaaae.eee org/ 1304. 5634.
Karren Yang, Wan-Yi Lin, Manash Barman, Filipe Condessa, and Zico Kolter. Defending multimodal fusion models against single-source adversaries. In 2021 IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pp. 3339-3348, 2021. doi: 10. N0.1109/CVPR46437.202 00335.
Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua Tenenbaum. CLEVRER: collision events for video representation and reasoning. CoRR, abs/1910.01442,, 2019. URL pettp://axxv.ooot
(oorg//b///99// abs/1910. 01442.
Youngjoon Yu, Hong Joo Lee, Byeong Cheon Kim, Jung Uk Kim, and Yong Man Ro. Investigat- ing vulnerability to adversarial examples multimodal data fusion in deep learning. CoRR, abs/2005. 109920 2020. URL https: wtwp/aaxivv..../ org/ aaabbb/// 2200/2005..997
Lei Yuan, Yalin Wang, Paul M. Thompson, Vaibhav A. Narayan, and Jieping Ye. Multi-source learning for joint analysis of incomplete multi-modality neuroimaging data. In Qiang Yang. Deepak Agarwal, and Jian Pei (eds.), The 18th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '12, Beijing, China, August 12-16, 2012, pp. 1149- 1157. ACM, 2012. doi: 10. 10.1145/2339530.2339111 URL https://ddo..oooe /tttss://ooi.org/00.11111 2339530.2333
Changqing Zhang, Zongbo Han, yajie cui, Huazhu Fu, Joey Tianyi Zhou, and Qinghua Hu. Cpm-nets: Cross partial multi-view networks. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Cur- ran Associates, Inc., 2019a. URL https: //tpce/eiiccgg..pcoooc... neurips. cctpppee///11/ pdf.
Wenwei Zhang, Hui Zhou, Shuyang Sun, Zhe Wang, Jianping Shi, and Chen Change Loy, Robust multi-modality multi-object tracking. CoRR, abs/1909.03850, 2019b. URL http::/arxiis org/abs//99t 99999. 03850.
Hang Zhao, Chuang Gan, Andrew Rouditchenko, Carl Vondrick, Josh McDermott, and Antonio Torralba. The sound of pixels. Proceedings of the European conference on computer vision (ECCV), pp. 570-586, 2018.
Stephan Zheng, Yang Song, Thomas Leung. and Ian Goodfellow. Improving the robustness of deep neural networks via stability training. CoRR, abs/1604.04326, 2016. URL http: //arxiv. www.ai..gg..as////aooaao...aaaaa 04326.
15


Reviews:
[{"review_id": "vDV2QTbuPL", "reviewer": "Reviewer_L857", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the contradiction in the existing multimodal robustness literatures. Intuitively, multimodal models are supposed to be more robust than unimodal model as extra redundancy is provided by multimodal data. However, some existing works find that multimodal integration may be more vulnerable to attack, noise or corruption. To understand this contradiction, the authors make a theoretical analysis using information theory. By formally giving the definition of complementary information, the authors establish the theoretical guarantee to quantify the different Bayes error rate of multimodal and miss-/noisy modality cases, and provide a metric to quantify the complementariness of multi-modalities. Experiments on synthetic and real world dataset show the value of measuring complementariness. \n\n--- post-rebuttal ---\\\nThanks for the authors' response. I have read the response and other reviewers' comments. Both reviewer TcW7 and I have a similar concern about the influence of pre-trained model. The authors propose two possible ways of constraining the feature extraction part to be the same. While it is still unclear to me for the case when two to-be-compared multi-modal approaches have different feature extraction modules. Despite this, I still believe there is value of being a pioneering work exploring modality complementariness, but slightly lower my score by accounting the review confidence weight.", "summary_of_the_review": "The studied problem of how modality complementariness affects multimodal robustness is interesting. This paper provides a new angle to look at multimodal robustness. I have two general questions regarding the metric. Please see them in the weakness section.", "strength_and_weaknesses": "Strength: \n\n-The studied problem of how modality complementariness affects multimodal robustness is interesting. This paper provides a new angle to look at the multimodal robustness. \n\n-The authors a thorough theoretical analysis of the interconnection of modality complementariess and robustness in multimodal setting.  \n\n-The empirical results clearly demonstrate that the proposed metric is able to quantify how complementariess of each multimodal dataset. It provides an opportunity to make a fair comparison under similar complementariness for future studies. \n\n \n\nQuestions/Weakness: \n\n-The proposed metric provides an opportunity to make a fairer comparison of multimodal robustness among the existing methods. I’d like to see an analysis of the existing popular multimodal robustness approaches by applying the proposed metric. \n\n-Although network architecture, training routines are out of this paper’s scope, the proposed complementariness metric is measured based on pretrained models for feature extraction. A more powerful base model usually extracts richer features. I am not clear on how to make a fair comparison if two multimodal models have different backbone feature extractors.  ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Mi1VW5W-dpK", "reviewer": "Reviewer_EKpK", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The article investigates the robustness of multi-modal learning by introducing the concept of modality complementariness. The analysis is based on information theory, presenting the increasing Bayes error in the condition of missing and noisy modality. Then, a dataset-wise metric is proposed to measure the complementariness of each dataset. Furthermore, a dataset with more complementary information is designed in this paper. More experiments are applied to support this theory with this metric.", "summary_of_the_review": "This paper examines the concept of Complementariness in multi-modal learning paradigm. Although the perspective is nice, some deeper insights had better to be proposed to enrich the theory. ", "strength_and_weaknesses": "Strengths:\n1. Although multi-modal complementariness has been focused on, this paper provides an intuitional solution (i.e. information theory) to explain the effect of this characteristic. \n2. The notation and proof of this article are clear.\n\nWeakness:\n1. This work only focuses on Bayes error on a fixed model and shows how complementariness affects robustness. That is a strong assumption and thus the application scenarios are limited. What if the model takes in noisy or incomplete multi-modal data when training?\n2. The experiment setting seems weak and less supportive. Through Figure 3 and Table 1, it is disturbing whether the deterioration is caused by complementariness or other nature of the dataset itself.  \n3. Details of these datasets had better be provided.\n4. Some special meanings are best represented in the figure. For example, the meaning of $\\Tau_Z$ when modality missing is quite confusing. Should it vary when data suffer from contamination?\n5. This paper focused on two modalities. Then, how to define the metric in datasets with more than two modalities? \n6. The word Complementarity may be more commonly used to denote complementary properties than Complementariness.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "KqSXjOUUzrb", "reviewer": "Reviewer_EDnT", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper gives an information-theoretical abstraction to represent  the modality robustness through the proposed complementary metric. The deduction is based on the assumption that models are anticipated to be more robust due to the possible redundancy between modalities. The authors also do some controlled experiments and try to support their theoretical analysis.\n", "summary_of_the_review": "I recommend accepting the paper (rating 6). I liked the abstraction proposed by the authors and particularly liked the way authors make connections between the complementariness with missing modality and noisy modality. Although this paper has some drawbacks for practical application, the contribution is actually enough. Ratings can be improved further if the authors could solve my questions, especially the use of label information.\n", "strength_and_weaknesses": "Pros:\n\n1. This work presents a very detailed theoretical analysis of multi-modal robustness. The complementary is estimated through a litter modification on MINE with a novel information metric.\n2. The organization of the paper is good in general.\n\nCons:\n\n1. My main concern with this paper is its usability. To analyze the complementariness, the methods need the label information beforehand. However, as we have known the label information, one could directly train the multi-modal network and percept the robustness intuitively. I think the use of label information is an obvious drawback of this work.\n2. Some works focus on the reliability of the modality [1], which could also learn the uncertainty (importance) of each modality, which is also a kind of robustness. In contrast, the proposed method could only analyze the robustness of the datasets, ignoring an important perspective, i.e., improving the representation of the network.\n3. Many recent multi-view works about information theory are missing [2-5]. All these methods attempt to establish the relationship between the different modalities and the label information in an unsupervised manner, although these methods are based on some extra assumptions, e.g., mutual redundancy [3,5] or containing view-specific information [2]. Especially, [3,4] also discuss the usability of the missing modality from information theory. The paper could do a better job by citing these relevant works and clarifying the differences between this work and them.\n\n[1] Trusted Multi-View Classification, ICLR'21\n\n[2] Self-supervised Learning from a Multi-view Perspective, ICLR'21\n\n[3] Dual Contrastive Prediction for Incomplete Multi-View Representation Learning, TPAMI'22\n\n[4] COMPLETER: Incomplete Multi-view Clustering via Contrastive Prediction, CVPR'21\n\n[5] Learning Robust Representations via Multi-View Information Bottleneck, ICLR'20", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "HQ3VEZxIGB", "reviewer": "Reviewer_TcW7", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper empirically studied the connection between the complementariness of modalities and multimodal model robustness. The authors pointed out that the multimodal models are more robust as the multimodal data are less complementary and otherwise. The conclusion is theoretically verified along with well-defined experiments. Moreover, the author proposed a new metric for measuring the complementary of multimodal data that may help improve the robustness of the multimodal model.\n\n-------after rebbuttal\nI read the resp from the authors, but only some concerns of mine have been addressed. Hence, I rise my score to 5. In fact,  if there is 4, I will only rise to it. ", "summary_of_the_review": "This paper studied the connection between the multimodal data complementary and the model robustness. The paper is well-written. However, the contribution of this paper is limited.", "strength_and_weaknesses": "Strengths:\n1.\tThis paper studied a good question about what affects multimodal robustness. The conclusion is well verified and may help improve model robustness.\n2.\tThe paper is written clearly and is easy to follow.\n\nWeaknesses:\n1.\tHowever, my major concern is that the contribution is insufficient. In general, the authors studied the connection between the complementary and the model robustness but without further studies on how to leverage such characteristics to improve model robustness. Even though this paper could be the first work to study this connection, the conclusion could be easily and intuitively obtained, i.e., when multimodal complementary is higher, the robustness is more delicate when one of the modalities is corrupted. Except for the analysis of the connection between complementary and robustness, it is expected to see more insightful findings or possible solutions.\n2.\tThe proposed metric is calculated on the features extracted by some pre-trained models. So the pre-trained models are necessary for metric computing which is contradictory to that the metric is used to measure the multimodal data complementary. In addition, in my opinion, the metric is unreliable since the model participates in the metric calculation and will inevitably affect the calculation results.\n3.\tThere are many factors that will affect the model's robustness. The multimodal data complementary is one of them. However, multimodal data complementary is not solely determined by the data itself. For example, classification on MS-COCO data is obviously less complementary than VQA on MS-COCO data. As mentioned by the author, the VQA task requires both modalities for question answering, accordingly the complementary is determined by the multimodal and the target task. However, I didn't see much further discussion about these possible factors.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject