Quantifying and Enhancing Multi-modal Robustness with Modality Preference
Authors: Zequn Yang
Yake Wei
Ce Liang
Di Hu
Venue: ICLR 2024 poster (2024)
Keywords: Robustness
Multi-modal learning
Modality preference
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: We theoritcally investigate the essential component related to the multi-modal robustness, and provide insight for the limitation caused by modality preference, which can be alleviated by our proposed training strategy.

Abstract: Multi-modal models have shown a promising capability to effectively integrate information from various sources, yet meanwhile, they are found vulnerable to pervasive perturbations, such as uni-modal attacks and missing conditions. To counter these perturbations, robust multi-modal representations are highly expected, which are positioned well away from the discriminative multi-modal decision boundary. In this paper, different from conventional empirical studies, we focus on a commonly used joint multi-modal framework and theoretically discover that larger uni-modal representation margins and more reliable integration for modalities are essential components for achieving higher robustness. This discovery can further explain the limitation of multi-modal robustness and the phenomenon that multi-modal models are often vulnerable to attacks on the specific modality. Moreover, our analysis reveals how the widespread issue, that the model has different preferences for modalities, limits the multi-modal robustness by influencing the essential components and could lead to attacks on the specific modality highly effective. Inspired by our theoretical finding, we introduce a training procedure called Certifiable Robust Multi-modal Training (CRMT), which can alleviate this influence from modality preference and explicitly regulate essential components to significantly improve robustness in a certifiable manner. Our method demonstrates substantial improvements in performance and robustness compared with existing methods. Furthermore, our training procedure can be easily extended to enhance other robust training strategies, highlighting its credibility and flexibility.

Published as a conference paper at ICLR 2024
QUANTIFYING AND ENHANCING MULTI-MODAL Ro- BUSTNESS WITH MODALITY PREFERENCE
Zequn Yang', Yake Wei', Ce Liang', Di Hu
Gaoling School of Artificial Intelligence, Renmin University of China' {zqyang, yakewei, liangce158, dihu)@ruc edu. en
ABSTRACT
Multi-modal models have shown promising capability effectively integrate information from various sources, yet meanwhile, they are found vulnerable to per- vasive perturbations, such as uni-modal attacks and missing conditions. To counter these perturbations, robust multi-modal representations are highly expected, which are positioned well away from the discriminative multi-modal decision boundary. In this paper, different from conventional empirical studies, we focus on a com- monly used joint multi-mode modal framework and theoretically discover that larger uni-modal representation margins and more reliable integration for modalities are essential components for achieving higher robustness. This discovery can further explain the limitation of multi-modal robustness and the phenomenon that multi-modal models are often vulnerable to attacks on the specific modality. More- over, analysis reveals how the widespread issue, that the model has different preferences for modalities, limits the multi-modal robustness by influencing the essential components and could lead to attacks on the specific modality highly effective. Inspired by our theoretical finding, we introduce a training procedure called Certifiable Robust Multi-modal Training (CRMT), which can alleviate this influence from modality preference and explicitly regulate essential components to significantly improve robustness in a certifiable manner. Our method demonstrates substantial improvements in performance and robustness compared with existing methods. Furthermore, our training procedure can easily extended to enhance other robust training strategies, highlighting its credibility and flexibility.
INTRODUCTION
As data are often presented from different perspectives, like text, images, and audio, how to effectively exploit and integrate information from multiple sources becomes important. This has given rise to the concept of multi-modal learning, which serves as a potent approach that enables a more comprehensive understanding of complex concepts and facilitates more effective knowledge acquisition for different sources of information (Wei et al. 2022). Nowadays, multi-modal learning has demonstrated its remarkable ability in various tasks, including scene understanding (Antol et al. 2015 Yang et al.
2022
al. 2022a). and emotion recognition (Tripathi et al. 2018 Chudasama et 2022)
However, weal-world data are often perturbed, such as attacks, and missing modality, which impacts the performance of multi-modal model (Kumar et al.) 2020). As result, multi-modal robustness, which refers to the model's ability to defend against such perturbations, has received increasing attention recent studies (Bednarek et al. 2020 Vishwamitra et al., 2021). Unlike data from a single modality, multi-modal data can be perturbed across all modalities. Therefore, for robustness, multi- modal models should have the ability to resist attacks on both individual and multiple modalities. Nevertheless, experiments have suggested that mulli- models could perform badly when encountering perturbation (Noever & Noever, 2021) or missing modality (Yu et al. 2020 Ma et al.) 2022). Based these observations about the vulnerability of the multi-modal model, how to achieve a robust multi-modal model arises and attaches more focus.
To address this, previous methods improve the training strategies to obtain robust multi- modal model (Liang et al.) 2021 Ding et al. 2021). Specifically, certain studies extend uni-modal robust training strategies, like adversarial training and mixup, learn a discriminative multi-modal de- cision boundary (Li et al. 2022b; Maheshwari et al. 2023). Others take steps like cross-modal
"Corresponding author

---

Published as a conference paper at ICLR 2024
alignment (Tian & Xu 2021 to enhance the connection among modalities and obtain compact and robust multi-modal representation. However, even if they empirically improve robustness to some degree, they still lack in-depth theoretical analysis to understand the resilience of multi-modal models to perturbations, which is vital for safety-critical applications. More importantly, a universal phenomenon can be observed for these robust training methods that the adversarial attack on specific modalities could be more effective than others. As shown in Figure I] the P attack is more effective on modality #a than modality #o for both Joint Training and three widely used robust training methods on the Kinetics Sounds dataset.
To deeply understand multi-modal robustness and elucidate this phenomenon, we first depict the multi- modal decision boundary by integrating uni-modal representation margins. Then, we derive lower bound for the perturbation radius that the multi-moda model can consistently defend against. We discover that larger uni-modal margins coupled with reason- able integration are crucial for enhanced multi-modal robustness. With the above theoretical results, we investigate the pervasive issue (Wang et al.] [2000] Peng et al.) (022) that multi-modal models exhibit a pronounced preference for particular modality. This preferred modality profoundly influences the model's s decision, then resulting the above robust- ness phenomenon. On one hand, when a specific preferred modality is substantial enough to rely upon, the model will be reluctant to learn from other modal- ities, leading to the imbalance problem (Huang et al.. 2022; Wu et al., 2022). This imbalance problem hin- ders the enhancement of the uni-modal margin on the other modality, thus limiting the robustness. On the other hand, since the multi-modal model heavily relies on the preferred modality, the corresponding factor used in modality integration becomes larger, which will amplify the variation of the uni-modal margin in decision-makinn preferred modality is vulnerable, the multi-modaa model becomes more vulnerable to multi- attack. Further, this preference for vulnerable modality makes attacks on the preferred modality significantly more effective than other ones, explaining the observation in FFiuure T
65-
NZTTH PV
63-
50
Figure 1: Accuracy of different multi-modal robust training methods compared with Joint Training (JT) baseline under fg-PGD attack with range of radius for modality #u (vi- sion) and #a (audio) respectively on Kinetics Sounds dataset. Results show that all these methods are more vulnerable to attacks on the specific modality #a.
Hence, in case the
Since the essential components of robustness have an interrelation with each other, directly applying regulation is hard to guarantee higher robustness. To address this, we employ an orthogonal baasegd framework that formulates an alternative bound, which eliminates the interrelation and explicitly presents the integration. Building upon our theoretical analysis, we introduce two-step Certifiable Robust Multi-modal Training (CRMT) procedure to ensure progressively superior robustness. Initially, we redefine uni-modal margins related to the reliability of the modality in this framework. Then we propose to regulate the unreliable modality by enlarging its margin, which can alleviate the imbalanced problem brought by modality preference. Further, our approach adjusts the integration of modalities considering the improvement of certified bound. These steps not only mitigate the large gap between robustness against attack on each modality but also credibly guarantee higher multi- modal robustness. To validate our method, we conduct extensive experiments to present the advanced robustness against both uni-modal and multi-modal attacks. Our contributions are as follows:
1. We focus on a commonly used multi-modal model, offering invaluable insights into the essential components influencing multi-modas robustness.
2. We present analyses highlighting how multi-modal preference limits the multi-modal robust- ness and contributes to the vulnerability of multi-modal models towards specific modalities.
3. Drawing from our thcoretical findings, we introduce a two-step training procedure, alleviat- ing the limitation brought by modality preference. Our method can effectively enhance both performance and robustness over three real-world multi-modal datasets.
2 RELATED WORK
Multi-modal Robustness Analysis. Recent studies have highlighted the vulnerability of deep neural networks (DNNs) attacks and perturbations (Goodfellow et al.) 2014 Madry et al. (017).
2

---

Published as a conference paper at ICLR 2024
raising significant concerns regarding their deployment. With the presence of multiple modalities, the forms of perturbations include uni-modal attacks, modti- attacks (Schlarmann & Hein 2023), and modality missing (Lee et al., 2023). One possible way to enable the models to resist these perturbations is to design robust training strategies to obtain a reliable decision boundary that is distanced from samples (Li et al.
202T). Certain methods, such as multi-modal adversarial
training (Li et al. (2022b) and incorporating uni-modal tasks (Ma et al.) 20022). have proven pertinent in this context. On the other hand, studies focus on the latent consistency among modalities and propose to enhance modality interaction like alignment (T'a et al., 2018) to obtain compact and robust representation (Bednarek et al. 2020). Among these empirical works, we identify universal phenomenon that multi-modal models are commonly vulnerable to a certain modality {Liang et al., 202TD, which currently lacks a comprehensive theoretical explanation. This motivates us to theoreti- cally figure out the essential components determining the multi-modal robustness and explain the vulnerable modality phenomenon.
Multi- imbalance problem. Multi-modal learning is a significant approach for achieving a richer understanding across various tasks (Baltrušaitis et al. 2018 Jiang et al. 2021). However, even with the potential for a more comprehensive latent representation from multi-modal models (Huang et al., 2021), their performance might not always surpass that of the best uni-modal counterparts (Wang ee al. 2020, Huang et al. 2022). Due to the inherent differences between modalities, a multi-modal model might prefer or lean towards a modality that is easier to learn, potentially overlooking others (Wu et al. 2022). In response, various strategies are proposed to strengthen the learning of uni-modalities, thereby enhancing the generalization of multi-modal model (Wang et al., 2020 Peng ct al. 2022, Fan et al. 2023] Xu et al. (003b)). However, these researches do not elucidate how this modality preference problem impacts robustness against perturbations, which is our main focus.
Certified robustness. To describe the robustness of the model, certified robustness is introduced to describe the size of permissible perturbations that model can consistently defend against. Techniques such as randomized smoothing (Cohen et al. [019 Rosenfeld et al. 2020; Yang et al.] 2021] Salman
al., 2019) and interval bound propagation Zhang et al. [018]] Lyu et al. 2021 Xu et al. (2020) are
widely utilized to relax the model, which assists in determining the certificate bounds. Additionally,
the Lipschitz constant can serve as an intuitive metric to illustrate how perturbations influence the
decision of models (Weng et al. 2018 Leino et al. 2021). However, these methods are tailored for inputs with only a single modality but fail to focus on the primary concerns with multiple modalities.
In our research, we establish certified robustness for multi multi-modal modal models and determine the essential components influenced by modality preference that limit the robustness.
3 METHOD
3.1 PRELIMINARIES
Multi-modal Framework. We consider general K-way classification problem with vectorized input sample æ (x()).(1())) Ied++d2 consisting of two modalities, and ground truth label y E [K]. We consider the commonly used joint learning framework in multi-modal learning, where all of the modalities are projected into the shared space for downstream tasks (Huang et al.) 2022). Concretely, the uni-modal representations are extracted by encoders (('") and concatenated to form the joint representation, which is then mapped to the output space using a linear classifier, where W E IR (ddim(()))) ) (d)) and b E R are the weight matrix and bias respectively, and dim(-) represents the dimension. The logits output of the multi-modal model can be denoted as f((x) W WI o (zx()) + b w (x(1) + (22) (2)) (zx) + b, where W(m))
R K xd) is the part of classifier W related to the m-th modality.
Multi-modal robustness. To assess the certified robustness of a multi-modal model, we can measure the radius of the smallest perturbation that shifts sample to the decision boundary. In other words, any perturbation that falls within this radius can be defended against.
Definition 3.1. Suppose the multi-modal model h can correctly classify the sample æ, i.e. Vk +
y. (x) > hx (z), where h& denotes the logit score of the k-th class. The robustness radius of the multi-moda model towards the sample can be defined as:
P(x) min
I| x' 2'112
s.t. 3j + y, h2'))²) = hz(2')
(1)
x²'
3

---

Published as a conference paper at ICLR 2024
Mepresossation
Regularzanionn
Robusi
Oudps
10113
Concor
Playing vielin
iprco
$1.00
Regularinatinn
Modallty Encodor
Modaatyyspwojiic
Ondhogonal siarsjfar with wetglar
Clonafar
Encoder
Figure 2: Illustration of traditional multi-modal joint learning framework (Baltuuaitii et al. 2018 (left) and our framework introducing orthogonality into each uni-modal classifier (right). Our framework can be easily applied to explicit regularization achieve larger certified robustness.
Equation I seeks to identify the smallest perturbation, denoted as æ s,' that results in reaching the decision boundary between the ground truth and its nearest class j. Hence, any smaller perturbation can always be defended against. Here, we use -n-ormm to measure the radius, reflecting the overall size of the perturbation (Tsuzuku al. 2018 Carlini & Wagner 2017).
3.2 CERTIFIED ROBUSTNESS FOR MULTI-MODAL MODEL
In this section, we endeavor to uncover the essential components that impact multi-modal robustness. Referring to Figure I there is a notable variation in the effectiveness of perturbations on different modalities. This observation prompts us to distinguish the differences within each uni-modality. Therefore, we introduce the concept of uni-modal margin, which quantifies the distance between a uni-modal representation and the uni-modal decision boundary. To elaborate further, (W (mm)
)ō w!"")) classes y and k for m-th modality. The uni-modal representation margin can be defined as:
signifies that sample x (m) is positioned on the decision boundary between
Definition 3.2. Given the uni-modal encoder (((m) and the classifier W(m), the margin represen- tation space between ground truth y and other label k is defined as:
(W - (1)) &.
Jo (m))
(m) (xe (T))
W,
W
(m)) 4-
In this context, larger margin indicates the uni-modality is more reliable in distinguishing these two classes. Using this margin definition, we can re-examine the constraint conditions in [Equation which outline the perturbed sample located on the multi-modal decision boundary where y and the closest class j are tied in the output space:
hy((") hy(x') of ( (a)1)) + (2) 5) (x + 3, =
(3)
Inspired by Equation 3] we observe that the multi-modal decision boundary can be described as
integration of different uni-modal margins of the perturbed sample with factors cfTh)) ||
w/(")/2 ||2 and constant term 3; by by. The integration factors c.m")) quantify how variations in the uni mni-modal margin influence multi-modaa decisions. When perturbing a sample x towards the decision boundary, larger factor e)
(m) indicates altering the margin of the corresponding modality more effective. Meanwhile, when the sample perturbed, how the uni-modal margin varies also depends on the size of the perturbation. Thus, we propose to introduce the Lipschitz constant (m) for the uni-modal margin. This Lipschitz constant is the smallest constant limit the local variation range of certain function (Finlay al. 2018), which is given by:
((m) c. c. (mm))) (m)) (m)
[mmm
Then, we can provide the lower bound for the perturbation radius for multi-modaa robustness.
Theorem 3.3. Given an input x with ground-truth label E [K] and the closest label j + (( (m)) as the representation margin for m-th modality with Lipschitz constraint f(m)) and the
4

---

Published as a conference paper at ICLR 2024
integration factor Ej (m)) Define o' the perturbed sumple, and xr' as the perturbation. The lower bound for the perturbation radius can be described as:
1 (0 (x + (2) ( (x( +B3
P(x) min x x x'1122
VicTi 301 + (c) ; 22)
(5)
where jøy s.t. (x +
(x + Bg = 0.
The detailed proof can be found in the Appendix 7.1. Based on the certified bound above, we deduce that the multi-modal robustness depends three primary factors: uni-modal representation margins <Sm)) integration c!m)) and bias difference By. Firstly, robustness increases proportionally with the enhancement of the uni-modal representation margin <. Secondly, the integration cj is related to both uni-modi margins and the uni-modal Lipschitz constant, the expected integration for robustness requires considering both. Thus, a reasonable choice of integration factor can benefit higher robustness. Thirdly, when the sample is perturbed, the bias difference term 13 stays invariant, since it depends on class and j rather than the specific sample. Based on this fact, the bias difference considered in our analysis about model robustness. In nutshell, we recognize that uni-modal margins and the integration of modalities are two essential components. In the following section, we will analyze how these essential components vary and influence the robustness of multi-modal models, especially under modality preference.
3.3 ANALYSIS ABOUT MODALITY PREFERENCE FOR ROBUSTNESS
Since modalities have different amounts of information, some specific modalities might hold more significance than others in recision-making processes (Gat al., (021). As result, it is widely recognized that multi-modal models tend to show preference for and predominantly rely on specific modality (Huang et al. 2022. Wu al. 2022). However, this preference or over-reliance on the specific modality poses challenges for achieving robust multi-moda model. We delve deeper into how such preferences impact the two essential components of the certified bound in Eqqatiion 5]
Uni-modal representation margin. As shown in Equation 5 a larger uni- modal margin c) ((m) (xx (m)) determines higher certified robustness. However, when the learned information in the preferred modality is sufficiently reliable, the multi-modal model is reluctant to learn more information from other modalities (Wu et al. 2022; Huang et al., 2022). Thus, the modality preference leads to an imbalance problem hindering the development of uni-modal representations, resulting in narrower representation margin and ultimately constraining the certified robustness of the multi-modal model.
Integration of modalities. According modality preference, the decision of the multi-modal model highly depends on specific modality. Thus, considering the integration of modalities, the
(m) which could amplify the variation of the uni-modal margin in multi-modal decision-making Since the preference only determined by whether the modality with ideal discriminative ability, the multi-modal model
preferred modality contributes more and is allocated a larger integration factor cy
could prefer a vulnerable modality, which has a larger 7 ((m)) Thus, the perturbation for this preferred but vulnerable modality leads to larger variations in multi-modal margins, which is further amplified
in decision decision-making. Motivated by this phenomenon, we define (m) = cf ((m)) (mm)) as the vulnerability indicator of modality When the multi-modal model exhibits preference for vulnerable modality 1, there is significant imbalance in this indicator, with (1(1) >> 17(2), Consequently, we observe:
(1) ( + ( P(x) 2 Vin + (772)
(2)
+ +3, ~
(2) g 9 (o) + c9 53 (2) + BB
(6)
(n()
That is to say, the modality preference on vulnerable modality leaves an unreasonable imbalance on r7(m), thus the indicator. In this way, distinctly attacking the vulnerable modalities alone is enough to obscure the model. Here further provide the multi-modal robustness under uni-modal attack case:
robustness is highly dependent the modality with larger vulnerability
5

---

Published as a conference paper at ICLR 2024
Proposition 3.4. Following the setting in Theorem 3.3, w.Lo.g. considering the perturbation on 1-th modality x the lower bound for the uni-modal perturbation radius can be described as:
1 (x) + 6,
(x) (2) +
min - 2000
" where jty s.t. (F + cf (
(7)
=0
The lower bounds of different uni- uni-modal perturbations share identical numerators but differ in denominators, the vulnerability indicator (m()) The larger indicator on preferred modality reduces
the lower bound for perturbations on this modality, thus making attacks on this preferred modality more effective, explaining the observation in Figure I
3.4 CERTIFIABLE ROBUST MULTI- MULTI-MODAL TRAINING
Based on the above analysis, target to improve the uni-modal representation margin ()()) and adjust the integration of modalities o regulation for higher certified robustness. However, these regulations are intricately linked with the linear classifier W, potentially leading to conflicts in optimization objectives. As result, stably enhancing the certified robustness presents challenges. Additionally, determining the Lipschitz constant for the margin is computationally demanding. To address these challenges, linear classifiers as our framework. Detailedly, we ensure that in each modality, the class-specific
propose adopt orthogonality (Huang et al.) 2018) within uni-modal
vectors ( k [K] are unit and orthogonal. Since the valuable information between modalities different, we apply the weight (m) E R-K to lead the model focusing on more reliable modalities. Integrating these weights can enhance the model's ' ability to effectively utilize the valuable information from each modality. Therefore, learning of uni-modal representations and integration of modalities can be decoupled. The score corresponding to the k-th class can be expressed as:
he(()) = @
() (xx d vir(2)
(2)
(2) ()(2) (xx (2) + bR.
(8)
where (m)) E (XXdii((s is the matrix with orthogonal rows, satisfying W(m) (W
Ig- We can use (=((m) (m)) the uni-modal score, which represents whether the uni-modal J.
representation is well learned toward the k-th class. With this framework in place, we are able to
express the new Lipschitz constant (m) for the uni-modal score on class k in the m-th modality as:
(m(()) of (m)) (x:!)) w-(m)) (im()))(m))) (a (m))
S
#/ ((m))
(m)
x((m
(9)
With these definitions, we can derive the certified bound for the multi-modal perturbation radius, which is distinctly tailored to the framework employing orthogonal classifiers:
Theorem 3.5. Given an input x with ground-rruth label E [K] and the closest label j + y, the ((m) and orthogonal classifier W(m), the mrodllitysspectiie weight a (m), the Lipschitz constant {j
the difference of the bias 3, = by - by. The lower bound for the perturbation radius with the orthogonal based framework can be described as:
P(x) 2 Σm=1 (ay (m)) (mm) ((m)(((())) - aj
(mn) (m(m)) ((m)) (x())))
Em=l (m)) =(m) + (m) (mm))(mm))
(10)
a
where i 1 s.t. hy(x') -h (2x).
See Appendix 7.2 for proof. With this lower bound, can discuss designing regulation method to explicitly enhance the certified robustness. Firstly, as analyzed Section 3.3, the imbalance problem brought by modality preference impacts the unreliable modal representation margin. To explicitly regulate the uni-modal encoder and classifier independent from integration, we redefine the margin the difference in uni-modal score between ground truth and the other label, which also reflect the reliability of uni moi-mtdllity. Then we propose to enlarge the relatively small margin through regularization, which expressed follows:
Wr(m)) (((m) W((m) (()) (2)))a(((1)) (m))
max
min
(11)
Wmm) m) mikey

---

Published as a conference paper at ICLR 2024
Table Experimental results of adversarial accuracy results. Bold and underlined values indicate the top and the runner-up results. Combined with JTT, MMAT, and Mixup, our proposed CRMT-based methods can improve both performance and robustness.
Attack Datasets
w/o
FGM
E2 PGD UCF101
KS
UCF101
VGGS
KS
UCF101
VGGS
KS
VGGS
JT
0.643
0.742
0.496
0.288
0,506
0.157
0.268
0.431
0.056
GB
0.704
0.784
0.529
0.371
0.432
0.279
0.344
0.205
0.174
OGM
0.651
0.743
0.498
0.292
0.492
0.190
0.270
0.377
0.069
PMR
0.689
0.742
0.503
0.350
0.455
0.166
0.331
0.292
0.059
MSEFM
0.627
0.721
0.492
0.390
0.483
0.187
0.376
0.243
0.115
MMAT
0.656
0.728
0,509
0.514
0.609
0.413
0.507
0,598
0.409
Mixup
0.669
0.717
0.507
0.347
0.413
0.278
0.327
0.192
0.135
CMRT 0.758
0.789
0.526
0.491
0.515
0.248
0.468
0.433
0.124
CMRT-AT
0.762
0.759
0,538
0.608
0.614
0.422
0.602
0.602
0.414
CMRT-Mix
0.744
0.769
0.537
0.514
0.430
0.328
0.491
0.191
0.184
Based on Equation 11 we propose regularization term, refining the maximum using the LogSumExp function to facilitate better optimization:
N
Exsr exp(Ii/.)) (xm) (m(()))
L1
log
(12)
N
exp(W,) y. (mp ((m)) ( (x: (mm))
1=1
m==
which is detailed in the Appendix 7.3. Furthermore, we can adjust the integration of different modalities by enlarging the lower bound in Equation 10 Subsequently, we propose two-step training procedure called Certifiable Robust Multi-moda Training (CRMT), which can credibly obtain a robust multi-modal model. The training procedure of CRMT as follows:
Step 1: optimize with cross-entropy loss and margin regularization with term min (m)) W(() (0(3) pL1 + NV EN CE(((x:)), ys), where CE is the cross-entropy loss function.
Step 2: fix W (m²) ((m)) update a (m) to approach higher certified robustness:
min L2 N Σ r(x;), where r(x:) is the lower bound in Equation 10
As shown in Appendix 8.3, we further demonstrate that only one iteration of our method can achieve considerable robustness, without a huge consumption of training cost.
4 EXPERIMENTS
4.1
SETUPS
Dataset. We evaluate our method on different datasets including Kinetics-Sounds (Audio + Vision) (Arandjelovic & Zisserman 2017). UCF101 (Optical flow RGB) (Soomro al., 2012), and VGGSound (Audio + Vision) (Chen et al. 2020). We use the backbone ResNet18 (He et al. 2016) as the encoder for each uni mii-oodaityy. Details about these datasets are presented in Appendix 8.1.
Multi-modal models. In this study, the comparison methods are selected to improve multi-modal learning and be suitable for the multi-modal joint training strategy. Comparison methods can be divided into two distinct groups: Firstly, methods address the imbalance problem caused by modality preference: Gradient Blending (GB) (Wang et al. 2020), On-the-fly Gradient Modulation with generalization enhancement (OGM) (Peng et al. 2022). Prototypical Modal Rebalance (PMR) (Fan al., 2023). Secondly, methods aim at improving multi-modal robustness: Multi-Modal Adversarial Training (MMAT) (Li et al.] (022b). Multi-modal mixup (Mixup) (Madry et al. [017]] [i et al.) (0026)), MinSim+ExFMer (MSEFM) (Tian Xu [02T]). Our method can extended different training strategies, denoted as Certifiable Robust Mull--modal Training with Joint Training (CRMT-JT), CRMT with Adversarial Training (CRMT-AT), and CRMT with Mixup (CRMT-Mix).
Attack methods. Following previous work (Tsuzuku et al. 2018 (ingla et al. 2021). we select Fast Gradient Method (FGM) (Goodfellow et all,20114 and l, Projected Gradient Descent (E2- PGD) (adry et al. 2017) two attack methods with attack size 0.5, which are widely used
7

---

Published as a conference paper at ICLR 2024
Table Performance against distinct uni-modal attack methods on KS dataset.
o Clean
FGM
f2-PGD
Missing modality
Method
Attack
#v
#a
#v
#a
#v
#a
Bascline
JT
0.643
0.616
0.143
0.616
0.110
0.480
0.230
GB
0.704
0.658
0.195
0.656
0.157
0.483
0.430
Imbalance
OGM
0.651
0.624
0.147
0.624
0.116
0.463
0.207
PMR
0.689
0.649
0.181
0.649
0.152
0.484
0.315
MSEFM
0.627
0.577
0.320
0.576
0.314
0.466
0.239
Robustness
MMAT
0.656
0.633
0.390
0.632
0.369
0.482
0,262
Mixup
0.669
0.628
0.191
0.626
0.147
0.450
0.256
CRMT-JT 0.758
0.685
0.384
0.682
0.327
0.560
0.591
Ours
CRMT-AT
0.762
0.703
0.488
0.698
0.464
0.547
0.626
CRMT-Mix
0.744
0.706
0.377
0.704
0.320
0.572
0.566
for verifying multi-modal robustness with norno For uni-modal attack, we also introduce attacks FGM and P2D with e = 1.0, and missing on uni-modality.
4.2 ROBUSTNESS VALIDATION ON MULTI-MODAL. AND UNIAMODAL ATTACK
Robustness against multi-modal aftacks. We validate the robustness of our method under multi- modal attacks. Based on the experimental results presented in Table I we have identified four key observations that warrant attention. Firstly, while imbalance methods effectively enhance performance on clean samples, robustness methods can demonstrate more notable defense capabilities against vari- ous attacks. Secondly, the imbalance method GB can be superior to the robustness method MSEFM under some situations. That is because GB introduces a multi-task method to enhance the learning of uni- modaliyyy and improve the uni-modal margin, thus it can enhance the robustness methods analysis. Thirdly, our proposed CRMT-basee methods surpass the performance of the compared methods across these datasets in most cases. This superiority stems from our approach to addressing the imbalance problem through improving uni-m modil representation margins, and the certificate adjustment of modality-specific weights for heightened certified robustness. Fourthly, the improvement of results for CRMT-AT and CRMT-Mix suggests that training procedure serve as valuable component applicable to other robust training methods.
according
Robustness against distinct uni-modal attack. To substantiate the efficacy of our CRMT methods, we conducted additional experiments on the KS dataset, involving a broader range of different uni- modal attacks. As demonstrated in Table 2 the absence of modality #a leads to larger performance decline than #o, since it is more preferred by multi-modal models. As shown in Table 2 previous modal methods demonstrated poor performance when subjected to attacks on the preferred modality #a, aligning with our analysis. In contrast, our CRMT-based methods consider addressing the imbalance problem introduced by modality preference and adjusting integration, thus making the model more robust against uni-modal attacks. Furthermore, when encountering different uni-modal attacks, our CRMT-based approach shows superior performance and consistently ensures higher robustness in various scenarios. This strongly demonstrates the effectiveness and versatility of our proposed method in enhancing the robustness of multi-modal models.
(a) (M) /g(0) in MMAT.
(b) (b) in CRMT-AT stcp-1.
(0) in CRMT-AT.
Figure 3: Evaluation of the ratio of vulnerability indicators between modality #e and #a (preferred).
We illustrate the ratio in MMAT, CRMT-AT, and CRMT-AT with only the first training procedure.
8

---

Published as a conference paper at ICLR 2024
Table Extension results of adversarial accuracy with transformer-basel method on VGGS dataset.
Attack
Clean
Uni-modal attack
Multi-modal attack
Method
o/o
FGM #v
FGM #a
2 #v
22DPG #a
FGM
42DP
MMT 0,465
0.433
0.259
0,431 0,428
0.228 0.237
0.331
0.326
CRMT-MMT
0.471
0.440
0.276
0.344
0.340
Imbalance on uni-modal vulnerability indicators. We ver- ify the conclusion drawn in Section 3.3 that the imbalance prob- lem introduced by modality preference leads to the vulnerability of attack on specific modality, as illustrated in Figure 1 To achieve this, we compare MMAT with our CRMT-AT approach. As depicted in Equation 7| 17 1n(m) is utilized as measurement to assess the robustness of multi-modal models against uni- modal perturbations, and we employ the ratio of uni-modal vulnerability indicators (())/((a)))))) (see Figure 3). can be demonstrated that our approach diligently works to reduce the imbalance in the indicator 77. effectively darkening the heated map. Furthermore, according to the robustness accuracy, our ap- proach significantly reduces the disparity in attack performance between the two modalities (see FFiuure 4). This evidence serves to illustrate the efficacy our method mitigating the imbalance problem and ultimately enhancing multi-moda robustness.
Figure 4: This figure presents the robustness accuracy against uni- modal attacks with different sizes, where the dotted line signifies the difference in robustness accuracy between two modalidlsssss.
4.3 VALIDATION FOR EFFECTIVENESS AND SCALABILITY
Ablation studies. To delve deeper into the efficacy of our training procedure, we conduct an ablation analysis, aiming to elucidate the contribution of each component of the proposed training procedure toward the overall result. When compared with the baseline Joint Training (JT), our results include the orthogonal- based framework and our proposed training procedure. We report findings related to solely using the orthogonal framework (OJT), CRMT focusing exclusively on each step (CRMT-step- 1 and CRMT-step-2), and full proposal, CRMT-JT, which incorporates both steps. As depicted in FFigure 5) it is evident that the orthogonal- framework does not contribute to improved robustness. Meanwhile, both step-1 and step-2 of the training procedure can enhance robustness, particularly in the face of larger- size attacks. Furthermore, the results highlight that each step consistently enhances
0.7
CRMIT-opp
CRNT oops CINTUIT
0.
Figure Ablation studies of our methods on the UCF101 dataset, revealing the effect each part we introduced.
robustness, making CRMT-JT superior in multi- modal robustness.
Extension studies to transformer. We further provide experiment results our methods extended to the Multi-Modal Transformer-based framework with hierarchical attention (Xu et al.) 2023a) (MMT) on the VGGS dataset. Both visual and audio class tokens are concatenated and linearly projected into the output space. We mainly evaluate the robustness under the Transformer model accurately, hence we train from scratch instead of using the pre-trained model. To validate our method, only introduce the CRMT procedure step-1 I this transformer architecture as comparison. That is because both audio and visual input influence each token, which is beyond our assumption. Based on the results in Table 3] our method also improve the robustness of transformer- architecture in most cases, which proves the broad prospects of our approach.
CONCLUSION
In this study, we present the essential components for modtt robustness and delve into the limitations imposed by modality preference. Additionally, explore why multi-modal models exhibit vulnerabilities to attack specific modalities. Further, we introduce the Certifiable Robust Multi- modal Training procedure, novel approach explicitly designed to boost the certified robustness.

---

Published as a conference paper at ICLR 2024
6 ACKNOWLEDGEMENT
This research was supported by National Natural Science Foundation of China (NO.62106272), the Young Elite Scientists Sponsorship Program by CAST (2021QNRC001), and Public Computing Cloud, Renmin University of China.
REFERENCES
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pp. 2425-2433, 2015.
Relja Arandjelovic and Andrew Zisserman. Look, listen and learn. In Proceedings of the IEEE international conference on computer vision, pp- 609-617, 2017.
Tadas Baltrušaitis, Chaitanya Ahuja, and Louis- Morency. Multimodal machine learning: survey and taxonomy. IEFE transactions on pattern analysis and machine intelligence, 42): 423-443, 2018.
Michal Bednarek, Piotr Kicki, and Krzysztof Walas. On robustness of multi-modal fusion-robonn perspective. Electronics, 9(7):1152, 2020.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 ieee symposium security and privacy (sp), pp. 39-57. Icce, 2017.
Honglic Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. Vggsound: A large-scale audio- visual dataset. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 721-725. IEEE, 2020.
Vishal Chudasama, Purhayan Kar, Ashish Gudmalwar, Nirmesh Shah, Pankaj Wasnik, and Naoyuki Onoe. M2fnet: Multi-modal fusion network for emotion recognition conversation. In Proceed- ings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 4652 4661, 2022.
Jeremy Cohen, Elan Rosenfeld, and Zico Kolter. Certified adversarial robustness via randomized smoothing. In international conference on machine learning, pp. 1310-1320. PMLR, 2019.
Wenhao Ding, Baiming Chen, Bo Li, Kim Eun, and Ding Zhao Multimodal safety-critics scenarios generation for Docision-making algorithms evaluation. IEEE Robotics and Automation Letters, 6(2):1551-1555) 2021.
Yunfeng Fan, Wenchao Xu, Haozhao Wang. Junxiao Wang, and Song Guo. Pmr: Prototypical modal rebalance for multimodal learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 20029-20038, 2023.
Chris Finlay, Adam M Oberman, and Bilal Abbasi. Improved robustness adversarial examples using lipschitz regularization of the loss. 2018.
Itai Gat, Idan Schwartz, and Alex Schwing. Perceptual score: What data modalities does your model perceive? Advances in Neural Information Processing Systems, 34:21630-21643, 2021.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6577 2014.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition. pp. 770-778, 2016.
Lei Huang, Xianglong Liu, Bo Lang, Adams Yu, Yongliang Wang, and Bo Li. Orthogonal weight normalization: Solution to optimization over multiple dependent stiefel manifolds in deep neural networks. In Proceedings of the AAAI Conference Artificial Intelligence, volume 32, 2018.
10

---

Published as a conference paper at ICLR 2024
Yu Huang, Chenzhuang Du, Zihui Xue, Xuanyao Chen, Hang Zhao, and Longbo Huang. What makes multi-modes learning better than single (provably). Advances in Neural Information Processing Systems, 34:10944- 2021.
Yu Huang, Junyang Lin, Chang Zhou, Hongxia Yang. and Longbo Huang. Modality competition: What makes joint training of multi-modal network fail in deep learning (provalll) arXiv preprint arXiv:2203. 12221, 2022.
Xingyu Jiang, Jiayi Ma, Guobao Xiao, Zhenfeng Shao, and Xiaojie Guo. A review of multimodal image matching: Methods and applications. Information Fusion, 73:22-71, 2021.
Deepak Kumar, Chetan Kumar, Chun Wei Seah, Siyu Xia, and Ming Shao. Finding achilles' heel: Adversarial attack on multi-modal action recognition. In Proceedings of the 28th ACM International Conference on Multimedia, pp. 3829-38377 2020.
Yi-Lun Lee, Yi-Hsuan Tsai, Wei-Chen Chiu, and Chen-Yu Lee. Multimodal prompting with missing modalities for visual recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14943-14952, 2023.
Klas Leino, Zifan Wang, and Matt Fredrikson. Globally-robust neural networks. In International Conference on Machine Learning, pp. 6212-6222. PMLR, 2021.
Guangyao Li, Yake Wei, Yapeng Tian, Chenliang Xu, Ji-Rong Wen, and Di Hu. Learning to answer questions in dynamic audio-visual scenarios. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19108-19118, 2022a.
Juncheng B Li. Kaixin Ma, Shuhui Qu, Yao Huang, and Florian Metze. Audio-visual event recognition through the lens of adversary. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 616-620. IEEE, 2021.
Juncheng B Li, Shuhui Qu, Xinjian Li, Po-Yao Bernie Huang, and Florian Metze. On adversar- ial robustness of large-scale audio visual learning. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 231-235. IEEE, 2022b.
Paul Pu Liang, Yiwei Lyu, Xiang Fan, Zetian Wu, Yun Cheng, Jason Wu, Leslie Chen, Peter Wu, Michelle A Lee, Yuke Zhu, al. Multihench: Multiscale benchmarks for multimodal representation learning, arXiv preprint arXiv:2107.0755 2021.
Zhaoyang Lyu, Minghao Guo, Tong Wu, Guodong Xu, Kehuan Zhang, and Dahua Lin. Towards evaluating and training verifiably robust neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4308-4317, 2021.
Mengmeng Ma, Jian Ren, Long Zhao, Davide Testuggine, and Xi Peng. Are multimodal transformers robust missing modality? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18177-181 186, 2022.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv: 1 1706006083 2017.
Harsh Maheshwari, Yen-Cheng Liu, and Zsolt Kira. Missing modality robustness in semi-supervised multi-modal semantic segmentation. arXiv preprint arXiv:2304. 2023.
David A Noever and Samantha E. Miller Noever. Reading isn't believing: Adversarial attacks on multi-modal neurons. arXiv preprint arXiv:2103.10480, 2021.
Xiaokang Peng, Yake Wei, Andong Deng, Dong Wang, and Di Hu. Balanced multimodal learning via on-the-fly gradient modulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8238-8247 2022.
Elan Rosenfeld, Ezra Winston, Pradeep Ravikumar, and Zico Kolter. Certified robustness to label- flipping attacks via randomized smoothing. In International Conference on Machine Learning, pp. 8230-8241. PMLR, 2020.
11

---

Published as a conference paper at ICLR 2024
Hadi Salman, Jerry Li, Ilya Razenshteyn, Pengchuan Zhang, Huan Zhang, Sebastien Bubeck, and Greg Yang. Provably robust deep learning via adversarially trained smoothed classifiers. Advances in Neural Information Processing Systems, 32, 2019.
Christian Schlarmann and Matthias Hein. On the adversarial robustness of multi-modal foundation models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3677-3685, 2023.
Sahil Singla, Surbhi Singla, and Soheil Feizi Improved deterministic 12 robustness on cifar- and cifar-100 arXiv preprint arXiv:
wwx.iv2218..000 108, 040.0 2021.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv: 2012.
Yapeng Tian and Chenliang Xu. Can audio-visual integration strengthen robustness under multimodal attacks? In Proceedings of the IEEF/CVF Conference Computer Vision and Puttern Recognition, pp. 0pp55601-611, 2021.
2021.
Samarth Tripathi, Sarthak Tripathi, and Homayoon Beigi. Multi-modal emotion recognition on iemocap dataset using deep learning. arXiv preprint 1xXiv:1804.05788. 2018.
Yao-Hung Hubert Tsai, Paul Pu Liang, Amir Zadeh, Louis-Philippe Morency, and Ruslan Salakhutdi- nov. Learning factorized multimodal representations. arXiv preprint arXiv: 10XX.00000006700 2018.
Yusuke Tsuzuku, Issei Sato, and Masashi Sugiyama. Lipschitz-margin training: Scalable certification of perturbation invariance for deep neural networks. Advances in neural information processing systems, 31, 2018
Nishant Vishwamitra, Hongxin Hu, Ziming Zhao, Long Cheng, and Feng Luo. Understanding and measuring robustness of multimodal learning, arXiv preprint arXiv:2112.12792 2021.
Weiyao Wang, Du Tran, and Matt Feiszli. What makes training multi-modal classification networks hard? Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pp. 12695-12705, 2020.
Yake Wei, Di Hu, Yapeng Tian, and Xuelong Li. Learning in audio-visual context: A review, analysis, and new perspective. arXiv preprint arXiv:2208.0955 2022.
Lily Weng, Huan Zhang, Hongge Chen, Zhao Song, Cho-Jui Hsieh, Luca Daniel, Duane Boning, and Inderjit Dhillon. Towards fast computation of certified robustness for relu networks. In International Conference on Machine Learning, pp. 5276-5285. PMLR, 2018.
Nan Wu, Stanislaw Jastrzebski, Kyunghyun Cho, and Krzysztof J Geras. Characterizing and overcoming the greedy nature of learning in multi-modal deep neural networks. In International Conference on Machine Learning, pp. 24043-24055. PMLR, 2022.
Kaidi Xu, Zhouxing Shi, Huan Zhang, Yihan Wang, Kai-Wei Chang, Minlie Huang, Bhavya Kailkhura, Xue Lin, and Cho-Jui Hsieh. Automatic perturbation analysis for scalable certi- fied robustness and beyond. Advances in Neural Information Processing Systems, 33:1 33:1129-1144 1141, 2020.
Peng Xu, Xiatian Zhu, and David A Clifton. Multimodal learning with transformers: survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023a.
Ruize Xu, Ruoxuan Feng, Shi-Xiong Zhang, and Di Hu. Mmcosine: Multi-modal cosine loss towards balanced audio-visual fine-graine grained learning. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023b.
Pinci Yang, Xin Wang, Xuguang Duan, Hong Chen, Runze Hou, Cong Jin, and Wenwu Zhu. Avqa: A dataset for audio-visual question answering on videos. In Proceedings of the 30th ACM International Conference on Multimedia, pp. 3480-3491, 2022.
Zhuolin Yang, Linyi Li, Xiaojun Xu, Bhavya Kailkhura, Tao Xie, and Bo Li. On the certified robustness for ensemble models and beyond, arXiv preprint arXiv: 10873, 2021.
12

---

Published as a conference paper at ICLR 2024
Youngjoon Yu, Hong Joo Lee, Byeong Cheon Kim, Jung Uk Kim, and Yong Man Ro. Investigating vulnerability to adversarial examples on multimodal data fusion in deep learning. arXiv preprint arXiv:2005.10987, 2020.
Huan Zhang, Tsui-Wei Weng, Pin-Yu Yu Chen, Cho-Jui Hsich, and Luca Daniel. Efficient neural network robustness certification with general activation functions. Advances in neural information processing systems, 31, 2018.
13


Reviews:
[{"review_id": "Ky9FfK2aQH", "reviewer": "Reviewer_yeWY", "summary": "In this paper, the authors tackle the challenge of improving the robustness of multi-modal models against perturbations, such as uni-modal attacks and missing modalities. They provide valuable theoretical insight, emphasizing the importance of larger uni-modal representation margins and reliable integration for modalities in achieving higher robustness. They introduce a training procedure, Certifiable Robust Multi-modal Training (CRMT), which effectively addresses modality preference imbalances and enhances multi-modal model robustness. Experimental results validate the superiority of CRMT in comparison to existing methods, demonstrating its versatility and effectiveness. Overall, this paper contributes to the field by providing a theoretical foundation and a practical method for enhancing the robustness of multi-modal models.", "questions": "N/A", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Overall, the paper advances multi-modal robustness understanding and presents a practical solution in CRMT with strong empirical results, and the potential for broader applications in the ML/Multimodal community.\n- The paper offers a fresh perspective on multi-modal robustness, emphasizing the importance of larger uni-modal representation margins and reliable integration within a joint multi-modal framework.\n- The research is methodologically sound, with well-designed experiments and clear presentation.\n- The authors effectively communicate complex concepts, enhancing accessibility.", "weaknesses": "The paper included some results on transformer as fusion models, particularlly the Multi-Modal Transformer-based framework with hierarchical attention on the VGGS dataset. However, all experriments, especailly the one with transformerr adopt training from scratch and did not consider any pre-training strategies, such as uni-modal pretraining, or multi-modal pretraining. It will be interesting to consider such methods as baselines and also to see how much CRMT can help to improve. \n\nAlso, except for experimenting, it will be good if authors can discuss how does their method generalize to other fusion mechanisms, besides late fusion.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uuPktQCWhe", "reviewer": "Reviewer_zQSE", "summary": "The paper studies adversarial robustness for multi-modal learning by building a new lower bound for the perturbation radius through uni-modal margins and the Lipschitz constraint. Based on the proposed lower bound, a two-step adversarial training framework has been provided to improve the robustness of multi-modal learning. Experimental results on three benchmark datasets were provided regarding multiple attack methods, compared with several strong baselines.", "questions": "Please refer to the questions raised in the *Weakness* section. Plus, the reviewer is interested in the following questions:\n- Can the proposed method apply to multiple modalities larger than 2?\n- What's the selection criterion in choosing datasets for the experiment? \n- Are the provided theoretical results applicable to vision-text data? Any empirical evidence? \n- Could the proposed method be incorporated into the pre-trained multi-modal model (e.g., CLIP or BLIP)?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- **New findings**: While discussing a new lower bound with the Lipschitz constraint is nothing new for adversarial robustness, the proposed method provides theoretical and insightful analyses of how the attack on a preferred modality would impact the overall robustness. This is a practical and common problem in multi-modal integration, as one modality often dominates the others. \n- **Good presentation**: The reviewer enjoyed reading the presentation of the proposed method, where each step was well demonstrated with theoretical supports and clearly developed through proper treatments. One minor suggestion is to provide a pseudo `Algorithm` to outline the method, as well as add a `Remark` to better summarize and explain the training steps. \n- **Decent experiment design**: Despite some minor issues, the experiment is overall well-designed and sufficient by 1) comparing with two groups of strong baselines, 2) adopting multiple attack methods (e.g., FGM, PGD, and missing modality), and 3) providing detailed ablation study and model discussions.", "weaknesses": "- **Missing implementation details**: I may have missed something; however, I did not find any implementation details about the multi-modal encoders. What are the backbones used in the experiment? Can the proposed method apply to different backbones? \n- **Unclear model-specific weights/classifiers**: The exact role of introducing model-specific weights $a^{(m)}$ is somewhat unclear to me. How will it be used to guide the orthogonal classifier of each modality? Also, it remains unclear to me how the proposed eventually gets the prediction result upon different modalities's classifiers. \n- **Lacking empirical evidence**: One main motivation of the proposed approach is one modality may be more vulnerable than the others.  While the adversarial accuracy (between uni-modal and multi-modal attacks) could support this observation empirically, it would be more convincing to provide more evidence that can be used to back-up the theoretical results, such as plotting the vulnerability indicator ($\\eta$) values, visualizing the perturbation radius over modalities, etc.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yefVXVRptk", "reviewer": "Reviewer_BCRJ", "summary": "This work employs an orthogonal-based framework that formulates an alternative bound, eliminating the interrelation and explicitly presenting integration.  Building on the theoretical analysis, they introduce a two-step procedure called Certifiable Robust Multi-modal Training (CRMT) to progressively enhance robustness.", "questions": "No", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(1) Following a more comprehensive analysis, the researchers furnish compelling evidence that demonstrates the constraining effect of multi-modal preference on the robustness of multi-modal systems, which contributes to the vulnerability of multi-modal models to specific modalities.\n\n(2) Building upon their theoretical insights, they present a two-step training protocol designed to alleviate the limitations stemming from modality preference. The suggested approach significantly enhances both the performance and robustness of multi-modal models across different real-world multi-modal datasets.", "weaknesses": "(1) Since FGM and PGD are the two white-box attacks chosen in the adversarial robustness experiments, why not consider the stronger white-box Auto Attack?  It is suggested to add the experiment results about Auto Attack in Section4.\n\n(2) \"Robustness against multi-modal attacks\" mentioned In Section 4.2, since multi-modal attacks are considered, the experimental results in Table 1 only consider single-mode attacks (#a,#v). Is the method proposed in this paper effective when co-attacks (both modality attacks) are existing?  In [1,2], more effective multi-modal attack methods are proposed than uni-modal(such as #a and #v) attack. Can the proposed method effectively resist these multi-modal attack methods? It is suggested that the relevant experiments should be added to Section 4.2, otherwise the conclusion of \"Robustness against multi-modal attacks\" is somewhat not convincing.\n\n\n[1]\tZhang J, Yi Q, Sang J. Towards adversarial attack on vision-language pre-training models[C]//Proceedings of the 30th ACM International Conference on Multimedia. 2022: 5005-5013.\n[2]\tLu D, Wang Z, Wang T, et al. Set-level Guidance Attack: Boosting Adversarial Transferability of Vision-Language Pre-training Models[J]. arXiv preprint arXiv:2307.14061, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kNbacmHtzc", "reviewer": "Reviewer_hY8P", "summary": "This manuscript concerns the robust multi-modal representation learning, which are positioned well away from the discriminative multi-modal decision boundary. To address this issue, they theoretically discover that larger uni-modal representation margins and more reliable integration for modalities are essential components for achieving higher robustness. Inspired by our theoretical finding, we introduce a training procedure called Certifiable Robust Multi-modal Training (CRMT), which can alleviate this influence from modality preference and explicitly regulate essential components to significantly improve robustness in a certifiable manner. Experiments validate the effectiveness.", "questions": "refer to the weakness", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe multi-modal robustness learning is meaningful and challenging. The paper is well-written and the proposed method is easy to understand.\n2.\tThe authors theoretically discover that larger uni-modal representation margins and more reliable integration for modalities are essential components for achieving higher robustness.\n3.\tExperiments on various datasets validate the proposed method.", "weaknesses": "The manuscript claims that they focus on the commonly used joint multi-modal framework, more multi-modal fusion method, and different multi-modal backbones should be compared. For example, the early fusion, and hybrid fusion strategy. On the other hand, different modalities can employ various backbones, the reviewer is curious about the influence of different backbones, and more ablation studies are expected.\n\nIn related work and comparison methods, more state-of-the-art multi-modal robustness approaches should be introduced and compared.\n\nHow can this setup be extended to three modalities? More explanations and experiments are needed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)