RollingQ: Reviving the Cooperation Dynamics in Multimodal Transformer
Authors: HaoTian Ni
Yake Wei
Hang Liu
Gong Chen
Chong Peng
Hao Lin
Di Hu
Venue: ICML 2025 poster (2025)
Keywords: Multimodal Learning
Dynamic Fusion
Modality Imbalance
Primary area: deep_learning

Abstract: Multimodal learning faces challenges in effectively fusing information from diverse modalities, especially when modality quality varies across samples. Dynamic fusion strategies, such as attention mechanism in Transformers, aim to address such challenge by adaptively emphasizing modalities based on the characteristics of input data. However, through amounts of carefully designed experiments, we surprisingly observed that the dynamic adaptability of widely-used self-attention models diminishes. Model tends to prefer one modality regardless of data characteristics. This bias triggers a self-reinforcing cycle that progressively overemphasizes the favored modality, widening the distribution gap in attention keys across modalities and deactivating attention mechanism's dynamic properties. To revive adaptability, we propose a simple yet effective method Rolling Query (RollingQ), which balances attention allocation by rotating the query to break the self-reinforcing cycle and mitigate the key distribution gap. Extensive experiments on various multimodal scenarios validate the effectiveness of RollingQ and the restoration of cooperation dynamics is pivotal for enhancing the broader capabilities of widely deployed multimodal Transformers. The source code is available at https://github.com/GeWu-Lab/RollingQ_ICML2025.

RollingQ: Reviving the Cooperation Dynamics in Multimodal Transformer
Haotian Ni Yake 2e4
Hang Liu Gong Chen Chong Peng Hao Lin 6
Di Hu 234
Abstract
trušaitis et al., 2018; Zhang et al., 2024) to achieve a com- prehensive understanding. Each modality provides unique information, and the challenge of effectively fusing these modalities has been emphasized in prior works (Gao et al., 2020; Liang et al., 2022). Current research primarily ex- plores two fusion paradigms: static fusion and dynamic fu- sion (Zhang et al., 2024). Static fusion applies fixed weights to different modalities during inference (illustrated Fig- ure (a)), assuming that the information provided by each modality remains consistent (Zhang et al., 2024). However, this assumption often breaks down in real-world scenar- ios, where modality quality can vary significantly across samples. Dynamic fusion, by contrast, directly addresses this issue by dynamically adjusting weights to each modal- ity based on the specific characteristics of the input data (illustrated in Figure 1(b)). To enable dynamic fusion, mul- timodal Transformers (Vaswani, 2017: Xu et al., 2023) have emerged a powerful scheme, leveraging the inherent at- tention mechanism identify and focus the most infor- mative and task-relevant tokens in the input (Clark, 2019; Kovaleva, 2019). Using self-attention and cross-attention fusion layers, these models can adaptively and effectively integrate information from different modalities, achieving superior performance tasks such as audio-visual learn- ing (Nagrani et al., 2021; Chumachenko et al., 2022), senti- ment analysis (Wankhade et al., 2022), and visual question answering (Yu et al., 2019).
Multimodal learning faces challenges in effec- tively fusing information from diverse modalities, especially when modality quality varies across samples. Dynamic fusion strategies, such as at- tention mechanism in Transformers, aim to ad- dress such challenges by adaptively emphasiz- ing modalities based on the characteristics of in- put data. However, through amounts of carefully designed experiments, we surprisingly observed that the dynamic adaptability of widely-used self- attention models diminishes. Model tends to pre- fer one modality regardless of data characteristics. This bias triggers a self-reinforcing cycle that pro- gressively overemphasizes the favored modality, widening the distribution gap in attention keys across modalities and deactivating attention mech- anism's dynamic properties. To revive adaptabil- ity, we propose a simple yet effective method Rolling Query (RollingQ), which balances atten- tion allocation by rotating the query to break the self-reinforcing cycle and mitigate the key dis- tribution gap. Extensive experiments on various multimodal scenarios validate the effectiveness of RollingQ and the restoration of cooperation dynamics is pivotal for enhancing the broader ca- pabilities of widely deployed multimodal Trans- formers. The source code is available at Github.
To validate the practical performance of static and dynamic fusion paradigms, we conducted experiments on the widely used audio-visual dataset, Kinetic-Sound (Arandjelovic Zisserman, 2017). Surprisingly, dynamic fusion imple- mented through a self-attention layer, often considered more flexible and adaptive, achieves an accuracy of 67.0 00, which underperforms the static fusion's 's accuracy of 68.0'. To in- vestigate this unexpected outcome, we first examine whether this attention- based dynamic fusion paradigm can adaptively adjust weights as anticipated, by analyzing the attention scores in the model's fusion layer. However, as shown in Figure I(c), the model allocates disproportionately high attention to the audio modality, regardless of data character- istics. To further verify this behavior, we replaced the audio input with Gaussian noise, which contains no information.
1. Introduction
Multimodal learning focuses on extracting and integrat- ing information from data across different modalities (Bal-
BBeihang University, Beijing, China Gaoling School of Arti- ficial Intelligence Renmin University of China, Beiing, China "Beijing Key Laboratory of Research on Large Models and Intelligent Governance, Beijing, China 'Engineering Research Center of Next-Generation Intelligent Search and Recommen- dation. MOE, Beijing, China "Xiamen University. Xiamen, China Tennent, Shenzhen, China. Correspondence to: Di Hu <dihu @ rww...u.u.cc.........
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
'See Table 1 for more details.

---

RollingQ: Reviving the Cooperation Dynamics in Multimodal Transformer
2
aa
KineticSound
KineticSound
u n H
w" x
H. wº, X + w X
140
sum
1.4
concer
1.0
wa E IRd
w x x X Rd
w" E Rdd
w" == X X - R
.908 +8.9 +6.0
+0.20 060 325
100
x2
X (b) Dynamic Fusion
(a) Static Fusion
(c) Attention Score
(d) Key Distribution
Figure 1. (a): Illustration of multimodal static fusion, where the modality weights w" and w" are fixed values after learned. (b): Illustration of multimodal dynamic fusion, where the modality weights w and w" are determined by the input data x" and x". (c): Attention scores assigned to each modality on the Kinetic-Sound dataset( (randel(Aradiielov & Zisserman, 2017). The left half shows normal samples, while the right half shows samples where the audio modality's 's input is replaced with Gaussian noise. (d): Distribution of keys in the attention layer for different modalities Kinetic-Sound. The red dots represent the keys when the audio modality is replaced with Gaussian noise.
Despite this, the model remains excessive attention to the perturbed audio modality in Figure I(c), indicating that such dynamic fusion paradigm is overly biased and fails to adapt attention score according to the informativeness of different modalities. This breakdown in multimodal attention adaptiv- ity undermines the intended advantages of dynamic fusion and degrades overall model performance.
cretely, could facilitate dynamic multimodal coopera- tion by mitigating the over-reliance on the biased modality, This Rolling Query (RollingQ) algorithm rotates the cur- rent query towards an anchor, which is designed to allocate higher attention scores to the unbiased modality rather than the biased one. After rotating the query to the anchor, the learning of this new query will encourage the key distri- bution across modalities to narrow the gap between them. Consequently, during optimization, the formerly underuti- lized modality gains more momentum, enabling it to ob- tain more informative features. This rebalancing gradually equalizes the quality of features across modalities. Further- more, we validate the effectiveness of our Rolling strategy through extensive experiments and provide detailed obser- vations demonstrating how RollingQ restores the dynamic adaptability of multimodal Transformers.
To investigate the underlying cause of this counterintuitive phenomenon, we analyzed the distribution of attention keys for each modality and their cosine similarity with the query of the class token, which directly determines the attention scores. As shown in Figure ((d), the degradation of the dynamic property in the attention mechanism arises from a significant disparity in the attention key distributions across modalities. A modality is biased: the query of the class to- ken, which determines the prediction of the model, remains significantly similar to the keys of the biased modality even when it contains no information. To further understand the factors driving this distribution gap, we performed both the- oretical analysis and empirical verification. Our findings reveal self-reinforeing cycle driven by the greedy nature of multimodal learning (Wu et al., 2022), where the model con- tinuously emphasizes the biased modality in feed-forward stage and optimizes its features through backward propa- gation. Over time, this feedback loop exacerbates the dis- tribution disparity: the biased modality accumulates higher attention scores and more informative features, while other modalities receive diminished attention and remain under- optimized. As a result, regardless of the data characteristics, the biased modality consistently receives higher attention scores due to the distribution gap, thereby significantly im- pairing the attention mechanism's 's dynamic adaptability.
Our contributions are summarized as follows: identify the deactivation of the dynamic property in multi- modal Transformers, which we attribute to a self-reinforcing cycle during training, resulting in a significant distribu- tion gap between the attention keys of different modali- ties. Secondly, we propose a simple but effective algorithm, Rolling,, which effectively disrupts this self-reinforcing cycle by rotating the query and rebalancing the attention mechanism. F Finally, through extensive experiments, we demonstrate that RollingQ not only restores the cooperative dynamics of multimodal Transformers but also significantly improves their performance across diverse datasets.
Firstly, we
2. Related Works
2.1. Multimodal Learning
To break the self-reinforcing cycle and revive the cooper- ative dynamics of multimodal Transformers, we propose simple yet effective strategy: Rotating the Query! Con-
Multimodal learning aims to leverage data from multiple modalities, where the fusion paradigm serves as the core to integrate multimodal information (Gao et al., 2020). Based
2

---

RollingQ: Reviving the Cooperation Dynamics in Multimodal Transformer
on the ideology of aggregate information within the fea- ture space, several works explore various fusion schemes to effectively combine features under a late fusion struc- ture (Zadeh et al., 2017; Liu et al., 2018). Besides, instead of simply aggregating, some works focus on enhancing the multimodal interaction thus fusing information within modalities at an earlier stage. Specifically, they have in- vented special transfer modules or exchange mechanisms between encoders to introduce interaction (Joze et al., 2020; Wang et al., 2020b). Meanwhile, behind these fusion struc- tures, researchers found that there is always modality favored by the model, hence it dominate the overall learning process, leading to imbalance multimodal learning ( (Wang et al., 2020a; Huang et al., 2022). To address this, several methods have been proposed to mitigate learning imbalance through optimization adjustments, data filtration, and spe- cialized objectives (Peng et al., 2022; Fan et al., 2023; Xia et al., 2023; Wei et al., 2024; Yang et al., 2024; 2025; Huang et al.. 2025). However, these approaches primarily focus on the static fusion paradigms. In this work, we examine a more complex paradigm, dynamic fusion, investigating the side effects of imbalanced feature quality on both attention score allocation and encoder optimization.
their dynamic capabilities and leads to better performance.
3. Method
3.1. Preliminary
In this work, we focus on exploring the attention mechanism within multimodal Transformers. To simplify the analysis and emphasize the core dynamics of attention, we use a single self-attention layer, which is both representative and widely adopted for theoretical analysis. As for different fusion paradigms within multi-layer Transformer, we also provide experiment verifications in Appendix C.4. For con- venience, we denote the dataset as D = {z1, ye Beeie,, (1)) where each sample in the dataset consists of a pair of differ- ent modalities H; = (xf,, xx),, using audio and visual data here for example. Two encoders are denoted by h", $", with parameter f*, 0", so the transformed features 2", z" follows 2!" Th' ( m E {a, In order ap- ply the attention mechanism for fusion, the modality fea- tures should be a sequence of tokens with embedding di- mension d and sequence length LTd for each modality, i.e. zi"h E R RR.." xd . E {aa v}. We follow Dosovitskiy (2020) and add a special [elass] token as a query vector denoted as Zets Ed, which will eventually be used as the output feature for task completion.
2.2. Multimodal Transformers
Multimodal Transformers (Xu et al., 2023), which leverage the attention mechanism to integrate information from mul- tiple modalities, are widely used across various fields since they can dynamically select informative and task-relevant tokens to accomplish the task. Some works directly con- catenate the input sequence of different modalities and pass them together into the Transformer blocks (Li et al., 2019; Kim et al., 2021). However, identifying the redundancy and computational expense associated with the long input sequence, others focus on structural modifications unlock the full potential of Transformer models (Gao et al., 2019; Tsai et al., 2019; Wu et al., 2021). For example, some ap- proaches use Transformer blocks to fuse features provided by two unimodal encoders (Chumachenko et al., 2022). Na- grani et al. (2021) and Recasens et al. (2023) limit modality interactions to specific tokens, thus compressing the interac- tion information for greater efficiency. Further, identifying that some tokens may be irrelevant to the task, Wang al. (2022) propose a specialized scoring network to evaluate and select the most informative tokens for task completion.
Within the attention layer (Vaswani, 2017), we have ma- trix WQ, W W.V E gdxdd that map current input into queries, keys, and values, respectively. We denote these as KTn = 2 W K Q!T = w," V.m = 2, 2m"V m E {a, v). Specifically, we use q = 2dl We E Rd denote the query for brevity. Hence, the output of the fusion layer can be simplified as follows:
A, 9KT vd
(1)
h; = softmarl (A,)V,
(2)
where K; = [Ko, K;), V₁ = V..Vy.). Finally, we use a linear classifier f(-) to obtain the prediction &N = fh,).
From Equation 1, we know that the attention score for each modality is determined by the dot product of the q and K?". Since Km E RL xd is sequence of keys, we express it as KTM the A; can be expanded into a sequence form as A, [akg ((G (...... ((,,.)) "h....).......
[k:..1):..., (....)....h...mm))) Similarly,
Despite their reported superior performance on various tasks (Yu et al., 2019; Wankhade et al., 2022), the internal workings of the attention mechanism within fusion mod- ules-and whether they function intended-remain derexplored. In this work, we delve into the learning process of the attention module, shedding light on how the attention mechanism attributes weights across different modalities. We further propose the RollingQ algorithm, which restores
Given that softmax function as a normalization function (Bri- dle, 1989), attention score for modality m is determined by Σ
remain un-
By averaging the keys of modality m. Vd
1=1
km
km
j==
(6,5)
m E (a,u),
(3)
L.
3

---

RollingQ: Reviving the Cooperation Dynamics in Multimodal Transformer
have:
CREMAD
MOSEI-VT
12.5
Σ 1=1
(GJ)
LM |allallk7
12c0000...
(4)
16.2
12.5
Here, |1a/22, ||,; represent the no-norms of q, k. re- spectively, while coso denotes the cosine similarity be- tween y and kmr.
A10 -215
-550
-225
305
6.5
020
025
1520 -335 1020 535 0.0 E35 Dao
3.2. A Self-reinforeing Cycle Undermines Cooperation Dynamics
Figure 2. The distribution of average key on CREMA-D and MO- SEI(V+T); the x-axis refers to the cosine similarity with query while the y-axis refers the L2-norm of key.
To explore the underlying causes of the deactivated coopera- tion dynamics, we delve into the training process, providing a theoretical analysis of attention allocation during the feed- forward stage and the optimization of unimodal encoders during backward propagation.
tions across modalities and disproportionately high attention score for biased modality.
At the beginning of training, since both modalities' features lack task-relevant information, the attention scores for each modality primarily depend on initialization. Specifically, we usually initialize the [class] token using truncated nor- mal distribution with expectation E [| = 0 (Dosovitskiy, 2020). Inspired by Chen et al. (2018) and Geshkovski et al. (2023), we regard both query g and the average key Rej as a sample from its corresponding random variables Q, Rm and distribution Q, kmm. We can denote it as follows:
Meanwhile, modality bias will also influence the backward propagation for unimodal encoders. We use L 9) to denote the loss function where y represents the ground truth label and ý is the prediction generated by the linear classifier f()), following 9 = f(h). We can integrate the Equation 1 and Equation 2 to the form below:
LKK KTTT IVS.V.).
h4 = =sofmaa
(6)
Vd
Q ~ Q, Km RT
(5)
the current loss L(ys ((h))).
The gradient of modality m's encoder parameter is:
From the [class] token to query following q Reto W. we has [[Q] = 0. Due to the empirical fact that Q and KT are independent variables, their expectation can be directly multiplied. resulting in E[ = E[QK*] = 0 at deduce the
aL aL 0f ah, 0z," 30 Of Th, 02m d6m
the training's 's initial phase. Therefore, following proposition:
According to Equation 7, we can find that 0L 27 and 0f ah.
shared for both modalities and 0z." depends on the modality- 30
Proposition 3.1. At the start of training, both modalities receive similar attention scores.
specific encoder th which is not influenced by the attention layer. Hence, the only difference in the gradients across modalities arises from 8h,
As training progresses, the situation evolves during the feed- forward stage. Due to intrinsic differences between modal- ities, a modality may be favored by the model and pro- vide higher quality features over time, becoming the biased modality. Here, we assume audio is the biased modality.
82
Using s(-) to denote softmar()), with Equation 6, we can expand the gradient
ahg
to
Os( ( (oKK/KKKK KKIT
OK?"
9K3 KKT ov.T
KYIT
Taking the greedy nature of multimodal deep neural net- works into account (Wu al., 2022), the model tends to prioritize the modality when it provides higher quality fea- tures. As result, the biased modality accumulates higher at- tention scores, which are determined by ΣØ= gkfa, With va Equation 4, this greedy nature increases cosine similarity between the modality a's average key by and while si- multaneously decreasing the cosine similarity between the modality e's average key by and q. Consequently, this leads to significant disparity in the average key distribu-
Ja aKm
+
(8)
02m
Vd
Ozme
At the backward propagation for encoders, the biased modal-
ity a receiving higher attention score s( 2(2/22 "K"""""" ), has greater gradient s( [o] J of Equation 7 and Equation 8. It, in turn, reinforces the opti- mization of its corresponding unimodal encoder parameters 6ah In contrast, the unbiased modality v receive less gra- dient from s( q K K.K'" av."
Va
vd "Bs augmenting the gradient
8V""
leading under-optimized 02.
Vd.
4

---

RollingQ: Reviving the Cooperation Dynamics in Multimodal Transformer
90"
Algorithm Rolling Query Algorithm.
A
Input: Training Dataset D = {(=,,"" x Ji
K.
135%
-E
45
number of batch NB. iter epoch T,, hyperparameter p. B. model parameters 0m, m E (), [class] token Rets and Wº, W in attention layer
qr
Initialize R =
30
Query Rotation
25
for 1 to T do
20
15
for j 1 to NB do
10
Sample Batch Bj Feed-forward with g Ects WetsWR R
180
0
Update model parameter pT
if j = NB then
Figure 3. The illustration of RollingQ algorithm.
Calculate the average key using Equation Calculate E[K") mean k(m) in B3
encoder parameters 0" Consequently, this dynamic further exacerbates the inequality between feature qualities. To verify this analysis, we provide visualization gradients of unimodal encoders in Appendix A.2.
Calculate E[Q] = mean(q) in Bj
Calculate the AIR indicator by Equation 9
if ATR 2 3 then
Freeze the model parameter tm
In summary, the modality bias triggers a self- reinforinngg cycle: during the feed-forward stage, biased modality ac- cumulates more attention score due to its more informative feature, while in the backward propagation, the higher atten- tion score provides more momentum to optimize the biased modality's s encoder parameters. Consequently, this cycle not only creates significant distribution disparity between Kt and K but also amplifies the inequality of feature quality between the two modalities.
Calculate a by Equation 11
Calculate rebalance position q by Equation 10
Calculate rotation matrix RG by Equation 13 Update rotation matrix I = R( ((.a.cel))))) Unfreeze the model parameter t"
end end end for end for
Due to the significant distribution gap, the & and ky main- tain considerable differences regardless of data character- istics, where kū consistently has higher cosine similarity with g than kg- As shown Figure 2, when the biased modality replaced with Gaussian noise on the CREMA-D and CMU-MOSEI datasets, we observe large distribution gap between the modalities. Notably, the noise input is mapped to marginal position within the original distri- bution, yet it still exhibits higher cosine similarity to compared to the unbiased modality. As a result, the model over-rely on the biased modality, leading to the deactivation of the cooperation dynamies in multimodal Transformers. Further verifications and visualizations on more datasets are provided in Appendix A.
reinforcing cycle. Firstly, to detect the occurrence of this cycle, we can evaluate and monitor the distribution dispar- ity across modalities. Next, to break the cycle, we must suppress its two driving factors: unequal feature quality and disproportionate attention scores. While feature quality is inherently determined by the modality itself, making it difficult evaluate and control, we focus modulating the attention scores. Specifically, we expect to find rea- sonable anchor that could assign higher attention scores to the unbiased modality rather than the biased one. We then rotate the current query towards this anchor and let the new query learn in this region, encouraging the equalization of feature quality and reduction of distribution disparity across modalities.
3.3. Rolling Query Algorithm
Evaluation of distribution gap. Referring to Equation 3, we transform the attention score into cosine similarity form. Based on the experiments observations and consid- ering the LayerNorm layers used in the model, we hold k ||k"" 1k, La L in practice. Thus, the primary factor determining the attention score the cosine similarity cost," between the average key km of the modality and the query q. To quantify the distribution gap, we define the Attention Imbalance Rate (AIR) indicator:
Based on the above analysis, attention fusion tends to over- rely the biased modality, losing its cooperation dynamics. It is therefore crucial to revive its dynamic property. To address this issue, we propose simple yet effective Rolling Query (Rolling() algorithm, which requires a few increase on model parameters and computation complexity".
The key idea to revive the cooperation dynamics is to find an effective method for identifying and breaking the self-
Al R E|coso" cos0") 1-
"See Table details
5

---

RollingQ: Reviving the Cooperation Dynamics in Multimodal Transformer
Table Validation on CREMAD (Audio+ Kinetic-Sound (Audio+ Visual) and MOSEI (Visual+Text with Transformer hackbone.
The best results are presented in bold. We use 1 to show the improvement in performance compared to not implementing the RollingQ.
Dataset
Fusion
CREMA-D Kinetic-Sound CMU-MOSEI (V+T)
Metric
Layers
Acc
Acc
Acc
Audio
47.6
53.9
Unimodal
Visual
36.3
57.0
47.1
Text
0
60.9
Concat
49.3
68.0
62.8
Static
OGM (Peng et al., 2022)
51.2
68.2
62.7
PMR (Fan et al., 2023)
50.1
68.2
63.0
Vanilla MT
48.8
67.0
62.7
Vanilla MT*
51.5
69.1
62.2
MulT (Tsai et al., 2019)
62.4
Dynamic
MBT (Nagrami al., 2021)
51.5
72.2
63.0
JMT (Nagrani et al., 2021)
50.7
67.7
62.6
MMML (Wu et al., 2024)
52.0
69.8
62.8
Vanilla MT+Rolling
51.9 (t 3.1)
69.3 (1 2.3)
63.2 (f 0.5)
Vanilla +
2
52.2 (f 0.7)
70.1 (1 1.0)
62.9 (t 0.7)
Ours
MulT (Tsai et al., 2019)+Rolling
62.5 (t 0.1)
MMML (Wu et al., 2024)+Rolling
52.7 017)
70.7 (1 0.9)
63.2 (f 0.4)
hyperparameter 3 serves as the threshold for A.R. For those ||TR| 2 3, consider the distribution gap signifi- cant, indicating disproportionate attention scores.
Tanh(-pAIR)) < O, resulting in 0.5 and >> 0.5. Thus, the modality will have lower impact the deter- mination of gs shown in Equation 10.
The balance anchor and the rotation matrix. To break the seiforeinn cycle, the query q is rotated to rebal- anced anchor do that could balance the allocation of attention scores and narrow the distribution gap between K' and K", suitable position should favor the unbiased modality while considering the biased modality based the AIR indicator. This enables the underutilized modality to gain more mo- mentum during optimization, helping it learn higher quality features. As a result, the quality of features across modal- ities can gradually equalize during backward propagation, reducing the disparity in their key distributions during the feed-forward stage. To determine (b, calculate it using the expectation form, which allows for easy extension to multi-layer scenarios as we discussed in Appendix B, of the average key distributions "[ and the query distribution E[Q], with a weight a::
To enable the current query to learn around the rebalance an- chor as we wished, a rotation matrix Rg E Rd is essential. Since we already ensure the L2-morm of rebalance anchor q and query q is the same by Equation 10, we can cal- culate RG by Singular Value Decomposition (Baker, 2005) with rebalance qu and the expectation of current query distribution B[Q): Rs SVD(E qb)),
(12)
which statisfied qb E(Q]R
Finally. for the query after rotation gr. its value can be obtained by multiplying rotation matrix R3 on g. where we can obtain:
gr R
(13)
E -
Overall, by leveraging the AIR indicator to detect the self- reinforcing cycle and learning new rotated query around rebalance anchor with a rotation matrix, Rolling can mit- igate over-reliance on the biased modality and revive the cooperation dynamics in multimodal Transformer.
Øh (ar
+ (1
|[[[Q]] (10)
|E
while the weight is derived, indicating AIR:
1 [ + Tanh((pAAA))
(11)
4. Experiments
where p is an hyperparameter holds p >>>00 0.
4.1. Dataset and Experiment Settings
This ensures that the decreases the influence of the biased modality on the output. For instance, for AIR > which indicates that modality a the biased modality, receiv- ing a higher attention score than modality v, we will have
Datasets. CREMA-D (Cao et al., 2014) is an audio-visual dataset designed for emotion recognition, covering com- mon emotions. Kinetic-Sound (Arandjelovie & Zisserman,
6

---

RollingQ: Reviving the Cooperation Dynamics in Multimodal Transformer
Kinetic-Sound, epoch-11 11
Kinetic-Sound, epoch- 11
Kinetic-Soudd
Kinetic-Sound
SEATEST
023
-075 -2.0
-23 0:0 035
632
0.5
100
Aga -015 -002
-025
130
1000
2004
1203
4000
5250
(a) Before Rotation
(b) After Rotation
(c) Training Process
(d) Attention Score
Figure 4. (a): The distribution of modalities before Rolling Query on the Kinetic-Sound dataset. (b):The distribution of two modalities after Rolling Query on Sameetcc-oundd dataset. (c): The variation of attention score on each modality during the training process on Kinetic-Sound datasct when implementing the RollingQ algorithm. (d): Attention scores assigned cach modality on the Kinetic-Sound dataset when implementing the RollingQ algorithm. The left half shows normal samples, while the right half shows samples where the audio modality's 's input is replaced with Gaussian noise.
2017) consists of 31 human action classes selected from the Kinetics dataset, which includes 400 action categories derived from YouTube videos. CMU-MOSEI (Zadeh al., 2018) is multimodal dataset that integrates audio, visual, and textual data.
tiTrans*. We implement the RollingQ algorithm on these two representative methods to reveal its effectiveness and extend it to MulT (Tsai et al., 2019) and MMML (Wu et al., 2024) on its last attention layer to demonstrate the general applicability of our approach, with implementation details shown in Appendix B. The results are presented in Table 1.
Settings For the CREMA-D and Kinetic-Soumt datasets, we use a 4-layer ViT-B/16 (Dosovitskiy, 2020) as the back- bone, initializing with pre-trained weights from ImageNet- 21k (Ridnik et al., 2021). For CMU-MOSEI (Zadeh al., 2018), adopt 4-layer vanilla Transformer following the preprocessing and settings outlined in Liang et al. (2021) I). As for training, we use SGD as optimizer and use cosine scheduler. The learning rate is set to Ic-3 with batch size 64 for all experiments. More detailed experimental settings can be found in Appendix C.I.
Based on empirical results, we make several observations. First, compared to all unimodal models, multimodal mod- els consistently outperform them, demonstrating that in- tegrating information from multiple modalities is indeed beneficial. Besides, dynamic fusion methods do not always outperform static fusion methods, including the simple con- catenation approach. This may be due to optimization chal- lenges and the potential loss of dynamic properties as we explored in this work. Moreover, our RollingQ algorithm shows significant improvement over the original method, including the vanilla multimodal Transformers and special- ized multimodal models like MulT (Tsai et al., 2019) and MMML (Wu et al., 2024), which indicates the prevalence of deactivated dynamic properties in multimodal Transformers and it is pivotal to revive the cooperation dynamics in multi- modal Transformers. Finally, our method yields comparable results to imbalance techniques in static fusion scenarios, although RollingQ algorithm does not directly enhance the learning of unimodal encoders. Unlike more complex Trans- former architectures, such MBT (Nagrani et al., 2021) and JMT (Waligora et al., 2024), which require additional pa- rameters and specialized modules, our Rolling algorithm is grounded in a simple yet effective idea, demonstrating strong performance without the need for such complexities.
4.2. Comparison with Static and Dynamic Fusion
To validate the effectiveness of our RollingQ method in reviving the dynamic properties of the attention mecha- nism and improving overall performance, we conducted experiments on various datasets and compared our method with previous fusion techniques. We use vanilla concate- nation as representations of static fusion approaches and include unimodal models as basclines. Besides, several methods, aiming at improving the performance of the static fusion paradigm by modulating the optimization for uni- modal encoders, are considered, including OGM (Peng et al., 2022) and PMR (Fan et al., 2023). From the per- collect several
spective of the dynamic fusion paradigm, attention-based methods from prior work for comparison, including MulT (Tsai et al., 2019), MBT (Nagrani et al., 2021), JMT (Waligora et al., 2024) and MMML (Wu et al., 2024). Finally, we use a single-layer attention for multi- modal fusion method, named Vanilla MultiTrans, and a 2-leyrr Transformer fusion module, denoted by Vanilla Mul-
4.3. What Has Rolling( Done to Revive Cooperation Dynamics?
Effective of rotation. First, we visualize the distribution of average keys before and after the Rolling Query during
7

---

RollingQ: Reviving the Cooperation Dynamics in Multimodal Transformer
training. As shown in Figure 4(a) and Figure 4(b), after the Rolling Query, the biased audio modality and the un- biased visual modality exchange their similarity with the query. Specifically, the rotated query will learn around a new region, that could assign more attention to the visual modality, while slightly decreasing the similarity with the audio modality like Figure 4(b), which will not significantly impair the learning of audio modality but encourage the learning of visual modality. As result, the audio modal- ity, which still maintains more informative features, will no longer reinforce the optimization of its encoder, while the visual modality's 's optimization is enhanced.
Table 2. The Pearson correlation hetween the attention score and whether it is a noise input. The coef closer to I or -1 indicates a stronger relation between attention score and input, while the p < 0.01 means that the coef is trustworthy.
Dataset Correlation
CREMA-D
Kinetic-Sound
coef
p
coef
p
Vanilla MT
0.52
< 0.01
0.44
< 0.01
Vanilla MT+Rolling
0.76
<.001
0.78
< 0.01
Table 3. Experiment results when perturbing the audio modality (the biased modality) with Gaussian noise following Liang et al. (2021) on Kinetic-Sound Dataset.
Evolution after rotation. After rotating the query, the new query learns in new rebalance region. Due to the fact that audio modality remains higher quality features, the new query tends to consecutively increase attention score to audio modality as we discussed in Section 3.2. To verify this analysis, we visualize the variation of attention scores on different modalities during the training process. As shown in Figure 4(c), the attention score of audio modality drops in certain iterations, while increasing latter. This phenomenon ensures that the query remains learnable based on the input data, indicating that the learning of attention allocation is not hindered by Rolling Query.
Noise level
Vanilla MT
Vanilla MT + RollingQ
0.00
67.0
69.3
0,25
62.7 4.3)
67.2 (4 1.9)
0.50
52.9 (4 14.1)
58.2 (L 11.1)
0.75
43.2 (L 23.8)
47.5 (L 21.8)
1.00
34.7 (L 32.3)
40.6 () 28.7)
attention (Rawal et al., 2023) are provided in Appendix C.2.
Additionally, we visualize the distribution of averaged at- tention keys across modalities. As shown in Figure 5, the RollingQ algorithm significantly reduces the distribution gap between modalities, compared to Figure (dd) Regard- ing the noise input (represented by the red dots), it now shows lower cosine similarity to the query than to the av- erage keys of both modalities. Consequently, the attention mechanism allocates less attention to the relatively uninfor- mative input in the biased modality as expected.
Cooperation dynamics. we investigate whether RollingQ can restore the cooperative dynamics of the attention mecha- nism. To test this, we replace the input of the biased modal- ity with Gaussian noise and observe the results. First, by averaging the attention scores for both normal and noise samples, can assess whether the attention layer assigns appropriate weights based on the current input. As shown in Figure 4(d), on the Kinetic-Sound dataset, replacing the audio modality with noise significantly reduces the attention score for the audio modality and increases the attention score for the visual modality. Compared to the disproportionately high attention observed in Figure Icc), the model with the RollingQ algorithm demonstrates improved inter-modality cooperation. Further tests verifying cooperation dynamics based on attention like adding attention mask or QUAG
Furthermore, we conduct a correlation analysis between the noise input and attention scores in models with and without the RollingQ algorithm. Using Pearson correlation analy- sis on both the CREMA-D and Kinetic-Sound datasets, we assess whether the model can effectively recognize sample- wise variations in modality quality and adjust the attention scores accordingly. As shown in Table 2, after applying the RollingQ algorithm, the correlation coefficient increases significantly. This indicates that the attention mechanism becomes more sensitive to the data characteristics of each modality and can dynamically adjust attention scores. These findings confirm that the RollingQ algorithm successfully re- stores the cooperation dynamics in multimodal Transformer.
CREMA-D
Kinetic-Sound
22
25
4.4. Test-time Adaptation for Noisy Biased Modality
To further and more comprehensively verify that RollingQ can restore cooperative dynamics within multimodal Trans- formers, thereby enhancing multimodal performance under more general conditions, we conduct experiments with noisy biased modality on Sinetic-Sound. We perturb the biased audio modality with Gaussian noise at various noise levels, as described in Liang et al. (2021), rendering the biased
-100 -575 -306
-633
630
0J5
430
335
150
2:30 -075 -5.0 -005 300
335 635 10e
Figure 5. The distribution of two modalities is key when imple-
menting the RollingQ algorithm. The red dots denote the audio modality with noise interference.
8

---

RollingQ: Reviving the Cooperation Dynamics in Multimodal Transformer
Table The accuracy, model parameters, and time complexity analysis on CREMA-D dataset. The GFLOPs are obtained from the thop library.
Table 7. Experiment results of our method with ResNet back- bone on CREMA-D.
Method
Acc
Method
Acc
Parameters
GFLOPs
Audio
60.1
Vanilla MT
48.8
59.87M
1489.13
Visual
40.1
MBT
51.5
114.21M
2746.90
Concat
61.4
MMML
52.0
77.88M
1828.29
Vanilla MT
60.8
JMT
50.7
62.23M
1494.87
Vanilla MT + Rolling
62.6
Vanilla MT + Rolling
51.9
60.46MM
1489.20
Table 5, Abaltion of different batch size on CREMA-D dataset. Batch Size
Vanilla MT
Vanilla MT RollingQ
on CREMA-D dataset with batch size around 16, 64, and 256 with all other hyperparameters fixed. As shown in Table 5, the proposed RollingQ algorithm consistently out- performs the baseline Vanilla MT by 0.7% 3.1%. The results ensure the stability of RollingQ.
16
49.1
50.6
64
48.8
51.9
256
47.8
48.5
modality unreliable for final predictions. As shown in Ta- ble 3, both methods experience performance drop as the noise level increases. However, RollingQ consistently out- performs the baseline, demonstrating its ability to better leverage both modalities and facilitate their cooperation. Be- sides, RollingQ exhibits lower accuracy drop rate than the baseline, indicating that by reviving cooperation dynamies, RollingQ enables the model to better handle variations in data quality across modalities, enhancing its robustness to perturbations and leading to superior performance. Addi- tionally, to evaluate RollingQ's performance under more severe and systematic distribution shifts, we conduct mul- timodal out-of-distribution (OOD) detection experiments following Dong al. (2024) in Appendix C.3.
Ablation of encoder layers. To ensure that the analysis and effectiveness of RollingQ are consistent across different backbone scales, we conducted an ablation study on Kinetic- Sound dataset where the performance varied significantly across different encoder layers. As shown in Table with encoder layers ranging from 2 4, and 6 and all other hyperpa- rameters fixed, RollingQ consistently shows improvement compared to baseline Vanilla MT, verifying its effectiveness.
4.7. Adaptation to Other Backbone
To verify the generalization of our approach, we employed ResNet18 (He et al., 2016) and transformed the feature maps into tokens by flattening the temporal and spatial dimensions following (Carion et al., 2020; Huang et al., 2020). As shown in Table 7, our method consistently outperforms the baseline, demonstrating its effectiveness across different backbones and its potential to extend various models.
4.5. Complexity and Efficiency
Since RollingQ only needs an additional rotation matrix to work effectively, it is simple method that requires few in- creases on parameters and computational cost. To verify the detailed additional cost of RollingQ, we record the model parameters and GFLOPs on experiments of CREMA-D dataset. As shown in Table 4, the proposed RollingQ algo- rithm requires 1% increase parameters and 0.1% increase on GFL.OPs, while achieving considerable improvements on its baseline vanilla multimodal Transformer and gaining comparable and even better performance over other special designed Transformer architectures.
5. Conclusion and Discussion
In this work, we identify the deactivation of cooperation dy- namics in multimodal Transformer based on the observation of self-attention fusion layer, perform theoretical analysis and experimental verifications, and propose the RollingQ a algorithm restore its dynamic properties.
Future work and limitations. Our theoretical analysis fo- cuses primarily on a single attention layer, while real-world Transformer models typically involve multiple layers, where the dynamics are more complex. Thus, further research is ex- pected to explore how to better model, analyze, and restore cooperative dynamics in multi-layer Transformers. Addi- tionally, our algorithm does not directly enhance the feature quality of unimodal encoders, which has been effectively achieved in previous static fusion approaches. Therefore. combining our method with those previous approaches to enhance and balance feature quality while addressing the deactivation issue an avenue for future exploration.
4.6. Ablation Study
Ablation of batch size. To test the sensitivity of RollingQ toward batch size decision, we conducted an ablation study
Table Ablation of different VIT encoder layers on Kinetic-Sound
dataset.
Encoder-Lan Layer
Vanilla MT
Vanilla MT + Rolling
2
57.6
58.5
4
67.0
69.3
6
73.6
74.6

---

RollingQ: Reviving the Cooperation Dynamics in Multimodal Transformer
Acknowledgements
Dong, H., Zhao, Y., Chatzi, E., and Fink, O. Multiood: Scal- ing out-of-distribttion detection for multiple modalities. arXiv preprint arXiv:2405. 17419, 2024.
This work is supported by National Natural Science Founda- tion of China NN0.62102772). This work also supported by Public Computing Cloud, Renmin University of China, and fund for building world-class universities (disciplines) of Renmin University of China.
Dosovitskiy, A. An image is worth 16x16 words: Trans- formers for image recognition scale. arXiv preprint arXiv:2010. 1929, 2020.
Fan, Y., Xu, W., Wang, H., Wang, J., and Guo, S. Pmr: Prototypical modal rebalance for multimodal learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20029-20038, 2023,
Impact Statement
This paper presents work whose goal advance the field of Machine Learning. There are many potential societal consequences of our work, none which feel must be specifically highlighted here.
Gao, J., Li, P., Chen, Z., and Zhang, J. A survey deep learning for multimodal data fusion. Neural Computation, 32(5):829-864, 2020.
References
Gao, P., Jiang, Z., You, H., Lu, P., Hoi, S. C., Wang, X., and Li, H. Dynamic fusion with intra-and inder-modaltty attention flow for visual question answering. In Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6639-6648, 2019.
Arandjelovic, R. and Zisserman, A. Look. listen and learn. In Proceedings of the IEEE international conference computer vision, pp. 609-617, 2017.
Baker, K. Singular value decomposition tutorial. The Ohio State University, 24:22, 2005.
Geshkovski, B., Letrouit, C., Polyanskiy, Y., and Rigollet, P. A mathematical perspective on transformers. arXiv preprint arXiv:23122 10794, 2023.
Baltrušaitis, T., Ahuja, C., and Morency, L.-P. Multimodal machine learning: A survey and taxonomy. IEEE trans- actions on pattern analysis and machine intelligence, 41 (2):423 4434234433,222 2018
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual lear- ing for image recognition. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Bridle, J. Training stochastic model recognition algorithms as networks can lead to maximum mutual information estimation of parameters. Advances in neural information processing systems, 2, 1989.
Huang, C., Wei, Y., Yang. Z., and Hu, D. Adaptive uni- modal regulation for balanced multimodal information acquisition. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 25854-25886 2025.
Cao, H., Cooper, D. G., Keutmann, M. K., Gur, R. C.,, Nenkova, A., and Verma, Crema-d: Crowd-sourced emotional multimodal actors dataset. IEEE transactions affective computing, 574):377-390, 2014.
Huang, Y., Lin, J., Zhou, C., Yang, H., and Huang, L.
Modality competition: What makes joint training of multi- modal network fail in deep learning (provabbly)) In Inter- national conference on machine learning, pp. 9226-9259, PMLR, 2022.
Carion, N., Massa, F., Synnaeve. G.. Usunier, N., Kirillov, A., and Zagoruyko, S. End-to-end object detection with transformers. In European conference computer vision, pp. 213-229. Springer, 2020.
Huang, 7., Zeng. 7.,, Liu, B., Fu, D., and Fu, J. Pixel-bert: Aligning image pixels with text by deep multi-modal transformers. arXiv preprint urXiv:2004.00845 2020.
Chen, R. T., Rubanova, Y., Bettencourt, J., and Duvenaud, D. K. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018.
Joze, H. R. V., Shaban, A., luzzolino, M. L., and Koishida, K. Mmtm: Multimodal transfer module for cnn fusion. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 13289-13299, 2020.
Chumachenko, K., Iosifidis, A., and Gabbouj. M. Self- attention fusion for audiovisual emotion recognition with incomplete data. In 2022 26th International Conference on Pattern Recognition (ICPR), pp. 2822-2828. IEEE, 2022.
Kim, W., Son, B., and Kim, I. Vilt: Vision-and-language transformer without convolution or region supervision. In International conference on machine learning, pp. 5583- 5594. PMLR, 2021.
Clark, K. What does bert look at? an analysis of bert's attention. arXiv preprint 1rXiv:/906.000 04341, 2019.
Kovaleva, O. Revealing the dark secrets of bert. arXiv preprint arXiv: /00:::008800000 08593, 2019.
10

---

RollingQ: Reviving the Cooperation Dynamics in Multimodal Transformer
Kuehne, H., Jhuang, H., Garrote, E., Poggio, T., and Serre, T. Hmdb: large video database for human motion recog- nition. In 2011 International conference on computer vision, pp.
linguistics. Meeting, volume 2019, pp. 6558. NIH Public Access, 2019.
IEEE, 2011.
Vaswani, A. Attention all you need. Advances in Neural Information Processing Systems, 2017.
Li, L. H., Yatskar, M., Yin, D., Hsieh, C.-J., and Chang, K.-W. Visualbert: A simple and performant baseline for vision and language. arXiv preprint Stlv:1908.035555 2019.
Waligora, P., Aslam, M. H., Zeeshan, M. O., Belharbi, S.,
Koerich, A. L., Pedersoli, M., Bacon, S., and Granger, E. Joint multimodal transformer for emotion recognition in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4625- 4635, 2024.
Liang, P. P., Lyu, Y., Fan, X., Wu, Z.,, Cheng, Y., Wu, J., Chen, L.,, Wu, P., Lee, M. A., Zhu, Y., al. Multibench: Multiscale benchmarks for multimodal representation learning. Advances in neural information processing systems, 202I(DBI):1, 2021.
Wang. W., Tran, D., and Feiszli, M. What makes training multi-modal classification networks hard? In Proceedings of the IEEE/CVF conference on computer vision and patlern recognition, pp. 12695-12705, 2020a.
Liang, P. P. Zadeh, A., and Morency, L.-P. Founda- tions and trends in multimodal machine learning: Prin- ciples, challenges, and open questions. arXiv preprint arXiv:2209.00 03430, 2022.
Wang. Y., Huang, W., Sun, F., Xu, T., Rong, Y., and Huang, J. Deep multimodal fusion by channel exchanging. Ad- vances in neural information processing systems, 33: 4835 4845, 2020b.
Liu, Z., Shen, Y., Lakshminarasimhan, V. B., Liang, P. P., Zadeh, A., and Morency, L.-P. Efficient low-rank mul- timodal fusion with modality-specific factors. arXiv preprint arXiv: arXiv:/806.000 1806.0006404 2018
Wang, Y., Chen, X., Cao, I.,, Huang, W., Sun, F., and Wang, Y. Multimodal token fusion for vision transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 12186-12195, 2022.
Nagrani, A.. Yang. S., Arnab, A., Jansen, A., Schmid, C.,, and Sun, C. Attention bottlenecks for multimodal fusion. Advances in neural information processing systems, 34: 14200-14213, 2021.
Wankhade, M., Rao, A. C. S., and Kulkarni, C. A survey on sentiment analysis methods, applications, and challenges. Artificial Intelligence Review, 55(7):5731-5780 2022.
Peng, X., Wei, Y.. Deng, A., Wang, D., and Hu, D. Balanced multimodal learning via on-the-fly gradient modulation. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 8238-8247, 2022.
Wei, Y., Feng, R., Wang, Z., and Hu, D. Enhancing multi- modal cooperation via sample-level modality valuation. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 27338-27347, 2024.
Rawal, I. S., Matyasko, A., Jaiswal, S., Fernando, B., and Tan, C. Dissecting multimodality in videoqa transformer models by impairing modality fusion. arXiv preprint arXiv:2306. 230.003306..888, 2023.
Wu, N., Jastrzebski, S., Cho, K., and Geras, K. J. Charac- terizing and overcoming the greedy nature of learning in multi-modal deep neural networks. In International Con- ference on Machine Learning, pp. 24043-24055 PMLR, 2022.
Recasens, A., Lin, J., Carreira, J., Jaegle, D., Wang, L., Alayrac, J.-b., Luc, P.. Miech, A.. Smaira, L., Hemsley, R., et al. Zorro: the masked multimodal transformer. arXiv preprint arXi:2300..09595 2023.
Wu, Y., Zhan, P., Zhang, Y., Wang, L., and Xu, Z. Multi- modal fusion with co-attention networks for fake news detection. In Findings of the association for compula- tional linguistics: ACL-IJCNLP 2021, pp. 2560-2569, 2021.
Ridnik, T., Ben-BenBaaee nen-Baruch, E., Noy, A., and Zelnik-Manor, L.. Imagenet-21k pretraining for the masses, arXiv preprint arXiv:2104. 10972, 2021.
Soomro, K., Zamir, A. R., and Shah, M. Ucf101: A dataset of 101 human actions classes from videos the wild. arXiv preprint arXiv: 22/. 2012.
Wu, Z., Gong, Z., Koo, J., and Hirschberg, J. Multimodal multi-loss fusion network for sentiment analysis. In Pro- ceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp.
Tsai, Y.-H. H., Bai, S., Liang, P. P., Kolter, J. Z., Morency, L.-P., and Salakhutdinov, R. Multimodal transformer for unaligned multimodal language sequences. In Proceed- ings of the conference. Association for computational
2024.

---

RollingQ: Reviving the Cooperation Dynamics in Multimodal Transformer
Xia, W., Zhao, X., Pang, X., Zhang, C., and Hu, D. Balanced audiovisual dataset for imbalance analysis. arXiv preprint DrXiv:2302.10912 2023.
Xu, P., Zhu, X., and Clifton, Multimodal learning with transformers: survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(10):12113-1213, 2023.
Yang. Y., Wan, F., Jiang, Q., and Xu, Facilitating multi- modal classification via dynamically learning modality gap. Advances in Neural Information Processing Sys- tems 38, 2024.
Yang, Y., Pan, H., Jiang, Q. Xu, Y., and Tang, J. Learn- ing to rebalance multi-modal optimization by adaptively masking subnetworks. IEEE Trans. Pattern Anal. Mach. Intell., 2025.
Yu, Z., Yu, J., Cui, Y., Tao, D., and Tian, Q. Deep modular co-attention networks for visual question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6281-6299 2019.
Zadeh, A., Chen, M., Poria, S., Cambria, E., and Morency, L.-P. Tensor fusion network for multimodal sentiment analysis. arXiv preprint arXiv:
2017.
Zadeh, A. B., Liang, P. P., Poria, S., Cambria, E., and Morency, L.-P. Multimodal language analysis the wild: Cmu-mosei dataset and interpretable dynamic fu- sion graph. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2236-2246, 2018.
Zhang. Q., Wei, Y., Han, Z., Fu, H., Peng, X., Deng, C., Hu, Q. Xu, C., Wen, Hu, D., al. Multimodal fusion low-quality data: A comprehensive survey. arXiv preprint 1844:72404.1844, 2024.
12

---

RollingQ: Reviving the Cooperation Dynamics in Multimodal Transformer
A. Assumption and Analysis
A.I. Attention and Average Key Distribution
To holistically and systematically validate our assumption and analysis discussed in Section 3.2, which mainly states that the modality bias in the multimodal training process triggers self-reinforcing cycle that leads to inequality in feature quality and unreasonable attention score, we conduct experiments on more datasets including CREMA-D, Kinetic-Sound, CMU-MOSEI(A+TT) CMU-MOSHI(V+TT UCo-roo et al., 2012), HMDB51 et al., 2011), whose modality ranging from audio, RGB, text and optical flows and exhibiting variation in modality sequence length.
As shown in Figure 6, with different combinations of modalities and data set scales, the attribution of the unreasonable attention score agrees with our previous study. Besides, the visualization of average key distribution, where the noise input consistently has higher cosine similarity, further validates our assumption and theoretic analysis towards cooperation dynamics under imbalance multimodal learning.
CREMA-D
Kinctic-Sound
verges
43
17
10
-300
-635
-058
-0.5
0.30
135
152
12
680
12.0
12.
12.0
125
200
10mple
CMU-MOSE((VT)
CMU-MOSEI(AT)
1.0
3.0
50
Tweg
Tweglebeayy. 31
19
880
-
UCF-101
HMDB51
40
1.c
1.0
V Veggtttbeee
10
9.0
-918-233
080
Figure 6. The visualization of attention score and average key distribution on CREMA-D, Kinetic-Sound, CMU-MOSEIV CMU- MOSHIA+T), UCF-101, HMDB51 datascts, including audio (A), RGB (V), text (T) and optical flow (O) modalities. The left figure for each dataset denotes the average key distribution of different modalities and the red dots represent samples replacing the biased modality by Gaussian noise. The right figure for each dataset reveals the attention score each modalities with normal or noise inputs.
13

---

RollingQ: Reviving the Cooperation Dynamics in Multimodal Transformer
A.2. Gradients Evolution of Unimodal Encoders
To verify the analysis on gradients posted in Section 3.2, we conduct experiments on CREMA-D and Kinetic-Sound to monitor the gradients of each unimodal encoder. As shown in Figure 7, the gradient of the audio encoder increases significantly during the mid-stage, when the attention score begins to accumulate in the biased modality. This results in noticeable gap between the modalities. As the total loss decreases over time, the gradients of both modalities drop, but their relative relationship remains. This further validates our theoretical analysis, showing that the biased modality consistently receives more optimization momentum. The gradient of the weaker modality never exceeds that of the biased modality, which can be explained by Equations 7 and 8. Since the only difference in gradients for each modality is @h, and the loss is the multimodal loss, where "one modality becoming converged" equals "the multimodal model becoming converged." the total loss becomes very small. This small loss cannot provide enough momentum for the biased modality to optimize effectively. Hence, the gradient of the biased modality will consistently greater than the weak one.
Gradient of Each Encoder on CREMA-D
Gradient of Each Encoder on Kinetic-Sound
2.5
2.5
2.0
2.0
1.5
1.5
1--Mom
1.0
1.0
0.5
0.5
0.0
500
1000
1500
2000
2500
3000
0.0
0
2000
4000
6000
8000
10000
iter
iter
Figure 7. The L2-Norm of gradient of the audio encoder and the visual encoder on CREMA-D and Kinetic-Sound datasets.
B. Implement Details of The RollingQ Algorithm
Rotation time limits for training stability. In Algorithm 1, we calculate and update the rotation matrix whenever the absolute value of the AIR indicator exceeds its threshold. However, as /3 becomes smaller, and the time required for rotation increases significantly. Based on our observation, for all dataset the RollingQ algorithm will rotate the query for about 3-5 times. However, the frequently changing query vector will impair training stability, resulting an not optimum performance. To address this, we set maximum rotation limit in the RollingQ algorithm ensure more stable learning. Specifically, the maximum rotation time is set to 1 for CREMA-D and MOSEI, and to for Kinetic-Sound, which is larger dataset.
Extension to multi-layer transformer. The RollingQ algorithm and the scenario we discussed are based on a single attention layer. However, as the number of layers increases in a multi-layer transformer, the situation becomes more complex than what we've analyzed so far. To effectively implement our RollingQ algorithm in such cases, we train the multi-layer attention model progressively, applying the RollingQ algorithm to a specific layer when it becomes the last attention layer in the model. For example, in a two-layer transformer fusion block, we train only the first block initially, while temporarily dropping the second block for several epochs. During this phase, we apply the RollingQ algorithm the first block's 'ss attention layer. After certain number of epochs, the RollingQ algorithm to the attention layer of the second transformer block, since its query can directly influence the prediction. In two-layer setup, the query is primarily influenced by the current input, rather than being irrelevant as in the single-layer case. Therefore, we calculate the rebalance anchor using the expectation the query, as shown in Equation 10, and apply the rotation matrix to the query. This rotation moves the query's 's distribution toward the anchor region, rather than leaving it in its original position.
begin training the second block well. At this point, apply
14

---

RollingQ: Reviving the Cooperation Dynamics in Multimodal Transformer
C. Experiments
C.1. Experimental Setting
Datasets. CREMA-D(Cao et al., 2014), (rrndjilo & Zisserman, 2017), and CMU-MOSEI(Zadeh et al., 2018) are prominent multimodal datasets for emotion recognition and sentiment analysis. CREMA-D contains 7,442 short video clips (2-3 seconds) from 91 actors, annotated with six basic emotions through crowd-sourcing. KineticSound features 31 human action classes from the Kinetics dataset, with 10-second clips manually annotated via Mechanical Turk. MOSEI, the largest of the three, includes over 23,500 sentence-utterance video clips from 1,000+ YouTube speakers, covering 250 topics with aligned language, vision, and sound features. It provides 7-class sentiment annotations and multi-label emotion intensities across six aspects.
Backbones. Our main experiments are conducted using 4-layer ViT(Dosovitskiy, 2020) with pretrained weights from ImageNet-21k (Ridnik et al., 2021) as the backbone, with evaluations on the KineticSound and CREMA-D datasets. For the audio and visual encoder, we adopt a ViT architecture to process multiple frames as input. Following the standard ViT approach, we extract and flatten patches from the input frames, which are then fed into the transformer layers for feature extraction. For MOSEI, we adopts vanilla transformer as encoder following settings of (Liang et al., 2021).
Training settings, For the KineticSound datasets, which consist of 10-second video clips, we extract frames at fps and uniformly sample frames per clip as visual inputs. The audio data is transformed into spectrograms of size 257x1 257x1,004 ,0044 using librosa, with a window length of 512 and an overlap of 353. For CREMA-D, which contains shorter clips, we extract I frame per clip and process the audio into spectrograms of size 257x299, maintaining the same window length and overlap. This approach ensures consistency across datasets while leveraging the strengths of ViT for both visual and audio modalities. As for CMU-MOSEI dataset, follow the preprocessing and settings of (Liang et al., 2021), using the extracted feature provided by (Liang et al., 2021). For training, the batch size is set to 64 for both MOSEI, CREMA-D and KineticSound. The learning rate is fixed at le-3, and SGD is used as the optimizer. The embedding dimensions are 120 for MOSEI and 768 for CREMA-D and KineticSound, and the cosine annealing scheduler is applied across all settings. Training is initialized from scratch for MOSEI, while pretrained models are used for CREMA-D and KineticSound, with all models trained for 30 epochs. This comprehensive setup ensures consistency and highlights the adaptability of the ViT backbone to different datasets and fusion methods.
C.2. Validation with Attention Mask and QUAG A Attention
We ablate RollingQ through mask or average attention score inspired by QUAG Attention (Rawal et al., 2023). As shown in Table 8, when masking one modality, RollingQ outperforms the baseline from 0.2 to 3%. This confirms RollingQ's ability to enhance unimodal feature quality. When using averaged attention scores, in CREMA-D mudio-dominant) vanilla MT exhibits a performance degradation of 2.3% due to over-reliance on audio features. For Kinetic-Sound (relative balance), vanilla MT's performance increases by 1.2% since the model learns an unreasonable attention score due to the self-reinforcing cycle. Conversely, RollingQ maintains stable performance (with less than 1% drop), demonstrating it's leveraging complementary modality information and robustness.
Table 8. Ablation on Kinetic-Sound dataset. Mask V refers force the attention score of visual modality to by adding attention mask, and vice versa. Besides, "uni_A " and "uni_V" refers to the performance of training solely with unimodal model.
Method
Vanilla MT Vanilla MT+Rolling uni A uni V
A (Mask V)
46.6
47.4 (f 0.8)
53.9
V (Mask A)
40.3
42.1 (f 1.8)
57.0
origin
67.0
69.3
-
Furthermore, to explore how RollingQ influence the intra- and inter-modality interactions learned by model, we explore on the baseline with transformer fusion blocks and conduct complete QUAG (Rawal et al., 2023) tests (unimodal, crossmodal, audio-avg, and video-avg) with and without RollingQ. As shown in Table 9, the RollingQ algorithm exhibits more performance drops around 2.7% to 5.0% across all types of QUAG tests compared to baseline. This indicates that Rolling can not only fully leverage both modalities faithfully but also learn comprehensive multimodal interactions.
15

---

RollingQ: Reviving the Cooperation Dynamics in Multimodal Transformer
Table 10. Results on OOD henchmarks with HMDB51 as In-Distribution dataset following the preprocessing and settings of Multi- DoDgDong et al., 2024). General Entropy (GEN) is used to obtain the confidence for OOD data.
OOD-Dataset Metrics/Methods
HMDB
UCF
IDAcc
FRR95 +
AUROC
IDAcc 1
FRR95 +
AUROCT
Vanilla MT
51.6
77.3
64.4
48.7
77.9
70.4
Vanilla MT + RollingQ
54.0 (1 2.4)
74.9 (4 2.4)
68.6 (1 4.2)
51.8 (1 3.1)
76.9 (4 1.0)
73.2 (1 2.8)
Attention score of each modelity when fusion at 2nd leyer
Attentor mocaley when leyer
Attontion
when fusion
0.
altertion
0.0
0.4
0.2
1000
18100
4000
2500
0.0
200
3000
1300
2003
2500
2000
2300
Encodor with 2nd layer
Gradiann
Encoder
fusion layer
Graclernt Encoder when loyor
500
3000
200C
2000
2500
200C
300
10.0
1500
2010
2505
3000
300
1.00
1500
2020
2303
2000
Figure The Lo-Norm gradient of audio encoder and the visual encoder and attention score across modalities on CREMA-D Kinetic-Sound datasets.
Table 9. QUAG operations (Rawal et al., 2023) on Kinetic-Sound dataset, the more drops indicates the more leverage. uni A and uni V refers to performance training solely with unimodal model.
Method
baseline
baseline+Rolling
uni_A
uni_V
origin
69.1
70.1
unimodal
67.8 (} 1.3)
63.8 (L 6.3)
crossmodal
65.5 (L 3.6)
63.8 (1 6.3)
avg-audio
64.8 (4 4.3)
62.9 (4 7.2)
53.9
avg-visual
65.6 (1 3.5)
62.1 (4 8.0)
57.0
C.3. Verification OOD Benchmarks
OOD is challenging task where distribution exhibits distribution shifts compared to training distribution. It a effective way to further verify the dynamic property of multimodal transformer to adaptively assign attention score. Hence, conducts multimodal OOD experiments following (Dong et al., 2024). Both Far-OOD and Near-OOD cases discussed in (Dong et al., 2024) are involved here to present comprehensive evaluation. Specifically, we use HMDB51 1 for Near-OOD test and use HMDB51 and UCF-101 for Far-OOD test.
As shown in Table 10, our RollingQ algorithm outperforms the baseline on metrics, showing that it can case the modality bias with strong generailzation ability and effectively fuse the multimodal feature even for distriiiittiiiioo samples.
C.4. Extension to More Fusion Paradigms
Multimodal transformers are designed under different fusion paradigms such as early fusion, mid fusion and late fusion. We perform further validation of the effectiveness of our proposed RollingQ algorithm under earlier fusion paradigms.
16

---

RollingQ: Reviving the Cooperation Dynamics in Multimodal Transformer
Attertion modality when fusion
2. Aiteetion score modal by tusion 3rd
10
4th
attertin roee
drewin Vole
61.6
adtettan
0.4
6.0
300
1000
2500
2000
2200
2000
0.0
500
1000
1500
2000
2500
3000
Gradient Each Encoder with wher Fuston 2nd layer
Graitiet of Parh Poroder winh wben fosinn And
Grodient of Each Encocer with when "usion at layer
3000
15000
2000
100001
1C00
2500
2000
Figure The L2-Norm gradient of audio encoder and visual encoder attention score across modalities on CREMA-D Kinetic-Sound datasets when implement RollingQ.
Table 11. Performance of vamilla multimodal transformer (Vanilla MT) and RollingQ across different fusion layers, where "2nd" refers to fusion from the 2nd transformer block.
Fusion Layer
Vanilla MT
Vanilla MT RollingQ
2nd
42.3
42.5 (1 0.2)
3rd
41.1
42.2 (f 1.1)
4th
41.4
43.7 (1 2.3)
Specifically, we adopts the same 4-layer ViT backbone on CREMA-D dataset but start to fuse features from two modalities by different layer from 2nd, 3rd and 4th layer. Firstly, we monitor the gradient of unimodal encoders and the attention which are the two driving factors of the self-reinforcing cycle. As shown Figure 8, the gradient difference declines and the attention becomes balanced and more unstable the fusion becomes earlier. Meanwhile, we monitor the gradient and attention after applying RollingQ. As shown in Figure 9, after applying RollingQ, both the attention score and the gradients across modalities are closer, indicating the ability of RollingQ to rebalance the feature quality and revive the cooperation dynamics provided by attention mechanism.
Afterwards, we conduct systematically comparison under different fusion layers. As shown in Table 11, The results reveals that our method can applied to earlier fusion paradigms achieving better performance around (2.3% 1.1% %% 0.2%), but it's more suitable for late fusion paradigm as we discussed the imbalance phenomenon is not significant under earlier fusion paradigms.
17


Reviews:
[{"review_id": "SAFqjQ25xr", "reviewer": "Reviewer_61wP", "summary": "The paper identifies the issue of the self-reinforcing cycle toward the majority modality in multimodal learning. To address this, the authors propose a query rebalance rotation method that disrupts the cycle and rebalances the attention mechanism. Experimental results and visualizations demonstrate the effectiveness of the proposed method.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer 61wP, \n\n**Thank you very much for your affirmation and constructive comments.**\n\nWe carefully considered your comments and conducted corresponding experiments.\n\n**Q1: The change of gradients during training.**\n\nThank you for your question. From the optimization perspective, previous works on imbalanced multimodal learning have extensively discussed the evolution of gradients under a late fusion structure with concatenation or summation fusion, which shows similarities to our setup[1, 2]. A well-established observation is that the superior modality always gains more optimization momentum, as its gradient is higher than that of the weaker modality. \n\nHowever, our setup differs from previous simpler structures where we adopts more complex transformer blocks for multimodal fusion, and your question prompted us to create a clearer visualization to provide more convincing evidence for our theoretical analysis.\n\nAs shown in [Figure 3](https://anonymous.4open.science/r/ICML-2025-Rebuttal-07BE/Figure3.png), the gradient of the audio encoder increases significantly during the mid-stage, when the attention score begins to accumulate in the biased modality. This results in a noticeable gap between the modalities. As the total loss decreases over time, the gradients of both modalities drop, but their relative relationship remains. This further validates our theoretical analysis, showing that the biased modality consistently receives more optimization momentum. The gradient of the weaker modality never exceeds that of the superior modality, which can be explained by Equations 7 and 8 in our paper. Since the only difference in gradients for each modality is $\\frac{\\partial h_i}{\\partial z_i^m}$ and the loss is the multimodal loss, where \"one modality becoming converged\" equals \"the multimodal model becoming converged,\" the total loss becomes very small. This small loss cannot provide enough momentum for the superior modality to optimize effectively. Hence, the gradient of the superior modality will be consistently greater than the weak one. \n\n[1] Peng, X., et al. \"Balanced multimodal learning via on-the-fly gradient modulation.\" CVPR 2022.\n\n[2] Fan, Y., et al. \"Pmr: Prototypical modal rebalance for multimodal learning.\" CVPR 2023\n\n**Q2: Cold-Start and pretraining's influence on self-reinforcing cycle.**\n\nThank you for your question. As discussed in the Experiments section, we are not using cold-start but using a 4-layer ViT-B/16 as the backbone, initialized with pre-trained weights from ImageNet-21k. For the MOSEI dataset, we use the Vanilla Transformer without pretraining. The results shown in Table 1 and the visualizations in Figure 4 demonstrate that even under pretraining conditions, the self-reinforcement cycle still exists and harms the overall performance of multimodal transformers.\n\n**Q3: Limited Improvements** \n\n> Q3.a: Significance tests to prove the gain is statistically meaningful.\n\nThank you for your suggestion. To prove the significance and stability of QRR, we selected 10 random seeds and conducted repeated experiments with the same settings on the CREMA-D and Kinetic-Sound datasets. By applying Pearson correlation analysis as shown in [Table 6](https://anonymous.4open.science/r/ICML-2025-Rebuttal-07BE/Table6.png), with 0 denotes vanilla multimodal transformer while 1 denotes applying QRR algorithm, the coefficient is 0.765 for CREMA-D and 0.698 for Kinetic-Sound indicates its statistically significant on performance improvements with p-value < 0.01 ensures the confidence of the analysis. We'll further accomplish a more systematic analysis of pretraining's influence during the discussion period.\n\n> Q3.b: Limited improvements.\n\nThanks for mentioning this. However, the QRR, which requires only: \n-  **1%** increase on parameters\n-  **0.1%** on GFLOPs \n\nAs shown in [Table 7](https://anonymous.4open.science/r/ICML-2025-Rebuttal-07BE/Table7.png). It achieves considerable improvements on its baseline vanilla multimodal transformer (accuracy increase by **3.1%** on CREMA-D and by **2.3%** on Kinetic-Sound), while gaining comparable and even better performance over other specially designed transformer architecture with much more requirements on computation resources and increased complexity. Hence, the improvements are significant especially taking the computation and time cost into consideration.\n\nAs for CMU-MOSEI, we acknowledge this is an difficult dataset for multimodal sentiment analysis. With all of our comparison methods achieving limited improvements and previous research [3] with three modalities input could only achieve around 1.5% improvements, our QRR algorithm performs relatively significant increment.\n\n[3] Paul Pu Liang et al. \"Multibench: Multiscale Benchmarks for Multimodal Representation Learning.\" NeurIPS, 2021\n\nIf you have any valuable questions or constructive suggestions that could help you understand our work, please tell us and help to produce a higher quality paper.", "claims_and_evidence": "Overall, the claims are well-supported. For example, in the introduction, each claim is supported by recent references or empirical results. Additionally, the investigation into the superior performance of static fusion over dynamic fusion is interesting and highlights the motivations of the paper.", "methods_and_evaluation_criteria": "-", "theoretical_claims": "1. The authors mention that as training progresses, the superior modality gains more attention and is better optimized. However, is this always the case, even in the late training stage when training is nearly converged? I believe that even in the early stages, the superior modality, having a higher gradient for backpropagation, would converge more quickly and reach a (sub-)optimal point, at which its gradient becomes smaller than that of the weaker modality. Could you clarify this reasoning?\n2. The setting in Section 3.2 appears to be under cold-start training, meaning the models are trained from scratch. As far as I know, CREMA-D and CMU-MOSEI are highly biased multimodal datasets, where the feature quality of one modality is significantly better than the others. What if multimodal transformers were pre-trained on more balanced and diverse datasets? Would the self-reinforcing cycle still persist in that case?", "experimental_designs_or_analyses": "1. In table 1, compared with baselines, the improvements are limited. On CREMA-D and MOSEI, the proposed method achieves only a 0.2% gain, while on Kinetic-Sound, its performance is even worse than the baselines.\n2. Given the limited improvement, significance tests are necessary to determine whether the gain is statistically meaningful.\n3. Section 4.3 is interesting, and Figure 4 is clear and effectively demonstrates the impact of the proposed QRR module in enhancing multimodal learning.", "relation_to_broader_scientific_literature": "-", "essential_references_not_discussed": "-", "other_strengths_and_weaknesses": "-", "other_comments_or_suggestions": "Overall, I am inclined to accept the paper, as it presents insightful findings and visualizations, even though the proposed method does not outperform the state-of-the-art.", "questions_for_authors": "Please refer to my comments above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TCRKfftzoW", "reviewer": "Reviewer_FKSM", "summary": "This paper focuses on fusion strategies in multimodal transformers, identifies issues in dynamic fusion, proposes the QRR algorithm, and validates its effectiveness in restoring cooperation dynamics and improving performance through experiments", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer FKSM, \n\n**We appreciate your time and great efforts in reviewing.**\n\nWe carefully considered your comments on the validation of assumptions, lack of in-depth analysis and extension to benchmarks and methods and conducted corresponding experiments and theoretic analysis.\n\n**Q1: The validation of the deactivation of the cooperation dynamics assumption is not adequate since the datasets and settings are limited.**\n\nThanks for pointing out. In the previous version, we provide visualization and analysis of Kinetic-Sound for this assumption. To holistically and systematically validate our assumption, which mainly states that the modality bias in the multimodal training process triggers a self-reinforcing cycle that leads to inequality in feature quality and unreasonable attention score, we conduct experiments on more datasets including CREMA-D, Kinetic-Sound, CMU-MOSEI(A+T), CMU-MOSEI(V+T), UCF-101, HMDB51, whose modality ranging from audio, RGB, text and optical flows and exhibiting variation in modality sequence length. As shown in [Figure 1](https://anonymous.4open.science/r/ICML-2025-Rebuttal-07BE/Figure1.png), with different modalities combinations and dataset scales, the unreasonable attention score attribution agree with our previous study. Besides, the visualization of average key distribution, where the noise input consistently has higher cosine similarity, further validates our assumption and theoretic analysis towards cooperation dynamics under imbalance multimodal learning.\n\n**Q2: Lacks in-depth analysis**\n\n> Q2.a: in-depth analysis of self-reinforcing cycle.\n\nIn addition to the observation of attention score during training dynamics, we monitor the gradient of unimodal encoders to verify our proposition of a self-reinforcing cycle. As shown in [Figure 3](https://anonymous.4open.science/r/ICML-2025-Rebuttal-07BE/Figure3.png), the gradient of both modalities's encoder are similar during the start time. However, the biased modality (audio) encoder increases significantly during the mid-stage, when the attention score begins to accumulate in the biased modality. Later, both of the gradients drop due to the minimization of total loss. This observation further verifies our theoretical analysis and ensures the existence of the self-reinforcing cycle. \n\nMoreover, from the perspective of fusion settings, we visualize the gradient and distribution of attention scores on different modalities under comprehensive fusion paradigms. As shown in [Figure 2](https://anonymous.4open.science/r/ICML-2025-Rebuttal-07BE/Figure2.png), even as the starts of the fusion layers varied, the gradients of the biased unimodal encoder and attention score towards biased modality consistently exhibits considerable greater than the unbiased one, verifying our assumption and analysis of the self-reinforcing cycle.\n\n> Q2.b: in-depth analysis of significance and stability.\n\nTo evaluate the stability and result significance of our method. We conduct repetitive experiments over CREMA-D and Kinetic-Sound with only differences in random seeds. By applying Pearson correlation analysis as shown in [Table 6](https://anonymous.4open.science/r/ICML-2025-Rebuttal-07BE/Table6.png), with 0 denotes vanilla multimodal transformer while 1 denotes applying QRR algorithm, the coefficient is 0.765 for CREMA-D and 0.698 for Kinetic-Sound indicates its statistically significant on performance improvements with p-value < 0.01 ensures the confidence of the analysis.\n\nBesides, ablation on batch size ensures the stability of our method. As results shown in [Table 5](https://anonymous.4open.science/r/ICML-2025-Rebuttal-07BE/Table5.png), with batch size varying from 16 to 256 on CREMAD, the QRR consistently outperforms baseline methods by **0.7% ~ 3%**, revealing the stability of our method.\n\n**Q3: Extension**\n\n> Q3.a: Extension to more benchmarks\n\nIn addition to existing benchmarks and testing, We adopt MultiOOD [1] for the Out-of-Distribution benchmark and further validate the performance of our QRR method. Here, we consider both near-OOD and far-OOD scenarios discussed in MultiOOD. As results shown in [Table 2](https://anonymous.4open.science/r/ICML-2025-Rebuttal-07BE/Table2.png), our QRR algorithm consistently outperforms the baseline on all metrics and fully tests the effectiveness of our method from another perspective. \n\n[1] Hao, D., et al. \"MultiOOD: Scaling Out-of-Distribution Detection for Multiple Modalities\", NeurIPS 2024. \n\n> Q3.b: Combine QRR with existing methods\n\nDue to the time constraints, we haven't done enough attempts to combine our method with existing multimodal methods, which would be conducted during discussion period.\n\nIf you have any valuable questions or constructive suggestions that could help you understand our work, please tell us and help to produce a higher quality paper.", "claims_and_evidence": "The paper's claims are supported by some evidence, but the universality in broader scenarios lacks sufficient proof as experiments are based on specific datasets and settings", "methods_and_evaluation_criteria": "The QRR algorithm is reasonably designed, and the benchmark datasets are suitable. However, the evaluation lacks indepth analysis of dataset characteristics.", "theoretical_claims": "The theoretical analysis is clear.", "experimental_designs_or_analyses": "The experimental design is comprehensive, yet the analysis lacks indepth statistical methods to determine result significance and stability.", "relation_to_broader_scientific_literature": "The QRR algorithm offers a novel approach to addressing the issue of modality imbalance, differing from prior methods that optimized single-modal encoders, thereby extending the existing research framework and providing new insights for multimodal fusion.", "essential_references_not_discussed": "none", "other_strengths_and_weaknesses": "**Strengths:** The research addresses a problem of practical significance by proposing a solution to the fusion challenges encountered in the practical application of multimodal Transformers. The QRR algorithm is straightforward yet effective, requiring no additional training loss and enhancing performance without increasing model complexity, which demonstrates its innovativeness.\n\n**Weaknesses:** I feel that the paper lacks in-depth analysis and does not adequately validate its assumptions. The experimental section employs too few benchmarks, and I would like to see whether the proposed method can provide assistance to existing multimodal models.", "other_comments_or_suggestions": "none", "questions_for_authors": "none", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5XOzCWpVux", "reviewer": "Reviewer_gn7r", "summary": "The paper analyzes an important problem of modality biases in multimodal learning setup. The authors find that the dynamic property of attention is lost during multimodal training; that is, rather than weighing the modalities per-instance, the models just focus on a single (biased) modality, which is overemphasized during training, leading to distribution gaps. To this, author propose Query Rebalanced Rotation (QRR) algorithm that rebalances the query to “revive” the dynamic property of attention by rotating the query vector towards an anchor that provides higher weights to the unbiased modality, thus reducing the modality bias in attention.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer gn7r,\n\n**Thanks a lot for your valuable review, suggestions, and questions.** \n\n**Q1: Extension to more fusion paradigms**\n\n> Q1.a: Analysis of cooperation dynamics. \n\nTo analyze cooperation dynamics across fusion methods, we monitor the gradient of unimodal encoders and the attention which are the two driving factors of the self-reinforcing cycle. As shown in [Figure 2](https://anonymous.4open.science/r/ICML-2025-Rebuttal-07BE/Figure2.png), the gradient difference declines and the attention becomes balanced and more unstable as the fusion becomes earlier.\n\n> Q1.b: Systematic comparison and QRR behaviors.\n\nFollowing descriptions in Appendix B, we train the model progressively and apply QRR. The comparison results are shown in [Table 1](https://anonymous.4open.science/r/ICML-2025-Rebuttal-07BE/Table1.png), where Vanilla MT denotes the vanilla multimodal transformer. The results reveals that our method can be applied to earlier fusion paradigms achieving better performance around (2.3% / 1.1% / 0.2%), but it's more suitable for late fusion paradigm.\n\n**Q2: Verification and validation.** \n\n> Q2.a: Results on OOD benchmarks.  \n\nThanks for providing a novel aspect for testing QRR. We adopt MultiOOD [1]. As shown in [Table 2](https://anonymous.4open.science/r/ICML-2025-Rebuttal-07BE/Table2.png), our QRR algorithm outperforms the baseline on all metrics, showing that our QRR algorithm can ease the modality bias during training dynamics. \n\n[1] Hao, D., et al. \"MultiOOD: Scaling Out-of-Distribution Detection for Multiple Modalities\", NeurIPS 2024. \n\n> Q2.b: Comparison with QUAG[2]\n\nQUAG is designed based on VideoQA tasks using **frozen unimodal encoders**, which sets it apart from our work. Besides, QUAG performs analysis on averaging attention score but our analysis focus on feature space of key, query in attention.\n\n[2] Rawal, Ishaan Singh, et al., “Dissecting Multimodality in VideoQA Transformer Models by Impairing Modality Fusion”. ICML 2024\n\n> Q2.b: QRR might not leverage both modalities due to modality bias.\n\nIn this work, we focus on modality bias that one modality dominates the training process[3], leading to inequality modality feature, unreasonable attention and sub-optimal performance. Hence, we propose QRR to balance the learning of unimodal encoders, revive the cooperation dynamics to leverage both modality.\n\nWe ablate QRR through mask or average attention score inspired by QUAG[2]. As shown in [Table 3,4](https://anonymous.4open.science/r/ICML-2025-Rebuttal-07BE/Table3%20&%204.png), when masking one modality, QRR outperforms the baseline from 0.2 ~ 3%. This confirms QRR's ability to enhance unimodal feature quality. When using averaged attention scores, in CREMA-D (audio-dominant), vanilla MT exhibits a performance degradation of 2.3% due to over-reliance on audio features. For Kinetic-Sound (relative balance), vanilla MT's performance increases by 1.2% since the model learns an unreasonable attention score due to the self-reinforcing cycle. Conversely, QRR maintains stable performance (with <1% drop), demonstrating it's leveraging complementary modality information and robustness.\n\n[3] Peng, X., et al. \"Balanced multimodal learning via on-the-fly gradient modulation.\" CVPR 2022.\n\n**Q3: Supporting details on the stability, efficiency, and performance**\n\n> Q3.a: Batch size ablation.\n\nThrough batch size ablation on CREMA-D in [Table 5](https://anonymous.4open.science/r/ICML-2025-Rebuttal-07BE/Table5.png), QRR maintains stable performance improvements (0.7%-3.1%) compared to baseline from 16 to 256 batch size, demonstrating remarkable training stability.\n\n> Q3.b: Limited performance improvements. \n\nCompared to methods requiring additional parameters and specialized modules, QRR only requires：\n- Parameter increase: **1.0%**\n- FLOPs increase: **0.1%**\n\nas shown in [Table 7](https://anonymous.4open.science/r/ICML-2025-Rebuttal-07BE/Table7.png), while accuracy achieve **3.1%** improvements on CREMA-D and **2.3%** improvements on Kinetic-Sound compared to vanilla multimodal transformer. , QRR is a simple yet effective method that achieves comparable and even better results with less computation cost.\n\n**Q4: Details on paper writing and figures**\n\n> Q4.a: Notation in figure 2.\n\nIn a single fusion layer setting, $q_b$ and $q_r$ are the same. As the number of fusion layers increases, the query $q$ becomes influenced by the input and is no longer a static value. Hence, for each input $q$, the rotation matrix $q_b$ can not ensure to rotate the $q$ to $q_b$, but should be located near this region.\n\n> Q4.b: Detailed captions, missing of related works[5,6] and header of the paper.\n\nThanks for pointing this out, we'll add relative discussion in our revised manuscript. \n\n[5] Buch, Shyamal, et al. \"Revisiting the \"video\" in video-language understanding.\" CVPR 2022.\n\n[6] Pătrăucean, Viorica et al., “Perception Test: A Diagnostic Benchmark for Multimodal Video Models”. NeurIPS 2023", "claims_and_evidence": "I am not convinced why the rotation of the query vector towards the defined anchor would help reduce the modality bias. Further, the results supporting this claim are not consistent and thorough (elaborated below)", "methods_and_evaluation_criteria": "- It is known that the quality of multimodal representations is heavily influenced by the fusion strategy (early/mid/late). In the paper, QRR is based on a very simple late fusion strategy. However, many works adopt early fusion strategies. The paper misses out on (a) systematic comparison with these methods, (b) evolution of cooperative dynamics in these setups, and (c) difference in QRR behaviours in these setups.\n- Does QRR work for the right reasons? The paper, while claims multimodal debiasing, does not provide any results on challenging OOD benchmarks. Multimodal datasets can have biases, and there are multiple challenging benchmarks for evaluating the true multimodal performance of the models. Therefore, performance on biased benchmarks is not representative of the unbiasedness property of the model. The closest to this result in the paper are the noise_T experiments, which are more of a sanity check than an OOD benchmark.", "theoretical_claims": "Not applicable", "experimental_designs_or_analyses": "For CREMA-D dataset, the authors were able to achieve high performance with just a single frame and the audio. This is concerning. This all the more emphasizes that the benchmarks have biases. Being truly multimodal means that the modal *needs* to leverage both the modalities faithfully. However, in this case it could be leveraging weak biases in the multiple modalities, and combining them to produce the answer. Therefore, I strongly suspect that while QRR might re-weigh the attention towards “unbiased modality”, the small gains in performance in some cases might be because t inadvertently makes the model leverage multimodal biases", "relation_to_broader_scientific_literature": "I believe that the findings of the paper regarding unimodal biases are in congruence with the related works in the domain", "essential_references_not_discussed": "- The related works section of the paper does not discuss the necessary works on unimodal and multimodal biases in datasets like [1], that is extremely relevant for such studies\n- Other important works like QUAG and QUAG-attention [2] perform very similar analysis on average key per modality and should be (a) acknowledged in the relevant works, and (b) utilized as a verification that QRR is indeed leveraging both the modalities. Further, datasets like CLAVI [2] and Perception Test [3] could be used as debiased test datasets\n\n\n[1] Buch, Shyamal, et al. \"Revisiting the \"video\" in video-language understanding.\" CVPR 2022. \\\n[2] Rawal, Ishaan Singh et al., “Dissecting Multimodality in VideoQA Transformer Models by Impairing Modality Fusion”. ICML 2024 \\\n[3] Pătrăucean, Viorica et al., “Perception Test: A Diagnostic Benchmark for Multimodal Video Models”. NeurIPS 2023", "other_strengths_and_weaknesses": "**Strengths**\n1. The finding that static fusion techniques might be on-par with dynamic fusion like attention based techniques is indeed surprising and re-affirms the difficulty in taming multimodal self-attention\n2. The paper is aiming to tackle an extremely important problem of biased representation is multimodal learning which can be proven very valuable to the community\n3. QRR is a generic method that can be applied to multiple domains\n3. The paper is cohesively written and easy to follow\n\n**Weaknesses** \\\nLack of convincing results: \n1. No results on OOD benchmarks (see above) and validation tests (see above)\n2. No consistent and significant improvement in accuracy", "other_comments_or_suggestions": "- Should it be q_b instead of q_r in figure 2?\n- For most of the graphs, the legend is not explained. Would be nice to have detailed captions.\n- The running header of the paper isn’t updated", "questions_for_authors": "- What is the effect of batch size on the QRR? How sensitive is it?\n- What is the overhead of QRR (computational complexity and/or increase in run-time/FLOPs)?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper identifies a weakness in attention-based multimodal fusion, that is, when one modality becomes biased, it disrupts the dynamic weighting in Transformers. The authors propose Query Rebalance Rotation to rebalance attention queries to prevent over-reliance on any single modality. QRR restores attention flexibility and improves multimodal learning performance, especially when modality quality varies. Experimental results show its superior performance when compared to other SOTA works. Based on the comments, recommendations, and post-rebuttal discussion, the authors’ point-to-point rebuttal addressed most of these concerns so that no significant issues remain. This paper is acceptable in its current form.
Decision: Accept (poster)