Test-Time Selective Adaptation for Uni-Modal Distribution Shift in Multi-Modal Data
Authors: MingCai Chen
Baoming Zhang
Zongbo Han
Wenyu Jiang
Yanmeng Wang
Shuai Feng
Yuntao Du.
Bingkun BAO
Venue: ICML 2025 poster (2025)
Keywords: test-time adaptation
multi-modal learning
Primary area: general_machine_learning->transfer_multitask_and_metalearning
TL;DR: A study on test-time adaptation for uni-modal distribution shift, an under-explored scenario where the distribution shift occurs in only one modality.

Abstract: Modern machine learning applications are characterized by the increasing size of deep models and the growing diversity of data modalities.  This trend underscores the importance of efficiently adapting pre-trained multi-modal models to the test distribution in real time, i.e., multi-modal test-time adaptation.  In practice, the magnitudes of multi-modal shifts vary because multiple data sources interact with the impact factor in diverse manners.  In this research, we investigate the the under-explored practical scenario *uni-modal distribution shift*, where the distribution shift influences only one modality, leaving the others unchanged. Through theoretical and empirical analyses, we demonstrate that the presence of such shift impedes multi-modal fusion and leads to the negative transfer phenomenon in existing test-time adaptation techniques.  To flexibly combat this unique shift, we propose a selective adaptation schema that incorporates multiple modality-specific adapters to accommodate potential shifts and a ``router'' module that determines which modality requires adaptation. Finally, we validate the effectiveness of our proposed method through extensive experimental evaluations.
Code available at https://github.com/chenmc1996/Uni-Modal-Distribution-Shift.

Test-Time Selective Adaptation for Uni-Modal Distribution Shift in Multi-Modal Data
MingCai Chen 1
Baoming Zhang" 2
Zongho Han Wenyu Jiang? Yanmeng Wang'
Shuai Feng?
Yuntao Du Bingkun Bao
Abstract
Modern machine learning applications are char- acterized by the increasing size of deep models and the growing diversity of data modalities. This trend underscores the importance of efficiently adapting pre-trained multi-modal models the test distribution in real time, i.e., multi-modal test-time adaptation. In practice, the magnitudes of multi-modal shifts vary because multiple data sources interact with the impact factor in diverse manners. In this research, we investigate the the under-explored practical scenario uni-modal dis- tribution shift, where the distribution shift influ- ences only modality, leaving the others un- changed. Through theoretical and empirical anal- yses, we demonstrate that the presence of such shift impedes multi-modal fusion and leads to the negative transfer phenomenon in existing test- time adaptation techniques. To flexibly combat this unique shift, we propose selective adapta- tion schema that incorporates multiple modality- specific adapters to accommodate potential shifts and a "router" module that determines which modality requires adaptation. Finally, we vali- date the effectiivness of our proposed method through extensive experimental evaluations. Code available http://giuuub.comm/mmmmm////// Shift.
O
0
(a) No shift
Uni-modal distribution shift
Figure I. A self-driving equipped with complementary camera and LiDAR sensors navigating under different conditions. In the right figure, the icy road surface primarily impacts the LiDAR sig- nals, while poor light conditions mainly affect the camera signals.
1. Introduction
In recent years, multi-modal learning has emerged as a crucial area in both academic research and practical applica- tions due to its remarkable ability to process wide range of data types (Huang et al., 2021; Xu et al., 2023). In the open-world environment across diverse applications, dis- tribution shifts constantly occur and significantly degrade the performance of static models (Liang et al., 2024; Shi et al., 2024; Han et al., 2023). Existing research on distribu- tion shifts, especially
multi-moda data, often implicitly assumes "global" distribution shift where all modalities experience distribution changes. However, many real-world corrupting factors impact only specific modalities, e.g. light changes can solely shift the distribution of camera signals, while the LiDAR data remains immune as shown in Fig. 1. For the first time, we term this type of shift uni-modal distribution shift, which is distinet from the previously con- sidered global distribution shifts. Nevertheless, this setting is of great practical significance. In many cases, comple- mentary data modalities are often preferred, yet they are likely to be subject to different disturbances. We theoreti- cally show how the uni-modal distribution shift undermines the attention-basee multi-modal model in Sec. 3.3.
"Equal contribution 'Nanjing University of Posts and Telecom- munications "State Key Laboratory for Novel Software Technol- ogy at Nanjing University, Nanjing University 'College of Intelli- gence and Computing. Tianjin University "Joint SDU-NTU Centre for Artificial Intelligence Research (C-FAIR) School of soft- ware, Shandong university. Correspondence to: Yuntao Du <yun- taodu @ sdu. adu..edd.cn.... Bingkun Bao <bingkunbao @ njupt.edu.cn>>
Methodologically. address distribution shifts, Test-Time Adaptation (TTA), which involves continuously updating models to adapt the shifted data distribution in test envi- ronments, has become cutting-edge approach (Wang et al., 2020: Cao et al., 2025; Yang et al., 2024). Therefore, our work also focuses on utilize TTA to solve the uni-modal distribution shift problem. Subsequently, we investigate the adequacy of existing TTA methods for addressing the
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Test-Time Selective Adaptation for Uni-Modal Shift in Multi-Modal Data
(c) Kinefics50 with video shift
(b) VGGSound with audio shift
cal analysis (i.e., negative transfer).
48
We propose a simple but effective method using lightweight adapters for each modality's feature repre- sentation. A learnable "router" designed to automat- ically activate the adapter for the shifted modality and disable the adapters for unshifted modalities.
30
46-
007O1 note
20
10-
42-
Our method is validated through extensive experiments on the uni-modal distribution shifted datasets, and the results show that our approach achieves superior perfor- mance. The comprehensive experimental setup guaran- tees the robustness and generalizability of our method.
ETA
SAR
Spurce
SAR
mathod
method
Figure TTA methods' performance on uni-modal distribution shift. The introduction of adaptation techniques on the unshifted test data results in limited performance gain or even degeration (Pre-trained model (Source) vs. Tent, ETA, and SAR).
2. Related Works
uni-modal distribution shifts. Recent TTA and multi-modal TTA techniques- such as Tent (Wang et al., 2020), ETA (Niu et al., 2022). SAR (Niu et al., 2023), and READ (Yang perform adaptation on every
2.1. Test-Time Adaptation
Distribution shift easily happens between training and test data. Meanwhile, adaptation to the test distribution is lim- ited by various factors, with the most prominent ones being the lack of test data labels and the limited computational resources for further tuning. TTA enables flexible, online adaptation to the current test distribution, attracting increas- ing attention in the field.
et al., mpdelity-specific encoder or the modality fusion module utilizing techniques like batch re- ne-nomaalzzaiin or entropy minimization (detailed in Sec. 2). However, when distribu- tion shift does not occur in all modalities, these approaches lead to redundancy or even negative effects, resulting in overfitting in unshifted modalities to which the original model already generalizes. We perform experiments on multi-modal datasets with uni-modal distribution shift, and discovery limited performance gain. Tent and SAR even exhibit the negative transfer phenomenon (Rosenstein et al., 2005), as shown in Fig. 2. In short, uni-modal distribution shift presents a unique challenge that common TTA meth- ods struggle to address: How to flexibly adapt to the shift in any modality without harming the multi-modal fusion of other unshifted modalities?
Early efforts (Wang et al., 2020; Lim et al., 2023) posit that the statistics in the batch nommalization layers embody distribution knowledge. For mini-batch of data x in the middle batch normalization layer, the activation å is given by: =-B[X] y + B, where E[X] and V[X] are the vvxxj
estimated mean and variance over the data distribution (For illustration, we omit the constant for numerical stability). During test-time, the mean and variance are set as the mov- ing average of training batches' statistics. Here, 7 and , are learnable scale and shift parameters. The re-normalization technique is used to adapt to the new distribution. During test-time, instead of fixing the running statistics (mean Mrun and standard deviation Trun) computed during training, Ad- abn (Li et al., 2017) recomputes the batch statistics for every test batch. The adjusted batch normalization formula for test data x becomes: & ---a 7 + B. This way, it utilizes the statistics from the test distribution. In contrast, TENT (Wang et al., 2020) optimizes the affine parameters and / in batch normalization layers by minimizing the entropy of the prediction probability. ETA (Niu et al., 2022) excludes unreliable and redundant samples from the entropy mini- mization calculation. SAR (Niu al., 2023) performs sam- ple selection from the gradient aspect, removing samples that cause large gradients and encouraging model weights to reach a flat minimum. However, most of the TTA meth- ods do not consider multi-modal scenarios. These methods can typically be deployed, for instance, in the encoder of each modality or the multi-modal fusion module. But in the context of the new uni-modal distribution shift within
To achieve this goal, we propose straightforward yet ef- fective approach for flexible modality-specific adaptation. mainly uses a learnable "router" that determines which modality should be adapted. On top of that, we introduce a lightweight adapter for each modality's 's feature represen- tations accommodate potential distribution shifts test time. By allowing the router to automatically activate one adapter while disabling others, we achieve flexible selective adaptation. All components are updated in an end-to-end manner during test time using simple self-training tech- niques. Consequently, our method effectively enables safe TTA for uni-modal distribution shifts in multi-modal data. It is verified on multi-modal datasets, namely Kinetics50 (Kay et al., 2017) and VGGSound (Chen et al., 2020), with 21 types of uni-modal distribution shifts across different modalities. The main contributions of this work are:
We identify the unique challenges of uni-modal shift in multi-modal data through theoretical analysis (i.c., large fluctuations in pross-modal attention) and empiri-
2

---

Test-Time Selective Adaptation for Uni-Modal Shift in Multi-Modal Data
multi-modal data, these methods would suffer from negative transfer issues.
unshifted audio
shifted video
2.2. Test-Time Adaptation for Multi-Modal Data
Multi-modal learning has emerged as promising approach for understanding the world through different data modal- ities. Moreover, publicly available large-scale pre-trained multi-modal models have been widely applied. However, in the open world, the dynamic and continuously changing data distribution damages the applicability (Schwag et al., 2019; Zhou, 2022; Cao et al., 2023; Xiong et al., 2024; Cao al., 2025). Thus, there is an urgent need for the adapta- tion of corresponding TTA techniques to efficiently adapt to new multi-modal data. In MM-TTA (Shin et al., 2022), a co-training-stye (Blum & Mitchell, 1998) modal pseudo-labeling followed by inter-modal pseudo-label re- finement, is introduced for 3D semantic segmentation. To bridge the gap distribution shift between modalities. It maintains two norm statistics: one is directly updated by the test data, another is slowly updated with momentum from the direct-updated batch norm parameter. The work focuses on the situation where distribution shifts occur within all modalities, yet adaptations should be separately adjusted. Compared with our method, it manually assigns different de- grees of shift to specific modalities. In contrast, our method performs dynamic routing in an end-to-end and learnable manner. READ (Yang al., 2024) investigates multi-modal shifts and includes preliminary experiments on uni-modal shifts. It identifies the "reliability bias" problem, indicating that when certain modalities are affected by distribution shifts, the information discrepancies between modalities are amplified. To address this, READ modulates the attention- based fusion layers. However, rather than reducing attention on the shifted modality, we aim fully extract informa- tion from them through modality-specific adaptation, which utilizes the shifted data more efficiently.
audio encoder
visual encoder
router c
identity
adapter
identity
adapter
cross-modal fusion layer
Classifier
Figure 3. The architecture of our model.
k. Additionally, the conditional distribution of the target label Y given the input x11), x2) x (M) re-
mains the same, i.e., Ps(Y (1)(1)) x2 x (M)
Pr (x(1, x2) x...),,((M)))
In TTA for uni-modal distribution shift, the model tasked with generating predictions for streaming multi-modal data. Simultancously, model updates are carried out to adapt to the uni-modal shifted distribution.
3.2. Architecture of Pre-trained Model
For simplicity, we will illustrate our approach using two modalities: audio and video. This framework can be eas- ily extended to more general multi-modal cases. Suppose there is model pre-trained on labeled multi-modal source dataset. As shown in Fig. 3, the model comprises modality- specific encoders (visual encoder f((') and audio encoder f()),, cross-modal fusion layer f(mm) and a classifier f(e), Both f(a) and f(+)) are transformer encoders (Vaswani et al., 2017) that map the input modality into series of tokens. The cross- modal fusion module f(m) is mainly self-attention layer (Vaswani et al., 2017).
3. Method
3.1. Problem Definition of Uni-Modal Distribution Shift
In multi-modal learning, data is represented through diverse modalities such as text, images, and audio. Uni-modal distri- bution shift occurs when the distribution of only one modal- ity changes between the source and target domains, while the others remain unchanged. Notably, we do not know which modality the shift occurs in. This presents unique challenges as the model must perform selective adaptation. Its formalization is: The joint distribution of the multi-modal source domain is given by (s(x!!), (2) (M), Y). For the target domain, it is (r(()), (2) x (M), Y).
3.3. Theoretical Analysis
With the problem setting and model architecture established, we initiate on theoretical analysis aimed at understanding the implications of uni-modal distribution shift on multi- modal fusion. We begin our analysis from the core module of mudti-modal fusion: the self-attention mechanism. plays pivotal role in multi-mode fusion by aggregating
Here, the marginal distribution of the k-th modality in
the target domain is shifted, i.e., (x( 4 (k())
while for the other modalities. (x) = Ps( vi +
3

---

Test-Time Selective Adaptation for Uni-Modal Shift in Multi-Modal Data
intra-modal and inter-modal token representations with dif- ferent weights. Mathematically, it is defined by the formula:
the single shifted modality). The variance of that change tends to be greater than that of two inter-modal tokens, i.e., one token from shifted modality and another token from the unshifted modality (We first prove the expectation of the change is zero in the appendix). This implies that the attention logits within the shifted modality have large fluc- tuations (e.g., high-variance inputs for the calculation of softmax of attention scores) after the shift occurs. As a result, the softmax outputs become more "peaked" within the shifted modalities' self-attention. Consequently, it be- comes more likely that fewer cross-modal attention events occur, especially between the shifted and unshifted modali- ties. Ultimately, this has a negative impact on multi-modal fusion.
Attention(@(=), (=)).(z)))
softmax {(()){())*))
(1)
V(z),
vdl
where query Q(-), key K-), and value V() are the linear transformation of the tokens' representation z, dg is the scale factor. The attention logit, a crucial determinant in the self-attention, is defined as the inner product between the query and key matrices prior to the softmax function. Its value depends on the representations of two tokens involved in the computation. Formally, we define the attention logit:
Definition 3.1 (Attention Logit). The atlention logit (AL) is defined as the inner product between query and key ma- trices:
In addition the above analysis, we also theoretically demonstrate that, under non-zero-meat shift, rrosl fusion faces similar challenges from the expectation per- spective, as detailed in the appendix.
z,) = Q(z,) ((z)) (WoW 2 (2)
3.4. Selective Adaptation with Model-specific Adapters and Router
where 2; E F and zj E Rf are pair of embedding of tokens involved in the self-attention calculation, and Wº, W* E FFF are the linear projection matrices.
Given that an unknown modality would be going through shift and harm multi-modal fusion, we propose to flexibly adapt the distribution shift of different modalities. We intro- duce two lightweight components on top of the base model: (I). Model-specific adapters: For the two modalities, we use two learnable matrices, o(") and (("), to fit the possi- ble domain shift. (II). Router: Router is parameterized by modality shift semaphore $ = [s(*), s(*]]: A vector whose length equals the number of modalities, indicating the prob- abilities of a modality shift. In our case, the vector size is 2. During test-time, the two components for selective adaptation are updated end-to-end.
However, the occurrence of a distribution shift in any modal- ity can disrupt this mechanism. Without loss of generality, we analyze the shift in the audio modality: Token repre- sentations of the shifted audio are corrupted, and we de- note it as 2!a) and =ta) different from the tokens from unshifted modality = The distribution shift would result in the changes of attention logit between two shifted audio tokens: AL(*(") _ AL(z!"), =(()). The change be- tween one shifted audio token and unshifted one video is token: AL(2!)) (e)) AL(z)! (
Proposition 3.2. Under the zero-mean additive shift as- sumption (Kim et al., 2020):
3.5. Forward Pass
Feature Extraction During test-time, the test samples arrive in batches (ax(a), (n)) E B. Our model's forward process, illustrated in Fig. 3, first, extract features from the test data using the modality-speciff encoders:
sup [ ( ,2k (0)) AL(2( .kk ))
(a)
(v)
change of emess moddll AI. after shift
< (D AL( 2 AL (a) (a)
(()) = (2)(a)))
change of intra intra-shited modal AL after shift
(4)
200) f( (1).
when [[[lz( (0) < [[ (0) 123 + E1 } 1121 + [I
(0)
1|
(3)
Selective Adaptation Next, each of these features multi- plies the corresponding modality adapter to form the distribution-alignedl features:
where sup(.) is the least upper bound. Ej is the noise on (1.]. E[|lz)
token 2) E[[ ("))1 E||l("))
(1)1 [[2]. E|
(a)
are the expected value of the squared norm of the token from
(r) (v) (z)()
(0) + +I),
different modalities and noise, respectively.
(5)
5(() - ( + +I,
Proof. See the appendix for the formal proof.
Remark 3.3. The presence of uni-modal distribution shift leads to specific changes in the attention logit of two to- kens from the shifted-moot modality (both tokens belonging to
where / is an identity matrix make the adaptation works in a "residual" way. It allows the adapter & to build on the identity mapping and learn only the necessary increments.
4

---

Test-Time Selective Adaptation for Uni-Modal Shift in Multi-Modal Data
We use the modality shift semaphore to decide whether to use the original or the adapted features. To provide better exploration of the solution space with differentiable approximation to a categorical distribution, we input the modality shift semaphore S = [>(*), s(o) into the Gumbel- Softmax function (Jang et al., 2017). Concretely, we first sample a Gumbel noise vector g:
3.6. Adaptation through Self-Training
To produce supervision signals without annotations, we use the self-training techniques that common in TTA and other relevant tasks (Sohn et al., 2020; 2013).
Self-Trainin Loss We first generate pseudo-label based on the model's predictions P. The class index corresponding to the most confident prediction is used as the pseudo-label
g = - log()),
(6)
i = arg max; (p); Additionally, we threshold each sam-
ple's loss according to the maximum probability to ensure accurate self-supervision:
where u ~ (O, and u has the same dimension as S. We then add this Gumbel noise to our input logits to obtain the noisy logits. We apply the softmax function to the semaphore to get the selection weights w:
Leetf = 1 (mmax()), > T) P),
(10)
where threshold r filters out less confident predictions. 1()) is the indicator which transforms the bool value in it to I or O. CF(() is the cross-entropy loss function (The loss is averaged in each batch. We omit this for simplicity of illustration.).
w softmax ((S + g)/T),
where 7 is scaling temperature. To make the obtained weight decides whether to use the original or the adapted features, we obtain a convex combination of the original feature and the adapted feature. The weights w act as coef- ficients for the adaptation choice, determining how much of the corresponding modality's feature should be adapted:
Balance Loss Following (Yang et al., 2024; Zhou et al., 2023), we use the negative entropy loss term Loal to regu- larize a balanced class distribution of the test batch. The details are in the appendix.
x(0) = (r))) 2(1) + (1-w ((())) 2(1),
(8)
Overall Self- Self-Training Objective The final adaptation ob- jective combines these two losses:
2(0) = (o) (a) + (1 - (e)) 2(())
The choice here is implemented in soft way to achieve sufficient training for both adapters. When (+()) R 1, the output features zz() are more similar to the adapted feature 2(,), while (()) ~ O causes z(() to be close to the original feature ((a)
L = Lhat + affselff
(11)
where a is a weight between the two terms. By minimiz- ing this combined loss, the model can benefit from both the regularization of the class distribution via Loa! and the exploitation of test data through pseudo-labeling in Tesetf-
Multi-Modal Fusion Subsequently, the features pass through an attention layer, which follows the well-known design from (Vaswani et al., 2017), before entering the clas- sifier to yield the class prediction p::
4. Experiments
4.1. Experiment setting
p = Softmax (( (mean()(m) (z()) (z(1), (())
(9)
Dateset Construction To validate the effectiveness of our method, we perform comparison experiments and on two multi-modal datasets, Kinetics50 (Kay et al., 2017; Peng et al., 2022) and VGGSound (Chen et al., 2020), with diverse domain shifts. The dataset construction and cor- ruption follow the procedures in (Hendrycks & Dietterich, 2019; Yang et al., 2024). Specially, we use the Kinetics50 dataset, which is 50-class subset of the Kinetics dataset and contains 29,204 YouTube videos of human actions (e.g.. wrestling or eating cake), 15 types of video shifts are in- troduced. These include Gaussian noise, shot noise, im- pulse noise, defocus blur, glass blur, motion blur, zoom blur, snow, frost, fog, brightness, elastic, pixelate, contrast, and JPEG. Additionally, 6 types of audio corruptions-Gaussie noise, traffic noise, crowd noise, rainy noise, thunder noise, and windy noise-are applied. The videos in Kinetics50 are trimmed to a duration of 10 seconds as per (Yang et al., 2024). For the VGGSound dataset, which consists of 14,046
where mean(-) calculates the mean features of all tokens output by the fusion layer.
Traverse Inference Schema Test-time adaptation, unlike the classic supervise training framework, must produce pre- dictions for performance evaluation as soon as the data arrive. If the router has not fully converged in the early data stream, performance may be compromised. To ad- dress this issue, we propose a straightforward schema called traverse inference. In this approach, predictions for test- ing are generated through an ensemble of multiple forward passes, each activating one of the adapters. For instance, are
both forward processes for (5(a), 2(+) and traversed. The final predictions given by choosing the most confident predictions from these outputs. It is worth noting that this schema is only for inference, adaptation only needs one forward routed by the selective adaptation schema.
5

---

Test-Time Selective Adaptation for Uni-Modal Shift in Multi-Modal Data
Table 1. Comparisons with SOTA methods on Kinetics50-C henchmark with corrupted video modality. "I.F" refers late fusion, "AF" refers to attention- fusion. The results are the mean values among 5 random seeds, and the best results are highlighted in bold. Performance of other methods are from (Yang et al., 2024).
Noise
Blur
Weather
Digital
Methods
Avg.
Gauss.
Shot
Impul.
Defoc.
Glass
Mot.
Zoom
Snow
Frost
Fog
Brit.
Contr.
Elas.
Pix.
JPBG
Source
31.8
33.4
31.7
64.0
54.3
67.5
61.9
50.9
54.8
38.4
72.3
44.0
60.2
61.7
56.4
52.2
MM-TTA
46.2
46.6
46.1
58.8
55.7
62.6
58.7
52.6
54.4
48.5
69.1
49.3
57.6
56.4
54.6
54.5
Tent
28.6
29.8
28.3
63.4
51.1
67.7
61.7
46.5
51.3
24.5
72.3
38.6
60.7
61.8
54.9
49.4
LF
ETA
31.8
33.3
31.6
64.2
54,6
67.7
62.2
51.3
54.7
38.1
72.5
44.2
60.4
62.0
57.0
52.4
SAR
31,9
33.3
31.7
63.8
54.0
67.7
50.7
54.5
38.8
72.3
44.0
60.3
62.0
56.5
S2.2
READ
34.0
34.5
33.8
65.3
57.7
68.7
64.9
56.1
57.5
41.1
73.2
48.7
62.9
64.6
59.2
54.8
Source
46.8
48.0
46,9
67.5
62.2
70.8
66.7
61.6
60.3
46.7
75.2
52.1
65.7
66.5
61.9
59.9
Tent
46.3
47.0
46.3
67.2
62.5
71.0
67.6
63.1
61.1
34.9
75.4
51.6
66.8
67.2
62.7
59.4
ETA
46.8
47.6
47.1
67.2
62.7
70.6
67.2
62.3
60.9
46.7
75.2
52.4
65.9
66.8
62.5
60.1
AF
SAR
46.7
47.4
46.8
67.0
61.9
70.4
66.4
61.8
60.6
46.0
75.2
52.1
65.7
66.4
62.0
59.8
READ
49.4
49.7
49.0
68.0
65.1
71.2
69.0
64.5
64,4
57,4
75.5
53.6
68.3
68.0
65.1
62.5
Ours
52.6
52.3
52.0
68.7
68.0
70.7
68.8
65.2
66.6
64,3
74.6
57.4
70.5
69.0
66.2
64.5
10 10-second videos of everyday audio events labeled into 309 classes (such as snake rattling or pheasant crowing), the same uni-modal shifts as in Kinetics50 are constructed.
ous cortuption types such as noise, blur, weather, and dig- ital in the video modality, and noise and weather in the audio modality, our method achieves the best average re- sults. In the Kinetics50-C benchmark with corrupted video modality (Table 1), across all corruption types like Gaussian, shot, and impulse noise, defocus, glass, motion, zoom blur, snow, frost, fog, brightness, contrast, elastic, pixelate, and JPEG compression, our method has the highest average ac- curacy (64.5). Similarly, in the audio-corrupted Kinetics50- C benchmark (left part of Table 3), for noise (Gaussian, traffic, crowd) and weather (rain, thunder, wind) corrup- tions, our method attains the best average aceuracy (71.5), In the VGGSound-C benchmark, whether with corrupted video (Table 2) or audio (right part of Table 3) modalities, our method leads in terms of average performance.
Model and Training Protocol For fair comparisons. all compared methods use the same backbone models, pre- trained parameters and training protocol. The CAV-MAE (Gong et al., 2023) model is pretrained on the training sets of Kinetics50 and VGGSound dataset (To simulate the TTA setting, the TTA algorithm does not access training data. The original training set of Kinetics50 and VGGSound, as source domain, are only used for pre-training). The hyperparameters and other details are in the appendix.
To comprehensively verify the ef- fectiveness of our methods, We compare our method with four SOTA TTA methods, including the direct transfer of single modality TTA methods, namely Tent (Wang et al., 2020), ETA (Niu et al., 2022), and SAR (Niu et al., 2023), on the multi-modal TTA setting and the multi-modal TTA methods: MM-TTA (Shin et al., 2022) and READ (Yang et al., 2024). Following (Yang et al., 2024), report the these methods' results with two versions of modality fusion: Late fusion (LF), e.g., directly emsembling the outputs of modality-specifiic encoders as the input to the classification head. iileettion-bssee Fusion (AF), e.g., the outputs of the modality specific encoders are first fused through an atten- tion layer, and the resulting fused representation is then fed into the classification head.
Compared Methods
For the Kinetics50-C dataset, video can be considered a dominant modality, In the case of video-related corruptions (Table 1), the performance gap between our method and others is more pronounced. This indicates more substantial gain in the dominant modality shift scenario. On the other hand, for the VGGSound-C C dataset with video shift (Table 2), the performance improvement of our method over SOTA is relatively less. This shows that on VGGSound-C with vide shift, there is less room for performance cnhancement compared to the dominant modality shift cases like video in Kinetics50-C.
In summary, the experimental results clearly demonstrate that our method outperforms SOTA across different modal- ity shift types and datasets, with more significant gain in dominant modality shifts, while also highlighting the performance limitations in certain datasets and modality- corruption combinations.
4.2. Experimental evaluation
Q1: How is the performance of our method compared with SOTA TTA method?
The presented tables 1, 2, and 3, compare our method with SOTA techniques across different datasets (Kinetics50-C and VGGSound-C) and modality shift types (corrupted video and audio modalities). In all three tables, for vari-
Q2: How is the shifted data behaves from the attention view?
In our experiments as shown Fig. 4, we analyze the atten-
6

---

Test-Time Selective Adaptation for Uni-Modal Shift in Multi-Modal Data
Table Comparisons with SOTA methods on VGGSound-C benchmark with corrupted video modality.
Noise
Blur
Weather
Digital
Methods
Avg.
Gauss.
Shol
Impul.
Defoc.
Glass
Mot.
Zoom
Snow
Frost.
Fog
Brit.
Cuntr:
Elas.
Pix.
JPBG
Source
37.7
36.5
37.8
52.7
51.3
55.2
53.7
51.9
52.3
50.4
55.3
45.2
52.5
51.7
52.3
49.1
MM-TTA
7.1
7.3
7.3
44.8
41.5
48.0
45.5
27.4
23.5
30.5
46.9
24.2
40.3
40.7
45.7
32.0
Tent
7.6
6.8
7.2
53.1
52.1
55.5
54.5
52.6
32.7
16.0
55.9
16.6
52.6
54.2
53.1
38.0
LF
37.7
36.5
37.7
53.2
52.3
56.0
52.9
51.0
55.0
53.5
52.3
52.7
SAR
37.7
36.4
37.7
52.8
51.5
55.5
53.9
51.9
52.5
50.4
55.4
44.8
52.7
51.8
52.3
49.2
READ
42.1
41.5
42.1
49.3
50,9
53,5
52.5
50.6
52.1
51.1
54.0
46.2
52.5
49.1
50.2
49.2
Source
52.8
52.7
52.7
57.2
57.2
58.7
57.6
56.4
56.6
55.6
58.9
53.7
56.9
55.8
56.9
56.0
Tent
52.7
52.7
52.7
56.7
56.5
$7.9
$7.2
55.9
56.3
56.3
58.4
54.0
37.4
56.2
56.7
55,8
ETA
53.0
52.8
53.0
57.2
57.1
58.6
57.8
56.3
56.8
56.4
59.0
54.1
57.4
56.1
57.0
56.2
AF
SAR
52,9
57.2
58.6
56.7
55.9
58.9
57.0
56.0
57.0
56.1
READ
53.6
53.6
53.5
57.9
57.7
59.4
58.8
57.2
57.8
55.0
59.9
55.2
58.6
57.1
57.9
56.9
Ours
53,9
53.9
54.0
57.6
58.0
59.0
58.6
56.9
57.0
56.6
59.8
54.7
58.6
56.7
57.9
56.9
Table Comparisons with SOTA methods on Kinetics50- (left) and VGGSound-CC C (right) benchmarks with corrupted audio modality.
Kinetics50-C
VGGSound-C
Methods
Noisc
Weather
Noise
Weather
Avg.
Avg.
Gauss.
Traff.
Crowd,
Ruin
Thund.
Wind
Gauss.
Traff.
Crowd.
Rain
Thund.
Wind
Source
71.1
67.8
67.4
67.4
70.6
68.6
68.8
29.5
17.1
22.6
17.3
33.7
20.6
23.5
MM-TTA
70.8
69.2
68.5
69.0
69.8
69.4
69.4
14.1
5.2
6.4
6.9
8.6
4.5
7.6
Tent
71.1
68.6
67.8
67.4
71.2
68.9
69.2
6.4
2.1
2.9
1.9
9.5
3.1
4.3
ETA
71.2
67.9
67,5
67.8
70.9
68.7
69.0
28.8
17.1
22.4
17.4
33.8
20.4
23.3
SAR
71.1
67.5
67.4
67.4
70.6
68.6
68.8
28.5
16.6
22.4
17.4
33.7
20.2
23.1
READ
71.3
68.5
68.5
68.4
71.8
69.0
69.6
36.4
25.3
28.9
27.3
35.6
26.6
30.0
Source
73.7
65,5
67.9
70.3
67.9
70.3
69.3
37.0
25.5
16.8
21.6
27.3
25.5
25.6
Tent
73.9
67.4
69.2
70.4
66.5
70.5
69.6
10.6
2.6
1.8
2.8
5.3
4.1
4.5
ETA
73.7
66.1
68.5
70.3
67.9
70.1
69.4
39.2
26.1
22.9
26.0
31,7
30.4
29.4
SAR
73.7
65.4
68.2
69.9
67.2
70.2
69.1
37.4
9.5
11.0
12.1
26.8
23.7
20.1
READ
74.1
69.0
69.7
71.1
71.8
70.7
71.1
40.4
28.9
26.6
30.9
36.7
30.6
32.4
Ours
74.5
69.6
70.5
71.4
72.0
71.0
71.5
41.5
31.8
30.9
32.6
38.9
32.6
34.7
tion mechanisms our proposed method in the context of multi-modal fusion, particularly when dealing with cor- rupted video inputs. Shifted video features would exhibit smaller similarity the unshifted audio modality. This discrepancy led reduced attention allocation to the audio modality, ultimately resulting in poorer multi-modal fusion outcomes. When examining the attention maps generated by the pre-trained model, in the first row of Fig. 4, we note that there was a reduction in video-to-audio attention, as evidenced by the darker color in the top right of the first row method demonstrated recovery
Table 4. The comparison of number of trainable parameters. "Tent's computation cost represents series of methods that tunes the parameters normalization layer.
Time cost (seconds epoch)
Methods of Parums (million)
Kinetics50-C
VGGSound-C
Tent" READ Ours
0.226
54.56
0OM
1.772
35.43
190.70
1.180
35.96
191.76
of figures. In contrast, in attention dynamics: the video modality not only attended to itself but also engaged more effectively with the audio modality, as highlighted by the lighter color in the top right of corresponding figures of the second row.
Q3: How is the efficiency of our method compared with other TTA method?
Test-time training methods prioritize computational effi- ciency as they aim to optimize model performance without incurring excessive resource costs, particularly in environ- ments with limited GPU memory or time constraints, In this context, we evaluated various approaches in Table including Tent, READ, and our proposed method. Tent, which adjusts normalization layer parameters, boasts the fewest trainable parameters at 0.226 million. However, it significantly increases computational demands, leading to out-of-memann (OOM) errors on the VGGSound- dataset.
Furthermore, our method consistently achieved smaller vari- ance across all five video shifts as shown in the number in each figure in Fig. 4, indicating more stable and reliable attention pattern. These results suggest that our approach not only enhances the attention allocation towards relevant modalities but also contributes to more coherent and con- sistent fusion of information across modalities.
This inefficiency stems from the necessity to save gradient
7

---

Test-Time Selective Adaptation for Uni-Modal Shift in Multi-Modal Data
Gaussian noise
shot noise
impulse noise
defocus blur
glass blur
video
Vidaa
videa
89.20
91:41
88.26
103.01
105.35
audio
oipne
-5
video
audio
video
audio
video
video
-10
-15
81.08
85.64
83.59
93.96
102.84
audio
audio
audio
audio
-00
video
audio
video
audio
video
audio
video
audio
video
audio
Figure 4. Attention map (logit) of the modli fusion layer Kinetics50-C dataset with five vidco shifts. The first row shows the pre-trained model's 's attention maps, the second row is our method's attention maps. The number the middle is average variance over the whole dataset. The first 196 tokens are from the video modality, the last 512 tokens from the audio modality. The lighter the color, the bigger the attention logit as shown the right color bar.
graphs for shallow layers, which complicates the training process and escalates memory usage. In contrast, both our method and READ, despite having more parameters- million and 1.772 million pereetivellydemmosnite computational efficiency. They require less GPU memory and avoid OOM issues, highlighting critical finding that higher parameter count does not inherently compromise source efficiency. This analysis underscores the importance of balancing parameter count and computational demands in the design of effective test-time training methods. The detailed hardware information is provided the appendix.
Table 5. Ablation study on Kinetics50 C with five video shifts.
Ablation
Gauss.
Traff.
Crowd.
Rain
Thund.
greater
52.6
52.3
68.0
-Gunbel-sotmma
52.0
51.9
51.4
67.8
68.0
-Traverse inference
51.5
51.5
51.1
67.2
67.5
-Selective adaptation
50.3
50.7
50.5
65.1
65.7
Source model
46.8
48.0
46.9
67.5
62.2
53.00
52-
52.50
Q4: Are the components of our method effective?
Acourss
52.25
In our ablation study, we systematically strip off three com- ponents one by one from our proposed method to evaluate their contributions to performance. We first try to change the Gumbel-softmal to direct softmax function. Then, we removed the test-time ensemble schema, followed by the selective adaptation schema. which utilizes the adapted fea- tures across all modalities. Without these adapters, the model reverts to the pre-trained source model. The results, summarized in Table 5, demonstrate the effectiveness of the components in our method.
Shot
temperaturr
Figure 5. Sensitivity test of loss coefficient batch size.
tests in the appendix.
5. Conclusion
Two plots in Fig. show the sensitivity test of two hy- perparameters. The first plot shows that as the batch size increases from 128, the accuracy for both Gaussian and shot noise declines. This indicates that the batch size has strong influence the model's performance. Even though layer normalization is used, the significant impact of the batch size implies that the update rate of our model, which related to the batch size, is crucial in the TTA task. The second plot shows that our method is not sensitive to the temperature for Gumbel-softmax, We show more sensitivity
In this paper, we systematically define and study the prob- lem of test-time uni-modal distribution shift. We find that previous methods not efficiently transfer to this new setting, primarily due negative transfer phenomenon. Without knowledge of which modality will experience distribution shift, blindly applying adaptation methods to modality-specine modules is not only redundant but also po- tentially has negative effect. Furthermore, we theoretically analyze how uni-modal distribution shift hinders a stable attention mechanism modall fusion. To address the unique challenges posed by this novel problem, we propose
8

---

Test-Time Selective Adaptation for Uni-Modal Shift in Multi-Modal Data
simple method with the core idea of selective adaptation.
Chen, H., Xie. W., Vedaldi, A., and Zisserman, A. Vg- gsound: A large-scale audio-visual dataset. In ICASSP 2020-2020 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 721-725. IEEE, 2020.
The model maintains router to choose which modality to adapt and modality-speciife adapters in a self-training fashion. As result, our method effectively achieves test- time adaptation for uni-modal distribution shift. We conduct extensive experiments to demonstrate the superiority of our method. From the perspectives of attention maps and com- putational cost, we also prove the practical significance of our method. We anticipate that the proposed new problem has both application and research value. By addressing these challenges, the broader implications of our work can be further expanded, leading to more reliable and deployable multi-modal models.
Gong, Y., Rouditchenko, A., Liu, A. H., Harwath, D., Kar- linsky, L., Kuehne, H., and Glass, J. R. Contrastive audio-visual masked autoencoder. In The Eleventh In- ternational Conference on Learning Representations, 2023. URL https:://////......tt http::///pppeevveeeeeeee net/ forum? Ad=QPtMRyk5rb.
Han, Z., Zhou, G., He, R., Wang, J., Wu, T. Yin, Y., Khan, S., Yao, L., Liu, T., and Zhang, K. How well does gpt-4v (ision) adapt to distribution shifts? a preliminary investi- gation. arXiv preprint arXiv:2312.07424, 2023.
Acknowledgements
This work was supported by the National Natural Science Foundation of China under Grants 62325206, the Key Re- search and Development Program of Jiangsu Province un- der Grant BE2023016-4, Natural Science Research Start- up Foundation of Recruiting Talents of Nanjing University of Posts and Telecommunications (Grant No. NY224061, NY224103), and National Natural Science Foundation of China (Grant No. 624B2100).
Hendrycks, D. and Dietterich, T. Benchmarking neural network robustness to common corruptions and perturba- tions. arXiv preprint arXiv: 1903. /2261, 2019.
Huang, Y., Du, C.,, Xue, Z., Chen, X., Zhao, H., and Huang, L. What makes multi-modal learning better than single (provably). Advances in Neural Information Processing Systems, 34: 14:10444-0956, 2021.
Impact Statement
Huang, Y., Lin, J., Zhou, C., Yang, H., and Huang, L. Modality competition: What makes joint training of multi- modal network fail in deep caaming?(provallee national conference on machine learning, pp. 9226-9259, PMLR, 2022.
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
In Inter-
Jang, E., Gu, S., and Poole, B. Categorical reparame- terization with gumbel-softmax. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL https: / // //oeerreiiew ew net. forum? dim.mk338.........
References
Blum, A. and Mitchell, T. M. Combining labeled and unlabeled data with co-training. In Proceed- ings of the Eleventh Annual Conference Computa- tional Learning Theory, pp. 92-100, 1998. doi: 10. 2145/279943.279992. URL https//s:://dii org//000......... 1145/279943.2702
Kay, W., Carreira, J., Simonyan, K., Zhang, B., Hillier, C., Vijayanarasimhan. S., Viola, F., Green, T., Back, T., Natsev, P., et al. The kinetics human action video dataset. arXiv preprint arXiv: 001.//055009955 2017.
Cao, H., Xu, Y., Yang. J., Yin, P., Yuan, S., and Xie, L. Multi-modal continual test-time adaptation for 3d seman- tic segmentation. In Proceedings of the IEEE/CVF In- lernational Conference on Computer Vision (ICCV), pp. 18809-18819, October 2023.
Kim, Y., Soh, J. W., Park, G. Y., and Cho, N. Transfer learning from synthetic to real-noise denoising with adaptive instance normalization. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pp. 3479-3489. Computer Vision Foundation / IEEE, 2020. doi: 10.1109/CVP22220020000000 URL
Cao, H., Xu, Y., Yang, J.. Yin, P., Ji, X., Yuan, S., and Xie, L. Reliable seatial-temporal voxels for molti-modal test-time adaptation. In Leonardis, A., Ricci, E., Roth, S., Russakovsky, O., Sattler, T., and Varol, G. (eds.), Computer Vision - ECCV 2024, pp. 232-249, Cham, 2025. Springer Nature Switzerland. ISBN 978-3-031- 73390-1.
httts:://opeaace com/content_com
thecvf.
Transfer_ httpaaaeeeeennnneeooosstttttttee
Meal-Noise_Dennoisinng http:iiiiiinnntttttaaapt.tteeeee Thstance__cVP__200__pppe html.
9

---

Test-Time Selective Adaptation for Uni-Modal Shift in Multi-Modal Data
Lee, D.-H. Pseudo-label:
The simple and efficient semi- supervised learning method for deep neural networks. In Workshop on challenges in representation learning, ICML, volume 3, 2013.
Shin, I., Tsaí, Y.-H., Zhuang, B., Schulter, S., Liu, B., Garg, S., Kweon, I. S., and Yoon, K.-J. Mm-tta: multi-modal test-time adaptation for 3d semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16928-16937, 2022.
Li Y., Wang, N., Shi, J., Liu, J., and Hou, X. Revis- iting batch normalization for practical domain adapta- tion. In Sth International Conference on Learning Rep- resentations, ICLR 2017, Toulon, France, April 24-26, 2017, Workshop Track Proceedings. OpenReview.net, 2017. URL https: http//ppeeeeiieeeeeeeeeeeee net/ forum? id=Hk Jdk.
Sohn, K., Berthelot, D., Li, C.-L., Zhang. Z., Carlini, N. Cubuk, E. D., Kurakin, A., Zhang, H., and Raffel, C. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. In Advances in the 34th In- ternational Conference on Neural Information Processing Systems, NIPS' 20, 2020. ISBN 978 3781713829546
Song, G., Wang, S., Huang, Q. and Tian, Q. Similarity gaussian process latent variable model for multi-modal data analysis. In Proceedings of the IEEE international conference on computer vision, pp. 4050-4058, 2015.
Liang, J., He, R., and Tan, T. A comprehensive survey test-time adaptation under distribution shifts. Interna- tional Journal of Computer Vision, pp. 1-34, 2024.
Lim, H., Kim, B., Choo, J., and Choi, S. TTN: A domain- shift aware batch normalization in test-time adaptation. In The Eleventh International Conference on Learning Rep- resentations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. Uttht:ps: openreview. net/ forum? www.rrummmiii@ffem fuudmV1.Q.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need. In Advances in the 30rd International Conference Neural Information Processing Systems, pp. 0p.5998--000 2017. URL ht pwweeedicc...... neur meurips.cc/ppper/// 7/ahah/ (r5ee224544ddee)1ffoddol Aaaa html.
Niu, S., Wu, J., Zhang, Y., Chen, Y., Zheng, S., Zhao, P., and Tan, M. Efficient test-time model adaptation with- out forgetting. In International conference on machine learning, pp. 16888-16905. PMLR, 2022.
Wang, D., Shelhamer, E., Liu, S., Olshausen, B., and Darrell, T. Tent: Fully test-time adaptation by entropy minimiza- tion. arXiv preprint arXiv:2006. 10726, 2020.
Niu, S., WU, J., Zhang, Y., Wen, 7., Chen, Y., Zhao, P., and Tan, M. Towards stable test-time adaptation in dynamic wild world. arXiv preprint arXiv:2302.11240 2023.
Xiong, B., Yang, X., Song, Y., Wang, Y., and Xu, C. Modality-collaborati test-time adaptation for action recognition. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 26732-26741, June 2024.
Peng, X., Wei, Y., Deng, A., Wang, D., and Hu, D. Balanced multimodal learning via on-the-fly gradient modulation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8238-8247, 2022.
Xu, P., Zhu, X., and Clifton, D. A. Multimodal learning with transformers: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(10):12113-1213 2023.
Rosenstein, M. T., Marx, Z., Kaelbling, L. P., and Dietterich, T. G. To transfer or not to transfer. In Neural Informa- tion Processing Systems, 2005. URL https::////. / /api. semanticschole www.aaarrrrrrooorreeeeeee otg//oorpsID::97777
Yang, M., Li, Y., Zhang. C., Hu, P., and Peng. X. Test- time adaption against multi- reliability bias. In The Twelfth International Conference on Learning Represen- tations, 2024.
Sehwag, V., Bhagoji, A. N., Song, L., Sitawarin, C., Cul- lina, D., Chiang, M., and Mittal, P. Analyzing the robustness of open-world machine learning. In Pro- ceedings of the 12th ACM Workshop on Artificial In- telligence and Security, pp. 105-116, 2019. doi: 10.
Yasarla, R., Valanarasu, J. M. J., Sindagi. V., and Patel, V. M. Self-supervised denoising transformer with gaussian pro- cess. In Proceedings of the IEEE/CVF Winter Confer- ence on Applications of Computer Vision, pp. 1474-1484, 2024.
1145/3338501.3357777 URL https: /ttes:://doo.ooo /tt.i.og//////oo///////o///o 10.1145/333888 33077377013333222.... 3357372.
Shi, H., Xu, Z., Wang, H., Qin, W., Wang, W., Wang, Y., Wang, Z., Ebrahimi, S., and Wang, H. Continual learning of large language models: A comprehensive survey, arXiv preprint ar 2404 16789, 2024.
Zhou, Z., Guo, L.-Z., Jia, L.-H., Zhang, D., and Li, Y.- F. Ods: Test-time adaptation in the presence of open- world data shift. In International Conference on Machine Learning, pp. 42574 42588. PMLR, 2023.
10

---

Test-Time Selective Adaptation for Uni-Modal Shift in Multi-Modal Data
Zhou, Z.-H. Open-environment machine learning. National Science Review, (8)::wac113,, 2022.

---

Test-Time Selective Adaptation for Uni-Modal Shift in Multi-Modal Data
A. Theoretical Analysis
A.1. Analysis of Cross-Modal Attention under Uni-Modal Distribution Shift with Zero-Mean Additive Noise
The self-attention mechanism lies at the core of multi-modal fusion. This mechanism is fundamental as it allows the model to selectively focus on different regions within and across modalities. It aggregates intra-modal and inter-modal token representations with distinct weights, following the formula:
((2))(=)²) V(z), vdx
Attention(Q(2)) ((=), softmax
(12)
where query (.), key (-)) and value (-) are the linear transform on the tokens' representation dy is the scale factor. The attention logit, which determines the relative importance of token pairs in the self-attention mechanism, is defined as the inner product between the query and key matrices prior to the softmax operation. Its value varies depending on the representations of a pair of tokens. Formally, we define the attention logit in Definition 3.1.
Definition A.1 (Attention Logit). The attention logit (AL) is defined as the inner product between query and key matrices:
AL(z,, ((,)) = (wowk")
(13)
where 2, F and 2j R are pair of embedding of tokens involved in the self-attention calculation, and the linear projection matrices W W e FxF are considered to be constant.
However, the presence of distribution shift in any modality can have adverse effects on this self-attention mechanism. Without loss of generality, we analyze the case of a shifted audio modality. Under the uni-modal distribution shift, tokens from the shifted modality are corrupted. We denote the corrupted tokens as =((²) and 2((0) and the tokens from the unshifted modality as x/
Following the work of distribution shift with additive noise (Kim et al., 2020), assume that, after the distribution shift, zero-mean noise is added to the tokens of the input modality (Song et al., 2015; Huang et al., 2022; Yasarla et al., 2024), Specifically,
$ = z + where E[s]
0 x
(14)
The noise is independent of the distribution of the clean token z and is independent across different tokens.
(t) Let =/_ and k represent tokens from different modalities, where 2(a) is corrupted to z((²) by E;- We can show that the expectation of the attention logit remains unchanged after the corruption:
E Q( (<>("))" Q(2))) (0) (< (1))7]
(15)
= -B( +e:)WRWK (=(e))T - -( w/ (=l)))
(16)
E|swoIKK (= (
(17)
(Els)),[W&& ( (2)
(18)
0
(19)
Let 2(() and zf0) represent tokens from the same modality. The expectation of the attention logit also remains unchanged.
E Q (0) K Q(= -(E)))
(20)
= E [|( +=)) (z) +E²)" -2-"- "" (z)0)r] E fe (2)0)) +2 +2 wo of tow&Wkk of]
(21)
(22)
(T] + [B
+ B[W" ss])
(23)
0
(24)
12

---

Test-Time Selective Adaptation for Uni-Modal Shift in Multi-Modal Data
Since the expectation of the attention logit of the noisy token is the same that of the clean token, we then turn our attention to the variance.
The variance of the difference between the attention logit of tokens from different modalities:
D D k( Q(2 )
(25)
= =D(aW )7
(26)
= E[ (E.W (x (=)))T²² )} ( 2
(27)
| (e:Wo R² (z (r) 2 ill
(28)
= BlE:
(1)) " w.e (x 1T11
(29)
2 maz
B (
(30)
(31)
max
B B(tr
(32)
maz
B B(tr(sTTe(<())
(33)
maz
0 maz
1r (r(TssTe:
(34)
< maz
2
1r Lr(BleTe.)
(35)
(36)
||(ecc | (=])
(37)
Bl
18 (2())
(38)
maz
2 maz 111
(39)
The combination of linear transformation wºW*" can be decomposed by singular value decomposition into UEVT, where maz >> 0 represents the largest singular value of Σ (excluding the trivial case of W&w is all-zero matrix). The inequality ||aU'VT [|2 <= omaz ||l]| is used to obtain the upper-bound in Eq. 30. efTe; denotes positive semidefinite outer product and tr(-) is the trace. The inequality 0 I tr(AB) I tr(A)tr(B B) for positive semidefinite matrices is used to deduce Eq. 35. We denote Eq. 39 as sup(D) Q( ( K(z())) K ("))) - (((]())(((
Similarly, for the variance of the attention logit bias between tokens from the same modality:
D Q ) Q(2("))) (
(40)
= =D(EWweIK' (z) + + w of +eWW9wKK }
(41)
= =D(aweepx (z)a))r] + DD (0) wo eg + Dee, o7]
(42)
+ 2Coor @Coo[s,W/91 wW R" (2)
WOWKT
5/ + 2Coov (s, w (= (2("))T, ww sf)
(43)
+ +Coov
E, G (
(44)
D DEeWowKK
(2) + WK
+ 1D e W gf]
Eg
(45)
= T (0) ()) + E w k eg)
B[(,. eg)
(46)
S 5
B Illa |B|z]] (z) (z) (0) + 2 |B
|l
|E (2(")) - [E[e 1ll2
|Bie,e
(47)
2
|[e,e] ||l2 ||[2)|) (2))) ( + 2 |f(e,
Illa
||||||| (z. (={") ( + x² 2 |F(e,e
|[E(e,
(48)
13

---

Test-Time Selective Adaptation for Uni-Modal Shift in Multi-Modal Data
B[z] (0) ) + maz Bleref) || (=) ) a man Rlese Eleys))
mas
(49)
- [ E|ll (a) |2 + 2 man Bl =||l2 [2 maa 11 Bill,,
E|
(50)
Denote Eq. 50 as sup
1D
@ (0) | (z)
(2) ( { 77)
We have
sup (D (2(4 = oma 12 | + E|| 1 + B1|l, 12 || 1
Q(2 (1) K(z} ) )r) dus (D ( -((==) ){(2)
(51)
(52)
We then state the following proposition:
Proposition A.2. Under the additive shift assumption (Kim et al., 2020):
sup(Ii( A -2())) - , -= < (up
00)
AT(E! ,2(00) - (T
(0) ,0,,)
(a)
change crcoossmmdall AL after shift
change of intro shited modal after shift
(53)
when E||l 121 < = (1)
EEI + Bill,
where sup(-) is the least upper bound. Ej is the noise related to token 2.00 are the expected value of the squared norm of the token representations from the different modality and noise, respectively.
Bill2
(0) 18) E11 13) E112,
Eille, [8]]
In practical terms, the expected value of the squared norm of the token representations gives us a measure the average "energy" or magnitude of the vector across its possible realizations.
A.2. Expectation Analysis of Cross-Modal Attention under Uni-Modal Disdistribution Shift with Non-Zero-Mean Additive Noise
In this section, we further analysis the case with non-zero-mean noise. The expectation of the attention logit between uncorrupted tokens as follow.
[E[ (a) W B B[2, ( (ef ,00))) El (a) W RT (B((z) )) )\²
(54)
(2
(55)
The matrix (o)) (())) denotes the covariance between z! w and 2) w Similarly, we have additive noise as:
2/(0 =/²0
(a) (0) + E;; 2/ = 2/ +
(56)
where the noise & is independent of tokens' feature and follows an independent and ddentically-distribee (i.i.d.) distribution across different tokens. Unlike Sec. A.I, its mean zero. The expectation of the attention logit between the clean token and corrupted token is as follows:
[r]
e 0(2 + (())))))))))
(57)
= Eiz 9 K" (")= + (. Ele.
(58)
The expectation of the attention logit between corrupted tokens is as follows:
E E[Q(E!")
(59)
=E[Q(= + +a))(())) +
(60)
= ((El)"""""")")"""""""""""""""""" (=) + tx( + (E(E,)))
(61)
= (B) W (
(62)
14

---

Test-Time Selective Adaptation for Uni-Modal Shift in Multi-Modal Data
We consider the change in attention logits after corruption, egg. Q(E()) ( Q(2, ) K(=
and
Q( (a) (( Q(2 ((2( {(()))) The expectation of ((2 -( yT is:
B {[Q(2)")
Q(2 ) [ = [[ (a) + {()((=) [ - (0) )(( [7
(63)
Ble 1))
(64)
The expectation of (( )(())))))))))) Q(( ((() is:
E { K( (T) Q(=) ((()) (z)
(65)
(0) =,,)²] E[ (0) [((z) [0 [[ (a) ((2)
(66)
"l RT (E(e,)) + B K" (Elz)"))² Ble Km (E(e,))*
(67)
k The combination of linear transformation W Wy* can be decomposed by singular value decomposition into UEVT, where mas > 0 represents the largest singular value of Σ (excluding the trivial case of W x" is all-zero matrix). Based on the Cauchy-Schwand inequality, i.e., Ka, bi] S ||a||/||/|l and the properties of singular values, i.e., ||aU/V/ ||2 <= T [aar||a]] ||||2 we can calculate the least upper bound of the absolute values of the two expectations mentioned above.
E ) Q(2, K(( )² = ||B| W K
(B(=)"))
(68)
[|B|:/ W K"
112 ||E (M) 11l2
(69)
maar [E(e]]l2
||
(70)
Similarly,
E ( )
(71)
= E|e;| (B(2)"))) + E[z (d) w R (Bley))* + Ele,] R (Ble))*
(72)
o maz El
|
(0) 11 + o man || lll ||E|e]lla + man [[exlll2 ||E|,
(73)
Proposition A.3. According to Eq. 70 and Eq. 73, we obtain the relationship of least upper bound between the absolute expected attention logit difference of toooaalit-t---usshiffeed-- Eq. 74.
as change of shira-- AI. after shift
and mo-dalyy-----mmmhiiiiee
sup E A 22 AL(2(") (0) 2
(1)
sup( [[ALLE (2)) )
change of crozs- modal AL after shift
(74)
when 1[2((1112 ||E|2] (0))())]22
[E[2 112
|B[ey]lla
Remark A.4. The relationship in the above least upper bounds indicates that the attention logit between noisy tokens has larger extreme values. This reaches to the conclusion of damaged multi-modal fusion similar to A.2. We can also have the conclusion from 7q.73 that when the magnitude of noise ([B(e]]l2) is greater, the difference between the extreme values of corrupted intra-modal attention logit and cross-modal attention logit will also be larger. This explains the experimental phenomenon of the heavier the noise in TTA, the worse the performance (Yang et al., 2024).
B. Single Modality Experiments
Previous TTA methods such as Tent (Wang et al., 2020), ETA (Niu et al., 2022), SAR (Niu et al., 2023), and READ (Yang al., 2024) utilize techniques like batch re-normalization, entropy minimization, and sample selection. Blindly applying TTA methods each poddaliy-sppecfiii encoder can lead to redundancy even harm when distribution shift is present, resulting in overfitting in unshifted modalities that the original model already generalizes to. This challenge renders previous test-time adaptation methods inadequate, particularly those that apply adaptation without considering the disparities between modalities. As shown in Fig. 6, to observe the performance gap in individual modality-specific modules, we perform experiments on single modality datasets with no shift, and discovey the negative transfer (Rosenstein et al., 2005) phenomenon. This mimics the failure of TTA when they operate on modality-speciffe encoder for unshifted modality.
15

---

Test-Time Selective Adaptation for Uni-Modal Shift in Multi-Modal Data
(a) Unshifted CIFAR-10
(b) Unshifted ImageNet
25.0
22.5
Error rate
Error rate
20.0
2-
17.5
Source
Tent
ETA
SAR
Source
Tent
ETA
SAR
Method
Method
Figure 6. The special "negative transfer" phenomenon on the unshifted modality. The introduction of adaptation techniques on the unshifted data results in performance degeration (Source model vs. Tent, ETA, and SAR).
C. Partial Adaptation Trends
To investigate how the model adapts to synthetic data shifts specific modalities, we design experiments where we explicitly control shifts in either the video or audio data in Table 6. When synthetic shifts are introduced to the audio data during inference (e.g., Gaussian or traffic noise on audio), model prioritizes audio adaptation in 62-69% of cases, relying less on video adaptation (31-38%). Conversely, under video shifts (e.g.. Gaussian shot noise on video), the model adapts to video in 53-56% of cases, minimizing reliance audio adaptation (44 46%).. The results are:
Table 6. Audio and Video Shift Percentages
Audio Shift
Percentage
Video Shift
Percentage
Gaussian Noise
62%
Gaussian Noise
53%
Traffic
69%
Shot Noise
56%
Notably, not all samples with shifted modality follow the expected route. For example, even with video shifts, 44-47% of predictions still rely on the adapted audio data. We suspect this could be attributed to two reasons: 1) Modality imbalance, where certain modalities exert a greater influence on predictions in multi-modal tasks, makes the model tend to learn from the dominant modalities. 2) Convergence dynamics of the adapter and router. They may not receive sufficient adaptation on one iteration over the test set. It's important note that making predictions with partially adapted model a characteristic of test-time adaptation. This observation highlights potential area for further research.
D. Adaptation Details
D.1. Hyper-parameter:
We mainly have the following hyper-parametedd The coefficient and threshold of self-training loss, the softmax temperature, for the shift on one modality on each dataset we keep the temperature
the batch size. We use one set of to 0.001 and loss coefficient as 0.5 across all experiments). For Kinetics50-C with video shift, the threshold as 0.9, the batch size 16. For Kinetics50-C with audio shift, the threshold as 0.9, the batch size 64. For VGGSound-C with video shift, the threshold as 0.8, the batch size 1 128. For VGGSound-C with audio shift, the threshold 0.8, the batch size 64. During test-time, our model is updated using the Adam optimizer. We implement the network on GeForce RTX(TM) 3090 GPU and Intel(R) Core(TM) 19-10900K CPU @ 3. 70GHz. For the software information and other experimental settings, please refer to our code https://gituub.comccheee 1aeaaammc99999 1996699i- Distrl--DDttitttoon
D.2. Balance Loss
The class balance loss is caculated as:
(pt).
Poum
(75)
16

---

Test-Time Selective Adaptation for Uni-Modal Shift in Multi-Modal Data
where Pt corresponds the i-th sample. The balance loss, Loat is then defined as its negative entropy:
Lbal entropy (Psum).
(76)
This loss serves to regularize the distribution of the model's confidence across different classes, promoting more even distribution of probabilities.
E. Sensitivity Test
We avoided heavily tuning the loss coefficient and confidence threshold as it simply follows the self self-training loss.
(b)
(b)
54
Accuracy
52
53
50
Gaussian Noise
48
Shot Noise
52
0.85 0.95
0.5
0.85
Confident threshold
Confident threshold
Figure 7. Sensitivity test of the self-training loss Kineties50-C and VGGSound-C with video shifts.
The plots in Fig. which vary the confidence threshold for the self-training loss on two dataset. We find that the model can achieve competitive performance with only the balance loss on the Kinetics50-C dataset. However, on the VGGSound-C dataset, removing the self-training loss results in about 2% drops. We also observe consistent performance for varying loss coefficient as shown in Fig. 8.
53.5
Gaussian Noise
Shot Noise
Accuracy
52.5
52.0
51.5
0.8 Loss coefficinnt
Figure 8. Sensitivity test of the on self-training loss on Kinetics50-C with video shifts.
17


Reviews:
[{"review_id": "cQwv5gZ4a0", "reviewer": "Reviewer_hbmC", "summary": "The paper tackles the problem of multi-modal adaptation—the existing method of test time Domain adaptation struggles to adapt to other modalities. The authors propose to highlight this phenomenon and tackle this challenge with a \"router\" enabling the selection of the adaptation if needed. The selective adaptation method is tested over two datasets and compared to 4 other SOTA methods.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to All Reviewers, AC and SAC\nThanks for all Reviewers' valuable suggestions and the efforts of AC and SAC. \n\nReviewer dkfb, UDgQ and 5Yv6 all acknowledge the studied problem (we quote: \"_novel and practically relevant_\", \"_interesting real-world problem_\" and \"_challenging and practical scenario_\"). \nReviewer hbmC, dkfb, and UDgQ all give positive comments on our methodology (we quote: \"_excellent idea_\", \"_effectively mitigates negative transfer_\" and \"_promising solution_\"). \nFurthermore, our theoretical and empirical results are appreciated by all Reviewers (we quote: \"_sufficient theoretical analysis_\", \"_mathematically sound_\", \"_trustworthy_\", “_strong empirical validation_” and “_extensive experiments_”).\nLastly, all Reviewers find our paper well presented. \nWe'll integrate the Reviewers' suggestions into our revision. Next, we respond to all of the questions.\n\n# Response to Reviewer hbmC\n\n## W1 [Clarity of Figure 3]\nWe appreciate this constructive feedback. To enhance clarity, we have expanded the figure caption to explicitly describe the router’s role and the architecture’s flow. The new caption reads:\n```\nFig. 3: The architecture of our model. The original multi-modal data is processed through modality-specific encoders to extract feature representations. A router then dynamically decides whether to adapt each modality. For instance, in the case of video corruption, the router prioritizes video adaptation (red path) while preserving audio features (blue path). Finally, the extracted features are fused and passed to the prediction head.\n```\n\n## W2 & Q1 [Study of router's adaptation weights]\nWe thank the reviewer for raising this insightful question. Indeed, selective adaptation is a critical aspect of our framework, and we have conducted additional analyses to address this point. To investigate how the model adapts to synthetic data shifts in specific modalities, we design experiments where we explicitly control shifts in either the video or audio data. Our key findings are as follows:\n### Modality-Specific Adaptation Routes:\nWhen synthetic shifts are introduced to the audio data during inference (e.g., Gaussian or traffic noise on audio), the model prioritizes audio adaptation in  62–69% of cases, relying less on video adaptation (31–38%). Conversely, under video shifts (e.g., Gaussian or shot noise on video), the model adapts to video in 53–56% of cases, minimizing reliance on audio adaptation (44–46%). The results are:\n| Audio shift | Percentage |\n|-|-|\n| Gaussian Noise  | 62%  |\n| Traffic | 69%  |\n\n| Video shift | Percentage |\n|-|-|\n| Gaussian Noise  | 53%  |\n| Shot Noise  | 56%  |\n\n### Partial Adaptation Trends:\n\nNotably, not all samples with a shifted modality follow the expected route. For example, even with video shifts, 44-47% of predictions still rely on the adapted audio data. We suspect this could be attributed to two reasons: 1) Modality imbalance, where certain modalities exert a greater influence on predictions in multi-modal tasks, makes the model tend to learn from the dominant modalities. 2) Convergence dynamics of the adapter and router. They may not receive sufficient adaptation on one iteration over the test set. It's important to note that making predictions with a partially adapted model is a characteristic of test-time adaptation. This observation highlights a potential area for further research.\n\nIn conclusion, these results highlight the model’s ability to dynamically prioritize modalities based on their reliability under distribution shifts. We will include these findings and detailed visualizations in the revised manuscript, along with a discussion of potential explanations. \n\n### Suggestion [Term \"router\"]\nWe acknowledge that the \"router\" could cause confusion and the adaptation selector could be more straightforward. We’ll make the change accordingly in the revision.\n\nThank the reviewer again for prompting the analysis. We believe it strengthens the paper’s contribution by demonstrating the adaptability and interpretability of our framework.", "claims_and_evidence": "Plots and good experiments support the claims of the paper.", "methods_and_evaluation_criteria": "The method uses some existing methods (i.e., losses) but adds the selective adaptation that makes sense for the problem of heterogeneous domain adaptation. The method is tested over two datasets of multi-modalities.", "theoretical_claims": "I didn't have time to check the proof thoroughly.", "experimental_designs_or_analyses": "The experiments are appropriately designed and show good performance and efficiency in the proposed method.", "relation_to_broader_scientific_literature": "The paper is well related to broader literature.", "essential_references_not_discussed": "Seems good.", "other_strengths_and_weaknesses": "Strengths:\n- The paper is easy to follow.\n- The proposition of having a selective adaptation is an excellent idea.\n- The experiments are well done and seem trustworthy.\n\nWeaknesses:\n- Figure 3 is complex to read at the beginning of the paper. It is not clear what the role of the router at the beginning. It can be nice to have a more detailed figure caption and marble put a color legend or something like that.\n- Study of router's adaptation weights (cf questions)", "other_comments_or_suggestions": "In my opinion, the term \"router\" is misleading and is only used at the beginning of the paper. Maybe using a more transparent term like \"Selective Adaptation\" or something like that can be better.", "questions_for_authors": "- You do not study the selective adaptation in your experiments. In the ablation study, you show that the selection is crucial. I would like to know whether the proportion of data is getting adapted or not. Maybe with the visualization of the data, we can understand which type of data needs to be adapted or not.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Hq7cOSxkEX", "reviewer": "Reviewer_dkfb", "summary": "This paper proposes a novel approach to handling multi-modal test-time adaptation when only one modality undergoes distribution shift. The authors introduce the concept of uni-modal distribution shift, highlighting its adverse effects on multi-modal fusion and the potential for negative transfer. To address this issue, the paper proposes a selective adaptation framework that integrates modality-specific adapters with a routing mechanism that dynamically determines which modality requires adaptation. The effectiveness of the proposed method is validated through extensive empirical evaluations on datasets exhibiting uni-modal shifts, demonstrating superior performance compared to existing TTA methods.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer dkfb\n\nThanks for the reviewer’s positive comments and constructive suggestions! We have carefully considered each point and revised our work accordingly. Below is our detailed response:\n\n## W1 [Theoretical justification router’s effectiveness]: \nWe agree that the theoretical underpinning of our router mechanism could be further developed. Our current understanding is in line with modality-specific attention mechanisms: The router (followed by cross-attention fusion) essentially learns to enhance attention computation conditioned on shift detection statistics (Eq. 8). Besides, we notice that Li et al. [1] give a theoretical study on this direction very recently, showing how the MoE-like structure help in multi-tasks or multi-domain scenario.  Drawing on the ideas presented in the article, we anticipate that theories related to 'shift-specific MoE' could serve as a starting point. \n\nWhile time constraints prevent full theoretical development in this submission, we would welcome the reviewer’s suggestions on specific theoretical directions to prioritize in our revision or future work.\n\n[1] Hongbo Li, et al. \"Theory on Mixture-of-Experts in Continual Learning.\" The Thirteenth ICLR. 2025\n\n## Suggestion regarding literature review & W2 [Long-term adaptation]:\nThanks for bringing the related works to our attention. \nIn terms of the Adabn, the reviewer might overlook our discussion in the related work where we discussed Adabn: \n```\nThe Adabn (Li et al., 2017) recomputes the batch statistics for every test batch ....\n```\n\nAs for the long-term adaptation stability, while our primary contribution remains in multi-modal shift rather than continual adaptation, we fully agree this is critical for real-world deployment. We now explicitly position our work as complementary to lifelong adaptation research: We have added a limitation section at the last of our manuscript as follows with the discussion regarding the lifelong/continual shift over time mentioned in the ref [A] along with our setting’s limitation of encounter more complex shifts within multiple modalities:\n\n```\nSection 6. Limitations\nWhile our work provides a novel view for the unique uni-modal distribution shifts in multi-modal test-time adaptation, it is crucial to acknowledge several limitations. \n\nFirst, our current formulation primarily focuses on scenarios where a distribution shift occurs in only one modality at test time. Although this setting aligns with many practical applications, real-world environments often involve more complex combinations of distribution shifts across modalities. For instance, cases where two modalities experience concurrent shifts while the third remains unchanged, or shifts with varying magnitudes, pose significant challenges. Our theoretical and empirical analyses reveal that existing methods, including our proposed approach, struggle to disentangle conflicting signals from multiple shifted modalities, leading to suboptimal fusion and adaptation performance. This limitation underscores the need for more sophisticated mechanisms to diagnose and disentangle multi-modal shifts dynamically.\n\nBesides, our method operates under the assumption of single mini-batch test-time adaptation, where adaptation occurs incrementally on small, temporally coherent batches. While this setup is practical for scenarios with transient shifts, it does not account for long-term or continual distribution shifts that evolve over extended periods [1-2]. For example, in real-world deployments, modalities may drift gradually or exhibit recurring shifts, requiring adaptation strategies that balance plasticity (to learn new patterns) with stability (to retain prior knowledge). Our method does not explicitly address catastrophic forgetting or the accumulation of adaptation errors over time, which remain critical open challenges.\n\nThese limitations, however, highlight promising directions for future research. The failure of existing methods in multi-shift scenarios motivates the development of adaptive routing mechanisms that can scale to higher-order modality combinations and dynamically prioritize shifts based on their severity. Similarly, extending our selective adaptation idea to continual or long-term settings could involve memory-augmented architectures or meta-adaptation strategies to stabilize performance over time. We hope our work inspires the community to explore these frontiers, ultimately advancing the robustness of multi-modal systems in increasingly complex real-world environments.\n\n[1] Wang, Qin, et al. \"Continual test-time domain adaptation.\" Proceedings of the IEEE/CVF Conference on CVPR. 2022.\n[2] Brahma, Dhanajit, and Piyush Rai. \"A probabilistic framework for lifelong test-time adaptation.\" Proceedings of the IEEE/CVF Conference on CVPR. 2023.\n```\n\nWe appreciate the reviewer's suggestions, which have strengthened our paper's literature review and clarified its limitations.", "claims_and_evidence": "1. Uni-modal distribution shift disrupts multi-modal fusion. Supported by both theoretical analysis and empirical results, demonstrating performance degradation when conventional TTA methods are applied indiscriminately across modalities.\n2. Selective adaptation enhances test-time robustness. Experimental findings indicate that the proposed router-based approach mitigates negative transfer and improves adaptation efficacy.\n3. The proposed method surpasses state-of-the-art (SOTA) TTA techniques. Performance evaluations on Kinetics50-C and VGGSound-C show consistent gains over Tent, ETA, SAR, MM-TTA, and READ.\n4. Selective adaptation is computationally efficient. The paper presents comparative analyses of computational cost, showing that the approach maintains efficiency while improving accuracy.", "methods_and_evaluation_criteria": "The study utilizes the Kinetics50-C and VGGSound-C datasets, incorporating 21 types of uni-modal distribution shifts. Evaluation metrics include classification accuracy across different corruption types and computational overhead analysis. The experimental setup is well-aligned with real-world scenarios, as it effectively models cases where only one modality undergoes degradation.", "theoretical_claims": "This paper provides a theoretical examination of the impact of uni-modal distribution shifts on self-attention-based multi-modal fusion. The derivations are mathematically sound, illustrating how attention weight distributions are disrupted under such shifts.", "experimental_designs_or_analyses": "The experimental methodology is well-structured, including (1) Comparative evaluations across multiple corruption types (e.g., Gaussian noise, motion blur, weather conditions). (2) Ablation studies to assess the contributions of key components, such as the router, Gumbel-softmax selection, and test-time inference schema. And (3) computational efficiency assessments, demonstrating the balance between accuracy improvements and resource consumption.", "relation_to_broader_scientific_literature": "This work builds upon prior advancements in TTA (Tent, ETA, SAR) by introducing a modality-selective adaptation approach. It also extends research in multi-modal fusion by addressing negative transfer through targeted adaptation. Compared to MM-TTA and READ, which apply adaptation broadly across modalities, this work is among the first to tackle selective adaptation for uni-modal shifts.", "essential_references_not_discussed": "The literature review is comprehensive, but the discussion could be expanded to include, (1) Adaptive batch normalization techniques (AdaBN, BN-statistics adaptation) as alternative strategies for domain adaptation. (2) Continual learning paradigms that manage selective forgetting, which may offer insights into selective adaptation mechanisms.", "other_strengths_and_weaknesses": "**Strengths:**\n\n1. Introduces uni-modal distribution shift, a novel and practically relevant problem.\n\n2. The router-adapter mechanism effectively mitigates negative transfer.\n\n3. Strong empirical validation demonstrating state-of-the-art performance.\n\n4. The paper is well-written and easy to follow.\n\n**Weaknesses:**\n\n1. The theoretical justification of the router’s effectiveness could be further developed.\n\n2. Long-term adaptation stability remains unexplored. [A]\n\n[A] A Probabilistic Framework for Lifelong Test-Time Adaptation.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "See weakness above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "20KUJ5ahwR", "reviewer": "Reviewer_UDgQ", "summary": "This paper addresses the challenge of uni-modal distribution shifts in multi-modal learning, where only one modality experiences distribution changes at test time. The authors propose a selective adaptation framework comprising modality-specific lightweight adapters and a learnable router to dynamically activate adaptation for the shifted modality. Theoretical analysis examines how uni-modal shifts disrupt cross-modal attention mechanisms, arguing that intra-modal attention logit variance increases more significantly than cross-modal variance under additive noise assumptions. Experiments on Kinetics50 and VGGSound datasets with synthetic corruptions (e.g., noise, blur) demonstrate improved accuracy over existing test-time adaptation methods.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer UDgQ\n\nThanks for the reviewer’s positive comments and constructive suggestions! We have carefully considered each point and revised our work accordingly. Below is our detailed response:\n\n## W1 [Novelty of method] \n\nWe emphasize that our design offers a straightforward solution to the novel challenge of uni-modal shift. While adapters and routing mechanisms are established concepts, our innovation lies in _their specific integration to tackle uni-modal distribution shifts—a critical challenge we analyze theoretically and validate empirically_. Our methodological contributions include: \n### Residual Adapter Design\nUnlike standard adapters, our architecture explicitly decouples shift-agnostic base features from shift-specific components through residual connections as in Eq. 8, enabling targeted adaptation without catastrophic forgetting.\n### Shift-Aware Router\nThe shift-aware router introduces a gating mechanism that dynamically selects adapters based on modality-specific shift detection, a crucial capability missing in prior routing approaches.\n\nThis integration uniquely addresses scenarios where only one modality shifts, a common real-world issue under-explored in prior work.\n\n## W2 & Q3 [Experiments on multi-modal datasets with different modality combinations]:\n\nFirst, we would like to bring to the reviewer’s attention that most current multi-modal TTA research focuses on Kinetics50 and VGGSound and with various multi-modal shifts. Nevertheless, we acknowledge the importance of more diverse multi-modal distribution shift benchmarks.\nTherefore, we further investigate a real-world distribution shift within multi-modal datasets by examining the transfer from CMU-MOSI to CMU-MOSEI, which includes audio, text, and video data for sentiment analysis. The CMU-MOSEI paper highlights factors such as varying face detection methods that can contribute to visual distribution shifts. Our experiments on this dataset aim to: 1). provide evidence that many real-world shifts resemble uni-modal shifts and 2). further validate the effectiveness of our method. For instance, some video clips may be more face-centered, indicating a video shift while other modalities remain largely unchanged. To support this claim, we calculate the Maximum Mean Discrepancy distance between each modality to measure their discrepancies:\n\n|Modality|MMD|\n|-|-|\n|audio|0.0042|\n|vision|0.1379|\n|text|0.0105|\n\nThe significant vision shift (14× larger than audio/text) mirrors real-world scenarios where single modalities shift.\nWe then simply apply our \"selective-adaptation\" architecture on a recent work CASP (due to the difference between tasks, we choose to swiftly adopt our core idea to the existing codebase to present the performance in a short rebuttal period). The following is our results:\n\n|Model|ACC ↑|F1 ↑|MAE ↓|\n|-|-|-|-|\n|GC (CVPR 22)|67.12|67.40|1.22|\n|RF (ICLR 24)|66.84|67.39|1.27|\n|CASP (AAAI 25)*|67.02|67.05| 1.34|\n|Ours|__67.74__|__67.73__|__0.85__|\n*reproduced using its official code\n\nThese results help demonstrate our method's effectiveness on broader ranges of modality combinations or more diverse multi-modal datasets.\n\n## W3 [Sensitivity of hyperparameter] \nWe thank the reviewer for this observation. We avoided heavily tuning the loss coefficient as it simply follows the self-training loss. Nevertheless, we do agree that a sensitivity analysis of α would provide technical insights. We will include this analysis in the revision to strengthen the empirical validation of our method. The results are as follows:\n\n|α|0.0|0.1|0.2|0.5|0.8|0.9|0.95|1.0|\n|-|-|-|-|-|-|-|-|-|\n|Accuracy|51.9|52.0|52.1|52.6|53.1|52.4|52.4|52.4|\n\nIn conclusion, we observe robust performance for α∈[0,1]:\n\n\n## Q1 [Model collapse]\n+ Probabilistic sampling (Eq. 6) in Gumbel-Softmax introduces randomness to prevent deterministic routing.\n+ In each of our experiments, only one modality is shifted at a time, so the 'model collapse', i.e., consistently selecting one modality is not necessarily detrimental.\n\n## Q2 [Shifts of multiple modalities]\nGood question! Although our setting aligns with many practical applications, real-world environments often involve more complex combinations of distribution shifts across modalities. This poses significant challenges and requires further research. For now:\n+ The selection of adapters is conducted in a soft way (Eq. 8), i.e., they may function simultaneously and allow partial contributions from multiple modalities.\n+ We have added a limitation section at the last of our manuscript with the discussion regarding our setting’s limitation of encountering more complex shifts within multiple modalities or continual shift environments. Please check our reply to Reviewer dkfb.\n\n## Suggestions [misuse of symbol]\nWe have corrected the issue. We thank your meticulous review! \n\nWe thanks the Reviewer's suggestions that help strengthen our paper’s demonstration of its applicability.", "claims_and_evidence": "The primary claims of the paper are that:\n1.Uni-modal distribution shifts degrade multi-modal fusion by amplifying intra-modal attention variance.\n2.Existing TTA methods suffer from negative transfer when applied to unshifted modalities.\n3.The proposed selective adaptation using modality-specific adapters and routers improves robustness to such shifts.\n\nThe authors conducted experiments on the Kinetics50 and VGGSound datasets, applying various uni-modal offsets such as noise and blur. The results show that existing TTA methods struggle with uni-modal offsets, but the improvements of the proposed selective adaptation methods are not always significant. Some baseline methods remain competitive in some cases, and the effectiveness of the method varies depending on the dataset and type of offset. The theoretical analysis outlines the impact of uni-modal offsets on multimodal fusion, but the practical implications remain somewhat abstract and lack a deep connection to real-world applications.", "methods_and_evaluation_criteria": "The paper proposes a selective adaptation framework that incorporates modality-specific lightweight adapters and a learnable router to determine which modality to adapt during test time. The router employs a Gumbel-softmax mechanism and the model is updated via self-training with pseudo-labels. Evaluation is conducted on standard multi-modal benchmarks (Kinetics50 and VGGSound) under uni-modal shifts such as noise and blur, with classification accuracy as the main metric. Although Experiment Q3 provides some analysis on computational efficiency by reporting parameter counts and runtime, the evaluation remains basic and does not thoroughly address scalability or potential trade-offs in more challenging scenarios.", "theoretical_claims": "This paper provides a sufficient theoretical analysis of how uni-modal transitions affect multimodal fusion, particularly self-attention mechanisms. However, the theoretical insights can be further extended to explain the generalizability of the approach to other types of transitions or scenarios beyond experimental testing.", "experimental_designs_or_analyses": "The experiments are conducted on two multi-modal datasets (Kinetics50 and VGGSound) with uni-modal corruptions (e.g., noise, blur). However, the evaluation is limited as it does not extend to other types of multi-modal datasets or compare on multiple multi-modal pre-trained models. Additionally, it lacks a sensitivity analysis of the key hyperparameter α (Eq. 11), significantly restricting insights into the method’s broader applicability.", "relation_to_broader_scientific_literature": "The paper positions its contribution within the broader context of test-time adaptation and multi-modal learning. It correctly highlights the limitations of existing methods in handling uni-modal adaptation for audio and video, but it would be more helpful if more comparisons or discussions were provided on models handling other modality data.", "essential_references_not_discussed": "To the best of my knowledge, the authors have cited several key works in TTA and multi-modal learning", "other_strengths_and_weaknesses": "Strengths:\n1.This paper presents an interesting real-world problem, uni-modal shifts, which is relevant for multi-modal systems deployed in dynamic environments.\n2.The proposed method effectively combines modality-specific adapters and a learnable router, offering a promising solution to the problem in the context of audio and video modalities.\nWeaknesses:\n1.The novelty of the approach is somewhat limited, as adapters and routing are well-established concepts in the literature.\n2.The experiments are relatively limited, focusing mainly on the video and audio modalities using two datasets. This does not adequately demonstrate the method's applicability to a broader range of modality combinations or more diverse multi-modal datasets.\n3.The sensitivity analysis of the key hyperparameter α (Eq. 11) is notably missing, which limits understanding of how changes in this parameter affect the model's performance under different conditions.", "other_comments_or_suggestions": "I suggest exploring more diverse and complex datasets to test the method's scalability. Additionally, discussing potential limitations or failure cases of the method would provide a more balanced view.\nThe repeated use of the variable τ (Eq. 7 and 10) is confusing.", "questions_for_authors": "1.How does the router avoid mode collapse (e.g., always selecting one modality)?\n2.The method seems to be strictly designed for handling single-modal shifts; how would it address cases where multiple modalities experience shifts? \n3.Could you provide the performance of the method on multi-modal datasets with different modality combinations?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "F1eufmlVDa", "reviewer": "Reviewer_5Yv6", "summary": "This paper address uni-modal distribution shift in multi-modal data, where the distribution shift influences only one modality. They demonstrate that the presence of such shift impedes multi-modal fusion and leads to the negative transfer phenomenon in existing test-time adaptation techniques through theoretical and empirical analyses. Finally, a selective adaptation schema is proposed adapters and a “router” module. Experiments on two datasets highlight its superior performance.\n\n\n\n## update after rebuttal\n\nAlthough the author said they would update the conceptual contribution and adjust claims, the severe misclaim and limited performance improvement in some scenarios make me keep the original score.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer 5Yv6 \n\nThanks for the constructive suggestions! We have addressed each point with careful consideration and revised our work accordingly. Below is our detailed response:\n\n## W1 [Claims of setting] \nWe agree that [1] has explored multi-modal shifts more broadly. However, our work explicitly identifies __uni-modal shifts as a distinct and critical subclass of multi-modal shifts__, which necessitates tailored solutions. Our conceptual contributions lie in:\n* [1] studies shifts in any subset of modalities (e.g., partial or all), formulating to the form of $p_s(x)≠p_t(x)$ as shown in its \"Sec. Problem Formulation\". Whereas our work specifically models shifts occur in one modality $p_t(x^{(k)}) \\neq p_s(x^{(k)})$ and $p_t(x^{(i)}) = p_s(x^{(i)}), \\forall i \\neq k$. We also discussed why [1] is inefficient in our setting in the related work.\n* We further highlight ambiguities of terminology in existing literature, e.g., ref [1] and [2] both focus on \"multi-modal shift\". However, [2], different from [1], studies \"shifts happens in all modalities\".  To resolve this inconsistency, we introduce a precise definition of uni-modal shifts to differentiate them from broader multi-modal shifts. \n* Most importantly, we put our emphasis on \"__uni__\" is we believe its broad practical implication, as we stated in the introduction, many real-world corrupting factors impact only specific modalities.\n\nWe have reframed our conceptual contribution as __identifying the unique challenges of uni-modal shift via theoretical and empirical analysis__, while emphasizing its practical significance.\n\n## W2 [Novelty of methods] \nBuilding on W1, our analysis reveals that uni-modal shifts undermine cross-modal fusion and induce negative transfer. \nWhile adapters and routing mechanisms are established concepts, our innovation lies in their _specific integration to tackle the unique challenges of uni-modal distribution shift_:\n### Residual Adapter Design\nUnlike standard adapters, our architecture explicitly decouples shift-agnostic base features from shift-specific components through residual connections as in Eq. 8, enabling targeted adaptation without catastrophic forgetting.\n### Shift-Aware Router\nThe shift-aware router introduces a gating mechanism that dynamically selects adapters based on modality-specific shift detection, a crucial capability missing in prior routing approaches.\n\n## W3 [Experiments for autonomous driving tasks] \nWith due respect, we offer the following clarifications:\nWhile we use autonomous driving as an illustrative example, our experiments span diverse tasks (action recognition on Kinetics50, event recognition on VGGSound) with applications extending beyond autonomous driving.\n\nWe acknowledge the need for diverse multi-modal shift benchmarks (though most current multi-modal TTA research focuses on the two datasets, Kinetics50 and VGGSound, with various multi-modal shifts). However, reproducibility challenges with MM-TTA (unreleased code) prompted us to validate our approach on other datasets. We choose CMU-MOSI→CMU-MOSEI, real-world sentiment analysis datasets with possible modality-specific shifts: Both datasets include audio, text, and video data. The CMU-MOSEI paper highlights factors such as varying face detection methods that can contribute to visual distribution shifts. For instance, some clips may be more face-centered, indicating a video shift while other modalities remain largely unchanged. To support this claim, we calculate the Maximum Mean Discrepancy (MMD) between each modality to measure their discrepancies:\n\n|Modality|MMD|\n|-|-|\n|audio|0.0042|\n|vision|0.1379|\n|text|0.0105|\n\nThe significant vision shift (14× larger than audio/text) mirrors real-world scenarios where single modalities shift. Our experiments on this dataset aim to: 1) provide evidence that many real-world shifts resemble uni-modal shifts and 2) further validate the effectiveness of our method. We then simply apply our \"selective adaptation\" architecture on a recent work CASP (due to the difference between tasks, we choose to swiftly adopt our core idea to the existing codebase to present the performance in a short rebuttal period). The following are the results:\n\n|Model|ACC ↑|F1 ↑|MAE ↓|\n|-|-|-|-|\n|GC (CVPR 22)|67.12|67.40|1.22|\n|RF (ICLR 24)|66.84|67.39|1.27|\n|CASP (AAAI 25)*|67.02|67.05|1.34|\n|Ours|__67.74__|__67.73__|__0.85__|\n*reproduced using its official code\n\nThese results help demonstrate our method's effectiveness on more diverse tasks. \n\n## S1 [term router]\nWe apologize for the oversight. We have strengthened its description in Sec. 3.5 Selective Adaptation, where we detail its role in computing gating weights based on shift severity.\n\n## Q1\nPlease check our reply to Reviewer UDgQ W3 [Sensitivity of hyperparameter].\n\nWe thank the reviewer for the constructive feedbacks, which has strengthened our paper. We remain open to further revisions.", "claims_and_evidence": "The authors clain that \"In this research, we define a new practical scenario as uni-modal distribution shift, where the distribution shift influences only one modality, leaving the others unchanged.\" However, their setting is not new and exactly the same as the previous work [1].\n\n[1] Test-time Adaption against Multi-modal Reliability Bias (ICLR 2024)", "methods_and_evaluation_criteria": "The authors motivate the paper with the application of a self-driving car equipped with complementary camera and LiDAR sensors (Figure 1 and intro). However, they only experiment on action recognition datasets with video and audio. It is important to also validate the method on autonomous driving tasks as in MM-TTA [2].\n\n[2] Mm-tta: multi-modal test-time adaptation for 3d semantic segmentation (CVPR 2022)", "theoretical_claims": "Yes, Proposition 3.2. No issues found.", "experimental_designs_or_analyses": "The experiments lack autonomous driving tasks such as multimodal segmentation as in [2].\n\n[2] Mm-tta: multi-modal test-time adaptation for 3d semantic segmentation (CVPR 2022)", "relation_to_broader_scientific_literature": "The setting is not new and already discussed as in the previous work [1]. The idea of adapters and router is also not new and widely used in the literature [3][4]. \n\n[1] Test-time Adaption against Multi-modal Reliability Bias (ICLR 2024)\n\n[3] Clip-adapter: Better vision-language models with feature adapters (IJCV 2024)\n\n[4] Sparse Mixture-of-Experts are Domain Generalizable Learners (ICLR 2023)", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strengths\n1. The paper addresses the multimodal test-time adaptation problem, which is a challenging and practical scenario.\n2. The paper is well written and easy to follow.\n3. The paper provides extensive experiments, showing the effectiveness and versatility of the proposed method.\n\nMajor Weaknesses\n1. The claimed new practical scenario is not new and exactly the same as the previous work [1].\n2. The idea of adapters and router is also not novel and widely used in the literature [3][4]. \n3. The paper is motivated with the application of a self-driving car equipped with complementary camera and LiDAR sensors (Figure 1 and intro). However, they only experiment on action recognition datasets with video and audio. It is important to also validate the method on autonomous driving tasks as in MM-TTA [2].\n\n[1] Test-time Adaption against Multi-modal Reliability Bias (ICLR 2024)\n\n[2] Mm-tta: multi-modal test-time adaptation for 3d semantic segmentation (CVPR 2022)\n\n[3] Clip-adapter: Better vision-language models with feature adapters (IJCV 2024)\n\n[4] Sparse Mixture-of-Experts are Domain Generalizable Learners (ICLR 2023)", "other_comments_or_suggestions": "The \"router\" module is not clearly defined in Section 3. I assume it is equation 7 but the author never mentioned it.", "questions_for_authors": "How is the method sensitive to alpha in equation 11?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper addresses unimodal distribution shifts in multimodal data, proposing a selective adaptation framework with modality-specific adapters and a router module. Reviewers acknowledge its practical relevance, noting strong empirical performance on Kinetics50-C and VGGSound-C datasets, where it outperforms baselines like Tent and READ in scenarios with single-modality corruption. 

Theoretical analysis highlights how unimodal shifts disrupt cross-modal attention, and the router-adapter design effectively mitigates negative transfer. However, concerns persist: limited novelty is raised, as the scenario and adapter-router concept partially overlap with prior work (e.g., READ). Experiments are criticized for focusing on video/audio modalities instead of broader multi-modal tasks (e.g., autonomous driving), and the router’s theoretical justification is deemed underdeveloped. While authors address some gaps (e.g., sensitivity analysis, CMU-MOSI/MOSEI results), the distinction from READ remains unclear, and improvements in some cases are marginal.

The work’s focus on a critical real-world problem and empirical validation justifies its contribution.
Decision: Accept (poster)