Partition-Then-Adapt: Combating Prediction Bias for Reliable Multi-Modal Test-Time Adaptation
Authors: Guowei Wang
Fan Lyu
Changxing Ding
Venue: NeurIPS 2025 spotlight ()
Keywords: test-time adaptation
multi-modal domain shifts
Primary area: general_machine_learning
TL;DR: Existing TTA methods mainly focus on single-modal domain shifts and often produce suboptimal results under multi-modal domain shifts due to severe prediction bias. To address this, we propose a novel Partition-Then-Adapt (PTA) method.

Abstract: Existing test-time adaptation (TTA) methods primarily focus on scenarios involving domain shifts in a single modality. However, they often prove ineffective when multiple modalities simultaneously undergo domain shifts, as they struggle to identify and utilize reliable samples within testing batches amid severe prediction bias. To address this problem, we propose Partition-Then-Adapt (PTA), a novel approach combating prediction bias for TTA with multi-modal domain shifts. PTA comprises two key components: Partition and Debiased Reweighting (PDR) and multi-modal Attention-Guided Alignment (AGA). Specifically, PDR evaluates each sample’s predicted label frequency relative to the batch average, partitioning the batch into potential reliable and unreliable subsets. It then reweights each sample by jointly assessing its bias and confidence levels through a quantile-based approach. By applying weighted entropy loss, PTA simultaneously promotes learning from reliable subsets and discourages reliance on unreliable ones. Moreover, AGA regularizes PDR to focus on semantically meaningful multi-modal cues. Extensive experiments validate the effectiveness of PTA, surpassing state-of-the-art method by 6.1\% on Kinetics50-MC and 5.8\% on VGGSound-MC, respectively. Code of this paper is available at https://github.com/MPI-Lab/PTA.

Combating Prediction Bias for Reliable Multi-Modal Time Adaptation
Guowei Wang' Fan Lyu²
Changxing Ding
South China University of Technology
National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences eegw. weggwwang scut. edu. cn, fan. fan.1yu@cripac ia. ac. chxding@scut. edu, cn
Abstract
Existing test-time adaptation (TTA) methods primarily focus on scenarios involving domain shifts in a single modality. However, they often prove ineffective when multiple modalities simultaneously undergo domain shifts, as they struggle identify and utilize reliable samples within testing batches amid severe prediction bias. To address this problem, we propose Bartiton-ThhenAA approach combating prediction bias for TTA with multi-modal domain shifts. PTA comprises two key components: Partition and Debiased Reweighting (PDR) and multi-modal Intention-Guider Alignment (AGA). Specifically, PDR evaluates each sample's predicted label frequency relative to the batch average, partitioning the batch into potential reliable and unreliable subsets. It then reweights each sample by jointly assessing its bias and confidence levels through a quantile-base baase approach. By applying weighted entropy loss, PTA simultaneously promotes learning from reliable subsets and discourages reliance on unreliable ones. Moreover, AGA regularizes PDR to foeus on semantically meaningful multi-modal cues. Extensive experiments validate the effectiveness of PTA, surpassing state-of-the-art method by 6. % Kinetics50-MC and 5.8% VGGSound-MC, respectively. Code of this paper is available at httpss//gihubb.com//PPLLLLL
(PTA), a novel
Introduction
Pre-trained models tend encounter performance degradation when deployed due to source-target domain shifts. To mitigate this, TTA updates model parameters during inference using online data, making it essential for applications like visual understanding [3]] and embodied intelligence [G] [8] Recent studies have expanded TTA to multi-moda. scenarios (MM-TTA), such multi-modal action recognition and event classification (5]]))))) However, they primarily focus on single-modal domain shifts in multi-modal tasks, where only modality undergoes shift while others remain unaffected. In the real world, multi-modal domain shifts are more commonly encountered. For instance, variations in lighting and background noise degrade the performance of cameras and acoustic sensors used in autonomous driving systems 4 40). In this paper, we investigate the challenging problem of simultaneous domain shifts across multiple modalities MM-TTA scenarios.
Existing MM-TTA methods typically address single-modal domain shifts multi-modal tasks, but they fall short in handling multi-modal domain shifts. This is mainly due to difficulties in identifying and effectively leveraging reliable samples. From Fig. [ (a), we observe that as more modalities become contaminated, the pre-trained model gradually loses reliability in multi nulti-modal fusion, causing fused representations of different classes to overlap. Consequently, as shown in Fig. (b), this results in biased predictions. In the context of TTA, updating the pre-trained model with biased predictions
Conresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Audio:
Video: Accuracy:81.6%
Audio: Video:
Accuracy:67.7%
Cangayy
Accuracy44.4%%
Audio: Accuracy:9.35
(b)
Marmallaed Entropy
Clean
Comppedd
Clean 13nta
Singla-madal Douain Shifts
Multi-modal Docaain Shifts
Figure 1: (a) I-SNE 33 visualizations of fused representations under clean, single-modal, and multi-
modal domain shifts. (b) Proportions of predicted labels under the same domain shift conditions as (a).
(c) Comparison of true positive rates for test data grouped by normalized entropy bins (interval: 0.1).
All experiments are conducted on the Ki)))))SSO-C(MC) [43] with the pre-trained CAV-MAE 43
model. The audio and video noise types are "Crowd" and "Gaussian Noise", respectively.
rapidly accumulates errors, leading to collapsed trivial solutions 26 i.e., limiting predictions to narrow subset of classes. One popular way to mitigate this is perform model adaptation using only high-confidence samples [43 50, 9). However, this strategy frequently struggles to adapt to multi-modal domain shifts. As illustrated in Fig. (c).
observe notable drop in the true positive rate for high-confidence samples, indicating that many of the selected "reliable" samples are actually false positives. In this case, updating the pre-trained model with these "reliable" samples amplifies the risk of error accumulation. Therefore, relying exclusively on prediction confidence fails to provide a reliable basis for sample selection and reweighting during TTA under multi-modal domain shifts.
To address the above challenges, we propose a Partition- Then-Adape (PTA) method, which discovers and leverages reliable samples for TTA in the presence of multi-modal domain shifts. PTA consists of two key components, i.e., Partition and Debiased Reweighting (PDR) and multi-modal Attention- Guided Alignment (AGA). Specifically, given test samples under multi-modal domain shifts, PTA first quantifies the prediction bias level for each sample. This bias is calculated based on how frequently the sample's 's predicted label appears relative the average frequency of predicted labels within the batch. Samples with lower prediction bias are identified as potentially reliable, while those with higher bias are deemed potentially unreliable. It then employs quantile-based reweighting scheme in the multi-modal adaptation that jointly assesses prediction bias and confidence levels, assigning positive and negative weighting factors for reliable and unreliable samples. By separately applying weighted entropy minimization and maximization to each group, it simultaneously promotes learning from reliable samples and discourages reliance on unreliable ones. Since the entropy maximization for unreliable samples may drive the attention map toward an undesired uniform distribution, potentially causing the model to attend to non-discriminative or noisy signals, we further introduce multi-modal attention- guided alignment (AGA) method to mitigate this risk. Specifically, AGA uses the attention map distributions of reliable samples guide the update of those from unreliable samples through maximum mean discrepancy-bast unreliable samples remains aligned with the semantically meaningful cues from reliable samples, rather than being misled by undesired uniform distributions. The contributions of this work can be summarized as follows:
regularization. It ensures the attention maps distribution of
(1) We show that solely believing in prediction confidence may induce error accumulation under multi-mode domain shifts in TTA.
(2) We propose novel PTA method, which consists of Partition and Debiased Reweighting (PDR) and multi-modal Attention Guided Alignment (AGA). PDR offers reliable sample selection and reweighting scheme that combats prediction bias for MM-TTA. AGA further regularizes PDR to focus semantically meaningful multi-modal cues.

---

(3) PTA consistently outperforms all state-of of-toff-tth-aaa advantages under mulli-modda domain shifts.
the-art art methods on tasks, with pronounced
2 Related Work
Test-Time Adaptation. TTA 37 19] aims to adapt pre-trained model to the target domain using online data without ground-truth labels. Traditional works typically address four core challenges of
38 41], catastrophic forgetting [40], and miscalibrated batch normalization statistics 45] To mitigate noisy pscudo-labels and biased entropy, existing methods employ sample sclection/reweiihting 25 26 [6], pseudo-label refine-
i.e. noisy pscudo-labels [ 24 biased entropy [25 16
ment G and robust prototypes learning [5 48 39. For catastrophic forgetting, memory banks [40], 5 22
Fisher regularization [25] 33), and teacher-student frameworks [4]]
34 [8 commonly
adopted to achieve prediction consistency. Meanwhile, miscalibrated statistics are typically corrected
via source-target normalization statistics mixup 45 49] Although effective in single-modal set-
tings, their direct application to multi-modal scenarios often yields suboptimal results, as real-world multi-modal domain shifts involve more complex noise patterns (9].
Multi-Modal Test-Time Adaptation. Different from traditional TTA studies, which primarily handle single-modal data, MM-TTA addresses the challenge of domain shifts in multi-modal data. Pioneer works (29] 2 I attribute the challenges to intra- and cross-modal error accumulation, and conduct similar operations with traditional single-modal TTA methods, such as pseudo-labeling [37 20) and teacher- student guidance 440 5). A recent work [43) advocates that the key of MM-TTA is mitigating increasing information discrepancies, i.e., reliability bias, and proposes reliable fusion and robust adaptation as solution. Its followers introduce attention bootstrapping [50], mutual information sharing 9 to further boost the performance. However, they focus on addressing single-modal domain shifts, where only one modality undergoes domain shift while others remain intact. In this paper, we demonstrate the robustness of our method on both single-modal and more practical and challenging multi-modal domain shifts.
3 Method
3.1
Problem Formulation
We define the problem using the example of video and audio co-classification for clarity. Consider a multi-modal pre-trained model Me = (dv. F). where d and de are the encoders for video modality and audio modality a, respectively, and the block F is the fusion component with a prediction head. Let the output logits be p(x() = MM( for online mini-batch X, where (<x,,.)) (20 =.)).... (x,, ). For sample
X, the prediction can be During MM-TTA, the model updates the tunable parameters θ C θ with self- self-supervised loss functions, e.g., entropy loss.
X' = {a; y
computed by ((x) = arg max (softmax
Previous MM-TTA methods 43 10) focus single-modal domain shifts, where only one modality is corrupted, which may deviate from real-world scenarios. this paper. we focus on multi-modal domain shifts in MM-TTA, and propose novel Partition-Then-Aee (PTA) method. As shown in PTA consists of Partition and Debiased Reweighting (PDR) and multi-moda
Fig. 2 Alignment (AGA). PDR identifies reliable and unreliable candidates and reweights their contributions, and AGA regularizes PDR to focus on semantically meaningful multi-modal cues.
Ivention-Guided
3.2 Partition and Debiased Reweighting
Sample reweighting is a widely adopted strategy in traditional TTA, designed to balance the influence of reliable and unreliable samples. Building on this approach, previous MM-TTA methods select and reweight samples simply guided by low entropy [25] 43 9 high softmax confidence [50]. They perform well when only one modality undergoes domain shift among multiple modalities, allowing the pre-trained model to leverage reliable, unaffected modalities to preserve prior knowledge 43 thereby maintaining the reliability of selected samples.
However, as we demonstrate in the experimentation section (Tables I and the effectiveness. of existing reweighting strategies becomes less effective under multi-moue domain shifts compared to single-modal domain shifts. In such scenarios, corrupted modalities compromise the reliability of
3

---

Moutt-mmdal
Partition Debiased Rewcighting (Sec. 3.2)
Allgnment
Prediction Bias Level
x
@
Joint Assemeet
Los and
Bla
Vidro
Audio
Kig- (1)
Alg:
Alleziiu Maps
Eq.
+
Pacton Mlock
Ex
W
Wg
Mare
Atteriiun
Praduction Contidence Level
300
Numn Feod
Comptted
Video/Audoo Token
Reliabie
Unedlable
Turabii
*
Figure 2: Overview of PTA. PTA first partitions the online data into two subsets, and and jointly evaluates sample importance considering their prediction bias and confidence levels. It then adapts the pre- trained model by weighted entropy minimization and multi-modal iitention-guided alignment.
prior knowledge, degrade fused representations, and constrain predictions a narrow set of classes. Unlike single-moda modal settings, noise in single modality can propagate through the fusion process, disrupting the entire multi-modal system [9. As more modalities are affected, the collapse of prior knowledge leads to a sharp decline in the true positive rate of confident predictions and a more pronounced increase in false positives. Consequently, confidence or entropy becomes an unreliable indicator for identifying and reweighting samples under multi-modal domain shifts. In such cases, adapting the model further increase the risk of prediction bias. To address this challenge, we design Partion and Debiased Reweighting (PDR) strategy that effectively combats prediction bias under multi-modal domain shifts.
PTA first identifies reliable and unreliable samples, and respectively promote and restrain their contributions for better adaptation. Given a batch of multi-mode samples X, with corresponding predicted categories y. For x X, the frequency of predicted label can be expressed as (xx) B (ū: = 5). Here, 1-)) is the indicator function that equals 1 when the condition is true. For samples in the miiii-batch X, we define their bias level of prediction as:
( = 2 - Z(x) = { - 7 2 - 7(x2),
[(5n))
where 2 = ExEx ((x) is the mean frequency of the predicted labels, therefore representing the average bias level of all samples in the batch. Let the prediction confidence of X' be denoted as partition X' into two subsets: X for less
Based on Eq. ( biased predictions (Z 2 0) and X for more biased predictions (Z < 0), yielding corresponding pairs of bias levels and confidences: (z+, K+) and (2 IC-). Building upon this partitioning, introduce novel approach that leverages the sample-wist wise bias levels for achieving more reliable reweighting adaptation.
(C() max maxxsofttaxx((("")
To identify the reliability of test samples, we reweight those in X using their corresponding bias and confidence levels, 2 and IC+, via a quantile-based importance weighting scheme: Q(Z+) and Q(K+), where Q(-) denotes the quantile ranking operation inspired by [15] 32]. Specifically, we first sort all elements in Z and AC, then compute each element's quantile by averaging the positions of its first and last occurrences in the sorted sequence, normalized by the total number of elements. Z and KC+ serves to
The details can be found in Algorithm This quantile-basee operation normalize sample influence and suppress outliers by mapping raw scores to a relative scale. By jointly ranking bias and confidence, it highlights samples that are both confident and less biased, enabling more reliable and robust adaptation. For a test sample x E X. the reweighting indicator function can be formulated as,
I(x) {-s, (Q(Z)()) Q((C(())
E x
The reliable samples in X are those with less prediction bias and high confidence, and the unreliable ones in X' are those with large prediction bias. We set the weight of X to -s (s >= 0), to reject samples with obviously biased predictions during entropy minimization. Therefore, the weighted entropy loss can be formulated as,
H(X) I((X) p( P(x).
(3)

---

Algorithm Quantile Ranking
Algorithm 2 PTA
1: Input: Sequence T
1: Input: Trained model Me. Data batch
2: Output: Quantile sequence Q
2: Calculate the prediction logits: p(t'
3: Twinted <- sort(T).
Me'
4: for each E do
3: Reweighting each element in X using
5: Find first index j where Tiomedd =
Eq. (1. Eq. (2), and Algorithm[[]
6: Find last index & where Tsomed [k]
4: Calculate the weighted entropy loss using
7: Compute Qt + j+t 2/T 8: Append Qt to Q
Eq. (3)
5: Calculate the MMD-based loss via Eq.
9: end for
6: Calculate the overall loss using Eqq 5
10: Return Q
7: Update the tunable parameters é of Ma
In summary, PTA takes both bias and confidence levels into account. Consequently, in the presence of numerous false positives, the high bias levels help reduce the corresponding weights, thereby mitigating error propagation during MM-TTA.
3.3 Multi-modal Attention-Guided Alignment
In Eq. (3). we introduce an entropy-based objective that assigns positive weights to reliable samples x and negative weights to unreliable ones X'= aiming minimize the entropy of x while maximizing that of X' X X++ has robust prior knowledge compared to that of X' and we will verify in Section 4.3 This mechanism, however, introduces potential risks. In multi-modal learning, attention-based fusion [36] (23), especially self- and cross-modal attention, is widely used to integrate information from different modalities. Under the entropy maximization objective, the attention map A associated with X' may converge toward a uniform distribution during adaptation. This can cause the model to attend to -on-disrriminative or noisy signals, which in turn interferes with the reliability and effectiveness of A in the fusion process. To mitigate this issue, based on the partition above, we propose a multi- modal Eeention-Guiden Alignment (AGA), which ensures that attention remains aligned with semantically meaningful multi-moda cues. We explicitly enforces the model to learn semantically meaningful multi-modal cues and push it as guide for model adaptation via AGA. Specifically, we model the distribution of A+ and A by mapping them into high-dimensional Reproducing Kernel Hilbert Space (RKHS) with Gaussian kernel g. and compute the regularization based on maximum mean discrepancy [2 (MMD) as follows:
R(A* AA) = Em (g(m, m')] An n')] 2 ----AAAAAAA------ n)], (4)
where (m, m') are positive pair drawn from At, and (n,n') are the negative pair drawn from A The first two terms of Eq. 4 encourage similarity among positives (negatives), promoting consistency within each respective group. And the third term regularizes the model by guiding A with A Specifically, it aligns the distributions of A with those of A+ As result, to minimize Eq. the model has to focus on the semantically meaningful multi multi-modal cues from At, while suppressing the knowledge from A
3.4 Overall Optimization
Finally, we combine Eq. (3) and Eq. with coefficient factor À the overall loss function:
C(X) = H(xx))))))))))))))( + R(A+,A7
(5)
For the clarity of our design, we summarize PTA in Algorithm 2 When the multi-modal system receives online data X, it encodes the data of each modality as modality-specifie tokens, sends tokens to the fusion block, and outputs the corresponding logits. We use PTA to separate X' into x and X based on the distribution of predicted class, and then reweight their contribution using a quantile-based weighting approach. Moreover, we enforce the model learn semantically meaningful multi-n molti- cues using AGA.
5

---

Table I: Accuracy comparison with SOTAS Kinetics50-MG for modal domain shifts (severity
level 5 for all tables except Table 3 and 5
The best performances are highlighted.
Goues.
Impul.
Defoc.
Motion
Frost
Fog
Elastic
JPGG
Avg.
Source
12.92
13.05
$7.20
36 $6.94
41.70
13
20,47
55.29
18.28
37.77
31.76
Tox
6.72
7.03
6.68
27.01
29.00
34,45
17.50
22
8.30
53,42
9.95
36.02
28.71
29. 79.79
23.72
ETA
P61
12.90
13.90
13.00
38.83
59.3 99.36
47.42
43.71
32.50
53.13
19:90
57.13
18:07
44. 30
41.48
39.75
$3,02
MMTIA
20
6.56
9.23
3.15
32.17
34.06
42.57
40.35
24.06
28.02
11,62
55.53
12.88
40.93
35.
35.17
27:97
ABPEM
50
12:27
13.16
12:24
40.59
41.08
50.05
45.92
53.00
57.21
19.19
58.41
20,02
46.25
40.72
38. 05.55
33.91
SuMi [9]
12.38
13.51
12.75
51.21
36.99
46.11
42.25
29.30
56
19.10
55.76
17.92
41.90
37.78
36.55
31.44
READ
50.12
26.28
42.97
PIA
21.93
47.72
52.55
40.25
43.57
39,66
59.99
27.32
50.35
50.86
47.59
41.47
Table 2: Aceuracy comparison with SOTAs on VGGSound-Mr for multi-modal domain shifts.
Gouss.
Shot
Impul.
Defoc.
Glass
Motion
Zoom
Snow
Frost
Fog
Brit.
Coutr.
Elustic
Pixel
JPBG
Avg.
Source
4,67
4.77
4.65
9.27
9.04
11.75
12.85
9.03
11.78
9.30
15.82
12.82
10.05
11.87
9.56
0.74
0.95
1.16
1.03
0.79
1.09
1.30
1:01
ETA
25
6.72
6.86
13.34
14.49
17.15
18.22
13.30
16.48
13.64
23.30
7.77
19.92
15:45
18.18
14.10
MMTIA
2.41
2.79
2.44
4.15
3.63
5:24
5.80
3.63
3.75
2.14
8.25
2.14
4.72
4,76
6.46
4.14
ABPEM
50
4.59
5.75
4.47
12.62
13.70
16.49
15,82
12.72
14.62
14.25
20.05
7.62
16.51
12.12
15.38
12.43
SuMi (1)
4.68
4.76
4.60
9.24
9.22
11:47
12:35
9.10
11.40
8.69
15.16
5.97
12.55
9.88
11.12
9.34
READ
7.43
7:71
7.65
13:01
13.71
15.28
14.58
12.65
13.79
12.88
16.57
15.75
13.16
13.71
12.50
PTA
10,89
18.08
18.91
19,06
21,08
22.51
20.97
18.29
4 Experiment
4.1 Experimental Setting
Datasets. To comprehensively validate MM-TTA, we conduct experiments on benchmarks featuring both synthetic and real-world domain shifts. For symthetic shifts, we apply 15 types of corruptions [ to the video modality and types of corruptions 43 to the audio modality of Kinetics50 [4] and VGGSound A. following the protocol in 43 This results in total of 90 combinations for each benchmark, egg., Kinetics50-MC and VGGSound-MC. Each corruption type is applied at 5 severity
levels. For real-world domain shifts, we choose CMU-MOSI [46], CMU-MOSEI [47), and CH-
SIMS 44 for evaluation, each comprising three modalities (e.g., text, audio, and video). See Appendix A for more details.
Baselines, We compare our method with state-of art (SOTA) methods, including TENT 37
ETA 225, MMTTA [29], READ (43), SuMi [91 and ABPEM (50). Details are in Appendix C
Implementation details. For experiments on synthetic multi-modal domain shifts, we adopt the
pre-trained model from [43] based the CAV-MAE architecture [8) following 43 g 50 The learning rate and batch size are set to 2--4 and 32 for Kinetics-C, and le-4 and 64 for VGGSound-C. respectively. For the experiments
real-world domain shifts, provide pre-trained models for
CMU-MOSI [46), CMU-MOSEI [47), and CH-SIMS 44 following the training protocol [10]. The learning rate and batch size are
1e-3 and 24. Details are in Appendix [ By default, hyper- parameters are set as 0.5, À = 1. Following 4 50) we update query/key/value transformation matrices of the attention layer in the fusion block. We run all experiments with 5 random seeds on one NVIDIA 4090 GPU, and report the average accuracy.
4.2 Comparison with SOTAS
Robustness to synthetic multi-modal domain shifts. We present the results of the most challenging task, where online data from both audio and video modalities is corrupted with the highest noise severity (level 5), in Tables D and Each number denotes the average performance over six audio corruptions under a specific video corruption. As the rate of true positives experiences a significant drop, existing methods that employ confidence 4K] 50) or entropy 25 9 the criterion for sample reweighting demonstrate limited improvement under multi-modal domain shifts. In comparison, PTA outperforms all existing SOTA methods across all corruption combinations. Specifically, it achieves 9..1%% % and 8.73% performance gains compared to the pre-trained model on Kinetics50-MC and VGGSound-Me respectively. This demonstrates the effectiveness of PTA in jointly assesses the contribution of each data based on their prediction confidence and bias levels, and the importance of leveraging the semantically meaningful multi-me modal cues guide model adaptation.
Robustness to real-world multi-modal domain shifts. We summarize the empirical results on real-world multi-modal domain shifts in Table where "A "A" denotes the source domain to the target domain. We observe that existing methods 43 25 50 that depend on confidence or entropy as the criterion to select and reweight samples bring marginal improvements compared to the pre-trained model.

---

Table 3: Accuracy comparison with SOTA methods CMU-MOSI, CMU-MOSEL, and CH-SIMS. We denote CMU-MOSI, CMU-MOSEI, and CH-SIMS as MOSI, MOSEI, and SIMS for clarity.
MOSI-MOSEI
MOSI-SIMS
Avg.
MOSEI -MOSI
MOSEI-SIMS
Avg.
SIMS-MOSI
SIMS-MOSEI
Avg.
Souree
34,42
24,00
39.61
77.00
30.05
53.55
51.86
31.42
41.64
Tenr
37
51.15
23.50
57.31
76,72
30.15
53.44
52,04
29.11
40.58
ETA
25
55.04
25.31
57.33
77.66
31.05
54.34
52.32
32.67
42.45
25.80
40.76
ARPEM 10
56.72
23.80
40.30
76.83
31:25
54.04
$2.10
35.86
43.98
SaMi 0
56.37
24.40
40.39
78.11
30.62
54.57
$2.53
36.36
44,45
READ
4
55.63
24.00
39.82
76.97
29.80
53.39
52.10
35.32
43.71
PTA
55.34
Table 4: Comparisons on the key components of PTA. "Baseline + PTA leverages only X'+
Gauss.
Impul.
Defoc.
Motion
Zoom
Snow
Fog
Elastic
Pixll
JPEG
Avg.
Raveline
9.03
9.07
34.03
43.16
40.08
27.97
12.87
55.04
13.77
40.40
34.65
34.55 85.88
7X:11
+PIA
10.77
11.09
11.16
43.74
41.80
50.04
46.94
34.18
39.11
21.04
58.42
16.95
47.61
45.76
43.14
34.82
+PIA
20.65
21.65
20.66
15.85
44.63
51.34
48.92
40.11
43.43
39.23
58.71
27.07
49.77
49.68
46.92
40.57
+PTA+AGA
21.93
22.98
22.11
47.72
45,92
52.55
49.31
40.25
43,57
39.66
89.99
27.32
50.35
50.86
47.59
41.47
EN
60.6
32.6
112
(a)
(b)
(C)
(d)
Figure 3: (a) Comparisons on existing sample reweighting methods. (b) The distribution of the weighting factors computed by comparison methods. The prediction confusion matrix of the (c)
pre-trained model and (d) PDR on video corruption "Defocus" and audio corruption "Crowd".
This is because their selected "reliable" samples contain plenty of false positives, which negates the potential improvement. In comparison, PTA outperforms all SOTA methods, and improve the pre-trained model by 2.56% on average. These improvements are mainly attributed to the effective handling of sample reliability under multi-modal domain shifts. These results demonstrate the rational design and effectiveness of PTA.
4.3 Ablation Study
All experiments in this subsection are performed on Kinetics50-MC (severity level 5).
Contribution of each component. To analyze the importance of each component, we evaluate the baseline with the proposed partition and debiased Reweighting (PDR) and multi-modal attention- guided alignment (AGA) in Table 4 The baseline Tent [37], modified update the same tunable parameters as ours. The results indicate that PDR is an effective sample reweighting solution under multi-modal domain shifts, indicating that the joint assessment of prediction bias and confidence levels are important. The combination of baseline and PDR results in further improvements, supporting the rationale that simultaneously promoting the knowledge in reliable samples and restraining the error propagation of unreliable ones effectively aids model adaptation. Moreover, the integration with AGA consistently improves the performance, demonstrating that guiding the optimization of A- with the semantically meaningful multi-modal cues from A+ is effective.
Comparison with sample reweighting methods. To verify the effectiveness of PDR, we compare with SOTA sample reweighting methods in MM-TTA, including low entropy (LE) [43] and single- modal assistance (UA) O with entropy-based reweighing (EW)
9 25 and robust adaptation (RA) 43 The results are summarized in Fig. 3 (a). We observe that sample reweighting based methods outperform those relying solely on sample selection (i.e., "LE"), indicating that they partially mitigate the prediction bias issue. Unlike previous methods that follow single-modal TTA methods [ and rely solely on confidence or entropy as smaple selection and reweighting signal, PDR introduces a comprehensive measurement. Empirical evidence supports that PDR is effective in combats prediction bias under multi-modal domain shifts, as it simultaneously leverages the respective strengths of both reliable and unreliable samples.
7

---

2.0.0
av
all
av+va
BV++Va
41.5
180
125
0.48
210
41.2
0.5
1:0
Accuracy
41.0
0::00
40.9
40.9
40.9
80
%
40.7
40.7
(a)
(b)
Figure 4: (a) Comparisons on variants of AGA. "none" denotes the performance with the absence of
AGA. (b) The distribution of attention maps for four blocks on clean, A and respectively.
NORTTER
43.4
41.5
41
41.4
415
40
00.6
403
35.2
31.5
0.3
0.7
0.5
0.
2.0
35
Misvev
29>
6.0002
0.0003
0.05
0.3
0.35
(11)
(b)
(c)
(d)
(4)
Figure 5: Comparisons on (a) s, (b) X, (c) Q, and (d) learning rate. (e) Analysis on Z.
Analysis of PDR. We further conduct a deeper analysis of PDR by visualizing the distribution of weighting factors for three SOTA methods in Fig. (b), and providing the prediction confusion matrix for both the pre-trained model and PDR in Fig. 3 (c). As shown in Fig. (b). "UA + EW" assigns high weights the selected candidates, regardless of whether they are false positives or not. This increases the risk of error accumulation. As we demonstrate in Fig. (c), the rates of true positives in predictions run low under multi-modaa domain shifts, and the increasing false positives may counteract the positive contribution. "LE + EW" and "LE + RA" mitigate this issue by conservatively assigning lower weighting factors for each sample. However, this brings limited performance gains as shown in Fig. 3 (a). In contrast to these methods, PDR explicitly assigns negative weights to potential false positives, while also promoting the importance potential true positives based on a joint assessment of prediction bias and confidence levels. This guarantees a significant improvement compared to previous methods.
As illustrated in Fig. 3 (c), the pre-trained model experience severe prediction bias dealing multi- modal domain shits. This bias can be substantially alleviated by equipping the pre-trained model with PDR, which is proven by Fig- (d). Specifically, the false positive rates for the top two biased classes are 24.24 and 24.04, respectively, whereas PDR reduces them to 5.66 and 6.74. This confirms that the design of PDR mitigating false positives is effective.
We also compare our choice Z with static values in Fig. 5 (c). We note that static values are not ideal, overly large small values can introduce noise. The results demonstrate 7 is appropriate for representing the average bias level within the batch.
Analysis of AGA. To investigate the mechanism of AGA, we compare the performance for different attention blocks and their combinations Fig. (a), and provide the distribution of attention maps for each block computed by clean, At, and A The attention map (denoted as "all") can be divided into self-attention blocks (denoted as "aa" and "av"") and cross-attention blocks (denoted as "av" and "vv"". From Fig. 4 (a), observe that the performance gain achieved by applying AGA on different attention blocks or their combination varies. In specific, "aa" and "va" achieve the best results for single attention block, while "vv" and "av" achieve modest improvement. As evidenced in Fig. (b). the attention maps "aa" and "va" in A+ have distributions closer to the clean versions, whereas "vv" and "av" show distributions farther from them. This explains the necessity of guiding the optimization of A with the semantically meaningful multi-modal cues in + In practice, one can effectively choose the attention blocks with semantically meaningful multi-modal cues by analyzing the variance of each block since the optimal ones are less sensitive to multi-modal domain shifts.
Analysis of hyper-parameters We summarize the ablation for hyper-parameters, including in Eq. 2). X in Eq. (5), the learning rate, and the tunable parameters Ô, in Fig. 5 (a)-(d) respectively. We

---

Table 5: Accuracy comparison with SOTA methods Kinetics-MC for multi-modal domain shifts. "A(X)-V(Y)" (x, Y
5)) denotes the noise severity level of audio and video modalities.
A()-V())
A(2)-(2)
A(3)-(())
A(4)-V(4)
A(3)V(5)
A(5)-((3
A(4)-((5)
A(5)-V(4)
Source
61.97
54.84
48.76
40.32
33.16
47.98
32.56
39.60
47.59
48.12
Tont
59.77
49.00
41.12
32.51
23.66
41.84
23.53
32.44
41,05
40.51
ETA
25
64.37
57.32
51.29
42.13
34.85
49.87
34.09
41.12
49.69
50.02
MMTTA
23
61.75
53.28
46.07
36.92
28.77
46.26
28.40
36.49
45.14
45.49
ABPEM
50
66.35
60.21
54.85
46.14
38.86
53,20
38.34
45.13
53.32
53.46
SuMi
9
62.84
55.16
48.94
40.69
32,04
47.64
31.57
38.88
47.88
48.12
READ
43
66.57
61.17
55.51
45.12
40.80
53.87
39.82
46.34
53.64
53.88
PIA
66.88
61.72
57.07
49.83
43.75
55.39
42.71
48.84
55,30
55.56
Table 6: Accuracy comparison with SOTA methods on Kinetics50-C with corrupted video.
Gruss.
Shot
Empal.
Defoc
Glass
Mation
Zoom
Snow
Frost
Fog
Brit.
Coat::
Elistic
Pixel
JPEG
Avg.
Source
18.84
69.36
18.92
67.47
61.3 61.38
70.83
66.19
61.18
61 .6.26
45.35
76,04
51.64
65.99
68.55
65.71
60.58
62.
71.96
63
23.31
69.83
59.68
67.71
64.18
MMTTA
48.60
67.90
61.70
71.41
66.30
62.86
36.58
75.95
51.43
70.05
66.88
60.56
ABPEM
50
50.25
51.32
50.59
67.64
65.37
71.96
68.10
63.93
65 05.52
50.99
76.08
52.33
68.98
69.30
67.75
63.34
SuMi (9)
89.23
69.59
19.52
67.50
62.15
71.08
66.76
61.65
61.3 .8
46.39
76.07
51.64
66.96
68.36
66.64
61.01
READ
43
51.91
52.28
51.77
67.95
65
71.63
69.08
64.81
65 89
61.67
76.37
54.21
69. 3G
70.03
68.46
64.08
PIA
52.93
53.00
68.51
66.57
72.36
64.36
67.11
64.14
75,92
55,05
69.15
70.23
68.41
64.68
Table 7: Comparison Kinetics50-C with corrupted audio.
Gauss.
Crowd.
Thund.
Avg.
Source
73.88
65.38
67.87
69.99
68.39
70.39
69.32
Tent 37
74.08
68.60
70.08
70.67
67.00
71.31
70.29
74.12
70.38
70.91
70.68
MMTTA
29
74.36
67.31
69.53
70.91
68.77
70.95
70.31
74.12
69.20
72.32
70.40
71.09
SuMi 9
73.87
66.19
68.28
70.38
70.58
70.00
69.88
Source
ETA MM READABFEM PIA
READ
43
74.12
69.30
70.02
70.57
72.64
70.85
71.25
PTA
73.52
70.00
70.40
70.31
73.36
70.79
71.40
Figure 6: Efficiency comparisons.
denote the query/key/value transformation matrices of the attention layer and the Layer Normalization (LayerNorm) module in the pre-trained model as W WoRV and WLN, respectively. Fig. (a)-(d) demonstrate the robustness of our method to hyper-parameters.
4.4 Further Analysis
Exploration on changing environment. Considering that the noise severity levels each modality may fluctuate in dynamic environments, further categorize multi-modal domain shifts into four distinct cases (See Appendix G): video and audio noise levels change synchronously; @ video noise level is constant, audio noise varies independently; @ video noise level varies randomly, audio noise level matches it synchronously; @ both video and audio noise levels vary randomly and independently. The results summarized in Table 5) We observe that the performance of the pre-trained model ("Source"" varies substantially across D-G, highlighting the importance of assessing the impact of varying noise severity levels in each modality. Specifically, for D- PTA achieves the highest performance, with average improvements of 7.4%, 9.3%, 7.7%, and 7.4% over the pre-trained model, respectively. These empirical results demonstrate the effectiveness and robustness of method in dynamic environments, delivering the best performance under changing environment.
Comparison single-modal domain shifts. We evaluate the performance of comparison methods under single-modal domain shifts, with the results for corrupted video and audio presented in Tables 7 respectively. Our method improves performance by 4.1% and 1. % on corrupted video and audio modalities, respectively, over the pre-trained model. This demonstrate PTA's robustness to both single- and multi-moda domain shifts, whereas existing methods often fail the latter case.
Efficiency comparisons. Following 43 50 we report the number samples processed per second (samples/sec) and the total count of trainable parameters, i.e., in millions (M), to enable efficiency comparisons Fig. 6 In summary, our method exhibits similar efficiency compared to SOTA methods.

---

Conclusions
In this paper, investigate a practical challenge: multi-modal domain shifts for TTA. We reveal that existing single-modal methods struggle to identify and utilize reliable samples within batch amid severe prediction bias. To address this challenge, we propose Pertitonn-Then-Aee comprises Partition and Debiased Reweighting (PDR) and multi- modaal Ereetton--Gidde Alignment (AGA). PDR partitions online data into potential reliable and unreliable subsets based on prediction bias, assessed via the uniformity of their predicted label distribution. It then employs quantile-based reweighting strategy that adjusts sample contributions in entropy optimization by jointly considering prediction bias and confidence levels. AGA further regularizes PDR to focus semantically meaningful multi- mollil cues by aligning the attention map distributions unreliable subsets with those of reliable ones through maximum mean discrepancy regularization. Extensive experiments demonstrate the effectiveness PTA, addressing both single- and challenging multi-modal domain shifts for TTA in multi-modal tasks.
(PTA), which
Acknowledgement
This work was supported by the National Natural Science Foundation of China under Grant 62476099, 62076101 and 62406323, the Postdoctoral Fellowship Program of CPSF (No. GZC20232993), the China Postdoctoral Science Foundation (No. 2024M753496), and Guangdong Basic and Applied Basic Research Foundation under Grant 2024B1515020085 and 2023A1515010001 the Guangdong Provincial Key Laboratory of Human Digital Twin under Grant 2022B 121001004, the TCL Young Scholars Program.
10

---

References
[1] Haozhi Cao, Yuecong Xu, Jianfei Yang. Pengyu Yin, Xingyu Ji, Shenghai Yuan, and Lihua Xie. Reliable spatial-temporal voxels for multi-modal test-time adaptation. In ECCV, 2024.
[2] Haozhi Cao, Yuecong Xu, Jianfei Yang, Pengyu Yin, Shenghai Yuan, and Lihua Xie. Multi- modal continual test-time adaptation for 3d semantic segmentation. In /CCV, 2023.
[3] Dian Chen, Dequan Wang, Trevor Darrell, and Sayna Ebrahimi. Contrastive test-time adaptation. In CVPR, 2022.
[4] Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. Vggsound: A large-scale audio-visual dataset. In ICASSP, 2020.
[5] Mario Döbler, Robert A Marsden, and Bin Yang. Robust mean teacher for continual and gradual test-time adaptation. In CVPR, 2023.
[6] Junyu Gao, Xuan Yao, and Changsheng Xu. Fast-slow test-time adaptation for online vision- and- language navigation. In ICML, 2024.
[7] Taesik Gong, Jongheon Jeong. Thewon Kim, Yewon Kim, Jinwoo Shin, and Sung-Ju Lee. Note: Robust continual test-time adaptation against temporal correlation. In NeurlPS, 2022.
[8] Yuan Gong, Andrew Rouditchenko, Alexander H Liu, David Harwath, Leonid Karlinsky, Hilde Kuehne, and James Glass, Contrastive audio-visual masked autoencoder. In ICLR, 2023.
[9] Zirun Guo and Tao Jin. Smoothing the shift: Towards stable test-time adaptation under complex multimodal noises. ICLR, 2025.
[10] Zirun Guo, Tao Jin, Wenlong Xu, Wang Lin, and Yangyang Wu. Bridging the gap for test-time multimodal sentiment analysis. In AAAI, 2025.
[11] Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint 1xXiv:/903. /2261, 2019.
[12] Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In ICLR, 2019.
[13] Raza Imam, Hanan Gani, Muhammad Huzaifa, and Karthik Nandakumar. Test-time low rank adaptation via confidence maximization for zero-shot generalization of vision- Dision-language models. In WACV, 2025.
[14] Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijaya- narasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint 1705. 2017.
[15] Roger Koenker and Gilbert Bassett Jr. Regression quantiles. Econometrica: journal of the Econometric Society, 1978.
[16] Jonghyun Lee, Dahuin Jung, Sachyung Lee, Junsung Park, Juhyeon Shin, Uiwon Hwang, and Sungroh Yoon. Entropy is not enough for test-time adaptation: From the perspective of disentangled factors. In ICLR, 2024.
[17] Jingjing Li, Zhiqi Yu, Zhekai Du, Lei Zhu, and Heng Tao Shen. A comprehensive survey source-free domain adaptation. IEEE TPAMI, 2024.
[18] Yafu Li, Xuyang Hu, Xiaoye Qu, Linjie Li, and Yu Cheng. Test-time preference optimization: On-the-fly alignment via iterative textual feedback. arXiv preprint arXiv:2501. 2025.
[19] Jian Liang, Ran He, and Tieniu Tan. A comprehensive survey on test-time adaptation under distribution shifts. IJCV, 2025.
[20] Jian Liang, Dapeng Hu, and Jiashi Feng. Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. In ICML, 2020.
11

---

[21] Jian Liang, Dapeng Hu, Yunbo Wang, Ran He, and Jiashi Feng, Source data-absent unsupervised domain adaptation through hypothesis transfer and labeling transfer. IEEE TPAMI, 2021.
[22] Fan Lyu, Kaile Du, Yuyang Li, Hanyu Zhao, Zhang Zhang. Guangcan Liu, and Liang Wang. Variational continual test-time adaptation. arXiv preprint wrXiv:2402.08182, 2024.
[23] Arsha Nagrani, Shan Yang, Anurag Amab, Aren Jansen, Cordelia Schmid, and Chen Sun. Attention bottlenecks for multimodal fusion. In NeurlPS, 2021.
[24] Chenggong Ni, Fan Lyu, Jiayao Tan, Fuyuan Hu, Rui Yao, and Tao Zhou. Maintaining consistent inter-class topology in continual test-time adaptation. In CVPR, 2025.
[25] Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Yaofo Chen, Shijian Zheng, Peilin Zhao, and Mingkui Tan. Efficient test-time model adaptation without forgetting. In ICML, 2022.
[26] Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Zhiquan Wen, Yaofo Chen, Peilin Zhao, and Mingkui Tan. Towards stable test-time adaptation in dynamic wild world. In ICLR, 2023.
[27] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In JCML, 2021.
[28] Ziqi Shi, Fan Lyu, Ye Liu, Fanhua Shang. Fuyuan Hu, Wei Feng, Zhang Zhang, and Liang Wang. Controllable continual test-time adaptation. In ICME, 2025.
[29] Inkyu Shin, Yi-Hsuan Tsai, Bingbing Zhuang, Samuel Schulter, Buyu Liu, Sparsh Garg, In So Kweon, and Kuk-Jin Yoon. Mm-tta: multi-modal test-time adaptation for 3d semantic segmentation. In CVPR, 2022.
[30] Samarth Sinha, Peter Gehler, Francesco Locatello, and Bernt Schiele. Test: Test-time self- training under distribution shift. In WACV, 2023.
[31] Damian Sójka, Sebastian Cygert, Bartiomiej Twardowski, and Tomasz Trzciński. Ar-tta: simple method for real-world continual test-time adaptation. In ICCV, 2023.
[32] Natasa Tagasovska and David Lopez-Paz. Single-mode model uncertainties for deep learning. In NeurlPS, 2019.
[33] Mingkui Tan, Guohao Chen, Jiaxiang Wu, Yifan Zhang, Yaofo Chen, Peilin Zhao, and Shuaicheng Niu. Uncertainty-calibaati test-time model adaptation without forgetting. IEEE TPAMI, 2025.
[34] Jiaxu Tian and Fan Lyu. Parameter-selecttve continual test-time adaptation. In ACCV, 2024.
[35] Laurens Van der Maaten and Geolfrey Hinton. Visualizing data using t-sne. JMLR, 2008
[36] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In NeurIPS, 2017.
[37] Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. In ICLR, 2021.
[38] Guowei Wang and Changxing Ding. Effortless active labeling for long-term test-time adaptation. In CVPR, 2025.
[39] Guowei Wang, Changxing Ding, Wentao Tan, and Mingkui Tan. Decoupled prototype learning for reliable test-time adaptation. IEEE TMM, 2025.
[40] Qin Wang, Olga Fink, Lue Van Gool, and Dengxin Dai. Continual test-time domain adaptation. In CVPR, 2022.
[41] Zehao Xiao, Shilin Yan, Jack Hong, Jiayin Cai, Xiaolong Jiang, Yao Hu, Jiayi Shen, Qi Wang, and Cees GM Snock. Dynaprompt: Dynamic test-time prompt tuning. In ICLR, 2025.
12

---

[42] Hongliang Yan, Yukang Ding, Peihua Li, Qilong Wang, Yong Xu, and Wangmeng Zuo. Mind the class weight bias: Weighted maximum mean discrepancy for unsupervised domain adaptation. In CVPR, 2017.
[43] Mouxing Yang, Yunfan Li, Changqing Zhang, Peng Hu, and Xi Peng. Test-time adaptation against multi-modal reliability bias. In ICLR, 2024.
[44] Wenmeng Yu, Hua Xu, Fanyang Meng, Yilin Zhu, Yixiao Ma, Jiele Wu, Jiyun Zou, and Kaicheng Yang. Ch-sims: A chinese multimodal sentiment analysis dataset with fine-grained annotation of modality. In ACL, 2020.
[45]
Longhui Yuan, Binhui Xie, and Shuang Li. Robust test-time adaptation in dynamic scenarios. In CVPR, 2023.
[46] Amir Zadeh, Rowan Zellers, Eli Pincus, and Louis-Philippe Morency. Multimodal sentiment intensity analysis in videos: Facial gestures and verbal messages. IEEE Intelligent Systems, 2016.
[47] AmirAli Bagher Zadeh, Paul Pu Liang, Soujanya Poria, Erik Cambria, and Louis Couis-Philippe Morency. Multimodal language analysis in the wild: Cmu-mosei dataset and interpretable dynamic fusion graph. In ACL, 2018.
[48] Ce Zhang, Simon Stepputtis, Katia Sycara, and Yaqi Xie. Dual prototype evolving for test-time generalization of vision-language models. In NeurIPS, 2024.
[49] Bowen Zhao, Chen Chen, and Shu-Tao Xia. Delta: degradation- fully test-time adaptation. In ICLR, 2023.
[50] Yusheng Zhao, Junyu Luo, Xiao Luo, Jinsheng Huang, Jingyang Yuan, Zhiping Xiao, and Ming Zhang. Attention bootstrapping for multi-modal test-time adaptation. In AAAI, 2025.
13

---

Appendix
This appendix provides additional results, figures, and graphs better illustrate method. Specifi- cally, we provide detailed information on the benchmark, the pre-trained model, and the implemen- tation in Sections A [ and[C] respectively. Additionally, Section preseenss further experimental results, including detailed main experiment and comparisons under continual settings. Section B explores different design variants of our method. Section F provides the theoretical analysis of PDR. We also include illustrations of changing environments with varying severity levels, attention maps, and confusion matrices facilitate intuitive understanding G Finally, we discuss the limitations and broader impacts of our work in Sections H and respectively.
A Benchmark Details
We construct two benchmarks based on Kinetics50 14 and VGGSound 4 evaluate the perfor- mance of SOTA methods under synthetic multi-modal domain shifts during test rest-time adaptation. The examples are shown Figure 7 Moreover, we test the comparison methods for real-world multi-modal domain shifts on CMU-MOSI [46]. CMU-MOSEI [47 and CH-SIMS 44
Cless
Clean
Giossimo
Impubec Neise
Defocus Rlur
Crowd
Guussian
Frost
Traffic
Wind
Rrightness
Compression
Figure 7: Examples for corrupted video and audio data.
Kinetics. The Kinetics dataset is comprchensive and high-quality benchmark designed for recog- nizing human actions in vidcos. It contains approximately 500,000 video clips that span 600 distinct action classes, with each class having at least 600 clips. Each video clip is around 10 seconds long and is associated with a single action label. The videos were sourced from YouTube. In line with the previous works 43 9 [50), focus a subset of the Kinetics dataset, which includes 50 action classes, comprising 2,466 test pairs.
VGGSound. The VGGSound dataset is large-scale audio-visual correspondence benchmark con- sisting of short audio clips extracted from videos uploaded to YouTuue. All videos are captured "in the wild", ensuring that there is clear correspondence between the audio and visual content, meaning that the sound source is visually identifiable. Each video this benchmark has a fixed duration 10 seconds. Following privious work pairs.
43
50 obtain 14,046 testing visual-audio
50 we introduce 15 types of synthetic noise [1I. including "Gaussian Noise", "Shot Noise", "Impulse Noise", "Defocus Blur". "Glass Blur". "Motion Blur". "Zoom Blur". "Snow". "Frost", "Fog", "Brightness", "Contrast", "Elastic Transform", "Pixelate". and "JPEG Compression", to corrupt the video modality data. For the audio modality data, we introduce type synthetic noise and types of real-world noise, comprising "Gaussian Noise", "Paris Traffic Noise", "Crowd Noise", "Rainy Noise", "Thunder Noise" and "Windy Noise". Each corruption type applied at five levels of severity. In this way, a total of 90 types of corruption combinations prepared for Kineties50-MC and VGGSound-MC.
Kinetic550-MC and VGGSound-Mrr Following 43
14

---

CMU-MOSL CMU-MOSI 46 benchmark is widely used benchmark for multimodal sentiment analysis, comprising 2.199 short video clips from YouTube. Each clip includes aligned text (tran- scripts), audio, and facial visual data, annotated with sentiment intensity (ranging from -3 to +3) and binary labels.
CMU-MOSEL. CMU-MOSEI [47 benchmark is a large-scale benchmark for multimodal sentiment and emotion analysis, containing 23,454 video clips from YouTube. Each clip includes synchronized text, audio, and visual modalities, annotated with sentiment scores (ranging from -3 to +3) and six discrete emotions (happiness, sadness, anger, fear, disgust, surprise).
CH-SIMS. CH-SIMS 44] is multimodal sentiment analysis dataset for mandarin, featuring 2,281 video clips from Chinese TV shows and vlogs. It provides text, audio, and visual data, annotated with both continuous sentiment scores (ranging from 1 to +1) and binary labels.
For sentiment recognition, as three datasets has different emotion labels, we categorize continuous sentiment scores into three classes: scores less than 0 as "Negative" scores equal to 0 as "Neutral", and scores greater than 0 as "Positive". In this way, we are able to test the model, which is pre-trained on the source domain, on the target domain. We adopt the pre-processed data of CMU-MOSI, CMU-MOSEI, and CH-SIMS provided by [10].
B Pre-trained Model Details
For the experiments on synthetic multi-modal domain shifts, we use the pre-trained CAV-MAE model provided by [43] consistent with previous works 19 [0]. Specifically, the model is pre-trained on the respective training sets of Kinetics and VGGSound, respectively. In other words, Kinetics50 and VGGSound serve as the source domains, while Kinetics50-MC and VGGSound-MC represent the target domains.
For the experiments on real-world multi-modal domain shifts, we train three models separately for the CMU-MOSI [46), CMU-MOSEI [47), and CH-SIMS [44] benchmarks. An Adam optimizer with a learning rate of lc-4 is adopted. The batch size is set to 24, and each model is trained for 30 epochs. We the validation set of each dataset select the best pre-trained model.
C Implementation Details
TENT. For TENT 337. we use the Adam optimizer with a learning rate of Ic-4 on Kinetics-MC and VGGSound-MM We set the tunable parameters as those in LayerNorm Module. The implementation follows the official code
ETA. For ETA 25 we use the Adam optimizer with learning rate of 1e-4 on Kinetics-MC and VGGSound MC. We set the tunable parameters as those in LayerNorm module. Moreover, we set the exponential moving average factor, the cosine similarity threshold, and the entropy threshold to 0.9, 0.05, and 0.4 x IC(C), respectively. Here, C is the number of classes. The implementation follows the official c
MMTTA. For MMTTA (29,, We use the Adam optimizer with a learning rate of le-4 on Kinetics-MC and VGGSound-MC We set the tunable parameters those in LayerNorm module. Moreover, we set the exponential moving average factor for the teacher model as 0.995. The implementation follows the official codo
READ. For READ 4 we use the Adam optimizer with a learning rate of le-4 on Kinetics-MC and VGGSound-NM -CC. The tunable parameters are set to the query/key/value transformation matrices of the attention layer in the fusion block. The implementation follows the official code
SuMi. For SuMi 1, we use the Adam optimizer with a learning rate of le-4 and le-5 on Kinetics- MC and VGGSound-MC, respectively. The tunable parameters are set to the query/key/value transformation matrices of the attention layer the fusion block. The implementation follows the official codd
ABPEM. For ABPEM 9 we use the Adam optimizer with a learning rate of Ic-4 on Kinetics-Me and VGGSound-MC. The tunable parameters are set to the query/key/valuc transformation matrices
htnp://gituub.oom/mt http:///rm.gggglaa////AAA/
htts://ww..ne labs. www.ccoommmmmmmMm...
READ
http://gthub.com/zguuSSMMMM
15

---

of the attention layer in the fusion block. We re- re-implement ABPEM based on their original paper since the code public available.
D More Experimental Results
Main experiment details. We report the detailed results of comparison methods total 90 corrup- tion combinations, and summarize the results in Table I0 In summary, our method achieves the best results across all corruption combinations, significantly outperforms the SOTA methods.
Comparison on continual settings. Continual test-time adaptation [40] (CTTA) introduce a chal- lenging setting, where pre-trained model operates in non-stationary and continuously changing environments, with the target domain distribution changing over time. All previous methods do not consider this setting; instead, they focus a fully test-time adaptation (FTTA) [77] scenario, where the pre-trained model can restore its parameters when distribution shift changes. We test our method and SOTAs the CTTA setting with multi-modal domain shifts, and report their performance in Table 8 We observe that most methods fail to handle CTTA, mostly because the accumulated errors render catastrophic forgetting 44). In comparison, ETA [25] and PTA still outperform the pre-trained model with different adaptation strategies. Specifically, ETA (25) limits the data used in loss computation very small subset, which means the pre-trained model could be unchanged when the online data contains extensive noise. On the other hand, PTA effectively handles sample reliability, preventing error accumulation, and eventually outperforms the pre-trained model by 11.9%.
Comparison on different batch sizes. In real- world scenarios, online data is often insufficient, and times, only limited amount is available, posing significant challenges for effective model adapta- tion. Previous research also indicates that batch size is a key factor influencing overall performance during TTA 266 Therefore, we validate our method with different batch sizes, and summarize the performance in Fig. 8 To adapt to extremely small batch sizes, we set a memory queue to store history predictions to compute Eq. ( and Eq. (2. In other words, Z and IC are transformed from an online manner to offline manner this case. From Fig. 8, we observe that most methods fail to handle small batch sizes, because their weighting factors become severely unreliable. On the contrary, method shows robustness to small batch sizes, consistently outperforms the pre-trained model ("Source") by 7.2% on average.
Comparison CIFAR-11 1EFAR-10110--6 and ImageNet-C. We experiments single-modal datasets, such as CIFAR-10/100 average accuracy over 15 types of coruptiion for these benchmarks in Table 13 We observe that method outperforms READ on single-modal datasets and matches specialized single-modal methods (surpassing ImageNet-C). Crucially, it excels all SOTAs in challenging multi-modal domain shifts.
[22 and ImageNet-C [12] using CNN-based backbones. We report the
Comparison on multi-modal domain shifts using CLIP. We also run experiments on CLIP (ViT- B-16) [27]. To simulate multi-modal domain shifts, we use ImageNet-C [12] for the image branch and apply Gaussian Noise to the text branch. The results are presented in Table 14 These results indicate that our method outperforms others in the majority of cases by substantial margin, thereby confirming its effectiveness on vision- -asoun-lan models.
Table 8: Accuracy comparison with SOTA methods on Kinetics50-MC with corrupted video and audio modalities (severity level 5) in continual settings.
Causs.
Shot
Inggal.
Defac:
Class
Mution
Zoom
Snow
Frost
Fog
Brit.
Conir.
Elastic
Pisel
JPEG
Avg.
Ten:
37
6.72
2:21
1.97
1.99
1.97
1:97
1.97
1.97
2.07
1.97
1.90
1.97
1.57
1.57
2.07
2.32
ETA
x1
12.90
13.29
12.43
36.39
42.10
51.07
49.60
36.51
40.57
36.74
57.25
26.36
54.89
47.88
44.73
37.55
26
12.18
6.34
5.34
6.17
3.44
3.0%
1.96
4.18
SuMi
12.38
6.63
2.67
3,62
3.37
2,83
2.51
2.48
2.15
2.67
2.77
2.20
2,47
3.58
ABPEM 50
12.27
12:65
12:71
29.51
29.00
33.82
32.47
25.57
28.82
26.04
36.70
13.78
33.72
32.07
30.15
26.12
PTA
20.55
26.38
27.72
47.78
45.20
52.63
18.22
38.16
42.38
32.12
53.38
25.11
44.70
44.48
39.48
39.24
16

---

B31C1 500 =
batch F25 2
B3tck site wi4
Borch s25
Botch Fac -
37.9
18.1
39.1
Accirecy the
.3.0 31.0
26:2
27.6
20.4
25.2
21.7
10.7
Figure 8: Performance comparisons on different batch sizes.
Table 9: Accuracy comparison with the original and variants (denoted as *) of PDR on Kinetics50-MC with corrupted video and audio modalities (severity level 5).
Gauss.
Shot
Impul.
Defoc.
Glass
Motion
Zoom
Saow
Frost
Brit.
Courr.
Elastic
Pixel
JPEG
Avg.
Source
13.95
13.05
36.94
45
41.79
30.36
55.29
42.30
38.
57.77
AGA
21.78
23.18
22.35
45.37
51.07
48.57
39.08
42.93
27.36
49.17
49.63
46.31
40.74
PDR+AGA
21.93
22,98
22.11
47.72
45.92
52.55
49.31
40.25
43.57
39.66
59.99
27.32
50.35
50.86
47.59
41.47
Table 10: Accuracy comparison with SOTA methods on KC with corrupted video and audio modalities (severity level 5). "Audio-c" denotes the corruption type of audio modality. The best performances are highlighted.
Audio CCowd
Gauss.
Shot
Impul.
Defoc.
Glass
Motion
Zoom
Snow
Frost
Fog
Brit.
Contr:
Elastic
Pixel
JPEG
Avg.
Source
9.34
9.54
9.35
31,99
35.45
41.31
38.78
25.37
29.90
19.99
54,03
16.09
41.15
36.97
38.47
29.38
Tent
4.83
5,07
5.15
13.60
35.53
25.86
28:38
16.74
22.32
8.06
41.32
22.48
26.88
21.38
8.86
9.15
54.53
18.02
45.25
50.75
19. 12.49
43.53
31.01
MMTTA
22.86
39.36
11.44
42.77
26.78
READ
8.61
9.45
9.87
38.88
47.88
42.91
26.03
57.91
19:09
44.62
42.73
33.12
SuMi [
8.66
9.21
9.03
$2.66
35.97
43.54
30
20.88
19:33
54.38
16.16
40.69
37.03
38. 18.81
20 55
ABPEM
50
7,20
7.10
6,68
39.55 98
38.95
47.70
42.93
32.60
35.74
21.58
57.65
17.87
44.57
42.72
42.05
32.36
PTA
20.02
20.74
19.75
45,90
43,50
51.16
47.13
$8.99
42.01
38.59
59.32
26.29
49.13
49.57
46.39
30.90
GGaneic
Giusss
Shot
Tergul.
Defoc
Glass
Metion
7aom
Snow
Frost
Frge
Rrit:
Contr:
Flastic
Pixel
IPRG
Avg.
Sonre
17.96
19.33
18.55
47.56
40.97
$4.29
48. 48
52.71
35.67
21.94
60.17
24.62
43.98
48.26
44.26
17:34
Tent
11.70
11.96
11.60
45.
24.34
$4.14
50
14.04
20.81
8.74
60.95
13.45
25.03
45.13
37.96
29.26
ETA
25
18.17
19.61
18.74
48.34
43.45
$4.70
50.64
35.12
26
21.90
61.29
24.74
46.12
50.93
46.25
38.43
MMTTA bj
15.07
16.92
15.58
47.11
36.96
54.35
49.99
20.41
28
12.73
60:92
18.41
35.65
45.86
42.45
33.55
READ
21,02
23.01
22.03
50.51
45.35
56.42
51.94
39.00
25.51
61.65
29.70
50.31
52.87
47.60
41.19
SuMi
9
17.85
19.26
18.53
47.44
40.21
54.66
43.713
20.67
60.24
43.57
36.85
ARPFM
20.80
22.59
64
45.07
15.00
$3.23
47.54
39.65
26.33
27.76
27.23
51.87
50.07
54.10
43.80
47.52
44.22
32.33
53.79
54.86
Audio-ciRais
Gauas.
Shot
Impul
Defoc.
Motion
Zoom
Snow
Frost
Fog
Coatr:
Elastic
Pixel
JPEG
Avg.
Source
12.96
14,06
13.32
40.49
38.04
40.70
68
32.68
19:45
54.51
14.49
43.35
39.34
38.89
32.02
Teur
4.18
4.35
4.09
29:90
33.75
51.75
19:08
7..65
35.18
17:76
26.21
23:24
FTA
12.81
13.96
13.40
42.31
90
33.57
18. 4B
56.54
13.92
45.51
40.75
33.
MMT A
5,61
3.65
5.33
38.67
36. 26.99
48.76
41.01
24.02
29.54
10.76
54.64
6.11
42.09
35.21
35.85
28.04
READ
15.66
16.95
15:99
24.77
41. 58
51..00
44.66
54.17
36.76
27.30
57.24
19.68
46.28
44.44
45.05
35.95
SaMi 9
12.39
13.24
12.98
40.00 90
37.55 16.86
48.64
40.78
20. 50
32.47
17.60
$4.49
13.22
43.30
18
38.86
31.69
ABPEM
50
15.25
16.52
15.80
44.46
41.39
50.37
44
34.10
36.82
21.30
56.95
10,97
46.15
44.79
43.05
34,82
PTA
201.6
21.23
211088
46.56
45.48
51.60
47.91
35.33
42.28
38.26
58.21
24.87
50.04
50.11
47.24
41.16
Mulii-c/Thumlle
Guuss
Shot
Terpul.
Defoe:
Glass
Metion
7oom
Snow
Fross
Figg
Rrit.
Contr:
Flastic
Pisel
JPRG
Avp.
Source
14,84
15.59
14,83
33.41
35.91
43.22
42.75 79
52.27
32.92
23.73
53.35
20.00
42.26
31.91
29.36
31.06
Tent
7,26
7,80
7,20
15.21
17.68
23.52
23.82
20.73
27.46
10.95
37.03
12.00
33.89
14.16
13.18
18.13
ETA 25
14.76
15.52
14.47
35.76
38.68
43.79
33.55
33.83
23.64
20.09
44.87
34.11
29.99
32.32
MMTTA 20
8.71
9.31
8.72
19.95
24.75
31.79
52.5 55
31.44
15.33
40.79
19.58
23.62
13.79
12.61
16.06
42.67
47.70
32.42
47.35
32.33
31.19
SuMi
13.84
14.95
13:94
33.75
37.001
45.12
43.78
31.83
32.91
22.96
55.01
19.29
41.91
21.25
301200
ABPEM
50
8.43
9.03
8.37
28.59
43.03
51.08
47.95
37.6/7
40.17
31.71
59.03
22.97
47.58
17.97
16.18
31.34
PTA
24.40
26.08
24.07
50.91
49.25
55.33
51.03
42.90
45.96
42.201
62.35
30.58
53.29
53.70
51.01
44.19
Audin-e/Traffic
Giuss.
Shot
Imruul.
Defose
Glass
Motion
7aom
Snow
Frost
Frgg
Rrit.
Contr:
Flastic
Pisel
JPRG
Avg.
Source
9.46
10.60
9.58
29
32.72
37.35
37.55 53
25. 40
27.58
18:53
$2.61
13.35
40.14
32.78
33.62
27.36
Tenr 577
3.73
5,89
3.N6
18.32
32.70
33.10
33.67
12.15
16.31
6.48
54.19
6.79
36.79
24.9%
33.51
21:75
ETA 25
9,73
10.68
9.75
30.35
35.5 35.59
40.79
39.52
28.30
29.
17.57
54.53
13.51
41.70
35.60
36,57
28.50
MMTTA 2D
7,83
8.24
7.59
244
35.63
35.70
37.50
19:21
23.24
9.06
$4.30
8.97
41.05
30.59
34.16
25.03
READ
10.82
11.54
10.36
36.11
37.29
45.17
42.55
30.24
33.18
24.77
56.85
15.89
43.54
35.81
38.66
31.72
SuMi 9
9,23
10.26
9.41
28.06
37.88
37.95
16.66
13.37
39.99
$2.49
33.9%
27.17
ARPFM
8.82
9.96
45.23
43.63
11.62
35.03
30.42
18.67
19.27
18.86
42.20
49.20
46.79
37.87
41.01
35.53
58.35
23,85
46.77
47.20
44.55
36.29
AuWid
Gauas.
Shot
Impul.
Detoc.
Glass
Motion
Zoom
Snow
Fog
Elastic
Pisel
JPBG
Avg.
Source
12.96
14.57
12.87
40.71
38.57
47.34
42.93
33.71
32.50
19.18
21.12
42.90
44.16
33.51
Tear
7.10
19.34
29.59 29.59
21.25
7.94
$7.00
13.57
43.92
44.76
28.54
FTA
13.05
13.87
12.44
41.35
40.63
48.55
78
16
34.00
18.33
$7.79
30.80
44.78
45.87
43.63
34.32
MMT A
7,59
8.43
7.18
40.47
36.02
47.48
41.80
29.46
27.50
10.40
57.65
16.98
43.22
45.61
42.69
30.50
READ
14.94
17.25
15.36
43.67
41.5
49.41
45.77
37.22
36.5%
21:62
58.56
24.85
46.22
46.66
43.95
36.23
SuMi 1
12.30
14.11
12.79
40.47
37.5 97.92
47.03
42.74
33.28
31.88
17.35
57.20
31.30
42.93
43.98
42.13
33.16
AВРEM 50
13.12
13:75
11.91
43.83
30.85
49.67
45.84
37.30
36.80
10.92
59.45
24.119
46.18
40.66
43.79
34,88
PTA
21.87
22.81
21,85
46.88
45.02
$1.08
49.88
39.61
2.64
39.13
$9.39
25.99
49.05
49.73
47.05
41L73
17

---

Table 11: Accuracy comparison with the original and three variants (denoted as MSE, L2, and KL) of AGA on Kinetic550-MC with corrupted video and audio modalities (severity level 5).
Gauss.
Sbot
Impul.
Defoc.
Glass
Mution
Zoomi
Scorw
Frust
Fog
Bril
Contr.
Elastic
Pixel
JPBG
Avg.
Source
12.92
13.95
13:05
37.20
36,91
15.30
41.79
30.36
31.88
20.47
55.29
18.28
12.30
35,90
37.77
31.76
21.52
22.68
21.85
47.04
45.09
52.02
42.66
37
59.94
26.73
49:23
50.53
47.72
40.81
KL.
20.39
21.60
20.85
45.95
44.61
51:33
28.90
40.07
43.39
39.23
58.74
27.18
49.75
49.68
46.90
40.58
MMD
21.93
22.98
22.11
47.72
45,92
52.55
49.31
40.25
43.57
39,66
59.99
27.32
50.35
50.86
47.59
41.47
Table 12: Accuracy comparison with the original and two variants (denoted as feature- and logits- level) of AGA on Kinetics50-MC with corrupted video and audio modalities (severity level 5).
Causs.
Sbot
Impul
Defoc.
Glass
Motion
Zoom
Snow
Frust
Fog
Brit.
Contr.
Elastic
Piscl
JPGG
Arg.
Source
12.92
13.95
13.15
37.20
26.95
45.30
41.79
30.36
31.88
20.47
55.29
18.28
42.30
36, 90
37.77
31.76
Feature level
20.57
21.57
20.94
45.87
44.64
51.35
48.95
20..3
43,40
59. 38
58.74
27.01
49.77
$9.58
46.96
41.59
Peatun--mppleeve
20.66
21.71
30.66
45.88
44.65
51.34
48.93
10
43.41
59 22
58.74
36.
49.75
49.68
46
40.57
Logits level
20.44
21.61
20.55
15.58
51,22
48.74
39.55
39.34
56.76
26.88
49.74
40.29
Ancmmion-leve
21.93
22.98
22.11
49.31
50.86
Table 13: Comparison CIFARIO-C, CIFARIOO-C, and ImageNet-C.
Method
CIFAR-10-C CIFAR-100-C ImageNet-C
Source
56.5
53.6
18.0
TENT
37
81.7
68.5
34.7
ETA
25
81.9
68.9
39.7
READ
43
79.0
61.5
31.3
Ours
81.3
68.8
41.3
Table 14: Accuracy comparison using CLIP.
Method
Gauss.
Shot
Impul.
Defoc
Glass
Motion
Zoom
Frost
Fog
Brit.
Contr.
Elastic
Pixel
JPEG
Avg.
Source
10.7
11.7
11.3
22.1
14.6
23.5
21.2
30.4
29.1
33,6
5188
16.1
12.4
29.5
30.8
23.3
TENT 37
5.5
4.4
6.7
23.8
18.5
26.4
23.4
32.0
30.2
36.1
52.9
21.6
13.2
33.6
34.7
24.2
ETA 25
18.1
19.2
19.7
25.1
21.7
28.7
25.7
34.5
31.7
38.0
53.9
25.8
17.1
36.3
36.7
28.8
12.4
21.8
23.2
30.5
33.7
52.0
29.5
23.7
Ours
20.6
22.3
20.9
24.5
25.0
30.5
27.7
35.4
32.3
40.4
54.5
30.2
26.4
38.2
38.6
31.2
E More Ablation Studies
Variants on PDR. In the original design PDR, we take both prediction bias and confidence levels to formulate the reweighting indicator. In this subsection, we substitute softmax confidence scores of each data with their entropy. Specifically, we compute the quantile-basee importance weighting using prediction bias and entropy levels instead of with softmax confidence scores. We show the results in Table 9 is shown that softmax confidence scores are more effective in handling domain shifts, which aligns with the findings of previous methods (50).
Variants on AGA. AGA is originally computed by MMDD-based regularization on the attention maps. substitute MMD-based regularization with MSE, I.2, and KI. divergence, Table [1] Moreover, we change the attention maps to fusion
In this subsection, and compare their performance features, fusion feature maps, and output logits, and summarize their performance in Table I2 The difference between fusion features and fusion feature maps lies in whether average pooling is applied. From Table IT] we observe that both element-wise and distribution-wiss alignments underperforms MMD- MMD-based regularization, or static patterns in the attention maps, but rather by dynamically changing relationships across modalities. From Table 12] we observe that the substitution options yield similar results, likely due to the prior knowledge across modalities are crucial for robust adaptation, whereas regularizing features and aligning logits appears to be ineffective.
the robust prior knowledge is not characterized by specific values
F Theoretical Analysis
To strengthen the theoretical foundation of PDR, we provide a brief analysis of why prediction bias arises and how our design mitigates it. Suppose the output logits 2; [2ı1,---. =ik], have
the softmax probability Pa en
Fih
and the entropy loss Lent - D Ek=1 Pik log Pik- We
18

---

first derive the gradient of the softmax function with respect to the logits:
Opss = Pij (6zk - Psk), as HCm log Pij
well as gradient of the entropy loss with respect to the softmax outputs: 8p:;
By applying the chain rule, we obtain the gradient of the entropy loss with respect to the logits as
follows: dhm
2==1 1 if else
HLa..
Opu
- (- log Pij 1) Pgj (ōjz Pik), where the Kronecker
Bpu
delta: 8jt
This expression shows that entropy minimization encourages the model to produce increasingly (over) confident (i.e., low-entropy) predictions by amplifying the largest logit and suppressing the others, which is also discussed in previous works [26]. To address prediction bias in multi-
domain shifts, PDR partitions each data batch into biased and unbiased subsets using prediction
bias indicator (Eq. (I), which leverages batch-average prediction frequency to identify overconfident predictions, often false positives due entropy minimization. These biased samples regularized via entropy maximization (the second term Eq. (2,, which counteracts noise propagation by encouraging uniform probability distributions, as supported by thcoretical insights from [21 on entropy regularization under noisy conditions.
Additionally, balance bias and confidence levels and prevent issues like unreliable gradient from low-confidence predictions [25], we propose a quantile ranking strategy (Algorithm 2 This strategy reweights unbiased samples by assigning higher weights to confident predictions and lower weights to uncertain ones, ensuring stable and reliable test-time adaptation. 'he theoretical reasonabilityy of this approach stems from its alignment with robust optimization principles, where reweighting mitigates the impact of noisy outliers, as discussed in [25]. We demonstrate PDR's consistent stability and superior performance across challenging multi-modal domain shift scenarios, including continual settings (Table () and changing environments (Section 4.4p
G Visualizations
Attention map. We visualize the attention maps of clean data, A+, and A- is shown that the distributions of "audio-audio" block and "video-audio block in A+ are closer to those in clean data attention map compared the distributions A- This demonstrates that AGA effectively focuses on robust information dependencies while suppressing the influence of sensitive ones.
Confusion matrix. We provide more visualization for confusion matrix in Figure I0 It is demon- strated that our method mitigates the false positive issue in the pre-trained model and generates more true positive predictions, which explains its effectiveness.
Audia
Video
Audio
Video
Audio
Video
0,2000
0,2000
0.2000
μ:
μ::
0.1500
0.1500
0.1550
0.1000
0.1000
0.1000
μ: 0.36
4.17
-0.000
0.52
3.75
-0.000
μ:: 0.54
3.69
-.0000
a": 0.01
0": 4.83
a": 0.12
8": 3.66
a": 0.10
0": 3.52
Figure 9: Visualization on the attention maps for clean data (left), A (middle), and AA (right). The values are amplified by 10, 000 times for clarity. The number upon the blocks denotes the mean and variance. The corruption type for audio and video modalities are "Crowd" and "Defocus".
19

---

(a) 20.47 VS 39.66
(b) 30.36 vy 40.25
(c) 41.79 vs 49.31
Figure 10: Comparison on confusion matrix of predictions for the pre-traine pre-trained model (Top) and our method (Bottom). The corruption type for audio and video modalities are "Crowd" and "Fog" in subfigure (a). "Crowd" and "Snow" in subfigure (b), and "Crowd" and "Zoom blur" in subfigure (c), respectively. "X vs. Y" denotes the performance comparison between the pre-trained model and PTA.
Variant-1
Variant-2
Variant-3 3
Variant-4
Video/Audio Modality
Oncline Batch
Test Data Stream
Noisc Severity Level
Figure 11: Illustration of multi-modal domain shifts with changing severity levels.
H Limitations
Test-time adaptation (TTA) focuses on the deployment phase of a pre-trained model, emphasizing the need for efficiency. Although our method achieves similar efficiency compared to existing methods, as demonstrated in Fig. [6 there is still room for improvement in efficiency for the sake of real-world deployment. Due the limitation of computational resources, we only test our method on one GPU. However, it could be accelerated through parallel processing using multiple GPUs.
I Broader impacts
Test-time adaptation enables a model pretrained on source domain data to adapt to target domain data in real-time. It has broad applications in dynamic scenarios such as autonomous driving. To the best of our knowledge, our work does not have obvious negative social impacts.
20

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justifiicton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] generally preferable to "[No] it is perfectly accePTAble answer "[No] provided a proper justification is given (e.g.. "error bars are not reported because it would be too computationally expensive" "we were unable find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased binary way. we acknowledge that the answer is often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendíx. If you answer [Yes] a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided AGAos for your answers.
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims made in the abstract and introduction accurately reflect this paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
21

---

Justification:
We discuss the limitations of this paper in Appendix H
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approxinations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proof'
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We comprehensively disclose the implementation and training details of our method for readers to reproduce the results, We will release our code on github later.
Guidelines:
22

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We evaluate our method on opensource datasets which are available for anyone. We will release code on github later.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is accePTAble answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips.ccppullcc/guue http://iccggiieeeeeeeuuubiiiiiii
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
23

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We elaborate the experimental setting and details in Sec. 4.1 and Appendix C
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
We report the implementation details in Sec. 4.1
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula,
call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide information the computer resources in Sec.
Guidelines:
The answer NA means that the paper does not include experiments.

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: This paper conforms with the NeurIPS Code of Ethics in every respect.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the the broader impacts of this paper in Appendix !
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper poses such risks
25

---

Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We cite the creators or original owners of datasets that are used in this paper.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of in the package should be provided. For popular datasets, paaerssithcode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re- re-packaged, both the original license and the license of the derived asset (if has changed) should provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
This paper does not release new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
26

---

Answer:
[NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing. editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non stanndaddrrd components.
Please refer to our LLM policy (https://neut) /c/////reeeeecc//2222/////////// for what should or should not be described.
27


Reviews:
[{"review_id": "7PfL00M05V", "reviewer": "Reviewer_h6TZ", "summary": "This paper investigates the test-time adaptation problem in multi-modal scenarios where both single-modality and multi-modality domain shifts may exist. The authors propose an approach dubbed Partition-Then-Adapt (PTA) to address this new setting. PTA consists of two main components, i.e., Partition and Debiased Reweighting (PDR) and multi-modal Attention-Guided Alignment (AGA). PDR leverages the relative prediction frequency to split testing samples into reliable and unreliable subsets and adopts contrary adaptive weighting for the entropy minimization of the two sets. AGA can further regularize the adaptation of unreliable sets to focus on semantically meaningful multi-modal cues. Extensive empirical results demonstrate PTA consistently outperforms state-of-the-art TTA methods on multi-modal tasks.", "questions": "See weaknesses.", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n- The problem of test-time adaptation (TTA) is practical and significant. Moreover, according to the authors, the submission is the first to study multi-modal TTA with multiple domain shifts instead of single-modality shifts.\n\n- The proposed PTA method is novel, especially the AGA component, which specifically addresses the challenge of multi-modal adaptation with unreliable samples.\n\n- Experiments and ablation studies are very comprehensive. Extensive experiments demonstrate the superior performance of PTA over other TTA methods.\n\n## Weaknesses\n- The PDR component lacks solid support, such as a theoretical analysis of its reasonability.\n- The presentation lacks clarity. The introduction of PDR is confusing due to imprecise notation. The specific meanings of $Z$ and $K$ are unclear—for example, it’s not specified whether they refer to scalars or vectors, or to individual samples or mini-batches.", "was_revised": true, "final_justification": "I intend to retain my initial rating for this submission.", "rebuttal": "### Response to weaknesses\n\n$\\textbf{W1}$: Theoretical analysis.\n\n$\\textbf{A1}$：Thanks for your suggestion. To strengthen the theoretical foundation of PDR, we provide a brief analysis of why prediction bias arises and how our design mitigates it.\n\nSuppose the output logits are $\\mathbf{z}\\_i = [z_{i1}, \\dots, z_{iK}]$, we have the softmax probability $p\\_{ik} = \\frac{e^{z\\_{ik}}}{\\sum\\_{j} e^{z\\_{ij}}}$ and the entropy loss $\\mathcal{L}\\_{\\mathrm{ent}} = -\\frac{1}{N} \\sum\\_{i=1}^N \\sum\\_{k=1}^K p_{ik} \\log p\\_{ik}$.\n\nWe first derive the gradient of the softmax function with respect to the logits: $\\frac{\\partial p_{i j}}{\\partial z_{i k}}=p_{i j}\\left(\\delta_{j k}-p_{i k}\\right)$, \nas well as the gradient of the entropy loss with respect to the softmax outputs: $\\frac{\\partial \\mathcal{L}\\_{\\mathrm{ent}}}{\\partial p\\_{i j}}=-\\log p\\_{i j}-1$.\nBy applying the chain rule, we obtain the gradient of the entropy loss with respect to the logits as follows:\n$\\frac{\\partial \\mathcal{L}\\_{\\text {ent}}}{\\partial z\\_{i k}}=\\sum\\_{j=1}^K \\frac{\\partial \\mathcal{L}\\_{\\text {ent}}}{\\partial p\\_{i j}} \\cdot \\frac{\\partial p\\_{i j}}{\\partial z\\_{i k}}=\\sum_{j=1}^K\\left(-\\log p\\_{i j}-1\\right) \\cdot p\\_{i j}\\left(\\delta\\_{j k}-p\\_{i k}\\right)$, where the Kronecker delta: $\\delta_{j k} = \\{ 1 \\text{ if } j = k, \\text{ else } 0 \\}$.\n\nThis expression shows that entropy minimization encourages the model to produce increasingly (over) confident (i.e., low-entropy) predictions by amplifying the largest logit and suppressing the others, which is also discussed in [10].\nTo address prediction bias in multi-modal domain shifts, PDR partitions each data batch into biased and unbiased subsets using a prediction bias indicator (Eq. (1)), which leverages batch-average prediction frequency to identify overconfident predictions, often false positives due to entropy minimization. These biased samples are regularized via entropy maximization (the second term in Eq. (2)), which counteracts noise propagation by encouraging uniform probability distributions, as supported by theoretical insights from [a, b, 31] on entropy regularization under noisy conditions.\n\nAdditionally, to balance bias and confidence levels and prevent issues like unreliable gradient from low-confidence predictions [19], we propose a quantile ranking strategy (Alg. 1). This strategy reweights unbiased samples by assigning higher weights to confident predictions and lower weights to uncertain ones, ensuring stable and reliable test-time adaptation. The theoretical reasonability of this approach stems from its alignment with robust optimization principles, where reweighting mitigates the impact of noisy outliers, as discussed in [19]. We demonstrate PDR’s consistent stability and superior performance across challenging multi-modal domain shift scenarios, including continual settings (Table 8, Appendix D) and changing environments (Section 4.4). \n\nWe  will add the above theoretical analysis and discussion in the Appendix.\n\n\n[a] J. Liang, D. Hu, Y. Wang, R. He and J. Feng, \"Source Data-Absent Unsupervised Domain Adaptation Through Hypothesis Transfer and Labeling Transfer,\" in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 11, pp. 8602-8617, 1 Nov. 2022, doi: 10.1109/TPAMI.2021.3103390.\n\n[b] J. Li, Z. Yu, Z. Du, L. Zhu and H. T. Shen, \"A Comprehensive Survey on Source-Free Domain Adaptation,\" in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 46, no. 8, pp. 5743-5762, Aug. 2024, doi: 10.1109/TPAMI.2024.3370978.\n\n$\\textbf{W2}$: The meaning of $Z$ and $\\mathcal{K}$.\n\n$\\textbf{A2}$: $Z$ represents the frequency of predicted class for $x$, which is a scalar. $\\bar{Z}$ is also a scalar since it denotes the average frequency of the predicted labels within a batch. $\\mathcal{Z}$ is a vector, consting of all instances' bias level computed by Eq. (1). And $\\mathcal{K}$ is also a vector, representing the softmax confidence. We will clarify the meaning of $Z$ and $\\mathcal{K}$, and bold the vectors to improve the presentation for better clarity, following your advice."}, {"review_id": "Pk5cTtPM9J", "reviewer": "Reviewer_xcN3", "summary": "This paper introduces PTA for test-time adaptation under multi-modal domain shifts. The method contains two components, Partition and Debiased Reweighting (PDR) and multi-modal Attention-Guided Alignment (AGA). PDR splits the test batch into reliable and unreliable subsets using a bias indicator, and then employs quantile-based strategy to jointly consider prediction bias and confidence to reweight samples. AGA adds an MMD-based regularization loss on attention maps.", "questions": "See Weakness", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 1, "significance": 2, "originality": 2, "strengths_and_weaknesses": "## Strength\n\n- This paper is well written and follows a good structure.\n- Promising results are obtained in the studied benchmarks.\n- The effectiveness of the proposed two components is also shown.\n\n## Weakness\n\n- Some claims might not be convincing. For example, the authors claim “Fig. 1 illustrate the effectiveness of existing reweighting strategies diminishes under multi-modal domain shifts” in Line 116. However, Fig. 1 only evaluates the pretrained model, and dose not provide direct evidence or analysis for current methods (e.g., SuMi, READ) in multi-modal domain shifts. The logical connection about the motivation part is weak and requires more elaboration to be convincing.\n- The authors experimentally demonstrate the effectiveness of proposed sample reweighting strategy. However, the core idea of splitting samples into reliable/unreliable subsets based on confidence/bias indicators and give unreliable samples little or negative weights has been introduced by earlier methods (e.g. DeYO[1]). The authors are encouraged to conduct a deeper comparison and discuss more about how PDR differs from, or improves upon those prior reweighting strategies.\n- Some necessary citations might be missing and related work might not be sufficient. For instance, using weighted entropy loss in TTA (Line 152) has already been proposed in TTL[2].\n- Key experimental details are missing. (1) What does the “source model” denote? (2) There lacks essential details about CAV-MAE architecture, such as the number of attention layers used for fusion. (3) It is not clear how many attention layers are updated during adaption and what fraction of total parameters they represent. (4) In Table1, labeling the video+audio modality corruptions as Kinetics50-C may cause confusion with the single-modality Kinetics50-C benchmark. (5) It is not entirely clear what each number in Table 1 and 2 means? (e.g., average over six audio corruptions in Table 1?)\n- Baseline reproduction discrepancies. It's not clear why results of SOTA methods (e.g., SuMi, READ) differ from their original papers in Table 6 and Table 7.\n- Concerns about hyper-parameter fairness for baseline comparison. Figure 5 shows that learning rate and updated modules greatly influence TTA performance, yet PTA and baselines use different learning rates. It is encouraged to supplement comparisons with different learning rates and updating modules, which would make the comparisons more comprehensive.\n- Although the experiments demonstrate the method’s effectiveness, it remains unclear whether PTA can generalize to other single-modal and multimodal datasets (e.g, image-text). It is encouraged to conduct experiments to verify the method’s applicability to vision-language models (e.g., CLIP in TDA[5]) and image shift contexts (e.g., ImageNet-C in FOA[3],  CIFAR10-C in SoTTA[4]). Such experiments would validate PTA’s broader applicability.\n\n[1] Entropy is not enough for test-time adaptation: From the perspective of disentangled factors. In ICLR, 2024\n\n[2] Test-Time Low Rank Adaptation via Confidence Maximization for Zero-Shot Generalization of Vision-Language Models. In WACV, 2025\n\n[3] Test-Time Model Adaptation with Only Forward Passes. In ICML, 2024\n\n[4] Robust Test-Time Adaptation on Noisy Data Streams In NeurIPS, 2023\n\n[5] Efficient Test-Time Adaptation of Vision-Language Models. In CVPR, 2024", "was_revised": true, "final_justification": "I appreciate the authors’ response. I have read the rebuttal carefully, and it addresses most of my concerns.\n\nOne issue remains—W5. The single-modal domain-shift results in Tables 6 and 7 differ substantially from those reported in three prior papers (ABPEM [38], SuMi [9], and READ [31]), yet the manuscript does not provide sufficient implementation details to account for these gaps.\n\nIn the rebuttal, the authors suggest that “minor discrepancies … may arise due to differences in GPUs and random seeds,” but the average accuracies for ABPEM (65.0 vs. 63.34) and SuMi (63.9 vs. 61.01) do not seem to be “minor.”\n\nBecause I am not deeply familiar with the implementation specifics of the previous methods—and because the majority of my other concerns have been resolved, I have raised my overall score to 4 and reduced my confidence score to 2. I expect the authors to include a more detailed discussion of how previous baselines were re-implemented in the revised manuscript, which can better benefit the community.\n\n[38] Yusheng Zhao, Junyu Luo, Xiao Luo, Jinsheng Huang, Jingyang Yuan, Zhiping Xiao, and Ming Zhang. Attention bootstrapping for multi-modal test-time adaptation. In AAAI, 2025.\n\n[9] Zirun Guo and Tao Jin. Smoothing the shift: Towards stable test-time adaptation under complex multimodal noises. In ICLR, 2025.\n\n[31] Mouxing Yang, Yunfan Li, Changqing Zhang, Peng Hu, and Xi Peng. Test-time adaptation against multi-modal reliability bias. In ICLR, 2024.", "rebuttal": "### Response to weaknesses\n\n$\\textbf{W1}$: Evidence of some claims.\n\n$\\textbf{A1}$: Thanks for your valuable feedback. The empirical results in Tables 1-3 and 6-7 provide direct evidence that existing methods perform less effectively under multi-modal domain shifts compared to single-modal domain shifts. For example, compared to the pre-trained model, READ achieves average performance gains of 2.7% across two independent single-modal shift scenarios and 3.6% on multi-modal domain shifts.  In contrast, our method demonstrates superior gains of 3.1% (single-modal) and 9.7% (multi-modal) respectively, showing significantly stronger robustness.\nMoreover, we have provided a deeper analysis in Fig. 3 (a), which illustrates that existing reweighting strategies, such as UA+EW and LE+RA, are less effective under multi-modal domain shifts compared to our method. To strengthen the logical connection of the context, we will reframe Line 116 as \"However, as we demonstrate in the experimentation section (Tables 1-3 and 6-7), the effectiveness of existing reweighting strategies become less effective under multi-modal domain shifts compared to single-modal domain shifts.\"\n\n\n$\\textbf{W2}$: Comparison with DeYO.\n\n$\\textbf{A2}$: Thank you for pointing this out. We have already included a comparison and discussion of representative reweighting methods in Fig. 3 (a) and Lines 239–265. Specifically, while DeYO [1] introduces a selective reweighting strategy, it operates on a strict subset of low-entropy samples with high PLPD scores (see Eqs. 8–10 in [1] and its official implementation). This design overlooks the accumulation of prediction bias that is especially critical in multi-modal domain shifts, where confident but biased predictions are common. In contrast, our PDR module first quantifies batch-level prediction bias (Eq. (1)) to partition the data into potentially reliable and unreliable subsets, and then reweights them accordingly (Eq. (2)). This allows PDR to explicitly suppress false positives and promote reliable adaptation beyond confidence alone. Moreover, we have conducted a new experiment under multi-modal domain shift on Kinetics50 using the same learning rate and update modules. Our method achieves 41.5% accuracy, significantly outperforming DeYO’s 28.5%. We will incorporate these results and discussions into Section 4 and add DeYO to Fig. 3 (a) and (b) for a more comprehensive comparison.\n\n$\\textbf{W3}$: More citations.\n\n$\\textbf{A3}$: Thank you for the helpful suggestion. We acknowledge that TTL [2] also employs weighted entropy loss. However, the reweighting strategies differ: TTL focuses on vision-language tasks by reweighting original and augmented views within the same sample, while our method operates on multi-modal domain shifts by partitioning the batch based on prediction bias and confidence. Unlike TTL and ETA, which primarily rely on entropy, our PDR module explicitly identifies and suppresses biased predictions, a key issue under multi-modal shifts. We will cite TTL and include this distinction in the related work section to clarify our contribution.\n\n$\\textbf{W4}$: Experimental details.\n\nWe sincerely value your detailed comments.\n\n$\\textbf{W4.1}$: The ''Source model'' denotes the CAV-MAE pre-trained model (see Lines 198-199) for experiments on synthetic multi-modal domain shifts, and denotes a distinct transformer-based model [10] (see Lines 201-202) for the experiments on real-world domain shifts. The ''Source'' represents the performance of the pre-trained model for direct inference without any adaptation. We will add the meaning of ''Source'' in Line 196 and Appendix C.\n\n$\\textbf{W4.2}$: We adopt CAV-MAE as the backbone architecture following [9, 31, 38]. Specifically, CAV-MAE employs an encoder-decoder architecture pre-trained on large-scale video datasets using both contrastive learning and masked image modeling. Its encoders both comprise 11 modality-specific Transformer layers for feature extraction, followed by an additional layer for cross-modal fusion. For more details, please kindly refer to [8, 31]. We will incorporate the details of CAV-MAE in Appendix B.\n\n$\\textbf{W4.3}$: As we declare in Lines 204-205, ''Following [31, 38], we update query/key/value transformation matrices of the attention layer in the fusion block''. To be more specific, the total parameters are 1.8M (1% of total), which remain the same as [31, 38].\n\n$\\textbf{W4.4}$: To eliminate the confusion, we will change Kinetics50 with multi-modal domain shifts as Kinetics50-MC (multi-modal corruption) and remain Kinetics50 with single-modal domain shifts as Kinetics50-C following your advice.\n\n$\\textbf{W4.5}$: In Tables 1-2, each number denotes the average performance over six audio corruptions under a specific video corruption. We will add the meaning of each number to the captions of Tables 1-2.\n We also present detailed results regarding the performance under a specific audio corruption paired with a specific video corruption in Table 10 of the Appendix.\n\n$\\textbf{W5}$: Reproduction of comparison methods.\n\n$\\textbf{A5}$: Thank you for the observation. To ensure fair comparison, we reimplemented all methods using their official code (if available), under the same adaptation protocol [31] using their default hyper-parameter settings. We report the average over 5 random seeds. Minor discrepancies with the original papers may arise due to differences in  GPUs and random seeds.\nEven when compared to SuMi’s original results, our method outperforms it under video-modal shifts (64.7 vs. 63.9) and achieves comparable performance under audio-modal shifts (71.4 vs. 71.9). We guarantee that all methods are implemented under fair conditions.\nWe will clarify these implementation details and hyperparameter settings in Appendix C to ensure transparency and reproducibility.\n\n$\\textbf{W6}$: Hyper-parameters.\n\n$\\textbf{A6}$: Thank you for your insightful review. We use the baseline READ's default learning rate of 1e-4. As Fig. 5 (c) and (d) already show our method's performance with varying learning rates and updating modules, we assume you seek a comparison with other methods under identical settings. For clarity, we provide the following tables:\n|     LR    | 0.0001 | 0.0002 | 0.0003 | 0.0005 |\n|---------|--------|--------|--------|--------|\n| READ    | 35.4   | 35.8   | 36.0   | 34.8   |\n| Ours    | **40.8**   | **41.5**   | **40.9**   | **40.6**   |\n\n|    Params     | W\\_QKV | W\\_LN         | W\\_WKV&LN      |\n|---------|--------|---------------|----------------|\n| READ    | 35.8   | Out of Memory | Out of Memory  |\n| Ours    | **41.5**   | **39.2**          | **41.8**           |\n\nWe have the following observations: (1) our method consistently outperforms READ under the same learning rates and updating modules; (2) our method is memory-efficient, requiring significantly less GPU memory compared to READ.\n\n$\\textbf{W7}$: Add experiments on CIFAR-C and ImageNet-C.\n\n$\\textbf{A7}$: Thank you for your constructive and thoughtful feedback. Follow your advice, we run experiments on single-modal datasets, such as CIFAR10/100-C and ImgeNet-C using CNN-based backbones. We report the average accuracy over 15 types of corruptions for  these benchmarks in the following table:\n\n| Method | CIFAR10-C | CIFAR100-C | ImageNet-C  |\n|--------|-------|--------|-------|\n| Source | 56.5 | 53.6  | 18.0 |\n| TENT   | 81.7 | 68.5  | 34.7 |\n| ETA    | **81.9** | **68.9**  | 39.7 |\n| READ   | 79.0 | 61.5  | 31.3 |\n| Ours   | 81.3 | 68.8 | **41.3** |\n\nWe observe that our method outperforms READ on single-modal datasets and matches specialized single-modal methods (surpassing on ImageNet-C). Crucially, it excels over all SOTAs in challenging multi-modal domain shifts (see Tables 1-2).\n\nWe also run experiments on CLIP (ViT-B-16). To simulate multi-modal domain shifts, we use ImageNet-C for the image branch and apply Gaussian Noise to the text branch. For your convenience, we attach the performance below:  \n\n|  | Gauss. | Shot  | Impul. | Defoc. | Glass | Motion | Zoom  | Snow  | Frost | Fog   | Brit. | Contr. | Elastic | Pixel | JPEG  | AVG   |\n|--------|--------|-------|--------|--------|-------|--------|-------|-------|-------|-------|-------|--------|---------|-------|-------|-------|\n| Source | 10.7   | 11.7  | 11.3   | 22.1   | 14.6  | 23.5   | 21.2  | 30.4  | 29.1  | 33.6  | 51.8  | 16.1   | 12.4    | 29.5  | 30.8  | 23.3  |\n| TENT   | 5.5    | 4.4   | 6.7    | 23.8   | 18.5  | 26.4   | 23.4  | 32.0  | 30.2  | 36.1  | 52.9  | 21.6   | 13.2    | 33.6  | 34.7  | 24.2  |\n| ETA    | 18.1   | 19.2  | 19.7   | **25.1**   | 21.7  | 28.7   | 25.7  | 34.5  | 31.7  | 38.0  | 53.9  | 25.8   | 17.1    | 36.3  | 36.7  | 28.8  |\n| READ   | 15.4   | 14.4  | 12.4   | 21.8   | 14.0  | 23.2   | 20.9  | 30.5  | 29.3  | 33.7  | 52.0  | 15.8   | 12.3    | 29.5  | 30.7  | 23.7  |\n| Ours   | **20.6**   | **22.3**  | **20.9**   | 24.5   | **25.0**  | **30.5**   | **27.7**  | **35.4**  | **32.3**  | **40.4**  | **54.5**  | **30.2**   | **26.4**    | **38.2**  | **38.6**  | **31.2**  |\n\nIn summary, these additional experiments further demonstrate a broader applicability of our method."}, {"review_id": "FbvHsUAgtW", "reviewer": "Reviewer_Y3nk", "summary": "This paper highlights the limitations of using confidence or entropy-based measures for selective test-time adaptation, particularly in multi-modal domains. To address this, the authors propose a novel Partition then Adapt (PTA) strategy, which first clusters data before applying adaptation, aiming to improve robustness and reliability.", "questions": "Please see weaknesses and questions section.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n- The paper addresses an important and practical issue that can arise in multi-modal domains, where traditional confidence-based measures may fail.\n- The proposed method leverages attention mechanisms effectively to improve adaptation.\n- The ablation study is thorough and helps clarify the contribution of each component.\n- Extensive experiments across multiple datasets support the efficacy of the proposed approach.\n\n**Weaknesses and Questions**\n\n- The proposed method, PTA, quantifies prediction bias based on prediction frequency. Specifically, according to Eq. (1), if a sample's predicted label has a higher frequency than the average frequency of predicted labels within the batch, the sample is considered a biased (i.e., unreliable) one. While the empirical results based on this partitioning are convincing, it would be helpful to include a higher-level explanation or intuition for why partitioning reliable and unreliable samples based on prediction frequency should work. Could the authors provide supporting evidence or cite prior work that inspired this approach?", "was_revised": true, "final_justification": "The authors addressed the raised concern. I am keeping my score.", "rebuttal": "### Response to weaknesses and questions\n\n$\\textbf{WQ1}$: Supporting evidence.\n\n$\\textbf{A1}$: Thank you for the valuable suggestion. The intuition behind our use of prediction frequency stems from the well-documented effect of entropy minimization under domain shifts: it tends to amplify spurious confident predictions, often leading to a collapsed solution where only a few classes dominate [20]. This is especially problematic in multi-modal settings, where noisy modalities can further reinforce such bias.\nTo address this, we quantify prediction bias using the frequency of predicted labels within each batch. Samples assigned to dominant classes, i.e., those predicted more frequently than the batch average, are likely influenced by such bias. \nFig. 3 (c) and (d) demonstrate that our design effectively mitigates prediction bias in multi-modal domain shifts, significantly reducing false positive rates for the two most biased classes (24.2 $\\rightarrow$ 5.7 and 24.0 $\\rightarrow$ 6.7).\nMoreover, our empirical results in Fig. 5 (e) confirm that using batch-average frequency as the bias threshold outperforms static baselines.\nTo the best of our knowledge, we are the first to propose prediction frequency as a bias indicator for partitioning reliable/unreliable samples in the MMTTA setting. We will add this discussion and cite relevant works in the final version."}, {"review_id": "O5NAnGtO6O", "reviewer": "Reviewer_Jc1F", "summary": "In this paper, the authors address the problem of test-time adaptation (TTA) under multi-modal domain shifts, specifically in the context of video and audio co-classification using pre-trained video and audio encoders. While prior TTA methods for multi-modal settings primarily focused on single-modality domain shifts, the authors propose a novel method, called Partition-Then-Adapt (PTA), to tackle domain shifts in both modalities simultaneously.\n\nPTA consists of two key components:\n- Partition and Debiased Reweighting (PDR), which identifies reliable and unreliable samples in a test batch by quantifying prediction bias—measured as the frequency of each predicted label relative to the batch average. Samples with lower bias are considered reliable, while those with higher bias are considered unreliable. A quantile-based reweighting strategy is applied to focused on reliable samples and remove effect of unreliable ones.\n- Attention-Guided Alignment (AGA), which aligns the attention maps of unreliable samples with those of reliable ones by minimizing the maximum mean discrepancy (MMD) between their attention distributions.\n\nExperiments on Kinetics50-C and VGGSound-C demonstrate the effectiveness of the proposed method, showing consistent improvements over existing TTA approaches.", "questions": "- Why does the SuMi method fail to adapt the pre-trained model stably across all benchmarks?\nAccording to Tables 6 and 7, SuMi not only performs inconsistently under multi-modal domain shifts but also fails to improve performance under single-modal shifts. An in-depth analysis of SuMi's failure modes would benefit readers.\n\n- Why do multi-modal domain shifts degrade model calibration more severely than single-modal shifts, as shown in Figure 1(c)?\nThis observation is central to the motivation of PTA but is not sufficiently explained. Additional insights into why the interaction between modalities causes such a pronounced miscalibration—possibly due to conflicting noise patterns or misaligned feature distributions—would strengthen the justification for the proposed method.\n\n- Why is the proposed method still effective under single-domain shifts?\nWhile PTA is designed for multi-modal shifts, the results suggest that it remains effective in single-modality scenarios. A deeper explanation is warranted: for instance, does prediction bias still emerge due to label imbalance or temporal inconsistencies even in single-domain cases? Clarifying this would demonstrate the generality of the framework.\n\n- Can the proposed method outperform existing single-modality TTA methods on standard benchmarks such as CIFAR-C or ImageNet-C?\nIf so, a comparison table on these datasets would help illustrate the broader applicability of PDR and AGA, even beyond multi-modal settings. However, due to time constraints, this is a minor suggestion.\n\n- Can the proposed method also be applied to other multi-modal classification models?", "limitations": "Authors discuss the limitations in Appendix. However, I encouraged authors to fix the following points to improve the quality of the paper.\n\n1. I suggest the authors include the efficiency of the Source (i.e., the original pre-trained model without adaptation) in Figure 6. Since TENT involves only a few trainable parameters, I assume the Source would be slightly faster. However, readers may want a direct comparison of the efficiency (e.g., latency or parameter count) between the Source model and various TTA methods. This is a minor suggestion for improving clarity.\n\n2. Additionally, I recommend that the authors explicitly specify the unit for the number of trainable parameters shown in Figure 6. I assume it is in millions, but making this clear in the figure caption or axis label would avoid confusion. Moreover, in Tables 6 and 7, the authors only highlight (bold) their own method's results. To provide a fair comparison and improve readability, it would be better to bold the best values across all methods, not just the proposed one.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "### Strengths\n- The paper is well-structured and easy to follow.\n- The authors address the problem of multi-modal domain shifts, which has been underexplored in the literature.\n- The proposed method is well-motivated, and the Attention-Guided Alignment (AGA) component effectively mitigates the issue of uniform attention maps that can arise when PDR is used for adaptation.\n\n### Weaknesses\n- I believe that the method may require a large batch size to accurately estimate prediction bias and perform effective partitioning, which could limit its applicability in low-latency or memory-constrained environments.\n- The key assumption—that multi-modal domain shifts lead to performance drops even for high-confidence predictions (as shown in Figure 1(c))—is only validated using CAV-MAE. The generality of this observation across other model architectures is not validated.", "was_revised": true, "final_justification": "The authors' rebuttal has adequately addressed the major concerns raised in the initial review. Specifically, they demonstrate that PTA performs well across different architectures, including CLIP. Additionally, they show that PTA is robust to small batch sizes, which is important for real-world applications. Based on these clarifications and results, I am raising my initial rating to accept.", "rebuttal": "### Response to weaknesses\n\n$\\textbf{W1}$: Applicability to different batch sizes.\n\n$\\textbf{A1}$: Thank you for raising this important point. We agree that small batch sizes are common in real-world, low-latency scenarios and must be addressed. To this end, we evaluate our method under various batch sizes in Fig. 8 (Appendix D, Page 14). Results show that PTA remains robust and consistently outperforms SOTAs even with small batches. To mitigate bias estimation instability, we introduce a memory queue that stores historical predictions for computing Eq. (1) and (2). This allows more stable frequency estimation without requiring large batch sizes. This design ensures that PTA remains practical and effective under constrained settings.\n\n$\\textbf{W2}$: Clarification of some assumptions.\n\n$\\textbf{A2}$: Fig. 3 (c) presents an example illustrating that high-confidence predictions can become unreliable under multi-modal domain shifts. We adopt CAV-MAE in Fig. 3 (c) to ensure a fair comparison, as it is also used by previous methods [9, 31, 38]. The results clearly show that the positive rate of high-confidence predictions drops significantly under such domain shifts. Moreover, the empirical results reveals that addressing this prediction bias issue substantially improves adaptation performance  (see Tables 1-3). For results obtained with other model architectures, please kindly refer to Q/A5. Additional figures validating this observation with other model architectures will be included in the Appendix.\n\n### Response to questions\n\n$\\textbf{Q1}$: In-depth analysis of SuMi.\n\n$\\textbf{A1}$: Thank you for your constructive feedback. For fair comparison, we reimplement all comparison methods using their official code (if any) and report the average accuracy across 5 independent seeds under both single- and multi-modal shifts. SuMi outperforms the pre-trained model (Source) under single-modal domain shifts (61.01 vs. 60.58 in Table 6, and 69.88 vs. 69.32 in Table 7), but fails under multi-modal domain shifts (see Tables 1–2). The reason might originates from SuMi’s design. Specifically, the key components of SuMi are (1) uni-modal assistance (UA) with entropy-based reweighing (EW) and (2) KL-based mutual information sharing. \n\nFirst, as we demonstrate in the empirical results (Lines 239-248) and the analysis (Lines  249-260 and Fig. 3 (b)), UA+EW assigns high weights to the selected candidates, regardless of whether they are false positives or not, which may poses error accumulation. Second, the KL-based loss function may not provide meaningful mutual information under severe multi-modal domain shifts, as significantly performance drop of the pre-trained model (Fig. 1 (a)) occurs. As a result, the above properties could be the reason why SuMi achieves unsatisfactory performance for MMTTA. We will add the above discussion in Section 4.\n\n$\\textbf{Q2}$: Degradation of the pre-trained model.\n\n$\\textbf{A2}$: Thank you for your insightful suggestion. We agree with your hypothesis that multi-modal domain shifts degrade model calibration more severely than single-modal shifts, likely due to conflicting noise patterns or misaligned feature distributions across modalities. Multi-modal domain shifts degrade model calibration primarily because they induce complex interactions between modalities in the fusion block, leading to degraded discrimination of fused features (Fig. 1 (a)). These skewed feature representations bias the classifier toward a few dominant classes (Fig. 1 (b)). Consequently, the normalized entropy, computed from these biased predictions, significantly impairs model calibration (Fig. 1 (c)). We will improve the presentation in Lines 31-34 with the above discussion.\n\n$\\textbf{Q3}$: Explain the effectiveness of PTA under single-domain shfits.\n\n$\\textbf{A3}$: Thank you for your thoughtful comment. We agree that PTA effectively addresses prediction bias in single-modal domain shifts, though the issue is less pronounced than in multi-modal shifts. First, our experiments show that the pre-trained model’s performance degrades under both single- and multi-modal domain shifts (Fig. 1 (a)), leading to increasingly noisy (incorrect) predictions. Entropy minimization exacerbates this by sharpening low-entropy predictions, regardless of their reliability, which propagates noise and risks a collapsed trivial solution. Empirical results from Tent (Tables 1–2 and Table 6) confirm that excessive entropy minimization with noisy predictions can degrade performance compared to the pre-trained model (Source). Second, following prior works [9, 31, 38], we use randomly sampled batch data, ensuring no intentional label imbalance. This indicates that prediction bias primarily arises from noisy predictions, further amplified by entropy minimization, rather than imbalanced data.\n\n$\\textbf{Q4}$: Add experiments on CIFAR10/100-C and ImageNet-C.\n\n$\\textbf{A4}$: Thanks for your suggestion. We provide more experiments on CIFAR-10/100-C and ImageNet-C and report the average accuracy across 15 types of corruptions (severity level 5).  \n\n| Method | CIFAR10-C | CIFAR100-C | ImageNet-C  |\n|--------|-------|--------|-------|\n| Source | 56.5 | 53.6  | 18.0 |\n| TENT   | 81.7 | 68.5  | 34.7 |\n| ETA    | **81.9** | **68.9**  | 39.7 |\n| READ   | 79.0 | 61.5  | 31.3 |\n| Ours   | 81.3 | 68.8 | **41.3** |\n\nWe observe that our method outperforms READ on single-modal datasets and matches specialized single-modal methods (surpassing on IN-C). Crucially, it excels over all SOTAs in challenging multi-modal domain shifts (see Tables 1–2). We will add the comparison table on these datasets to help illustrate the broader applicability of our method following your advice.\n\n$\\textbf{Q5}$: Performance on other multi-modal classification models.\n\n$\\textbf{A5}$: Thank you for your valuable feedback. First, we validate our method’s effectiveness on synthetic multi-modal domain shifts (see Tables 1-2 and Tables 4-7) using CAV-MAE [8, 31] as the backbone. Second, we demonstrate our method's generalization to real-world domain shifts using a Transformer-based multi-modal classification model [10] as the backbone (see Table 3). Third, we prove our method's broader applicability using CLIP (ViT-B-16) as the backbone. To simulate multi-modal domain shifts, we use ImageNet-C for the CLIP's image branch and apply Gaussian Noise to its text branch.  We report the comparison table below:\n\n|  | Gauss. | Shot  | Impul. | Defoc. | Glass | Motion | Zoom  | Snow  | Frost | Fog   | Brit. | Contr. | Elastic | Pixel | JPEG  | AVG   |\n|--------|--------|-------|--------|--------|-------|--------|-------|-------|-------|-------|-------|--------|---------|-------|-------|-------|\n| Source | 10.7   | 11.7  | 11.3   | 22.1   | 14.6  | 23.5   | 21.2  | 30.4  | 29.1  | 33.6  | 51.8  | 16.1   | 12.4    | 29.5  | 30.8  | 23.3  |\n| TENT   | 5.5    | 4.4   | 6.7    | 23.8   | 18.5  | 26.4   | 23.4  | 32.0  | 30.2  | 36.1  | 52.9  | 21.6   | 13.2    | 33.6  | 34.7  | 24.2  |\n| ETA    | 18.1   | 19.2  | 19.7   | **25.1**   | 21.7  | 28.7   | 25.7  | 34.5  | 31.7  | 38.0  | 53.9  | 25.8   | 17.1    | 36.3  | 36.7  | 28.8  |\n| READ   | 15.4   | 14.4  | 12.4   | 21.8   | 14.0  | 23.2   | 20.9  | 30.5  | 29.3  | 33.7  | 52.0  | 15.8   | 12.3    | 29.5  | 30.7  | 23.7  |\n| Ours   | **20.6**   | **22.3**  | **20.9**   | 24.5   | **25.0**  | **30.5**   | **27.7**  | **35.4**  | **32.3**  | **40.4**  | **54.5**  | **30.2**   | **26.4**    | **38.2**  | **38.6**  | **31.2**  |\n\nIn summary, out method demonstrate its robustness on multiple multi-modal classification models.\n\n### Response to limitations\n\n$\\textbf{L1}$: Improve Fig. 6 by including the efficiency of the Source.\n\n$\\textbf{A1}$: We appreciate your feedback. The number of samples processed per second (samples/sec) by Source is 92.4.\nWe will revise Fig. 6 by including the Source in the comparison to improve clarity, following your advice.\n\n$\\textbf{L2}$: Improve Fig. 6, and Tables 6-7.\n\n$\\textbf{A2}$:  We appreciate your valuable suggestions. The unit for the number of trainable parameters in Fig. 6 is indeed millions (M). We will revise Fig. 6 by adding the unit to the y-axis for the number of trainable parameters and improve Tables 6-7 by highlighting the best-performence of comparison methods, following your advice."}, {"review_id": "XtG0T0cmgS", "reviewer": "Reviewer_UQ6f", "summary": "This paper tackles the under-explored problem of multi-modal test-time adaptation (MM-TTA) when all modalities simultaneously undergo domain shift. To mitigate prediction bias under this setting, the authors propose Partition-Then-Adapt (PTA),  which contains two key components: Partition and  Debiased Re-weighting (PDR) and Attention-Guided Alignment (AGA) . Extensive experiment show that PTA achieves good performance.", "questions": "Please refer to Weaknesses.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 1, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. This paper raises a meaningful question: how to design a test-time adaptation (TTA) scheme that mitigates domain shift when multiple modalities are involved. The authors observe that existing MM-TTA approaches focus only on single-modal domain shifts within multi-modal tasks; when all modalities are simultaneously shifted, performance degrades markedly. Hence, investigating how to develop an MM-TTA method for this more challenging setting is both timely and significant.\n\n2. The paper is well organised,  clear writing, and figures/tables support the narrative effectively.\n\n3. The authors benchmark PTA against MM-TTA methods, report clear SOTA gains, and conduct ablations on PDR, AGA, weighting strategies, and hyper-parameters, which help the reader understand each design choice.\n\n\nWeaknesses:\n1. Is “high class frequency ⇒ high bias” always valid? In PDR, samples whose predicted class appears more frequently than the batch mea  are deemed more biased. Yet class frequency can be high simply due to small batch size or genuine class imbalance in the target domain.  Please  provide results for PTA’s sensitivity to different batch sizes.\n\n2. Possible misconception in the MMD explanation (§3.3) \n    Minimising the squared MMD in Eq. (4) actually reducesthe distance between $A^+$ and $A^-$, i.e., makes the two distributions more similar.\n   However, the text claims the goal is to “push them apart”. Could the authors reconcile this apparent contradiction or correct the description?\n\n3. Definition of the confidence term $\\mathcal K$\n  Are they the maximum Softmax probabilities of the current classifier? Please add a precise definition.", "was_revised": true, "final_justification": "Based on the author's response and the comments of other reviewers, I gave the final score.", "rebuttal": "### Response to weaknesses\n\n$\\textbf{W1}$: Applicability to different batch sizes.\n\n$\\textbf{A1}$: Thank you for your insightful comment. We agree this is an important question as online data is often insufficient, and at times, only a limited amount is available. And addressing this problem would firmly validate the applicability of our method dealing with real-world scenarios. In fact, we provide results for PTA’s applicability to different batch sizes in Fig. 8, Appendix D, Page 14, where small batch size has been also discussed. To handle extremely small batch sizes, we implement a memory queue to store historical predictions for computing Eq. (1) and (2), effectively adapting the original computation method for offline use. Therefore, PTA still works well and consistently outperform SOTAs in such cases.\n\n$\\textbf{W2}$: The meaning of the third term in Eq. (4).\n\n$\\textbf{A2}$: Thank you for your helpful review. We acknowledge the typo in Line 174 —  \"penalizes\" should in fact be \"promotes\". To avoid any misunderstanding, we will reframe Lines 174-175 as ''The third term regularizes the model by guiding $\\mathcal{A}^-$ with $\\mathcal{A}^+$. Specifically, it aligns the distributions of $\\mathcal{A}^-$ with those of $\\mathcal{A}^+$.''\n\n$\\textbf{W3}$: Definition of the confidence term $\\mathcal{K}$.\n\n$\\textbf{A3}$: Thank you for your valuable feedback. We confirm that the confidence term ( $\\mathcal{K}$ ) represents the maximum softmax probabilities of the current classifier for a given batch data. We will clarify this definition in Lines 133-134 as ''Let the prediction confidence of $\\mathcal{X}$ be denoted as $\\mathcal{K}(\\mathcal{X}) = \\max(\\text{softmax}(\\mathbf p(\\mathcal{X})))$, which corresponds to the highest softmax probability produced by the current classifier for the input batch $\\mathcal{X}$.''"}]

Author rebuttal:

Meta-review: All reviewers support the acceptance of this paper. After a careful reading of the manuscript, the detailed rebuttal, and all reviewer comments, I agree with their overall assessment. The paper addresses an important and under-explored problem, multi-modal test-time adaptation under simultaneous domain shifts, and proposes a novel and well-motivated method (PTA) that is both empirically effective and theoretically grounded. The authors provide extensive experiments, thorough ablations, and insightful rebuttals that satisfactorily resolve most reviewer concerns. I encourage the authors to further refine the camera-ready version by incorporating the reviewers’ constructive suggestions, especially regarding clarity, reproducibility, and additional implementation details. Overall, this is a solid contribution to the area of test-time adaptation and multi-modal learning.
Decision: Accept (spotlight)