Test-time Adaptation against Multi-modal Reliability Bias
Authors: Mouxing Yang
Yunfan Li
Changqing Zhang
Peng Hu
Xi Peng
Venue: ICLR 2024 poster (2024)
Keywords: Test-time adaption
Imbalanced multi-modal learning
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: Reveal a new problem named reliability bias for multi-modal TTA, and propose a new method to achieve reliable fusion and robust adaption.

Abstract: Test-time adaptation (TTA) has emerged as a new paradigm for reconciling distribution shifts across domains without accessing source data. However, existing TTA methods mainly concentrate on uni-modal tasks, overlooking the complexity of multi-modal scenarios.
In this paper, we delve into the multi-modal test-time adaptation and reveal a new challenge named reliability bias. Different from the definition of traditional distribution shifts, reliability bias refers to the information discrepancies across different modalities derived from intra-modal distribution shifts. To solve the challenge, we propose a novel method, dubbed REliable fusion and robust ADaptation (READ). On the one hand, unlike the existing TTA paradigm that mainly repurposes the normalization layers, READ employs a new paradigm that modulates the attention between modalities in a self-adaptive way, supporting reliable fusion against reliability bias. On the other hand, READ adopts a novel objective function for robust multi-modal adaptation, where the contributions of confident predictions could be amplified and the negative impacts of noisy predictions could be mitigated. Moreover, we introduce two new benchmarks to facilitate comprehensive evaluations of multi-modal TTA under reliability bias. Extensive experiments on the benchmarks verify the effectiveness of our method against multi-modal reliability bias. The code and benchmarks are available at https://github.com/XLearning-SCU/2024-ICLR-READ.

Published as a conference paper at ICLR 2024
TEST-TIME ADAPTATION AGAINST MULTI-MODAL RELIABILITY BIAS
Mouxing Yang'
Yunfan Li'
Changqing Zhang?
Peng Hu'
Xi Peng
Sichuan University! Tianjin University (yangmouxing, yunfanli. gm, @angchhangging@:]1 jud.eeee.......uuuu.u.......
gm, penghu. fiighu.ml, ml, pengx. www.ggggggmmmm.mmmmmmm com,
ABSTRACT
Test-time adaptation (TTA) has emerged as new paradigm for reconciling dis- tribution shifts across domains without accessing source data. However, existing TTA methods mainly concentrate uni-modal tasks, overlooking the complexity of multi-modal scenarios. In this paper, we delve into the multi-modal test-time adaptation and reveal new challenge named reliability bias. Different from the definition of traditional distribution shifts, reliability bias refers to the informa- tion discrepancies across different modalities derived from intra-modal distribu- tion shifts. To solve the challenge. we propose a novel method, dubbed REli- able fusion and robust ADaptation (READ). On the one hand, unlike the existing TTA paradigm that mainly repurposes the normalization layers, READ employs a new paradigm that modulates the attention between modalities in self-adaptive way, supporting reliable fusion against reliability bias. On the other hand, READ adopts a novel objective function for robust multi-modal adaptation, where the contributions of confident predictions could be amplified and the negative impacts of noisy predictions could be mitigated. Moreover, we introduce two new bench- marks to facilitate comprehensive evaluations of multi-mode TTA under reliabil- ity bias. Extensive experiments on the benchmarks verify the effectiveness of our method against multi-modal reliability bias. The code and benchmarks are avail- able at https:://// / thub. com/ XLea rn -nnnnngSCUU 2024-02222222 - CLR-READ.
INTRODUCTION
Multi-modal pre-trained models (Radford et al., 2021; Girdhar al., 2023; Li al., 2023; Lin al., 2024) have shown great potential in various applications, being research focuses in both academic and industrial communities. After acquiring common knowledge from the source domain, pre- trained models could be customized into specific tasks through the attention mechanism ( (Vaswani al., 2017; Gong et al., 2023) that integrates knowledge from different modalities in the target domain. Although such a paradigm has achieved promising performance, its success heavily relies on the identical distribution between the source domain and target/test domain (Chen et al., 2023). However, as shown in Fig. 1(a), it is daunting to meet such a mild assumption, especially in open- world scenarios with unpredictable factors such as the changing weather (e.g. fog) and degenerated sensors (e.g. defocus) would lead to the distribution shifts (Hendrycks & Dietterich, 2019).
Toward achieving robustness against distribution shifts, numerous test-time adaptation (TTA) meth- ods have been proposed (Wang et al., 2021; Yu et al., 2023a; Niu et al., 2022). Most of them work by updating parameters of normalization layers in the source model (loffe & Szegedy, 2015; Ba al., 2016), hoping bridge the gaps between domains (Schneider al., 2020; Zhang al., 2022; Nado et al., 2020; Hu et al., 2021; Iwasawa & Matsuo, 2021). To this end, they usually minimize the entropy-based objective on the model predictions of unlabeled test samples. Despite the signif- icant success, almost all existing TTA methods are devoted to handling distribution shifts between domains while ignoring specific challenges multi-modal learning scenarios. Specifically, once some modalities are contaminated with distribution shifts, the information discrepancies between modalities would be enlarged, leading to reliability bias across modalities. For example, as shown
"Corresponding author.

---

Published as a conference paper at ICLR 2024
Camera
Foggy Highway
Night
Video
Audio
(c) Performance Degradation
Noisy Crossroad
Micraphone
Thudner
Growd
Wind
Video
Audio
(a) Distribution Shift
(b) Multi-modal Reliability Bias
(d) Unstable Entropy
Figure 1: Our observations. (a) Distribution shift: some sensors of autonomous vehicles might encounter different situations in the wild, leading to domain shifts in certain modalities. (b) Multi- modal reliability bias: due to the distribution shifts, some corrupted modalities will lose the task- specific information and suffer from reliability bias during cross-modal fusion compared to the un- corrupted counterparts. (c) Performance degradation: the video modality contaminated with relia- bility bias (Video-C) has poor recognization accuracy compared to the audio modality. Both vanilla attention-based fusion (AF) and late fusion (LF) manner give inaccurate predictions compared to the single-modality ones. Instead, the proposed self-adaptive attention-based fusion (SAF) could achieve reliable fusion thus guaranteeing the performance gain in multi-modal scenarios. (d) Unsta- ble entropy: once the more informative modalities are corrupted (e.g. video for action recognition), would be challenged to give accurate predictions. Consequently, the entropy of multi-modal pre- dictions would be unstable. In other words, the ratio of confident predictions would decrease while the noise might dominate the predictions. All results in the figures are from experiments conducted on subset of the Kinetics dataset (Kay et al., 2017) with foggy corruption in the video modality.
in Fig. 1(b), when an autonomous vehicle equipped with camera and audio sensors drives into foggy highway or noisy crossroad, either the visual or audio modalities would be corrupted. As result, the reliability balance across the modalities will destroyed, and the performance of the model would heavily degrade if each modality is equally treated as depicted in Fig. 1(c). Although some studies (Zhang et al., 2023; Peng et al., 2022) have been conducted toward imbalanced multi- modal learning, they mainly focus altering the training process with the labeled samples in the source domain, rather than adapting biased modalities during the test time.
Based on the above observations, in this paper, reveal adaptation, i.e., reliability bias. Different from the distribution shifts between domains, reliability bias refers to the information discrepancies across different modalities derived from the intra-modal distribution shifts. should be pointed out that is intractable to conquer the reliability bias problem using the existing TTA methods (Niu al., 2023; Shin et al., 2022) due the following reasons. First, it is impossible to completely reconcile the distribution shifts through updating the parameters of normalization layers. As result, it is inevitable to introduce reliability bias across modalities. Second, as shown in Fig. ((d). in the multi-modal scenarios, once the representative modality is corrupted, noisy predictions would dominate the adaptation process. As a result, simply minimizing the entropy on all predictions or only confident predictions might either lead to model overfitting or underfitting on the test data. To support our claims, we provide some empirical results in Section 4.2 and Fig. 3.
problem for multi-modal test-time
To achieve reliability-bias robust multi-modal TTA, we propose novel method, dubbed REliable fusion And robust ADaptation (READ). READ handles the reliability bias challenge by resorting to the following two-fold modules. On the one hand, instead of reconciling the intra-modality distri- bution shifts through repurposing normalization layers, propose modulating the attention-basse based fusion layers in addativee manner for reliable cross-man cross-modal fusion during test time. On the other hand, design novel objective function for robust multi-modal adaptation. In short, the objec-
2

---

Published as a conference paper at ICLR 2024
tive function can not only amplify the contributions of confident predictions but also prevent noisy predictions from dominating the adaptation process.
The major contributions and novelties of this work could be summarized as follows:
1. We reveal a new challenge for multi-modal test-time adaptation, i.e., reliability bias. In a word, reliability bias refers to the information discrepancies across different modalities, derived from the distribution shifts between domains.
2. To enjoy robustness against reliability bias, we propose a novel method named READ. Unlike most existing TTA methods that reconcile the distribution shifts by repurposing the normalization layers, READ achieves reliable fusion and robust adaptation by modulating the attention- based fusion layers in a self-adaptive manner under the support of novel objective function.
3. We provide two benchmarks amulti-modie action recognition and event classification) for multi-modal TTA with reliability bias. Extensive experiments on the benchmarks not only verify the effectiveness of our method but also give some observations for the community.
RELATED WORK
In this section, we briefly review some related topics to this work, ie., test-time adaptation, and imbalanced multi-modal learning.
2.1 TEST-TIME ADAPTATION
Test-time adaptation aims at bridging the gaps between source and target domains during test time without accessing the source data. Toward this goal, some test-time training methods (Liu et al., 2021; Sun et al., 2020) have been proposed, which additionally add self-supervised task in the training process. As a result, the source model could be adapted by performing the self-supervised task on test samples. Such paradigm needs to alter the training process and might be limited in the pre-trained model era. To remedy this, the fully test-time adaptation paradigm has emerged and plenty of methods have been proposed recent years, which could be roughly divided into the following categories. i) online TTA methods (Wang et al., 2021; Gao et al., 2023), which updates the specific model parameters (always the normalization layers) with the coming test samples by resorting to some unsupervised objectives such as entropy minimization on predictions. ii) robust TTA methods (Niu et al., 2023; Zhou et al., 2023), which considers some challenging and practical adaptation settings such label shifts, single sample, mixed domain shifts, etc. iii) Continual TTA methods (Gan et al., 2023; Wang et al., 2022) which aims to solve the continual and changing shifts along test time. iv) TTA beyond recognition (Shin et al., 2022; Lee et al., 2023), which focuses on applications beyond image classification such as multi-modal segmentation, pose estimation.
In this paper, focus online TTA and aim to achieve multi-modal test-time adaptation against modality reliability bias. Among the existing TTA studies, MM-TTA (Shin et al., 2022) might be most relevant to our work, while having the following main differences. i) Problem/motivaaiion dif- ferences. MM-TTA focuses on reconciling the distribution shifts between domains for 2D-3D joint segmentation tasks. In contrast, this work aims to handle the modality reliability bias challenge overlooked by the existing studies and validate the necessity and effectiveness in multi-modal sce- narios including audio-video event classification, and action recognition. ii) Approach/paradigm differences. MM-TTA achieves adaptation by updating the normalization layer like most existing TTA methods, while this work proposes modulating the attention-based fusion layers a self- adaptive way. iii) Objective function differences. MM-TTA adopts noise-filter cross-entropy loss whose pscudo labels are selected based on a set of slow-fast models. In contrast, we design a novel confidence- -wafid objective function, which would not only benefit the model optimization by ex- ploiting the confident predictions but also hinder the model from overfitting noise.
2.2
IMBALANCED MULTI-MODAL LEARNING
learning has emerged promising avenue for understanding the world, encompass- ing various tasks such as recognition, clustering, and retrieval across diverse views, media, or do- mains (Lin al., 2021; Yang al., 2022; 2021). Some recent studies have found that multi-modal
3

---

Published as a conference paper at ICLR 2024
Loss Gure
Image Encoder
06
Altunion
Layar Norm
Feed Forword
Layer Norm
Ciocoalfee
Wlng
Corrupted
Audio Encoder
+
Clean Audio
Salt Adoptive Attention
Confidenco Aware Loss
The pipeline of our method (READ). On the adaptation process, biased modality (e.g. corrupted video) and unbiased modality (e.g. clean audio) are input into two modality-specifie encoders, and the output embeddings are concatenated token-level for fusion. During the cross- modal fusion, the attention is calculated between the token embeddings in self-adaptive manner and severed as the reliability for fusion. After that, the multi-modal predictions are obtained by the classifier with the fused embeddings. The confidence-aware loss will amplify the contributions of the high-confident predictions (p > ?) by increasing the gradients and alleviate the influence of Low-confident predictions < 7) by reversing the gradients. Both the amplification and alleviation effects are proportional to the confidences of predictions.
Figure 2:
learning might not achieve better performance compared to the single-modal counterparts (Peng et al., 2022; Wang et al., 2020; Fan et al., 2023; Du et al., 2021; Wei et al., 2022). The essence behind the problem could be boiled down to the discrepancy/imballnce between modalities. Wang et al. (2020) observes that different modalities have various convergence rates. Motivated by the observation, Peng et al. (2022) makes deep analysis and finds that some modalities embrace more task-specific information under certain scenarios, e.g., audio for multi-modal sound localization. As a result, the more informative modalities might dominate the learning process, thus hindering the fitting of other modalities if each modality is equally during the optimization. Toward guarantee- ing the performance gain of multi-modality learning, some works have delved into learning with imbalanced multi-modal data. Wang al. (2020) uses auxiliary networks penalize the modali- ties by considering their overfitting behaviors, leading to a better fusion between modalities. Peng al. (2022) designs meadient-modulatee ent modalities according to their contributions to the network optimization. From the perspective of uncertainty learning, Zhang et (2023) proposes robust multimodal fusion method with theoret- ical guarantees, which estimates the certainty of each modality and accordingly achieves weighty cross-modal fusion.
strategy that adaptively adjusts the gradients for differ-
In this paper, we focus on the test-time reliable fusion across modalities, and the setting is signifi- cantly different from the existing imbalanced multi-modal learning studies. Specifically, the existing studies focus on learning with unbalanced modalities under the labeled source domain, while this work aims to adapt the source model with unlabeled multi-modal test pairs during test time.
3 METHOD: RELIABLE FUSION AND ROBUST ADAPTATION
In this section, elaborate the proposed method dubbed reliable fusion and robust adaptation (READ) for multi-modal test-time adaptation against reliability bias. As shown in Fig. 2, READ consists of the self-adaptive attention module achieve reliable fusion across different modali- ties, and the confidence-aware loss function for robust test-time adaptation. In the following, we first present the used notations and definition of the multi-modal reliability bias problem in Sec- tion 3.1, then introduce the self-adaptive attention module in Section 3.2, and finally elaborate on the confidence-awe aware loss function in Section 3.3.
3.1 NOTATIONS AND PROBLEM FORMULATION
Without loss generality, we take two modalities as showcase for clarity of presentation. For clarity, we Fe, (Joy, Jey, Sor, )o,} denote the source model that trained on the labeled
4

---

Published as a conference paper at ICLR 2024
training set {sf yi k- where fø; and fe; the specific Transformer encoders for modality and v. fer and Ce, are the multi-modal fusion layer and the following classifier, and s consists of tokens (s/) ) (h E ()).)) During training, the source model Fe, would (over)fit the distribution of training data, i.e., P(s). As result, in the inference/testing stage, the performance of Fe, would heavily degrade once the distribution shifts emerge due to weather changes, sensor degeneration, etc. In other words, P(s) 7 P(x) where are the unlabeled test multi-modal data. Test-time adaptation (TTA) (Yu et al., 2023a; Wang et al., 2021) aims to quickly reconcile the shifts for coming test samples through online updating the parameter of F from θ, to θ during test-time. To this end, most of them minimize the objective function below:
min L (p),
where Lute is the loss function, ô C θ denotes the learnable parameters (usually BN or LN) of the adapted model Fe. is the predictions of Fe for test-time multi-modal pairs (x", xx",, ) i.e., p = Ce. (fe. (z",z") =)") where z fe (x") and z" = fe- (x"). Under the unlabeled setting, most TTA methods usually use the entropy minimization objective as L or design another variants.
Although existing TTA methods have achieved great success, most of them focus on single-mot modality tasks and cannot handle the reliability bias problem in multi-modas scenarios. Concretely, in the wild, the corrupted modalities will lose some specifce information compared to the other clean ones as discussed in Introduction. As result, the clean modalities will be more reliable than the corrupted ones during the cross-modal fusion, í.e., the modality reliability bias. In Experiment, we empirically validate that the reliability bias problem would significantly degrade the performance of existing TTA methods. Therefore, our goal becomes achieving cross-modal fusion on the reliability biased modalities and performing robust test-time adaptation.
3.2 RELIABLE FUSION
To fuse the information across modalities, one widely-used solution is the t---ffusoon-bbseeee man- ner (Peng et al., 2022; Zhang al., 2023). Mathematically, given the test-time embeddings (z", zz), the multi-modal predictions p" obtained via late fusion could be formulated as follows,
pd If (Cg (mean(z")) + Cg (mean(z")) /2.
(2)
where mean denotes the token-wise mean operation, and Cg and C. are two modality-specifff classifiers. Clearly, the late fusion manner equally treats each modality whether they are reliable or not, being sensitive to the reliability bias problem as depicted in Fig.
As remedy, we propose the self-adaptive attention module to dynamically integrate information from different modalities. Specifically, the modality-specific embeddings z" and z" are first con- catenated at token-level, then projected into the query, key, and value matrixes. More formally,
Q Wee ([z"; iz)) + B
K Wex (lz; )) + Bøx,
(3)
Wer ([z":z"]) + Bgv,
where Wos and Bas (h E (Q, K, V)) are the projector and bias term inherited from the source model and updated during the test time, and [::-] denote the token-level concatenate operation. After that, the attention map could be calculated as follows,
QKT
A Softmax
(4)
Vd
where the cell of A,, denote the similarity value between the r-th token and I-token from z (h E {a, v))), and d is the latent dimension of the tokens.
Note that, the vanilla attention mechanism (Vaswani et al., 2017: Gong et al., 2023) usually keeps the parameters inherited from the source model, and performs information integration across modali- ties. Apparently, the distribution shift between training and test-time data might hinder the similarity estimation between tokens. As result, reliable fusion on biased modalities cannot be guaranteed. Instead, we hope that the model could focus more on the unbiased modalities and avoid the interop- eration from the bias. To this end, we propose repurposing the cross-modal attention-basee fusion
5

---

Published as a conference paper at ICLR 2024
layers in self-adaptive way. In other words, the parameters of We and B (h (Q.K,V) would be updated in order to adapt the test-time distribution. Thanks to the attention modulation, the model would focus on the unbiased modalities, leading to reliable cross- fusion during test time. We empirically verify the above claims in Fig. 3.
Finally, the predictions obtained through our self-adaptive iitention-based fusion layer could be formulated as below,
p
Ce. (mean (AV)).
(5)
3.3 ROBUST ADAPTATION
After cross-modal fusion on modalities with reliability bias, the challenge becomes achieving robust adaptation against the distribution shifts in multi-modal scenarios. One feasible solution is adopting the widely-used entropy minimization objective on either all predictions (Wang et al., 2021) only some high-confident ones (Niu et al., 2023). However, as discussed in Introduction, once some informative modalities (e.g., visual modality for action recognition task) are corrupted, the overall task-specific information would greatly reduce. As result, the accuracy of the predictions from the source model might decrease. At this time, either the vanilla entropy minimization objective or the noise-filter one would overfit the noisy predictions or underfit clean predictions, leading to degraded adaptation effects as verified in Fig. (dd and Tables 1-3.
As remedy, we propose novel confidence-awarc loss function for robust adaptation. Formally, given a mini-batch test predictions of size B, the loss function is designed as below:
Lra
1
Σ Pi log
e7
(6)
ps
where p. is confidence of the prediction p. i.e., Pt = max (6 (p. is the softmax operation and y is threshold for confident prediction division fixed constant in all our experiments. In the following, we mathematically show that why the loss function could achieve robust adaptation. To begin with, first plot the loss performance curve for clarity. As shown in Fig. 2, the loss embraces the following merits.
Remark 1. Lra will reduce non-monotonously for different predictions. Consequently, the high- confident predictions (p: > ",, possible clean) will contribute to optimization while the influence of low-confident predictions (p, 7, possible noisy) will be eliminated. Meanwhile, contributions of the high-cow coef-cenddeet predictions will be amplified with the increasing confidences, while the negative impacts of the low-confidence ones will be reduced with the decreasing confidences.
The above properties of our loss could be supported by the following Theorems.
Theorem 1. The gradient direction produced by Lra is non-monotonous.
Theorem 2. The gradient value will rise with increasing ps iff. ps E (r,1) or decreasing ps iff.. P: E (0. 7)
Due to the space limitation, we remove the according proofs into the Appendix. Thanks to the favorable properties of Cra. the robust adaptation could be achieved. On the one hand, the model would not overfit the low-confident predictions, thus preventing the noise from dominating the adap- tation process. On the other hand, the model will focus more on the high-confident predictions, thus benefiting the optimization.
Combining both the self-adaptive attention module and the confidence-awand loss function together, we could obtain the final objective function for multi-modal test-time adaptation as follows,
min ((zat),
where 6 = 109, Ok @V) C O, L Lra + Coais and Loa! = Σ Ex= (ct) log (c)) is an negative entropy loss term to make the prediction balance following Yu et al. (2023b); Zhou et al. (2023) where dt EE 8 (pfaf) and K is the class number.

---

Published as a conference paper at ICLR 2024
Table I: Comparisons with SOTA methods on Kinetics50-C benchmark with corrupted video modality (severity level 5) regarding the accuracy (%) metric. "Stat." and "Dyn." are the abbre- viation of "Statical" and "Dynamic", while "LN", "LF", "AF" and "SAF" denotes the layer normal- ization, late fusion, attention- datention-based fusion, and self adaptive attention- fusion, respectively. The results are the mean values among 5 random sceds, and the best results are highlighted in bold.
Noese
Blur
Weather
Digital
Methods
Glauss Shot Ingal. Defoc.
Glass Mor. Zoom SSom Frost Fog Brit. Brit. Coon:
Elas. Pix.
IPEG Ang.
Source (1Star. LP
3188
33.4
31.7
64.0
62,5
61.9
50.9
44.0
60.2
61.7
56.4
52.2
MM-TTA (Dyn LN)
46.2
46.6
46.1
58.8
55.7
62.6
58.7
$2.6
54.4
48.5
69.
49.5
57.6
56.4
54.6
54.5
Tent (Dya. LN)
26.6
29.8
63.4
61.7
46.5
51.
38.6
60.7
61.8
49.4
EATA (Dyn. LN)
31.8
33.3
31.6
6422
67.7
62:2
$1.3
54.7
38.1
72.5
44.2
60.4
62.0
52.4
SAR (Dyn. LN)
$1.9
33.3
31.7
63.8
54.0
62.7
61.8
50.7
54.5 38.8
72.3
44.0
60.3
62.0
56.5
52.2
(Dyn.
68.7
73.2
62.9
64.6
54.8
Source (Stat. (L.N&AF)
46.8 48.0
46.9
67.5
62.2
70,8
66.7
61.6
60.3
46.7 75.2
52.1
65.7 66,5
61.9
59.9
Tent (Dya. LN)
46.3
47.0
71.0
61.1
34.97544
66.8
59.4
EATA (Dyn. LN)
468
47.6
47.1
67.2
62.7
70.6
67.2
62.3
60.9
46.7
52.4
65.9
66.8
62.5
60.1
SAR (Dyn. LN)
46.7
47.4
46.8
67.0
61.9
704
66.4
61.8
60.6
46.0
52.1
65.7
66.4
62.0 59.8
READ (SAF)
49.4
71.2
75.5
53.6
68.3 68.0
62.5
4 EXPERIMENTS
In this section, we evaluate the proposed READ on the audio-visual joint action recognition and event classification tasks under multi-modal TTA with reliability bias. The organization of this section is as follows. In Section 4.1, we present the experiment settings including benchmark con- struction and implementation details. In Section 4.2, we compare READ with the state-of-the-ar (SOTA) TTA methods under different settings, revealing some observations. In Section 4.3, we per- form ablation studies and analytic experiments to give a comprehensive understanding on READ.
4.1 EXPERIMENT SETTINGS
To facilitate the investigation of multi-modal TTA with reliability bias, we construct two benchmarks based on the widely- used multi-modal datasets Kinetics (Kay et al., 2017) and VGGSound (Chen et al., 2020). For comprehensive studies, following Hendrycks & Dietterich (2019), we introduce 15 types of corruptions for the vidco modality and 6 for the audio modality. Each type of corrup- tion has five levels of severity. As a result, obtain Kinetics50-C and VGGSound-C benchmarks with either corrupted audio or corrupted video modalities. READ a general framework that could endow most existing visual-audio pre-trained models with robustness against reliability bias. With- out loss of generality, we choose the SOTA CAV-MAE (Gong et al., 2023) model pre-trained on web-scale audio-vice visual data as the backbone and fine-tune it on the training sets of Kinetics50 and VGGSound dataset, obtaining the corresponding source models. In other words, the training sets of Kinetics50 and VGGSound are the source domains while Kinetics50-C and VGGSound-C are the target domains. During the test-time adaptation phase, READ conducts online updates on specific parameters of the source models using the Adam optimizer. This process utilizes an initial learning rate of 0.0001 for every mini-batch of size 64 within a single epoch. The confidence threshold y in Eq. is fixed as e for all settings. All evaluations are run on Ubuntu 20.04 platform with NVIDIA 3090 GPUs. Due to the space limitation, remove more details into Appendixes B and C.
4.2 COMPARISONS WITH STATE-OF-TIIE-ARTS
We compare READ with four SOTA TTA methods including Tent (Wang et al., 2021), MMT (Shin et al., 2022), EATA (Niu et al., 2022), and SAR (Niu et al., 2023) under different settings. The results are presented in Tables 1-3 and 10-12 wherein the two blocks denote the source model trained with late fusion (Eq. 2) and attention-based based fusion, respectively. From the results, one could have the following observations and conclusions.
TTA methods using late fusion are most sensitive to the reliability bias, which could be attributed the equal treatments on each modality. MM-TTA with carefully-designed pseudo label generation strategy for late fusion cannot always achieve robustness, espe- cially when the representative modality biased (e.g., audio for VGGSound).
The bttet fusion can improve the robustness against reliability bias compared to late fusion. However, TTA methods using vanilla attention- based fusion can only achieve
7

---

Published as a conference paper at ICLR 2024
Metro
AF
Tent
Ours
Comution
V900o Pop
Audio-Trrff:
38
Severiy
Seuerty
(b) Altontoo V-A
[c] Altertton
Figure 3: Reliable fusion with respect to attention values. In the figure, "Tent" and "AF" denote the variants of adopting vanilla attention based fusion with and without parameter updating of LN, respectively. "Attention X-Y" (X, Y E (A, V)) indicates the configuration where query and key correspond to the tokens of modality X and modality Y in Eq. 4, respectively.
Table 2: Comparisons with SOTA methods on Kinetics50-C (left part) and VGGSound-C (right part) benchmarks with corrupted audio modality (severity level 5).
Noise
Weather
Noise Traff. Crowd, Raain Thand. Wind Avg.
Weather
Methods
Guuss. Traff. Crowd
RRinn Thind.
Wind Avg.
Gaass.
Source ((Stat.
67.4
29.5
22.6
20,6
MM-TTA (Dyn. LN)
70,8
69.2
68.5
69.0
69
69.4
69.4
14.1
52
6.4
6.9
8.6
4,5
7.6
Tent (Dyn. L.N)
71.1
67.8
67.4
68.9
6,4
2.9
EATA (Dya. LN)
71.2
67.9
67.5
67.8
70.9
68.7 69.0
28.8
17.1
22.4
17.4
33.8
20.4
23.3
SAR (Dyn. LN)
71.1
67.5
67.4
67.4
70.6
68.6
68.8
28.5
16.6
22.1
17.4
33.7
20.2
13.1
READ (Dyn. LN)
71.3
68.5
68.5
68.4
71.8
69.0
69.6
36.4
25.3
78.9
27.3
26.6
30.0
Source (Stat. (F))
73.7
65.5
67.
70.3
67.9
70.3 69.3
57.0
25.5
16.8
21.6
27.3
25.5
13..6
Tenr (Dyn. L.N)
73.9
69.
70.5
10.6
1.8
EATA (Dyn. LN)
73.7
66.1
68.5
70.3
67.9
70,1
69.4
39.2
26.1
22.9
26.0
31.7
30.4
29.4
SAR
68.2
37.4
READ (SAF)
74.1
69.0
69.7
71.1
718
70.7
71.1
40.4
28.9
26.6
30.9
36.7
30.6
32.44
negligible performance gains some cases, implying that the reliability bias problem can- not be simply addressed by adopting the widely-used TTA paradigm, i.e., updating the parameters of normalization layers.
The proposed confidence- loss could bring performance gain for both late fusion and attention-basee fusion. Applying the proposed SAF strategy with the loss could guarantee noise-resistant thus learning reliable attention for fusion. In other words, our READ could significantly improve the robustness against the cross-modal reliability bias.
Table 3: Comparisons with SOTA methods on VGGSound-C benchmark with corrupted video modality (severity level 5).
Noise Gauss, Shot Impal. Defoc.. Glass Mor. Zoom Snow Frost Fog Brit Coon: Elas. Pix.
Weather
Digital
Methods
JPBG
Avg.
Sounce ((Stas. LN) LFi
37.7
36,5
37,8
52.7
51.3
55.2
53.7
$1.9
52.3
50.4 55.3
45.
52.5
51.7
52.3
49.1
MM-TTA (Dyn. I.N)
44.8
41.5
48.0
27.4
23.5
46.9
40.3
40.7
32.0
Teur (Dya. LN)
76
688
7.2
53.1
52.1
55.5
54.5
52.6
52.7
56.0
16.6
52.6
54.2
53.1
38.0
(Dyn. 1N)
56.0
52.4
51.0555.
53.5
52.3
49.5
SAR (Dyn. LN)
37.7
36.4
31.7
52.8
51.5
55.5
53.9
51.9
52.5
50.455.4
44.8
52.7
51.8
52.5
49.2
READ (Dyn. LNN
42.1
41.5
42.1
49.3
50.9
53.5
52.5
50.6
52.1
51.1
54.0
46.2
52.5
49,
50.
49.2
Source (Stat, (1 AF))
52.8
52.7
57.2
58.
57.6
56.4
58.9
53.7
56.9
558
56.9
56.0
* Tent (Dya. LN)
52.7
52.7
52.7
56.7
56.5
57.9
57.2
55.9
56.3
56.3 58.4
$1.0
57.4
56,2
56.7
55.8
(Dyn. LN)
58.6
59.0
57.4
56.2
SAR (Dyn. LN)
$2.9
52.8
52.9
57.2
57.1
59.6
57.6
56.3
56.7
58.9
58.9
$4.0
57.0
56.0
57,0
56.1
READ (SAF)
53.6
53.6
53.5
57.9
57.7
39.4
58.8
57.2
57.8
55.0
59.9
55.2
58.6
57.1
57.9
56.9
4.3 ABLATION AND ANALYTIC STUDIES
In this section, all the experiments are performed under Kinetics50 datasets with either fog noise on video or traffic noise on audio at severity level unless otherwise stated.
Ablation studies. To verify the importance of each design, we investigate the variants of the method in Table 4, where one could have the following observations. First, Tent using SAF cannot always improve the robustness (e.g., 34.9 to 22.7). predictions
Table 4: Ablation studies on Kinetics50-C benckmark. Pink denote the default setting.
Varinns
Video-fog
Audio-aatic
which could be boiled down to the in multi modal TTA as depicted in Fig. (dd). At that time, up- dating the parameters of the cross- moss-aftentionn might make model
Tem (SStd. EN & SAF)
22.7
6900
Ours ((Stat. LN) & AF)
50.9
67.4
Ours (DDy, LN) SAF)
58.1
69.3
Ours (1851.
574
69.0
8

---

Published as a conference paper at ICLR 2024
overfitting noise. Second, the proposed loss could also improve the robustness of the existing TTA paradigm that mainly updates the normalization layers (e.g.,34.9 to 50.9). Third, applying the normalization layer updating mechanism our SAF could slightly improve the performance (e.g., 57.4 to 58.1). Note that, considering the efficiency, we maintain "(Stat. LN) & SAF" as our default setting. Due space limitation, more comprehensive results are moved to Appendix D.1.
Superiority of Reliable Fusion at Test-time. As mentioned in Introduction, some imbalance modal- ity learning studies (Zhang et al., 2023) could alle- viate the modality reliability problem by alerting the training process in the source domain. To verify the necessity of reliable fusion at test-time, we compare our method with the recently proposed SOTA method (QMF (Zhang et al., 2023)) on all kinds of corruptions. In short, QMF could estimate the confidence of each modality by adopting an elaborately designed training pipeline and such paradigm could naturally applied to handle reliability bias. As shown in Fig. 4, our method could achieve robustness superiority against both audio and video modalities compared to QMF, although the latter alters the training process in the source domain.
10 plan
OlADdo Cocuples
Figure 4: Comparisons between different fusion manners. In the figure, "LF", "AF", and "QMF" denote the variants of late fu- sion, vanilla attention- based fusion, and the training-time robust fusion of QMF, respectively.
Reliable Fusion across Different Situations. In this experiment, we further verify the claim in Introduction that the reliability bias cannot be completely eliminated by simply updating the nor- malization layers like existing TTA methods. From Fig. 3, one could observe that "Tent" would perform slightly robust fusion effects compared to "AF", which could be attributed the narrowed domain gap by repurposing the LN. In contrast, our method shows remarkable improvements in the reliability estimation (attention value) on both video or audio bias situations with varying severities, which verifies the necessity of self-adaptive attention-based fusion paradigm for multi-modal TTA.
Visualization on the Self- adaptive Attention. The effec- tiveness of our SAF module could be verified from Fig. 5 with the following observations.
Clean
Video-Fog
Audio-Trafh Tralfic
5.8
35.9
First, adapting with clean/nearly-non- shifted test data, our SAF could maintain the importance between audio and video modalities (8.0 v.s. 41.5). Note that, the video is more informative for Kinetics50 under the action recognition task. Second, once the one modality is corrupted, SAF could make the model focus more on another rather reliable modalities, e.g., from 8.0 to 10.5. Meanwhile, the clean modality will reduce its attention to the corrupted one while the corrupted modality will increase the attention to the clean one, e.g., from 30.2 to 23.6 and 3.7 to 5.1. Due to space limitations, we place more results in Appendix D.7.
3.5
42.0
Figure 5: Visualization on the self-adaptive attention. The
blocks of the top left and bottom right denote the self-attention between audio and videos, respectively. The blocks of the top right and bottom left denote the cross-attention from audio to video and video to audio, respectively. The number upon the blocks denotes the mean of attention values across the adapta- tion process, which is amplified by 10, 000 times for clarity.
CONCLUSIONS
In this paper, we formally study the multi-modal test-time adaptation. By delving into the distri- bution shifts in multi-modas scenarios, we reveal that the intra-modal distribution shifts will result in the information discrepancies across modalities, i.e., the modality reliability bias challenge. To address this challenge, READ adopts the self-adaptive attention module to achieve reliable fusion across different modalities, and the confidence-awart loss function for robust adaptation. For com- prehensive evaluations,
provide two new benchmarks with multi-modal reliability bias. In the future, we plan to explore more specific problems for multi-moud modal TTA with reliability bias under different applications and scenarios.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This work was supported in part by NSFC under Grant U21B2040, 62176171; and in part by the Fundamental Research Funds for the Central Universities under Grant CJ202303.
10

---

Published as a conference paper at ICLR 2024
REFERENCES
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv:1607.06433 2016.
Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. Vggsound: large-scate audio- visual dataset. In ICASSP, 2020.
Shuo Chen, Jindong Gu, Zhen Han, Yunpu Ma, Philip Torr, and Volker Tresp. Benchmarking ro- bustness of adaptation methods on pre-trained vision-language models. arXiv:2306. 02080, 2023.
Chenzhuang Du, Tingle Li, Yichen Liu, Zixin Wen, Tianyu Hua, Yue Wang, and Hang Zhao. Im-
proving multi-modal learning with uni-modal teachers. arXiv:2106. 1059,, 2021.
Yunfeng Fan, Wenchao Xu, Haozhao Wang, Junxiao Wang, and Song Guo. Pmr: Prototypical modal rebalance for multimodal learning. In CVPR, 2023.
Yulu Gan, Yan Bai, Yihang Lou, Xianzheng Ma, Renrui Zhang, Nian Shi, and Lin Luo. Decorate the newcomers: Visual domain prompt for continual test time adaptation. In AAAI, 2023.
Jin Gao, Jialing Zhang, Xihui Liu, Trevor Darrell, Evan Shelhamer, and Dequan Wang. Back to the source: Diffusion-driven adaptation to test-time corruption. In CVPR, 2023.
Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all. In CVPR, 2023.
Yuan Gong, Andrew Rouditchenko, Alexander H Liu, David Harwath, Leonid Karlinsky, Hilde Kuehne, and James Glass. Contrastive audio-visual masked autoencoder. In ICLR, 2023.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common cor- ruptions and perturbations. In ICLR, 2019.
Xuefeng Hu, Gokhan Uzunbas, Sirius Chen, Rui Wang, Ashish Shah, Ram Nevatia, and Ser-Nam Lim. Mixnorm: Test-time adaptation through online normalization estimation. arXiv:21 10. 1478, 2021.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by
reducing internal covariate shift. In JCML, 2015.
Yusuke Iwasawa and Yutaka Matsuo. Test-time classifier adjustment module for model-agnostic domain generalization. NeurIPS, 2021.
Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijaya- narasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv: 77..0005990. 2017.
Taeyeop Lee, Jonathan Tremblay, Valts Blukis, Bowen Wen, Byeong-Uk Lee, Inkyu Shin, Stan Birchfield, In So Kweon, and Kuk-Jin Yoon. Tta-cope: Test-time adaptation for category-level object pose estimation. In CVPR, 2023.
Junnan Li, Dongxu Li. Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In ICMI, 2023.
Yijie Lin, Yuanbiao Gou, Zitao Liu, Boyun Li, Jiancheng Lv, and Xi Peng. Completer: Incomplete multi-view clustering via contrastive prediction. In CVPR, pp. 11174-11183, 2021.
Yijie Lin, Jie Zhang, Zhenyu Huang, Jia Liu, Zujie Wen, and Xi Peng. Multi Multi-granularity correspon- dence learning from long-term noisy videos. In ICLR, 2024.
Yuejiang Liu, Parth Kothari, Bastien Van Delft, Baptiste Bellot-Gurlet, Taylor Mordan, and Alexan- dre Alahi. Tit++: When does self-supervised test-time training fail or thrive? In NeurIPS, 2021.
Zachary Nado, Shreyas Padhy, D Sculley, Alexander D' Amour, Balaji Lakshminarayanan, and Jasper Snoek. Evaluating prediction-time batch normalization for robustness under covariate shift. arXiv:2006. 10963, 2020.
11

---

Published as a conference paper at ICLR 2024
Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Yaofo Chen, Shijian Zheng, Peilin Zhao, and Mingkui Tan. Efficient test-time model adaptation without forgetting. In JCML, 2022.
Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Zhiquan Wen, Yaofo Chen, Peilin Zhao, and Mingkui Tan. Towards stable test-time adaptation in dynamic wild world. In ICLR, 2023.
Xiaokang Peng, Yake Wei, Andong Deng, Dong Wang, and Di Hu. Balanced multimodal learning via fly gradient modulation. In CVPR, 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
Steffen Schneider, Evgenia Rusak, Luisa Eck, Oliver Bringmann, Wieland Brendel, and Matthias Bethge. Improving robustness against common corruptions by covariate shift adaptation. In NeurIPS, 2020.
Inkyu Shin, Yi-Hsuan Tsai, Bingbing Zhuang, Samuel Schulter, Buyu Liu, Sparsh Garg, So Kweon, and Kuk-Jin Yoon. Mm-tta: multi-modal test-time adaptation for 3d semantic segmenta- tion. In CVPR, 2022.
Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei Efros, and Moritz Hardt. Test-ting training with supf--upen for generalization under distribution shifts. In ICML, 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurlPS, 2017.
Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. In ICLR, 2021.
Qin Wang, Olga Fink, Luc Van Gool, and Dengxin Dai. Continual test-time domain adaptation. In CVPR, 2022.
Weiyao Wang, Du Tran, and Matt Feiszli. What makes training multi-modal classification networks hard? In CVPR, 2020.
Yake Wei, Di Hu, Yapeng Tian, and Xuelong Li. Learning in audio-visual context: A review, analysis, and new perspective. arXiv:2208. 09579, 2022.
Mouxing Yang, Yunfan Li, Zhenyu Huang, Zitao Liu, Peng Hu, and Xi Peng. Partially view-aligned representation learning with noise-robust contrastive loss. In CVPR, 2021.
Mouxing Yang, Zhenyu Huang, Hu Peng. Taihao Li, Jian Cheng Lv, and Xi Peng. Learning with twin noisy labels for visible-infrared person re-identification. In CVPR, 2022.
Yongcan Yu, Lijun Sheng, Ran He. and Jian Liang. Benchmarking fest-time adaptation against distribution shifts in image classification. arXiv:2307.03133 2023a.
Zhiqi Yu, Jingjing Li, Zhekai Du, Lei Zhu, and Heng Tao Shen. A comprehensive survey on source- free domain adaptation. arXiv:2302.118033 2023b.
Marvin Zhang, Sergey Levine, and Chelsea Finn. Memo: Test time robustness via adaptation and augmentation. In NeurlPS, 2022.
Qingyang Zhang, Haitao Wu, Changqing Zhang, Qinghua Hu, Huazhu Fu, Jocy Tianyi Zhou, and Xi Peng. Provable dynamic fusion for low-quality multimodal data. In ICML, 2023.
Zhi Zhou, Lan-Zhe Guo, Lin-Han Jia, Dingchu Zhang, and Yu-Feng Li. Ods: Test-time adaptation in the presence of open-wort world data shift. In ICML, 2023.
12

---

Published as a conference paper at ICLR 2024
APPENDIX
A PROOFS TO THEOREMS
In this section, we present detailed proofs for Theorems and in the main paper. Without loss of generality, omit the batch size and subscript of pi, and obtain the form of Lra p log (ey/p). Theorem 1.
The gradient direction produced by f is non-monotonous.
Proof 1. The gradient of Lra W.K., is in the form of
ŌCre Op
log (7) - log (p).
(8)
Clearly, Oppa/Op > Off. p <, and 0Cra/ /pp O 7 p. and OCra/Op 0 iff. p = Therefore, is the stationary point of Lya and Lro. is non-monotonous.
Theorem 2. The gradient value will rise with increasing ps iff. ps (1: 1) decreasing ps iff.
E (0, 7)
Proof 2. The second-order gradient Lra w.r.t. is in the form of
Lro
8L
Op
0p2
Öp
1
0.
P
Therefore, 0Lra/ OOp decreases monotonically for E (0. Given that DOpa/Op > 0 if < and 0Opa/Op < i < p. 10Cra/0p will rise either with increasing ps (pr (7, 1)) or with decreasing (p: E (0,7)) 7)).
13

---

Published as a conference paper at ICLR 2024
B MORE DETAILS ABOUT THE BENCHMARKS
We construct two benchmarks for multi-modal TTA with reliability bias upon the VGGSound (Chen al., 2020) and Kinetics (Kay et al., 2017) datasets. The two datasets are widely used for multi- modal event classification and action recognition. To be specific,
VGGSound (Chen et al., 2020) is a large-scale video dataset that consists of 309 diverse classes and contains broad spectrum of everyday audio events. All videos within the VGGSound dataset are "in the wild," meaning that they were captured in real-world set- tings, and the audio in the videos corresponds to the visual information, making the source of sound visually apparent. In other words, the audio modality in the datasel will con- tain more task-specific information for the event classification lask, compared to the visual modality. Each video in this dataset has a fixed duration of 10 seconds. Due to the changes in video availability, we downloaded 14, 046 evaluation videos and thus obtain 14, 046 testing visual-audio pairs.
Kinetics50 is a subset of Kinetics (Kay et al., 2017) dataset. Specifically, Kinetics is a comprehensive collection of YouTube videos, covering diverse set of 400 distinct human action classes. Human actions depicted in these videos have been meticulously annotated through manual efforts employing Mechanical Turk. Due to the characteristic of action recognition, video modality in the dataset will contain more information compared to the audio modality. Additionally, all videos have been trimmed to a standardized duration of 10 seconds, centered around the specific action, ensuring consistency and relevance in the dataset. In our experiments, following Peng et al. (2022), we randomly select 50 classes from the dataset, obtaining the subset of Kinetics, i.e., Kinetics50, with 29, 204 training pairs and 2, 466 test pairs.
To comprehensively evaluate modality bias, introduce different distribution shifts on the video and audio modalities for the test sets of VGGSound (Chen et al., 2020) and Kinetics (Kay et al., 2017) datasets. For the video corruptions, we follow Hendrycks kinds of corruptions into the video, and each corruption is with 5 kinds of severity levels for exten- sive validations. Specifically, the corruptions on video modality include "Gaussian Noise", "Shot Noise", "Impulse Noise", "Defocus Blur". "Glass Blur", "Motion Blur". "Zoom Blur", "Snow", "Frost", "Fog", "Brightness", "Elastic", "Pixelate", "Contrast", and "JPEG". Similar to the video modality, we add 6 kinds of common audio noise' with 5 kind of severity levels captured in the wild. Specifically, the corruptions on audio modality include "Gaussian Noise", "Paris Traffic Noise", "Crowd Noise", "Rainy Noise". "Thunder Noise" and "Windy Noise". The case of the 15 video corruption types and 6 audio corruption types are visualized in Fig. 6 and Fig. 7.rsppcttvely. Fi- nally, we obtain the corresponding corrupted benchmarks, named VGGSound-C and Kinetics50-C.
Dietterich (2019) apply 15
hhttss://freesoundd.or
14

---

Published as a conference paper at ICLR 2024
Gaussian Noise
Shot Noise
Impulse Noise
Defocus Blur
Glass Blur
Motion Blur
Zoom Blur
Snow
Frost
Fog
Brightness
Contrast
Elastic
Pixclate
JPEG
Figure 6: Visualization of various visual corruption types on the constructed Kinetics- C benchmark.
Crowd
Raw Audio
Thunder
Wind
Figure 7: Mel spectrogram visualization of the raw audio and the corresponding audio corruption types on the constructed Kinetics-C benchmark.
15

---

Published as a conference paper at ICLR 2024
C MORE DETAILS ABOUT THE BACKBONE
In the implementation, we use the CAV-MAE (Gong et al., 2023) model as the backbone. CAV-MAE adopts an encoder-decoder-lik architecture that is pre-trained on large-scale video data with both the contrastive learning and mask image modeling paradigms. The CAV-MAE encoder consists of 11 Transformer layers dedicated each modality for the modality feature extraction, alongside one Transformer layer for cross-moda fusion. The input to the CAV-MAE encoder involves 10- second video clips containing both video and corresponding audio data. For the video stream, CAV- MAE samples 10 frames within each video clip and randomly selects one frame feeding into the visual Transformer encoder. For the audio stream, each 10-second audio waveform is converted into one spectrogram and then inputted to the audio Transformer encoder.
During the fine-tuning phase, we maintain the visual and audio encoders of the pre-trained model and add one randomly initialized classification head upon them. The fine-tuned model is regarded as the source model and denoted as "Source (Stat. (LN & AF))". Here, "Stat." is the short of "statical" that represents the frozen state of the layer normalization (LN) and attention-baas based fusion (AF) lay- ers during the test-time phase. To investigate the robustness of different fusion manners, we design another variant of the source model that utilizes 12 Transformer layers for feature extraction and per- forms vanilla late fusion (LF) between the classification logits of each modality. The corresponding model variant denoted "Source ((Stat. LN) & LF)". During the test-time adaptation phase, unless otherwise specified, all baselines update the parameters of all normalization layers rooted in the source model, i.e., referred as "Dyn. LN" where "Dyn." is the short of "Dynamic". In con- trast, as depicted in Fig. 2, our default approach in the READ framework involves updating only the parameters of the last Transformer layer (referred to as the AF layer) in a self- self-adaptive manner. We dub this paradigm self-adaptive attention-based fusion, abbreviated as "SAF". SAF essentially repurposes the standard AF operation through modulating the parameters within the attention layer with the guidance of the proposed objective function (Eq. 7). As result, the model would focus more on the unbiased modalities, leading to reliable cross-modal fusion during test time.
16

---

Published as a conference paper at ICLR 2024
D MORE EXPERIMENT RESULTS
D.1 MORE ABLATION RESULTS
We additionally present more comprehensive ablation results on Kinetics50 with both corrupted audio modality and video modality.
Table 5: Comprehensive ablation studies on Kinetics50-C with corrupted video modality.
Noise
Blur
Weather
Digital
Methods
Gauss. Shot Impul. Defoc. Glass Mot. Zoom Snow Frost Fog Brit. Coontrr Elas, Pix. JPEG Avg.
Tent ((Stat. LN) & SAF)
45.3
45,7
45.1
66.6
58.1
70.5 65.8
60.8
57.2 22.7 75.2
48.6
66.1 63.7 53.4
56.3
Ours ((Dyn. LN) & AF)
47.8
48.2
47.6
67.7
64.2
71.0
68.2
63.9
62.6
50.9 75.3
53.3
66.9 67.8 63.8
61.3
Ours (Dyn. LN) & SAF)
49.5
49.8
49.1
68.1
65.8
71.2
69.1
65.1
64.8
58.1 75.4
54.2
68.8
68.7
65.3
62.9
Ours (Stttt. LN) SAF)
49.4
49.7
49.0
68.0
65.1
71.2
64.5
57.4 75.5
53.6 68.3 68.0
62.5
Table 6: Comprehensive ablation studies on Kinetics50-C with corrupted audio modality.
Noise
Weather
Methods
Gauss. Traff. Crowd, Rain Thund. Avg.
Tent ((Stat. LN) SAF)
73.7
69.0
69.6
70.4
69.2
70.6
70.4
((Dyn. LN) AF)
73.8
67.4
69.0
70.6
70.5
70.3
70.3
((Dyn. LN) SAF)
73.9
69.8
72.2
71.2
((Stat. L.N) & SAF)
69.0
69.7
71.1
71.8
71.1
D.2 RESULTS ON THE MIXED SEVERITY SETTING
To further investigate the robustness of our READ, we conduct more experiments the Kinetics50- C benckmark under the settings of mixed severity, comparing with Tent (Wang et al., 2021) and SAR (Niu et al., 2023). To be specific, we create test pairs for each corruption type by blending severity levels from to 5, resulting in 5N test pairs, where N represents the original size of the test data. After that, we shuffle the obtained test pairs and randomly choose pairs for each corruption type. The results are depicted in Fig. 8, indicating the effectiveness of READ in addressing cross- modal reliability bias across various corruption types exhibiting mixed severity levels.
Tent:
Tent: 71.0%
SAR: 68.8%
75
Ours: 70.2%
Ours: 72.7%
74
70
Accuracy (%)
71
65
68-
60
55
83
gaies
implose
defocus
giaas
motion
zoce
Srow
frest
log
bright
contrast
elantic
pinel
loeg
rain
ture
wind
(a) Video Compptinn
(b) Audio Corupttin
Figure 8: Performance comparison among Tent, SAR, and our READ on the Kinetics50 C bench- mark under mixed severity levels. The legend key provides an overview of the average performance of each approach across various corruption types.
D.3 RESULTS ON THE MIXED DISTRIBUTION SHIFTS
We explore the efficacy of our READ approach in a more challenging scenario, í.e., mixed distri- bution shifts, which is in line with the continual TTA setting (Gan al., 2023; Wang et al., 2022). In this setting, both baseline methods (Tent and SAR) alongside our READ continually adapt to evolving corruption types, and the averaged performance across all corruption types are reported.
17

---

Published as a conference paper at ICLR 2024
To ensure comprehensive evaluations, vary the severity levels from to 5. The results are sum- marized in Fig. 9. Although READ not dedicatedly designed for the mixed distribution shifts challenge, it still achieve remarkable robustness. The results underscores the adaptability and silience of READ.
72.0
71.5
Accuraty Ho)
83
Acunay 55
71.0
80
70.5
45
70.0
50
09.5
45
09.0
40
Sevwrity
by
(0) Video Comuption
(b) Audio Corruption
Figure 9: Performance comparison among Tent, SAR, and our READ on the Kinetics50-C beneh- mark with mixed corruption types. The legend key provides an overview of the average performance of different severity levels.
D.4 INFLUENCE OF THE
In this section, investigate the influence of the only hyper-parameter (i.e., threshold y in Eq. 6) in our approach. To this end, we vary in the range of 1..1, 0.3, 0.,, e 0.4, 0.5] and perform corresponding experiments on the Kinetics50-C benchmark with fog and traffic corruptions. The results on Fig. 10 illustrates the stability of READ across varying threshold values of
Mudio-Treffic
Video-Fog
70
68-
64
62
60
58
54
0.1
0.2
0.3
0.4
0.5
Threshold in Eq.
Figure 10: Sensitiveness analysis of our READ against the hyper-parameter on the Kinetics50-C benchmark with fog and traffic corruptions.
D.5 EFFICIENCY COMPARISONS
Different from most TTA methods that updates the parameters of normalization layers, our READ repurpose last one Transformer layer of CAV-MAE (Gong et al., 2023) model as elaborated Sec- tion 3. In this section, we compare the efficiency of the two paradigms. To this end, we choose the
CAV-MAE model as source model (i.e., source (Stat. (LN AF))), and conduct experiments on the VGGSound-C benckmark. We measure both the size of learnable pa- rameters and the GPU time during the test-time adaptation phase. Table 7 highlights that our READ accomplishes adaptation in less time. The efficiency of READ can be attributed to its module repur- posing approach. Although the normalization layer updating scheme occupies fewer parameters, demands more time for propagation.
18

---

Published as a conference paper at ICLR 2024
Table 7: Efficiency comparisons among different approaches the VGGSound-C benchmark.
Method
#params (M)
GPU time (14,046 pairs)
Tent (Dyn, LN)
0.2
209.5 seconds
EATA Oyn. LN)
0.2
207 seconds
SAR (Dyn. LN)
0.2
286.1 seconds
READ (SAF)
1.8
134.1 seconds
D.6 DIFFERENT MODULE REPURPOSE SCHEMES
In our default approach, we update W and Bex (h E Q, within the last Transformer layer of the source model to ensure reliable fusion. This section explores the impact of different repurposing schemes. To this end, we design three variants: one that updates only the query and key projection layers, another that updates only the value projection layers, and third that updates the final classification head. Table illustrates that the default setting, updating the query, key, and value projection layers simultaneously, exhibits significant performance superiority. Modulating the classification head demonstrates minimal effectiveness (e.g.. from 46.7 to 49.1). In contrast, the attention modulation scheme achieves adaptive fusion between discrepant modalities. mitigating the multi-modal reliability bias problem (e.g., from 46.7 7 to 51.7). Moreover, modulation on the query, key, and value projection layers introduces additional parameters for reliable fusion, resulting in further improvements in robustness (e.g., from 46.7 7 to 57.4).
Table 8: Comparisons between different modulation schemes on the Kinetics50-C benchmark with severity level 5.
Corruption Video-Fog Audio-Traffic
Source
QK
MI.P
QKV (ours)
46.7
51.7
53.6
49.1
57.4
65.5
68.8
67.2
66.7
69.0
D.7
MORE VISUALIZATION RESULTS THE ATTENTION MATRIX
Clean
Video-Fog
Audio-Traffic
00/V
30.9
9.4
26.4
30.2
AF
0905A
4.3
39.9
40.7
4PV
4P/VV
7.7
30.8
9.4
26.4
7.1
32.4
Tent
3.7
41.5
4.2
40.0
3.6
41.5
Audio
Video
Audio
Video
Ausio
Video
Figure 11: Visualization on the attention matrix of source model with vanilla attention-based fusion (AF), and the model adapted by Tent (Wang et al., 2021) with dynamic LN.
D.8
MORE COMPARISON RESULTS ON DIFFERENT SEVERITY LEVELS
19

---

Published as a conference paper at ICLR 2024
Table 9: Performance of different approaches on VGGSound and Kinetics50-C benchmark without any corruptions.
Method
Source (Stat. (LN&AF))
Tent
EATA
SAR
READ
VGGSound
62.6
63.1
63.1
63.5
Kinetics50
82.3
82.1
82.3
82.3
82.2
Table 10: Comparisons with SOTA methods on Kinetics50-C benchmark with corrupted video
modality (severity level
Noise Gause Shot Impul. Defoc Glass Mot. 7oom Snow Frost Fog Brit Coontr. Elas. Pix.
Blur
Weather
Digital
Methods
JPFG Avg.
Source ((Stat, LN) & LF)
46.6
47.8
46,9
71.0
63.4
74.4
68.1
62.1
58.9
65.4
77.6
68.2
76.1
77.1 73.0
65.1
MM-TTA (Dyn. LN)
48.8
50.8
50.6
66.0
60.6
70.9
63.5
59.8
56.3
58.1 75.1
59.3
72.2
74.7
68.7
62.4
(Dyn. LN)
46.6
44.9
64.6
56.5
77.9
76.3
77.0
73.2
EATA (Dyn. LN)
48.2
47.3
70.8
63.9
74.6
68.4
62.3
58.9
65.4 77.8
68.1
76.0
77.0
73.0
65.2
SAR (Dyn. LN)
46.7
47.9
47.0
70.6
63.3
74.4
68.2
62.3
58.9
65.2 77.7
68.0
76.0
77.0
72.7
65.1
READ (Dyn. LN)
49.3
50.0
49.4
71.1
65.7
75.0
70.3
64.5
61.5
67.1 78.
69.5
76.6
77.2
73.7
66.6
Source (Stat. (LN&AF)
54.1
54.8
54.6
73.5
68.3
76.6
71.5
69.2
64.7 69.5 79.3
72.1
77.6
79.4
75.4
69.4
Tent (Dyn. L.)
54.2
55.1
55.2
73.6
69.6
76.8
71.9
69.5
65.6 70,2 79.4
72.9
78.3
79.2
75.3
69.8
EATA (Dyn. LN)
54.4
54.9
55.0
73.4
69.1
76.5
71.6
69.2
65.1 69.5 79.5
72.3
77.7
79.1
75.2
69.5
SAR (Dyn. LN)
54.8
55.0
73.1
68.2
76.4
71.1
69.1
64.8 69.4
72.0
77.4
79.1
75.0
69.2
(SAF)
56.4
79.3
75.6
Table 11: Comparisons with SOTA methods on Kinetics50-C (left part) and VGGSound-C (right part) benchmarks with corrupted audio modality (severity level 3).
Noise
Weather
Noise
Weather
Methods
Gauss. Traff. Crowd, Rain Thund. Wind Avg.
Gauss. Traff. Crowd, Rain Thund. Wind Avg.
Source ((Stat. LN) & LF)
74.2
68.8
68.7
66.7
71.6
70.4
70.1
39.6
23.8
25.0
28.7
36,5
26.9
30.1
MM-TTÀ (Dyn. LN))
72.8
69.6
68.9
68.7
70.7
70.3
70.2
13.8
7.1
7.6
16.2
10.6
5.4
10.1
Tent (Dyn. LN)
74.2
69.0
69.6
64.8
71.9
71.1
70.1
11.2
4.1
3.4
5.2
12.8
5.1
7.0
EATA (Dyn. LN)
74.1
68.8
69.1
67.3
71.8
70.6
70.3
40.3
23,9
24.7
28.7
36.5
26.9
30.2
SAR (Dyn. LN)
73.9 68.8
68.9
66.7
71.6
70.3
70.0
39.9
23.6
24.9
28.7
36.4
26.8
30.0
(Dyn. LN)
74,2
69.6
69.0
72.7
71.0
29.9
33.2
34.6
Source (Stat. (LN&AF))
75.9 64.4
68.7
70.3
67.9
70.3
69.3
42.1
29.4
19.5
27.6
31.2
29.4
29.9
(Dyn.
73,9
70.1
4.7
7.8
EATA (Dyn. LN)
76.0
65.7
68.9
69.8
69.1
72.1
70.3
46.7
30.5
28.0
31.4
35.4
33.8
34.3
SAR (Dyn. LN)
76.0 64.6
68.7
69.3
68.6
72.2
69.9
43.1
17.3
8.3
29.0
31.6
30.5
26.6
READ (SAF)
76.4
69.6
70.8
72.0
72.6
72.3
72.3
47.3
32.7
29.9
33.2
38.3
33.7
35,8
Table 12: Comparisons with SOTA methods on VGGSound-C benchmark with corrupted video modality (severity level 3).
Noise Gauss.
Blur
Weather
Digital
Methods
Impul. Defoc, Glass
Zoom Snow
Fog Brit, Coontr Elas, Pix.
JPEG Avg.
Source ((Stat. LN) LF)
45.6
45.3
45.4
55.7
54.0
55.4
55.1
53.7
53.4 58.5
58.3 58.1 56.5
MM-TTA (Dyn.
15.8
50.4
44.3
48.4
41.4
28.1
46.5
52.0
52.0
$2.0
51.6
Tent (Dyn. LN)
19.8
18.4
55.9
55.3
57.3
55.9
55.3
45.3
34.8 58.4
56.4
58.4
58.4
57.1
46.9
EATA (Dyn. LN)
45.8
45.6
45.7
56.3
55.2
58.0
56.0
55.8
54.4
54.5 58.9
55.3
58.8
58.5 57.1
54.4
SAR (Dyn. LN)
45.4
45.2
45.2
55.8
54.3
57.7
55.6
55.3
53.9
53.7 58.5
54.2
58.5
58.2 56.7
53.9
READ (Dyn. LN)
46.0
46.0
46.3
53.0
52.9
56.3
54.1
53.8
53.3
53.0
58.0
53.8
57.7
56.8
55.1
53.1
Source (Stat. (I.N& & AF)
54.7
54.6
54.7
59.3
58.4
60.4
59.0
58.3
57.4
57.8
61.3
58.0
61.0
60.9
60.0
158.4
Teat (Dyn. LN)
54.4
54.3
54.4
58.8
57.8
59.7
58.5
57.6
56.9
58.6
60.6
58.0
61.0
60.9
60.0
58.1
(Dyn. LN)
59.4
58.4
61.3
SAR (Dyn. LN)
54.8
54.6
54.7
59.4
58.3
60.3
58.9
58.3
57.5
58.2
61.2
58.3
60.9
60.8
59.9
58.4
READ (SAF)
55.3 55.4
55.4
60.0
59,1
61.1
59.8
59,2
58.5
59.3
61.9
59.8
61.5
61.5
60.7
59,2
20


Reviews:
[{"review_id": "tpk5GFU4AJ", "reviewer": "Reviewer_Hrwp", "summary": "This work proposes a method for multi-modal test-time adaption (TTA) in the presence of cross-modal reliability bias. While there are numberer of works in test-time adaption, most of them focus on single-modality tasks and few works consider the practical multi-modal scenarios. In contrast, the authors investigate the characteristics of multi-modal TTA and reveal the task-specific cross-modal reliability bias setting where the information between modalities is unbalanced during test time derived from the distribution shifts across domains in some modalities. The authors conduct analysis experiments finding that the unreliable cross-modal fusion and noisy predictions hinder the robustness against cross-modal reliability bias. As a remedy, this paper proposes a novel method dubbed reliable fusion and robust adaption (RFRA). The idea of RFRA is straightforward. A self-adaptive attention is employed to achieve reliable cross-fusion during test time and a robust loss is adopted to prevent the prediction noise from dominating the adaption process. The authors experimentally validate their method on two new benchmarks against several baselines showing reasonable improvements.", "questions": "My questions are mainly in the efficiency comparisons between the proposed SAF module and the TTA baselines, and some clarification on statements. Moreover, I wonder about the effect of the proposed method on the settings of MIXED SEVERITIES and MIXED DISTRIBUTION SHIFTS. Certainly, this is optional during the rebuttal time because the settings are out of the scope of the paper, but I think the results would strengthen this work.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. This paper has a good motivation. The authors focus on multi-modal test-time adaption and reveal a NEW task-specific challenge (i.e., cross-modal reliability bias) for the first time. This paper first empirically proves that the existing test-time adaption methods cannot tackle the cross-modal reliability bias problem. Furthermore, the authors also investigate the effect of adopting the existing unbalanced multi-modal learning methods to handle the cross-modal information discrepancy.  The unbalanced multi-modal learning methods handle the unbalanced multi-modal data during training time with the help of labeled data, which resembles the test-time training paradigm in the domain adaption community. The results indicate that the multi-modal learning methods cannot take superiority for the challenge. In other words, the authors support the claim that designing an elaborated method for the cross-modal reliability bias during test time is essential.\n\n2. The authors take an in-depth study on the cross-modal reliability bias challenge and find that robust cross-modal fusion and noise resistance are essential to achieve multi-modal TTA against reliability bias. To this end, the authors design a novel method dubbed reliable fusion and robust adaption (RFRA). On the one hand, RFRA achieves reliable fusion through the self-adaptive fusion module. On the other hand, RFRA employs an elaborately designed objective function to achieve noise-robust adaption. Figs. 1 and 2 clearly depict the motivation and key idea of the paper. Overall evaluation, this paper is with strong motivation, a technical sound approach, extensive experiments, and good writing.", "weaknesses": "1. The authors have conducted extensive evaluations on two newly constructed benchmarks regarding the most challenged setting (severity 5 in this paper), and the results indeed verify the effectiveness of handling the cross-modal reliability bias challenge. Even so, I think some challenging TTA settings that are orthogonal to the cross-modal reliability bias might help to improve the practicalness and impact of this work. First, in the practical multi-modal scenarios, the severity of distribution shifts might dynamically vary. It would make the work more practical if the authors could additionally investigate the robustness of the proposed method under the setting MIXED SEVERITIES. Furthermore, it is also common that the corruption types continually vary in the wild, resulting in the demand for continual test-time adaption (CTTA). The proposed approach would be more solid and universal if the proposed method could work in the MIXED DISTRIBUTION SHIFTS setting (i.e., continual TTA setting).\n\n2. Some details are missing. It is not clear how many layers of the attention module are used and repurposed during test time adaption in the proposed approach. And how many parameters do these self-attention layers account for? It is encouraged to supplement comparisons with the TTA baselines regarding the number of modulated parameters and the adaption time, which would make the comparisons more comprehensive.\n\n3. The results of the default setting can be added to the ablation tables for clear clarification. In the current form, the readers need to compare the main table (1,2,3) and the ablation tables for contrast.\n\n4. I carefully read the paper and found two potential mathematical typos. $\\theta_{s}^{m}$ in Line 5, page 4. $\\partial$ in Eq. 9.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZzCyWF5w52", "reviewer": "Reviewer_r3k4", "summary": "This paper delves into test-time adaption under the multi-modal setting and reveals an interesting and practical challenge, namely, reliability bias. In the wild, it is common that some modalities would suffer from distribution shifts compared to their counterparts in the source domain. As a result, the task-specific information across the modalities would be more inconsistent, thus contributing to the reliability differences for different modalities. Extensive empirical studies have been conducted to investigate the impact of the reliability bias using different cross-modal fusion strategies. To achieve robust multi-modal TTA against reliability bias,  the authors propose a novel method, dubbed reliable fusion and robust adaption (RFRA). Different from the existing TTA methods that mainly repurpose the normalization layers to achieve adaption, RFRA modulates the attention module to achieve reliable cross-modal fusion during test time. Besides, RFRA adopts a new objective function with desirable mathematical properties to combat with noise during adaption. To highlight the necessity of developing reliability-bias robust multi-modal TTA, the authors construct two new benchmarks with different settings of reliability bias based on the Kinetics and VGGSound datasets. Finally, the authors validate the effectiveness of the proposed method and give a deep analysis of the reliability bias challenge.", "questions": "My primary concerns revolve around the lack of comprehensive experiment details and the design of the modulation strategies, as highlighted in the weaknesses.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Revealing a new problem. This paper studies a new and practical challenge in the context of multi-modal test-time adaptation, namely, reliability bias. In the wild, it is evitably to introduce distribution shifts in some modalities.  As a result, the task-specific information across the modalities would be more inconsistent, thus contributing to the reliability differences for different modalities. The authors design extensive experiments to investigate the influence of reliability bias under different multi-modal fusion manners and the results validate the necessity of handling reliability bias for multi-modal TTA. I think the revealed challenge would bring some insights to the TTA community.\n2. Constructing meaningful benchmarks. To highlight the necessity of developing reliability-bias robust multi-modal TTA, the authors construct two new benchmarks with different settings of reliability bias based on the Kinetics and VGGSound datasets. Concretely, the benchmarks consist of both video and audio modalities and each modality is with corruptions of different levels so that the reliability bias is simulated. On the one hand, the corruption on video modality follows ImageNet-C, which ensures comparison fairness. On the other hand, the incorporation of audio corruption types extends the utility of this research to the audio domain with TTA. Besides,  the benchmarks encompass diverse multi-modal tasks, including action recognition and event classification, thus providing a comprehensive evaluation and supplementing existing multi-modal test-time adaptation tasks. \n3. Innovative paradigm for TTA. This paper proposes a new paradigm for TTA, namely, repurposing the attention layers during test time. Intuitively, the parameter updating of normalization layers like most existing TTA methods can only handle the distribution shifts between domains. In contrast, the parameter modulation of attention layers would help learn the importance difference between modalities, resulting in the reliable fusion for multi-modal TTA. The authors perform extensive experiments to show the superiority of the proposed new paradigm against reliability bias.", "weaknesses": "1. While the paper is well-written, it lacks essential details about CAV-MAE, such as the number of attention layers used for fusion. As many evaluations rely on the CAV-MAE framework, providing this information is crucial for the readers and reviewers to fully grasp the methodology. Additionally, visualizing the corruption types on both video and audio modalities would enhance the paper's clarity and help readers better understand the benchmarks.\n2. The paper introduces the modulation of Q, K, and V parameters to address reliability bias. However, it's not clear why the authors chose to update all parameters of Q, K, and V simultaneously. Explaining this choice and discussing the possibility of updating only the MLP parameters of Q and K would provide valuable insights. Furthermore, while the novel attention layer repurposing is effective for reliability bias, the paper should address whether this approach comes at the cost of efficiency for test-time adaptation.\n3. The paper introduces a crucial hyper-parameter, the confidence threshold ($\\gamma$) in Equation 6. While this threshold is fixed in all experiments, it's essential to include ablation studies to explore the sensitivity of the proposed method to variations in this hyper-parameter. A more comprehensive analysis would provide a deeper understanding of the method's robustness.\n\nMinor: Some of the figures, such as Figure 1c and Figure 3b, suffer from low image clarity. Improving the quality of these figures would enhance the paper's visual presentation and make the findings more accessible to readers.\n\nOverall evaluation, I think this paper is above the bar of ICLR, regarding motivation and novelty.\n\n\n----------------------------------\n\nUpon reviewing the response, I note that my concerns have been effectively addressed. Considering the consensus of other reviewers, I wholeheartedly recommend this work with a high level of approval.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BmHoYJ7HBA", "reviewer": "Reviewer_9FmQ", "summary": "This paper introduces a novel approach for addressing reliability bias in multi-modal test-time adaptation (TTA), a challenge arising from information disparities between modalities due to distribution shifts. To investigate the impact of reliability bias, the authors conduct comprehensive analyses involving various multi-modal fusion strategies and state-of-the-art TTA methods. The results underscore two pivotal aspects of effective TTA against reliability bias: dynamic information integration across modalities and noise-resilient adaptation across domains. To tackle these challenges, the authors devise a self-adaptive attention module to facilitate reliable cross-modal fusion and a confidence-aware loss function to ensure robustness against noisy predictions. Furthermore, this paper contributes two benchmark datasets focusing on multi-modal action recognition and event classification. Extensive comparison experiments against existing TTA methods and imbalanced multi-modal learning methods validate the effectiveness of the proposed method.", "questions": "My questions mainly lie in some unclear experiment analysis and  the generalizability of the proposed approach to a broader range of severity levels.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. This paper studies a new challenge (i.e., reliability bias) for multi-modal test-time adaption. Test-time adaption methods aim at adopting the pre-trained model from the source domain to the target domain in real-time and most existing of them focus on single-modality tasks against domain shifts. On the one hand, this paper takes the more complex multi-modal scenarios into consideration. On the other hand, the authors study and tackle the reliability bias challenge. \n\n2. The authors conduct extensive experiments to validate the importance of developing robust TTA methods against reliability bias. On the one hand, the existing cross-modal fusion methods (late fusion, attention-based fusion, etc.) would suffer from reliability bias and cannot achieve reliable cross-modal fusion. On the other hand, the existing TTA method cannot completely reconcile the distribution shifts by updating the parameters of normalization layers, leading to surviving reliability bias across modalities. Furthermore, the authors show that simply handling reliability bias during test time takes more superiority compared to the imbalanced multi-modal learning methods that alter the training process to handle the problem.\n\n3. The proposed method is novel and technically sound. First, the authors focus on the characteristics of multi-modal TTA and design the self-adaptive attention module that repurposes the attention layers during test time for achieving reliable cross-modal fusion. I think the design would inspire the community to design task-specific parameter modulation instead of solely updating the parameters of normalization layers following most existing methods. Second, to achieve robustness against heavy noise during adaption, the authors propose the robust loss function which not only eliminates the influence of noisy predictions but also boosts utilization of the clean predictions with theoretical guarantees.", "weaknesses": "Although this paper is well-motivated and extensively validated, I still have the following concerns or suggestions, hoping to make the paper more clear and solid.\n\n1. The experiment results are mainly obtained under the setting of severity 5. To establish the method's generality, it is encouraged to expand the empirical results across a spectrum of scenarios, including different severity levels. A broader array of experiments, encompassing various severity levels, would not only fortify the method's reliability but also enhance the comprehensiveness of this study. Moreover,  it would be beneficial to evaluate how the method performs in the context of test-time adaptation (TTA) under unbiased reliability conditions. Specifically, investigating the method's effectiveness in both (i.d.d. and non-i.d.d. scenarios would render it more practical and versatile.\n2. The paper contains extensive analysis and experiments, but some settings require further clarification. For instance, it's not entirely clear what \"Attention A-V\" means in Figure 4 and how the results demonstrate the method's robustness. In the analysis of Figure 5, claims are made about the importance of maintenance between audio and video modalities, but the contrast with clean results is not evident. Additional clarification or supplementary results are needed to support these claims.\n3. There are a few typos and vague statements in the paper, such as \"trafﬁc noise in the audio modality\" in the caption of Figure 1, \"information bias\" on Page 2, and inconsistent notations like \"(Stat. LN) & AF\" or \"Stat. (LN & AF)\" in Tables 1-3. These should be corrected for clarity and consistency.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "W5VxWNuHWT", "reviewer": "Reviewer_kzfx", "summary": "This paper studies the problem of multi-modal test-time adaption (TTA) under the challenge of reliability bias, which refers to the information discrepancies across different modalities due to the distribution shifts between domains. The paper proposes a novel method, RFRA, which consists of two modules: a self-adaptive attention module for reliable fusion across modalities, and a confidence-aware loss function for robust adaption. The paper also provides two new benchmarks for multi-modal TTA with reliability bias based on Kinetics and VGGSound datasets. The paper shows that RFRA outperforms several state-of-the-art TTA methods on these benchmarks under various corruptions.", "questions": "The primary questions for the rebuttal primarily arise from the \"weaknesses\" section. It would be highly appreciated if the authors could provide further explanations regarding the experiments and address the raised concerns, which will strengthen the paper. Overall, I recommend accepting this paper.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1. This paper is well-written, well-organized, and easy to follow.\n2. The paper addresses a novel and important problem, i.e.,  multi-modal TTA with reliability bias, which has not been well-studied in the literature.  Accordingly, the paper proposes an effective method, RFRA, which leverages self-adaptive attention and confidence-aware loss to achieve reliable fusion and robust adaption across modalities.   Moreover, the confidence-aware loss is simple but effective and enjoy the non-monotonous gradient property.\n3. The paper provides two new benchmarks for multi-modal TTA with reliability bias, which could facilitate future research on this topic. The paper conducts extensive experiments and ablation studies on these benchmarks to demonstrate the effectiveness and superiority of RFRA over existing TTA methods.", "weaknesses": "1. The paper lacks some experimental details, such as clarifying the terms “Statical” and “Dynamic” in Table 1 and explaining the distinctions between the proposed Self-Adaptive Attention and traditional self-attention. It is unclear what these terms mean and how they affect the performance of different methods. The paper should provide more definitions and discussions on these terms. Moreover, the paper should elaborate on how the Self-Adaptive Attention differs from the conventional self-attention in terms of design, implementation, and advantages.\n2. The confidence threshold $gamma$ work as the important parameter in the confidence-aware loss.  However, the influence of the confidence threshold $\\gamma$ on TTA performance is not explored, and it would be beneficial to understand its role more explicitly. The paper could conduct more experiments and analysis to show how different values of $\\gamma$ affect the accuracy and robustness of TTA. The paper could provide provide some insights on how to choose an appropriate value of $\\gamma$.\n3. How does the attention value in Figure 4 calculated? Some explanation is needed. The paper should provide more details on how to compute the attention value for each modality pair in Figure 4.\n4. An outlying problem, how does the audio information used in autonomous vehicle? As most scenarios of autonomous vehicle might use the visual sensor and could your method used in such case? \n5. There is a typographical error in Eq. (4); “A” should be bold. This is a minor mistake that can be easily corrected.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)