Smoothing the Shift: Towards Stable Test-Time Adaptation under Complex Multimodal Noises
Authors: Zirun Guo
Tao Jin
Venue: ICLR 2025 Poster (2025)
Keywords: Test-Time Adaptation
Multimodal Noises
Transfer Learning
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: We reveal a new challenge named multimodal wild test-time adaptation.

Abstract: Test-Time Adaptation (TTA) aims to tackle distribution shifts using unlabeled test data without access to the source data. In the context of multimodal data, there are more complex noise patterns than unimodal data such as simultaneous corruptions for multiple modalities and missing modalities. Besides, in real-world applications, corruptions from different distribution shifts are always mixed. Existing TTA methods always fail in such multimodal scenario because the abrupt distribution shifts will destroy the prior knowledge from the source model, thus leading to performance degradation. To this end, we reveal a new challenge named *multimodal wild TTA*. To address this challenging problem, we propose two novel strategies: sample identification with interquartile range **S**moothing and **u**nimodal assistance, and **M**utual **i**nformation sharing (SuMi). SuMi smooths the adaptation process by interquartile range which avoids the abrupt distribution shifts. Then, SuMi fully utilizes the unimodal features to select low-entropy samples with rich multimodal information for optimization. Furthermore, mutual information sharing is introduced to align the information, reduce the discrepancies and enhance the information utilization across different modalities. Extensive experiments show the effectiveness and superiority over existing methods under the complex noise patterns in multimodal data. Code is available at https://github.com/zrguo/SuMi.

Published as a conference paper at ICLR 2025
SMOOTHING THE SHIFT:
TOWARDS STABLE TIST-TIME ADAPTATION UNDER COMPLEX MULTIMODAL NOISES
Zirun Guo Tao Jin Zhejiang University
z zrguo. cségmail. com
ABSTRACT
Test-Time Adaptation (TTA) aims to tackle distribution shifts using unlabeled test data without access to the source data. In the context of multimodal data, there are more complex noise patterns than unimodal data such as simultaneous cor- ruptions for multiple modalities and missing modalities. Besides, in real-world applications, corruptions from different distribution shifts always mixed. Ex- isting TTA methods always fail in such multimodal scenario because the abrupt distribution shifts will destroy the prior knowledge from the source model, thus leading to performance degradation. To this end, we reveal a new challenge named multimodal wild TTA. To address this challenging problem, strategies: sample identification with interquartile range Smoothing and unimodal assistance, and Mutual information sharing (SuMi). SuMi smooths the adaptation process by interquartile range which avoids the abrupt distribution shifts. Then, SuMi fully utilizes the unimodal features to select low-entropy samples with rich multimodal information for optimization. Furthermore, mutual information shar- ing is introduced to align the information, reduce the discrepancies and enhance the information utilization across different modalities. Extensive experiments on two public datasets show the effectiveness and superiority over existing meth- ods under the complex noise patterns in multimodal data. Code is available at https://gil whwghu..hh........ com wrruo//uuuuu
propose two novel
INTRODUCTION
Deep learning has achieved remarkable success and has been widely adopted across a variety of applications (Touvron et al., 2023; Podell et al., 2024; Yan et al., 2024; Lin et al., 2024). How- ever, these models often struggle when faced with data distributions that differ from their training data. For example, in real-world scenarios, unexpected environmental changes and noises always occur such as weather changes and data corruption. When encountering such domain shifts, model performance can degrade rapidly (Hendrycks & Dietterich, 2019). To address this challenge, many adaptation techniques such as domain adaptation (Zhu et al., 2023) and domain generalization (Zhou al., 2023a) have been proposed enhance the robustness of models. One of the most challeng- ing settings is Test- Test-Time Adaptation (TTA) (Wang et al., 2021; Niu et al., 2022), where the model must adapt to a target domain without access to any source domain data and labels of target data. Recently, numerous promising test-time adaptation methods (Niu et al., 2023; Yang et al., 2024; Lee et al., 2024; Chen et al., 2024; Guo et al., 2024b) have shown great results.
However, the majority of existing TTA methods have focused on unimodal scenarios. In comparison to unimodal tasks, multimodal tasks often face more complex noise patterns, such as simultaneous noise corruption across multiple modalities or missing modalities. In this work, we broadly catego- rize multimodal noise scenarios into two types (shown in Figure (a)): weak cutttf-Diitribttion (OOD) samples, where only one modality is corrupted by noise, and strong OOD samples, where multiple modalities are corrupted by noise or missing modality issues occur. Considering a multi- modal sentiment analysis system, audio corruption can arise from factors like the speaker's 's accent,
"Corresponding author

---

Published as a conference paper at ICLR 2025
Tent
SAR
SOTTA
DeVO
CEMA
SuMi (Ouns)
Clean Data
Accuracy
40
Weak ODD Data
[b] Performance under four strong 000 sconarics
Wiaa
Strong OOD
Strong source
Tent
(Durs)
(c) Performance under weak 000 and strong 0OD sconarios
Strong OOD Data
Same distribution
Mixed distribution
Mixed source
Accuracy
40
20
(Durs)
Noise patterns of multimodal data
(d) Performence under same and mixeed distribution shifts
Figure 1: Illustration of our task where the target domain includes various domain shifts including weak OOD and strong OOD samples. The performances of existing methods degrade significantly on this challenging task, worse than the source model. We get these results Kineties50-C.
dialect, and background noise. Video corruption can occur due to lighting variations and diverse facial features. Furthermore, audio corruption can lead to transcription errors text, creating text noise and leading to simultancous domain shifts (strong OOD) across modalities. As shown in Fig- ure 1(b) and (c), we observe the performance of existing TTA methods can degrade significantly when faced with the more complex noise patterns encountered in multimodal scenarios, especially in the case strong OOD samples. The huge distribution gap between the source domain and the strong OOD data will damage the prior knowledge in the source model, thus leading to performance degradation. Additionally, in real-world dynamic environments where the target domain includes various types of distribution shifts (known as wild TTA), the performance of existing TTA methods always fail (Niu et al., 2023). To address the challenge in wild TTA, Niu et al. (2023) propose sharpness-awe aware and reliable entropy minimization method further stabilize TTA. However, as shown in Figure ((d), in the context of multimodal wild TTA where the target domain includes var- ious distribution shifts, the results are still not satisfying. recent work READ (Yang et al., 2024) explores the reliability bias in multimodal data during test time. READ proposes that when one of the modalities is corrupted, the reliability balance across the modalities will destroyed, which leads to a heavy performance degradation of the model. However, it only discusses the weak OOD situations and overlooks the more complex noise patterns in multimodal data. Besides, it is based on the mild TTA setting where test samples have the same distribution shift type.
Based on the above observations and the limitations of existing methods, in this paper, we reveal a new challenging task named multimodal wild TTA where the target domain includes various types of distribution shifts including weak OOD samples and strong OOD samples. To address the chal- lenging problem, we propose two novel strategies: sample identification with interquartile range Smoothing and unimodal assistance and Mutual information sharing (SuMi). To avoid the abrupt distribution shifts which could destroy the prior knowledge from the source model, we propose to smooth the adaptation process with interquartile range. Besides, we fully utilize the unimodal infor- mation select low-entropy samples with rich multimodal information. Furthermore, we propose the mutual information sharing to align information between different modalities which can reduce discrepancies across different modalities and enhance the information utilization of different modalities. Our main contributions can be summarized follows:
We show that the complex noise patterns in multimodal data will make existing TTA meth- ods fail. To this end, we propose new practical and challenging task named multimodal
2

---

Published as a conference paper at ICLR 2025
Unmoddal Assrtance
Iill Drill I Iill
Boraja"y piilett"
Video Engoder
Layens
Gu
Datele") e Sole) di 11 di
Layera
Area'
Overal Opcinizztoon
Encoder
Audio
Fusion Layers
Intefel Sele). fkll mill Thn Thil
Cie - tu popeeepee
Figure The overview of SuMi.
wild TTA where the target domain includes various types of distribution shifts including weak OOD samples and strong OOD samples.
We propose novel method SuMi, consisting of sample identification with interquartile range smoothing and unimodal assistance, and mutual information sharing.
SuMi outperforms all the baselines consistently and significantly in weak, strong and mixed OOD domains, Additionally, we build two benchmarks for multimodal wild TTA.
RELATED WORK
Test-time adaptation aims to update the source model without source domain data and labels of the target domain data. Test-time training, such TTT (Sun et al., 2020) and TTT+ (Liu et al., 2021) trains source model with both supervised and self-supervised objectives in the training stage to enhance test-time adaptation. These methods depend on proxy tasks and assume that the training process is controllable, which limits the scope of applications. Therefore, fully test-time adaptation methods (Wang et al., 2021; Niu et al., 2022; Zhou et al., 2023b; Yuan et al., 2023; Gong et al., 2023a; Park et al., 2024) are proposed to adapt the model only in test-time, without intervening in the training stage. Tent (Wang et al., 2021) proposes to entropy minimization normalization layers of the model. Furthermore, EATA (Niu et al., 2022) and SAR (Niu et al., 2023) propose the sample selection criteria for entropy minimization. More recently, Lee et al. (2024) show that using entropy alone as a measure of confidence is insufficient and propose to use a combination of entropy and the proposed PLPD metric to identify samples. Chen al. (2024) proposes dynamic unreliable and low-informative sample exclusion method for entropy minimization.
update the
However, existing works focus on the unimodal TTA. Compared to unimodal scenarios. multimodal data face much more complex patterns of noise in real-world applications, such as simultancous corruptions and missing modalities (Guo al., 2024c). Shin al. (2022) proposes framework to generate cross-modal pseudo labels as self-training signals. Guo et al. (2024b) propose mul- timodal TTA approach but focus on the multimodal regression tasks. recent work (Yang et al., 2024) explores the multimodal TTA and proposes reliable fusion and robust adaptation to address in- formation discrepancies in multimodal data. However, it only discusses the situations where there only one modality corrupted. When there are multiple modalities corrupted missing, the huge and abrupt distribution gap between the source domain and the target domain will make the method fail. Additionally, it focuses the single domain adaptation. In contrast, explore a more practical and challenging wild TTA where the target domain includes various types of corruption.
3 METHODOLOGY
3.1 PROBLEM FORMULATION
Without loss generality, we take two modalities as an example for clarity of presentation. Let Mg = =du, (uzy with parameter 0 be the source model pre- trained on the source domain dataset
D Dumuree {a;; y1) ) where Φuy Pug are the encoders of modality in and u2, F is the multi-
3

---

Published as a conference paper at ICLR 2025
modal fusion layers with prediction head, 2; (ax" [x and N₂ is the number of samples. TTA
aims to fine-tune the source model Ma on the target domain dataset D Diarget = {x } where the
labels and the source dataset are unavailable. Existing TTA methods (Niu et al., 2023; Yang et al.,
2024) update the parameter by minimizing the entropy of test domain data:
C
Enty (z) = -po(2) po(p)(() =
Σ po p)); log po(
(1)
(=1
where Po = (0f) = (o))))))) "............ (po())))....P(()) is the probabilistic distribution outputted by the model Ma and C is the number of classes.
In this paper, we reveal a new challenging task named multimodal wild TTA. Specificcll,. we broadly categorize multimodal noise scenarios into two types: weak OOD samples, where only one modality is corrupted by noise, and strong OOD samples, where multiple modalities are corrupted by noise or missing modality issues occur. Multimodal wild TTA considers a more practical and challenging TTA setting where the target datasets contain various types of distribution shifts including both weak OOD samples and strong OOD samples. The overall architecture of SuMi is presented in Figure 2.
3.2 SAMPLE IDENTIFICATION WITH INTERQUARTILE RANGE SMOOTHING AND UNIMODAL ASSISTANCE
3.2.1
INTERQUARTILE RANGE SMOOTHING
Many existing TTA methods rely on selecting low-entropy samples for entropy minimization (Niu et al., 2022; Lee et al., 2024; Chen al., 2024). However, when the target data is a mixture of various types of distribution shifts, including weak OOD samples and strong OOD samples, the performance of the model would degrade significantly. For example, in Figure 3(a), we present the performance of three different settings of the adaptation process. We can observe that directly adapting the model to the strong OOD domain will yield much poorer performance than a model adapted on weak OOD domain. The main reason is that there is a huge distribution gap between the source domain and the strong OOD domain. Therefore, a direct adaptation would destroy the prior knowledge of the source model and lead instability. In comparison, when first perform adaptation on the weak OOD domain before the strong OOD domain, the performance of the model will improve. This phenomenon inspires that a smoothing adaptation process under the wild TTA and complex noise patterns of multimodal data is much better than an abrupt adaptation process.
Motivated by the above observations, we propose an interquartile range smoothing method for dy- namic sample identification during the adaptation process. Interquartile range (IQR) is a measure of statistical dispersion, which is the spread of the data (Dekking et al., 2006). We give brief definition of IQR below:
range
Definition 1 IQR is the difference between the 75th and 25th percentiles of the data. The data is divided into four rank-ordered even parts via linear interpolation which are denoted as Q2 (lower quartile), Q2 (median) and Q3 (upper quartile). IQR is calculated as IQR = Q3 Q1-
IQR is often used identify unstable samples or outliers in a dataset. Specifically, according to Tukey's 's rule (Tukey et al., 1977), the stable sample set X₂ is selected as:
3 IQR and x S Q3 + 3 IQR)
X, fz| Q1
2
2
To smooth the adaptation process, we modify the above equation slightly and select the samples Hg (z) as:
2 [())QQ and S Q3 + 3 マ (())QR)
3
Hi (x) = (h 2 Q h "1 u0 h 2h Pu, (x:"), h² Quo (x:22
(3)
where t is the current iteration, is the parameter of the model, f(t) is the smoothing function, [.] is the concatenation operation and h is the representation of the sample. For simplicity, linear smoothing and set ft) = t /iteer where iter is the total iteration. We use the representations instead of the raw inputs because the representations are informative dense vectors that contain less
the
4

---

Published as a conference paper at ICLR 2025
30442
BMACSS dring
10.00
12242
14.00
120,00)
190300
(a)
(b)
(c)
Figure 3: (a) Performance different adaptation settings on strong OOD samples. (b) I-SNE visu- alizations (Van der Maaten & Hinton, 2008) of features during adaptation. (c) Performance using different quantiles multimodal and unimodal entropy. Results are obtained on Kinetics50-C.
noise and unrelated information than the raw inputs. At iteration t. we use the selected data H2 (x) for adaptation. In Figure 3(b), we visualize the sample identification process using the source model. From the figure, we can observe that at first several iterations, most weak OOD samples are selected. With the increase of i, the data for adaptation is also increasing, including more and more strong OOD samples. This smoothing process enables gradual adaptation to the strong OOD samples and various types of distribution shifts, avoiding the abrupt distribution gaps which could destroy the prior knowledge of the source model. Additionally, h is a vector. Therefore, in practice, we select for adaptation if 3 + (1 ff percent of the values in h satisfy Equation for stability.
3.2.2 UNIMODAL ASSISTANCE
IQR smoothing aims help the model preserve the prior knowledge in the source model and grad- ually adapt to the strong OOD samples and various types of distribution shifts. However, this pro- cess can not distinguish the high-quality samples that benefit the entropy minimization. Therefore, we introduce a novel sample identification method for multimodal data. As suggested in previous work (Niu al., 2022: 2023; Chen al., 2024), low-entropy samples will benefit the entropy mini- mization while high-entropy samples, due its uncertainty, will adversely affect the process. However, the context of multimodal data, apart from multimodal entropy, there is unimodal entropy we can utilize to help the adaptation. As shown in Figure 3(c), conduct experiments using unimodal en- tropy and multimodal entropy. We can easily observe that the multimodal low-entropy samples will yield much better performance than high-entropy samples. However, for audio and video modality, the samples of (20, 40] interval yield better results than samples of [0, 20). This indicates that for unimodal entropy, lower entropy does not necessarily correlate with better performance. When only one modality exhibits low entropy, it indicates that the sample does not rely on multimodal data for accurate prediction, implying has low informative value for multimodal optimization. Conversely, unimodal sample with slightly higher entropy shows dependence on multimodal data for accurate prediction, indicating that contains rich multimodal information.
Inspired by the above observations, we propose sample identification method with unimodal as- sistance to select low-entropy samples with rich multimodal information. Specifically, our method employs the following identification criteria:
Sa(x) {ær Ento(x)) S 7/ and (Ents(x")) ) + wuEnt 2 2m3
(4)
where "/m and Yuu are the pre-defined threshold for multimodal and unimodal entropy and is trade- select low-entropy samples with
off between modalities. By limiting the multimodal entropy, high certainty and fewer noises. Meanwhile, by limiting the unimodal entropy, we can ensure the samples selected contain rich multimodal information, excluding low-informative samples.
3.3 MUTUAL INFORMATION SHARING
A recent study (Yang al., 2024) reveals a challenge in multimodal TTA, known as reliability bias, which refers to the information discrepancies across different modalities. During the adaptation process, this discrepancy often leads to imbalanced utilization of each modality (Guo et al., 2024a). In strong OOD situations, missing modality cases could occur or multiple modalities could be cor- rupted. Therefore, the imbalanced phenomenon could be enlarged. How to balance the adaptation
5

---

Published as a conference paper at ICLR 2025
Algorithm SuMi
1: Input: Source model Ma (dur F), target dataset D Diarget
{x: J adaptation
iterations T and a series of hyperparameters.
2: for t to T do
Sample
3: x (x x=
tiargeti
4: Calculate the representations h 11 2 h Uuz (x*11.
h22 h22 = Ous (x*2);;
5: Calculate Q1 and Q3: Q1 quantile 0.25). Q3 quantile(h. 0.75);
6: Calculate IQR: IQR = Q3 Qı:
7:
Select samples Hg (æ: using Equation 3;
8:
Calculate entropy of multimodal outputs and unimodal outputs;
9:
Select samples (x) from (a) using Equation 4;
10:
Calculate the entropy in Sar);
11:
if < to then
12:
Caleulate mutual information sharing loss using Equation 6;
13:
end if
Calculate the loss L(x) using Equation 8;
15: Update the affine parameters of the model Ma;
end for
across different modalities under strong OOD domains (especially missing modality cases) is very important. To address this problem, we propose a simple yet effective method, mutual information sharing, align information between different modalities. Concretely, for modality u, can obtain its probabilistic distribution as P (æ* = softmax(F(áu.. (x**)), For simplicity, we will p represent Pg (ar") in the following context. We define the complementary probabilistic distribution of p as
M 1 - -P'--"" 5=1
p U.
(5)
where M is the number of modalities. For two modalities, pu 2 and p p To improve the alignment between different modalities, we can minimize the KI. divergence (Kullback & Leibler, 1951) between the probabilistic distribution pd and its complementary distribution p" However, if one modality is severely corrupted, minimizing the KL divergence might influence the clean modality. Therefore, add multimodal distribution p' softmax (ManaaxxMMMGa (M (x)) to improve the robustness and stability. Therefore, we can represent the mutual information sharing loss as:
1 Liii = DKI(p 1 (p"" + +p")) + (Rr.
1 +p"))
2
2p
2p
Σ
Pi log
Σ 2 log
P + P.
P.
+ p.
where C' is the number of classes and p: is the i-th value of p. Mutual information sharing can help the model connect and align the information between different modalities. Through mutual information sharing, when there are corrupted modalities including missing modalities, information from other modalities could be utilized to enhance the predictions.
3.4 OVERALL OPTIMIZATION
Following previous TTA methods (Niu et al., 2022; Lee et al., 2024), we add weighting term emphasize the contributions of samples during adaptation. Specifically, the weighting term is calculated as
org (<( exp[Ento(z)) Entp]
where Ento is pre-defined normalization factor (Niu et al., 2022). In summary, we can denote the overall loss function as:
L(x) = (e0 (z) + ALmis
(8)

---

Published as a conference paper at ICLR 2025
Table 1: Accuracy comparison with SOTA methods on Kinetics50-C with corrupted video modality (severity level 5). We report a over five random seeds, Bold: best results. Underline: second best results.
Noise
Weather
Digital
Gunss.
Shot
Impul.
Defoc:
Cllass
Motion
7oom
Snow
Frout
Fog
Brit.
Cietr
Flastic
Pixel
JPEC
Avg
Source
47.7
28.5
47.4
674
61.0
21.1
66.1
60.77
62.1
45.5
75.9
31,9
65.1
67.8
65.7
60.2
Tent
49.3
48.4.15
67.84a1
62.2435
71.9.a
67.8.1
63.
63.5.45
23.0azs
50.l
67 67,8.4
70.5.e:
59.8
EATA
49.0asa
50.3
49.2,1:
67.9.a
71.6.a3
63.4as
64.4.z3
47.0aa3
52.2as3
70.3.ss
67.9.13
61.9
48.6u33
49.8ass
48.7a31
67.9
62.7a34
71.8a33
67.9.1
63.4x44
64.2,13
24.0aas
75.9
S1.1a
08.0a.
70.6uss
67.2a13
60.1
SoTTA
48.3433
49.8.33
48.5..1
67,94s
62.5433
21.9-1
67.8.1
63:2-a:
64.0.a
27.6
75.2413
51.3.e1
67.8.3
20.4.45
67.5.11
60.3
DeYO
48.5.33
49.8.sc
48.7.14
68.0.11
63.0aa4
71.9.a
68.1
63.5.t4
64.4.z1
21.4
75.9
50.6.s
68.6Gz1
70.8.es
67.5a13
60.0
CEMA
48.4a33
49.4.a5
67.8.13
62.7.a4
71.7
67.811
63.4.14
64.2.1
22.8aas
75.7a24
50.7,13
67.9a1
70.5.s2
67.3,13
READ
13.
50.8.05
19.8.17
67.9.t5
65.1
72.2.s:
09.
61.8.s
66.2.m3
56.8.1
76.2:3
548.14
68.
70.7.e:
68.9:1
63.5
SuMi
30.1.34
30.7u83
30.4;31
68.2
65.0g33
72.2
09.7224
63.7
67.0ga3
56.5.s3
77.1
35.2.14
09.3g13
71.2,z2
68.9;12 63.9
Table 2: Accuracy comparison with SOTA methods on Kinetics50-C with corrupted audio modality and strong OOD scenarios (severity level 5).
Noise
Weather
Strong
OOD
Gauss.
Traff.
Crowd
Rain
Thund.
Wind
Avg.
Both
Vmiis
Amiss
Mix
Avg.
Source
74.9
65.4
67.9
70.0
68.5
70.7
69.6
30.8
27.9
44.5
16.9
30.0
74.8.05
68.2ies
20.3 m0
71.4.01
66.7.os
71.7-o1
13.1aca
92.s:
21.3.a
1.2ss
11.2
EATA
74.9
68.01ez
70.0aea
71.2
70,0203
71.
70.9
32.3
28.61a3
45.3
119.13
30.3
SAR
74.8
68.4.aa
20.3.ag
71.240:
68.9.01
71.9.m
70.9
13.6.r
10.3,e:
23.5
3.8.45
12.8
ScTTA
74,8
68.41e4
70.1.aa
71.aaa
69.2na
7188.a
70.9
14.1aca
98.s
24.1aaa
19aas
12.5
DeYO
74.8
68.6.ea
20.3.ea
71.3205
70.4.03
72.0.m.
71.2
14.9
12.
27.6
2.1
14.2
74.8
67.8.e4
69.5.e
20.5.ea
71.6.a
169
1344e:
30.3
18,a6
15.6
READ
24.9
69.1
20.3.a2
714ma
72.8.0a
71.3
31.6
31.
27.5
44.3aa5
13.7
29.1
SuMi
75.
68.9
70.6.0
716621
72.8 72.
71.9
34,
388
48.6
18.4.14 33.4
Table Accuracy comparison with SOTA methods on VGGSound-C with corrupted video modality (severity level 5).
Noise
Blur
Weather
Digital
Guuss.
Shot
Impul.
Defoc.
Glass
Motion
Zoum
Snow
Frost
Fog
Bri.
Contr.
Blasic
Pisel
JPEG
Avg.
Sounce
53.0
52.9
53.0
57.2
57.3
58.6
57.5
56.3
56.5
55.
59.2
53.7
57.2
5644
57.3
56.1
Tent
53.0za:
53,2
52.9
56.3111
56.3
57.6.83
56.8
55.41e3
56.02a3
56.2.s1
58.3,1:
53.5.a3
57.3
56.7
56.8113
55.8
EATA
53.S.s:
53.7
53.S.s:
37,
57.
58
38.2
57.7
56.0
56.6111
36.7en
59...:
54.3-a1
18.1
57.3
57.5.:1
56.5
SAR
52.9
23.1
$2.9.1:
56
56.3
56.211
57.4
56.71:
55.4113
56.Das
56.21s
58.2
53.5
57.4
56.721
$6.111
55.7
ScTTA
32.9.;: 53.2
52.9.;
56.6213
568811
57.9,32
43.
55.2213
56.123
5633,3
$9.4,33
S38.13
57.6
56.2
56..13
55.9
DeYU
33.0).;: 53.1.31
56.5.1
56.5.3
53.7
55.4.c1
56.0.a3
56.3.93
S3.6.11
57.6
57.0.es
57.0.:3
CEMA
52.8.a
52.94a3
$2.9.s
56.5.
56.4.3
57.6
56.8.1
53.4.1
56.24.1
56.2441
58.4.:
53,5.41
57.8
56.8.e8
56.0.:
55.8
READ
52.9.s:
52.8
$2.8.
57,2411
57.3.15
58.8433
38.1413
56.4.11
57.5.45
$7.4.41
20.3.1
544.45
578
56.6.41
57.2.13
56.4
SaMi
54.0.,
$4.3
5388
58.2.1
584..
59.4..3
5677
57.5..1
58.2.
57.6..
$9.4.;
548..1
59.0
57.5.e1
58.2.;
57.3
where (-) is an indicator function and À is a trade-off between the two losses. One point worth emphasizing is that for strong OOD adaptation, we only add the mutual information sharing loss Liii in the first to iterations during the adaptation process. The reason is that with the increase of iteration, the IQR smoothing will include more and more strong OOD samples where multiple modalities are corrupted which could damage the information sharing and the performance of the model. For weak OOD adaptation, we add mutual information sharing loss for all the iterations. Overall, Algorithm 1 presents the outline of our method.
4 EXPERIMENTS
4.1 EXPERIMENTAL SETTINGS
Datasets. We use two widely used multimodal datasets, Kinetics50 (Kay et al., 2017) and VG- GSound (Chen et al., 2020) for evaluation. Following previous work (Hendrycks & Dietterich, 2019; Yang et al., 2024), we introduce 15 different types of corruptions and 6 types for audio to simulate the distribution shifts in real-world applications. Each type of corruption has five levels of severity. For strong OOD samples, we introduce four different types: Both (both modalities are corrupted), Vmiss (video modality is missing), Amiss (audio modality is missing), and Mix (one modality is missing and the other is corrupted). Details of datasets and the corruptions are presented in Appendix A. As result,
obtain the corrupted datasets Kinetics50-C and VGGSound-C.
Implementation Details. For the source model, we use the pre-trained CAV-MAE (Gong et al., 2023b) following Yang et al. (2024). We use Adam optimizer with learning rate of le-4/1e-5 and
7

---

Published as a conference paper at ICLR 2025
Table 4: Accuracy comparison with SOTA methods VGGsound-C with corrupted audio modality and strong OOD scenarios (severity level 5). We report avg Over five random seeds. Bold: best results. Underline: second best results.
Noise
Weather
Strong OOD
Gauss.
Traff.
Crowd
Rain
Thund.
Wind
Avg.
Both
Viisss
Amiss
Mix
Avg.
Source
37.2
21.2
16.9
21.3
27.4
25.6
25.0
9.4
28.0
18.9
6.0
15.6
6.0.ea
L6
1.1
1.7.00
3.200
2.3.01
2.6
0.8
18.3.gs
1.0.00
0.1.os
EATA
112
25.0-as
28,8
32.3,34
34,5m02
33.2=ms
32.5
[5.2p04
29.2-1
19.6
5.8
17.4
SAR
10.9.07
2.L.a3
1.0.s0
2.0.a0
3.2.0
2.3-01
3.6
1.1400
19.8.
1,5400
0.3.os
5.7
SoTTA
13.8
10.1in3
8.4us
4.2ans
6dusa
3...1
7.7
24
20.4an3
4/4aa1
1.1.08
7.1
DeYO
7.0zas
1,5.03
2.Zasa
3.5an3
68ana
4,2m1
4.2
0.6ar
20.8a1
288na
0/Aas
6.2
CEMA
68808
1.9
200aa
2.9m13
44s02
39
3.7
0.920
19.
1.9481
0.1g00
5.7
READ
27 2nd
22.
19.0am2
21.6,a5
23.6114
21.0anz
22.4
10.
27.9
15.3
4.5
14.5
SuMi
41.9
26.
27.9a02
31.6,as
37.1gng
24.1
33.2
184ggg
31.8
21.7
6.7
19.7
158
194M
(a) Kinetics50 (weak)
(b) Kinctics50 (strong)
(c) VGGSound (weak)
(d) VGGSound (strong)
Figure 4: Comparison with SOTA methods on corrupted data of different severity levels. weak:
average accuracy of 21 different types of weak OOD distribution shifts. strong: average accuracy of 4 different types of strong OOD distribution shifts.
2312
(a) Kinetics50 (sever. 5) (b) VGGSound (sever. 5) (c) Kinetics50 (mixed)
(d) VGGSound (mixed)
Figure 5: Comparison with SOTA methods on mixed corrupted data with ten different ratios of strong OOD samples. (a) and (b): severity level 5. (c) and (d): mixed severity.
batch size of 16/64 for Kinetics50-C and VGGSound-C, C, respectively. The multimodal threshold Ym in Equation and the normalization factor Ento in Equation 7 are set to 0.4 x In C following Niu et al. (2022) by default where C' is the number of task classes. The unimodal threshold % in Equation 4 set to e- by default. The smoothing coefficient /3 is set to 0.6/0.9, the weighting term À is set to 5.0 and the unimodal assistance μ is set to 1.0 by default for Kinetics50-C and VGGSound-C. For strong OOD adaptation, we set the mutual information sharing term 40 as iter /2. Following previous work (Niu al., 2023; Gong et al., 2023a; Chen et al., 2024; Guo et al., 2024b), we update the affine parameters of normalization layers.
4.2 COMPARISON WITH STATE-OF-THE-ARTS
We compare our method with Tent (Wang et al., 2021), EATA (Niu et al., 2022), SAR (Niu et al., 2023), SoTTA (Gong et al., 2023a), CEMA (Chen et al., 2024), DeYO (Lee et al., 2024) and READ (Yang et al., 2024).
Single Domain Results. We report the accuracy of 21 different types of weak OOD corruptions and 4 different types of strong OOD corruptions at severity level 5 on Kinetics50 and VGGSound in Table 2, and 4. For weak OOD samples, SuMi outperforms existing SOTA methods on most of the distribution shifts and achieves consistent good performances. For strong OOD samples, most of the existing SOTA methods perform even worse than the source model. On both datasets, only EATA performs better than the source model slightly. On the most noisy distribution "Mix" where
8

---

Published as a conference paper at ICLR 2025
SSurre TAME BFATA bwid 18T1 RMM
60
70
30
Deticc
Thandl
Wed
Video Corruptions
(b) Audio Corruptions
(c) Strong OOD
Figure 6: Comparison with SOTA methods on mixed severity level on Kinetics50-C.
one of the modality is missing and the other is corrupted, EATA also has a performance degrada- tion, performing worse than the source model. In comparison, SuMi outperforms other methods consistently and significantly on all the four distribution scenarios, indicating its effectiveness and superiority in dealing with the complex noise patterns in multimodal data. Furthermore, we com- pare SuMi with SOTA methods at different severity levels and present the results in Figure 4. From the figure, we can observe that at different severity levels, most of the methods can work well on weak OOD samples while fail on strong OOD samples. However, SuMi can still perform well and achieve the best results on corrupted datasets at all the four severity levels, which demonstrates its generalization ability.
Mixed Domain Results. In Figure 5, we present the results of different methods on datasets with ten different portions of strong OOD samples. Figure 5(a) and 5(b) presents the results at severity level 5. We can observe that all the methods can perform well when the ratio of strong OOD samples low. However, with the ratio increasing, the performance of most of the methods degrade rapidly, performing worse than the source model. The reason is that the huge distribution gap between the source domain and strong OOD domain destroy the prior knowledge of the source model, thus leading to degradation of the model. In comparison, SuMi smooths the process by interquartile range smoothing and outperforms the SOTA methods consistently. From Figure 5(c) and 5(d) where mixed severity level cases are added, we can reach the same conclusion. Moreover, in Figure 6 and 10, we present the results on corrupted data with mixed severity level samples on both datasets. From the table, provements, outperforming other SOTA methods in most of the distribution shifts. Additionally, on strong OOD distribution shifts, other methods always fail while SuMi can still perform well. These results indicate the effectiveness of SuMi.
observe that mixed severity level, SuMi still achieve consistent im-
4.3 ABLATION STUDY
Contributions of different components. In Table 5, we present the results of our ablation exper- iments. We can observe that IQR smoothing brings the most improvements to the model. This because IQR smoothing can bridge the gap between the source domain and strong domain, avoid- ing the abrupt distribution shifts which could destroy the prior knowledge from the source model. Unimodal assistance aims to select low-entropy samples with rich multimodal information for opti- mization and can also enhance the performance of the model. With these strategies combined, the performance of the model is further enhanced.
Exploration of unimodal assistance. Equation 4 and Figure 2, we divide the samples into four areas. We can consider the four areas Area 1 (low-entropy samples with rich mul- timodal information), Area 2 (high-entropy samples with rich multimodal information), Area 3 (low-entropy samples with little multimodal information) and Area 4 (high-entropy sam- ples with little multimodal information). We present the performance of these four areas on Kinetics50-C in Table ples for optimization will yield better results. Based on low-entropy samples, rich multimodal information will further help to optimize the multimodal models and achieve better results.
From the table, we can observe that selecting low-entropy sam-

---

Published as a conference paper at ICLR 2025
Table 5: Ablation study of different components in SuMi comrupted data with 50% of strong OOD samples at different severity levels. IQR, UA and MIS represents IQR smoothing, unimodal assistance and mutual information sharing, respectively.
Kinetics30-C
VGGSound-C
IQR
UA
MIS
severity 3
severity
mixed severity
severity
severity
mixed severity
37.1
31.7
36.4
25.7
23.5
25.3
52.1
45.1
$1.9
33.8
30.4
33.1
49.4
39.4
46.2
31.1
27.4
31.2
47.4
38.1
45.6
29.8
26.1
28.4
58.0
51.2
$7.4
36.9
56.0
49.7
56.7
34.2
32.1
34.0
54.3
44.6
51.3
33.4
29.8
32.1
59.3
59.1
35.1
60
Kinelika
VGdSound
45
90
0.8
0.9
1.2
(a) Impact p
(b) Impact of X
Figure 7: Ablation experiments of hyperparameters on the two datasets.
Additionally, we explore the trade-off coefficient μ Equation 4 and present the results on both datasets in Figure 7(a). From the figure, we can observe that with the increase of μ. the performance improves on Kinetics50-C and drops on VGGSound-C. This is because Kinet- ics50 a video modality dominant dataset and VGGSound is an au- dio modality dominant dataset. From the results, know that adding more weight to the dominant modality will yield poorer performance because unimodal assistance aims select samples with rich multi- modal information. Therefore, adding weight to the weak modality will help utilize the multimodal features. Besides, the performances with different are stable, indicating the stability of the strategy.
Table Performance of samples in different areas on Kinetics50-C.
Ratios
Arca
13.1%
39.4
Area
78.5%
27.6
Area
5.3%
32.1
Area 4
3.1%
24.3
Exploration of X. To explore the mutual information sharing, select several À in Equation 8 and present the results on both datasets in Figure 7(b). We can observe that increasing the weight term will improve the performance slightly. Besides, the results in the table demonstrate the stability of mutual information sharing across varying values of A.
CONCLUSION
In this paper, we propose a new practical and challenging task named multimodal wild TTA. To address this problem, we propose sample identification with interquartile range smoothing and uni- modal assistance, and mutual information sharing (SuMi). SuMi bridges the gap between the source domain and strong OOD domain by smoothing the adaptation using interquartile range. Besides, SuMi leverages unimodal features to select low-entropy samples with rich multimodal information for optimization. Finally, mutual information sharing is proposed to further align the information and reduce the discrepancies across different modalities. We conduct extensive experiments on two widely used multimodal datasets where SuMi outperforms existing TTA methods significantly and consistently, indicating its effectiveness. Ablation experiments are then conducted to validate the contributions of each component.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENT
This work was supported in part by Public Welfare Research Program of Ningbo under Grant No. 2024S062 and Yongjiang Talent Project of Ningbo under Grant No. 2024A-161-G.
REFERENCES
Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. Vggsound: large-scale audio- visual dataset. ICASSP 2020 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 721-725 2020.
Yaofo Chen, Shuaicheng Niu, Yaowei Wang, Shoukai Xu, Hengjie Song, and Mingkui Tan. To- wards robust and efficient cloud-edge elastic model adaptation via selective entropy distilla- tion. The Twelfth International Conference on Learning Representations, 2024. URL
https: wttp//eeeeeeeeeeeeeeeeeeeeeeeeee net wwt.:/eeettdddeeeeeeeeeeeeeeeeee [orumtld=vePaNu3uee
Frederik Michel Dekking, Cornelis Kraaikamp, Hendrik Paul Lopuhaä, and Ludolf Erwin Meester. A Modern Introduction to Probability and Statistics: Understanding why and how. Springer Science & Business Media, 2006.
Taesik Gong, Yewon Kim, Taeckyung Lee, Sorn Chottananurak, and Sung-Ju Lee. SoTTA: Robust test-time adaptation noisy data streams. In Shirty-seventh Conference Neural Information Processing Systems, 2023a.
Yuan Gong, Andrew Rouditchenko, Alexander H. Liu, David Harwath, Leonid Karlinsky, Hilde Kuehne, and James R. Glass, Contrastive audio-visual masked autoencoder. In The Eleventh International Conference on Learning Representations, 2023b. URL httpps://ppeneraiie
net/forum? wwwpdppPPMRRkkk
Zirun Guo, Tao Jin, Jingyuan Chen, and Zhou Zhao. Classifier-giided gradient modulation for enhanced multimodal learning. In The Thirty-eighth Annual Conference Neural Information Processing Systems, 2024a.
Zirun Guo, Tao Jin, Wenlong Xu, Wang Lin, and Yangyang Wu. Bridging the gap for test-time multimodal sentiment analysis. arXiv preprint arXiv:2412.07121,, 2024b.
Zirun Guo, Tao Jin, and Zhou Zhao. Multimodal prompt learning with missing modalities for sentiment analysis and emotion recognition. In Lei Ku, Andre Martins, and Vivek Sriku- mar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pp. 1726-1736. Bangkok, Thailand, August 2024c. As- sociation for Computational Linguistics. doi: 10. 18.18653/v 111865//1/2024.acc 1/2024.acl-loog.9999 URL https: / //aclanthology. org/ 22242222222222222 acl-long. 94.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common ruptions and perturbations. In International Conference on Learning Representations, 2019, URL https: wwwpreview..weccc//ep .et/ foorumiid-H-a H..dHHHHH666 iC
Will Kay, João Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijaya- narasimhan, Fabio Viola, Tim Green, Trevor Back, Apostol Natsev, Mustafa Suleyman, and An- drew Zisserman. The kinetics human action video dataset. ArXiv, 0tos//705.06955 2017.
Solomon Kullback and Richard A Leibler. On information and sufficiency. The annals of mathe- matical statisties, 22(1):79-886 86, 1951.
Jonghyun Lee, Dahuin Jung, Saehyung Lee, Junsung Park, Juhyeon Shin, Uiwon Hwang, and Sun- groh Yoon. Entropy is not enough for test- adaptation: From the perspective of disentangled factors. The Twelfth International Conference on Learning Representations, 2024.
Wang Lin, Jingyuan Chen, Jiaxin Shi, Zirum Guo, Yichen Zhu, Zehan Wang, Tao Jin, Zhou Zhao, Fei Wu, YAN Shuicheng, et al. Action imitation in common action space for customized action image synthesis. The Thirty-eight Anmual Conference on Neural Information Processing Systems, 2024.
11

---

Published as a conference paper at ICLR 2025
Yuejiang Liu, Parth Kothari, Bastien van Delft, Baptiste Bellot-Gurler Taylor Mordan, and Alexan- dre Alahi. Tit++: When does self-supervised test-time training fail or thrive? In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 21808-21820. Curran Associates, Inc., 2021.
Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Yaofo Chen, Shijian Zheng, Peilin Zhao, and Mingkui Tan. Efficient test-time model adaptation without forgetting. In International conference on machine learning, pp. 16888-16905. PMLR, 2022.
Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Zhiquan Wen, Yaofo Chen, Peilin Zhao, and Mingkui Tan. Towards stable test-time adaptation in dynamic wild world. In The Eleventh International Conference on Learning Representations, 2023. URL htthttps:// oppnrevieweeeeeeeeeeeee net forum?
id=g2YzaF75TJ.
Hyejin Park, Jeongyeon Hwang, Sunung Mun, Sangdon Park, and Jungseul Ok. Medbn: Robust test-time adaptation against malicious test samples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5997-6007, 2024.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhom, Jonas Müller, Joe Penna, and Robin Rombach. SDXL: Improving latent diffusion models for high-resolution image synthesis. In The Twelfth International Conference Learning Representations, 2024. URL petps://openreett wtt.:eiwwwweeeeeeeeeeeee..e.eeee wtw.eeeennet/oooiddddiedde.
Inkyu Shin, Yi-Hsuan Tsai, Bingbing Zhuang, Samuel Schulter, Buyu Liu, Sparsh Garg, In So Kweon, and Kuk-Jin Yoon. Mm-tta: multi-modal test-time adaptation for 3d semantic segmenta- tion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 16928-16937, 2022.
Yu Sun, Xiaolong Wang, Liu Zhuang, John Miller, Moritz Hardt, and Alexei A. Efros. Test-time training with self-supervision for generalization under distribution shifts. In ICML, 2020.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023.
John Wilder Tukey et al. Exploratory data analysis, volume Springer, 1977.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008
Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. In International Conference Learning Repre- sentations, 2021. URL http::::///peneeeew..eeee./
net (ot
Xin Wang, Devinder Kumar, Nicolas Thome, Matthieu Cord, and Frédéric Precioso. Recipe recog- nition with large multimodal food dataset. In 2015 IEEE International Conference on Multimedia and Expo Workshops (ICMEW), pp. 1-6, 2015. doi: 10.1 10.1199/CMEE.2201..71055
Weicai Yan, Ye Wang. Wang Lin, Zirun Guo, Zhou Zhao, and Tao Jin. Low-rank prompt interaction for continual vision-language retrieval. In Proceedings of the 32nd ACM International Conference on Multimedia, pp. 8257-8266, 2024.
Mouxing Yang, Yunfan Li, Changqing Zhang, Peng Hu, and Xi Peng. Test-time adaptation against multi-modal reliability bias. In The Twelfih International Conference on Learning Representa- tions, 2024.
Wenmeng Yu, Hua Xu, Fanyang Meng, Yilin Zhu, Yixiao Ma Jiele Wu, Jiyun Zou, and Kaicheng Yang. CH-SIMS: A Chinese multimodal sentiment analysis dataset with fine-grained annotation of modality. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 3718-3727, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/0220.au 343. URL https: / antho wow.. org 2020 aall-mainnnmmmmmmm n.
1/2020.acl main.
12

---

Published as a conference paper at ICLR 2025
Longhui Yuan, Binhui Xie, and Shuang Li. Robust test-time adaptation in dynamic scenarios. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp- 15922- 15932, 2023.
Amir Zadeh, Rowan Zellers, Eli Pincus, and Louis-Philipee Morency. Multimodal sentiment in- tensity analysis in videos: Facial gestures and verbal messages. IEEE Intelligent Systems, (1))); 82-88 88, 2016. doi: 10.1109/MIS.2016.90
Kaiyang Zhou, Ziwei Liu, Yu Qiao, Tao Xiang, and Chen Change Loy. Domain generalization: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(4):439644455,
2023a. doi: 10.1 209/
Zhi Zhou, Lan-Zhe Guo, Lin-Han Jia, Dingchu Zhang, and Yu-Feng Li. Ods: Test-time adaptation in the presence of open- world data shift. In International Conference Machine Learning, pp- 42574-42588, PMLR, 2023b.
Jinjing Zhu, Haotian Bai, and Lin Wang. Patch-mix transformer for unsupervised domain adapta- tion: A game perspective. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recog- nition (CVPR), pp. 3561-35711 2023.
13

---

Published as a conference paper at ICLR 2025
Original
Brightness
Contrast
Defocus Blur
Elastic
Fog
Frost
Gaussian
Glass
Impulse
JPEG
Motion
Pixel
Shot
Snow
Zoom
Figure 8: Fifteen different types of noises in videos.
A DETAILS OF DATASETS
Kinetics50 (Kay et al., 2017). The Kinetics dataset is large-scale and high-quality dataset for human action recognition in videos. The dataset consists of around 500,000 video clips covering 600 human action classes with least 600 video clips for each action class. Each video clip lasts around 10 seconds and labeled with single action class. The videos are collected from YouTube. Following Yang et al. (2024), training pairs and 2,466 test pairs
subset of Kinetics which consists of 50 classes, 29,204
VGGSound (Chen et al., 2020). VGGSound is a large-scale audio-visual correspondent dataset consisting of short clips of audio sounds, extracted from videos uploaded to YouTube. All videos captured "in the wild" with audio-visual correspondence in the sense that the sound source is visually evident. Each video in this dataset has a fixed duration of 10 seconds.
To evaluate the performance under different distribution shifts, we introduce total of 25 different types of distribution shifts. These distribution shifts can be divided into two groups: weak OOD distribution shifts and strong OOD distribution shifts.
For weak distribution shifts, we divide them into video corruptions and audio corruptions. Following previous work (Hendrycks Dietterich, 2019), we introduce 15 different types of video corruptions as shown in Figure 8. They include "Gaussian Noise" (Gauss.), "Shot Noise" (Shot), "Impulse Noise" (Impul.), "Defocus Blur" (Defoc.), "Glass Blur" (Glass), "Motion Blur" (Motion), "Zoom Blur" (Zoom), "Snow" (Snow), "Frost" (Frost), "Fog" (Fog), "Brightness" (Brit.). "Contrastive" (Contr.), "Elastic" (Elastic), "Pixelate" (Pixel) and "JPEG" (JPEG). Following Yang et al. (2024), introduce six types of audio corruptions shown in Figure 9. They include "Gaussian Noise" (Gauss.), "Paris Traffic Noise" (Traff.). "Crowd Noise" (Crowd), "Rainy Noise" (Rain), "Thunder Noise" (Thund.) and "Windy Noise" (Wind).
For strong distribution shifts, in this paper, we introduce four types of corruptions. They include "Both Modality Corruptions" (Both), "Audio Missing" (Amiss), "Video Missing" (Vmiss) and
14

---

Published as a conference paper at ICLR 2025
Clean audio
Gaussian noise
Rain
Crowd
204
Thunder
Traffic
Wind
Figure 9: Six different types of noises in audio.
50
60
50
10
(a) Video Corruptions
(b) Audio Corruptions
(c) Strong OOD
Figure 10: Comparison with SOTA methods on mixed severity level on VGGSound-C.
"Missing and Corruption" (Mix). Both represents the both modalities are corrupted. Mix represents that of the modality is missing and the other is corrupted. For missing modality, substitute any missing modalities with zero vectors. This allows us to maintain the input dimensions required by the network while enabling to process the available data effectively.
B DETAILS OF METHOD AND IMPLEMENTATION
IQR can effectively capture the central tendency and variability of the data. For IQR calculation, we can use different metrics for data ranking such as magnitude (Fuclidean Norm) and specific dimension comparison. However, there are some drawbacks of these metrics. For example, large components in the vectors will disproportionately affect the magnitude. Specific dimension com- parison ignores other dimensions which may be important and does not represent the overall vector well. To combine multiple dimensions, we calculate the min and max all the h element by element to obtain hmin and hmaz- Then, obtain the Q1 and Q3 through linear interpolation. Addition- ally, we follow previous work (Yang et al., 2024) and add a negative entropy loss term to balance the prediction.
15

---

Published as a conference paper at ICLR 2025
850
Stume FATA
RRAD
6SM
81.0
Souce &CATA READ a5.M
7200
Souree
83.0
80.0
23.0
76.0
12.0
71.
800
64.0
60.0
75.0
760
200n
FoR
CanT
Correst
(a) Video corruption
(b) Audio corruption
[e) Text corruption
(d] Strong OOD
Figure 11: Performance comparisons CMU-MOSI.
65.0
READ
85.0
Source EATA READ SUMI
80.0
85.0
05.0
55.0
$0.0
35.0
20.0
10.0
10
Bonh
Viniss
Twntss
(a) Image cortuption.
(b) Textt corruption
Strong
Figure 12: Performance comparisons on UPMC-Food 101.
C MORE EXPERIMENTAL RESULTS
C.I GENERALIZATION ABILITY OF SUMI
To further validate the generalization ability and robustness of SuMi, conduct experiments two additional datasets. The first dataset is CMU-MOSI (Zadeh et al., 2016). MOSI encompasses three modalities (text, image, and audio), enabling evaluate the performance of SuMi across a dataset with more than two modalities. The other dataet is UPMMC-Food10 (Wang et al., 2015), which is image-text dataset for food classification.
We introduce four types of corruptions for text modality. Specifically, we introduce random dele- tion of word character (RD), random insertion of word or character (RT), word shuffling (WS) and sentence permutation (SP). Random deletion of word or character randomly removes words or characters from sentences to simulate noise the data. Random insertion inserts random words or characters into sentences, which can disrupt the original meaning. Word shuffling randomly shuffle words within a sentence to change the sentence structure while retaining some semantic meaning. Sentence permutation changes the order of sentences in a paragraph to simulate context shifts. For strong OOD on MOSI, we use Corrn to denote that n modalities are corrupted, missn to denote modalities are missing and corr+miss to denote both missing modalities and corruption modalities are present.
We use the stacked transformer blocks trained on MOSI dataset. Then, fine-tune the model on corrupted MOSI. For Food 101, we use the CLIP image encoder and text encoder as the modality- specific encoders followed by fusion classification head. Then train the model Food 101 and adapt the model on the corrupted Food 101. The results are presented in Figure and Figure 12. We can observe that in dataset with more modalities and in the common vision- languala task, SuMi can also outperform existing methods, demonstrating its effectiveness.
16

---

Published as a conference paper at ICLR 2025
Table 7: Results on real-world distribution shifts.
MOSI- SIMS
SIMS MOSI
Method
ACC
F1
ACC
F1
Source
39.2
39.1
40.1
45.5
EATA
40.5
41.2
40.4
45.7
READ
42.0
42.5
40.9
46.9
SuMi
44.2
44.7
41.6
47.8
86
62
40
46
46
43
(a) 8
(h) to
Figure 13: Ablation experiments on B and to on Kinetics50-C.
C.2 REAL-WORLD DISTRIBUTION SHIFTS
To evaluate the robustness of SuMi in addressing real-world distribution shift, follow the setting of Guo et al. (2024b) and conduct experiments on two datasets: CMU-MOSI (Zadeh et al., 2016) and CH-SIMS (Yu al., 2020). Specifically, CMU-MOSI and CH-SIMS multimodal sentiment analysis datasets which include three modalities. They contain different topics of conversations, different speakers, and different recording environments which can all be seen as real-world distri- bution shifts. In pratice, we consider the task as binary classification task and use the cross entropy loss. We use stacked Transformer blocks as the backhone and pre-train the model on CMU-MOSI and CH-SIMS as the source model for the setting MOSI-> SIMS and SIMS MOSI, respectively. Table presents the results. We can observe that in real-world distribution shifts, SuMi can still outperform existing methods, showing its robustness.
C.3 MORE ABLATION EXPERIMENTS
Exploration of 3 in IQR smoothing. In interquartile range smoothing, we set ,3 for more stable selection. Here, we select different values of 3 and present the results Figure 13(a). From the figure, we can observe that the performances across varying 3 are stable.
Exploration lo in mutual information sharing. For strong OOD adaptation, we add mutual information sharing in the first iterations to avoid the impact of strong OOD samples which could damage the mutual information sharing. We select several different to to conduct experiments and present the results formances are all better than the model without mutual information sharing which indicates the effectiveness of mutual information sharing strategy. Besides, with the increase of to, the perfor- mance improves before dropping when to Siter This shows that with the adaptation process, the strong OOD samples also increase which could bring many noises and damage the mutual informa- tion sharing. Moreover, the performances across varying to are stable, demonstrating the stability of our method.
Kinetics50-C in Figure 13(b). From the figure,
observe that the per-
Exploration of smoothing process. In Equation 3, we opt for simple linear smoothing process for clarity. Here, we provide a deeper analysis of the smoothing process. In addition to the linear smoothing, we provide the results of the logarithmic and exponential functions. Specifically, for
logarithmic function,
((t)
log( (e-1))
+ 1) and for exponential function we use f())
ster
17

---

Published as a conference paper at ICLR 2025
Table 8: Performance with different smoothing functions on Kinetics50-C.
[())
Linear
Exponential
Logarithmic
Acc
59.1
59.5
58.7
exp ((ie2) 1. We present the results in Table 8. From the table, we can observe that using function improve the performance of the model slightly. From the
ft) exp (1m2) properties of the exponential function, can be seen that the function grows slowly when the variable t is small and quickly when the variable is large. For logarithmic function, it grows quickly when the variable t is small and slowly when the variable is large. This indicates that slowing down the smoothing process the initial phase helps the model's performance. Additionally, can observe that the function will not affect the performance drastically, indicating the effectiveness of smoothing process itself.
18


Reviews:
[{"review_id": "C3eBSC1vDZ", "reviewer": "Reviewer_HsAx", "summary": "This paper presents SuMi, a novel approach for test-time adaptation (TTA) that aims to address distribution shifts in multimodal data. Through sample selection and mutual information sharing, the authors propose an approach that adapts effectively to complex multimodal noise patterns under distribution shifts.", "questions": "The authors could respond to the identified weaknesses by addressing the following aspects:Limited Multimodal Data Diversity, Insufficient Comparative Analysis with Sample Selection-Based TTA Methods, Limited Scope in Addressing Diverse Distribution Shifts, Fragmented Methodological Motivation.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 1, "strengths": "The paper introduces two main contributions—(1) a dual filtering mechanism to refine sample selection and (2) a mutual information loss\nstrategy to promote information alignment across modalities. These strategies aim to address limitations in current TTA approaches, particularly in multimodal data scenarios.\nThe authors validate their method across various corruption scenarios, including challenging out-of-distribution samples. This broad\ntesting highlights SuMi’s robustness and effectiveness under multimodal noise conditions.", "weaknesses": "1.Although the experiments demonstrate the method’s effectiveness, both datasets are limited to video-audio modalities. It remains unclear whether SuMi can generalize to other multimodal datasets, such as image-text pairs. This raises a question of the method’s applicability to vision-language models and other multimodal contexts (e.g., image and point cloud as in MM-TTA[1]).\n2.The proposed sample selection strategy shows promising results in the ablation study. However, comparable sample selection-based approaches (e.g., DEYO, EATA) are not sufficiently compared. A comparison of DEYO/EATA with (IQR + UA) , as well as experiments\ncombining DEYO/EATA and MIS, would provide clearer insights into the distinct benefits of SuMi.\n3.The experiments primarily focus on corruption shifts, but multimodal scenarios often encounter various other types of distribution shifts. Including datasets that reflect shifts beyond corruption could offer a more comprehensive evaluation of SuMi’s robustness.\n4.The paper’s three main components—sample identification with IQR, unimodal assistance, and mutual information sharing—lack a\ncohesive, unified framework. The combination of these techniques appears to address separate aspects of TTA, but the rationale for exclusively selecting these methods is insufficiently articulated. Greater clarity on the connections and unique insights of each\ncomponent could enhance the motivation and provide readers with more convincing reasoning for the proposed approach.\n[1] Shin, Inkyu, et al. \"Mm-tta: multi-modal test-time adaptation for 3d semantic segmentation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "f5UpEs3cPO", "reviewer": "Reviewer_d57C", "summary": "This paper introduces a novel task called multimodal wild TTA and presents a method named Sample Identification with Interquartile Range Smoothing and Unimodal Assistance and Mutual Information Sharing (SuMi). SuMi addresses the challenges of bridging the source domain and out-of-distribution scenarios through interquartile range smoothing. It effectively selects low-entropy samples rich in multimodal information for optimization and employs mutual information sharing to align different modalities. Experiments on two popular multimodal datasets show that SuMi significantly outperforms existing TTA methods, confirming its effectiveness.", "questions": "I would suggest that the authors consider discussing the setting of the new task and the practical applications of its motivation in detail. In real-world scenarios, would algorithms truly continue to be utilized under conditions where sensor failures or malfunctions occur? From my perspective, this task appears to resemble a combination of a modality missing and a TTA subtask. The proposed approach seems to specifically address these two subtasks.", "limitations": "", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The author proposes a TTA task that seems to be more applicable, which is helpful for the subsequent development of TTA. The paper is well-motivated and easy to follow. And the author's related work review is relatively comprehensive. From the experiments, the proposed SuMi has achieved significant performance improvement in new TTA task.", "weaknesses": "1. My main concern is the motivation behind the paper. Is this innovative task based on a pseudo-motivation? Is it truly feasible for this task to occur in real-world application scenarios? See the Questions part for details.\n2. Under real conditions of TTA, the number of modalities is generally more than two. I noticed that the authors discuss a multimodal form in equation (5), but at other times they refer only to a two-modality form. Can these be expanded into a general multimodal setting? Is it possible to validate the effectiveness of the method across multiple modalities in the experiments?\n3. The authors provide a comprehensive discussion of the parameters. However, there is no explanation for why the 'Interquartile' form of IQR was chosen. In this specific task, other quantile ranges might yield better results. Additionally, the paper's improvement is limited to gradually smoothing with the number of iterations, which is merely a simple linear process. There is no discussion of a more theoretical or effective smoothing process.\n4. The paper mentions that the selection of hyperparameters (such as $\\mu$) relies on prior information about the dominant modality of the datasets. However, in TTA, we cannot know the prior information of the data. Could this directly impact the performance of the method across different datasets? Is there a more robust hyperparameter-selection mechanism?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LPY1Em1UVo", "reviewer": "Reviewer_pfaw", "summary": "This paper investigates multimodal test-time adaptation (TTA) and introduces a new method, SuMI. Specifically, it extends current multimodal TTA research into a more challenging setting, termed wild multimodal TTA, where test data may include weak or strong out-of-distribution (OOD) contamination. To tackle this problem, the authors propose a technically robust approach involving reliable data selection and cross-modal discrepancy elimination. Extensive experimental results confirm the effectiveness of the proposed method.", "questions": "Please see the weaknesses", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Given the prevalence of multimodal data in the real world, studying multimodal test-time adaptation (TTA) methods for pre-trained models is highly relevant, especially in the era of foundation models. Beyond existing multimodal research, such as READ (Yang et al., 2024), the authors extend this work into a more challenging and general setting—wild multimodal TTA. I believe this extension could significantly advance test-time adaptation research within the community.", "weaknesses": "1. Figure 2 needs revision; it’s currently unclear what is being input to the fusion layers.\n2. Figure 3(c) lacks clear explanations for the X and Y axes.\n3. The settings (weak/strong OOD) should be labeled in Tables 1 and 3.\n4. The variable $\\mu$ is not found in Equation 4. Did you mean $\\gamma_{\\mu}$?\n5. It’s unclear how the proposed method handles cases where certain modalities are missing. From my understanding, if one modality is missing, the correspondence predictions may not be achievable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nzO8MPnzms", "reviewer": "Reviewer_Xmz9", "summary": "This paper mainly focus on the task of test-time adaptation. Different from previous works, the author consider a much tougher circumstance with both weak and strong OOD samples. Performance of existing methods will inevitably degrade for huge distribution gap. To combat with this gap, the authors introduce the interquartile to smooth the sharp gap. Moreover, unimodal entropy is utilized to suggest rich multimodal information. Since strong OOD may contain missing modality problem, the author further a mutual information sharing strategy to complement the distribution. Experiments on several datasets under various domains demonstrate the performance of the proposed method.", "questions": "Please refer to cons above.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The paper is written straight-forward, which is easy to follow.\n2. The researched TTA problem is a popular problem, while the authors make a further step to discuss under both weak and strong OOD situations.", "weaknesses": "1. While I understand the general framework of the method, certain parts seem to lack clarity. For instance, in section 3.2.1, I can learn that the purpose is to select those samples with lower adaptation gap, however, the ranking process for samples is never introduced. In equation 3, vector $h$ is directly used for comparison, but the comparison method—whether by calculating magnitude, norm, or another metric—is not specified. Definition of IQR also lacks clear clarification. Besides, the motivation part (Figure 3) is suggested to be placed in the Intro section, since it's more vivid and persuasive than experiment results (Figure 1). \n2. My main concern about this work is novelty. The method is mainly composed of three parts: interquartile mainly smooth the distribution gap, unimodal entropy selects samples with high quality, and mutual information sharing combats with missing modality. Every part seems to be orthogonal, with no clear interaction between them. Please explain how separate parts benefit each other in detail, or does every part just simply make their own contribution?\n3. While I am mostly ok with the experimental results, I have concern about the setting of the experiment. In my view, missing modality is a more common issue in multimodal circumstances with more than two modalities. Discussing such problem only on simulated datasets with two modalities is somehow narrow. It is suggested for the author to conduct experiments on dataset with more than two modalities.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)