Towards Robust Multimodal Open-set Test-time Adaptation via Adaptive Entropy-aware Optimization
Authors: Hao Dong
Eleni Chatzi
Olga Fink
Venue: ICLR 2025 Poster (2025)
Keywords: Test-time Adaptation
Multimodal Learning
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: We introduce Adaptive Entropy-aware Optimization (AEO), a novel framework designed to tackle Multimodal Open-set Test-time Adaptation (MM-OSTTA) for the first time.

Abstract: Test-time adaptation (TTA) has demonstrated significant potential in addressing distribution shifts between training and testing data. Open-set test-time adaptation (OSTTA) aims to adapt a source pre-trained model online to an unlabeled target domain that contains unknown classes. This task becomes more challenging when multiple modalities are involved. Existing methods have primarily focused on unimodal OSTTA, often filtering out low-confidence samples without addressing the complexities of multimodal data. In this work, we present Adaptive Entropy-aware Optimization (AEO), a novel framework specifically designed to tackle Multimodal Open-set Test-time Adaptation (MM-OSTTA) for the first time. Our analysis shows that the entropy difference between known and unknown samples in the target domain strongly correlates with MM-OSTTA performance. To leverage this, we propose two key components: Unknown-aware Adaptive Entropy Optimization (UAE) and Adaptive Modality Prediction Discrepancy Optimization (AMP). These components enhance the model’s ability to distinguish unknown class samples during online adaptation by amplifying the entropy difference between known and unknown samples. To thoroughly evaluate our proposed methods in the MM-OSTTA setting, we establish a new benchmark derived from existing datasets. This benchmark includes two downstream tasks – action recognition and 3D semantic segmentation – and incorporates five modalities: video, audio, and optical flow for action recognition, as well as LiDAR and camera for 3D semantic segmentation. Extensive experiments across various domain shift situations demonstrate the efficacy and versatility of the AEO framework. Additionally, we highlight the strong performance of AEO in long-term and continual MM-OSTTA settings, both of which are challenging and highly relevant to real-world applications. This underscores AEO’s robustness and adaptability in dynamic environments. Our source code and benchmarks are available at https://github.com/donghao51/AEO.

Published as a conference paper at ICLR 2025
TOWARDS ROBUST MULTIMODAL OPEN-SET TEST- TIME ADAPTATION VIA ADAPTIVE ENTROPY-AWARE OPTIMIZATION
Hao Dong*
Eleni Chatzi
Olga Fink²
ETH Zürich
'EPFL
[hao. dong,
chatzi baug ethz.ch, Aiga.fingepfl.co
ABSTRACT
Test-time adaptation (TTA) has demonstrated significant potential in addressing distribution shifts between training and testing data. Open-set test-time adaptation (OSTTA) aims adapt source pre-trained model online to an unlabeled target domain that contains unknown classes. This task becomes more challenging when multiple modalities are involved. Existing methods have primarily focused on uni- modal OSTTA, often filtering out cow-confieecce samples without addressing the complexities of multimodal data. In this work, we present Adaptive Entropy-aware Optimization (AEO), novel framework specifically designed to tackle Multimodal Open-set Test-time Adaptation (MM-OSTTA) for the first time. Our analysis shows that the entropy difference between known and unknown samples in the target domain strongly correlates with MM-OSTTA performance. To leverage this, we propose key components: Unknown-aware Adaptive Entropy Optimization (UAE) and Adaptive Modality Prediction Discrepancy Optimization (AMP). These components enhance the model's ability to distinguish unknown class samples during online adaptation by amplifying the entropy difference between known and unknown samples. To thoroughly evaluate our proposed methods in the MM- OSTTA setting, we establish new benchmark derived from existing datasets. This benchmark includes two downstream tasks action recognition and 3D semantic segmentation and incorporates five modalities: video, audio, and optical flow for action recognition, as well as LiDAR and camera for 3D semantic segmenta- tion. Extensive experiments across various domain shift scenarios demonstrate the efficacy and versatility of the AEO framework. Additionally, we highlight the strong performance of AEO in long-term and continual MM-OSTT, settings, both of which are challenging and highly relevant to real-world applications. This underscores AEO's robustness and adaptability in dynamic environments. Our source code is available at httpss/ggitub..co//oongaaa//////
INTRODUCTION
Test-time adaptation (TTA) significantly enhances the robustness and adaptability of machine learning models by enabling source pre-trained model to adapt to target domains experiencing distribution shifts (Wang et al. 202T). This adaptability is crucial for ensuring the applicability of models in real-world scenarios, such as autonomous driving and action recognition. To address the challenges posed by distribution shifts, a variety of TTA algorithms have been developed (Niu et al.) 2022 Yuan al. 2023 Gong et al. 2024). These algorithms adapt specific model parameters using incoming test samples through unsupervised objectives such as entropy minimization and pseudo-labeling However, most of these algorithms are designed for unimodal data, particularly images (Li et al.) 2003)). As real-world applications increasingly demand the processing of multimodal data, extending these approaches to support multimodal TTA across various modalities, including audio-video (Kazakos) al. 2019) and LiDAR-camesn -ammerr (Dong al. 2022). has become essential. In response this, several methods, such as READ (Yang et al. 2024) and MM-TTA (Shin et al. 2022), have been proposed to address the complexities inherent in multimodal TTA.

---

Published as a conference paper at ICLR 2025
wiluws Inose
ualoons
Seaes
30-
loown
Tixt
(Cras)
Disalt
22-
18-
12-
20-
6.335 0.190 0.135 0.212 0125 0:200 0.25 0.300 6.100 0.125 0.190 D.1 6:200 8:220 0:556 0.205 0,300
BAC
EPIC-Kitdoon
Fantox
Sooc
Scare
(4) MM-OSTTA with Tent
(b) MM-OSTTA with AEO (Curs)
(c) H-acore wio and wi TTA
Figure 1: (a) Tent minimizes the entropy of all samples, making it difficult to separate the prediction score distributions of known and unknown samples. (b) Our AEO amplifies entropy differences between known and unknown samples through adaptive optimization. (c) As a result, Tent negatively impacts MM-OSTTA performance while AEO significantly improves unknown class detection.
fundamental assumption in TTA is the alignment of label spaces between the source and target domains. However, real-world applications like autonomous driving (Blum et al. (019)) often involve target domains containing novel categories not present in the source label space (Nejjar et al. 2024). As a result, models adapted under this assumption may struggle with samples from these novel categories, significantly reducing the robustness of existing TTA methods (Fig. D This scenario, where the target domain contains unknown classes present in the source domain, is referred to as open-set TTA. Several unimodal open-set TTA approaches, including OSTTA (Lee et al. 2023) and UniEnt (Gao al., 2024), have been developed. However, OSTTA assumes that confidence values for unknown samples are lower in the adapted model than in the original model, which may not hold in Multimodal Open-Set TTA (MM-OSTTA) settings. UniEnt relies heavily on the quality of the embedding space to accurately detect unknown classes. The goal of MM-OSTTA is to adapt pre-trained multimodal model from the source domain previously unseen target domain with the same modalities but including samples from wiknown classes. The key challenge of MM-OSTTA is efficiently leveraging complementary information from diverse modalities improve adaptation and unknown class detection areas where current unimodal open-set TTA methods fall short.
Building on our observation that the entropy difference between known and unknown samples in the target domain is strongly correlated with the MM-OSTTA performance where larger en- tropy differenne results in better detection of unknown classes - we introduce the novel Adaptive Entropy-awan Optimization (AEO) framework. AEO is designed to amplify the entropy difference between known and unknown samples during online adaptation and consists of two key modules: Unknown-aware Adaptive Entropy Optimization (UAE) and Adaptive Modality Prediction Discrep- ancy Optimization (AMP). UAE dynamically assigns weights to each sample based on an entropy threshold and automatically determines whether minimize maximize the entropy for each sample. AMP adjusts prediction discrepancies between different modalities adaptively. encourages diverse predictions between modalities for unknown samples while maintaining consistent predictions for known samples. Together, these modules enable AEO to significantly amplify entropy differences between known and unknown samples during online adaptation, leading to substantial improvements in unknown class detection (Fig. [ and Fig. 2}
To comprehensively evaluate the MM-OSTTA task, develop new benchmark derived from existing datasets. This benchmark includes two downstream tasks action recognition and 3D semantic segmentation and incorporates five modalities: video, audio, and optical flow for action recognition, well LiDAR and camera for 3D semantic segmentation. Extensive experiments conducted across various domain shift scenarios demonstrate the efficacy and versatility of the proposed AEO framework. Furthermore, we evaluate AEO in challenging yet practical long-ferm and continual MM-OSTTA settings. AEO is robust against error accumulation and can constantly optimize the entropy difference between known and unknown samples multiple rounds of adaptation, a capability essential for real-world dynamic applications. Our contributions can be summarized follows:
We explore the novel field of Multimodal Open-Set Test-time Adaptation, a concept with significant implications for real-world applications. MM-OSTTA involves adapting pre- trained multimodal model from a source domain to a target domain that shares the same modalities but includes samples from previously unknown classes.
2

---

Published as a conference paper at ICLR 2025
To address MM-OSTTA, propose Adaptive Entropy- Optimization, which ef- fectively amplifies the entropy difference between known and unknown samples during online adaptation. Additionally, establish new benchmark based on existing datasets to comprehensively evaluate our method in the MM-OSTTA setting.
The effectiveness and versatility of our approach are validated through extensive experiments across two downstream tasks and five modalities, well as in challenging long-term and continual MM-OSTTA scenarios.
2 MULTIMODAL OPEN-SET TEST-TIME ADAPTATION
Multimodal Open-set Test-Time Adaptation (MM-OSTTA) aims to adapt a pre-trained source model to target domain that experiences both distribution shifts and label shifts across multiple modalities. Let Ds {(x,, 3/ ) represent the source domain dataset with label space Cs. which follows the distribution Px where each sample consists of M modalities, denoted as { k
== represent the target domain dataset with label space Cr and distribution Pxy. Let f - RC denote a neural network trained on the source distribution Pay, where the number of classes in Cs. In MM-OSTTA, f consists of M feature extractors
1. .M). Similarly, let Dr- = (x; 3/
go and classifier h(-). Each feature extractor (-) processes modality k to produce an embedding
z and the classifier fi(-) combines these embeddings to generate a prediction probability p.
ò = fx)())) = 6(h((91(x") 9M ()) = ((h((()) I)),
(1)
where 8(.) denotes the softmax function. Additionally, we include separate classifiers hx(.) for each modality k. yielding sppelity-specifie prediction probabilities & = ((hx((x(=*))
Given well-trained multimodal source model f(x) Ds. MM-OSTTA aims to adapt this model to the target domain Dr, where Py # Pxy- Unlike traditional closed-set TTA, which assumes C3 Cr. MM-OSTTA operates under the condition Cs C Cr, meaning the target domain may contain samples from unknown classes not present in the source domain. In addition to adapting the model and making predictions, MM-OSTTA involves generating a prediction score S(x) for each sample and employing an unknown class detector (r), defined as:
known S(x) 2 17
(x)
unknown S(x) < 1)
where 17 is a predefined threshold. Samples with S(x) < n are classified as unknown. We use the Maximum Softmax Probability (MSP) (Hendrycks & Gimpel 2017) as S(x) by default.
3 METHODOLOGY
3.1 CORRELATION BETWEEN ENTROPY AND MM-OSTTA PERFORMANCE
We begin by exploring the relationship between prediction entropy, defined as H -E. Pe log pe. and the performance of MM-OSTTA. Using pre-trained model on the source domain, we generate predictions on the target domain without performing any adaptation. The target domain consists of both known and unknown samples. To quantify this relationship, we calculate the average prediction entropy for known (HRnown) and unknown samples (Hanknown) separately, and then compute the difference (Hunknown: Hknown). We evaluate this entropy difference across various domain-shift scenarios using the EPIC-Kitchens (Damen et al. 2018)) 8) dataset and analyze its correlation with an MM-OSTTA performance metric (FPR95), which measures the unknown class detection ability. A lower FPR95 indicates better performance. As shown in Fig. 2 there is a strong correlation between the entropy difference and MM-OSTTA performance, with a larger entropy difference corresponding difference suggests that unknown samples exhibit significantly greater entropy than known samples, making them easier to differentiate.
lower FPR95. This observation is intuitive, higher entropy
Tent (Wang et al. 202T) minimizes the entropy for all samples, regardless whether they belong to known unknown classes, which inadvertently decreases the entropy difference between these
3

---

Published as a conference paper at ICLR 2025
classes. This reduction in entropy difference leads to diminished performance, demonstrated in Fig. and Fig. 6 To overcome this limitation and amplify the entropy difference between known and unknown samples during online adaptation,
propose the Adaptive Entropy-aware Optinization (AEO) framework. AEO consists of two primary components: Unknown-aware Adaptive Entropy Optimization (UAE) (Sec. (.2) and Adaptive Modality Prediction Discrepancy
Optimization (AMP) (Sec. 3.3
3.2
UNKNOWN-AWARD ADAPTIVE ENTROPY OPTIMIZATION
As discussed in Sec. 3.1 improving MM- OSTTA performance requires increasing the en- tropy difference between known and unknown class samples, i.e., maximizing the entropy of unknown samples while minimizing that of known samples. Tent (Wang et al. (021) min- imizes the entropy of all samples, failing to en- hance this difference. Some approaches (Niu et al. 2023 Yang et al. 2024) apply entropy minimization selectively to high-confident sam- ples, but still struggle to increase the entropy dif- ference. The first step in effectively optimizing entropy for both known and unknown samples is reliably identify potential unknown samples. To address this, we introduce the Unknown- aware Adaptive Entropy Optimization (UAE) loss, which adaptively weights and optimizes each sample based on its prediction uncertainty. The UAE loss is defined as:
MMG-O6TTA Pefformmce (FPR064)
OMA
Source
90
OMA Test
OMA ARD
70
00
80
30
0.0 0.2 0.4
0.6
0.8
1.0
1,2
Kutnewy D'Clerence between Known and Unkmowwn Sumples
Figure 2: The entropy difference between known and unknown samples is positively correlated with the MM-OSTTA performance. Tent minimizes the entropy of all samples, regardless of whether they are known or unknown, thereby failing to increase entropy differences and leading to poorer perfor- mances. In contrast, our AEO amplifies entropy differences via adaptive optimization, significantly improving unknown detection. Different shapes represent different domain-shift scenarios.
W Wada = T Tanh(3 (3. (H()) a)),
(3)
And = - H()) Wada, (4)
where Tonh is the hyperbolic tangent function, W is the adaptive weight assigned to each sample, (f) is the normalized entropy of prediction P. computed as H = -C. P. log poloog(() (C)) with C' being the number of classes. The parameters and B3 are hyperparameters that control the entropy threshold and scaling, respectively.
The function (z) is positive when x >> 0 and negative when x < 0. Therefore, the UAF loss CddaEnt maximizes (f) when HQ) > a (i.e. when prediction confidence is low, indicating the sample is likely unknown) and minimizes H when (ÿ) (i.e. when prediction confidence is high, indicating the sample is likely known). Moreover, Tanh(1) asymptotically approaches 1 as increases and converges to 11 as x decreases, resulting in higher weights for samples with very high or very low (Ø) (i.e. those that most likely to be known or unknown). When ( is close to o, the model is uncertain about whether the sample is known or unknown, and there a higher risk of wrong predictions. In such cases, the assigned weight approaches O, effectively neutralizing the potential negative impact of uncertain samples. In this manner, our UAE loss adaptively optimizes the entropy for each sample, enhancing the separation between known and unknown samples to ensure more reliable predictions. More discussion on the importance of W ada are in Appendix C.13
3.3 ADAPTIVE MODALITY PREDICTION DISCREPANCY OPTIMIZATION
To further enhance the entropy difference between known and unknown samples, we introduce Adaptive Modality Prediction Discrepancy Optimization (AMP), which optimizes the predictions across different modalities. To achieve this, AMP first employs adaptive entropy loss, similar to the UAE, that increase the entropy of predictions from each modality when confidence low, and decrease it when confidence is high. The loss is defined
LddaEnt =- 2 (H + H())) Wada;
(5)
where Wadn is the adaptive weight calculated in Eq. (3) Additionally, we propose maximiz- ing the prediction discrepancy between modalities for unknown samples to encourage uncertainy
4

---

Published as a conference paper at ICLR 2025
(i.e.. diversifying predictions across modalities increases the uncertainty in the final prediction). Conversely, for known samples, we enforce consistency across modalities to ensure confident predictions (i.e., confident predictions should exhibit consistent outputs across all modalities). To achieve this, we define the adaptive modality pre- diction discrepancy loss as:
MM-08177 Peferrmece (FPRRF))
25
ORAD UAE
AdaDis (Dis(f) ,²)) W ada:
(6)
UAE+AMP
where W is the adaptive weight from Eq. (3) and
20
Add Dis emphasizes samples with either very high or low (()). Dis(-) measures the prediction discrep-
66-
ancy between two modalities, with L, distance being the default choice. As illustrated in Fig. 3 training with AMP further increases the entropy difference,
60-
35
30
leading to improved performance. We also include
46
negative entropy loss term Lpi to ensure diversity in predictions (Zhou et al. 2023
3.5 Eatcopy Dufference bewwen Knowa and Unlcnown Samples
Yang et al.
2024):
Figure 3: Training with AMP further ampli- fies the entropy difference between known and unknown samples, leading to improved performances.
Div Σ
Pe log Pe,
(7)
where Pe is the accumulated prediction probability for class over one batch. The final loss is computed as the weighted sum of the previously defined losses:
LAO = AdaEnt + (C + LddaDis) + D2C
(8)
As shown in Fig- our AEO significantly amplifies the entropy difference between known and unknown samples test time, resulting in substantial performance improvements.
4 EXPERIMENTS
We evaluate our proposed method across four benchmark datasets: EPIC-Kitchens and Human- muimal-Cartoon (HAC) for multimodal action recognition with domain shifts, Kinetics- 100-C for multimodal action recognition under corruptions, and the nuScenes dataset for multimodal 3D semantic segmentation in Day-to-Night and USA-Singapore adaptation scenarios.
4.1 EXPERIMENT SETTINGS
Datasets. For domain adaptation experiments, we utilize the widely adopted EPIC-Kitchens (Damen al. 2018)) and HAC (Dong et al., 2023) datasets. Both datasets offer three modalities: video, audio, and optical flow. The EPIC-Kitchens dataset comprises eight actions ("put', 'take', 'open', ''llose' "wash', 'cut', 'mix', and 'pour') recorded in three distinct kitchens, forming three domains D1, D2, and D3. The HAC dataset includes seven actions ('sleeping'. 'watching TV', 'cating', 'drinking'. 'swimming'. 'running', and 'opening door') performed by humans (H), animals (A), and cartoon (C) figures, resulting in three distinct domains: H, A, and C. In our experiments, models are pre-trained on source domain and adapted samples as unknown classes for EPIC-Kitchens and vice versa. To prevent class overlap, we exclude the 'open' class samples from EPIC-Kitchens dataset and the 'opening door' class from the HAC dataset when used as unknowns.
target domain online. For the open-set setting, we treat HAC
For the corruption robustness experiments, models are trained on clean datasets and adapted to corrupted test sets. We create the Kinetics- 100-C dataset, which includes video and audio modalities, following the approaches outlined Heeddrycks & Dietterich (2019) Yang al. (2024). Kineties- 100-C consists of 100 classes selected from Kinetics-600 dataset (Carreira et al. 2018). with 21181 videos for training and validation, and 3800 videos for testing. We apply six types of corruptions on videos (Gaussian, Defocus, Frost, Brightness, Pixelate, and JPEG) and audios (Gaussian, Wind, Traffic. Thunder, Rain, and Crowd), generating six distinct corruption shifts. For example, Defocus (v) Wind (a) indicates defocus corruption on video and wind corruption audio. All experiments are conducted under the most severe corruption level (Hendrycks & Dietterich 2019). For the open-set setting, utilize HAC as the unknown classes, applying the same corruption types in
5

---

Published as a conference paper at ICLR 2025
Kinetics- resulting in the HAC-C dataset. By applying identical comupption types, we create open-set samples from a matching domain shift but with unknown classes.
For multimodal 3D semantic segmentation, utilize the nuScenes dataset (Caesar et al. 2020), which includes LiDAR and camera modalities. We examine two realistic adaptation scenarios following Jaritz et al. 20200: (1) Day-to-Night adaptation: LiDAR exhibits minimal domain shift due to its active sensing capabilities (emitting laser beams that remain largely unaffected by lighting conditions). In contrast, the camera, functioning as a passive sensor, experiences significant domain gap due to poor light at night, leading to substantial changes in object appearance. (2) USA-Singapore country-to-coune
adaptation: The domain gap may vary for both LiDAR and camera modalities. For certain classes, the 3D shape may shift more significantly than the visual appearance, while for others, the reverse may hold true. the open-set setting, we designate all vehicle classes as unknown. During training, unknown classes are labeled as void and ignored. During inference, the objective is to segment the known classes while simultaneously detecting unknown classes. Further illustrations and dataset details are provided in Appendix B.4
Evaluation Metrics. To evaluate the model's adaptation performance on known data, we use accuracy (Acc) for classification tasks and mean Intersection over Union (IoU) for segmentation tasks. To assess the model's ability to robustly detect unknown classes, we measure the area under the receiver operating characteristic curve (AUROC) and the false positive rate of unknown samples when the true positive rate at 95% (FPR95) for unknown samples. As our objective is to achieve good balance between the classification aceuracy of known classes and the detection accuracy of unknown classes, we reformulate a novel version of H-score, defined as the harmonic mean of Acc, AUROC, and FPR95:
3
H-score
Acc
AU ROC I- F9595
Since lower FPR95 indicates better performance, use FPR95 for the H-score calculation in Eq. (9). AUROC provides a global measure of how well the model distinguishes between known and unknown classes across all possible thresholds, making it suitable for tasks requiring balanced performance across thresholds. FPR95 evaluates the model's 's performance at a specific recall level (95% TPR), which is particularly important in applications requiring high recall, such as fraud detection or outlier detection. To comprehensively evaluate the model under the open-set setting, both FPR95 and AUROC are included in our H-score calculation. For the segmentation task, we replace Acc with IoU to calculate H-score.
Baseline models. We compare our method against two unimodal TTA methods, Tent (Wang et al. 2021) and SAR (Niu et al. (2023), as well as two unimodal open-set TTA methods, OSTTA (Lee et al. 2023 and UniEnt (Gao et al. (004)), along with one multimodal TTA method, READ Yang et al.
2024). Due to space limitations, additional implementation details are provided in Appendix B.2 and
Appendix B.3
4.2 COMPARISONS WITH STATE-OF-THE-ART
Robustness under domain shifts. We first conduct comprehensive experiments on domain adaptation benchmarks, where a model is trained single source domain and then adapted online to target domain with significant distribution shifts. Tab. prresnnts results from the EPIC-Kitchens dataset using video and audio modalities. Unimodal TTA methods, such as Tent (Wang et al. 2021 l) and SAR (Niu et al. 2023), underperform in the multimodal open-set TTA setup, revealing their limited adaptability in complex scenarios involving multiple modalities and unknown classes. Similarly, OSTTA (Lee et al. 2023), a unimodal open-set TTA method, struggles to achieve robust performance. highlighting the inherent challenges of multimodal open-set TTA contrast, UniEnt (Gao et al. 2024), another unimodal open-set TTA method, performs well in this setup. The SOTA multimodal TTA method READ (Yang et al.) 2024) demonstrates competitive performance, improving the Source baseline H-score by 3.55%. READ achieves this by focusing entropy minimization on high- confidence predictions while mitigating the noise from low confidence ones. Our proposed AEO framework demonstrates strong robustness in the challenging open-set setup, significantly improving the Source baseline H-score by 22.07%. Notably, in the DI + D3 adaptation, AEO improves FPR95 metric by relative value of 59.31%, %, which is crucial in applications requiring high sensitivity.

---

Published as a conference paper at ICLR 2025
DI - D2
DI -+ D3
DE - DI
1D2 -+ D3
Acct
FPR95]
AUROC!
Hacaret
Ace1
FFR95,
ALROC
H sesoore
Asct
FPR35]
AUBOC1
H score|
Ace1
FPRSEL
AUROCT
HI-scoee
48.54
85.29
58.31
27.75
43.31
63.09
33.82
62.13
49.83
06.72
77.31
48.08
Tent
43.45
13.01
SAR
49.12
88.94
63.50
50.30
93.84
57.58
15.03
80.87
74.38
54.40
64.96
13.75
OSTTA
47.09
97.04
13.94
7.72
50.15
97.57
39.10
11.06
48.79
90.29
64.45
21.58
44.39
89.82
67.34
22.12
UniEat
17.46
73.10
76.67
42.08
45.5
62.90
85.36
49.50
47.37
45.65
$7.96
58.97
51.94
41.82
90.71
63.20
READ
47.73
79.63
68,
35.44
48
78.46
68.69
36.81
49.90)
73.36
75.94
42.41
54.30
67.67
77.87
48.21
ABO
48.68
46.87
70.15
D2
Acch
FPRO5]
AUROC1
H-scoref
Acct
FFR95L
AUROC
H-score")
Aect
AUROC1
Hiscoret
Source
16.31
90.19
59.67
21:38
58.43
89.06
57.26
23.81
50.01
79.12
65.94
$4.11
Tent
46.01
91.05
57.45
19.88
51.53
92.39
48.17
17:49
47.11
92.82
51.85
16.52
SAR
45.75
85.79
62.17
27.20
50.97
90.88
20.31
89.14
62.79
22.48
OSTTA
60.41
23.63
18.99
UniEa
49.14
85.39
63.21
26.47
66.07
READ
49.65
83.72
66.19
31.03
57.50
83.36
61.97
32.04
51.28
77.20
69.86
57.06
ABO (Ours)
19.14 75.43
72.79
40.11
56.33
79.48
68.43
36.99
50.93
48.65
85.58
56.18
Table 1: Multimodal Open-set TTA with video and audio modalities on EPIC-Kitchens dataset.
Mron (Vilewannio)
Meon (vllowillon)
Rean (low-sudioo)
Mrow (Vile*+ashssillll)))
FPRU5] AUIROC1 H-scoref
EPRO5, ALROCC H-scowe
FPRU5]
AURROC
H-scoref
FPR95L
AUROCT H-screof
Source
56,36
57.81
37:11
79,25
76.31
37.68
Tent
59.05
85.19
59.75
28.87
58.61
86.71
57.78
26.55
44.32
89.01
55.04
21.12
58.86
36.91
53.60
25,42
SAR
60.66
78.95
63.88
35.94
50.17
85.92
59.70
27.19
43.75
62.32
61.29
27.43
60.56
82.34
59.74
31.35
OSTTA
58.84
82.46
63.005
32.58
58.45
86.45
60.95
25.85
41.75
88.26
55.17
21.63
58.52
84.38
57.29
29.13
UniEat
59.34
75.59
63.35
37.48
00.09
71.51
66.02
41.34
75.17
68.90
37.00
58.23
72.13
68.25
42.05
READ
58.78
72.83
59.03
40.17
36.28
74,26
41.32
ABO (Qurs)
59.53
66.75
72.50
59.38
74.96
49.23
44.29
69.44
42.62
72.82
48.50
Table 2: Multimodal Open-set TTA with different combinations of video, audio, and optical flow
modalities on HAC dataset.
To assess the generalizability of our proposed method, we further evaluate it on the HAC dataset using different modality combinations:
video+flow. flow aldio, and video+ aaleoo-auio as presented in Tab. 2 For each modality combination, we consider six adaptation scenarios: H H. C A. The results averaged over six splits (detailed
- A. H + C, A + H. A A. results are available from Tab. 20 to Tab. 23 Consistent with our findings on the PitC-Kis
dataset, most existing TTA methods struggle to generalize effectively in the challenging multimodal open-set TTA setup. While UniEnt (Gao et al. 2024) and READ Yang et al., 2024) perform well and surpass the Source baseline, other TTA methods fail to achieve robust performance, underscoring the complexities of multimodal open-set TTA. In contrast, our method demonstrates strong robustness across all modality combinations, significantly improving the Source baseline H-score by 13.70% 12. 12.12%. 9. 9.12% and 10.82% for the respective modality setups. This showcases the effectiveness of our approach in handling diverse multimodal scenarios under challenging open-set conditions.
Robustness under corruption. We evaluate our method on the challenging Kinetics- cor- ruption benchmark, which introduces various corruptions to both video and audio modalities. The results are summarized in Tab. The pre-trained model struggles to generalize to corruptions such as Gaussian (v) Gaussian (a) and Frost (v) Traffic (a), leading to very low H-scores. contrast, our proposed AEO adapts effectively to these corruptions in an online manner, improving the H-score over the Source baseline by 32.78% and 26.50% respectively. Conversely, methods like Tent (Wang et al., 2021) and SAR (Niu et al. (023) exhibit severe performance degradation under these corruptions. Our AEO consistently demonstrates robustness across all types of corruptions, achieving an average H-score improvement of 22.41% % over the Source baseline.
Long-term MM-OSTTA. Models deployed in real-world sce- narios continuously encounter test samples over extended pe- riods and must make reliable predictions at any time. Recent work by Lee al. (2023) shows that most existing TTA meth- ods perform poorly in long-term settings, often degrading to performance levels worse than non-updating models. Following the methodology of Lee et al.] (2023), we simulate long-term TTA by repeating the adaptation process for 10 rounds without resetting the model. The results, summarized in Tab. f show a 7.67% increase in H-score after long-term adaptation using our AEO. This improvement demonstrates that our method is robust against error accumulation and its ability to continuously optimize the entropy difference between known and unknown samples (Fig. 44. In contrast, most of the baseline methods suffer from significant performance degradation during long-term adaptation.
70-
Saumee
ABO spods)
ABO (10 opecial
50
40-
30-
H
C-
Figure 4: AEO continuously opti- mizes entropy difference between known and unknown samples, re- sulting in a substantial reduction of FPR95 after 10 adaptation epochs.
7

---

Published as a conference paper at ICLR 2025
Delocus (v) Wind G))
Frost (V) + Tralfic (4)
Brightnsss (v)+ "Thunder (A)
Pixelate (V) - Rain (o)
Acct
[PR95]
AUROCI
Huscure
Ace1
FFR95.
ALROC H score"
AscT
FPRS51
AUROCT H score|
Acct
FPRSSL
AUROCT
H1-score"
60.74
72.64
71.93
13.24
23.20
59.50
60.01
70.63
72.11
69.38
46.56
Tent
85.87
18
18.77
SAR
63.32
89.45
64.89
23.81
95.71
54.57
11.09
76.76
91,42
61.84
20.58
93.37
55.33
16.46
OSTTA
62.76
81.39
65.19
35.29
51.42
86.45
61.65
27.41
76.11
87.92
67.70
27. 17:10
75.05
79.29
72.57
39.79
UniEat
63.53
62.50
80.20
54.67
50.26
79.39
71.30
36.39
77.13
46.06
$7.65
69.90
74.79
56.58
84,14
62.13
READ
64.24
59.08
80.19
57.17
54.95
67.47
75.62
48.26
77.74
42:21
87.8
72.19
75.63
45.32
87.03
69.77
63.47
73.35
JPEGTTV
(0)
Caussian
Ciossinn (6)
Acch
PPRO5]
AUROC1
Hiscoref
Ace?
FFR95L
AURO
H-score")
Aect
AUROC1
Hiscoret
Source
61.11
70.11
69.38
46.70
13.05
98.18
13,99
1.62
52,96
76.20
64.34
37.66
Tent
70.13
93.11
53.47
16.84
35.82
97.26
49.38
7.26
61.85
92.38
54.40
17.58
SAR
10.05
93.71
58.22
15.75
39.08
97.16
54.57
7.58
62.69
93.47
58.24
15.88
OSTTA
69.79
UniEa
69.53
79.67
51.40
56.02
3.93
68.56
READ
21.58
48.42
84.76
06.44
42.97
74.95
69.44
38.66
64.52
56.24
80.00
58.75
ABO (Ours)
21.05
45.87
$7.12
68.14
40.82
75.53
67.34
37.40
63.88
54.13
82:22
60.07
Table 3:
Multimodal Open-set TTA with video and audio modalities on Kinetics- with
conrupted video and audio modalities (severity level
Maoo
Mean (HAC)
Maae (Kinetics. II0-C)
Aech
FPR-95
AURDC1
H-scoret
Acct
FPR95|
AUROC1
H-score?
Acct
FPRO5
AUROC1
H-scoret
Source
56.36
78.24
36.68
34,61
Source
56.36
78.24
56.68
34,61
52.96
76.70
64,84
37.66
Test
56.11
7.68
Tent
55.48
88.41
54.79
23.13
52.08
94.97
42.70
11:74
SAR
57.68
97.40
6.87
87.81
58.92
95.47
36.42
OSTTA
58.36
88.00
63.22
23.37
OSTTA
55.55
84.36
61.06
29.77
57.53
92.36
50.62
17.05
UniEnt
50.91
80.27
59.85
30.07
UniEat
58.56
80.68
62.52
32.30
59.62
68.33
77.99
46.41
READ
53.83
77.10
61.17
36.97
READ
57.76
73.49
65.00
41.85
61.10
48.29
81.19
62.32
ABO (Ouns)
56.81
55.18
AED (Ours)
60.38
78.89
42.62
87.22
Table 4:
Long-term Multimodal
Table 5: Continual Multimodal Open-set TTA on HAC and
Open-set TTA on HAC dataset.
Kinetics-100-C (severity level 5) datasets.
Day
Night AUROC1
USA - Singupore
loUt
FFR95L
H-scond
loUt
FTR9S]
AUROC
H--soee1
Source
41.76
47.97
75.11
53.76
54.27
43.358
53,49
599.81
Tent
50.50
79.86
$2.80
47.92
46.75
82.78
40.32
47.03
81.84
53.67
50.99
59.14
MM-TIA
39.98
52.86
79.30
50.99
51.42
44.13
83.43
60.87
AEO (Ours)
42.04
44.90
82.56
55.51
55.04
41.11
81.57
63:87
Table 6: Multimodal Open-set TTA with LIDAR and camera modalities on nuScenes dataset.
Continual MM-OSTTA. Real-world machine perception systems operate in environments where the target domain distribution evolves continuously. Recent work by [Wang et al. (2022b) has highlighted that most existing TTA methods perform poorly in continual settings. Following the setup outlined in Wang al. (2022b), simulate continual TTA by sequentially adapting the model across changing
domains (e.g., H À - C. for the HAC dataset and similarly for Kinetics- 100-C) without resetting the model. The results, presented in Tab. 5 (detailed results are in Tab. [8] and Tab. [9), demonstrate that our method maintains robust performance under this challenging setup, even improving the H-score. In contrast, most baseline methods, particularly unimodal TTA methods (Wang et al. 2021 Niu et al. 2023), exhibit significant performance degradation.
Scaling to segmentation task. To further demonstrate the versatility of our AEO method beyond action recognition task involving video, audio, and optical flow modalities, we conduct experiments on a novel 3D semantic segmentation task that utilizes LiDAR and camera modalities. As shown in Tab. 6 baseline methods such as READ (Yang et al.] (024) and MM-TTA (Shin et al., 2022) struggle to outperform the source model. In contrast, our AEO method consistently demonstrates strong open-set performance, improving FPR95 with up to 8.22% over the Source baseline. This significant improvement highlights the effectiveness of AEO in maintaining robust performance across diverse tasks and modalities, underscoring its potential for reliable deployment in real-world segmentation applications.
4.3 ABLATION STUDIES AND ANALYSIS
Ablation on each proposed module. We conducted comprehensive ablation studies to evaluate the contribution of each proposed module, as detailed in Tab. 7 The results indicate that incorporating UAE effectively increases the entropy difference between known and unknown samples, thereby enhancing detection performance for unknown classes. Additionally, integrating AMP maximizes the prediction discrepancy between different modalities for unknown classes, fostering uncertainty in predictions and further improving unknown class detection. These two modules are complementary, and their combined implementation achieves the highest performance across all datasets.
8

---

Published as a conference paper at ICLR 2025
HAC
EPIC--Kichens
Kinstics
UAB AMP Acct
FPR95)
AUROC1
H-score1
Acct
FPR95)
AUROC1
H-score
AecT
FPR95L
AUROCT
H-scorsf
56.36
18.24
56.68
34.61
50.01
79.12
38.11
52.96
76.70
64.84
59.76
67.58
71.53
47.79
51.07
50.12
84.91
55.01
63.49
55.44
81.30
59.01
56,74
69.33
70.66
45.49
45.79
63.08
80.74
48.03
58.81
56.59
79.90
56.38
59.53
66.75
72.50
48.31
50.93
48.65
85.58
56.18
63.88
54.13
82.22
60.07
Table 7: Ablation on each proposed module with video and audio modalities.
Moon (1ND+15N)
Moan (SAM)
Mean (SimMMDG)
Aect
FPR95,
AUROCT
sesor
Aect
FPR95,
AUROCT
Hsswvee
Auct
FPR95]
AUROCT
Ilscuree
Source
14.29
74.61
62.14
38.39
Source
60.25
79.54
55.50
31.87
58.87
78.91
51.21
33,77
Teni
59.80
85.52
59.18
27.58
Tent
61.24
84.06
59.04
29.77
60.27
80.79
60.44
34.47
SAR
60.08
82.67
62.86
30.40
SAR
62.23
77.80
64.06
35.78
60.54
78.72
63,02
36.20
OSTTA
59.04
84,42
63.06
28.41
OSTTA
61.19
77.49
63,72
38.24
59.80
77.87
63,59
36.59
Uni
57.60
69.47
70.03
42.75
UniEnt
61.32
76.42
63.97
36.90
62.44
70.22
65.53
44.82
READ
58.00
73.48
66.89
40.89
READ
60.48
71.74
67.26
43.86
59.93
70.16
67.62
45.71
AEO (Ouns) 59.54
66.87
74.14
47.62
ARO (Ours)
60.63
67.06
72.13
48.20
60.93
66.26
70.86
49.16
Table 8: Ablation on different ar-
Table 9:
Ablation on different pre-trained models using
chitectures using video and flow
video and audio modalities on HAC dataset.
modalities on HAC dataset.
Ablation on hyperparameters in Wada. We evaluate the sensitivity of our method the hyper- parameters in Wada by varying one hyperparameter at time while keeping the others fixed. Our findings, illustrated in Fig. [ demonstrate that our method consistently outperforms the best baseline, READ, across all parameter settings. These results indicate that our approach is robust and less sensitive to variations in hyperparameter choices.
Applicability on different architectures. To demonstrate the robustness of our AEO method across different network architec- tures, we conducted experiments by mod- ifying the backbone networks. Specifically, we replaced the video backbone with In- flated 3D ConvNet (I3D) (Carreira& & Zis- serman 2017) and the optical flow back- bone with the Temporal Segment Network (TSN) (Wang et al. 2016). As illustrated in Tab. AEO consistently achieves signif- icant performance improvements in MM- OSTTA across these alternative architectures. This consistency underscores the versatility and effectiveness of our approach, regardless of the underlying network design.
Blat Tlaekre
Dest Bowr
Gau
48-
43-
40-
Hostee
42-
40
0.5
0.0
0.8
0.9
Figure 5: Parameter sensitivity analysis using video and audio modalities on the HAC dataset.
Robustness different pre-trained models. previous experiments, we pre-trained the model using the standard cross-entropy loss on the training set of each dataset. To assess the robustness of our AEO method across various training strategies, we also pre-trained models using advanced optimization techniques such as Sharpness-aware Minimization (SAM) (Foret et al. 2020) and SimMMDG (Dong et al., 2023). As demonstrated in Tab. b our method successfully adapts to these differently pre- pre-trained models, consistently achieving the best performance in all configurations.
Prediction score distributions before and after TTA. Fig. 6 iillustrates the prediction score distribu- tions for known and unknown classes generated by various baseline methods on the EPtC-Kitchens dataset, both before and after applying TTA. Without TTA (Fig. 6 (a)), the score distributions of known and unknown samples significantly overlap, resulting in poor performance in detecting un- known classes. Tent (Wang et al. 2021) minimizes the entropy of all samples indiscriminately. and fails to reduce the separation between known and unknown distributions (Fig. (b)). In contrast, our method achieves better separation between the score distributions of known and unknown classes (Fig. (c)), leading to improved performance in unknown class detection. Fig. Iillusrates the model prediction entropy for known and unknown samples during online adaptation. Our AEO continuously optimizes the entropy batch after batch to improve the MM-OSTTA performance.
Different ratios of unknown samples. In real-world scenar- ios, the proportion of unknown samples can vary significantly. To evaluate how this variability impacts our method's perfor- mance, we conducted experiments on the HAC dataset (A -> H) using video and audio modalities. Specifically, we adjusted the proportion of unknown class samples in each batch, rang- ing from 20% to 80%, and present the corresponding H-scores
0.2
0.4
0.5
0.6
0.8
Source
20.83
41.77
57.68
42.27
19.87
UniFEnt
53.28
63.49
53.46
58.07
40.90
READ
37.23
58.37
57.76
59.49
55.50
ABO (Owr)
53.78
68.18
65.26 67.77 69.58
Table 10: Ablation on different ra-
tios of unknown samples.

---

Published as a conference paper at ICLR 2025
16-
caleeon laom
allo Jana
calami
Iarm
6-
C-
Dreety
12
4-
0.53
0.50
0.70
0.30 0325
1200 0.20 0.00 0.20
9.000 0.200
2.300 Bume
9.25
9:000
2.30 8.225 0.30 0.2ss 0:30 a.ts 0.50 0325 0.30
Stee
(0) DI D2 (No ITA)
(b) 02 (Temt)
(e) - 122 (Ours)
Figure 6: Prediction score distributions of different baseline methods the EPIC-Kitchens dataset before and after TTA. AEO achieves better separation between the score distributions of known and unknown classes, leading improved performance in unknown class detection.
0.
013
2.00
0.2
Octone Buss
Oulixe Bunb
OaAen Baid. (c) DJ D2 (Oun)
(3) DJ D2 (No TTA)
(b) D1 - D2 (Tent)
Figure 7: Model prediction entropy for known and unknown samples during online adaptation. Tent fails to reduce the separation between known and unknown distribution. In contrast, our AEO continuously optimizes the entropy batch after batch to improve the MM-OSTTA performance.
The results indicate that our method remains robust across different proportions of unknown class samples, consistently achieving the highest H-scores in all scenarios.
in Tab. I0
Robustness under mixed distribution shifts. In this setup, the test data originate from multiple shifted domains that are mixed together during adaptaion (Niu et al. 2023). complicating the problem further. As shown in Tab. 11 our AEO consistently achieves best performance terms of H-score, suggesting its effectiveness under dif- ferent challenging scenarios. In contrast, several bascline methods, such Tent (Wang et al.) 2021), SAR (Niu jtt al., (023), and UniEnt (Gao et al., 2024 suffer from significant performance degradation.
Mean
FPR95.
AUROC* H-scoref
Source
56.36
78.24
56.68
34,61
Tent
58.27
91.34
55.75
19.88
SAR
57.62
89.43
58.40
22.53
OSTTA
57.56
81.33
64.40
33.17
Uni
57.92
81.64
59.48
29.27
READ
56.67
73.15
66.17
42.78
AEO (Ours)
58.87
65.54
74.52
49.85
Table 11:
Ablation under mixed distribu-
tion shifts on HAC dataset using video and audio.
CONCLUSION
In this work, tackle the challenging task of Multimodal Open-set Test-time Adaptation (MM- OSTTA) for the first time. Motivated by the observation that the entropy difference between known and unknown class samples positively correlates with MM-OSTTA performance, we propose Adaptive AEO Entropy-aware Optimization (AEO). AEO consists of two key components: Unknown-awart Adaptive Entropy Optimization (UAE) and Adaptive Modality Prediction Discrepancy Optimization (AMP). Together, these components increase the entropy difference between known and unknown class samples during online adaptation a complementary manner. We conduct extensive experiments on the newly introduced benchmark, encompassing two downstream tasks and five different modalities, to demonstrate the efficacy and versatility of our proposed AEO. Furthermore, AEO achieves promising results in both long-te and continual MM-OSTTA settings, where the adaptation process is repeated over multiple rounds and the target domain distribution evolves over time. These results underscore the robustness and adaptability of AEO in dynamic, real-world environments.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
The authors acknowledge the support of "In-service diagnostics of the catenary/pantograhh and wheelset axle systems through intelligent algorithms" (SENTINEL) project, supported by the ETH Mobility Initiative.
REFERENCES
Hermann Blum, Paul- Paul-Edouard Sarlin, Juan Nieto, Roland Siegwart, and Cesar Cadena. Fishyscapes: A benchmark for safe semantic segmentation in autonomous driving. In ICCVW, 2019.
Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In CVPR, 2020.
Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. In CVPR, 2017.
Joao Carreira, Eric Noland, Andras Banki-Horvall Chloe Hillier, and Andrew Zisserman. A short note about kinetics-600. arXiv preprini urXiv: 1800:01 00..008400 2018
Tiago Cortinhal, George Tzelepis, and Eren Erdal Aksoy. Salsanext: Fast, uncertainty-awee aware semantic segmentation of lidar point clouds. In ISVC, 2020.
Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray. Scaling egocentric vision: The epic-kitchens dataset. In ECCV, 2018.
Andrija Djurisic. Nebojsa Bozanic, Arjun Ashok, and Rosanne Liu. Extremely simple activation shaping for out-of-distribution detection. arXiv preprint arXiv:2209.0988 2022.
Hao Dong, Xianjing Zhang, Jintao Xu, Rui Ai, Weihao Gu, Huimin Lu, Juho Kannala, and Xieyuanli Chen. Superfusion: Multilevel lidar-camera fusion for long-range hd map generation. arXiv preprint arXiv:2211.15666 2022.
Hao Dong, Ismail Nejjar, Han Sun, Eleni Chatzi, and Olga Fink. SimMMDG: A simple and effective framework for multi-modal domain generalization. In NeurlPS, 2023.
Hao Dong, Eleni Chatzi, and Olga Fink. Towards multimodal open-set domain generalization and adaptation through self-supervision. In ECCV, 2024a.
Hao Dong, Yue Zhao, Eleni Chatzi, and Olga Fink. Multiood: Scaling out-of -iitti--iitiiittionn detection for multiple modalities. NeurlPS, 2024b.
Xuefeng Du, Zhaoning Wang, Mu Cai, and Yixuan Li. Vos: Learning what you don't know by virtual outlier synthesis. In ICLR, 2022.
Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. ICCV, 2019.
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimization for efficiently improving generalization. arXiv preprint arXiv:2010001412, 2020.
Yulu Gan, Yan Bai, Yihang Lou, Xianzheng Ma, Renrui Zhang, Nian Shi, and Lin Luo. Decorate the newcomers: Visual domain prompt for continual test time adaptation. In AAAI, 2023.
Zhengqing Gao, Xu-Yao Zhang, and Cheng-Lin Liu. Unified entropy optimization for open-set test-time adaptation. arXiv preprint 24XY::2444.06663 2024.
Taesik Gong, Yewon Kim, Taeckyung Lee, Som Chottananurak, and Sung-Ju Lee. Sotta: Robust test-time adaptation on noisy data streams. In NewrIPS, 2024.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
11

---

Published as a conference paper at ICLR 2025
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness common corruptions and perturbations. arXiv preprint arXiv:1903.12261, 2019.
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. In ICLR, 2017.
Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. In ICLR, 2019.
Dan Hendrycks, Steven Basart, Mantas Mazeika, Andy Zou, Joe Kwon, Mohammadreza Mostajabi, Jacob Steinhardt, and Dawn Song. Scaling out-of-distribution detection for real-world settings. ICML, 2022.
Maximilian Jaritz, Tuan-Hung Vu, Raoul de Charette, Emilie Wirbel, and Patrick Pérez. xmuda: Cross-modal unsupervised domain adaptation for 3d semantic segmentation. In CVPR, 2020.
Evangelos Kazakos, Arsha Nagrani, Andrew Zisserman, and Dima Damen, Epic-fusion: Audio-visual temporal binding for egocentric action recognition. In ICCV, 2019.
Donghyun Kim, Yi-Hsuan Tsai, Bingbing Zhuang, Xiang Yu, Stan Sclaroff, Kate Saenko, and Manmohan Chandraker. Learning cross-modal modal contrastive features for video domain adaptation. In ICCV, 2021.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
Jungsoo Lee, Debasmit Das, Jaegul Choo, and Sungha Choi. Towards open-set test-time adaptation utilizing the wisdom of crowds in entropy minimization. In ICCV, 2023.
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting samples and adversarial attacks. In NeurlPS, 2018
Shawn Li, Huixian Gong, Hao Dong, Tiankai Yang. Zhengzhong Tu, and Yue Zhao. Dpu: Dynamic prototype updating for multimodal duito--distribution detection. arXiv preprint arXiv:2411.08221 2024.
Yushu Li, Xun Xu, Yongyi Su, and Kui Jia. On the robustness of open-world test-time training:
Self-training with dynamic prototype expansion. In ICCV, 2023.
Jian Liang, Ran He, and Tieniu Tan. A comprehensive survey on test-time adaptation under distribu- tion shifts. International Journal of Computer Vision, pp. 1-34, 2024.
Weitang Liu, Xiaoyun Wang, John D Owens, and Yixuan Li. Energy-based out-of-distribution detection. In NeurlPS, 2020.
Xixi Liu, Yaroslava Lochman, and Christopher Zach. Gen: Pushing the limits of softmax-based -uittf-diittibuutio detection. In CVPR, 2023.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint 1608 100.:966...3383,, 2016.
Jonathan Munro and Dima Damen. Multi-modal domain adaptation for fine-grained action recognition. In CVPR, 2020.
Ismail Nejjar, Hao Dong, and Olga Fink. Recall and refine: A simple but effective source-free open-set domain adaptation framework. arXiv preprint w.Xiv:2411.12555 2024.
Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Yaofo Chen, Shijian Zheng, Peilin Zhao, and Mingkui
Tan. Efficient test-time model adaptation without forgetting. In ICML, 2022.
Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Zhiquan Wen, Yaofo Chen, Peilin Zhao, and Mingkui Tan. Towards stable test-time adaptation in dynamic wild world. In ICLR, 2023.
Mirco Planamente, Chiara Plizzari, Emanuele Alberti, and Barbara Caputo. Domain generalization through audio-viste visual relative norm alignment in first person action recognition. In WACV, 2022.
12

---

Published as a conference paper at ICLR 2025
Bardia Safaci, VS Vibashan, Celso M de Melo, and Vishal M Patel. Entropic open active learning. In AAAI, 2024.
Inkyu Shin, Yi-Hsuan Tsai, Bingbing Zhuang, Samuel Schulter, Buyu Liu, Sparsh Garg, In So Kweon, and Kuk-Jin Yoon. Mm-Ita: multi-modal test-time adaptation for 3d semantic segmentation. In CVPR, 2022.
Yiyou Sun, Chuan Guo, and Yixuan Li. React: Out-of-distribution detection with rectified activations. In NeurlPS, 2021.
Yiyou Sun, Yifei Ming. Xiaojin Zhu, and Yixuan Li. Out-of-distribution detection with deep nearest neighbors. ICML, 2022.
Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. In ICLR, 2021.
Jindong Wang, Cuiling Lan, Chang Liu, Yidong Ouyang, Tao Qin, Wang Lu, Yiqiang Chen, Wenjun Zeng, and Philip Yu. Generalizing to unseen domains: survey on domain generalization. IEEE Transactions Knowledge and Data Engineering, 2022a.
Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool. Temporal segment networks: Towards good practices for deep action recognition. In ECCV, 2016.
Mei Wang and Weihong Deng. Deep visual domain adaptation: A survey, Neurocomputing, 312: 135-153, 2018.
Qin Wang, Olga Fink, Luc Van Gool, and Dengxin Dai. Continual test-time domain adaptation. In CVPR, 2022b.
Hongxin Wei, Renchunzi Xie, Hao Cheng, Lei Feng, Bo An, and Yixuan Li. Mitigating neural network overconfidence with logit normalization. In ICML, 2022.
Baochen Xiong, Xiaoshan Yang, Yaguang Song, Yaowei Wang, and Changsheng Xu. Modality- collaborative test-time adaptation for action recognition. In CVPR, 2024.
Jingkang Yang, Pengyun Wang, Dejian Zou, Zitang Zhou, Kunyuan Ding, Wenxuan Peng, Haoqi Wang, Guangyao Chen, Bo Li, Yiyou Sun, et al. Openood: Benchmarking generalized out-of- distribution detection. NeurIPS, 2022.
Mouxing Yang, Yunfan Li, Changqing Zhang, Peng Hu, and Xi Peng. Test-time adaption against multi-modal reliability bias. In ICLR, 2024.
Yongcan Yu, Lijun Sheng, Ran He, and Jian Liang. Stamp: Outlier-aware test-time adaptation with stable memory replay. arXiv preprint a7Xiv:2407.15773, 2024.
Longhui Yuan, Binhui Xie, and Shuang Li. Robust test-time adaptation in dynamic scenarios. In CVPR, 2023.
Yunhua Zhang, Hazel Doughty, Ling Shao, and Cees G.M. Snoek. Audio-adaptive activity recognition across video domains. In CVPR, 2022.
Zhi Zhou, Lan-Zhe Guo, Lin-Han Jia, Dingchu Zhang, and Yu-Feng Li. Ods: Test-time adaptation in the presence of ooen-world data shift. In ICML, 2023.
Zhuangwei Zhuang, Rong Li, Kui Jia, Qicheng Wang, Yuanqing Li, and Mingkui Tan. Perception- aware multi-sensor fusion for 3d lidar semantic segmentation. In ICCV, 2021.
13

---

Published as a conference paper at ICLR 2025
A
RELATED WORK
A.1 TEST-TIME ADAPTATION
Test-time Adaptation (TTA) seeks to adapt a pre-trained model on the source domain online, address- ing distribution shifts without requiring access either source data or target labels. This characteristic distinguishes TTA from domain generalization (Wang et al. (2022a) and domain adaptation (Wang & Deng 2018), giving it broader applicability (Liang et al. 2024). Online TTA methods (Wang et al. 2021 Yuan et al. 2023) update specific model parameters using incoming test samples based on unsupervised objectives such as entropy minimization and pseudo-labels. Robust TTA methods (Niu ct al. 2022 Zhou et al., 2023) address more complex and practical scenarios, including label shifts, single-sample adaptation, and mixed domain shifts. Continual TTA approaches (Wang et al., 2022b] Gan et al. 2023) target the continual and evolving distribution shifts encountered over test time. Additionally, several TTA techniques have been developed for specific tasks such as semantic seg- mentation (Shin et al. 2022) and action recognition (Yang et al. 2024), often involving multiple modalities.
A.2 OPEN-SET TEST-TIME ADAPTATION
Open-set test-time adaptation (OSTTA) addresses situations where the target domain includes classes absent in the source domain, presenting greater challenges due to the risk of incorrect adaptation to unknown class samples, which can cause significant drop in performance. OSTTA (Lee et al. (023) mitigates this by filtering out samples with lower confidence in the adapted model than in the original model. UniEnt (Gao et al., 2024) distinguishes between pseudo- known and unnuoo-n samples in the test data, applying entropy minimization to the pseudo-knon known data and entropy maximization to the pseudo-unt wreudo-unknown data. STAMP (Yu et al., (2024) optimizes over stable memory bank rather than the risky mini-batch, where the memory bank is dynamically updated by selecting low-entropy and label-consistent samples class-balanced manner. However, existing open-set TTA methods focus exclusively on unimodal settings, overlooking the complexities of multimodal scenarios.
A.3 MULTIMODAL ADAPTATION AND GENERALIZATION
Multimodal domain adaptation (DA) and domain generalization (DG) tackle the challenge of distri- bution shifts in multiple modalities, such as video and audio. For instance, MM-SADA (Munro & Damen 2020) proposes a sulf-supervieed alignment method combined with adversarial alignment for multimodal DA. Similarly, Kim et al. (2021) employ cross- cross-modal contrastive learning to align representations across both modalities and domains. Besides, Zhang et al. (2022) introduce an audio- adaptive encoder and an audio-infusee recognizer to mitigate domain shifts. RNA-Net 1 (Planamente
et al. 2022) addresses the multimodal DG problem by introducing a relative norm alignment loss that balances the feature norms of audio and video modalities. SimMMDG (Dong et al.) 2023) presents universal framework for multimodal domain generalization by separating features within each modal- ity into mspelity-specific and modality-shared components, while applying constraints encourage meaningful representation learning. Building on SimMMDG, MOOSA (Dong et al., 2024a) intro- duces two self-super tasks, Masked Cross-modal Translation and Multimodal Jigsaw Puzzles, to simultaneously address multimodal open-set domain generalization and adaptation. Unlike existing multimodal DA and DG methods, which typically aim to train models using both labeled source domain data and unlabeled target domain data simultaneously or to develop more generalizable neural networks from the source domain, our work focuses on improving the performance of pre-trained multimodal models during test-time by leveraging unlabeled online data from the target domain.
A.4 DUTSROFDDISTRIUUTION DETECTION
Outtof-Distribution (OOD) detection aims to identify test samples that exhibit semantic shifts without compromising in-distribution (ID) classification accuracy. Numerous OOD detection algorithms have been developed, which can be broadly categorized into post hoc methods and training-time time regular- ization YYang et al. 2022). Post hoc methods design OOD scores based on the classification outputs of neural networks, offering the advantage of case of use without modifying the training procedure or objective. Popular OOD scores include Maximum Softmax Probability (MSP) (Hendrycks &
14

---

Published as a conference paper at ICLR 2025
Gimpel 2017). MaxLogit (Hendrycks et al. 2022). Energy (Luu al. 2020), ReAct (Sun et al. 2021 ASH (Djurisic et al., 2022), Mahalanobis (Lee et al. 2018). N-Nearest Neighbor (KNN) (Sun et al. 2022), etc. Training-ting time regularization methods, such LogitNorm (Wei et al. 2022 address prediction overconfidence by imposing a constant vector norm on the logits during training. In contrast, Outlier Exposure (Hendrycks et al.) 2019) uses external OOD samples from other datasets during training to improve discrimination between ID and OOD samples. Additionally, Du et al. (2022) propose synthesizing virtual outliers for training-time regularization. While most existing OOD methods are designed for unimodal scenarios, recent work (Dong et al., (024b) introduces the first benchmark for multimodal OOD detection (Li et al.) 2024) along with the Agree-to-Disagres algorithm, which enhances multimodal OOD detection performance. However, traditional OOD detection methods assume that both training and test data originate from the same domain an unrealistic assumption in real-world conditions and TTA scenarios. In realistic settings, domain shifts between training and test data pose additional challenges for OOD detection.
B
FURTHER IMPLEMENTATION DETAILS
B. PSEUDO CODE
This section presents the pseudo-code for our AEO method. From Algorithm T test samples are coming batch by batch, For each sample, we first compute the prediction probability P from all modalities as well as ÿ from each modality k. We then obtain an initial prediction ý from á and calculate adaptive weight Wade for each sample. Next, we compute the losses from UAE and AMP to formulate the final loss CAsO and update model parameters using Adam optimizer. Finally, we output prediction as well as a corresponding score for each sample. Samples with scores below predefined threshold will be treated as unknown classes.
Algorithm The Pipeline of AEO
Input: Unlabeled test samples D7 for each online mini-batch X, do
{ a pre-trained multimodal model fa, (-).
Calculate و and Pa for all x E Xo.
Obtain the prediction ý = arg max. [ple for all Xo. Obtain the adaptive weight Wada for all x E X, by Eq.
Compute End in UAE by Eq. 4
Compute EndaEnts and Ada in AMP by Eq. and Eq. (
Compute final loss AAA using Eq. 8
Update O, with Adam optimizer.
end for
Output: The prediction and score {ū,, max. (x. Nr.
B.2 IMPLEMENTATION DETAILS ON ACTION RECOGNITION TASK
For the action recognition task, conduct experiments across three modalities: video, audio, and optical flow. We use the SlowFast network (Feichtenhofer et al. (019)) to encode video data and ResNet-18 (He al. 2016) for audio, and the SlowFast network's slow-only pathway for optical flow. The models are pre-trained on each dataset's training set using standard cross-entropy loss. The Adam optimizer (Kingma & Ba 2015)) is employed with learning rate of 0.0001 and a batch size of 16. Training is performed for 20 epochs on an RTX 3090 GPU, and the model with the best validation performance is selected. We also pre-train models using advanced training strategies such as Sharpness-aware Minimization (SAM) (Foret et al. (020)) and SiMMMDG (Dong et al. 2023), to evaluate TTA performance on different models. During open-set TTA, we construct mini-batches with equal numbers of known and unknown samples. We use batch size of 64 and the Adam optimizer with a learning rate of 2c-5 for all experiments. We update the parameters of the last layer in each modality's 's feature encoder well the final classification layer. To ensure faimess, update the same number of parameters for all baseline models. For hyperparameters W adn. we set 0.8 and 3 to 4.0. For hyperparameters in the final loss LAFO. we set both ?1 and 72 to 0.1.
15

---

Published as a conference paper at ICLR 2025
Video
Optical Flow
Audio
EPIC-Kitchens
Human-Animal-Cattonn
Kinetics-600
nuScenes Figure Illustrations of datasets used in benchmark. We include three action recognition datasets with video, optical flow, and audio modalities, as well as one 3D semantic segmentation dataset with LiDAR and camera modalities.
B.3 IMPLEMENTATION DETAILS ON SEGMENTATION TASK
For the 3D semantic segmentation task, we use ResNet-34 (He et al. 2016) as the backbone for the camera stream and SalsaNext (Cortinhal al. 2020) for the LiDAR stream. We adopt the fusion framework proposed in PMF (Zhuang et al. (2021) modifying it by adding an additional segmentation head the combined features from the camera and LiDAR streams. For optimization, we use SGD with Nesterov (Loshchilov & Hutter] 2016) for the camera stream and Adam (Kingma & Ba 2015) the LiDAR stream. The networks are trained for 50 epochs with batch size of 3, starting with learning rate of 0.0005, which decays 0 with cosine schedule. In the open-set setting, all vehicle classes are treated as unknown. During training, unknown classes are labeled as void and ignored. To prevent overfitting, flipping, random scaling, color jitter, 2D random rotation, and random cropping. During open-set TTA, we use batch size of and Adam optimizer with learning rate of 0.0001. We update the batch normalization parameters of both LiDAR and camera streams.
apply various data augmentation techniques, including random horizontal
B.4 MORE DETAILS ON BENCHMARK DATASETS
To comprehensively evaluate our proposed methods in the MM-OSTTA setting, we establish new benchmark derived from existing datasets. This benchmark includes two downstream tasks action recognition and 3D semantic segmentation and incorporates five different modalities: video, audio, and optical flow for action recognition, as well as LiDAR and camera for 3D semantic segmentation, as shown in Fig. g
EPIC-Kitchens (Damen al.) 2018). EPIC- PitCheis a large-scale egocentric dataset collected from 32 participants in their native kitchen environments. The participants recorded all their daily kitchen activities, with annotated start and end times for each action. We use a subset of the EPIC- Kitchens dataset introduced in the Multimodal Domain Adaptation work (Munro & Damen 2020). which includes 10, 094 video clips across eight actions ('put', 'take", 'open", 'close", 'wash" "cut'. "mix', and 'pour'), recorded in three distinct kitchens, forming three separate domains D1, D2, and D3. We use the provided video and audio modalities.
HAC (Dong et al. 2023). The HAC dataset, designed for multimodal domain generalization, features seven actions ( ('slceping' 'watching tv', 'cating', 'drinking', 'swimming', 'rumning', and 'opening door') performed by humans, animals, and cartoon figures, spanning three domains H, A, and C.
16

---

Published as a conference paper at ICLR 2025
Gaussian Noise
Defocus
Frost
Brightness
Pixelate
JPBG
Figure 9: Visualization of various visual corruption types on the constructed Kinetics- 100-C bench- mark.
Gaussian
Wind
Rain
Audio
Thunder
Traffic
Crowd
Figure 10: Mel spectrogram visualization of the raw audio and the corresponding audio corruption types the constructed Kinetics-100-C benchmark.
contains 3, 381 video clips and we use the provided video, optical flow, and audio modalities for our experiments.
Kinetics-600 (Carreira et al. 2018). Kinetics-600 large-scale action recognition dataset comprising approximately 480% videos across 600 action categories. Each video is a 10-second clip of action moment annotated from YouTube videos. our benchmark, we carefully selected a subset of 100 action classes from Kinetics-600 dataset, with each class comprising roughly 250 video clips, yielding a total of 24, 981 video clips. We use the provided video and audio modalities to create new -imetics-100-C corruption dataset (Fig. aadd Fig. I0) following the ideas in Hendrycks & Dietterich (2019) and Yang et al.) (2024). We apply six different types of corruptions (Gaussian, Defocus, Frost, Brightness, Pixelate, and JPEG) on videos and six others (Gaussian, Wind, Traffic, Thunder, Rain, and Crowd) on audios. We randomly combine them to generate different corruption shifts and all our experiments are conducted under the most severe corruption level of 5 ((Heddyycks & Dietterich] 2019)). For example, Defocus (v) Wind (a) means we add defocus corruption on video and wind corruption on audio.
mitigate the potential category overlap with the HAC
nuScenes (Caesar et al.) 2020). nuScenes is large-scale dataset designed for autonomous driving research. It includes data collected from autonomous vehicles equipped with a comprehensive sensor suite, including cameras, LiDAR, radar, GPS, and IMU. The dataset provides 360 coverage of urban environments in Boston and Singapore, offering diverse weather and traffic scenarios. includes 1, 000 driving scenes with rich annotations, such 3D bounding boxes for 23 object classes, enabling tasks like object detection, tracking, and scene understanding. The scenes are split into 28, 130 training frames and 019 validation frames.
17

---

Published as a conference paper at ICLR 2025
Defocus (v)
Wind ())
Frost (V)
Tralfic (4)
Brightnsss (v)
Thhnder (A)
Pixelate (V)
Rain (o)
Acch
[PR95]
AUROC1
H
Ace1
FFR95,
ALROC"
H score
AscT FPR55)
AUROCT H score!
Acc1
FFRSSL
AUROCT
H1-score"
13.82
67.74
25.05
66.663
66.42
73.76
53.58
SAR
OSTTA
13.47
75.24
72.09
44.20
99.53
91.39
61.28
20.10
78.34
78.54
73.70
41.38
77.39
66.05
78.25
54.39
UniEat
73.63
57.37
84.54
61.39
56.87
79.47
70.69
37.30
78.76
45.11
$3.06
70.98
78.16
54.37
84.42
64.44
READ
74.18
48.32
85.05
67.28
60.92
64.61
76.7%
52.00
79.08
40.18
88.31
73.74
78.18
41.87
88.02
72.54
61.11
Coossinn (8)
Acch
PPROS
AUROC1
Hiscoref
Ace?
FERSS,
AUROC
H-score")
Aech
FPKS5]
AUROC1
Hiscoret
Source
76.97
61.97
75.25
54.63
41.61
$3.13
55.46
23.75
65,40
72.12
69.16
45.06
Tent
78.16
89.47
60.27
24.12
64.00
93.08
55.15
16.83
71.66)
90.34
58.50
22.14
SAR
77.76
23,02
64.39
92.34
71.75
90.95
21:20
READ
77.95
43.47
$6.72
71.34
06.08
55.82
81.82
60.01
72.73
49.05
84.45
66.15
ABO (Ours)
78.08
42.37
88.06
72.26
65.63
54.50
82.74
60.85
72.59
47.34
85.26
67.33
Table 12:
Multimodal Open-set TTA with video and audio modalities on Sinetics-100-C with
corrupted video and audio modalities (severity level 3).
0.8
Ran
SAR
0.6
OSITTA
AIRO ((KES)
0.3
0.2
0.1
15
All 36 Possible Combination of Corraptions
Figure 11: Detailed results on all 36 possible combinations of corruptions. Our AEO achieves the best on 31 of them.
B.5 EXTENSION TO MORE MODALITIES
Our AEO framework is not limited to two modalities and can be easily extended to M modalities. Given a sample x with M modalities, we obtain prediction probabilities á from the combined embeddings of all modalities, and ø f2. fuh from each modality, all of which are of shape [1, CJ. where C' represents the number classes. this case, And still in its original form as in Eq. 4 and EndaEnts can be defined as:
1 M
AddaEnt
Σ
H( Waday
(10)
M
and CAdaDis becomes:
M-1
M
2
CAdaDis
Σ
Σ j=i+1
Dis(f'.p)
W
(1I)
M(M
The final loss is also the same as in Eq. 88. As shown Tab. and Tab. 23 our framework demonstrates strong performances when video, audio, and optical flow all available.
C ADDITIONAL EXPERIMENTAL RESULTS
C. MORE RESULTS ON KINETICS-1ON 100-C DATASET WITH DIFFERENT SEVERITY LEVELS
In previous experiments, we used the most challenging setup (severity level 5) for Kinetics-1000 In this section, evaluate the performance of various methods under milder corruptions (severity level 3). As shown in Tab. [2] our AEO consistently outperforms baselines and achieves the best results at severity level consistent with the findings severity level 5.
18

---

Published as a conference paper at ICLR 2025
Source Tent SAR OSTTA UniEnt READ AEO (ouns)
H-score 37.30 16.90 16.11
28.14 14
45.38
59.11
60.56
Table 13: Average H-score on all 36 possible combinations of corruptions.
Kinetics- 100-C
Acch
FPR95]
AUROC1
H-scoret
Source
61.06
73.09
68.82
43.51
UniEnt
66.87
59.55
80.31
56.37
READ
68.65
52.52
82.31
62.44
AEO (Ours)
68.39
49.85
84.70
64.39
Table 14: Multimodal Open-set TTA with video and audio modalities on Kinetics-100-C with corrupted video and audio modalities (severity level 5). The corruptions for open-set data (HAC dataset) are mixed.
C.2 EVALUATION RESULTS ON ALL 36 POSSIBLE COMBINATIONS OF CORRUPTIONS
In the previous experiments, we applied six types of corruption separately to video and audio, then randomly combined them to generate six distinct corruption shifts (c.g.. Defocus (v) Wind (a), Frost (v) + Traffic (a), Brightness (v) + Thunder (a), etc.). In total, there are 36 possible combinations. In this section, we comprehensively evaluate various baselines across all 36 combinations and present the results in Fig. and Tab. [3 Our AEO achieves the best performance on 31 out of 36 combinations and obtains the highest average H-score, demonstrating its robustness under diverse corruption scenarios.
C.3 ROBUSTNESS UNDER MIXED CORRUPTIONS FOR OPEN-SET DATA
By default, the corruptions applied to the open-set data (HAC dataset) are the same as those applied to the closed-set data (Kinetics-100) for each configuration on Kinetics-1004 100-C. For instance, the Defocus (v) Wind (a) configuration, Defocus corruption is applied to video and Wind corruption to audio for both the HAC and Kinetics- 100 datasets. In this section, we evaluate a more challenging setup, where the corruptions for the open-set data are mixed and randomly sampled from all six possible corruptions. For example, in the Defocus (v) + Wind (a) configuration, Defocus corruption is applied video and Wind corruption to audio for Kinetics-100 while one of the six possible corruptions is randomly assigned to each sample in the HAC dataset. As shown in Tab. 14 our AEO consistently outperforms the baselines in this challenging setup.
C.4 ROBUSTNESS UNDER DIFFERENT SCORE FUNCTIONS
The score function in Eq. 2 is flexible, with several options such as MSP (Hendrycks & Gimpel 2017), MaxLogit ( (Hendrycks et al. 2022), Energy (Liu et al. 2020), and Entropy (Liu et al. (023). In this paper, both video and audio, as shown in Tab. 15 Our AEO demonstrates low sensitivity to the choice of score functions, achieving comparable performance across different metrics.
MSP the default and evaluate other score functions the HAC dataset using
C.5 INFLUENCE OF Lpie TO THE PERFORMANCES
We investigate the impact of Lav Eq. the performances, negative entropy loss term widely used in prior works (Zhou et al. 2023 Yang et al. 2024) to promote diversity in predictions. As shown in Tab. [6] removing C results in performance on the EPIC-Kitchent dataset remaining comparable to the original, while significantly reducing performance the HAC and Kinetics-1000 100-C datasets. This demonstrates the critical role LDi in ensuring diversity predictions.
C.6 ABLATION ON EACH COMPONENT IN AMP
We analyze the contributions of each term in Adaptive Modality Prediction Discrepancy Optimization (AMP), specifically LddoEntx and AdaDii. The LddaEntx term adaptively maximizes minimizes the prediction entropy each modality and C AdaDis dynamically adjusts the prediction discrepancy between modalities based whether the samples are known unknown. As shown in Tab. 17 with
19

---

Published as a conference paper at ICLR 2025
MSP
MaxLogii
Finergy
Eniropy
Acef
59.33
59.53
59.53
59.53
FPR95
66.75
65.25
65.02
66.
AUROC1
72.50
73.07
73.31
73.52
H-scoref
48.31
49.69
49.86
49.04
Table 15: Ablation on different score functions on HAC dataset.
HAC
EPIC-
Kineties- 100-C
Acct
FPR951
H-scorot
Acet
FPR95,
AUROC'
H-scorel
FPR95)
AUROCT
wio Zin
61.74
73.05
46.62
48.79
45.78
$6.11
$6.40
61.47
06.26
78.25
50.15
w/ Lan
59.53
66.75
72.50
48.31
50.93
48.65
85.58
56.18
63.88
54.13
82.22
60.07
Table 16: Ablation on Lii the final performances.
HAC
Aoct
FPR95]
AUROC*
H-scoret
59.76
67.58
59.49
67.48
72.39
47.60
59.85
67.41
72.36
47.74
59.53
66.75
72.50
Table 17: Ablation each component in AMP HAC dataset.
Butt Bedia:
Bot Budia:
86-
66-
64
42
42-
0.0Ga.0
00050000 10
0.50
Figure 12: Sensitivity to loss hyperparameters and 72-
only LnddEntt or Add the performance is very close to using L Ada alone. The best results achieved when they are combined, which means AddaEnts and AddaDis are complementary to each other.
C.7 SENSITIVITY TO LOss HYPERPARAMETERS
We evaluate the sensitivity of our method to the hyperparameters 7/ and 7/2 Eq. (8) by varying one hyperparameter at time while keeping the others fixed. As shown in Fig. 12] our method consistently outperforms the best bascline, READ, across all parameter configurations. These results highlight the robustness of our approach and its low sensitivity to changes in hyperparameter settings.
C.8
MORE VISUALIZATIONS ON PREDICTION SCORE DISTRIBUTIONS
Fig.
13 presents the prediction score distributions generated by various methods on the EPIC- Kitchens dataset before and after TTA. The score distributions of known and unknown samples exhibit significant overlap without TTA (Fig, 13 (a), (d), and (g)), leading to poor detection of unknown classes. Tent (Wang et al.) 202T) minimizes the entropy of all samples, whether known or unknown, further compressing the score distributions (Fig. [3 (b), (e), and (h)) and making separation more difficult. In contrast, the score distributions produced by our method achieve better separation between known and unknown samples (Fig, 13 (c), (f), and (i)), thereby improving unknown class detection.
Fig. 14 depicts the model prediction entropy for known and unknown samples during online adapta- tion, evaluated batch by batch. Without TTA (Fig. 14 (a), (d), and (g)), the entropy values for known and unknown samples also show considerable overlap, resulting in weak unknown class detection. Tent [Wang et al. 202T) minimizes the entropy of all samples, further narrowing the gap between known and unknown samples (Fig. 14 (b). (e), and (h)). In contrast, the entropy values generated by
20

---

Published as a conference paper at ICLR 2025
ualsors knewn
20
knowe
50
Desllyy
10
10
0.18
&15e 0.123 8:150 suasc
(6) D1 - (No TTA)
(b) DI 9:13 (Tenl)
(c) DJ Se (Ours)
12-
15.0-
Deooby
Desatt
10.0
7.5
10x
2.5
8:000
0.0
(6) D2 Di (Ours)
knww
23
6.55 1.30 100
0.00
0255 Sure
0d0s $200
0.133
0.100
2.35
0300
3,22
0.20
0,273
0.00
1.100
6.25
0.19
0.00
0.30 Soom
0.23
0,203
0275
D2 TTA)
(Uurs)
Figure 13: Prediction score distributions of different methods the EPIC-Kitchens dataset before and after TTA.
our method enable better separation between known and unknown samples during online adaptation (Fig. 14 (c), (f), and (i)), thus enhancing the detection of unknown classes.
C.9 DIFFERENT RANDOM SEEDS
We run experiments three times using different seeds on HAC dataset using video and audio (A + H adaptation) and then calculate the mean H-score to demonstrate the statistical significance of our methods. As illustrated in Fig. [5] training with AEO is statistically stable and consistently outperforms the baselines across various random seeds. In contrast, most baseline methods exhibit instability, with large variances across different runs.
C.10 DETAILED RESULTS ON KINETICS- 100-C DATASET IN CONTINUAL MM-OSTTA SETTING
Tab. 18 presents the detailed results on the Kinetics- -inetics-100-C dataset under the continual MM-OSTTA setting, where the target domain distribution continually changes over time without resetting the model. We simulate continual TTA by repeating adaptation across continually changing domains (Defocus (v) Wind (a) + Frost (v) Traffic (a) - Brightness (v) + Thunder (a) + ...). Notably, our method achieves a 4.12% increase in H-score after continual adaptation, indicating its robustness to error accumulation and its ability to continually optimize the entropy difference between known and unknown samples. Instead, most baseline methods suffer from severe performance degradation after continual adaptation.
21

---

Published as a conference paper at ICLR 2025
Stors
10
60
00
02
140
Oeher Blcck
Oclleen Ruri
(4) DI 133 (No TTA)
(b) D1 D3 (Toit)
(c) DI D3 (Curs)
100
035
022
Oafer DanA
Deber Bark
Osten Brr.
(0) 102 DI (No TEA)
(e) L3) DI (Fent)
( D2 DI (Cuns)
1.0
100
140
140
000
123
(g) D2
(b) (Tent)
(Ous)
Figure 14: Model prediction entropy for known and unknown samples during online adaptation.
70
65
60-
66-
50
H
45-
40-
35-
30-
25
Source
Texl
SÁR
OSİTA UniEs READ ABO
Figure 15: Experiments using three random seeds HAC dataset using video and audio (A H adaptation). Foreground points in bold show results averaged across three different seeds while background points, shown feint, indicate results from the underlying individual seeds.
C.11 DETAILED RESULTS ON HAC DATASET IN CONTINUAL MM-OSTTA SETTING
Tab. 19 provides detailed results on the HAC dataset in the continual MM-OSTTA setting, where the target domain distribution continually changes over time and the model never reset. We simulate three continual TTA scenarios (H A - C, A + C + H. and C H - A). Similar to the results on Kinetics- 100-C, our method improves the H-score by 5.53% after continual adaptation, further demonstrating its robustness against eтror accumulation and its capability constantly optimize the entropy difference between known and unknown samples. Mcanwhile, most baseline methods show severe performance degradation after continual adaptation.
22

---

Published as a conference paper at ICLR 2025
Defocus (v)
Wind (a)
Prost (v)
Traffic (a)
Brighees (V) + Thunder (a)
Pisclate (N) + Kain
Aect
FPR55]
AUROCT
H-score?
Acct
FPR9S]
AUROCT H-sooee
Acc"
EPRO5
AUROCT
H-soxref
Ace!
FPRO5.
AUROC1
H-soxeef
Source
660.74
72.63
71.95
14.84
87.68
55.35
23.20
TX.9
59.50
79.15
00.01
72:11
66:35
46.50
Test
95.26
43.62
11.73
12.25
62.76
97.45
38.64
6.91
SAR
63.32
99.45
32.27
4.92
P0.11
94.18
14.03
97.37
28.08
6.96
OSTTA
62.76
$1.39
65.19
35.29
68.66
91.18
50.23
14.13
71.97
92.08
56.24
18.99
68.76
94.39
47.72
14,03
UniEm
63.83
62,50
80.20
54.67
48.66
74.97
76.15
40.74
73.18
46.54
88.74
68.86
71.97
58.18
85.34
68.66
READ
64.24
59.08
80.19
57.17
54.84
64.00
78.98
51
76 76
33.92
91.76
76.81
73.05
29.07
92. 20
77.45
AEO (Ous)
64.21
57.68
81.08
58.21
54.05
59.74
82.40
54,08
74.71
25.74
94.57
80.16
69.95
21.21
94.64
79.88
JPEG
(0)
(0)
Acct
FPR951
AUROC1
Accc
PPR95]
AUROC1
sicree
Acct
FPR955
AUROC1 H-scoref
Source
61:11
70:11
191.38
99.18
43:30
76.20
37.66
Temi
53.84
98. 13
34,23
5.15
21.74
97.95
34,05
5.33
52.08
94.97
42.70
11.74
SAR
58.95
97.08
25.98
7.54
20:34
96.55
31.22
8.09
54.79
95.47
36.42
10.89
OSTTA
62.13
91.32
46.19
14.03
30.92
97.82
38.14
5.80
57.53
92.36
50.62
17.05
UniEnt
65.37
09.32
82.05
49.93
35.00
98. 98.68
55,01
3.73
59.62
08.33
77:99
46.45
READ
66.
34.53
90.82
72.73
50.89
68.21
72.41
38,6
61.10
10
48.29
84.49
62.32
28.55
28
78.45
39.91
74
42.62
Table 18: Continual Multimodal Open-set TTA with video and audio modalities on Kinetics- 100-CC with corrupted video and audio modalities (severity level 5).
H -+ A
A O-c C
A -- C
C+ H
Aact
FPR95
AUROCC
H-scoref
Acct
FPROS
AUROCC
H-scoret
Acct
FPROS|
AURROC
H-scors
Aact
FPR95
AUROC!
H-scoref
Source
57.25
87.53
45.90
25.12
38.79
58.9 94
22.49
2.33
44.85
87.76
58.24
24.73
67.99
56.26
74.93
57.68
Tent
57.51
78.59
65.63
31.25
98.16
39.60
4.59
54.87
91.27
55.23
19.88
66.02
92.00
52.79
18.87
77.70
41.82
97.06
45.43
60:35
21.85
67.48
59.20
OSTTA
72.14
31.53
88.97
63.08
21.70
52.76
89.3
57.61
23.06
67.74
89.83
55.83
22.90
Unibat
60.1
78.04
64.02
38.57
39:71
59.63
29:20
1.09
52.21
93.75
55.99
15:23
65.25
85.94
68.75
29.70
READ
57 84
68.87
69. 58
47.03
43.65
87. 78
54.09
24. 30
54.14
84.10
58.05
10.41
68.95
61.21
71.49
55.27
ABO (Curs)
58.9
65.78
73.30
50.01
47.24
82.08
70.49
32.91
53.86
72,33
70.31
43.52
70.01
37.20
90.76
72.77
Meon
Aoct
FPRO5L
AUROCT
H-scoref
Acct
FPRO5]
AUROOC
H-scoret
Acct
FPROS]
AURROC
H-scors
Source
61:36
71.07
67:22
45.21
00:38
(3:23
70.31
$2.00
36.36
78.24
36308
34.61
Tent
62.94
78.73
65:22
38.35
59:71
91.83
50.29
18.86
55.48
88
54.79
23.13
SAR
66.91
74.41
66.88
43.49
62.80
94.81
50.27
13.13
58.81
87.81
59.92
23.89
OSTTA
78.30
64,44
82.89
53.24
32.15
55.55
84.36
61.06
UniEal
66.26
36
55
32.30
READ
69.79
69.55
46.98
69:21
66. 31
47.09
57.76
73.49
ABO (Ours)
61.10 61.28
79.70
55,58
66.89
43.60
$8.80
68.27
60.08
60.38
78.89
$3.84
Table 19: Continual Multimodal Open-set TTA with video and audio modalities on HAC dataset,
with corrupted video and audio modalities.
H A
H --s C
A A-1 H
A A-C C
Acc
FTR95)
AUROC1
[l-cure!
Agc1
FFR95,
AUROC
H-score!
Asct
FPR85)
AUROC1 [I-scure
Ace1
FPR95L
AUROCT
I1-scoret
57.28
43.00
57.68
58.24
24.73
Tent
57.51
78.59
37.83
94.03
47.25
14.11
70.58
87.82
26.67
91:27
55.23
19.88
SAR
58.50
77.70
68.39
39.19
16.32
95.04
18.67
12.31
70.22
72.39
68.88
46.17
55.13
90.26
60.35
21.85
OSTTA
55.63
76.82
72.14
40.01
44.49
89.98
$8.36
21.52
71.09
86.59
61.74
28.62
52.76
89.34
57.61
23.06
UniFa
60.15
78.04
64.02
38.57
40.99
98.71
31,86
3.61
68.85
64.10
72.81
53.46
52.21
93.75
55.99
15.23
READ
57.84
68.87
69.5
47.03
15.40
36.95
56.73
25,80
70.51
58.62
73.60
57.76
51.14
$4.10
58.05
30.43
57.84
72,16
45.59
21.95
72.33
43.52
Mron
Aech
FPROS)
AUROCC
H-sporef
Aost EPROS,
AUROIC
H-swors7
Aect
FIRRS]]
AUROCC H-scoref
Souaree
61.36
71.61
67.72
45.21
07.83
63.23
52.07
36.36
78.24
56.68
54.61
Tent
62,94
78.73
65.22
38.35
66.78
30.68
63.56
36.38
59.05
85.19
28.87
SAR
66.91
66.88
66.67
70.11
601.66
78.95
35.04
OSTTA
63.52
78.30
64.03
43.52
82.46
UniEar
66.26
64.38
76.11
53.28
67.55
54.53
79.28
60.72
59.34
75.59
63:35
37.48
READ
61.07
69.79
60.55
46.08
63.60
68.65
68.72
48.27
58.76
72.83
66.04
42.71
ABO (Qurs)
64.10
61.28
79.70
55.58
65.34
62.14
74.46
$4.40
59.53
66.75
72,50
48.31
Table 20: Multimodal Open-set TTA with video and audio modalities on HAC dataset.
1. DETAILED RESULTS ON HAC DATASET USING DIFFERENT COMBINATION OF MODALITIES
Tab. throuugh Tab. 23 summarize the detailed results on the HAC dataset using different combina- tions of modalities. Most existing TTA methods struggle to generalize effectively in the challenging multimodal open-set TTA setup. While UniEnt (Gao et al. (024) and READ (Yang et al.) (024) perform well and exceed the Source baseline, other TTA methods fail to achieve robust perfor- mance, underscoring the complexities of multimodal open-set adaptation. In contrast, our method demonstrates strong robustness across all modality combinations, significantly improving the Source baseline H-score by 13. 73.70 12.12% 9.12%, and 10.82% for the respective modality setups. These improvements highlight the effectiveness of our approach in handling diverse multimodal scenarios, even under challenging open-set conditions.
23

---

Published as a conference paper at ICLR 2025
H - A
H - C
A H
A-+
Acct
EPR95]
AUROCI
Hasaret
Ace1
FFR95,
ALROC
H H-score!
Asct FPR951
AUROC1
H scoree
Ace1
FPRS5L
AUROCT
H1-scoref
63.13
85.45
50.36
28.75
44.103
25.35
3.09
52.70
77.53
62.00
19
52.72
29.16
Tent
62.69
78.37
65.
10.05
14.35
SAR
63.58
75.06
69.16
42.68
46.78
95.40
50.39
11.60
70.87
87.53
27.17
52.02
96.69
44.8%
8.73
OSTTA
62.25
65,56
75.25
51.38
44.03
91
58.60
19.29
66.76
90:99
59.61
21.02
50.92
96.69
44,97
8.72
UniEat
64.24
84.33
62.54
51,46
45.40
99.06
33.38
2,63
70.66
36.57
58.95
71.99
53.22
77.57
62.07
37.74
READ
63.69
62.03
77.71
54.64
41.54
93.29
54.26
15.66
70.37
67.34
69.70
50.70
53.31
68.33
51,81
24.24
ABO
56.95
81.81
46.32
31.25
Acch
FPRO5]
AUROC1
H-scoref
Acet
RFR95L
AUROC
H-score")
Acct
AUROC1
Hiscoret
Source
57.82
67.56
70.46
48.14
61.01
65.34
12.26
50.79
57.80
75.69
58.15
57.11
Tent
66.33
82.84
64.21
33.74
61.49
83.00
63.97
33.08
58.61
86.71
57.78
26.55
SAR
64,96
65.34
33.63
62.50
77.92
66.34
39.33
60.17
85,92
59.70
27.19
OSTTA
66.11
89.55
63.13
60.60
UniEar
61.36
66,83
75.03
50.19
74.15
54.04
71.51
READ
62.58
70.44
71.20
46.98
62 09
08.32
71.60
48.79
59.03
74.96
06.05
40.17
ABO (Ours)
61.93
58.04
80.45
57.24
62.00
61.70
74.73
54,14
59.38
65.05
74,96
49.23
Table 21: Multimodal Open-set TTA with video and optical flow modalities on HAC dataset.
H
Accn
FPR55]
AUROCT
H-scoret
Acct
FPR05,
AUROC
H-scoee??
Aoct-
FPROS)
AUROC1
H-scoref
Acct
FPR95)
AUROCT
H.scoref
Source
57.40
65.34
71.75
49.83
29.00
92.83
45.70
15.37
54.72
78.73
69.16
57.62
40.26
39.34
57.20
22.04
Tent
54.75
71.96
70.43
44.04
29.78
98.15
40.91
4.52
55.37
88.18
57.74
25.00
35.85
96.60
47.36
8.74
SAR
55.74
51.75
79.50
57.07
29.50
96.12
50.35
9.01
54.87
93:73
58.80
15:41
35.66
96.97
45.59
7.89
OSTIA
52.76
70.20
73.31
45.35
29.04
42,94
6.09
52.05
83.20
59.64
31.41
35.39
43.84
6.60
UniEax
57.84
21.43
71.38
58.
23.22
READ
51.53
61.15
77.81
52.70
29.96
33.97
61.09
21.37
53,42
72.53
71.11
43.40
36.86
$6.58
$2.07
21.82
ABO (Ours)
54.42
52.21
83.02
58.53
$1.99
33.04
66.76
27.94
54.72
64.10
77:39
50.50
36.76
$6.31
53.01
25.19
C-
Mrot
Acct
FVR951
AUROC1
H--curet
Acc1
FFR95,
AUROCT
H--sore?
Asct
FPR851
AUROC1
Hl-scurc
Souruee
42.42
Tont
41.38
90:56
56.27
20.29
79
58.41
57.51
24.16
44.32
89.01
55.04
21.12
SAR
43.31
75.78
66:57
57.79
43.38
76.27
66.61
37.40
43.75
82.52
61.29
27.43
OSTTA
34,29
92.72
$4.98
16.57
41.83
$8.19
$6.34
23.75
41.75
88.26
55.17
21.63
UniEa
43.69
79.09
69
35.29
45.70
75 50
70.28
39.00
44.62
75.17
68.90
37.60
READ
67.80
46.91
75
39.06
36.38
(Ours)
79.03
45.81
Table 22: Multimodal Open-set TTA with optical flow and audio modalities HAC dataset.
A
H --C
A --
A
Acct
FPRUS
AUROCT
H-scoref
Acct FPRO5,
AUROC:
H-score
Acst FIRRUS]
AUROCT
H-scoref
Acct
FPRO5
AUROCT
H-scveof
64.31
X0.79
53.25
35.03
92.32
9:20
301.76
63.40
75.37
39.26
Tent
63,25
64.27
93.01
48.26
15.81
91.28
20.28
9.22
SAR
64.02
70.42
60.33 60.36
46.97
41.36
95.77
47.92
10.66
70.15
77.72
67.02
40.51
55.33
15
42.49
7.64
OSTTA
64.02
68.21
69.56
48.82
40.53
91.08
53.52
19.30
66.85
87.67
56.27
26.35
57.08
94.35 39
44.78
13.75
UniEa
63.91
74.28
68.10
43.35
58.88
95 95.96
44.73
10:15
68.71
43,40
85.23
08.25
50.92
08.20
71.56
46.11
READ
63.91
61.26
76.13
$4.95
42.74
87.22
59,05
25.30
66.35
60.99
73.51
55.23
51 51.38
83.92
60.28
30.54
39.32
Acch
PPRO5
AUROC1
Hiscoref
Acct
RP995,
AUROC*
H-score")
Aect:
FIRR05|
AUROC1
Hiscoref
Source
58.76
78.01
64.53
38.51
63.69
76.60
65,52
40,71
56.66
76.31
61.81
37.68
Tent
61.02
81.69
60.43
34.57
61.02
$1.33
56.97
30.93
58.86
86,91
53.60
25.12
SAR
67.84
74.62
66.36
43.42
64.08
'8.37
64.89
38.91
60..6
82.34
59.34
31.35
OSTTA
57.97
59.30
64.68
29.13
UniEat
60.85
79.52
69.03
37.62
66.11
71.41
70.85
16.72
58.23
72.13
68:25
42.03
READ
60.85
76.86
68.50
40.41
59.00
75.28
66.21
41.47
57.47
74.26
67.28
41.32
ABO (Ours)
64.31
63.23
78.41
$4.05
64.02
68.43
70.63
48.82
59.76
66.88
72.82
48.50
Table 23: Multimodal Open-set TTA with video, audio, and optical flow modalities on HAC dataset.
C.13
FURTHER DISCUSSIONS ON THE NOVELTY OF THE PROPOSED FRAMEWORK
C.13.1 COMPARISON WITH OOD DETECTION METHODS
The primary contribution of Liu al. (020 is the proposal of energy as an inference-time OOD score. While Liu et al. (2020) also uses energy as learning objective, assigning lower energies to in-distribution (ID) data and higher energies to OOD data, it relies on auxiliary OOD training data. This assumption knowing whether a sample is ID or OOD during training- dooe not hold in MM-OSTTA, where the task is to differentiate between ID and OOD samples without prior knowledge of their status. Therefore, the energy learning objective proposed in Liu et al. (2020) can't be used for MM-OSTTA directly. Additionally, Liu et al. (2020) assigns equal weights to all samples in the energy loss, which is impractical in MM-OSTTA. In MM-OSTTA, both distribution and label shifts are present, and the network often encounters samples with high uncertainty, making it unclear whether they are ID or OOD. Assigning equal weights such samples can hinder adaptation performance.
To address the first challenge, use entropy robust indicator of known and unknown samples to make an initial differentiation. Then we address the second challenge by assigning dynamic weights to samples based their distance to entropy threshold. This mitigates the impact of noisy or
24

---

Published as a conference paper at ICLR 2025
HAC
EPIC- PiPChKitchens
Linetics-100-C
Consistent Prediction
47.61
55.12
58.91
AEO (ours)
48.31
56.18
60.07
Table 24: Ablation consistent predictions across modalities to the final performances. The H-score is reported.
ambiguous samples, focusing optimization on reliable ones. These adaptations make our Unknown- aware Adaptive Entropy Optimization substantially different from the energy-based approach in Liu ct al. (020).
C.13.2 COMPARISON WITH MULTIMODAL TTA METHODS
Both Shin et al. 2022) and Xiong et al. (2224) address the multimodal closed-set TTA by introducing a consistency loss between predictions of different modalities to make them close. However, this approach can be detrimental for open-set pet TTA, especially when dealing with unknown samples.
For known samples, enforcing consistent predictions across modalities helps the model make confident and accurate final predictions, each modality supports the others. For unknown samples, enforcing consistent predictions across modalities can be detrimental. both modalities have predictions with high entropy unknown samples, enforcing consistent predictions across them has minimal impact. However, when the model suffers from overconfidence issues and outputs high prediction certainty (low entropy) for unknown class samples (Fig. (a)), enforcing consistent predictions make the model output final predictions with high certainty aligned with known classes, degrading the open-set performances.
To address this, we propose Adaptive Modality Prediction Discrepancy Optimization, which adap- tively enforces consistency discrepancy based whether sample likely belongs to known or unknown class. This approach ensures that consistency is enforced only when beneficial, while discrepancy is maintained for unknown samples to prevent misalignment with known classes. Our ablation study highlights the necessity of this adaptive strategy. Enforcing consistent predictions across all modalities degrades performance across all datasets, as shown in Tab. 24
C.13.3 COMPARISON WITH OPEN-SET METHODS
Safaei et al. 2024) and Gao et al. 2024) also utilized the entropy difference between known and unknown samples in open-set settings. However, our work goes beyond merely leveraging this difference. In Sec. 3.1 we provide a deeper and more comprehensive analysis of this phenomenon. Specifically, we investigate the relationship between entropy differences and the performance of MM-OSTTA across various baselines, identifying potential failure modes that previous works did not address. Building on these insights, we propose the Adaptive Entropy-aware Optimization (AEO) framework, specifically designed for the MM-OSTTA setup to amplify the entropy difference between known and unknown samples dynamically during online adaptation.
C.13.4 COMPARISON WITH SELECTIVE ENTROPY MINIMIZATION METHOPS
Both SAR (Nuu et al. 2023 and EATA Niu et al. 2022 select reliable samples according to their entropy values for adaptation, excluding high- oigh-entropy samples from adaptation and assigning higher weights to test samples with lower prediction uncertainties. While they achieve better performance than Tent, they face two key limitations in the multimodal open-set TTA setting:
Limited handling of unknown classes. SAR and EATA focus solely on minimizing the entropy of samples with lower prediction uncertainties (likely belonging to known classes) while leaving the high-entropy samples (potentially unknown classes) unaddressed. This selective approach restricts their ability to amplify the entropy difference between known and unknown samples. contrast, our proposed Adaptive Entropy-awan Optimization (AEO) dynamically optimizes the entropy of both known and unknown classes, resulting in a significantly larger entropy difference, as demonstrated in Fig. 2
Sensitivity to overlapping score distributions. In the challenging open-set setting, the initial score distributions of known and unknown samples are often closely aligned and difficult to distinguish, as
25

---

Published as a conference paper at ICLR 2025
shown in Fig. (a). In such cases, SAR and EATA could potentially minimize the entropy of unknown samples incorrectly, leading to degraded performance that may even fall below the Source baseline. AEO addresses this issue by progressively increasing the entropy difference between known and unknown samples, mitigating the risk of negative adaptation and ensuring more reliable performance.
Additionally, introduce Adaptive Modality Prediction Diserepancy Optimization, which exploits cross-modal interactions to further enhance the separation between known and unknown classes. These complementary approaches strengthen our framework's ability handle open-set challenges more effectively than selective entropy minimization methods.
C.13.5 PERFORMANCE IMPROVEMENT IN ACCURACY
While our AEO framework is indeed primarily designed to enhance unknown sample detection, does not compromise accuracy (Acc) for closed-set adaptation. On the contrary, results show that AEO achieves competitive accuracy across all datasets.
To clarify, the assertion that AEO sacrifices Acc for its advantages in unknown sample detection is not supported by our findings. For example, while READ excels on EPIC-Kitchens and Kinetics- 100-C, performs poorly the HAC dataset. Conversely, SAR performs well HAC dataset but poorly on EPIC-Kitchens and Sinetics-100-C datasets. Our AEO, however, consistently performs well across all datasets. To provide further clarity, we have computed the average Acc across all datasets (EPIC- Kitchens, HAC (vidco+audio, video+flow, flow+audio, vidco+audio+flow) and Kinetics- 100-C). As shown Tab. 25 our AEO achieves the highest overall performance in Acc, demonstrating its robustness.
Moreover, we have computed the average values other metries (FPR95, AUROC, and H-score) across all datasets. As shown in Tab. 26] AEO achieves the best overall performance across all metrics, with notable improvements in FPR95, AUROC, and H-score. While the improvement in Acc is relatively modest, it is significant that AEO balances both closed-set adaptation and unknown sample detection effectively.
In summary, work is the first to tackle the challenging and practical problem of Multimodal Open-set Test-time Adaptation. AEO does not trade Acc for improved unknown sample detection but instead achieves well-rounded performance across both closed-set and open-set adaptation objectives.
Source Tent SAR OSTTA Unitint READ AFO (ours)
Acct 53.02 54.97 56.00 54.65
55.77
55.77
56.30
Table 25: Average accuracy (Acc) across all datasets.
Accf
FPR93
AUROCT
H-scoret
Source
53.02
77.58
61.86
35.78
Tent
54.97
55.40
22.68
SAR
56.00
85.36
60.94
26.71
OSTTA
54.65
86.40
59.24
26.23
UniEnt
55.77
71.51
70.22
41.62
READ
55.77
72.14
69.36
42.81
AEO (Ours)
56.30 (+0.30)
(+9.58)
76.82
(+6.60)
50.82 (+8.01)
Table 26: Average value of different metrics all datasets.
26


Reviews:
[{"review_id": "W9XP4KcXrx", "reviewer": "Reviewer_wyfy", "summary": "This paper addresses the multi-modal open-set TTA setting, a challenging and less-explored area within TTA research. It aims to adapt a source pre-trained multi-modal model to an unseen target domain containing samples from unknown classes. They propose an Adaptive Entropy-aware Optimization (AEO) framework, which encourages the adapted model to amplify entropy difference between known and unknown samples. Specifially, UAE dynamically determines whether to minimize or maximize the entropy of each sample through a sample-wise weighting scheme, while AMP considers modalities to encourage diverse predictions for unknown samples and consistent predictions for known samples. The AEO framework shows its effectiveness on the proposed MM-OSTTA benchmark across various modalities.", "questions": "- Please refer to the weaknesses part.\n- In equation (3), (5) and (6), how is the entropy used for the adaptive weight W_ada calculated? Is it based on predictions from each individual modality or from the aggregated output? \n- For the corruption robustness experiments, there are six types of corruption for both videos and audios, resulting in 36 possible combinations (6*6) of corrupted test sets. Why are only six distinct corruption shifts used in the experiments?\n- In Table 7, what happens if the diversity loss in equation (8) is excluded?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper addresses the MM-OSTTA setting for the first time, which is a challenging but practical scenario in real-world applications.\n- The paper is well written, making it easy to follow the introduced setting and proposed algorithms.\n- The paper provides extensive experiments across two tasks and five modalities, showing the effectiveness and versatility of the proposed method.", "weaknesses": "- The observation in Sec. 3.1 is less impressive in the open-set settings. Previous works [1-2] on the open-set settings have already discovered and leveraged the entropy difference between known and unknown samples. \n\n- In lines 171-174, the authors argue that some selective entropy minimization methods struggle to increase the entropy difference. But, the evidence supporting this claim is insufficient; Figure 2 compares only simple baselines (source, tent), missing another promising baseline such as EATA [3].\n\n- EATA [3] also takes a sample-wise adaptive weighting scheme based on entropy values and achieves competitive performance. Such methods (e.g, EATA, SAR) seem to be able to increase the entropy difference between known and unknown samples, at least more effectively than TENT. What advantages does the proposed method offer over such selective entropy minimization methods?\n\n- In Sec. 3.3, it is straightforward to enforce consistent predictions across modalities for known samples. However, for unknown samples, what is the benefit of maximizing prediction discrepancy across modalities? The equation (6) seems to penalize (i.g., maximize the prediction discrepancy) samples with consistently high entropy (likely unknown) across modalities, although they are easily classified as unknown. A more in-depth explanation of the adaptive modality prediction discrepancy loss in equation (6) would be valuable.\n\n- One of the key purposes of TTA is to adapt the pre-trained model to make better predictions on unseen test data. Although the proposed AEO significantly improves unknown class detection (AUROC, FPR95), its accuracy gains are limited compared to the baselines. Also, the proposed framework primarily focuses on unknown class detection, showing limited capability in model adaptation itself.\n\n\n[1] Safaei et al., Entropic open-set active learning. AAAI'24. \\\n[2] Gao et al., Unified Entropy Optimization for Open-Set Test-Time Adaptation. CVPR'24. \\\n[3] Niu et al., Efficient test-time model adaptation without forgetting. ICML'22.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OHDyV8Btmc", "reviewer": "Reviewer_SHVx", "summary": "In this paper, the authors present Adaptive Entropy-aware Optimization (AEO) to tackle Multimodal Open-set Test-time Adaptation (MM-OSTTA), which consists of Unknown-aware Adaptive Entropy Optimization (UAE) and Adaptive Modality Prediction Discrepancy Optimization (AMP). To thoroughly evaluate the proposed methods in the MMOSTTA setting, the authors establish a new benchmark derived from existing datasets.", "questions": "What is the underlying reason for the performance gap between the proposed method and previous ones on Acc metric?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.The proposed method is easy to understand and the source code and benchmarks will be made publicly available.\n\n2.A new benchmark that incorporates five modalities: video, audio, and optical flow for action recognition, as well as LiDAR and camera for 3D semantic segmentation is proposed.\n\n3.The analysis on the entropy difference between known and unknown samples is reasonable.", "weaknesses": "1.It seems that the proposed Unknown-aware Adaptive Entropy Optimization has already been used in some of the existing works of wild TTA.\n\n2.In the experimental settings, the authors say that all experiments are conducted under the most severe corruption level 5. However, it may be unrealistic in real-world applications. What about the performance under more lower corruption level?\n\n3.More recent methods should be included for comparisons in Table 1 and 2, i.e., [1-2]\n\n4.It seems that the performance of the proposed method on Acc is generally inferior to previous methods in Table 1 and 2.\n\n\n[1]Two-Level Test-Time Adaptation in Multimodal Learning\n\n[2]COME: Test-time adaption by Conservatively Minimizing Entropy\n\nThere is one minor issue that will not affect the rating:\n\n1.It would be better for demonstration if the second row in Table 1 fills the entire page", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "znEXxacXvK", "reviewer": "Reviewer_uXos", "summary": "The paper presents a new problem setting for multimodal open-set test-time adaptation (MM-OSTTA), where the multimodal test data stream contains both known and unknown classes. The paper presents adaptive entropy-aware optimization (AEO) to tackle the MM-OSTTA problem. Intensive evaluations on various multimodal open-set scenarios demonstrate the effectiveness of AEO.", "questions": "- Please discuss how diverse open-set data (e.g., SoTTA [b]) would affect the methods and results.\n- How is the method sensitive to loss hyperparameters $\\gamma_1, \\gamma_2$?\n- Why did the authors select hyperparameter $\\alpha=0.7$ where 0.8 outperforms in Figure 5?\n- The paper used softmax probability as the prediction score, which does not align with the entropy-based adaptation method. Why did the authors decide to use softmax instead of entropy?\n\n\n----\n\nCitations\n> [b] Gong, T., Kim, Y., Lee, T., Chottananurak, S., & Lee, S. J. (2024). SoTTA: Robust Test-Time Adaptation on Noisy Data Streams. Advances in Neural Information Processing Systems, 36.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- Introduces a new problem setting of multimodal open-set TTA.\n- The proposed method (AEO) performs robustly among various datasets and scenarios by successfully discriminating between known and unknown samples.", "weaknesses": "- The paper needs better motivation/discussion of the new setting (MM-OSTTA) regarding how/why existing open-set TTA approaches fail in multimodal settings. \n\n- The method seems incremental, combining multiple loss terms of (1) entropy discrimination (similar to OOD detection works [a]), (2) entropy minimization (similar to TTA works), (3) discrepancy minimization across modalities, and (4) negative entropy loss (as cited in the paper).\n\n- The design of AMP with three objectives requires further justification (e.g., deeper ablation study of AMP for each loss function) and an explanation of how they connect to contribute.\n\n\n- The proposed method performs better with more unknown samples (Table 10), which is counter-intuitive and lacks proper explanation.\n\n\n\n\n----\n\nCitations\n> [a] Liu, W., Wang, X., Owens, J. D., & Li, Y. (2020). Energy-based Out-of-distribution Detection. Advances in Neural Information Processing Systems, 2020-December. https://arxiv.org/abs/2010.03759v4", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)