Test-Time Ensemble via Linear Mode Connectivity: A Path to Better Adaptation
Authors: Byungjai Kim
Chanho Ahn
Wissam J. Baddar
Kikyung Kim
HUIJIN LEE
Saehyun Ahn
Seungju Han
Sungjoo Suh
Eunho Yang
Venue: ICLR 2025 Poster (2025)
Keywords: test-time adaptation
domain adaptation
linear mode connectivity
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: Leveraging linear mode connectivity to enhance test-time adaptation through ensemble methods.

Abstract: Test-time adaptation updates pretrained models on the fly to handle distribution shifts in test data. While existing research has focused on stable optimization during adaptation, less attention has been given to enhancing model representations for adaptation capability. To address this gap, we propose Test-Time Ensemble (TTE) grounded in the intriguing property of linear mode connectivity. TTE leverages ensemble strategies during adaptation: 1) adaptively averaging the parameter weights of assorted test-time adapted models and 2) incorporating dropout to further promote representation diversity. These strategies encapsulate model diversity into a single model, avoiding computational burden associated with managing multiple models. Besides, we propose a robust knowledge distillation scheme to prevent model collapse, ensuring stable optimization and preserving the ensemble benefits during adaptation. Notably, TTE integrates seamlessly with existing TTA approaches, advancing their adaptation capabilities. In extensive experiments, integration with TTE consistently outperformed baseline models across various challenging scenarios, demonstrating its effectiveness and general applicability.

Published as a conference paper at ICLR 2025
TEST-TIME ENSEMBLE VIA LINEAR MODE CONNEC- TIVITY: A PATH TO BETTER ADAPTATION
Byungjai Kim'
Chanho Ahn' Saehyun Ahn' Seungju Han'
Wissam J. Baddar'
Kikyung Kim
Huijin Leeb
Sungjoo Suh'
Eunho Yang²
AA Center, Samsung Electronics Korea Advanced Institute of Science and Technology
[byungjai. kim, chanho. ahn, wisam. baddar, wisam.baddar, kk87 kk87.kim 1177.kimm@sssssuse com (huijin. lee, saehyun. ahnn, $j75.han sungjoo. suh) s @samsung com eunhoy@ml i kaist. st.ss.. att.ac.... kr
ABSTRACT
Test-time adaptation updates pretrained models on the fly to handle distribution shifts in test data. While existing research has focused on stable optimization dur- ing adaptation, less attention has been given to enhancing model representations for adaptation capability. To address this gap, we propose Test-Time Ensemble (TTE) grounded in the intriguing property of linear mode connectivity. TTE lever- ages ensemble strategies during adaptation: 1) adaptively averaging the parameter weights of assorted test-time adapted models and incorporating dropout to fur- ther promote representation diversity. These strategies encapsulate model diver- sity into single model, avoiding computational burden associated with managing multiple models. Besides, we propose robust knowledge distillation scheme to prevent model collapse, ensuring stable optimization and preserving the ensemble benefits during adaptation. Notably, TTE integrates seamlessly with existing TTA approaches, advancing their adaptation capabilities. In extensive experiments. integration with TTE consistently outperformed baseline models across various challenging scenarios, demonstrating its effectiveness and general applicability.
INTRODUCTION
machine learning advances with larger and more complex architectures, the demand for sub- stantial computational resources and extensive datasets rises. This trend has made off-the-shelf pre- trained models more valuable than training models from scratch. However, these pretrained models often struggle with data distributions that deviate from their training environments, underscoring the need for effective methods to adapt to diverse distribution shifts and maintain robust performance.
Test-Time Adaptation (TTA) has emerged as online adaptation method for handling distribution shifts in test data. By leveraging off-the-shelf models, TTA enables the adjustment of model param- to better align with test distributions (Wang al., 2021). Previous studies have highlighted the importance of maintaining optimization stability to prevent model collapse (Zhang et al. 20226]
Niu 2023 Lim et al. 2023). Addressing more practical environments, some research has tack- led stability issues related to dynamic distribution shifts that static models struggle to handle (Gong al. 2022 Wang et al., 2022 Yuan al. 2023). However, there has been limited exploration into enhancing model representations during test time to further improve adaptation capabilities.
In parallel, several studies have explored offline methods to enhance the adaptability of off-the-shelf models for out-of-diitrriuttin. Notably, empirical evidence suggests that fine-tuned models from pretrained model are 'linearly connected' (Neyshabur et al.) 2020). with the fine-tuning operation often approximated by first-order or linear expansions (Jacot et al., 2018 Wortsman et al.) 2022) Evei et al. 2022). This property enables straightforward techniques for improving domain general- ization. For instance, several approaches have constructed ensembles by averaging the weights of fine-tuned models (Rame et al. 2022 Wortsman et al. 2022), where these ensembles, built from a
"Corresponding Author

---

Published as a conference paper at ICLR 2025
single model, incur additional computational cost during inference. Recent work has also utilized high-level dropout rates to mitigate shortcut learning and encourage diverse collaboration of exist- ing representations (Zhang & Bottou, 2024). As result, robust fine-tuning seeks enhance the representation of off-the-shelf models under linear approximation conditions, demonstrating strong generalization performance across distribution shifts (Zhang et al. 2022a Zhang & Bottou 2023
This paper explores the intersection between TTA and robust fine-tuning, both enhancing model performance on out-of-diitribution but differing in their online and offline operation, respectively. Building on the insight that fine-tuning under linear approximation can streamline adaptation pro- cesses. we extend this advantage to TTA. Our preliminary study empirically shows that TTA models can exhibit the linear connectivity. This finding opens the door to incorporating advancements in offline domain generalization into TTA to enrich versatile representations in test time.
We propose Test-Timme Ensemble (TTE) that leverages ensemble strategies to dynamically enrich representations during online adaptation. TTE constructs an ensemble network by adaptively aver- aging the parameter weights of assorted TTA models. Notably, unlike prior methods that require multiple predictions to improve adaptation (Jang et al., 2022 Yuan et al. 2023). this simple weight averaging captures model diversity in a single model, boosting representation quality but reducing the computational burden of multiple model inference. To further promote diverse collaboration among the representations within TTA models, we incorporate dropout, ensuring it does not hinder adaptation or inference. However, existing TTA models often collapse, such as consistently assign- ing all samples the same class. Ensemble from such unstable models can lead to performance degradation. To address the instability, we propose de-biased and noise-robust knowledge distilla- tion schemes to stabilize the learning of TTA models within the ensemble. TTE is straightforward to implement and integrates effortlessly with existing TTA methods. By building on the foundational efforts, TTE achieves significant leap in adaptation performance.
Author contributions. 1) We reveal that TTA models exhibit linear mode connectivity, an in- triguing insight that simplifies and enhances the adaptation process. 2) Based on this insight, we introduce Test-Time Ensemble (TTE), novel and computationally efficient approach that not only enriches model representations but also stabilize TTA optimization through de-biased and noise- robust knowledge distillation. 3) TTE integrates effortlessly with existing TTA methods, enhancing adaptation in diverse scenarios and showing potential for applicability to future TTA methods.
2
PRELIMINARIES
This section covers the main categories of prior work and preliminary analysis that inspired the proposed TTE. Detailed discussions of related works are included in Appendix A
2.1 TEST-TIME ADAPTATION
Let f(0) a pretrained model with parameters 0 for the task X - y where training inputs Tay Xer and labels E V. TTA adapts the model ff to learn Xie - y only using out-of-distribution test inputs Fte Xte in online. The majority of objective functions in TTA are based entropy minimization (Wang et al., 202T), which aims to reduce uncertainty in the model's predictions, g = fatee 0). by adjusting the affine transformation layers of f. Specifically, Shannon entropy is adopted as (t) - p(O*) logp(("), where p(O") denotes the probability of class e with the softmax function p. Entropy minimization offers inndinggindeeendent method that effectively adapts off-the-shell models for TTA. However, it is prone to model collapse, especially under severe distribution shifts (Gong et al. 2022).
To address challenges in test-time adaptation (TTA), previous approaches have focused on ensuring stable optimization through methods such as selecting reliable samples (Niu et al. 2023 2022
Yuan et al.] 2023 Lee et al. (204)), maintaining prediction consistency (Wang et al. 2022 Zhang
et al. et al. 2023 Zhao et al. 2023), and using robust optimizers (Niu et al. 2023 Gong et al. 2024).
2022b Chen et al. 2022), manipulating affine transform statistics (Gong et al. [022] Lim
In contrast, our work shifts the focus to enhancing adaptation capability, a less explored aspect of TTA research. Some methods have aimed to improve adaptation through dense image augmentation Yuan et al. 2023 Döbler et al.) (023) or multiple predictions (Jang et al., 2022). Unfortunately, such strategies resulted in heavy computational complexity.
2

---

Published as a conference paper at ICLR 2025
TTA (CoUs5)) TTA (Defoc)
(Geuss) TTA
TIA (Geuss) TTA (Contr)
bucpubapece interpetat interpo witern
A25 on ofher 33 COTTuAAt1
13 comuuntooee
46
48-
407
65
14
45
43
oulput-space
-46
35.0
31.5
40.0
42.5
45.0
47.3
-2
40
30
42.5 40.0
4b.0
47.5
32.0
32.5 35.0
Snow
Figure 1: Accuracy trajectories of weight-space and output-space interpolated TTA models in ImageNet-C with 15 corruptions (level 5). The interpolation is performed with adjusting a from 0 to 1 in steps of 0.1. The x-axis shows accuracy for the target corruptions (or distributions) which two TTA models are adapted for and the y-axis for all non-targets. Interpolating model weights (solid line) and interpolating outputs (dashed line) often show similar trajectories.
2.2 CAN TEST-TIME ADAPTATION BE APPROXIMATED BY LINEAR EXPANSION?
Domain generalization under linear approximation. Several studies provide empirical evidence that fine-tuning a pretrained model can be approximated by a linear expansion when using a dataset much smaller than what is needed to train the network from scratch (Jacot et al. 2018 Maddox et al., (021). This linear approximation simplifies the process and offers a path way for applying straightforward strategies to enhance domain generalization. Some researchers have revisited model weight averaging, originally designed for convex problems (Rame et al. 2022; Wortsman et al., 2022), and introduced fine-tuning methods under the premise that fine-tuning primarily leverages existing representations in pretrained models (Evci et al. 2022, Zhang & Bottou 2024).
We view TTA as a form of domain generalization, where adaptation to currently shifted test samples seeks to improve performance on future samples with potentially different distributions. Building on previous work on fine-tuning with linear expansion, we investigate whether TTA, fine-tuning on domain-shifted test data, can also be approximated by linear expansion.
Preliminary analysis. We investigate whether TTA models can be approximated by a linear expan- sion. Using the setup from Frankle et al. (2020); Wortsman al. 2022), Linear Mode Connectivity TTA empirically demonstrated, as described by
(1 a) (0)) + (C))((2) Accpp(1-o)) - a1) 02)
where 8 and 0g are the weights of two TTA models adapted to different distributions, a is a mix- ing coefficient in the range (0, 1] and Acep is classification accuracy for certain distribution D. This property suggests that averaging the weights of two models can yield better performance than each model by mimicking output-space ensemble effect. The connection between weight-space and output-space averaging implies that 0, and 0g can be linearly approximated around the pretrained Og indicating that they are linearly connected (Fort et al. 2222)). Neural networks, being non-linear, usually do not benefit from weight interpolation, but linear mode connectivity enables enhanced performance. Appendix[B.. presents theoretical analysis for linear approximation in TTA.
Figure Sourepprees presents the preliminary results for linear mode connectivity, using a visualization method for robustness in distribution shifts (Recht et al., 2019 Taori al., 2020). Weight-space interpo- lation can provide higher accuracy both on target and non-target corruptions than single models, following the accuracy trajectories of output-space interpolation. Although trajectory differences grow the corruption properties become more distinct (noise and blur in Figure (a)), perfor- mance improvements are still valid. The results suggest that TTA models can be linearly connected across different distributions, indicating the applicability of domain generalization strategies based on linear approximation. Appendix BB. describes the details of the preliminary experiments. We also report additional preliminary results under various TTA scenarios in Appendix B.3
3

---

Published as a conference paper at ICLR 2025
Conventional TTA methods
Accuracy (3)
Applying drapout in TTE
ImageNet-C (Label Shifta)
fd
Lizil Oa. Na:
Luca Br: 0a)
Caiux
TENT - TTE
60.0
Ou. Oax
Oet
W
high
SAR
test
weight-apace ensemble
SAR + TTE
[x
blas-gulding score
low
1ha class
DeYO + TTE
63.2
soffmax
TTE objective
stop gradient
Lute
Lua
10etall
Figure 2: Overall framework of TTE. The ensemble network fe is constructed from fa using weight- space and dropout ensembles. Ensemble outputs De serve predictions and drive de-biased, noise- robust knowledge distillation to enhance the representation and stability of fa- The procedures integrate seamlessly with existing TTA methods, yielding significant accuracy improvements across three representative methods. Stronger methods achieve even higher performance through TTE.
TEST-TIME ENSEMBLE
We propose test-time ensemble (TTE) approach, designed to enhance the adaptation capabilities of existing TTA methods (Figure 2} TTE is built on teacher-student framework, comprising an adapter network fa and an ensemble network fe- The proposed TTE involves 1) constructing fe from orignal TTA models fa through adaptive ensemble strategies and 2) distilling the knowledge from fe to fa ensure stable optimization. The proposed knowledge distillation maintains linear mode connectivity between fa and fe during TTA and also mitigates model collapse, common issue in TTA processes. Further details are provided the following sections.
3.1 ENSEMBLE STRATEGIES FOR ENHANCING ADAPTATION PERFORMANCE
TTE uses the same input view for both fe and fa without augmentation. Instead of multiple pre- dictions with dense augmentations in teacher-student frameworks (Wang et al.) 2022 Yuan et al., propose computationally efficient ensemble strategies enhance the representation of Ja-
2023
Adaptive weight-space ensemble. Our preliminary results in Figure underscore two insights: 1) TTA models even adapted different distributions would have non-redundant representations and their ensemble are further beneficial for diverse distribution shifts. 2) Weight-space interpolation can emulate the rich representations of ensembles with a single model, eliminating the computational burden of multiple models at test time. In this paper, we propose an online weight-space ensemble approach, where the weights of fa iteratively updated via exponential moving average (EMA) as 0g < mQ + (1 mm with momentum m, where Og and 0g denote weights of Je and fa. respectively. Our preliminary findings suggest that using a wide range of momentum m does not hinder optimization but rather enhances domain adaptation. Thus, we propose adaptive EMA modulated by the divergence Lyki, which measures the probability distance between fo and fa-
771 = mg Lex//T
where my is a base momentum and 7 temperature controlling sensitivity to the divergence. Eq. promotes lower momentum to actively construct ensembles when Ja and fa have different repre- sentations. This adaptive scheme particularly valuable online TTA, where data distributions are unknown and subject dynamic shifts. The method for calculating the divergence Lyki is described in Section 3.2
Dropout ensemble. Dropout is traditionally used to introduce noise that hinders optimization, pre- venting overfitting, and is typically applied in long-range learning a non-linear function from scratch. However, recent studies have shown that applying dropout during fine-tuning create diverse collab- oration between existing features Zhang & Bottou| (2024), based on the assumption that fine-tuning pretrained model is near-linear process that primarily leverages existing representations Evei
4

---

Published as a conference paper at ICLR 2025
[a!
0.5
Shot
0.5
Shot notse
55-
Loss bernter betiveen and
14
TTA
Berrter betweeen and
030
12-
Ensemble
800
Hue Iverage sccuracy
50
Prsstctdd clase
10
0
Predited class
600
45-
TA Ensemble
0.2
400
C
400
35
Loss barrerr
0,7
2:00
C2
200
0.0
0.0
Online
Online
Online
Figure 3: Failure case study. Classification accuracy in applying the proposed ensemble strategies to a conventional TTA approach (Tent, WWng et al. (202)) The loss barrier between fa and fe is measured, as a common metric for assessing linear mode connectivity between them (Eq. 7 in Appendix). The predictions in (b) and (d) present the model outcomes of ensemble fe- Details of this study are included in Appendix D.I
al. [2022] Jacot et al. (2018). Building on this insight, TTE incorporates dropout in the penul- timate layer of fa during test-time adaptation as pa(sp) ha (dropout(saa>);)))) p)), where ho and Ga are the linear head and the backbone of fa. respectively, and p the dropout probability. This dropout encourages model diversity of fa during exploring distribution shifts in test data, which is then transferred to fe through weight-space ensemble.
3.2 DE-BIASED AND NOISE-ROBUS KNOWLEDGE DISTILLATION
Existing TTA models often collapse, such as consistently assigning all samples to improperly biased classes (Niu et al. 2023). To solve the issues, TTE employs robust knowledge distillation objective within a teacher-student framework for stable optimization, addressing a key degradation factor in output de: bias as a prominent aspect of model collapse.
De-biased distillation. Simply constructing ensembles with unstable TTA models can rather de- grade performance. Figure 3 illustrates failure cases: while ensemble improves performance under glass blur, it degrades as the TTA model collapses under shot noise, providing long-range biased predictions (Fig. 3()). terms of linear mode connectivity, the loss barrier spikes as the TTA model Ja begins to collapse (Fig. 3())). This indicates that the linear connectivity between the TTA model Sa and its ensemble fe disrupted on the loss surface, stopping the benefits of the ensemble for adaptation. To solve collapse issues, we introduce de-biased representation y'. for knowledge distillation to reduce improper bias in fo and maintain linear mode connectivity. First, we quantify the accumulated bias Cbias in fa by applying EMA to the first-order batch statistics of ya, follows
Cbias n2 Cbias + (1 (1-n) n).
I
Σ
Ja.s
(3)
M
where n is momentum constant, M is a batch size and Cbias is initialized as all zero. Due to the long-range effects of improper bias, Cbias converges into a specific distribution when model collapse manifests. Second, we identify bias-guiding samples from 9. by measuring the cosine similarity between Coias and each output
50-
Credtent rate tevo predicter)
1.00
La Lo (Oum))
23.5
18
8
0.85
16
Lap
Lay -aa (Oum)
15
400
500
1200
2000
0.50
4303
800
1200
1800
y. Tesazll Me.aa]" As s; increases, ÿe,; is
Du..
Orline batch
batth
more likely to intensify the bias in fa during distillation. Third, for bias-guiding samples, Do.s is adjusted to construct de-biased repre- sentations V The Ve,, is generally described with a weight function w(s) = max(0, a s) with scale a as follows
Figure 4: Comparisons of standard and reverse
KL divergence with Gaussian noise on ImageNet- C (Level 5), combined with the TTA objective (Tent, Wang et al. (202T)).
ye, Do.i w((s)) Chias-
5

---

Published as a conference paper at ICLR 2025
When 8; >> 0, ye.s is adjusted with Cbias to introduce the effects of label smoothing, increasing the probabilistic distance between Vo, and Chios during distillation. This adjustment acts as a regularizer for the unsupervised TTA process, helping to prevent model collapse.
Noise-robust distillation. Besides, to mitigate the impact of noisy predictions in De. which are used pseudo-labels for fa during distillation, we employ reverse KL divergence as the distillation objective. Figure 4 compares standard and reverse KL. divergence when they integrate with a con- ventional TTA approach. Reverse divergence improves accuracy over standard one by penalizing the gradient of incorrect predictions. True labels are used to distinguish correct from incorrect samples, and the gradient ratio calculated by dividing the gradient of incorrect predictions by that of correct ones. These findings are consistent with results observed in supervised learning with noisy labels Wang et al. 2019b). The mathematical analysis is included in Appendix [D.2
Consequently, the proposed knowledge distillation objective is mathematically defined the Kullback- (KL) divergence between the predictions of fa and fc as L (Ua: V1) LL (.). This term is also used to dynamically update the momentum value in Eq.
3.3 OBJECTIVE FUNCTION IN TTE
Consequently, the objective function for TTE combines the existing TTA objective Lita with the knowledge distillation objective Lykl. Note that Lito usually based on entropy Hūa)) in prior TTA approaches (Wang et al., 2021;; NNu et al. [023] Lee et al.) 2024). The combined objective can be interpreted as aiming to reduce the uncertainty of Ya while ensuring noise-robust alignment with the de-biased ensemble output from V'. as given by
Niu
Lete(Bo g.: 0a)
0a) + Lrki(Das 8a)-
(5)
To avoid over-tuned hyperparameter configurations, the two objective terms are assigned equal weights. Beyond basic entropy minimization, some prior approaches have proposed reliable sam- ple selection weighting strategies (Niu et al., [202]] Lee et al.) (022)), and robust optimization techniques (Niu et al.] 2003).. In our framework, we integrate these methods directly into the Lute objective. Algorithms outline how TTE performs with each prior approach in Appendix C.2
EXPERIMENTS
Datasets and models. We conducted experiments with four benchmark datasets: ImageNet-C (Apache-2.0 License) (Hendrycks & Dietterich 2019) assesses adaptation performance under 15 types of corruptions at five severity levels, reflecting extreme distribution shifts from the original Im- ageNet (Deng et al. 2209). All experiments were conducted at level 5, the most severe. ImageNet-S (MIT License) (Wang et al. 2019a) and ImageNet-R (MIT License) (Hendrycks et al., 202T eval- uated adaptation performance under natural distribution shifts. Unlike ImageNet-C's 's artificial cor- ruptions, ImageNet-S features sketch-style images for every ImageNet classes, while ImageNet-R includes diverse renditions of 200 ImageNet classes, such art, cartoons, graffiti, etc. ImageNet- V2 (MIT License) (Recht et al., 2019) consists of data sampled after a decade of progress on the original ImageNet dataset. It was used to measure performance unaffected by adaptive overfitting, offering a measure of adaptation performance to intrinsic shifts.
We evaluated two types of architectures: Vision Transformer Base (ViTBase) and ResNet-50 with Group Normalization (ResNet50-GN), Architectures with batch normalization were excluded due their batch size sensitivity and instability during the TTA process (Niu et al.) 2023 Mounsaveng et al. 2024). The models' parameters were initialized using pre-trained weights from the PyTorch Image Models library (Wightman) 2019). For adaptation, the affine parameters of normalization layers in each architecture were trainable. For the ensemble strategies in TTE, the temperature was set to 1.0 with a dropout ratio of 0.9 for ResNe150-GN, and T = 10.0 with a dropout ratio of 0.4 for ViTBase. The value of mg was fixed at 1.0 for both ResNet and ViT models. For de-biased knowledge distillation, n was set to 0.99 and a to 3.0. These settings were consistent across all test-time scenarios and baseline methods integrated with TTE to avoid over-tuned hyperparameter configuration. When integrating TTE with existing TTA approaches, we followed the original im- plementations and hyperparameter settings specified in their papers ensure accurate evaluation

---

Published as a conference paper at ICLR 2025
Table 1: Integration with previous TTA approaches. Classification accuracy (%) with Label Shifts and Batch Size setups (ImageNet-C. level 5). Underline depicts performance improvement when applying TTE. Bold numbers are the best results.
Gauss
Impal
Defoc
Zooen
Snow
Frost
Fog
Contr
JPEG
Avg
ResNer50-GGN
18.0
17.9
19.8
11.4
40.4
47.3
36.3
18.6
18.9
36.2
27.3
29.6
25.8
28.6
50.2
48.8
42.2
40.1
36.4
38.4
27.8
71.9
36.2
37.4
32.2
71.9
42.6
52.8
DeYo
781
44.3
42.9
11.4
16.6
$2.9
$2.0
9777
59.4
-01.8
DeYo TTE
43.0
44.0
34.4
46.0
55.2
73.4
54.4
ViTBase
9.4
23.4
27.0
15.8
26.3
54.7
49.9
30.5
44.5
13.3
59.7
45.4
47.9
55.4
54.4
59.0
55.3
62.0
69.88
46.6
20.5
55.2
54.2
59.0
58.0
44.1
69.8
76.2
600.9
57.2
SAR
48.6
48.4
55.8
55.2
59.4
63.2
62.4
76.4
66.4
62.4
67.0
60.5
DeYo
19.1
53.6
57.6
58.6
63.8
31.5
67.9
06.0
733.1
71.3
68.6
70.1
DoYo TIE
$4,0
54.7
55.2
58.9
59.7
68.3
66.7
73.9
78.0
68.3
69.3
70.3
65.2
Batch Size 1
Gnass
Impal
Defoc
Glass
Mosion
Zoom
Snow
Fiost
Fog
Brit
Contr
Blastic
Pixel
JPBG
AAg
ResNer3D-GiN
18.0
17.9
19.8
11.4
21.4
40.4
47.3
36.3
18.6
Tent TTE
3.1
3.7
16.6
5.2
20.0
17.7
25.1
72.0
46.2
8.1
41.6
33.8
31.2
53.5
52.3
$9.3
$1.3
47.8
23.4
26.5
23.9
18.4
15511
30.3
44.4
44.8
27.44
72.3
44.7
14.6
47.0
34.5
SAR TTE
25.9
28.6
26.7
23.7
17.7
30.8
32.4
48.0
46.1
42.1
72.2
45.2
34.2
47.7
56.1
38.5
DeYo
41.3
44.3
42.4
23.7
2.511
41.4
19.9
54.6
52.2
1.9
73.4
53.4
30.9
59.3
59.7
-22.2
DeYo + TTF
42.5
44.9
43.5
34.8
32.88
43.3
45.9
55.8
53.7
60..
73.4
54.4
$1.0
6019
60.4
511.5
ViTBase
9.4
6.7
8.1
29.1
23,4
34.0
27.0
15.88
26.3
47.4
54.7
43.9
30.5
44.5
47.6
29.9
Tent
43.2
1.6
44.0
52.6
48.9
55.8
51.2
22.3
31,5
67.0
75.0
64.9
54.3
67.2
64.4
48.9
Tonl TTE
49.2
.8.8
50.1
56.2
55.8
60.2
56.9
64.3
63.6
71.4
76.9
67.0
64.1
70.7
68.1
61.6
SAR
40.9
16.6
41.9
53.4
50.5
$7.4
52.9
59.1
57.2
68.9
75.5
65.6
58.1
68.3
65.9
56.9
SAR 4 TTE
43.6
40.4
44.3
55.2
53.1
59.2
55.4
61.5
61.9
70.7
76.7
66.7
61.9
70.2
67.3
59.2
DeYo
53.1
51.2
54.3
58.8
59.6
64.0
37.4
68.
66.4
73.7
78.3
68.2
68.5
73.7
70.5
63.1
DeYo A TTE
53.8
54.0
54.6
59.1
59.7
64.4
62.3
68.4
66.8
73.9
78.3
68.5
69.2
73.9
70.7
65.2
of TTE's effects. Comparative methods are introduced in each comparison section. Implementation
details are provided in Appendix C.I and C.2
4.1 COMPARISON STUDY
Integration with previous TTA approaches. TTE was applied to three representative TTA approaches to verify its effectiveness and general applicability: Tent (Wang et al. (021) introduced entropy mini- mization; SAR (Niu et al. 2023) added sharpness- aware entropy minimization for optimization stabil- ity: and De (Lee al., (024) introduced object- based sample weighting and selection. We followed the three wild test scenarios from Niu et al. (2023) using ImageNet-C: Label Shifts where batches are class-imbalanced with most samples belonging to the same class, Batch Size 1 where each batch con- tains only one sample, testing adaptation with minimal information, and Mix Shifts where batches contain samples from various distributions, testing adaptation with multiple shifts simultaneously.
Table 2: Classification accuracy (%) with
the Mix Shifts setup (ImageNet-C, level 5). Underline depicts improvement when apply- ing TTE. Bold presents the best results.
Methods
ResNe150-GN
ViTBase
No Adapt
30.6
29.9
* Tent
33.1
$2.3
Teat+TTE
38.7
57.2
SAR
38.1
57.1
SAR+TTE
39.1
57.5
DeYO
33.8
58.6
DeYO+TTE
42.9
68.6
Table I presents classification accuracy across 15 distributions in ImageNet-C with the two setups of Label Shifts and Batch Size 1. Table 2 shows classification accuracy for the Mix Shifts setup. The results along with the standard deviations are detailed in Table [4| and T3 Integrating TTE (+TTE) significantly improved accuracy compared using the original methods alone. Interest- ingly, while Tent initially performed worse than SAR, Tent+TTE achieved comparable results to SAR+TTE Label/Mix Shifts and even outperforms SAR+TTE in Batch Size (48.7% vs. 38.5% on ResNet50-GN, and 61 31.6% vs. 59.2% on ViTBase). This suggests that sharpness-awar optimiza- tion in SAR has less benefit for adaptation with small batch size. The DeYO+TTE delivered the best performance across all challenging scenarios, outperforming DeYO. Specifically, it achieved an average accuracy improvement of +9.9% in Label Shifts, +8.3% in Batch Size 1, and +1 +11.1% 1.1% in Mix Shifts ResNet50-GN and +3.9%, +2.1%, and +1.9% on ViTBase. Notably, +TTE remained stable across most setups while previous methods experienced model collapse in certain cases (c.g., shot noise in Label Shifts). The experiments showed that TTE enhance adaptation performance
7

---

Published as a conference paper at ICLR 2025
Table 3: Continual TTA with non-i..d. conditions. Classification accuracy (%) with ImageNet-C
(level 5). Bold numbers are the best results.
Adaptaaion Order (-)
Methods
Gauss
Defoc
Snow
Contr
Shot
Glass
Prost
Elastic
Impui
Motion
Fag
Pixel
Zooth
Brit
JPEG
Aorg
RedNertO.GG
18.0
19.8
40.4
36.3
19.8
11.4
47.3
18.6
17.9
21.4
33.6
28.4
24.9
69.3
52.3
50.6
Tent
3.9
1.8
1.6
0.1
0.1
0.1
0.1
0.1
0.1
0.1
0.1
0.6
CoTTA
23.5
5,5
2,0
0.7
0.2
0.2
0.1
0.1
0.1
0.1
2.2
SAR
33.1
16.8
44.7
44.6
42.3
18.8
45.7
37.8
39.7
9.3
3.1
0.7
5,4
1.2
23.0
DaYO
28.1
3.7
7.2
0.99
0.1
0.1
0.1
0.1
0.1
0.1
0.1
0.1
0.1
0.1
0.1
2.8
TTE (w. DeYO)
43.0
31.7
52.7
52.3
45.1
36.4
51.2
53.6
40.9
43.0
59.1
60.3
47.4
68.9
59.0
49.7
VTTBuse
9.4
29.1
43.9
6.7
23.4
26.3
30.5
34.0
47.4
44.5
27.0
54.7
47.6
29.9
Tent
30.9
7.1
CoTTA
34.1
3,4
SAR
46.5
54.2
54.7
38.7
38.9
45.3
45.1
37.5
40.0
45.8
50.0
37.3
58.6
48.6
46.0
D=YO
19.1
36.7
63.3
61.4
52.8
59.1
62.2
54.0
69.2
2.2
61.8
67.2
53.5
TTE DeYO)
54,0
55.2
64.6
62.5
54.2
55.9
66.8
52.0
59.1
68.5
69.9
55.2
74,9
68.4
61.6
Table 4: TTA with natural distribution shifts. Classification accuracy (%) with ImageNet- (S), ImageNet-Renditee (R), ImageNet- V2 (V2). Bold numbers are the best results.
(a) Label Shifts
(b) Batch Size
Aug
Avg
ResNe150-GN
29.2
40.8
68.9
46.3
ResNe150-GN
29.2
40.8
68.9
46.3
Tont
41.5
68.9
47.1
Tent
31.9
42.1
68.9
47.6
CoTTA
30.8
41.2
68.8
46.9
CoTTA
26.7
40.6
68.7
45.3
SAR
30.6
41.6
68.9
47.0
SAR
31.6
41.9
68.9
47.5
DeYO
34.6
44.4
68.9
49.3
DeYO
37.0
46.0
69.0
50.7
TTE (w. DeYO)
36.9
45.1
68.9
50.3
TTE DeYO)
39.1
47.1
69.0
$1.7
ViTBase
18.2
43.1
66.2
37,9
VTBBase
18.2
43.1
66.2
37,9
Tent
8.8
41.9
68.9
39.8
Tent
7.0
40.6
69.1
38.9
CoTTA
28.4
45.1
67.5
47.0
COTTA
24.0
46.0
66.9
45.6
DeYO
42.4
58.6
71.1
57.3
DeYO
43.6
38,8
71.2
58.2
TTF (w: DeYO)
43.6
59.1
71.1
57.9
TTE (w: DeYO)
45.0
61.3
71.3
59.2
even fürther when paired models have stronger representations. TTE would have potential to boost performance in future TTA methods.
Continual TTA with non-i.id. conditions. We extended the ImageNet- C scenarios to continual TTA with non-i.i.d distributions and classes. These challenging scenarios require models adapt to imbalanced classes (Label Shifts) across 15 corruptions continuously, setting where TTA models are prone collapse. For comparison, we included CoTTA (Wang et al. 2022), originally designed reports classification accuracy, with detailed results and standard deviations in Table I6 TTE consistently achieved the highest average accuracy, with 49.7% on ResNe150-GN and 61.6% on ViTBase, while other methods, including De YO, showed unstable and near-zero accuracy in later adaptation stages (all comparative methods on ResNet50 GN and Tent, CoTTA ViTBase). The results highlight that ensemble strategies improve model representations for better adaptation, while robust knowledge distillation stabilizes TTA optimization and prevents model collapse. Furthermore, we conducted these experiments with other baseline methods integrated with TTE, as shown in Table II and compared their performance in addressing the issue of catastrophic forgetting in Figure
for continual TTA with teacher-student framework. Table 3
Natural distribution shifts. We evaluated adaptation under natural distribution shifts by combining two challenging scenarios: Label Shifts and Batch Size 1. Table 4 presents classification accuracy for both ResNet50- GN and ViTBase. TTE proved highly effective on natural distribution shifts, consistently outperformed De with a +1.0% gain on ResNet50-Gr and +0.6% on ViTBase for Label Shifts, and +1 +1.0% on ResNet50-GN and +1.0% on ViTBase in the Batch Size 1. Notably, the ensemble strategies demonstrated strong benefits on ImageNet-R where natural shifts occur con- currently, outperforming De across all cases. However, in ImageNet-Ve V2, which features intrinsic shifts, TTE provided modest gains (0.0% to 0.1%) compared to Yo suggesting that further in- vestigation is needed for this type of shift. Additionally, Table T2 reports the results of other baseline methods integrated with TTE.
Computational complexity. Integrating TTE required only additional feedforward pass for fe, resulting in minimal computational overhead. Table | in Appendix compares the computational complexity with baseline models. Further discussion is provided in Appendix C3
8

---

Published as a conference paper at ICLR 2025
Table Impacts of TTE components on classification accuracy (%) Label Shifts with ImageNet-
C. DB: de- de-biasing in knowledge distillation, WSE: weight-space ensemble with adaptive momen- tum, RKL: reverse KL divergence, Do: dropout ensemble.
Methods
Impul
Defoc
Glass
Fog
Contr
IPEG
Avg
ResNe150.GN
19.8
19.8
37.7
32.8
36.3
23.7
20
35.6
51.7
72.6
49.0
57.6
42.5
42.5
32.7
59.2
59.5
59.7
+DB-WSE+RKL
42.6
44.9
34.4
33.0
12.9
444
54.0
52.7
59.5
73.3
53.7
19.9
59.6
59.7
49.9
43.0
45.3
44.0
34.4
3366
43.4
46.0
56.2
53.6
61.0
73.4
54.4
51.5
61.0
60.2
50.7
ViTBase
9.1
6.7
29.1
23.4
34.0
27.0
15.8
26.3
17.4
51.7
43.9
30.5
44.5
47.6
29.9
DeYO
49.1
35.9
53.6
57.6
58.6
63.8
37.5
67.9
66.0
73.1
77,9
66.5
68.6
73.5
70.1
61.3
:DB
49.3
47.8
504
57.0
$6.1
60.7
55.8
64.9
64.3
71.7
78.0
66.3
65.0
72.0
68.9
61.9
DB WSE
53.0
50.7
$4.1
58.7
59.0
63.3
59.6
61.3
66.1
73.4
78.2
68.5
68.2
73.2
69.9
64.2
-DB-WSE-RKL
53.4
53.3
54
SK.9
59.3
61.8
59.6
67.7
66
73.7
78.2
68.5
68.5
73.4
70.1
64.6
SSE+RKIADO
54.0
54.7
55.2
5M.9
59.7
64.5
62.1
68.3
66.7
73.9
78.0
6X.3
69.3
73.8
70.3
68.2
0.96
Constart S4 (woq
R
Window
0.94
Cark
Hoalic
Food
Zoom
JPRG
Figure 5: Momentum profile in Continual TTA with non i.id. conditions (left axis). Classification accuracy (%) of TTE, reported to compare constant and adaptive in this setup. (right axis)
4.2 ABLATION STUDY
Impact of the proposed components. We analyzed the contribution of each proposed component to improving the baseline DeYO. Table 5J shows classification accuracy as components are added sequentially. First, constructed a conventional teacher-student framework with DeYO, updated using EMA with m 0.999 and standard KL divergence for knowledge distillation, which are the standard setting in such frameworks (Performance for this setup not included). Adding de- biasing in knowledge distillation (+DB) prevented collapse and led to general gains of +1.7% on ResNet50-GN and +0.6% on ViTBase in average. Introducing the weight-space ensemble with adaptive momentum (+WSE) improved performance across all distributions, with average gains of +7.2% on ResNet50-GN and +2.3% on ViTBase. Replacing the standard KL divergence with reverse KL (+RKL) provided the average gains of +0.2% on ResNet50-GN and +0.4% on ViTBase. Lastly, applying dropout to fa (+DO) further improved average accuracy by +0.8% on ResNet50-GN and +0.6% on ViTBase.
Adaptive momentum. Table compares the proposed adap- tive momentum scheme with the conventional constant scheme across a wide range of m values Label Shifts for TTE with DeYO. The commonly used value of from other momentum-basee approaches (Wang et al. 2022 Yuan et al. 2023) did not yield significant ensemble et- fects, resulting in lower accuracy. In contrast, lower momen- tum values enhanced the ensemble effect, leading better srrformance-specfficllly
Table 6: Classification accuracy (%) with varying momentum val- ues in TTE (ImageNet-C, level 5).
0.999
ResNe150-GN
ViTBase
DeYO
40.8
61.3
0.999
62.0
0.99
49.3
65.0
64.8
= 0.5 for ResNet50-GN and m = 0.99 for ViTBase. The proposed adaptive momentum scheme, which lower momentum based probabilistic dis- tance, achieved the best performance for ViTBase. For ResNet, lower momentum generally worked well. These results consistent with preliminary study, which observed linear mode connec- riviry between TTA models.
no 0.5
50.7
64.6
Adaptive
65.2
To further verify effectiveness of the adaptive scheme, tested it in the challenging scenario of Continual TTA with non-i.idd conditions for ViTBase, where distribution shifts were sequentially changed. Figure illustrates the momentum values during TTA and compares classification aceu-

---

Published as a conference paper at ICLR 2025
ResNet50
ResNetS0
5q
40
40.9
Avcroae accursty
60
DeVo (PeNNe55 680 40
60.5
Devo OVITBwad 63.3
40
0.0
244
0.9
00
0.4
0.5
0.8
0.9
Figure 6: Hyperparameter Sensitivity. Average classification accuracy (%) in Label Shifts (ImageNet-C. level 5). If not specified in experiments, the default values were: dropout ratio 0.9, T = 15.0 for ResNet50-GN and dropout ratio = 0.4, r = 15.0 for ViTBase.
racy between constant and adaptive schemes. Note that the constant momentum value was set to m = 0.99, as it achieved the best performance in Table As shown in Figure 5 momentum values distributions, actively promoting ensemble construction. This dynamic adjustment led to a +1.8% accuracy improvement. For additional study, the proposed adaptive weight averaging is compared with stochastic weight averaging (Izmailov et al. 2018) in Appendix D.5)
were adjusted to lower levels during transitions
Ensemble hyperparameter sensitivity. Figure 6 illustrates the impact of varying ensemble hyper- parameters (with DeYO) on ViTBase and ResNet. It is noteworthy that TTE steadily outperformed DeYO across different hyperparameter values, demonstrating the low sensitivity of hyperparameter. For dropout, increasing the ratio generally enhanced accuracy compared to ratio of 0.0. However, the extreme case of high dropout ratio above 0.9 led to decreased accuracy in ViTBase while led to increased accuracy in ResNet. For the temperature controlling decaying factor in adaptive mo- mentum, TTE achieved the best performance with 1.0 for ResNet and r 10.0 for ViTBase.
Analysis for de-biasing scheme. The de- biasing scheme, which penalizes biased rep- resentations in knowledge distillation, is more closely related to optimization stability than to adaptation capability. To assess optimization presents the standard devia- tion of performance across three random seeds. As or controlling de-biasing degree in K' ap- proched 1.0, classification accuracy dropped and standard deviation increased, indicating the models struggled with biasing under certain random seeds. When the bias update rate ap- proached 1.0, accuracy decreased with greater standard deviation, suggesting the update was too slow to estimate bias properly. Extensive experiments with firmed the de-biasing scheme effectively prevented model collapse in diverse test-time scenarios. However, observed the sensitivity of the proposed de- de-biasing scheme with true-biased scenario where same class samples were fed into the model consecutively over 100 times. Further analysis is included in Appendix D.6
stability, Figure /
2:99
Figure 7: Average classification accuracy with standard deviation in Label Shits for ViT- Base, with varying hyperparameters in di-biasing scheme. The values for ensemble hyperparame- ters were set to dropout ratio=0.4 and T 15.0.
3.0 and 0.99 con-
5 CONCLUSION
We proposed a novel test-time adaptation (TTA) approach, TTE, that enhances the performance of existing methods through ensemble strategies. TTE consistently improved adaptation, demonstrated broad applicability, and remained computationally efficient. Extensive experiments confirmed its effectiveness, and notably, TTE exhibited exceptional stability, preventing model collapse across four datasets and four test-time scenarios. This approach holds significant potential for advancing future TTA developments.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Dian Chen, Dequan Wang, Trevor Darrell, and Sayna Ebrahimi. Contrastive test-time adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 295-305, 2022.
Arkabandhu Chowdhury, Mingchao Jiang. Swarat Chaudhuri, and Chris Jermaine. Few-shot image classification: Just use library of pre-trained feature extractors and simple classifier. In Pro- ceedings of the IEEE/CVE International Conference on Computer Vision, pp. 9445-9454, 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hier- archical image database. In 2009 IEEE conference on Computer Vision and Puttern Recognition, pp. Icee, 2009.
Mario Döbler, Robert A Marsden, and Bin Yang. Robust mean teacher for continual and gradual test- time adaptation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 7704-7714, 2023.
Nikita Dvomikk Cordelia Schmid, and Julien Mairal. Selecting relevant features from multi- domain representation for few-shot classification. In Computer Vision-ECCV 2020: 16th Eu- ropean Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part X 16, pp. 769-786. Springer, 2020.
Utku Evci, Vincent Dumoulin, Hugo Larochelle, and Michael C Mozer. Head2toe: Utilizing in- termediate representations for better transfer learning. In International Conference on Machine Learning, pp. 6009-6033. PMLR, 2022.
Stanislav Fort, Gintare Karolina Dziugaite, Mansheej Paul, Sepideh Kharaghani, Daniel M Roy, and Surya Ganguli. Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the neural tangent kernel. Advances in Neural Information Processing Systems, 33:5850-5861, 2020.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear mode con- nectivity and the lottery ticket hypothesis. In International Conference on Machine Learning, pp- 3259-3269, PMLR, 2020.
Mudasir Ganaie, Minghui Hu, Ashwani Kumar Malik, Muhammad Tanveer, and Ponnuthurai N Suganthan. Ensemble deep learning: A review. Engineering Applications of Artificial Intelli- gence, 115:105151, 2022.
Yossi Gandelsman, Yu Sun, Xinlei Chen, and Alexei Efros. Test-time training with masked autoen- coders. Advances Neural Information Processing Systems, 35:29374 29385, 2022.
Taesik Gong, Jongheon Jeong, Taewon Kim, Yewon Kim, Jinwoo Shin, and Sung-Ju Lee. Note: Ro- bust continual test-time adaptation against temporal correlation. Advances Neural Information Processing Systems, 35:27253-27266, 2022.
Taesik Gong, Yewon Kim, Taeckyung Lee, Sorn Chottananurak, and Sung-Ju Lee. Sotta: Robust test-time adaptation noisy data streams, Advances in Neural Information Processing Systems, 36, 2024.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common cor- ruptions and perturbations. arXiv preprint arXiv: 12261, 2019.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of 2021.
generalization. ICCV,
Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wil- son. Averaging weights leads to wider optima and better generalization. arXiv preprint 1801:01803.05507, 2018.
11

---

Published as a conference paper at ICLR 2025
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kemel: Convergence and gen- eralization in neural networks. Advances in Neural Information Processing Systems, 31, 2018.
Minguk Jang, Bac-Young Chung, and Hye Won Chung. Test-time adaptation via self-training with nearest neighbor information. arXiv preprint 2XXiv:207..1092 2022.
Jonghyun Lee, Dahuin Jung, Saehyung Lee, Junsung Park, Juhyeon Shin, Uiwon Hwang, and Sun- groh Yoon. Entropy not enough for test-time adaptation: From the perspective of disentangled factors. In International Conference on Learning Representations, 2024.
Wei-Hong Li, Xialei Liu, and Hakan Bilen. Universal representations: A unified look at multiple task and domain learning. International Journal of Computer Vision, pp. 1-25, 2023.
Hyesu Lim, Byeonggeun Kim, Jaegul Choo, and Sungha Choi. Tin: A domain-shift aware batch normalization in test-time adaptation. arXiv preprint arxiv:2302.05155 2023.
Yuejiang Liu, Parth Kothari, Bastien Van Delft, Baptiste Bellot-Gurlet, Taylor Mordan, and Alexan- dre Alahi. Ti+++ When does self-supervised test-time training fail or thrive? Advances in Neural Information Processing Systems, 34:2 20121180822220 2021.
Wesley Maddox, Shuai Tang, Pablo Moreno, Andrew Gordon Wilson, and Andreas Damianou. Fast adaptation with linearized neural networks. In International Conference Artificial Intelligence and Statistics, pp. 2737-2745. PMLR, 2021.
Saypraseuth Mounsaveng, Florent Chiaroni, Malik Boudiaf, Marco Pedersoli, and Ismail Ben Ayed. Bag of tricks for fully test-time adaptation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1936-1945, 2024.
Fangzhou Mu, Yingyu Liang, and Yin Li. Gradients as features for deep representation learning. arXiv preprint #xXi:20004005529 2020.
Behnam Neyshabur, Hanie Sedghi, and Chiyuan Zhang. What is being transferred in transfer learn- ing? Advances in Neural Information Processing Systems, 33:512-523. 12222,,,222 2020
Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Yaofo Chen, Shijian Zheng, Peilin Zhao, and Mingkui Tan. Efficient test-time model adaptation without forgetting. In International Conference Machine Learning, pp. 16888-16905. PMLR, 2022.
Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Zhiquan Wen, Yaofo Chen, Peilin Zhao, and Mingkui Tan. Towards stable test-time adaptation in dynamic wild world. In International Conference on Learning Representations, 2023.
Alexandre Rame, Matthieu Kirehmeyer, Thibaud Rahier, Alain Rakotomamonjy, Patrick Gallinari, and Matthicu Cord. Diverse weight averaging for oft-of-distribution generalization. Advances in Neural Information Processing Systems, 35: 10821-108366 2022.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International conference on machine learning, pp. 5389-5400. PMLR, 2019.
Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei Efros, and Moritz Hardt. Test-time train- ing with self-supervision for generalization under distribution shifts. In International Conference on Machine Learning. pp.
PMLR, 2020.
Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht, and Ludwig Schmidt. Measuring robustness to natural distribution shifts in image classification. Advances in Neural Information Processing Systems, 33:18583-18599, 2020.
Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. In International Conference on Learning Repre- sentations, 2021.
Haohan Wang, songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representa- tions by penalizing local predictive power. In Advances in Neural Information Processing Sys- tems, pp. 10506-10518 2019a.
12

---

Published as a conference paper at ICLR 2025
Qin Wang, Olga Fink, Luc Van Gool, and Dengxin Dai. Continual test-time domain adaptation. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp- 7201-7211, 2022.
Yisen Wang, Xingjun Ma, Zaiyi Chen, Yuan Luo, Jinfeng Yi, and James Bailey, Symmetric cross entropy for robust learning with noisy labels. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 322-330, 2019b. httpe///ggthaab..omm//gggg......
Ross Wightman. Pytorch image models.
pytorch- imagee models 2019.
Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, et al. Robust fine-tuning of zero-shot models. In Proceedings of the IEEE/CVF conference Computer Vision and Puttern Recognition, pp. 7959-7971, 2022.
Longhui Yuan, Binhui Xie, and Shuang Li. Robust test-time adaptation in dynamic scenarios. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 15922-15933 15932, 2023.
Jinnyu Zhang and Léon Bottou. Learning useful representations for shifting tasks and distributions. In International Conference on Machine Learning, pp. 40830-40850, PMLR, 2023.
Jianyu Zhang and Léon Bottou. Fine-tuning with very large dropout.
arXiv preprint
29X4v:2403.00946 2024.
Jianyu Zhang, David Lopez-Pas Paz, and Léon Bottou. Rich feature construction for the optimization- generalization dilemma. In International Conference on Machine Learning, pp. 26397-2641 1. PMLR, 2022a.
Marvin Zhang, Sergey Levine, and Chelsea Finn. Memo: Test time robustness via adaptation and augmentation. Advances in Neural Information Processing Systems, 11-38229-88644 2022b.
Bowen Zhao, Chen Chen, and Shu-Tao Xia. Delta: degradation-fire fully test-time adaptation. arXiv preprint arXiv:2301. 13018, 2023.
13

---

Appendix
Contents
Related work
15
B Details of the preliminary study
16
B. Theoretical analysis of linear mode connectivity in TTA
16
B.2
Preliminary study for linear mode connectivity
16
B.3 Expanding various TTA scenarios
16
Details of implementation
18
Baseline models
18
Algorithms
19
Computational complexity
20
Further analysis
21
D.I
Failure case analysis
D.2
Mathematical analysis of reverse Kl. divergence
21
D.3
Further experiments for Tent+TTT. and SAR+ITH
22
D.4
Further experiments for catastrophic forgetting
22
D.5
Further experiments for weight averaging strategy
23
D.6
Further experiments for long-range true-biased scenarios
23
Main results with standard deviation
25
Limitations and future works
26
14

---

Published as a conference paper at ICLR 2025
A RELATED WORK
Test-time adaptation (TTA), using pretrained models, has become essential for adjusting to distri- bution shifts in test data (Wang al. 2021). Many existing methods have focused on optimizing
stability to prevent model collapse amid dynamic distribution shifts, by selecting reliable samples
(Niu al. 2023; 2022 Yuan et al. 2023), ensuring consistency in predictions (Wang et al.
2022
Zhang et al. [202bb] Chen et al. (0022) manipulating affine transform statistics (Gong et al.
2022
Lim et a 2023; Zhao et al., (023) and using a robust optimizer (Niu et al., [023] Gong et al,
2024). However, enhancing model representation for adaptation capability remains less explored to
date. Similar to the proposed TTE, several researchers have adopted student-teacher networks and
employed exponential moving average strategy to update the teacher network (Wang al.) 2022
Yuan et al. 2023 Döbler et al. 2023 Chen et al., 2022). J. Unlike TTE, however, they typically have introduced weight-averaged network ensure optimization stability and minimize abrupt model changes by using a high momentum value for the average, close to 0.999. To achieve relevant rep- resentations, these methods have often utilized dense image augmentation relied on source data. Unfortunately, such strategies resulted in increased computational complexity.
Test-time training, similar to TTA, has been designed actively adjust pretrained model dis- tribution shifts using test data. Unlike TTA, which domain adaptation processes are independent of the training phase, test-time training starts by developing a training process that can be extended into a test phase for adaptation (Sun et al.] (2020). Most strategies have implemented a combined optimization of supervised and self-supervised learning during a training phase, subsequently per- forming the self-supervised learning with test data (Sun et al. 2020; Liu et al.) [021]] Gandelsman jet al., 2022). However, dependence on specific training procedures and architectures, designed for self-supervised objectives, could restrict the methods' applicability to various off-the-shelf models. Additionally, these methods sometimes demanded substantial computational resources due to the intensive augmentations used in self-supervised learning (Gandelsman et al. 2022)..
Fine-tuning under linear approximation has garnered significant attention as an efficient strategy for adapting complex pretrained models to downstream tasks. Research has shown that the later stages of training deep neural networks often stabilize within nearly-convex regions, suggesting that the landscape of the models' cost functions is more tractable (Izmailov et al., 2018) Frankle al., 2020 Fort et al. 2020). This observation persists even when fine-tuning large networks with datasets much smaller than those required to train a model from scratch. Further advancing this concept, fine-tuning processes have been approximated by a first-order Taylor expansion, transforming the process into linear system based on Neural Tangent Kernel (NTK) features (Maddox et al. [022]] Jacot et al., 2018). More recently. notable approaches have proposed averaging the weights of various fine-tuned models, effectively replicating the performance of output ensembles across both in-distribution and out-of-distribution datasets, and underscoring the near-linear characteristics of the fine-tuning process (Wortsman et al. 2022; Rame et al., 2022). Inspired by these intriguing findings, we aim to develop a TTA method based on the premise that test-time tuning of pretrained models can also exhibit near-linear characteristics.
Constructing versatile representations is crucial for improving generalization performance, espe- cially in distribution shifts. Researchers have increased the versatility of representations by diversi- fying architectures, datasets, and dyper-paramette (Dvornik al., 2020 Chowdhury et al.) 2021 Ganaie et al., 2022), rather than merely enlarging dataset sizes. Some studies have collected features from different models (Li al. 2003), while others have integrated diverse representations through weight averaging (Wortsman et al. 2022 Rame et al.) 2022). Notably, recent research has shown that redundant representations, although not beneficial for in-distribution performance, can signifi- cantly enhance ofstiiibbttttoionn generalization (Zhang & Bottou 2023). Building on this concept, the proposed components in TTE aim to cultivate versatile representations during test time, thereby enhancing performance against unexpected distribution shifts.
15

---

Published as a conference paper at ICLR 2025
B DETAILS OF THE PRELIMINARY STUDY
B.1 THEORETICAL ANALYSIS OF LINEAR MODE CONNECTIVITY IN TTA
Our preliminary experiments suggest that weight-space interpolation can mimic the effects observed in output-space interpolation. It is crucial, however, to understand the conditions under which weight-space interpolation can theoretically approximate output-space interpolation. Previous re- search (Mu al., 2020; |Mddoxx al. 2021 has proposed approximating the fine-tuning process using a first-order Taylor expansion, resu resulting in a linear system that operates on Neural Tangent Ker- nel (NTK) features (Jacot et al. 2018 Fort al. 2020). We extend this hypothesis to TTA. a specific form of fine-tuning, positing that it can similarly be approximated by linear expansion. Under this framework, weight-space interpolation approximates output- space interpolation correctly.
Proposition. If for any 0 within θ {(1 {11-a)0 + a02 [0. 1]), the function f(0) can be approximated linearly around 0g follows: f(0) f(Ou) + V (0,) Bg), then weight-space interpolation between 9 and 8g is equivalent to performing output-space interpolation. Here, 6 denotes the parameter of pretrained model.
Proof.
We initiate with oupput-space ensemble and retrieve weight-space ensemble
(1 - a) + af
(1-a) + a.V 0u) af(()) (VV 0o)
=f(0)) + V a) -0) + a(02 Bo))
(6)
=f(0)) + v.(()) ((1-a))) a02 0o))
f((11 a0) + a02)
B.2 PRELIMINARY STUDY LINEAR MODE CONNECTIVITY
We conducted a preliminary study to empirically investigate the linear connectivity between two TTA models and to suggest strong cues for the benefits weight-space interpolation for TTA. All experiments in the preliminary study were performed with ImageNet-C (severity level 5) by using Tent (Wang et al., 202I). The sample selection scheme suggested in Niu et al. (2023) was integrated with Tent prevent model collapse and clearly analyze the linear property in TTA. The details of experimental procedures are as follows:
1. We performed TTA for four corruptions individually, which were Gaussian noise, defocus blur, snow, and contrast.
2. Two TTA models were interpolated with a mixing coefficient a in both output and weight spaces.
3. Classification accuracy measured over all 15 corruptions in ImageNet-C (level 5), by using the predictions from the two types of interpolation.
4. We repeated the second and the third steps by changing a in [0, 1].
Accuracy values averaged across two target corruptions were used values in Figure I to show adaptation performance. Accuracy over all other corruptions were used as y values show gener- alization performance.
B.3 EXPANDING TO VARIOUS TTA SCENARIOS
Inspired by the results in Appendix B.2 we conducted an additional preliminary study to verify the effects of linear mode connectivity across various TTA scenarios. In this study, four corruptions (Gaussian noise, defocus blur, snow, and contrast) were selected target distributions, while the remaining corrppiins were considered as non-target distributions. We assessed weight-space inter- polation through two scenarios: single-instance TTA, where each model adapts one corruption,
16

---

Published as a conference paper at ICLR 2025
and continual TTA, where model sequentially updates across four target distributions, reflect- ing practical settings. The classification accuracy for both target and non-target corruptions was measured after TTA processes.
Single-instance TTA. We constructed a ensemble by averaging the parameters of four models with 0.25 mixing coefficients. Table compares the performance of each TTA model and their ensem- ble. Interestingly, the ensemble model achieved the best average performance on both target and non-target corruptions, suggesting that merely averaging model parameters can effectively capture diverse representations from multiple models.
Continual TTA. Model parameters yielding three intermediate models plus final model. Table 8 compares the performance of the final model against the ensemble of the four models with 0.25 coefficients. The ensemble models generally outperformed the continual models, demonstrating that linear mode connectivity remains valid in continual setups. Notably, even with the sample selection scheme in the baseline TTA mod- els, the continual model collapsed in the sequence of D S-> C-> G, providing near-zero accuracy, while the ensemble model mitigated the performance degradation.
saved before transitioning subsequent corruptions,
Table Classification accuracy (%) under single-instance TTA. Note that WSE stands for weight- space ensemble. G,D.S and C represent Gaussian noise, defocus blur, snow and contrast respec- tively. Bold and underlined numbers are the best and the second best results.
Tirget comuptions
Non-target cortuptions
Gaun
Detec
Imped
Glass
Motion
Blanic
Pidd
JPRG
Method
No adapt
9.4
29.1
15.8
43.9
24.6
6.7
8.3
23.4
34,0
27.0
26.3
47.4
51.7
30.5
44.5
47.6
31.8
39.5
TTT (D)
12.9
57.2
38.5
52.5
40.3
11.8
12.1
36.2
53.4
40.4
42.1
60.0
71.7
34.4
57.3
57.4
43.2
TIA (S)
21.0
35.5
60.9
37.4
38.7
16.3
20.8
31.4
43.3
32.0
58.2
52.6
74.7
40.8
55.7
57.8
44.0
29.0
61.4
WSB (G+D+S+C)
90.7
43.7
49.5
56.0
45.5
26.2
31.6
34.5
49.5
38.2
51.1
59.3
73.7
38.7
60.6
60.5
47.6
Table 8: Classification accuracy (%) under continual TTA with four corruption orders. Notations are identical to Table
Target conruptions
Now target coctuptions
Method
Cauns
Defic
Saree
Contr
Avg
Shor
Sonpul
Glass
Moilou
Zonn
Brit
Blatte
Pred
JPBG
Arg
No adopt
9.4
29.1
15.8
43.9
24.6
6.7
34.0
27.0
26.3
47.4
54.7
30.5
44.5
47.6
31.8
TTA (G S.S C)
38.7
$2.5
40.2
66.1
49.4
36.9
39.9
35.0
50.5
33.1
51.0
60.1
75.2
38.3
62.4
63.5
49.6
(G+D48+))
62.9
TTA (C- G-a D-+
40.4
42.5
$1.9
38.1
40.8
27.4
44.9
27.9
$1.7
74.2
35.2
62.4
WSP (C+G+D&S)
43.6
51.3
49.9
64.1
52.2
39.8
44.3
35.3
51.4
38.3
52.5
61.8
74.5
38.5
62.5
63.3
51.1
TTA (S+C+G+D)
44,0
55,8
56.3
62.2
54.6
44.5
38.1
53.7
41.4
52.8
61.3
75.8
41.5
61.4
63.7
52.3
56.9
75.6
TTA (D- S-+ C-+ G)
0,2
0.2
0.1
0.2
0.2
0.]
0.2
0.1
0.1
0.1
9.9
0.6
2.3
3.7
1.6
WSE (D+S+C++)
2.4
16.9
3.5
6.8
7.4
2.8
1.9
4.9
11.5
4,9
18.5
13.1
62.3
11.6
44,8
44.1
20.0
17

---

Published as a conference paper at ICLR 2025
C DETAILS OF IMPLEMENTATION
C.I BASELINE MODELS
We utilized pre-trained ViT-base and ResNet50-GN obtained from the publicly available PyTorch Image Models repository Wightman (2019). The public models used were trained on ImageNet- Ik for image recognition tasks. The implementations of the comparative methods were obtained from their public repositories and followed the guidelines outlined in their original papers. Additionally, when integrating TTE with existing TTA approaches, we followed the original implementations and hyperparameter settings specified in the respective papers to ensure accurate evaluation of TTE's effects. Further details are provided below.
Tent WWan et al. 2021 Tent used stochastic gradient descent (SGD) with momentum of 0.9 as the optimizer. The learning rate was set to 0.00025 for ResNe150-GN and 0.001 for ViTBase. For batch size of 1, the learning rates were adjusted to 0.00025/32 for ResNet50-GN and 0.001/64 for ViTBase. The trainable parameters included all affine parameters in the normalization layers.
COTTA /Wang al. 2022 used the Adam optimizer with learning rate of 0.0025 for ResNet and 0.001 for ViTBase. The method employed model-specific hyperparametens and reported their search for optimal values for ViTBase. The restoration factor was explored within the range [0.01,0.9], and the EMA smoothing factor a was searched within [0.1,0.001]. Based on performance, we selected p 0.7 and a 0.001 for ViTBase, while for ResNet, we followed the author's 's recommendation with p = 0.01 and = 0.001. Additionally, the augmentation confidence threshold Pth was set to 0.1. All weights in the architectures were trainable.
values for ResNet, prompting
SAR (Niu et al. 2023 uued SGD with a momentum of 0.9 as the optimizer. The learning rate was set to 0.00025 for ResNet50-GN and 0.001 for ViTBase. For a batch size of 1, the learning rates were adjusted to 0.00025/16 for ResNet50-GN and 0.001/32 for ViTBase. We set the sharpness threshold to م 0.05 and the entropy threshold to Eo 0.4 Im(1000). The learnable parameters included affine parameters in the normalization layers, while the top layers were frozen: layer 4 in ResNet and blocks 9-11 in ViTBase.
DeYO (Lee et al. 2024 used SGD with a momentum of 0.9 as the optimizer. The learning rate was set to 0.00025 for ResNet50 and 0.001 for ViTBase. For a batch size of the learning rates were adjusted to 0. 0.00025/16 for ResNet50-GN and 0.001/32 for ViTBase. The required hyperpa- rameters for YO are the entropy threshold Tem. the probability difference threshold TPLPD, and the normalizing factor Ento. We set TEnt = 0.5 x In (01000). TPLPD = 0.3, and Ent, 0.4 x In (1000).. The learnable parameters included the affine parameters in the normalization layers, while the top layers were frozen: layer 4 in ResNet and blocks 9-11 in ViTBase.
TTE (Ours) employed SGD with momentum of 0.9 as the optimizer. The learning rate set to 0.00025 for ResNet50-GN GN and 0.001 for ViTBase. For a batch size of 1, the learning rates were adjusted to 0.00025/ 16 for ResNet50-GN and 0.001/32 for ViTBase. The temperature T was set to 1.0 with dropout ratio of 0.9 for ResNet, and r = 10.0 with dropout ratio of 0.4 for ViTBase. For the de-biasing scheme, a 3.0 and n = 0.99 were used. Note that a linear ramp-up was applied to gradually increase from 0.0 to mg over initial 100 iterations for batch size of 64, and initial 6400 iterations for a batch size of I, where ensemble effects were insignificant. After the ramp-up phase, the active momentum scheme was applied. The trainable parameters in TTE were identical to those of the bascline model with which TTE was integrated.
00001
01
res 01-res 224
v0.1
httes:/ggithbb..com///y/////////
18

---

Published as a conference paper at ICLR 2025
C.2 ALGRITHTM
TTE was integrated into three existing methods to verify its effectiveness and broad applicability: Tent (Wang al. 2021), SAR (Niu et al. 2023), and DeYO (Lee et al. 2024). TTE was seamlessly incorporated by leveraging the optimization procedures and sample selection/ selection/veightinng mechanisms originally proposed in these methods. The algorithms for Tent, SAR, and De are detailed in Al- gorithm I 2 and respectively, with the parts introduced by TTE highlighted in blue. Importantly, TTE required few implementation lines, adding minimal computational burden.
Algorithm 1: TTE + Tent (Wang et al. 2021
Input: Test samples Dest = {ms) f adapter model (1 0a. p) with trainable parameters 0g C 0g and dropout ratio ensemble model fe(:;0e) with de aligned 8, bias vector Chiass step size momentum dehiasing parameters n, a.
Output: Predictions L.M {
Initialize 0g + Agg. 6g +- 0g. Obina 0:
for mn E Dien
Predict Do.i = fa (ar;; (). De.t = [e(;;; 0e) and compute entropy F. Fiia, 0a); Compute weight w(ge,1) and de-biased representation Ve
Compute KT Do.i 3 (aa) and total objective Lite:
CualBo,is Ve 0a) ) E(ia, + KL(B,,,1lBe.tt
Compute gradient g Vo, Lite (Bu, Ve,
Update 0, 0a n9:
Compute momentum m from Ey and update ensemble parameterss: Oe +- m -e m) Oa:
Compute first-order batch statistics MDo. - Σi Pa.45
Update bias vector: Chena + n Obiax + n) M0a.s
end
Algorithm 2: TTE SAR (Niu et al. 2023)
Input: Test samples Diest = } model fa (-; 0a. p) with trainable parameters 0, C 8a and a dropout ratio ensemble model (=(:).) with 6e aligned
bias vector Cbias step size 1 neighborhood size >> Teu > 0 in Eq. (2), ég > for model recovery. momentum m, debiasing parameters n, a.
Output: Predictions (
Initialize 8,, 86. 6g + moving average of entropy em Cbias
for æ;; E 'Den
Predict Do,i = fa (ar;; P). De,i fe(m;;; 62) and compute entropy E; E(io, 0u);
if E; then
continue:
Compute weight x) ) and de-hiased representation V Compute KL 19 0o) and total ubjective Cuee:
Lueljo, de)): E(ia, 0.) KI..a..llB6.5....
Compute gradient Va, Lue Ve,
10
Compute é()) Eq. (4);
Compute gradient approximation: g a.. Lite (Ma.. Wa
Update 0a 0g n0:
13
Compute momentum m from Fq 2 and update ensemble parameters: 0g +- m -e m)) 0::
14
Compute first-order hatch statistics Mia.. = ΣN Du.si
15
Update vector: China n Cbias (1 72) Mg..i
16
Om 0.9 cm 0.1 Elga,ci ((Oa)) if em 0 else Elw;; 0a {(0a));
17
if Cm < cp
18
Recover model weights: 6, + 6o. 0, + 0p
19
end
20
19

---

Published as a conference paper at ICLR 2025
Algorithm 3: TTE Ye (Lee et al. 2024)
Input: Test samples Dies = (z:) model (+; la. p) with trainable parameters C 0 and a dropout ratio ensemble model fe(:;0, 0.) with 0, aligned la. bias vector Cbias- object-destructive transformation A. step size 1) > 0, and hyperparameters Ento, Teat. 7HPP > O, momentum m, debiasing parameters nt, a.
Output: Predictions {9:: )
Initialize dn < 06. 0. 0o. Obias for æ; E PPvu
Predict Da,i = fa P), Dest fe(we;;)) 0e) and compute entropy E; E(gn,i;;
if E2 then
continue;
Obtain 2 = A(a;);
Compute pscudo-label probability difference (LPDy(x;; if PLPDD(aa;; æ!} < TPLPD then
continue:
11
12
Compute weight ( and de-biased representation Ve..
13
Compute L 12, 0a) and total ubjective Cite:
Lue Ho F. 0a) + KI KI,,,,,lllb.s5
14
Compute sample weight oxg,, (ug,( (at;):
Compute the overall loss LDDYO 0ug (m:) Litei Compute gradient g = Va. LDeYo:
16
17
Update O. Compute momentum from By and update ensemble parameters:: 0e m.me -c + (1 m) 0a:
n9:
18
19
Compute first-order batch statistics Mia., >> Du.ii
20
Update bias vector: Cbsaa 4- n Obias + n2) Mga.si
C.3 COMPUTATIONAL COMPLEXITY
Integrating TTE required only additional feedforward pass for fe, resulting in minimal computa- tional overhead. Table 9 details the computation and runtime required for adaptation with ViTBase under Gaussian noise (lmageNet-C,, level 5). ViTBase chosen for its superior performance ResNet. Integrating TTE introduced an additional network, Se, requiring only single feedforward pass compute De- adding minimal computational overhead. For example, single feedforward pass without adaptation took 3 minutes 57 seconds. That was why TTE integration resulted in about 4-minute increase in GPU time. contrast to CoTTA, which relied dense augmentations and multiple feedforward passes for knowledge distillation, TTE maintained efficiency by requiring only single pass.
Table Computational complexity and runtime for adaptation using ViTBase under Gaussian noise (ImageNet-C, level 5) with Label Shifts setup. The total sample size is 100,000. Bold numbers present accuracy gain by applying TTE.
Methods
#Model
#Forward
#Backward
Other Computation
GPU time (100k images)
Accuracy (%)
No adapt
100k
n/a n/a
$7 sec (x1..0 1.0)
Tent
100k
100k
8 min 12 sec (x2.1)
30.9
SAR
100k+75k
75k+72k
weight perturbation
min 46 see (x3.7)
46.6
D-YO
100k++2kk
75k
probability difference
11 min 49 (x3.0)
49.1
COTTA
2
200k+4499k
100k
anchor probability
81 min 14 (x20.5)
34.1
Ten+TTE
200k
100k
min 30 (x3.2)
46.6(+15.7)
SAR+TTE
2
200k+72k
72k+72k
Lyki Cbias
20 min 30 see (x5.2)
48. (8.6()2.0)
D-YO-TTE
200k+89k
72k
15 min 47 (x4.0)
54.0(+4.9)
20

---

Published as a conference paper at ICLR 2025
D FURTHER ANALYSIS
D.1 FAILURE CASE ANALYSIS
In this section, we analyzed failure cases when applying ensemble strategies to TTA alone. Two en- semble strategies were applied representative TTA method, Tent (Wang al. 202I): construct- ing an ensemble network Je through weight-space ensemble of the original TTA model Sa and using knowledge distillation with standard KI. divergence between outputs. Additionally, dropout was applied to the penultimate layer of fo to further enhance ensemble representations. The hyper- parameters for ensemble strategies were set as mg = 1.0, T = 1.0 and a dropout ratio 0.6.
We evaluated classification accuracy under the challenging Label Shift scenario ViTBase and ImageNet-C (level 5). To gain deeper insights, the loss barrier between fa and fe was measured as typical metric for assessing linear mode connectivity, following / Fort et al. (2020). The loss barrier was calculated along a linear interpolation path between the models in weight space, as described by
(R(a)) 1 (A(0.) (R(0.))+A((e)) + aE(U1]]
max
2
where P((0,) N En (1,a(;;; 0.).1) and a is interpolation coefficient in [0, 1]. Here, L is classification loss with input x and label y.
Table 10 compares TTA with and without the two ensemble strategies. Figure 3 visualizes the average accuracy profile and prediction distributions for two representative datasets. The results show that TTA is unstable in the challenging scenario and can sometimes perform worse than no adaptation, due to biased class predictions (we call it model collapse). Applying ensemble strategies could occasionally worsen performance by constructing ensembles with unreliable fa. Additional methods are needed to prevent the model from biasing.
Table 10: Classification accuracy (%) of ViTBase measured under Label Shifts (ImageNet-C, level 5). Red number indicates lower performance than without adaptation, signifying model collapse. Bold number highlights the best performance.
Gouss
Impul
Fog
Avg
Araatt
9.4
23.4
27.0
43.9
47.6
TTA
20.1
1.2
19.0
54.9
53.4
54.1
14.3
13.6
69.8
76.3
66.4
39.8
69.8
66.9
46.6
TTA Ensemble
47.0
1.4
32.7
58.5
348
59.5
55.4
14.9
17.6
70.8
76.9
66.7
62.0
70.4
67.5
50.2
D.2 MATHRMATICAL ANALYSTS OF REVERSE KI. DIVERGENCE
In this section, we mathematically analyzed the robustness of reverse KL divergence in the context of TTA. Since KL divergence is asymmetric, swapping its inputs resulted different effect. To understand this difference, we computed the gradients of both standard and reverse KI divergence when integrated with a typical TTA objective. Specifically, we used Tent (Wang et al.) (022]. which minimized Shannon entropy via Go log Ua). The objective Lite could be described in two ways as
( ( 9.)
for Standard for Reverse
Lite ja. V6)
(8)
( + KL 7..
where P(O.) log p().)) and K (T(p(()))
(()) = -. p(O.) (/pp((p)))))))()))))))))))))))))
Here, p(g.) denotes the probability of class e with the softmax function p. To simplify expressions, p(Dā) and p(i(") are denoted as of and d The gradient magnitude of the objective function is calculated for the weight 0g as
0Lus ( No. H.) 00,
- =
E.
-0--(11 (1+ log 9f + q.h /////.) for Standard log de
(9)
Σ.
for Reverse
21

---

Published as a conference paper at ICLR 2025
In standard KL divergence, the term q0 produces larger gradients for samples where de is high but qa is low. This term results in those misaligned samples being implicitly weighted more dur- ing gradient updates, compared to samples where 9h and %a align. While this weighting can be beneficial when de is reliable, it may lead to overfitting to noise if o. is noisy. contrast, re- verse KI. divergence lacks the 9 /1f term, treating all samples only based ensemble prediction (-En log (²"). Figure 4 supports the analysis by comparing classification accuracy and gradient ratio during TTA process. Reverse KL divergence reduces the gradient weight incorrect predictions while increasing it for correct ones.
D.3 FURTHER EXPERIMENTS FOR TENT+TTE AND SAR+TTE
To further validate the robustness of TTE, we revisited the experiments in Tables and 4 incor- porating other baselines integrated with TTE (e.g., Tent+ TTE and SAR+TTE). Tables 11 and T2 present the additional results alongside some previous findings, demonstrating consistency with the DeYO+TTE results and reaffirming the effectiveness of TTE.
Table 11: Continual TTA with non i.i.d. conditions. Classification accuracy (%) with ImageNet-C
(level 5). Underline depicts performance improvement when applying TTE. Bold numbers the best results.
Adoptation Order
+
Ganss
Shot
Impul
Defoe
Glass
Motion
Zoom
Snow
Frost
Fog
Rrit
Contr
Flastic
Pixel
JPEG
Avg
ResNet50-GG
18.0
19.8
17.9
19.8
11.4
21.4
24.9
40.4
47.3
33.6
69.3
36.3
18.6
28.4
52.3
30.6
Tent
3.9
1.8
1.6
11.1
0.1
0.1
0.1
0.1
0.1
0.1
0.1
0.1
0.1
0.1
0.1
0.6
Tent TTE
16.2
27.9
46.8
48.1
42.1
30.4
49.4
45.2
36.8
54.7
50.3
52.8
39.4
67.0
55.9
44.2
SAR
33.1
16.8
41.7
11.6
42.3
18.8
15.7
37.8
39.7
9.3
3.1
2.1
0.7
T22
23.0
SAR + TTE
35.9
28.3
46.9
18.9
43.1
30.7
50.1
45.6
38.2
35.5
51.5
54.4
39.7
67.8
57.0
14.9
DeYO
28.1
3.7
7.2
033
0.1
0.1
0.1
0.1
0.1
0.1
0.1
0.1
0.1
0.1
2.8
DeYO TTE
43.0
31.7
52.7
52.3
45.1
36.4
52.2
$3.6
40.9
43.0
59.1
60.3
47.4
68.9
49.7
ViTBase
30.5
54.7
26.3
0.1
0.1
59.9
49.5
51.3
64.5
68.4
SAR
15.8
SAR
62.8
51.5
57.7
69.8
75.3
61.5
69.2
TTE
55.2
64.6
62.5
54.2
52.0
68.5
69.9
Table 12: TTA with natural distribution shifts. Classification accuracy (%) with ImageNet-Sketch (S), ImageNet-Rendien Renditennn (R), ImageNet-V2 (V2). Underline depicts performance improvement when applying TTE. Bold numbers are the best results.
(a) Label Shifts
(b) Batch Size
Methods
S
Avg
Methods
S
Avg
ResNet50 GN
29.2
40.8
68.9
46.3
ResNet50. GN
29.2
40.8
68.9
46.3
Tant
30.8
41.5
68.9
47.1
Tont
31.9
42.1
68.9
47.6
Teat+TTE
33.4
42.1
68.9
48.1
Tent+TTE
36.2
44.1
68.9
49.7
SAR
30.6
41.6
47.0
SAR
31.6
41.9
47.3
SAR+TTE
32.5
41.9
47.8
SAR+TTE
31.7
41.8
47.5
DeYO
34.6
34.4
68.9
49.3
DeYO
37.0
46.0
69.0
50.7
DeYO-TTE
36,9
45.1
68.9
50.3
DeYO-TTE
39.1
47.1
69.0
51.7
ViTBase
18.2
43.1
66.2
37.9
ViTBase
18.2
43.1
66.2
37.9
8.8
41.9
68.9
39,8
Tent
7.0
40.6
69.1
38.9
TeR+-TE
36.9
52.1
69.1
$2.7
TeR++TTE
38.5
53.7
69.3
$3.8
SAR
17.8
43.1
68.3
43.8
SAR
26.4
44.7
69.4
46.8
SAR+TTE
37.0
51.9
68.8
52.6
SAR+TTE
33.7
50.4
68.9
$1.0
42.4
$7.3
43.6
58.2
DeYO-TTE
43.6
59.1
71.1
$7.9
DeYO+TTE
45.0
61.3
71.3
59.2
D.4 FURTHER EXPERIMENTS FOR CATASTROPHIC FORGETTING
The proposed TTE effectively prevents model collapse and demonstrates stable optimization across extensive TTA experiments. To further evaluate its stability, we conducted additional experiments to determine whether TTE can address catastrophic forgetting phenomenon where model exhibits severe performance degradation on source domain dataset after adaptation. To test this, we concur- rently measured the accuracy the clean ImageNet dataset immediately after each adaptation to a distribution in Table II Figure [ shows that integrating TTE successfully mitigates forgetting issues, whereas other baseline methods suffer from them.
22

---

Published as a conference paper at ICLR 2025
(a) ResNet50-GN
(b) ViTBase
Auerage accuracy
Usseline: B0.0
Dase linellne: 78.0
lenti IE
lenii+ HIE
Average
SAR+TTE
SAR-TTE
DevO TTE
DeYO-TTE
10
12
10
Adaptation order
Adaptation order (-)
Figure 8: Comparison preventing catastrophic forgetting on Continual TTA with non-i.id. con- ditions. Classification accuracy in-distribution data (ImageNet). Each point was measured after TTA on each distibibbtii data in Table
D.5 FURTHER EXPERIMENTS FOR WEIGHT AVERAGING STRATEGY
The proposed ensemble strategy is inspired by weight-averaging methods originally developed for offline domain generalization. In this section, we compare the proposed adaptive weight-averaging method with the stochastic rright-averaging (SWA) method (Izmailov al. 2018). For this com- parison, we create a variant of the TTE method by replacing the proposed ensemble strategy with SWA while keeping all other components identical. SWA focuses on the uniform averaging of TTA models generated during the TTA process and is implemented as follows:
Uswn
Renodels + w
Ugwa
(10)
Ronddls +
where moddels represents the number of ensemble models, Wswa and denote the parameters of the TTA model Ja and the ensemble model fe. respectively. The value of Temodels is determined by the frequency f of model selection during TTA. We extensively optimized f within the range [1, 20] and selected for ResNet50-GN and 5 for ViTBase to achieve maximum performance. Table [3 presents the comparison results on ImageNet-C under the Label Shifts setup. The results indicate that the proposed adaptive scheme achieves higher adaptation performance compared the SWA approach.
Table 13: Comparison study between different weight averaging schemes. Classification accuracy (%) with the Label Shifts setup (ImageNet-CC level 5). Note that swa stands for stochastic weight averaging. Bold numbers are the best results.
Label Shifts
Guuss
Shot
Impul
Defoc
Glass
Motion
Zoom
Snmw
Fros
Fog
Brit
Cuntr
Elastic
Pixel
JPEG
Avg
ResNe150-GN
18.0
19.8
17.9
19.8
11.4
21.1
21.9
40.4
47.3
33.6
69.3
36.3
18.6
28.4
52.3
30.5
DeYD
28.1
44.3
42.9
23.4
16.6
41.5
6.1
$2.9
$2.0
20.2
73.2
$3.0
37.7
60.0
59.4
40.8
-TTE (swa)
39.8
42.9
41.6
30.2
28.7
39.8
10.4
52.9
50.9
5S.4
73.3
51.6
45.1
$7.5
58.5
47.4
+ -TE (ours)
43.0
45.3
44.0
34.4
33.6
43.4
46.0
55.2
53.6
61.0
73.4
54.4
51.5
61.0
60.2
50.7
ViTBase
9.4
6.7
8.3
29.1
23.4
34.0
27.0
15.8
26.3
47.4
54.7
43.9
30.5
40.5
47.6
29.9
DeYO
49.1
35.9
$3.6
17.6
58.6
63.8
37.5
67.9
66.0
73.1
77.9
66.5
68.6
73.5
70.1
61.3
-TE (swa)
53.5
53.6
14.4
58.7
19.4
63.7
60.7
67.9
66.3
73.9
78.3
68.4
68.7
73.6
70.2
64.7
TTE (our))
$4.0
54.7
55.2
58.9
59.7
64.5
62.1
68.3
66.7
73.9
78.0
68.3
69.3
73.8
70.3
65.2
D.6 FURTHER EXPERIMENTS FOR LONG-RANGE TRUE-BIASED SCENARIOS
The proposed de-biasing scheme is designed to prevent the model from biasing towards specific classes, a key issue when models collapse. However, cases where true label stream is heavily biased (i.e.. model encounter the same class samples consecutively for over 100 iterations), assessed whether the de-biasing scheme could still perform effectively. To simulate extreme sce- narios, we combined a batch size of with the Label Shifts setup, where class-imbalanced streams persisted over 100 iterations.
Figure shows the average accuracy across all distributions in the Label Shifts setup for ViTBase. Unfortunately, the default de- de-biasing configuration (a = 3.0, n 0.99) penalized true-biased predictions, providing lower accuracy than the baseline DeYO. By reducing the de- de-biasing strength
23

---

Published as a conference paper at ICLR 2025
(a 1.0) and more gradually estimating model bias (n 0.999), TTE stabilized and achieved higher accuracy than DeYO. However, over-tuning hyperparameter based on test data performance could be controversial. Future work should explore additional methods to distinguish between true bias and false bias, which can lead to model collapse.
350
00)
Deto: 01.3
Figure 9: Average classification accuracy (%) in varing with hyperparameter of de-biasing schemes in Lable Shifts for ViT'ase (ImageNet-C. level 5). The extremely biased scenario is used to assume that same class samples are fed into the model consecutively over 100 times.
24

---

Published as a conference paper at ICLR 2025
E MAIN RESULTS WITH STANDARD DEVIATION
In this paper, experiments were conducted with three different random seeds: 2022, 2023, and 2024. The results presented in Tables I and include the mean values across these three repetitions. The corresponding standard deviations are provided in Tables 14
15 and 16 respectively.
Table 14: Integration with previous TTA approaches. Classification accuracy (%) with Label Shifts and Batch Size setups (ImageNet-C, level 5). Underline depicts performance improvement when applying TTE. Bold numbers are the best results.
Label Shifts
Gauss
Shot
Impul
Defoc
Glass
Motion
Zoom
Snow
Frost
Fog
Brit
Contr
Elastic
Pisel
JPEG
Avg
ResNet50-Gh GN
18.0
19.8
17.9
19.8
11.4
21.4
24.9
40.4
47.3
33.6
69.3
36.3
18.6
28.4
52.3
30.5
Temi
-627
3.9
4.6
4.7
16.4
22.16
6.0
27.2
=00
29.0
18.9
27.4
2.4
72.1
46.1
8,1
2003
52.4
56.1
2044
25.0
29.6
3005
2106
20.0
2010
2100
2105
SAR
33.1
36.5
35.2
2931
18.9
$1.19
20.8
33.3
29.8
27.8
44.9
35.2
71.9
46.6
7.6
20.11
52.1
56.2
=104
36.7
SAR
-02
38.4
2013
29-7
204
=001
5005
53.33
20.00
52.8
2300
=000
DeYo
28.1
44.5
42.
23.4
16.6
41.5
6.1
52.9
52.0
20.2
73.2
53.0
37.7
60.0
59.4
40.8
DeYo + TTE
43.0
2611
34.4
1311
33.6
1771
43.4
1773
11973
53.6
2711
13.1
73.4
1311
10115
10T
1100)
61.0
60.2
17005
50.7
100
VITRase
94
6.7
8.3
29
23.4
34.0
27.0
15.8
26.3
47.4
54.7
43.9
30.5
44.5
47.6
29.9
Tent
30.9
1.0
23.2
54.9
53.3
58.8
54.3
13.3
12.5
69
76.3
66.3
59.7
69.8
66.8%
47.
Tent TTE
46.6
2039
2009
2307
13.0
1:1 16
54.4
59.0
2.00
2014
2:8
19.15
76.4
am 16
66.3
36.33
3.036
2.005
67.0
6000
2.005
SAR
46.6
29.5
48.1
55.2
54.2
59.0
54.6
58.0
44.1
69 &
76.3
66.
60.0
69.7
66.6
57.5
SAR TTE
38.6
E130
16.4
E178
55.8
55.2
59.4
55.8
62.4
70.1
311
76.4
66.1
30122
62.4
31m
210m
70.0
62.0
2000
60.5
DeYo
29.1
35.9
53.6
57.6
53.6
63.8
37.5
67.9
66.0
73.1
77.9
06.5
68.6
73.5
70.
61.3
DeYo e TIE
5037
54,0
21100
$4.7
201A
58.2
58.9
3012
59.7
30075
64.5
3003
62.1
31113
68.3
2106
66.7
73.9
3912
3073L
75.0
39136
68.3
69.3
30075
73.8
2003
70.3
65.2
20111
BSI
Gauss
Shot
Impul
Defoc
Glass
Moxion
Zoom
Snow
Frost
Fag
Brit
Coazz
Elastic
Pixel
JPEG
Avg
ResNerbi-GiN
18.0
19.8
17.9
19.8
11.4
21.4
24.9
40.4
47.5
33.6
69.3
36.3
18.6
28.4
52.3
30.6
3.1
4.1
3.7
16.6
5.2.
27.2
20.0
17.7
25.1
1.9
73.0
46.2
8.1
52.7
56.3
24.6
Tent TTF
21.6
10h
43.9
10T
11m
18mh
T0TR
41.0
1718
TTT7)
59.1
73.1
12TR
101T
58.1
48.7
23.4
26.5
23.9
18.4
15.1
28.6
30.3
$4.4
44.8
27.4
72.3
44.7
14.6
47.0
56.1
34.5
SAR TTF
200
25.9
2031
28.6
2025
26.7
23.7
2002
17.7
30.8
=01
32.4
300
48.0
2003
46.
2500
42.1
-0
72.3
45.3
20.16
34,2
2003
47.7
2004
56.1
2005
38.5
41.3
44.2
42.4
23.7
25.1
41.4
54.6
52.2
1.9
73.4
53.4
39.9
59.7
42.2
39123
DoYo + TTE
-009
42.5
20339
44.9
$0.34
43.5
34,8
29.15
14.22
32,8
19.19
43.3
34..5
45.9
55.8
53.7
2012
20.12
60.5
15.05
73.4
54.4
$1.0
=033
60.9
20.04
20.1
6014
2:005
50.5
VTlBase
9.4
6.7
29.1
23.4
34.0
27.0
15.8
26.3
47.4
54.7
43.9
30.5
44.5
47.6
29.9
43.
1.6
44.0
52.6
48.9
55.8
51.2
22.3
21.5
67.0
75.0
64.9
$4.3
67.2
64.4
48.9
Jent TTE
2006
492
2009
48.8
E105
50
56.2
55.8
20176
60.3
56.9
64.3
63.6
2300
71.4
76.9
67.0
3016
64.1
2000
70.7
2000
68.1
200
61.6
20.9
$7.4
65.5
56.9
166
11011
1008
1300
1901
18.9
1806
1811
SAR TTE
43.6
40.4
24.3
55.3
1011
$3.1
59.2
1018
15.4
61.5
10.
61.9
1011
70.7
76.7
66.7
61.9
20.2
131
67.3
1075
59.2
DeYo
101
1815
18.1
10.0
68.1
1013
1816
180
101e
161
70.5
63.1
DeYo TTF
53.8
$4.0
54.6
59.1
19m
1076
59.7
64.4
1070
02.3
68.4
66.8
2003
73.9
1210
1016
78.3
68.6
69.2
73.9
1009
70.7
100
65.2
1010
Classification accuracy (%) with the Mix Shifts setup (ImageNet-C, level 5). Underline depicts performance improvement when applying TTE. Bold numbers are the best results.
Table 15: Integration with previous TTA approaches.
Methods
ResNer50.GN
ViTRase
No Adapt
30.6
29.9
Tent
33.1=012
$2.3
TeILTTE
38.7=n32
$7.2-033
SAR
38.1
57.1
SAR+TTE
29.1
$7.51051
De
33.8114
DeYO&TTE
42.92018
60.6.10.5
25

---

Published as a conference paper at ICLR 2025
Table 16: Continual TTA with correlatively sampling. Classification accuracy (%) with ImageNet-C
(level 5). Bold numbers are the best results.
Correlative Sampling Both of Domain and Cliss (Adlaptztion Order -+)
Ganss
Shot
Empul
Defoe
Glass
Motice
Zoom
Snow
Frost
Fog
Brit
Contr
Elastic
Pixel
JPEG
Arg
ResNertO-GEN
18.0
19.8
17.9
19.8
11.4
21.4
24.9
40.4
47.3
33.6
69.3
36.3
18.6
28.4
52.3
30.6
Teni
-0.3
3,9
1.079
1.5
10.19
1.6
10.00
01.1
133
11:1
100
11.1
1010
01.1
0.1
0.1
1,30
0,1
100
011
100
0.1
0.1
0,1
0.1
0.6
CeTTA
10:11
11.33
18:47
10:45
1
1om
J0.0
10d1
1.00
1.0m
10:0
SAR
38.1
JB73
18:30
16.8
1000
44.7
44.6
10:25
42.3
18.8
45.7
1,0.77
37.8
10:51
39.7
9.3
10:3
3,1
10:77
2.1
0.7
5.4
1.2
23,0
1:8.0
1,307
1it00
10:0
10:00
1000
1.3m
10tc
TTE (W. DeYO)
20031
43,0
1.039
31.7
52.7
2002
10:12
53.3
111.4
45.1
2001
36.4
$2.2
1022
1.9111
53.6
10..V
40.9
1.0.5
43.0
2.005
59.1
1.0.27
60.3
1.9225
47.4
68.9
59.0
1,02
1.1.0
49.7
50.5
6.7
30.9
COTTA
2239
14.1
2500
13.6
323
2.2
0.1
230
4:3
0.1
4010
0.1
40.0
0.1
40.0
0.1
-00
0.1
5:00
0.1
223
0.1
20.0
0.1
=0.0
0.3
-10
0.1
=00
0.1
40.33
3.4
46.6
49.0
38.9
+3.12
39
DeYO
49.1
1538
17
36.7
63.3
1331
61.4
18.45
52.8
49.9
1317
59.1
62.3
1.39
48.9
54.0
11.
61.5
69.2
18-71
2.2
64.3
67.2
53.5
DeYO)
-0.37
10.6
1800
10.31
1006
1003
1002
18.3
1099
10.76
Table 17: TTA with natural distribution shifts. Classification accuracy (%) with ImageNet-Sketch (S), ImageNet-Renditin (R), ImageNet- V2 (V2). Bold numbers are the best results.
Label Shifts
(b) Batch Size
Methods
S
V2
Methods
5
V2
Avg
ResNe150-GN
Tent
30.8:0.0
41.510.11
68.9
47.1
Tent
31.9.0.2
12001
68.91000
47.6-00.
CoTTA
3088:0.1
41.210.1
68.81001
46.91:00
CoTTA
26.7.00.
10.6:003
68.7.002
45.310.1
SAR
30.6:10.1
41.61000
68.9100
47.01:111
SAR
31.6.002
11.9.01
6891101
47.510.1
DeYO
34.6204
44.4 44.41.02
68.91011
49.319922
DeYO
37.020
46 06:00
09.010011
50.1103
ITE (W. DeYO!
36.9:00.0
45.1:00
68.91:011
50.3::5.1
TTE (w. DeYO)
39.1:0.11
47.1
09.0::000
$1.71011
ViTBase
18.2
43.1
66.2
37.9
ViTBase
18.2
43.1
66.2
37.9
Teni
8.8.0.5
41.9106
68.9100
39.819.3
Tent
7.02002
40 40.62:03
69.1:10.1
38.91901
CoTTA
28.4
45.1
67.540.1
47.0:1044
CoTTA
24.0=000
46.0::.0
66.91000
45.6:000
SAR
17.8122.7
15.1100
68.51101
43.81:10
SAR
26.1:25
4.720.
69.11022
46.81:10
DeYO
12.1200
58.6.10.3
71.1102
57.3.002
DeYO
43.6:00.5
59.82:0.3
71.2:102
58.21001
ITE DeYO)
43.6.00.1
59.1100.1
71.1102
57.9.0011
TTE DeYO)
45.0.0011
61.32002
71.310.1
59.2.000
F LIMITATIONS AND FUTURE WORKS
The proposed TTE, as TTA approach, performs optimization with limited, unlabeled data during inference. In scenarios where test data is identical to training data, over-tuning hyperparameters based on the performance of test data could be controversial. To avoid this risk, we maintained con- sistent hyperparameter settings across all experiments, although these settings were still determined by performance outcomes. Additionally, reliance on hyperparameter values tailored for specific nitoo-ddistribution characteristics may lead to performance degradation when encountering out-of- distribution scenarios not anticipated in pre-experimental setups. Therefore, developing an effective validation method that determines hyperparameters using either in-distribution data or a subset of -it---i-ditiitttionn data represents a valuable direction for future research.
26


Reviews:
[{"review_id": "p0GYzfBdjC", "reviewer": "Reviewer_drM4", "summary": "This paper introduces a novel test-time ensemble approach that can be seamlessly integrated with existing TTA models to enhance adaptation. Specifically, the proposed framework reduces domain gaps through two ensemble strategies: weight averaging of TTA models and dropout. Additionally, a knowledge distillation strategy is employed to mitigate both noise and bias for improving model robustness under different distribution shifts. \nExtensive experiments are conducted in different TTA scenarios to demonstrate the superiority of the proposed method over existing baselines.", "questions": "Please refer to the weaknesses above.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. Writing quality is good. The paper is well-structured, and clearly written. \n2. Good insights. This paper explores TTA as a domain generalization problem, uncovering linear connectivity within TTA models. This perspective suggests that domain generalization techniques could enhance model representations for TTA tasks. \n3. SOTA performance. The proposed method achieves the state-of-the-art performance via the integration with different TTA models in various scenarios.\n4. Ablations. Ablation experiments are provided to verify the effectiveness of the proposed modules.", "weaknesses": "1. Although the results presented in Tables 3 show the performance improvement achieved by the proposed framework in the continual TTA scenario, it is unclear how the method enhances baseline performance in later adaptation stages. Additionally, I would like to know if the proposed method addresses the issue of catastrophic forgetting in this context. \n\nAdditional question:\nIs it possible to extend the proposed benchmark construction method to dense prediction tasks, such as semantic segmentation? It would be very meaningful if it can be applied to various tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LdKU8Naiom", "reviewer": "Reviewer_xGvc", "summary": "The paper proposes a method called Test-Time Ensemble (TTE), which uses an ensemble strategy to dynamically enrich model representations during online test-time adaptation (TTA). TTE constructs an ensemble network by averaging the parameter weights of different TTA models, which are continuously updated using test data. This weight averaging technology captures model diversity and improves representation quality without increasing the computational burden of managing multiple models. TTE further combines dropout to promote diverse collaboration of representations within TTA models, and also proposes a debiased and noise-resistant knowledge distillation scheme to stabilize the learning of TTA models in the ensemble. TTE can be seamlessly integrated with existing TTA methods, enhancing their adaptive capabilities in various challenging scenarios.", "questions": "1. Why the results of CoTTA in Continual TTA with non-i.i.d. conditions are only 2.2% and 3.4%?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. TTE utilizes an ensemble strategy to dynamically enrich model representations during online test-time adaptation (TTA), which is an interesting approach.\n2. TTE constructs an ensemble network by averaging the parameter weights of different TTA models, and this weight averaging captures model diversity, improving representation quality without increasing the computational burden of managing multiple models.\n3. TTE further promotes the diversity of representations within TTA models by combining with dropout, and proposes a debiased and noise-resistant knowledge distillation scheme to stabilize the learning of TTA models in the ensemble.\n4. The experiments are extensive and the results are superior to other compared methods.", "weaknesses": "1. The TTE method involves multiple hyperparameters, such as the momentum coefficient , dropout ratio and temperature, which may affect the stability and generalization of the method. Further research on how to reduce the dependence on hyperparameters is crucial.\n2. TTE integrates many well-established techniques such as ensemble, dropout, knowledge distillation, which have been utilized in TTA or few shot learning. The combination has weaken the novelty of the paper, and the unique contributions of the paper should be classified.\n3. How to conduct the weight-space ensemble without adding computational complexity? Will the technique increase storage consumption?\n4. The knowledge distillation-based debiasing and anti-noise strategies proposed in the paper may not be able to completely solve the problem of noisy test data. How to solve the scenario that the pseudo labels are incorrect?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wjsfHys9Si", "reviewer": "Reviewer_NKQc", "summary": "The paper introduces the Test-Time Ensemble (TTE), a method designed for TTA using the theory of weights space ensemble, which can be used on top of different TTA methods. The authors show different results for TTA over corruptions with different baselines, and the method seems to work pretty well. Furthermore, the authors also provided results for continual TTA, which is interesting.", "questions": "Here, I am adding the questions that I find relevant for improving the work quality; some of them were already discussed in the Weaknesses section:\n\n- Could you clarify this a bit more in section 3.2? How is it important for the method?\n\n- Do you think the batch size can impact the results, especially the ones provided for the continual TTA?\n\nPlease consider answering the points on the weaknesses as well. Furthermore, I am open to discussion, and I think that the work has a good potential for the community.\n\nRebuttal period: After carefully reading all the answers provided by the authors, I feel that my questions were answered, and I don't have any additional questions. I am confident to change my decision from \"marginally above the acceptance threshold\" to \"accept, good paper\".", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The paper is well-written, easy to follow, and detailed. I liked how the authors presented the work and motivated toward the problem. Furthermore, the results are motivating, and the idea seems easy to implement on top of different methods (as demonstrated by the authors), which can be beneficial for the community if the authors also provide the full code for reproducibility.", "weaknesses": "Personally, I did not see many problems with the paper, but I would suggest the authors proofread again to avoid problems such as the following typo \"with lager and more complex\" -> \"with larger and more complex\" in the introduction or \"Adaptvie momentum\" ->  \"Adaptive momentum.\"  \n\nIf the authors work on the following points, it will improve a lot the quality of the work: \n- I am not so convinced by section 3.2, DE-BIASED AND NOISE-ROBUST KNOWLEDGE DISTILLATION. Could you clarify this a bit more in this section? And maybe make it more clear in the paper.\n\n-  In Equation 5, there is no hyperparameter to balance the terms. I think it should be included, right? \n\n- For Table 3, with continual TTA, the authors compared all other baselines with TTE with DeYO, but not TTE with other variants as well, and for continual TTA, I don't understand how the performance can still perform well in the direction of the adaptation without degrading too much as we can see in the other approaches (for example DeYO goes from 28.1 to 3.7 and then 7.2), for me it only make sense if you ensemble with zero-shot model as well (or reset the model weights) but if this is the case it should be done for all other baselines as well.\n \n- For Table 4, column V2 seems strange, as almost all results in a) are 68.9, even DeYO and DeYO with TTE. Could you also add more results with other batch sizes? (the batch size can play an important role in different algorithms, which can benefit DeYO and not the others. For instance, I recommend taking a look at the paper \"Bag of Tricks for Fully Test-Time Adaptation, IEEE/CVF Winter Conference on Applications of Computer Vision. 2024\",  which shows the role of batch size in some of the TTA algorithms.\n\n- I would suggest revisiting some of the baselines for Tab 1. I would also consider a baseline with other methods of the local ensemble as well, such as SWA with TTA, and for Tab 4. I would also add other methods with TTE (maybe in the supp. material).\n\n\nRebuttal period: All my points were answered well, and I am quite happy with the effort of the authors during the rebuttal phase.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "okaQm6KXx5", "reviewer": "Reviewer_sF3M", "summary": "This paper considers a new problem, test-time ensemble (TTE), which aims at using multiple models generated during TTA. This paper first formulates the test-time ensemble problem. The paper also proposes the weight average and dropout as the baseline methods to evaluate the performances. \n\nThe contributions can be summarized as: \n\n(1) The author revealed that TTA models exhibit linear mode connectivity, an in- triguing insight that simplifies and enhances the adaptation process.\n\n(2) The author introduced Test-Time Ensemble (TTE), a novel and computationally efficient approach that not only enriches model representations but also stabilize TTA optimization through de-biased and noise- robust knowledge distillation.\n\n(3) TTE integrated effortlessly with existing TTA methods, enhancing adaptation in diverse scenarios and showing potential for applicability to future TTA methods.", "questions": "Please see the weakness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "(1) This paper proposes the new problem, test-time problem, which is different from previous test-time adaptations. I believe this problem has some practical applications.\n\n(2) The paper proposes some simple baseline methods that can effectively address the problem.", "weaknesses": "(1) The analysis of test-time adaptation does not inspire the new methods. The moving average ensemble methods are popularly adapted in self-supervised learning and ensemble methods. I consider the Linear Mode Connectivity theory should tell the reason and the situation that the models generated during test-time adaptation. \n\n\n(2) Limited technical novelty: this paper proposes the two-branch structure and leverage the weight average to improve the performance. Similar techniques are implemented in https://github.com/huggingface/pytorch-image-models. I do not see anything new compared to what have been proved in image classification.\n\n(3) Unclear description. In section 3, the de-biased distillation subsection does not describe clearly where the bias comes from. I suggest the author should explain the bias again. Also, I can not understand what the connection between the spike phenomena of the accuracy curve and the bias.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)