Exploring Human-in-the-Loop Test-Time Adaptation by Synergizing Active Learning and Model Selection
Authors: Yushu Li
Yongyi Su
Xulei Yang
Kui Jia
Xun Xu
Venue: Accepted by TMLR ()

Abstract: Existing test-time adaptation (TTA) approaches often adapt models with the unlabeled testing data stream. A recent attempt relaxed the assumption by introducing limited human annotation, referred to as Human-In-the-Loop Test-Time Adaptation (HILTTA) in this study. The focus of existing HILTTA studies lies in selecting the most informative samples to label, a.k.a. active learning. In this work, we are motivated by a pitfall of TTA, i.e. sensitivity to hyper-parameters, and propose to approach HILTTA by synergizing active learning and model selection. Specifically, we first select samples for human annotation (active learning) and then use the labeled data to select optimal hyper-parameters (model selection). To prevent the model selection process from overfitting to local distributions, multiple regularization techniques are employed to complement the validation objective. A sample selection strategy is further tailored by considering the balance between active learning and model selection purposes. We demonstrate on 5 TTA datasets that the proposed HILTTA approach is compatible with off-the-shelf TTA methods and such combinations substantially outperform the state-of-the-art HILTTA methods. Importantly, our proposed method can always prevent choosing the worst hyper-parameters on all off-the-shelf TTA methods. The source code is available at https://github.com/Yushu-Li/HILTTA.

Published in Transactions on Machine Learning Research (12/2024)
Exploring Human-in-the-Loop Test-Time Adaptation by Synergizing Active Learning and Model Selection
Yushu Li Yongyi Su*1. Xulei Yang² Kui Jia
Xun Xut²
'ootth China University of Technology, Guangzhou, China
Institute for Infocomm Research (F2 A*STAR, Singapore
School of Data Science, The Chinese University of Hong Kong, Shenzhen, China
Reviewed OpenReview: http:://gpaaeeese..aa////u1o1100
Abstract
Existing test-time adaptation (TTA) approaches often adapt models with the unlabeled testing data stream. A recent attempt relaxed the assumption by introducing limited hu- man annotation, referred to as Maman-In-the-Logg study, The focus existing HILTTA studies lies selecting the most informative samples to label, a.k.a. active learning. In this work, we are motivated by pitfall of TTA, i.c. sensitivity hyper-parameters, and propose to approach HILTTA by synergizing active learning and model selection. Specifically, first select samples for human annotation (ac- tive learning) and then use the labeled data to select optimal hyper-parameterr (model selection). To prevent the model selection process from overfitting to local distributions, multiple regularization lechniques are employed to complement. the validation objective, ive. A sample selection strategy is further tailored by considering the balance between active learn- ing and model selection purposes. We demonstrate on 5 TTA datasets that the proposed HILTTA approach is compatible with off-che-shelf TTA methods and such combinations substantially outperform the state-of-the-art HILTTA methods. Importantly, our proposed method can always prevent choosing the worst hyper-parameters on all off-the-shelf TTA methods. The souree code is available at https:://githut howp:tuuuuuccommmuuuuuuuuuuuua Li/HILTTA
Test-Time Adaptation (HILTTA) in this
1 Introduction
Deep learning has demonstrated remarkable success in numerous application scenarios. Nevertheless, the disparity in distribution between training and testing data leads to poor generalization of deep neural net- works. In many real-world scenarios, the data distribution of testing data is often unknown until the inference stage (Sun et al. 2020 Wang et al. 2020) and the distribution may experience constant shift over the course of inference (Wang et al. 2022 Yuan et al. 2023 Press et al. 2023 Lee & Chang 2024 Su et al.) 2024a These practical challenges render traditional unsupervised domain adaptation and source- free domain adap- tation paradigms ineffective. To overcome the above challenges, test-time adaptation (TTA) emerges by adapting model weights upon observing testing data at inference stage (Wang ct al., 2020 Sun et al. 2020:
Wang et al. 2022 Su et. al., 2022 2024a). The success of TTA is often altributed the practice of aligning
testing data distribution with source domain distribution (Su et al. 2022
Kang et al.
2023 Wang et al.
2023a) and/or self-training on pscudo labels (Jang et al.) 2023 Su et al. 2024b Marsden ct al. 2024a
Despite achieving impressive results on a wide range of (asks, e.g. classífication
SSnn et al.
2020
image TTA
segmentation (Wang et al. 2023bb and object detection (Chen et al. 2023), the fully unsupervised
paradigm is still overwhelmed by remaining challenges including hyper-parameter tuning ZZao et al.
2023
continual distribution shift (Wang et al. 2022). non-i.i.d. (Gong al. 2022 Niu et al. 2023 Yuan et al.
"Equal contribution. This work done during Yushu Li and Yongyi Su's attachment with 1 visiting students. i Coorrepponding author. Email: alex. www.xuunnnnnnaa..oommmmmm

---

Published in Transactions on Machine Learning Research (12/2024)
2023 class imbalance (Su et al. 2024a), etc. Unfortumately, there are still no principled solutions to address
these challenges without relaxing the assumptioos.
Test Data Stream with Unknown Data Distribution
Gaussian
Snow
Motion
Pop a real-time
data batch
TTA Method
Candidate Models
Hyper-parameler value: w, ." WNa
Unsup. Train.
Model 0" ((w,)
Model 0 ('(wN,,.
e.g. Entropy Threshold r
Human-in-the-Loop TTA
Active Learning
Model Selection
D
(xy, (i
0 = argmin L (6m)
Model 0r-1
Sample Selection
Annolate Few-shol Samples
m
m C (1. Me)
K-Margin
Prediction
Sup. Train
Model 0
Human Annolation
Update for the next step
Figure 1: An illustration of Human-in-the-Loow Test-Time Adaptation framework. Upon collecting batch of testing data J=:.:.....N. we train multiple candidate models 0* (wm) using candidate typer-parameters {wm] We further select candidate samples for annotation through K-Margin. Model selection is achieved by choosing the syper-parameter minimizing validation objective. We repeat until the testing data stream concludes, following continual TTA framework.
Recently, efforts have been made to the exploration of introducing a limited annotation budget during test- time adaptation (SimATTA) (Gui et al. 2024). In this study, we refer to this concept test-time adaptation (HILTTA). Through theoretical analysis, SimATTA (Gui al.) 2024) demonstrated that adapting pre-trained model with carefully selected labeled samples the testing data stream could
substantially improve the effectiveness of TTA and prevent catastrophic forgetting (Kemker et al. 2018
Li & Hoiem 2017) upon adapting to continuously changing testing domains (Wang et al. 2022). Apart from proving the theoretical guarantees, the main focus of SimATTA (Gui et al., 2024) lies in developing a balanced incremental clustering approach to select the most informative samples on the testing data stream for annotation. Specifically, SimATTA lakes into consideration the model uncertainty and feature diversily for selecting the most informative samples to annotate on the testing data stream.
Despite the theoretical guarantee and superior empiri- cal observations, the underlying practice of utilizing la- bel budget by existing active HILTTA methods follows the traditional line of research on active learning (Sener & Savarese 2018 Saran et al., 2023). In this study, we aim explore the question: How can human-in-the- loop test-time adaptation be realized beyond an active learning perspective? To answer this ques- tion, first revisit the pitfalls of state-of-the-art test- time adaptation (Zhao et al. 2023). Among the multiple remaining pitsalls, we pinpoint hyper-parameter selection as the most nasty one due to the sensitivity to the choice of syper-parametted by many TTA methods. Hyper-parameter selection, also referred as model selection, aims choose the optimal hyper-paramcters for model training. Established paradigms towards model selection often rely on a held-out validation set and the model selection task is defined discovering the papaeeearsmeters trained upon which the model
Table 1: The differences between pro- posed human-in-the-loop test-time adaptation (HILTTA) and existing TTA settings.
Active
Model
Setting
Learning
Selection
Test-Time Adaptation (TTA)
Active Test Time Adaptation (ATTA)
Human-in-the -ite-iittthh-oopppp TTA
(HILTTA)
2

---

Published in Transactions on Machine Learning Research (12/2024)
displays optimal generalization performance the held-out validation set (Yang & Shami 2020 Inspired by the success of hyper-parameter optimization, we offer a fresh perspective on human-in--l teeen--ii-ttte-looe thoploop test-time adaptation by synergizing active learning and model selection. In specific, we present an HILTTA procedure with three steps. First, upon observing a testing sample, the model will make an instant prediction to min- imize inference latency. When a fixed number of lesting samples are observed, we employ data selection strategy to recommend a subset of testing samples for human oracle to label. The labeled data will then be utilized as the validation set for selecting the best sxper-parammeers. Finally, the model further updated with the labeled data in a fully supervised fashion. With the novel HILTTA steps, intend to make better utilization of the limited annotation budget compared to existing HILTTA approaches. We demonstrate the differences between our proposed HILTTA and related TTA settings in Tab. D
To enable the novel HILTTA procedure, we identify two major challenges. First, the validation objective deserves delicate design due to the small validation set. particular, without seeing the whole picture of testing data, naive model selection strategy may overfit the labeled testing data within a local time window, resulting in unstable and biased hyper-parameter selection. Second, the strategy for selecting testing samples to annotate must be well-calibrated for the task of HILTTA. Employing the data acquisition functions developed for generic active learning (Snner & Savarese 2018) or active tosting (Kosson et al. 2022 is suboptimal.
To tackle these challenges, we first propose a hyper-parameter regularization technique by considering the deviation of model predictions from frozen source model. We further combine the ranking scores of validation loss and the deviation as the scoring function for model selection. To accommodate the active learning strategy for HILTTA from model selection perspective, and sample diversity and combine the two objectives through a simple re-weighting, Sample selection is then achieved by maximizing the coverage in the re-weighted feature space. Extensive evaluations on TTA datasets demonstrate the effectiveness of the proposed HILTTA approach.
prioritize both model uncertainty
We summarize the contributions of this work as follows.
To the best of our knowledge, the first to approach Human-In-the-Looy Test-Time Adaptation (HILTTA) by synergizing active learning and model selection and propose off-the-shelf HILTTA method.
To prevent the model selection from overfitting to a small validation set and local distribution, we introduce multiple regularization techniques. We further identify an effective selection strategy for HII.TTA by considering the feature diversity and model uncertainty.
We demonstrate that the proposed method is compatible with state-of-the-ar TTA methods. The combined methods outperform dedicated active TTA methods and stream- based active learning methods on five test-time adaptation datasets.
2
Related Work
2.1
Test-Time Adaptation
The generalization of deep learning model is hindered by the distribution gap between source and target domain. Traditional approaches towards improving the generalization of deep learning models follow un- supervised domain adaptation (UDA) (Ganin & Lempitsky (015)) which adapts model weights to learn indiscriminative features across source and target domains. Simultaneously training both source and target domain data is infeasible in many real-world scenarios where access to source training data is pro- hibited due to privacy or storage overhead. Source-free domain adaptation (SFDA) targets this issue and adapts model weights to larget domain data only (Liang el al.) 2020: Liu et al. 2021 Yang et al. 2021 Liang et al. 2021). In contrast to the assumptions made UDA and SFDA, rest-time adaptation (TTA) the need for adapting pre-trained model unknown target domain at inference stage (Su et al. 2022 Sun et al. 22200) Wang et al., 2020p.. TTA has been demonstrated to improve model's generalization to out-of-distributioon lesting data and the success owns to self-training on unlabeled testing
emerges response
3

---

Published in Transactions on Machine Learning Research (12/2024)
data. To prevent the model from being mislead by incorrect pscudo labels, a.k.a. confirmation bias (Arzo et al. 2020 people choose to update only a fraction of model weights, e.g. classifier weights (Twasawa & Matsuo 2021) and normalization layers (Wang et al.) 2020), or introducing regularizations, e.g. distribution
alignment (Liu et al. 2021 Su et al., 2024b) and reprosentation regularization (Su et al. 2024a). More recent focus of lest-lime adaptation research lies on lackling challenges arising from real world lesting data, including continual distribution shift. (Wang et al., 2022), class imbalance (Su et al. 2024a), update with small batchsize (Niu et al.] 2023) and open-world testing data (Li et al. 2023 Zhou et al. 2023). Never- theless, existing TTA approaches are mainly built upon fully unsupervised adaptation protocol. We argue that sparse human annotation budgets might be available and exploiting annotation budget in TTA, referred to as comman-in-the-lon TTA, has gained attention from the community (Gui et al. 2024). Contrary to the traditional active learning perspective (Gui et al.) 2024) we identified an unexplored opportunity to treat labeled data as the validation set for model selection and we propose multiple techniques to improve the efficacy of human-in-the-loop test-time adaptation.
2.2 Active Learning
Active learning (AL) aims to select the most informative samples for Oracle to label for supervisod training. Prevailing approaches often prioritize uncertainty (Gal & Kendall 2017), feature diversity (Sener & Savarese 2018) or the combination of both (Ash et al. 2020). The traditional AL approaches often adopt a. batch- wise paradigm. When data arrive in a sequential manner stream-based AL (Saran et al.) 2023) proposed to quantify the increment of the determinant as the criteria for incremental sample selection. Active learning was integrated into test-time adaptation to tackle the forgetting issue (Gui et al. 2024). In contrast to tackling HILTTA from a pure active learning perspective, we proposc to synergize active learning and model selection. Selecting samples for model selection often relies on an orthogonal objective to active learning. In the related works, active testing (Kosen et al. 2021 2022 Matsuura & Hara 2023: Ochiai et al.; 2023) was developed to select subset of testing data that representative of the whole testing data. Active testing has been demonstrated to help label-efficient model validation. Nevertheless, active learning and active testing will prioritize supervised training and model validation respectively. We believe either is not enough for the purpose of synergizing active learning and model selection for test-time adaptation.
3 Methodology
3.1
Human-in-the-Loop Test-Time Adaptation Protocol
We first provide an overview of the existing test-time adaptation (TTA) evaluation protocol. TTA aims to adapt pre-trained model Mx;;)) 0) to a target testing data stream ID = (I,, Dr..ii-----... where nq and J. E [1...C} denote the testing sample and the associated unknown label, respectively. Existing test- time adaptation approaches often make an instant prediction á = |(<;; 0) on each testing sample and then update model weights 0 upon observing a cumulative batch of samples B, = {==,==1-..N, where refers to the t-th minibatch. Prevailing methods enable adaptation by self-training (Wang et al. 2020 Su et al. 2024b) or distribution alignment (Su et al., 2022). Unlike the existing assumption of totally unsupervised adaptation, additional human annotation IB; = (Mj = within each batch could be introduced (uui ct al. 2024) and TTA is implemented on the combine of large unlabeled data By = B, \ B/ and sparsely labeled data B/- We refer to the novel evaluation protocol as Human-in-the-LLop Test-Time Adaptation (HILTTA) throughout the study.
3.2 Model Selection with Sparse Annotation
We present the details for model selection with sparse human annotation on the testing data stream. The choice of hyper-parameters is known to have a significant impact on the performance of model training and generalization. W.L.o.g,, we define the task of model selection by introducing a fixed candidate pool of eyper-paramcters f2 {am]m=i...... with Nm options. The choice of hyper-parameters hinges on the sensitivity of respective TTA methods, e.g. pscudo label threshold, learning rate, and loss coefficient are commonly chosen hyper-parameters for selection. We further denote the discriminative model trained upon
4

---

Published in Transactions on Machine Learning Research (12/2024)
the apper-paramcters w as ((w). The model selection can be formulated bi-level optimization problem as follows, where Lir (::w) denotes the training loss dictated by the hyper-parameter w, Leat(:;)) denotes the validation loss/objection and D' refers to the labeled data.
min WER NI
(xa( Σ (ASAM))ED
s.t. ('(w) = arg min
Σ ()) N₂ R,ED
1
(1)
We interpret the above bi-level optimization problem as discovering the optimal hyper-parameter trained upon which the model weights 0* achieves the best performance the labeled dataset. Solving the above problem through gradient- method (Liu et al. 2018) is infeasible for TTA tasks where adaptation speed is major concern while gradient- methods require iteratively updating between meta gradient descent VaLral and task gradient descent VoLtr- Considering the task gradient descent only takes a few steps in a typical TTA setting, we opt for an exhaustive search for the hyper-parameters the discretized hyper- parameter space. In specific, we enumerate the candidate models adapted with different parameeesss {6m Im=...1. The candidate with the best validation loss is chosen the best model 0", follows.
Vm L...., Nm: 0" arg min
Σ
(has( 0),
(E
(2)
s.t. 0n = arg min
Σ Z.ED
(er(h(2;;;) wm)
N
The specific choice of training loss fir could be arbitrary off-the-shelf test-time adaptation methods while the design of validation loss Leaa deserves careful attention in order to achieve robust model selection.
3.3 Design of Validation Objective
In this section, take into consideration two principles for the construction of validation objectives. First, the validation loss should mimic the behavior of the model on the testing data stream. Assuming the limited labeled data lakes a snapshot of the whole testing data stream and classification is the task to be addressed, an intuitive choice is the cross-entropy loss as follows.
Hos (2,; 0m) = Σ 1 log ha( 0%,
(3)
cml
Alternative to the continuous cross-entropy loss, accuracy also characterizes the performance on the labeled validation set. However, the discretized nature of accuracy renders this option less suitable due many tied optimal hyper-parameters.
Regularization for Stable Model Selection: The stability of model selection heavily depends on the selection of the validation set and the variation of target domain distribution. Under a more realistic TTA scenario, the target domain distribution could shift over time, e.g. the continual test-time adaptation (Wang et al. 2 Such a realistic challenge may render cross-entropy loss less effective due to over-adapting local distribution. To stabilize the model selection procodure, we propose the following two measures.
Anchor Deviation for Regularizing Model Selection: We first introduce anchor deviation to regularize the model selection procedure. This regularization is similar to the anchor network proposed in Su et a (2024a) except that we use the anchor deviation to regularize model selection which is orthogonal to the purpose in Su et al. (2024a). Specifically, we keep a frozen source domain model, denoted as 0g. The anchor loss is defined as the L2 distance between the posteriors of the frozen source model and the m-th candidate model 0% A big deviation from the source model prediction suggests the model has been adaptod very specific target domain and further adapting towards one direction would increase the risk of failing to adapt to the continually changing distribution.
5

---

Published in Transactions on Machine Learning Research (12/2024)
Rane mm) No
Σ (EI.N)DDE
[1(a;;;)) - 1((ay 0%,)
(4)
Smoothed Scoring for Model Selection: To determine the optimal model to select, we combine the cross-cntropy loss and the anchor deviation as the final scorc. Direct summing the two metrics as the model selection score is sub-optinal due to the relative scale between the two metrics. Therefore, we propose x-min(xx) to first normalize the two metries as Norm(z) max(s)-min(z)> After the normalization, the relative size relationships of different candidate model scores are preserved, and a comparable validation loss can be conveniently obtained by addition, as below.
Leat (Is; 0%) (er (m)) + (1: s.L. Vm 1....,,.111...... [0], (ec( (%) Norm(Hea(wss5 Om)), Sane(1;;0 Norm(Ranc(x;; 0m)):
(5)
Considering that the testing data stream is often highly correlated, i.e. temporally adjacent samples are likely subject to the same distribution shift, the optimal poper-paramttermmoooe /ooeel should change smoothly as well. To encourage a smooth transition of the optimal model, we select the optimal model following an exponential moving average fashion as in Eq. c where we denote the moving average validation loss the i-th batch L and the chosen best model 8 is obtained the that minimizes the smoothed validation loss.
C.. (Ba)) B.
1-0--1(05 (0m)
3)
(h Un), Σ
B
(6)
0r = min L
m
3.4 Sample Selection for HILTTA
The sample selection strategy should balance the objectives of active learning and model selection for more effective HILTTA. We achieve this target from two perspectives. First, the selected samples should prioritize low-confidence samples for more elfective active learning, adopted by SimATTA (Gui et al., 2024) through incremental K-means. However, selecting redundant low-confidence samples may harm the effectiveness of model validation. This motivates us to select samples approximating the testing data distribution, similar to the objective of active testing (Kossen et al. 2021). Biasing towards either criterion would result in a suboptimal HILTTA. Our design mainly aims to balance the preference over selecting low-confidence and diverse samples.
A similar objective above was achieved by recent active learning approach, BADGE Ashh al. 2020 which takes the gradient of loss w.r.t. the penultinate layer feature the uncertainty weighted feature as 00gout (h(r) 0), where fout E IR DxC is the classifier weights. Hence, the gradient embedding E ERD D-C is of dimension D C. When the semantic space is substantially high, e.g. C. = 1000 for ImageNet (Deng et al.) 2009), the above gradient embedding results in extremely high dimension features. For instance, a typical penultimate layer feature 2048 channels and the gradient embedding could reach 2, 2,048,000 channels. Such high dimension prevents efficient clustoring operations, c.g. due to memory constraints, for sampling. To mitigale the ultra-high dimension challenge, we integrate uncertainty into feature representation by multiplying the uncertainty, measured by the margin confidence, to the feature representation as in Eq. where sort refers to the descending sorting.
n = sort [/1 0), h2(2;; 0) [o(x;; 9: = (1 - f1 + (m;;;
(7)

---

Published in Transactions on Machine Learning Research (12/2024)
The above design prioritizes the difference between the probability of the most confident and second most confident classes a quantification of model uncertainty. Such a design is superior to alternative designs, e.g. entropy, for the following reasons. First, deep learning models tend to be over confident with low entropy in general. Thus the gap between top and top 2 class probabilities may better suggest the model confidence than entropy. Moreover, the entropy may become less distinct for a classification task with many categorics as the randomness on non-ground-truth classes may affect the overall entropy calculation. Finally, our empirical observations in Sect. 4.3 also suggest the superiority of using the probability margin.
The samples to be selected for annotation are eventually obtained by conducting K-Center clustering on the uncertainty-weighred features within each minibatch, as in Eq. g which is referred to as K-Margin clustering. We select K samples BBj such that the maximal distance of all testing sample embedding is minimized. We deliberately exclude the selected samples in the previous minibatch, as opposed to SimATTA (Gui al. 2024)), because the labeled samples will serve as the validation set, and excluding new samples that may overlap with previously selected samples could undermine the effectiveness of model selection. The following problem can be solved via greedy selection algorithm (Sener
Sample Selection via K-Margin:
& Savarese 2018
min max min l1:: celb 8.1. | SK (. N ( CLEB
(8)
Our K-Margin approach with the K-Center clustering approach offers a computationally efficient alternative to K-Means++ while similarly balancing uncertainty and diversity in sample selection. Unlike +-Means++, which requires iterative center updates, K-Center clustcring selects points that maximize coverage by mini- mizing the maximum distance from each sample to its nearest center. This method provides representative subset with less computational overhead, which is more suitable in TTA settings. By focusing on high- uncertainty, diverse samples, K-Center clustering ensures a labeled subset that accurately reflects the test distribution. It guarantees that the labeled subset spans the feature space with fewer computations, thereby maintaining consistent approximation of the test distribution. This enhances model selection and super- vised training by creating robust validation set that captures broad feature variance without rodundant low-confidence samples.
3.5 Overall Algorithm for HILTTA
We propose two types of training losses for model updates and a validation loss for model selection. During the model selection stage, we generate multiple candidate models by training with unsupervised TTA loss, denoted Lgp following the off-the-shelf TTA methods. The optimal candidate model 0 is selected using arg min (O%)) After selecting the best model, we further refine it through supervised training,
optimizing cross-cntropy loss Cr The complete ftuaan-inther tnop Test-Time Adaptation algorithm is detailed in Alg. 1
4 Experiments
4.1 Experimental setting
Datasets: We select a total of five datasets for evaluation. The CIFARIO-C and CIFAR100- C (Hendrycks Dictterich 2018) are small-scalc corruption datascts, with 15 diffcrent common corruptions, each containing 10,000 corrupt images with 10/100 categories. For our evaluation in large-scale datasets, we opt for ImageNet-C (Hendryeks & Dietterich 2018), which also contains 15 different corruptions, each with 50,000 corrupt images in 1000 categories. Additionally, ImageNet-D (Rusak et al. 2022 is a style-transfer dataset, offering 6 domain shifts, each consisting of 10,000 images selected in 109 classes. Additionally, we
evaluate our method on the ModelNet440-C dataset (Sun et al. 2022). which includes 3, 180 3D point clouds affected by 15 common types of corruption.
Implementation Details: We evaluate TTA performance in the continual adaptation setting Wang al. 2022 Niu et. al. 2022). where the target domain undergoes continuous changes. For CIFAR-10-C and
7

---

Published in Transactions on Machine Learning Research (12/2024)
Algorithm 1: Human-in-the-Looog TTA
Input Source Model 06: Candidate
Hyper-parameter Q {wm]: Testing Data
Batchos {B₂}
Output: Predictions y {ú:}
for == to I' do
# Make Predictions:
Vx, B2, 01 h(z,; 10- 5 = SU # Oracle Annotation:
Select labeled subset B; by Eq. 8
for m = to Nm do
# Unsupervised Model Adaptation:
8th = arg min No
1 Σ (h(,;; 0); wm)
B,EB,
# Update Moving Average Validation Loss: Update Fluat by Eq. 6.
# Model Selection by Eq. 6:
o = arg min La (0%)
# Supervised Model Adaptation:
arg ming No
1
Σ
C (h(z,; um)
2i
return Predictions D;
CIFAR100-C datasets, we use a batch size of 200 and an annotation percentage of 3%, with pre-trained WideResNet-22 28 İZagouyko & Komodakis 2016) and ResNeXt-29 (Xie et al.) 2017] models, respectively. For ImageNet-C and ImageNet-D, we use a batch size of 64 for adaptation and an annotation percentage of 3.2% of the total testing samples, employing the ResNet-50 (He ct al., 2016) pro-trained model.
We the optimizer recommended in the original papers for each method's unsupervised training. We set the momentum & for 0.5. For supervised training, use the Adam optimizer with learning rate of 1e-5. We consider seven distinct values for each hyper-parameter category regarding model selection, detailed in Tab. S
Competing Methods: We evaluate state-of-the-ar off-the-shelf test-time adaptation approaches under both traditional unsupervised TTA and faaman-in-ther Loop TTA protocols, including TENT (Wang et al. 2020), PL (Tee al., 2013), SHOT (Liang al.) 2020), EATA (Niu et al. 2022 SAR (Niu al. 2023) RMT Döbler et al. 2023 For each of the above TTA method under HILTTA protocol, we embed the
TTA loss into the "I "Unsupervised Model Adaptation" step of the HILTTA algorithm in Alg, [ We further benchmark against existing active TTA methods, including SimATTA (Gui et al. 2024) and VeSSAL (Saran
et al. 2023). Specifically, we adapt VeSSAL for TTA by using the selected samples for supervised training. More dotails of the benchmarkod mothods are doferred to the Appendix.
4.2 Evaluations on HILTTA
We report the classification error rates for continual TTA in Tab. 2 For unsupervised TTA methods (Unsup. TTA), report three results: worst(Worst),, average (Avg.), and best (Best). The "Worst" and 'Best' results are derived from fixed dyper-parametent that yield the highest and lowest overall error rate, respectively. The 'Average' results are calculated by averaging the accuracy across models trained with all candidate typer-parameeers. For the competing active TTA methods (Active TTA), we maintain the same annotation rate as our method and use manually tuned typer-parameters to ensure a fair comparison. These methods, lacking model scloction capabilitics, result single error rate per dataset. In contrast, our TTA) automatically selects syper-parametenn using our proposed model selection strategy, also resulting in single error rate per dataset.
method Stuman-in-the-Iae
8

---

Published in Transactions on Machine Learning Research (12/2024)
Table Average classification error (lower is better) through ongoing adaptation varying domains. 'Worst', 'Avg.'. and "Best' indicate the worst, average, and best performance within the candidate hyper- parameter pool Ω. The improvement of error rate for each off-the-shelf TTA method after combining with
HIL is appended in the bracket as (ő). denotes methods under manually tuned typer-parameters. & de-
notes methods combined with our proposed HILTTA (with model selection). TWe adapt VeSSAL for ATTA
by combining TENT as an unsupervised training method.
ImageNet-C (%) 1
IetgeNet-D (%)
CIFAR100-C (%)
CIFARIO-C (%6
TTA Method
Worst
Avg.
Best
Worst
Avg.
Best
Worst
Avg.
Best
Wonst
Avg.
Best
Source
82.02
58.98
46.41
43.51
TENT (Wang at al.
2020
95.23
73.13
62.96
83.54
59.82
52.93
94.26
50.69
32.75
71.22
27.77
18.14
PLeLee al. 2013
89.78
81.48
67.75
63.06
56.59
52.45
37.31
34.67
32.76
19.95
19.11
18.34
Unsup.
TTA
SHOT Liang
2020
99.18
79.56
64.68
91.51
64.51
53.69
90.72
43.89
32.90
52.87
23.22
17.26
EATA Niu
62.41
51.30
42.72
19.57
SAR
Niu et al.
2023
90.60
70.70
62.20
56.19
53.37
51.72
39.00
84
31.90
20.22
19.10
17.45
RMT Dible
Dübler et al.
2022
96.77
72.28
60.45
88.80
58.92
50.
38.35
32.18 18
30.16
29.36
19.75
16.66
Active TTA*
VeSSAL Saran
62.78
SImATTA
Gui et 2024
61.09
49.26
31.75
17.66
HIL + TENT
(+14.78)
48.74
(+-11.08)
30.53 (4-20.16)
15.87 (-11.90)
Human-in-
HIL +
61.60 (+ 19.88)
49.01 (+7.58)
30.82 (+33.85)
16.97 (+-2.14)
thopLoop TTA (OURS)
HIL + SHOT
60.66 (+18.90)
48.98 (+-15.53)
31.13 (+12.76)
15.78 (+7.44)
HIL + EATA
55.13 (+5.13)
47.51
(+33667)
30.93 (+11.79)
16.40 (4-3.17)
HIL. + SAR
58.91 (+ (+111.9)
50.02 (+33335)
30.00 (+3.844)
16.87 (+2.23)
HIL + RMT
58.26 (+11.02)
16.28 (+12.64)
29.41 (+2.77)
16.43 (+3.30)
From the results, we make the following observations:
i) Existing unsupervised TTA methods, when applied without additional annotation, exhibit significant sensitivity to hyper-parameters. This sensitivity can lead to large variations performance (c.g., TENT's best accuracy at 62.96% versus its worst at 95.23% on ImageNet-CC.. In the worst case, performance can even fall below the source model without any adaptation. In contrast, the 'Best" performance across all methods are relatively close to each other. Nevertheless, selecting the best model without oracle knowledge is impractical in real-world scenarios. These findings highlight the persistent challenge of hyper- parameter sensitivily in TTA.
ii) Our proposed human-in-thhe-lop TTA consistently improves the performance of all off-the-shelf TTA methods against their unsupervised counterparts even with the best apper-parameters. This demonstrates the robustness of our approach, mitigating the catastrophic failures that can arise from hyper-paramen sensitivity.
iii) Compared to existing active TTA approaches, even when manually tuning the model selection issue, our proposed HILTTA consistently outperforms them across all four datasets. Notably, even a basic bascline like TENT achieves performance comparable to or better than existing active TTA methods when combined with our approach. These results highlight the effectiveness of integrating a small amount of labeled data through synergistic active learning and model selection, rather than relying solely active learning.
to address
4.3 Ablation & Additional Study
Unveiling the Impact of Individual Components: We conduct comprehensive ablation analysis using TENT the off- off-the-shelf TTA method in Tab. 3 Compared with not adapting to the testing data stream, fine-tuning with only the unsupervised training loss (Unsup. Train) could improve the performance, expected. When human annotation is introduced, using the cross-entropy loss Hos as validation objective (CE Valid.) does not guarantee a consistent improvement. As discussed in Zhao et al. (2023). this can be attributed to the challenges posed by the min-max oquilibrium optimization problem across time, resulting in a notable performance decline. Incorporating additional anchor regularization for model selection (referred to Anchor Reg.) consistently improves performance beyond the average error rate of the original TENT.

---

Published in Transactions on Machine Learning Research (12/2024)
This underscores the necessity of stronger regularization for model selection to achieve overall good TTA performance. Additionally, integrating FMA smoothing (FMA Smooth.) of validation loss and supervised training with labeled data (referred to as Super. Train) both contribute positively to enhancing TTA performance.
Table 3: Ablation study of HILTTA with TENT (Wang et al. 2020) as off- sh-thh-shelf TTA method.
Uusup.
CE Anchor EMA Super.
Errur Rate (%) &
Remarkk
Train Valid.
Reg.
Smooth Train
TetgeNet--C ImageNet-D CIFAR100-C CIFARIO-C
Source
$2.02
58.98
46.44
43.51
TENT
73.13
39.82
54.97
27.77
w/
HIL
90.52
62.66
67.82
21.36
HIL
V
v
/
65.95
52.87
47.55
17.79
W
62.61
52.61
34.18
17.79
48.74
30.53
15.87
Evaluation of Alternative Active Learning Strategics: To optimize annotation resource allocation, systematically evaluate various sample selection stralegies within the HILTTA protocol, as illustraatd We explore multiple approaches, including random sampling (Random), entropy-based selection (Entropy( Shannon, 1948)). ), feature diversity sampling (CoreSet (Sener & Savarese 2018)), stream-based active learning (VeSSAL [Saran et al. 2023)), ), the incremental clustering method introduced by Gui et al. (2024), and also our approach, K-Margin, uniquely combines uncertainty and feature diversity. Given that other active learning methods lack model selection capabilities, employ cross-entropy loss Hoe the validation objective. The results clearly demonstrate that K- K-Margin significantly outperforms other selection strategies in terms of average error rate across four datasets.
in Fig 2
ImageNet-C
ImageNet-D
CIFAR100-C
CIFAR10-C
80
Error Rate (%)
70-
52-
0
50
15-
16-
Random
Entropy
CoreSet
VeSSAL
ASE
OURS
Random
Entropy
CoreSitt
leSSAL
AsE
OURS
Random
Entrpy
Corefiet
Ve
ASE
Ic
OURS
Random
Entropy
CoreSiir
VLSSAL
AsE
cc
OURS
Figure 2: Comparison of different active learning strategies under HILTTA with TENT WWagg al. 2020 as TTA unsupervised model adaptation strategy. Average classification error is reported, where a lower value indicates better performance.
Evaluation of Alternative Model Selection Strategies: We further compare our approach against sev- eral statc-of-the-art model selection methods, including Entropy [Morerio et al., 2018). InfoMax (Muggrave et al. 2022), and MixVal (Hu al., 2023), which are widely recognized in Unsupervised Domain Adapta- tion. Additionally, consider the Oracle model selection strategy for online TTA, known PitTTA (Zhao) et al. 2023), which utilizes all labels within the streaming test dataset D, to perform model selection based on accuracy comparison. The results presented in Tab demonstrate that our proposed methods consis- tently surpass these alternative strategies, delivering comparable or superior performance to the best results obtained with fixed hyper-parameter settings. Notably, our HILTTA approach outperforms PitTTA (Zhao et al. 2023) while requiring significantly fewer annotations, making it a more practical and resource-eflicient solution.
10

---

Published in Transactions on Machine Learning Research (12/2024)
Table 4: Comparison of different model selection strategies under the HILTTA protocol with TENT (Wang ct al. 2020 as unsupervised model adaptation strategy.
Error Rate (%)
Strategy
ImageNet-C
ImageNet-D
CIFAR100-C
CIFARIO-C
Avg.
Oracle Worst
94.98
76.27
94.75
27.81
73.45
Oracle Best
58.49
48.55
30.11
15.75
38.23
Entropy
Morerio et al. 2018
94.65
73.95
94.11
26.66
72.34
InfoMax
Musgrave et al.
2022
94.56
77.00
93.14
28.81
73.38
MixVal
Hu et al. 2023
92.98
68.20
92.23
19.64
68.26
РirTTA
Zhao et 2023
60.75
47.95
47.99
16.23
43.23
HILTTA
(Ours)
58.35
48.74
30.53
15.87
38.37
Evaluations beyond 2D Image Recognition: To fur- ther evaluate the effectiveness of our method beyond 2D im-
age classification task, we conduct experiments on 3D point cloud classification using DGCNN (Wang et al. 2019 as
199 Pale Enne
2751
the backbone, adapting continuously across domains within the ModelNet44D-C (Sun et al., (022) dataset. We maintained the same pyrer-paramettr candidate set as out- lined in Tab. g and followed the experimental setup detailed in Tab. 2. with a batch size of 32 and an annotation per- centage of 3.2%. The results, presented in Fig. 3 show that unsupervised TTA still consistently outperforms the base- line without any adaptation (Source). However, due to the sensitivity hyper-parameters, all unsupervised TTA meth- ods exhibit substantial variation in performance with the "Best" and "Worst" typer-parameters. All off off-th TTA methods are consistently improved when integrated with our HILTTA method. Five out of the six methods wilness a more significant boost in performance, surpassing their unsupervised counterpart with "Best" ayper-parameters.
22.45
20100
TENT
Figure 3: Performance on the 3D point cloud classification task in ModeINet440-C dataset. Average classification error is reported.
Computation Efficiency: A major concern for test-time adaptation methods is the computation efficiency, We conduct empirical studies to measure the wall-clock time of both the inference and adaptation steps in HILTTA, utilizing single RTX 3090 GPU, Intel Xeon Gold 5320 CPU, and 30 GB of RAM. The results are presented in Tab. 5 In this experiment,
only perform human-in-the-loop annotation for every N batch on the testing data stream. The inference time required is fixed for all methods and the adaptation time varies. We demonstrate that the additional model selection procedure significantly decreases the error rate (27.77% + 15.87%) with times the overall time required (Ims v.s. 9.1ms). More importantly, the overall time required can be further reduced by conducting sparser human annotation (larger N). With N=10, we witness a 10% absolute error rate decrease (27.77% + 17. 17.13%) with only 2.8 times computation time. Considering that annotating single image would cost 3000 ms to 5000 ms, the adaptation time is negligible.
Table 5: Compulation efficiency under different human intervention frequencies.
w/o HIL
w/ HIL N-1
w/HIL N-3
w/HIL N-5
w/HIL N-10
Error Rate (%)
27.77
15.87
15.98
16.43
17.13
Inference Time (ms/sample)
0.40
0.40
0.40
0.40
0.40
Adaptation Time (ms/ ims/sample)
0.60
8.70
3.50
2.60
1.40
Detailed Analysis of Performance: We provide more detailed analysis against the selection of hyper- parameters. In particular, we investigate the performance of TENT with and without HIL and plot the results in Fig. TENT w/o HIT and TENT w/o HII. (Average) refer to applying TENT without additional
11

---

Published in Transactions on Machine Learning Research (12/2024)
ImageNet-C
ImageNet-D
(c) CIFAR100-C
(d) CIFAR10-C
70.0
Erer 195
Exror 196)
Error 1960
65.0
Error
02.3
50
5e5
10-4
2.5d4
3p-4
1033
5e5
104
2 304
5e-4
36.3
Se5
16-4
2 3.54
Se-4
10.5
Se5
1041
255-4
56-4
303
Learning Rate
Learning Rate
Learning Rate
Learning Rate
TENT w/o HIL
TENT w/o (Average)
TENT wHiL (Entropy)
TENT wHIL Model Sel. (Entropy)
TENT WHIL Sup. Train (Random)
TENT w/iL Mode Sel. (Random)
TENT wHIL (K-Margin)
TENT w/IL Model Sel. (K-Margin)
Figure 4: Performance TENT Wang et al. 2020 with and without HIL. Bold lines represent the per- formance with different hyper-parameter values, while dashed lines indicate the performance with model selection.
annotation and the average error rate respectively. TENT w/HIL Sup. Train (Kaandom/nntooy/KKSS and TENT w/HIL Model Sel. (Random/ (aaaloom//Etttooyy-MMgggii
refer to augmenting TENT with supervised training on Random/Entroge Bataa///ooroooooMrragge /KMMarin labeled lesting data and our proposed model selection respectively. The accuracy w/ HIL consistently surpasses all w/o HIL. Furthermore, random and entropy selection are consistently less effective than our proposed K-Margin selection for model selection purposcs. Crucially, our final HILTTA approach consistently achieves significantly better performance than the worst hyper- parameter, and on ImageNet-C, we even surpass the model using the best-fixed hyper-pa hyper-peraneter.
10"4
Learning Rate
Bi
Error(%)
80
60
Makal
40
10
Gaus.
Shot
Impul.
Defo.
Glass
Motion
Zoom
Snow
Frost
Fag
Brigh.
Contr.
Elast.
Pirel.
lpeg
Gaus.
Shot
Impul.
Defe.
Glass
Motton
Zoom
Snow
Frost
Pag
Brigh.
Centr.
Elast.
Pizel.
lpag
Figure 5: (Left) Average selected learning rate per corruption for TENT ImageNet-C. (Right) Average error rate per corruption for TENT ImageNet-C.
Temporal Evolution Analysis of HILTTA:
parameters over time. As shown in Fig. 5 the We present an analysis of the evolution of selected hyper- selected by our HILTTA model selection method always hover around the upper bound "Fixod Best" while competing methods deviate further away from the upper bound. The advantage is also reflected in the accumulated error plot. In particular, state- of-the-art model selection methods such MixVal (Hu al. 2023), susceptible to catastrophic failure they struggle to adapt to continuously changing domains.
Table 6: Select two hyper-parameters at the same time with PL on CIFARIO-C, error rate is reported.
LR/Threshold
0.1
0.2
0.4
0.6
0.8
AVG. Error
HILTTA (OURS)
le-5
17.17
17.12
17.19
17.22
17.22
le-4
17.08
17.10
17.01
17.13
17.04
1e-3
16.99
16.76
16.54
16.49
16.43
28.96
16.41
le-2
15.89
15.56
20.29
23.69
23.56
lc-1
22.41
81.89
88.64
88.89
88.78
Selecting Two hyper-parameters: In certain cases, more than one hyper-parameler needs to be selected. We evaluated HILTTA combined with PL (Lee et al. 2013) on CIFAR1O-C by selecting two
the self-training threshold and the learning rate. As shown in Tab. t our HILTTA achieved the 3rd best result among 25 candidate more than one typer-parameter.
sets. These results suggest that our method still effective with
12

---

Published in Transactions on Machine Learning Research (12/2024)
Performance Under Different Labeling Budgets: We investigate the effectiveness of HILTTA under different annotation budgets. Specifically, we vary the annotation ratio from 0% to 10%. As shown in Fig. 6 with the increase in annotation budget, we observe consistent improvement for all off-the-shelf TTA methods. Importantly all methods can benefit significantly with only 1% labeled data. This suggests that limited labeled data could substantially ease the challenges TTA.
TENT HIL+TENT
PL
SHOT HIL+SHOT
EATA HIL+EATA
Error Rate [%6]
26
19-
HIL+PL
24
18-
18-
1
20
18
18
16-
15
Labeling Budget (%)
Figure 6: Performance of different TTA methods combined with our proposed HILTTA under different labeling budgets on CIFAR1O-C, average error rates are reported (lower is better).
Performance Under Adjustable Human Intervention Frequency: We investigate the effectiveness of HILTTA under varying human intervention frequencies (N), as shown in Tab. / Instead of requesting human intervention for every batch of test streaming data, reduce the frequency by only requesting intervention for every N-th batch. Consequently, the label rate is reduced to 1/N. Model selection is also performed in every N batch, while for batches without human intervention, the previously selected hyperparameters are used for lest-time adaptation. Remarkably, even when N = 10, corresponding to a mere 0.3% label rale, HILTTA combined with TENT significantly outperforms the original TENT without human intervention, achieving an error rate of 18.61% compared to 27.77%. This demonstrates that even with very limited labeling budget and sparse human intervention, notable performance gains can be achieved. By employing adjustable human intervention frequency, there is need to annotate every batch, further enhancing HILTTA's practical impact.
Table 7: Performance of TENT (Wang et al. frequency (N) on CIFARIO-C.
2020 with HILTTA under different human intervention
Human Intervention Frequency (N) W/O HIL
10
Error Rate (%)
27.77
15.87
16.43
17.15
17.53
17.94
17.99
18.00
18.05
18.39
18.61
5 Conclusion
In this study, we introduce novel approach to human-in-the-loop test-time adaptation (HILTTA) by inte- grating active learning with model selection. Beyond utilizing active learning, we leverage the labeled data as a validation set, facilitating the smooth selection of prpee-parameters and enabling supervised training on a subset of the data. We also developed regularization techniques to enhance the validation loss and proposed new sample selection strategy tailored for HILTTA. By integrating HILTTA with multiple existing TTA methods, we observed consistent improvements due to the synergistic effects of active learning and model selection, outperforming both the worst-case rrper-parameter choices and average performance, well existing active TTA methods. Our findings offer new insights into optimizing limited annotation budgets in TTA tasks.
This research supported by the Agency for Science, Technology and Research (A (A*STAR) under its MTC Programmatic Funds (Grant No. M23L750021), the National Natural Science Foundation of China (NSFC) under Grant 62106078, and Sichuan Science and Technology Program (Project No. 2023NSFSC1421).
Acknowledgement:
13

---

Published in Transactions on Machine Learning Research (12/2024)
References
Eric Arazo, Diego Ortego, Paul Albert, Noel E O'Connor, and Kevin McGuinness. Pseudo- and confirmation bias in deep semi-supervised learning. In Proceedings of 2020 International Joint Conference on Neurul Networks, 2020.
Jordan T Ash, Chicheng Zhang, Akshay Krishnamurthy, John Langford, and Alekh Agarwal. Deep batch active learning by diverse, uncertain gradient lower bounds. In Proceedings of International Conference on Learning Representations, 2020.
Yijin Chen, Xun Xu, Yongyi Su, and Kui Jia. Stfar: Improving object detection robustness test-time by self-training with feature alignment regularization. arXiv preprint vrXiw:2303.19933 2023.
Francesco Croce, Maksym Andriushchenko, Vikash Sehwag, Edoardo Debenedetti, Nicolas Flammarion, Mung Chiang, Prateek Mittal, and Matthias Hein. Robustbench: a standardized adversarial robustness benchmark. In Thirly-fifth Conference on Neural Information Processing Systems Dalasets and Bench- marks Track (Round 2), 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proceedings of IEEE conference on Computer Vision and Pattern Recognition, 2009.
Mario Döbler, Robert A Marsden, and Bin Yang. Robust mean teacher for continual and gradual test-time adaplation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pallern Recognition, 2023.
Yarin Gal and Alex Kendall. Deep bayesian active learning with image data. In Proceedings of the Interna- tional Conference on Machine Learning, 2017.
Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In Proceedings of International Conference on Machine Learning. 2015.
Taesik Gong, Jongheon Jeong, Taewon Kim, Yewon Kim, Jinwoo Shin, and Sung-Ju Lee. NOTE: Ro- bust continual test-time adaptation against temporal correlation. In Proceedings of Advances in Neural Information Processing Systems, 2022.
Edward Grefenstetle, Brandon Amos, Denis Yarats, Phu Mon Htut, Artem Molchanov, Franziska Meier, Douwe Kiela, Kyunghyun Cho, and Soumith Chintala. Generalized inner loop meta-learning. anXiv preprint arXiv:1910.01 01777,, 2019.
Shurui Gui, Xiner Lú, and Shuiwang Ji. Active test-time adaptation: Theoretical analyses and an algorithm.
In Proceedings of International Conference on Leurning Representations, 2024.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, 2016.
Dan Hendryeks and Thomas Dietterich. Benchmarking neural network robustness common corruptions and perturbations. In Proceedings of International Conference on Learning Represendations, 2018.
Dapeng Hu, Jian Liang, Jun Hao Liew, Chubui Xue, Song Bai, and Xinchao Wang. Mixed samples as probes for unsupervised model selection in domain adaptation. In Proceedings of Advances in Neural Information Processing Systems, 2023.
Yusuke Iwasawa and Yutaka Matsuo. Test-time classifier adjustment module for model-agnostic domain generalization. Proceedings of Advances in Newral Information Processing Systems, 2021.
Minguk Jang, Sae- Young Chung, and Hye Won Chung, Test-time adaptation via self-training with nearest neighbor information. In Proceedings of International Conference on Learning Representations, 2023.
Juwon Kang, Naycong Kim, Donghyeon Kwon, Jungseul Ok, and Suha Kwak. Leveraging proxy of training data for test-time adaptation. In Proceedings of International Conference Machine Learning, 2023.
14

---

Published in Transactions on Machine Learning Research (12/2024)
Ronald Kemker, Mare McClure, Angelina Abitino, Tyler Hayes, and Christopher Kanan. Measuring catas- trophic lorgetting in neural networks. In Proceedings of the AATT conference on artificial intelligence, 2018.
Jannik Kossen, Sebastian Farquhar, Yarin Gal, and Tom Rainforth. Active testing: Sample-efficient model evaluation. In Proceedings of International Conference on Machine Learning, 2021.
Jannik Kossen, Scbastian Farquhar, Yarin Gal, and Thomas Rainforth. Active surrogate estimators: An active learning approach to label-efficient model evaluation. In Proceedings of Advances in Neural Infor- mation Processing Systems, 2022.
Dong-Hyun Lee et al. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on challenges in representation learning, ICML, 2013.
Jae-Hong Lee and Joon-Hyuk Chang. Continual momentum filtering on parameter space for online test-time adaptation. In Proceedings of International Conference on Learning Representations, 2024.
Yushu Li, Xun Xu, Yongyi Su, and Kui Jia. On the robustness of open-world lest-time training: Self- training with dynamic prototype expansion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023.
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on Pattern Analysis and Machine Intelligence, 2017.
Jian Liang, Dapeng Hu, and Jiashi Feng. Do really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. In Proceedings of International Conference on Machine Learning, 2020.
Jian Liang, Dapeng Hu, Yunbo Wang, Ran He, and Jiashi Feng. Source data-absent unsupervised domain adaptation through hypothesis transfer and labeling transfer. IEFE Transactions on Pattern Analysis and Machine Intelligence, 2021.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. In Proceedings of International Conference on Learning Representations, 2018.
Yuejiang Liu, Parth Kothari, Bastien Van Delft, Baptiste Bellot-Gurlet, Taylor Mordan, and Alexandro Alahi. Tu++: When does self-supervised lest-time training fail or thrive? In Proceedings of Advances in Newral Information Processing Systems, 2021.
Robert A. Marsden, Mario Döbler, and Bin Yang. Universal test-time adaplation through weight ensem- bling, diversity weighting, and prior correction. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2024a.
Robert A Marsden, Mario Döbler, and Bin Yang. Universal test-time adaptation through weight ensem- bling, diversity weighting, and prior correction. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2024b.
Mitsuru Matsuura and Satoshi Hara. Active model selection: A variance minimization approach. In NewrIPS 2023 Workshop on Adaptive Experimental Design and Active Learning in the Real World, 2023.
Pietro Morerio, Jacopo Cavazza, and Vittorio Murino. Minimal-entropy correlation alignment for unsuper- vised deep domain adaptation. In Proceedings of International Conference Learning Representations, 2018.
Kevin Musgrave, Serge Belongie, and Ser-Nam Lim. Benchmarking validation methods for unsupervised domain adaptation. arXiv preprint arXi::2208.07360, 2022.
Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Yaofo Chen, Shijian Zheng, Peilin Zhao, and Mingkui Tan. Efficient test-time model adaptation without forgetting. In Proceedings of International Conference on Machine Learning, 2022.
15

---

Published in Transactions on Machine Learning Research (12/2024)
Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Zhiquan Wen, Yaofo Chen, Peilin Zhao, and Mingkui Tan. Towards stable test-time adaptation in dynamic wild world. In Proceedings of International Conference on Learning Representations, 2023.
Takuma Ochiai, Keiichiro Seno, Kota Matsui, and Satoshi Hara. Active testing of binary classification model using level set estimation. In NewrIPS 2023 Workshop on Adaplive Experimental Design and Active Learning in the Real World, 2023.
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Procecdings of the IEEE/CVF International Conference on Computer Vision, 2019.
Ori Press, Steffen Schneider, Matthias Kümmerer, and Matthias Bethge. Rdumb: A simple approach that questions our progress in continual test-time adaptation. In Proceedings of Advances in Neural Information Processing Systems, 2023.
Evgenia Rusak, Steffen Schneider, Peter Vincent Gchler, Oliver Bringmann, Wieland Brendel, and Matthias Bethge. Imagenet-d: A new challenging robustness dataset inspired by domain adaptation. In ICML 2022 Shift Happens Workshop, 2022.
A kanksha Saran, Safoora Yonsefi, Akshay Krishnamurthy, John Langford, and Jordan T Ash. Streaming active learning with deep neural networks. In International Conference on Machine Learning, 2023.
Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. In Proceedings of International Conference on Learning Representations, 2018.
Claude Elwood Shannon. A mathematical theory of communication. The Bell system technicul journal, 1948.
Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi-supervised learning with con- sistency and confidence. Advances in neural information processing systems, 33:596-66 (5596668,, 2020.
Yongyi Su, Xun Xu, and Kui Jia. Revisiting realistic test-time training: Sequential inference and adaptation by anchored clustering. Proceedings of Advances in Neural Information Processing Systems, 2022.
Yongyi Su, Xun Xu, and Kui Jia. Towards real-world test-time adaptation: Tri-net self-training with balanced normalization. In Proceedings of the AAAI Conference on Artificial Intelligence, 2024a.
Yongyi Su, Xun Xu, Tianrui Li, and Kui Jia. Revisiting realistic test-time training: Sequential inference and adaptation by anchored clustering regularized self-training. IEEE Transactions on Pallern Analysis and Machine Intelligence, 2024b.
Jiachen Sun, Qingzhao Zhang, Bhavya Kailkhura, Zhiding Yu, Chaowei Xiao, and Z Morley Mao. Benchmarking robustness of 3d point cloud recognition against common corruptions. arXiv preprint arXiv:2201.12200 2022.
Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei Efros, and Moritz Hardt. Test-time training with self-supervision for generalization under distribution shifts. In Proceedings of International Conference on Machine Learning, 2020.
Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. Advances in newral information processing systems, 30, 2017.
Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. In Proceedings of International Conference on Learning Representa- lions, 2020.
16

---

Published in Transactions on Machine Learning Research (12/2024)
Qin Wang, Olga Fink, Lue Van Gool, and Dengxin Dai. Continual test-time domain adaptation. In Pro- ceedingg of the IEEE/CVI Conference Computer Vision and Pottern Recognition, 2022.
Shuai Wang, Daoan Zhang, Zipei Yan, Jianguo Zhang, and Rui Li. Feature alignment and uniformity for test time adaptation. In Proceedings of the IEFE/CVF Conference Computer Vision and Pattern Recognition, 2023a.
Wei Wang, Zhun Zhong, Weijie Wang, Xi Chen, Charles Ling, Boyu Wang, and Nicu Sebe. Dynamically instance-guided adaptation: backward-froe approach for test-time domain adaptive semantic segmenta- tion. In Proceedings of the IEEE/CVF Conference on Compuler Vision and Pallern Recognátion, 2023b.
Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E Sarma, Michael M Bronstein, and Justin M Solomon. Dynamic graph cnn for learning on point clouds. ACM Transactions on Graphics (tog), 2019.
Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, 2015.
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transforma- tions for deep neural networks. In Proceedings of the IEEE conference on Compuler Vision and Pallern Recognition, 2017.
Li Yang and Abdallah Shami. On hyperparameter optimization of machine learning algorithms: Theory and practice. Neurocomputingg 2020.
Shiqi Yang, Yaxing Wang, Joost Van De Weijer, Luis Herranz, and Shangling Jui. Generalized source- free domain adaptation. In Proceedings of the IEEE/CVP International Conference on Computer Vision, 2021.
Longhui Yuan, Binhui Xie, and Shuang Li. Robust test-time adaptation in dynamic scenarios. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In Procedings of the British Machine Vision Conference, 2016.
Hao Zhao, Yuejiang Liu, Alexandre Alahi, and Tao Lin. On pitfalls of test-time adaptation. Proceedings of International Conference on Machine Learning. 2023.
Zhi Zhou, Lan-Zhe Guo, Lin-Har Jia, Dingehu Zhang, and Yu-Feng Li. Ods: test-time adaptation in the presence of woen-world data shift. In Proceedings of International Conference on Machine Learning, 2023.
17

---

Published in Transactions on Machine Learning Research (12/2024)
A Experiment Details
A.1 Detailed Settings for Datasets
Following prior works (Döbler et al. 2023 Wang et al. 2022 Niu et al. 2022 we perform continual test time adaptation in the following datasets, detailed in follows.
CIFAR1O-C Heeddrycks & Dictterich, 2018 is small-scale corruption dataset, with 15 different common corruptions, each containing 10,000 corrupt images of dimension (3, 32, 32) with 10 categories. We evaluate severity level 5 images, with "Gaussian + shot + impulse + defocus + glass + motion + zoom + snow + frost + fog- + brightness + contrast + elastic - pixelate + jpeg' sequence.
CIFAR100-C Heendrycks & Dietterich 2018 is a small-scale corruption dataset, with 15 different common corruptions, each containing 10,000 corrupt images of dimension (3, 32, 32) with 100 categories. We evaluate severity level 5 images, with "Gaussian shot + impulse + defocus + glass + motion zoom + snow frost + fog + brightness contrast - clastic > pixclate jpeg' sequence.
ImageNet-CC (Hendrycks & Dietterich 2018) is large-scale corruption dataset, with 15 different common corruptions, each containing 50,000 corrupt images of dimension (3, 224, 224) with 1000 categories. We evaluate the first 5,000 samples with severity level 5 following Croce et al. (2021). h. The domain sequence is 'Gaussian - shot + impulse - defocus + glass - motion - z0om + snow + frost + fog + brightness + contrast + clastic > pixclate + jpeg".
ImageNet-D (Rusak al. 2022), as large-scale style-transfer dataset, is built upon Domain (Peng et al. 2019), which encompasses 6 domain shifts (clipart, infograph, painting, quickdraw, real, and sketch) The dataset focuses on samples belonging to the 164 classes that overlap with ImageNet. Following Marsden et al. (2024b), we specifically select all classes with one-to-one mapping from DomainNet to Image.Ntt, resulting in a total of 109 classes and excluding the quickdraw domain due to the challenge of attributing many examples to a specific class. After processing, contains domain shifts with each 10,000 images selected in 109 classes. The domain sequence 'clipart + infograph - painting - real sketch'.
ModelNet40-C (Sun et al. 2022), as 3D point cloud corrupted dataset for benchmarking point cloud recognition performance under corruptions, is built upon ModelNet:10 (Wu et al. 2015).. We use the highest corrupted level to evaluate the performance. The domain sequence is 'background + cutout + density + density inc > distortion > distortion rbf distortion rbf inv - gaussian - impulse + lidar - occlusion + rotation - shear > uniform + upsampling'.
A.2 Competing Methods
We mainly follow the official implementation of each method and use the same optimization strategy. But in certain cases, such as transferring from another protocol, we make minor changes to the configuration.
Source serves as the baseline for inference performance without any adaptation.
TENT (Wang et al., 2020). It focuses on updating batch normalization layers through entropy mini- mization. We follow the official implementatior[oor of TENT to update BN parameters.
PL (Lee et al., 2013). We implement self-supervised training unlabeled data updating all parameters by cross-entropy with entropy threshold E = 0 log(#class) For CIFAR10-C and CIFAR100-C, use SGD optimizer with 1e-3 learning rate, while an SGD optimizer with a 2.5e-5 learning rate for the rest.
SHOT (Liang et al.) 2020). It freezes the linear classifier and trains the feature extractor by balancing prediction category distribution, coupled with pseudo-label-basee self-training. We follow its official codo? to update the feature extractor but remove the threshold. For CIFARIO-C and CIFAR100-C, we use an SGD optimizer with a 1e-3 learning rate, while an SGD optimizer with 2.5e-5 learning rate.
18

---

Published in Transactions on Machine Learning Research (12/2024)
EATA (Niu et al., 2022). It adapts batch normalization layers using entropy minimization but intro- duces an additional fisher regularization term to prevent drastic parameter changes. We follow the official implementation
R of EATA to update BN parameters. We hold 2000 fisher training samples and keep the entropy threshold Eg = 0.4 x log(#class).
updates batch normalization layers with a sharpness-aware optimization approach. We mainly follow its official code but set en = 0 for CIFARIO-C and CIFAR100-C, preventing failure cases of frequently resetting the model's weight.
SAR (Niu et al., 2023):
RMT (Döbler et al., 2023). It adapts in teacher-student mode, incorporating cross-entropy consistency loss and contrastive loss. We follow the official implementation
ASE Kossen et al., 2022 uses surrogate estimators to sample in a uncertainty distribution to sample with low bias. We follow official implementation 6
SIMATTA (Gui et al., 2024). As an active test-time adaptation method, it selects labeled samples through incremental clustering and updates the model by minimizing cross-entropy. To reduce the compu- tational and memory cost of incremental clustering, experiments. For CIFAR10-C and CIFAR100-C, we set the lower entropy bound to 0.005 and 0.01 for the higher entropy bound eu- While for ImageNet-C and ImageNet-D,, we set the lower entropy bound ej to 0.2 and 0.4 for the higher entropy bound ex.
keep the maximum length of anchors to 50 in all
VeSSAL (Saran et al., 2023). As a stream- grream-based active learning method, it uses volume sampling for streaming active learning. We also follow its official implementation using feature embedding for stream- based data selection.
A.3 Hyper- papeerppraaeter Candidate set for Model Selection
Following Musgrave et al. 2022 Hu al. 20223, in order to evaluate the performance of our method on different types of hyper-parameters, we construct candidate sets Ω with seven distinct values for cach typer-parameter category related to model selection. We assess different byper-parameters for each TTA method: for PL and FATA, the self-training threshold; for TENT and SAR, the learning rate; and for SHOT and RMT, the loss coefficient factor. To ensure generality, we use the same l across diverse datasets, as summarized in Tab. g
Table 8: Overview of the TTA methods validated and their associated hyper- aherrppaammters Search Space
TTA method
Hlyper-parameter
Learning Rate
TENT
Wang et al.
2020
{5e-5, 1e-4, 2.5e-4, 5e-4, 1e-3, 2.5e-3, 5e-3]
"
Entropy Threshold
PL
Lee et al.
2013)
(0.05, 0.1, 0.2, 0.4, 0.6, 0.8, 1.0)
Loss Coefficient
SHOT
Liang al.
2020)
(0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0)
8
Learning Rate
EATA
Niu et al.
2022
{5e-5, 1e-4, 2.5e-4, 5e-4, 1e-3, 2.5e-3, 5e-3)
?
Learning Rate
SAR
Niu et al.
2023
{5e-5, 1e-4, 2.5e-4, 5e-4, 1e-3, 2.5e-3, 5e-3]
?
Loss Coefficient
RMT Döbler et al.
2023
(0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0)
ACE
6 httne//sihhhblooom/////rrsssssst
7 https////chhbbbommmmsmm///ASa
19

---

Published in Transactions on Machine Learning Research (12/2024)
B Additional Experimental Analysis
B.1 Ablation Study on the Impact of |
We evaluate the impact of the moving average momentum parameter, B, in Eq when applying TENT (Wang ct al. 2020) combined with our proposed HILTTA. The experiments were conducted across four datasets: ImageNet-Co ImageNet-DD, CIFAR100-C, and CIFARIO-C. We varied 8 over wide range, from 0.1 0.9, and observed in Fig. 7 that the performance remained robust throughout. This demonstrates that proposed HILTTA method is not sensitive to the choice of 3 values, maintaining stability across different settings.
60
50
Error Rate(%%)
40
mageNee
30
0.5
B
Figure 7: Performance of TENT (Wang et al. 2020 combined with our proposed HILTTA under different B on 4 different datasets.
B.2 Comparison to Bi-level Optimization
In some scenarios, the cost of collecting annotated data can outweigh the computational expense, making hyperparameter optimization via bi-level optimization (Liu al., 2018) viable approach. However, bi-level optimization has limitations, particularly its sensitivity to the meta-learning rate (the learning rate in the outer loop). We compare our method, HILTTA, with the approach proposed in (Liu el al. 2018). using Adam as the meta-optimizer. The Higher package (Grefenstette et al. 2019) is adopted for meta optimization. As shown in Fig while the best performance achioved by bi-level optimization is comparable to ours (59.34% vs. 58.35% етrоr rate), bi-level methods can be prone to collapse if the meta-learning rate is not carefully tuned. Additionally, bi-level optimization is unable to optimize hyperparameters that are non-differentiable.
TENT
Bi-level Optimization+
90
HIL+TENT
Eiror Rate(%)
80
70
16-6
56.6
16.5
5e-5
1c-4
5e4
1o3
Meta Learning Rate
Figure 8: Performance of TENT WWagg al. 2020 combined with our proposed HILTTA and bi-level
optimization on the ImageNet-C dataset.
20

---

Published in Transactions on Machine Learning Research (12/2024)
B.3 Comparison to Semi-supervised Learning Methods
To evaluate the effectiveness of our proposed HILTTA, compare it against semi-supervised methods, such as FixMatch (Sohn et al., 2020) and Mean'Teacher (Tarvainen & Valpola 2017), both using the same label rate of 3%. Additionally, we include ATTA methods like SimATTA (Gui et al. 2024) and VeSSAL (Saran et al. 2023), adapted to the ATTA setting by integrating them with TENT (Wang et al. 2020 also with label rate of 3%. As shown in Tab. 9 our HILTTA THNT++TENT) surpasses all other TTA, semi-supervised learning, and ATTA approaches. Semi-supervised methods typically require longer training times and multiple epochs to achieve convergence, making them less effective in stream-based tasks. It is worth noting that while both semi-supervised and ATTA methods rely manually tuned hyperparameters, HILTTA achieves superior performance with model selection.
Table 9: HILTTA Comparison with Semi-supervised Learning and Active TTA methods.
Setting
TTA
Semi-supervised Learning
ATTA
(ULTTA(Ous))
Method
TENT
FixMatch
MeanTTacher
SimATTA
VeSSAL
HIL + TENT
Error Rate(%)
27.77
17.03
16.59
17.66
17.47
15.87
Although semi-supervised learning and ATTA methods can achieve performance comparable to HILTTA, they suffer from signifieant sensitivity to hyperparameters. As shown in Fig. g we compare the performance of TENT, HIL+TENT, and FixMatch on the CIFAR10-C dataset with learning rate as the variable hyper- parameter. The results indicate that semi-supervised methods such as FixMatch are particularly sensitive changes in learning rate, highlighting the challenges of tuning these methods effectively. Our proposed HILTTA is not only aimed at improving performance but, more importantly, addressing hyperparameter sensitivity issues by synergizing active learning with model selection.
TENT
FixMatch
30
HIL:TENT
Error Rate 196)
5c-5
10-4
2.5e-4
50-4
10:3
2.50-3
56-3
Learning Rate (LR)
Figure 9: Performance of Fixmatch (ohn et al. 2020 and TENT (Wang al. 2020 combined with
proposed HILTTA the CIFAR10-C dataset.
B.4 Studies on the Role of Regularization in HILTTA
We evaluate the impact of the proposed anchor regularization and EMA smoothing techniques for model selection within the HILTTA framework. These methods designed to prevent the model from overfitting to the local data distribution in the testing data stream. While removing these regularizations may improve adaptation to single domain, it negatively impacts performance across future domains. As shown in Tab, IO in the continual TTA selting, model selection without regularization results in significantly worse performance compared to when regularization is applied (17.25% vs. 15.87%). This indicates that regularization is crucial for maintaining robust performance continual TTA. model selection without regularization leads significantly worse results (17.25% vs. 15.87%), highlighting the necessity of regularization for robust performance in continual TTA.
21

---

Published in Transactions on Machine Learning Research (12/2024)
Table 10: Comparison of HILTTA with TENT on CIFARIO-C, showing error rates with and without regu- larization.
Time
brigläness
Regulatization
slo8
Iroplte
dedoent
glees
untim
Irre
[og.
contrant
dasti
pkodate
Jpeg
Avg.
Without
25.04
20.21
28.57
12.22
29.67
13.56
11.45
15:57
14.59
13.64
8.26
11.29
20.44
14.96
19.37
17:25
With
25.93
20.13
27.49
11.59
27.69
12.48
9.54
14.04
13.01
12.06
6.59
8.52
18.21
12.46
18.26
15.87
B.5 Comparing with Knowledge Distillation and Ensemble Learning
We carried out additional experiments by comparing with ensemble learning and multi-teacher knowledge distillation methods. Specifically, for ensemble learning, we average the posterior predicted by each candidate model and make prediction with the average posterior. For multi-teacher knowledge distillation, the pseudo label predicted by the ensemble model for self-training. As observed from Tab. [T both ensemble learning and multi-teacher knowlodge distillation yiclds inferior rosults than HILTTA. This is most likely caused by incorporating poor candidate models (ensemble learning) and using poor pseudo labels for self- training, i.c. learning from noisy labeled data (multi-tcacher knowledge distillation).
use
Table 11: Comparison with Ensemble Learning and Knowledge Distillation with TENT on CIFARIO-C.
Method Average Performance on all candidate model Eusemble Learning
Muli teacher Knowledge Distillation
HILTTA (OURS)
Error
27.77
19.48
26.42
15.87
B.6 Comparing Adaptation Time & Annotation Time
We carried out empirical studies by measuring the wall-clock time lapses of the inference step (Inference Time), adaptation step (Adaptation Time), and human annotation (Annotation Time) per sample. Tn specific, the "Inference Time" refers to the time-lapse of pure forward pass, the "Adaption Time' refers to the time-lapse of backpropagation and weights update, and 'Annotation Time" refers to the estimated time lapse required for human to annotate images at a 3% labeling rate. We evaluate under three categories of annotation cost, i.c. 2 sec/sample, 5 sec/sample, and 10 sec/sample. As seen from Tab. 12, despite the adaptation time being higher than the inference time (8.7 ms v.s. 0.4 ms), the annotation time the most expensive step. These results suggest model selection and adaptation do not slow down the throughput under the HILTTA protocol.
Table 12: Comparing HILTTA and annotation time consuming with a 3% label rate on CIFARIO-C
Inference
Adaptation
Annotation (2 sec/sample)
Annotation (5 sec/sample) Aunotation (10 sec/sample)
Time (is/ /sample
0.40
8,70
60.00
150.00
300,00
B.7 Computation Complexity of Competing Active TTA Methods
We find that existing stream- based active learning (VeSSAL (Kossen et al. 2022 and active TTA (SimATTA
Gui al. 2024)) are inherently more computationally expensive than HILTTA due to the following reasons. First, VeSSAL adopts gradient embedding, i.e. g(æ;) = (x;f(x;;; 0), v:) 00,, where Or, E IR KRD refers to the classifier weights. This results in very high dimensional gradient embedding g( E R RDD. The probability of labeling sample pj involves calculating the inverse of covariance matrix, i.e. pt g(x,) ((2,)T1-1 ) s.t.
Ex,B (xx)) E DD Hence, VeSSAL can hardly deal with classification tasks with a large number of classes, e.g. ImageNel, where K is large and calculating a large matrix inverse is inherently lime-consuning, Second, SimATTA employs a K-Means clustering algorithm for the selection of samples to annotate. Since K-Means an itcrative approach the computation cost of SimATTA is relatively high as well. In contrast, our proposed sample selection is built upon K- Center clustering which sequentially selects the sample with the highest distance to cluster centers (greedy algorithm), the computation cost is significantly lower. An
22

---

Published in Transactions on Machine Learning Research (12/2024)
empirical study in Tab. 13 reveals that our method is about 3 times faster than VeSSAL and SimATTA in adaptation time.
Table 13: Error rate and adaptation wall time comparing to different methods CIEAR1O-C dataset
Methods
Frror
Adaptation Time (ms/sample)
VeSSAL
24.41
26.90
SimATTA
17.66
22.50
HILTTA (OURS)
15.87
8.70
B.8 Studies on Random Batchsizes
We recognize that cerlain scenarios, test data may arrive inconsistently and intermittently. Results from Tab. 14 demonstrate that our proposed HILTTA framework is adaptable to such varying batchsize. At each iteration, vary the batchsize randomly among the choices [50, 100, 150, 200]. This versatility leads to even greater improvements compared to fixed batch size. This enhanced performance stems from the fact that the optimal hyper-parameter values vary with different batch sizes, thus benefiting significantly from our adaptive model selection approach.
Table 14: Adaptation error with varions batchsize with TENT on CIFAR10-C.
Batchsize
w/o HIL
HILTTA
200
27.77
15.87 (+11.90)
Random choose from 10,1000150,,00)
35.89
16.39 (+19955)
B.9
Further Ablation Study with RMT
We have conducted wider experiments to analyze the ablation study with RMT (öbler et al. 2023 as the off-the-shelf TTA method in Tab. cxhiibiss a consistent trend with the findings presented in the manuscript, providing additional support for our analysis and demonstrating the effectiveness of proposed approach.
Table 15: Ablation study of HILTTA. Average classification error is reported for combing with RMT.
Unsup
CE
Auchor
EMA
Super. Train.
Error (%) 4
Remark
Train.
Valid.
regular.
Smooth.
ImageNNt-CC
ImageNet-D
CIFAR100-C CIFARIO-C
Source
82.02
58.98
46.44
43.51
RMT
72.28
58.92
32.18
19.75
w/ HIL
98.34
49.98
36.28
21.94
W/ HIL
51.40
31.53
HII.
v
61.33
50.23
30.75
17.33
HIL.
/
/
58.26
46.28
29.41
16.45
23


Reviews:
[{"review_id": "LMjRHkaKey", "reviewer": "Reviewer_6jpG", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strength\n- The proposed approach is novel and sounds reasonable.\n- Extensive evaluations including ablation study are provided. They validate the effectiveness of the proposed algorithm.\n- This paper is well-organized and explains the motivation and the idea clearly.\n\n### Weakness\n- Lack of theoretical guarantees/analyses on the proposed approach.\n- Lack of some validations on the proposed approach (see requested changes).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a novel human-in-the-loop test-time adaptation (HILTTA) approach. The proposed approach tackles one of the pitfalls of TTA methods, i.e., their sensitivity to the selection of hyperparameters, by integrating active learning into model selection. The proposed algorithm is simple: (1) collecting a small validation set (active learning), (2) running unsupervised model adaptation on a hyper-parameter set, (3) selecting a model, and finally (4) learning the selected model in a supervised manner. The technical contributions are (i) proposing this procedure, (ii) proposing a method for active learning, and (iii) proposing a metric for model selection. The paper also provides extensive evaluations on real-world datasets, which validate the effectiveness of the proposed algorithm.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "### Major  Comment\n- **Validating of the robustness to the selection of $\\beta$.** The proposed method has a hyperparameter, $\\beta$, in the EMA smoothing (Eq. (6)). Table 2 reports the importance of the EMA smoothing on the CIFAR100-C dataset. Since the proposed method tackles the sensitivity issue to the selection of the model hyperparameters and the EMA smoothing can be important on some datasets, the authors should validate the robustness of the proposed method to the selection of $\\beta$.\n\n- **Clarifying the problem setting or adding experimental results for gradient-based methods solving Eq. (1).** In real-world applications, collecting labels even on a small validation set might be too *time-consuming*. It is not realistic to expect human-experts to start annotating a requested validation set immediately when we ask them to annotate a small validation set. So, the cost of collecting annotated data can dominate the computational time in the whole process. Then, it can be *feasible* to use gradient-based methods (e.g., Liu et. al, 2018) to solve the problem Eq. (1). I think that the authors should add experimental results for gradient-based methods solving Eq. (1) or clarify the problem setting and motivation in this view point.", "broader_impact_concerns": "I don't have the broader impact concerns on this paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "8STxBSamGS", "reviewer": "Reviewer_bwks", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. The paper is well-structured, with a clear flow from problem definition to proposed solution and experiments.\n2. TTA is a critical area for real-world applications. \n3. Addressing hyper-parameter sensitivity to enhance the robustness of TTA methods in dynamic environments is interesting.\n4. The empirical results are comprehensive, demonstrating significant improvements over baselines across various corrupted datasets.\n\nWeaknesses:\n\n1. The theoretical aspects of the method were not discussed sufficiently.\n2. The distinction of the HILTTA and ATTA is the introduction of model selection processes, which is clearly stated in the paper, but the names themselves are not distinguishable enough. Furthermore, as what I observed, settings are mixed together, e.g., VeSSAL is adapted to TTA and denoted as ATTA. That may confuse or potentially mislead readers.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes Human-in-the-Loop Test-Time Adaptation (HILTTA) by integrating active learning and model selection to address sensitivity to hyper-parameters in existing test-time adaptation (TTA) methods. The framework involves selecting samples for human annotation and using labeled data for optimal hyper-parameter selection. Regularization techniques are introduced to prevent overfitting to local distributions during model selection, and a tailored sample selection strategy balances active learning and model selection objectives. Experiments on five TTA datasets demonstrate that HILTTA significantly improves performance compared to state-of-the-art TTA methods, mitigating the impact of poor hyper-parameter choices.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Theoretical discussions are required.\n2. A setting comparison table regarding the differences between settings Streaming active learning, TTA, ATTA, HILTTA, etc. is required.", "broader_impact_concerns": "Not observed.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "mMQ9XbYL3o", "reviewer": "Reviewer_VRqh", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n- The proposed method is reasonable.\n- The proposed method shows good performance under many situations.\n- The experiments are comprehensive. Many ablation studies have been conducted.\n\n## Weaknesses\n\n- The proposed method requires human intervention. Popular unsupervised TTA methods are easier to use and more attractive to practitioners than the proposed method.\n- It's not surprising that (semi) supervised methods outperform unsupervised methods. It would be better to compare the proposed method with more supervised methods (such as the ones in the online learning literature.)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes HILTTA, which combines Test-Time Adaptation with Human-in-the-Loop to stably select good hyperparameters and improve accuracy. The proposed method consistently outperforms various TTA-related approaches under ImageNet variant domain shifts.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- It would be better to elaborate the appeal message of the proposed method beyond accuracy and/or propose some countermeasures to alleviate the burden of human intervention.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The effectiveness of the proposed method has been studied through experiments on multiple datasets, and ablation studies show that all components of the proposed approach contribute effectively to its performance.
The authors appropriately addressed concerns raised by each reviewer, such as sensitivity to $\beta$ and the proportion of human intervention.
As a result, all reviewers agreed to accept the paper.
Decision: Accept (poster)