Ranked Entropy Minimization for Continual Test-Time Adaptation
Authors: Jisu Han
Jaemin Na
Wonjun Hwang
Venue: ICML 2025 poster (2025)
Keywords: continual test time adaptation
entropy minimization
masked image modeling
Primary area: deep_learning->robustness
TL;DR: We propose an efficient solution for continual test-time adaptation based on ranked entropy minimization.

Abstract: Test-time adaptation aims to adapt to realistic environments in an online manner by learning during test time.  Entropy minimization has emerged as a principal strategy for test-time adaptation due to its efficiency and adaptability. Nevertheless, it remains underexplored in continual test-time adaptation, where stability is more important. We observe that the entropy minimization method often suffers from model collapse, where the model converges to predicting a single class for all images due to a trivial solution. We propose ranked entropy minimization to mitigate the stability problem of the entropy minimization method and extend its applicability to continuous scenarios. Our approach explicitly structures the prediction difficulty through a progressive masking strategy. Specifically, it gradually aligns the model's probability distributions across different levels of prediction difficulty while preserving the rank order of entropy. The proposed method is extensively evaluated across various benchmarks, demonstrating its effectiveness through empirical results.

Ranked Entropy Minimization for Continual Test-Time Adaptation
Jisu Han Jaemin Na?
Wonjun Hwang .
Abstract
Test-time adaptation aims to adapt to realistic en- vironments in an online manner by learning dur- ing test time. Entropy minimization has emerged principal strategy for test-time adaptation due to its efficiency and adaptability. Neverthe- less, it remains underexplored in continual test- time adaptation, where stability is more impor- tant. We observe that the entropy minimization method often suffers from model collapse, where the model converges to predicting a single class for all images due to trivial solution. We pro- pose ranked entropy minimization to mitigate the stability problem of the entropy minimiza- tion method and extend its applicability to con- tinuous scenarios. Our approach explicitly struc- tures the prediction difficulty through a progres- sive masking strategy. Specifically, gradually aligns the model's probability distributions across different levels of prediction difficulty while pre- serving the rank order of entropy. The pro- posed method is extensively evaluated across var- ious benchmarks, demonstrating its effectiveness through empirical results. Our code is available at https::: : / gighuu...uteee. com/ /p/ 1 Han rem
Ooject
Moaking
(Original) Target Image
(10% Masked) Target Image
(20% Maskid Target Image
Accuracy
Masked Prediction
Prediction
Entropy
Original
10%
Masked Prediction
Prediction
Figure I. Our Intuition. We explicitly raise the prediction diffi- culty of the input images through the masking strategy. Based on the intuition that increased difficulty decreases prediction accuracy and increases entropy, we attempt maintain a rank ordering of entropy while improving consistency from original to masked pre- dictions. Our approach addresses problem of model collapse in entropy minimization methods in simple yet efficient way.
Continual test-time adaptation (CTTA) (Wang al., 2022) addresses the issuc of error accumulation in long-sequence domains. It mitigates the forgetting problem under contin- uous environmental changes and sequentially adapts to a stream data, facilitating the practical deployment of TTA. Recent studies on CTTA can be broadly categorized into two major approaches: entropy minimization (EM) (Wang et al., 2021; Niu et al., 2022; Zhang et al., 2025) and consistency regularization (CR) (Wang et al., 2022; Liu et al., 2024b:a). The EM approach minimizes the entropy of predictions and offers computational efficiency but suffers from instability due to the risk of trivial solutions, where predictions collapse into single class. In contrast, the CR approach employs a teacher-student framework (Tarvainen & Valpola, 2017), up- dating the model conservatively ensure stability but incurs high computational costs. As result, there is trade-off between efficiency and stability in these approaches.
1. Introduction
The real world is non-i.i.d., which demands real-time adap- tation of AI applications. Deep learning models have achieved remarkable progress in recent years; however, performance degradation caused by distribution shifts be- tween different domains limits the generalization capabili- ties (Shimodaira, 2000). Test-time adaptation (TTA) (Wang et al., 2021) has emerged as practical approach to address non-stationary environmental changes by enabling models trained on source domain to adapt an online manner unlabeled target data during the test-time.
I Ajou University KKorea Telecom 3 KKorea University. Corre- spondence to: Wonjun Hwang <wjhwang @ kor
Motivation. Our approach stems from the intuitive observa- tion that a model's predictions for an explicitly information- degraded image are inaccurate and have high entropy com- pared to its predictions for complete image (as illustrated
Proceedings of the nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Ranked Entropy Minimization for Continual Test-Time Adaptation
Tent (Gauss. T₁) Tent (Shot
Tent (Impulse Ta)
Tent (JPEG T1s)
100
90
Model
Tent(Ir=ee-4 1e-4)
Collapse
Tent(Ir=1e-3)
Model Collapse
Error (%)
REM(Ir-1e4)
70-
REM(Ir=1e-)
60
50-
REM (Gauss. T1]
REM (Shot Tx)
REM (Impulse T2)
REM (JPEG Tis)
40-
30
20
Shot
Shot
Impulse
bel
Motion
Zoom
Snow Frost Fog Bright contlagi Pixel
Pixel jREd
Adaptation Order (-)
Adaptation Order
(a) Continual Adaptation
(b) Prediction Diversity
Figure 2. Observation on model collapse in the entropy minimization approach. (a) Under the CTTA scenario, the EM approach (Tent) undergoes significant performance degradation critical point (adaptation order Ts, Impulse noise). (b) This phenomenon occurs because the model learns constant representations that do not depend on input images, leading to a collapse in prediction diversity. This is evidenced by class probabilities converging to single point when visualized a polar coordinate system. Our proposed method (REM) mitigates model collapse and maintains prediction diversity.
in Figure 1). Motivated by Zeno's Achilles and the tor- toise paradox (Huggett, 2002), this paper aims to prevent abrupt entropy reduction by gradually advancing the orig- inal predictions while reducing the gaps in explicit rank relationships. Here, an original prediction is represented by the tortoise, maintaining its leading position over masked prediction in the ranking relationship and learning through rank regularization. In contrast, masked prediction is analo- gous to Achilles, aiming to catch up with original prediction by learning through consistency regularization.
tendencies into ranked predictable one. For taking advan- tage of the ranked structure, we provide two interrelated methods. (1) First, we apply consistency loss by ensuring that predictions with higher masking ratio are similar to those with a lower masking ratio, thereby not only indirectly reducing prediction entropy but also enabling the model to learn contextual information from masked regions. (2) Second, we introduce ranking loss that ensures the entropy of predictions with a lower masking ratio remains lower than that of predictions with higher masking ratio. This ap- proach not only models uncertainty but also reduces entropy by incorporating object-specific information, preventing pre- dictions from being biased toward background information. The main benefit of our method lies in achieving the joint goals of stability and adaptability while maintaining the ef- ficiency of EM approaches with single model and without requiring additional models.
Observation. Figure 2 shows the phenomenon of model collapse in EM. Model collapse means that the EM approach converges to a prediction for constant class by trivial solution, resulting in the model produce a nonsensical prediction. This issue corresponds mode collapse in GAN (Goodfellow et al., 2014) and complete collapse (Hua et al., 2021) in self-supervised learning (SSL) within the CTTA. Model collapse occurs because the EM objective function minimized even when the model consistently predicts single class, regardless of the input. Since EM lacks stability to address the forgetting problem, it maintains performance by using low learning rate. However, this leads to limitations in both adaptability and robustness.
2. Related Work
2.1. Test-Time&& Adaptation
The concept of optimizing during test time to adapt to target domains was proposed in Test-Time Training (TTT) (Sun et al., 2020). However, TTT relies the use of source data and training loss, which may not generalize well to practi- cal applications. To address these limitations, Tent (Wang et al., 2021) proposed the concept of Fully TTA, which enables online adaptation to unlabeled target data without requiring access to source data. In this context, TTA fo- cuses on efficiency, typically by updating the normalization layers (Gong et al., 2022) adjusting predictions without training the model parameters to improve computational efficiency. (Iwasawa & Matsuo, 2021; Boudiaf et al., 2022).
Based on this observation, we propose a Ranked Entropy Minimization (REM). Specifically, we exploit the self- attention structure of ViT (Dosovitskiy et al., 2021) to mask patches with a high likelihood of containing objects (Bolya al., 2023; Son et al., 2024). The principal idea is to ex- plicitly enhance the prediction complexity of a sample by masking objects that domain invariant features. Building mask chain that sequentially obscures more patches based on the masking ratio transforms unpredictable prediction
2

---

Ranked Entropy Minimization for Continual Test-Time Adaptation
Entropy minimization approaches (Wang et al., 2021; Niu et al., 2022; 2023; Lee et al., 2024; Zhang et al., 2025) are evolving as primary solution for TTA. Among those, EATA (Niu et al., 2022) introduces sample filtering for un- certain predictions and regularization method to enhance stability. Furthermore, SAR (Niu et al., 2023) observes the phenomenon of model collapse by trivial solutions, and proposes adaptation to flat minima and filtering of large gra- dient samples. Building on these approaches, Ye (Lee et al., 2024) presents a criteria for additional sample selec- tion through image rearrangement, and COME (Zhang et al., 2025) proposes a conservative entropy minimization method to address the overconfidence problem.
3. Ranked Entropy Minimization
In this section, we introduce the CTTA setup and the trivial solution from the EM, then explain the proposed REM. Our method consists of mask chaining strategy and two loss functions for CR and EM. Masked consistency loss ensures consistency from predictions with a lower masking ratio to those with a higher masking ratio, while entropy ranking loss enforces ranking constraint from predictions with a higher masking ratio to those with a lower masking ratio.
3.1. Preliminaries
In CTTA setup, a target model ft is trained in an online manner, starting from a source model fs. The model adapts to sequential target domain data x; where each domain is represented in order by T E {1, 2,,,,... J. The evaluation protocol involves calculating the cumulative error based on the model's predictions p7 f((= while the model adapts to the target domains during test time using xi. The main difference between TTA and CTTA lies in whether the model is reset to the source model when a domain changes. In CTTA, the model is not reset, which makes the issue of catastrophic forgetting more significant.
2.2. Continual Test-Time Adaptation
Advancing from single-domain adaptation, CTTA (Wang et al., 2022; Brahma & Rai, 2023; Döbler et al., 2023; Liu et al., 2024b;a) emphasizes the requirement for adapt- ing to continuous domain shifts, thereby extending the practicality of TTA. This paradigm promotes rethinking the conventional protocol by encouraging deeper focus on stability. COTTA (Wang et al., 2022) addresses catas- trophic forgetting by introducing a consisteney loss be- tween the base model and weight-averaged model, while employing stochastic restoration of parameters based on the source model. PETAL (Brahma Rai, 2023) intro- duces a probabilistic framework for CTTA and a parameter restoration method leveraging the Fisher Information Ma- trix. ViDA (Liu et al., 2024b) presents a trade-off between stability and plasticity by designing an adapter that explic- itly separates invariannnv features and domain-specific features. Comtinual-MAE (Liu et al., 2024a) measures pixel uncertainty through Monte Carlo (MC) dropout to distin- guish object presence and enhances the representation of domain-invariant properties using masked autoencoder.
The EM method applies gradient descent using entropy S(r,) - -- c=11 Pt.cc Pr.e over the total number of classes C' as the objective function. Consequently, the gra- dient of ((r) with respect to the parameter is as follows:
aS())) 00,
Σ 1=1
C
(log Pr, + 1) Dr,co11 - -e,c) d2t.c
(1)
00,
where 2t, represents the logit for class e before applying
exp(2t.e) the softmax function, i.c., Pr.c E exp(ar,e) The trivial
30, = O arises under the following cases: (Case 1) Uniformly distributed probability (fr,c () ) and (Case 2) Perfectly confident predictions (fr.c E {0, 1J). In this case, since the initial source model's prediction distri- bution is not uniform, the likelihood of achieving a trivial solution due to a uniform distribution under entropy mini- mization is low. However, it is possible for singular class to result in perfectly confident predictions, which is experi- mentally observed in Figure 2.
solution for aS(p.)
Beyond not Forgetting. The recent state-of-the-ar CTTA methods achieve stability and mitigate performance degra- dation by adopting teacher-student frameworks or param- eter restoration from the source model. However, these approaches often neglect efficiency constraints, leading to excessive computational costs and memory overhead. This inefficiency contradicts the goal of TTA, which is to enable real-time adaptation under resource constraints. To address this issue and realign the framework with its intended pur- pose, protocols that account for computational time con- straints (Alfarra et al., 2024) and label delays (Csaba et al., 2024) have been proposed. We argue that efficiency in CTTA requires renewed attention and propose leveraging insights from pioneering approaches in TTA to improve both stability and computational efficiency. In this study, we integrate the strengths of EM and CR approaches to bal- ance efficiency and stability. Detailed efficiency analysis is provided in Section 4.4.
3.2. Explicit Mask Chaining
Conventional CTTA methods point out that augmentation policies may not be valid for dramatic domain shifts and decide whether to apply augmentation through prediction confidence (Wang et al., 2022). However, these methods fail to model the impact of augmentation policies on model pre- dictions, resulting in inefficiencies caused by the application of diverse augmentations. In order to model the predictions, we propose Explicit Mask Chaining, which incrementally
3

---

Ranked Entropy Minimization for Continual Test-Time Adaptation
Noise (Gaussian)
Blur (Defocus)
accuracy.
We define masked consistency loss (MCL) as follows:
Entrayy
95
0.20
Emor (%)
0.08
MN
0.08
70
LMCL Σ H(f(()))ss((A())))
(3)
60
50
105
20
60
100
Mask Ratio (16)
Mask Ratio (1)
Weather (Snow)
Digital (Jpeg)
where H(p, q) denotes the cross-entropy between two prob- ability distributions p and q. MN = {0, my, :,,...: mn is a set of mask ratios, where N is the number of mask chains, and sg denotes the stop-gradient operation.
100
Oner Entragy
0.12
Emer
Entrayy
0.12
A.18
Entropy
Error (%)
80
0.30
entropy
0.08
bdde
60
0.00
20
60
20
100
20
40
90
100
Comparison with EM. Unlike EM, which uses entropy as a loss function, MCL indirectly reduces prediction entropy by using the cross-entropy between masked predictions and either less- esss-maseed or unmasked predictions. This is de- signed to mitigate abrupt entropy changes and overconfident predictions, leading to the alleviation of model collapse.
Mask Ratio (561
Mask Ratio (9))
Figure 3. Empirical study according to masking ratio, We report the changes in error and entropy the masking ratio increases. Both entropy and error exhibit monotone increasing trend with respect the masking ratio, and observe that linearity becomes more pronounced, especially in regions with lower masking ratios.
Comparison with CR. To generate stable predictions dis- tinct from the target model, the CR approach requires ad- ditional teacher models and numerous forward passes for uncertainty estimation. In contrast, our method eliminates the need for uncertainty prediction by leveraging an explicit ranking structure and improves efficiency by generating diverse predictions via mask chains within a single model.
masks content containing domain-invarisn information for augmentation. To this end, we focus on the self-ateention structure of VIT for efficient object masking. Exploiting the self-attention structure allows us to cluster content by simi- larity between intermediate tokens (Bolya al., 2023), well as compute attention score for attention head (Son et al., 2024). Following pioneering research, we define an attention score A as follows:
3.4. Entropy Ranking Loss
Through the reduction of differences in ranked prediction distributions, MCL is designed to indirectly minimize pre- diction entropy. However, this may result in slower adap- tation due to small differences in prediction distributions or lead to biased predictions toward the background when learning from images with occluded objects. To comple- ment MCL and address these issues, we propose the entropy ranking loss (ERL) as follows:
Qhictelhhimmm Vd
A =
Softmax
where is the number of heads in the multi-head attention, Qh,els and Kn.ang are query for class token and key for image token of the h-th attention head, respectively. d denotes the dimension of each attention head.
The masked image (Imm Tmgg'.. ..yy J. defined for the top-m proportion of set A sorted in descending order, has mask ratios that satisfy the condition 0 < mi S m2 S I mmN- I 1. Figure shows the error and entropy across range of masking ratios for noise, blur, weather, and digital corruptions on ImageNetC (Hendrycks & Diet- terich, 2018). The results empirically confirm that explicit masking of objects results in lower accuracy and higher entropy as the masking ratio increases. In conclusion, we establish an explicit ranking relationship between entropy and accuracy by employing an incremental masking strategy for consecutive mask ratio.
MN max (0, sg(S(((rr))) -g,S((())))s(((((((())) + m),
LERI Σ
(4)
where is the margin. The purpose of ERL is to main- tain the principle that the entropy of predictions with a low masking ratio for objects should be lower than that of predictions with a high masking ratio. By maintaining an explicit ranked order, we prevent overconfidence in high masking ratio predictions, which could lead bias toward the background. This follows from prior findings that apply- ing ranking losses in neural calibration effectively mitigates overconfidence (Moon et al., 2020; Noh et al., 2023). Addi- tionally, ERL directly reduces the entropy of samples that violate the ranked order, thereby promoting faster adapta- tion and maintaining a structured entropy hierarchy across different masking ratios.
3.3. Masked Consistency Loss
Given an explicitly ranked prediction, we aim to make a high masking rate prediction with relatively low accuracy similar to a low mask rate prediction with relatively high
4

---

Ranked Entropy Minimization for Continual Test-Time Adaptation
Table Classification error (%) for ImageNet-to-lmagNNerc under CTTA scenario. Mean (%) denotes the average error across
15 target domains. Gain (%) represents the relative performance improvement compared to the source model.
Time
Method
Gaussian
shot
impulse
deforus
glass
mation
zokm
SHOW
Frost
feg
trightness
contrast
elastic trans
pixclate
ineg
Mean] Gain
Source (Dusovitskiy et al., 2021)
53.0
51.8
52.1
68.5
78.8
58.5
63.3
49.9
54.2
57.7
26.4
91.4
57,5
38.0
36,2
55.8
0.0
Pseudo--abee label (Lee, 2013)
45.2
40.4
41.6
51.3
53.9
45.6
47.7
40.4
45.7
93.8
98.5
99.9
99.9
98.9
99.6
61.2
-5.4
Tent (Wang et al., 2021)
48.9
49.2
65.8
73.0
54.5
58.4
44.0
47.
50.3
23.9
72.8
55.7
34.4
33.9
51.0
+4.8
CoTTA (Wang et al., 2022)
52.9
51.6
51.4
68.3
78.1
57.1
62.0
48.2
52.7
55.3
25,9
90.0
56.4
36,4
35.2
54,8
+1.0
VDP (Gan et 2023)
52.7
51.6
50.1
58.1
70.2
56.1
58.1
42.1
46
45.8
23.6
70.4
54,9
34.5
36.1
50.0
+5.8
SAR (Niu al. 2023)
49.3
43.8
58.2
46.1
51.8
41.3
41.8
23.8
$7.2
49.9
32.9
32.7
45.2
+10.6
PETAL (Brahma & Rai, 2023)
52.1
48.2
47.5
66.8
74,0
56.7
59.7
46.8
47.2
52.7
26.4
91.3
50.7
32.3
32.0
52.3
+3.5
VIDA (Liu et al., 2024b)
47.7
42.5
42.9
52.2
56.9
45.5
48.9
38
42.7
40.7
24.3
52.8
49.1
33.5
33.1
43.4
+12,4
Continual-MAF (Liu et al., 2024u)
46.3
41,9
42.5
51.4
54.9
43.3
40.7
34.2
35.8
64.3
23.4
60.3
37.5
29.2
31.4
42.5
+13.3
REM (Ours)
43.5
38.1
39.2
53.2
49.0
43.5
42.8
37.5
35.2
35.4
23.2
46.8
41.6
28,9
30.2
39.2
+16.6
Supervised
42.5
36.9
37.1
46.4
44.0
37.4
38.3
34,2
33.1
32.5
21.5
43.3
34.4
26.1
27.5
35.7
+20.1
Table Classification emror rate (%) ) for CITARI0+1-CIFARRIOO under CTTA scenario. Mean (%) denotes the average error rate across
15 target domains. Gain (%) represents the relative performance improvement compared to the source model.
Time
Method
Gaussian
shot
impaliee iss
defocus
glass
nation
zoom
snow
frost
Fog
brightreis
contrast
clastic trans
pixelate
ipeg
Mean./
Gain
Source (Dosoviskkyy et al., 2021)
60.1
33.2
38.3
19.9
35,5
22.6
18.6
12.1
12.7
22.8
5,3
49.7
23.6
24.7
23.1
28.2
0.0
Pscudo-label (Lee, 201)
59.8
52.5
37.2
19.8
35.2
2188
17.6
11.6
12.3
20.7
5.0
41.7
21.5
25.2
22.1
26.9
+1.3
Tent (Wang et al., 2021)
57.7
56.3
29.4
16.2
35,3
16.2
12.4
11.0
11.6
14.9
4.7
22.5
15.9
29.1
19.5
23.5
+4.7
CSTTA (Wang al., 2022)
58.7
51.3
33.0
20.1
34.8
20
15.2
11.1
11.3
18.5
4.0
34.7
18.8
19.0
17.9
24.6
+3.6
VDP (Gan et al., 2023)
$7.5
49.5
31.7
21.3
35.1
19.6
15.1
10.8
10.3
18.1
4.0
27.5
18.4
22.5
19.9
24.1
+4.1
SAR (Niu et al., 2023)
54.1
47.6
38.0
19.9
34.8
22.6
18.6
12.1
12.7
22.8
5.3
39.9
23.6
24.7
23.1
26.6
+1.6
PETAL (Brahma & Rai, 2023)
59.9
52.3
36.1
20.1
34.7
19.4
14.8
11.5
11.2
17.8
4.4
29.6
17.6
19.2
17.3
24.4
+3.8
VIDA (Liu et al., 2024b)
52.9
47.9
19.4
11.4
31.3
13.3
7.6
7.6
9.9
12.5
3.8
26.3
14.4
33.9
18.2
20.7
+7.5
Continual-MAA (Liu et al., 2024a)
30.6
18.9
11.5
10,4
22.5
13.9
9.8
6.6
8.8
4.0
8.5
12.7
9.2
14.4
12.6
+15.6
REM (Ours)
17.3
12.5
10.3
8.4
17.7
8.4
5,5
6.6
5.6
7.2
3.7
6.4
11.0
7.3
13.0
9,4
+18.8
Supervised
14.6
9.0
6.1
11.2
6.0
3.7
4.4
4.9
2.1
3.7
7.5
4.3
8.5
6.4
+21.8
3.5. Total Loss Function
CIFAR100G 100C. The source domains are ImageNet (Deng et al., 2009) and CIFAR (Krizhevsky et al., 2009), while the corre- sponding robustness benchmarks (Hendrycks & Dietterich, 2018), ImageNetC, CIFAR IOC, and CIFARIO0C, are used as the target domains. The suffix C in these datasets indi- cates corruption, which includes 15 types of corruptions, each with 5 levels of severity. Following (Wang et al., 2022: Liu et al., 2024b;a), we adopt target domains with level severity across all 15 corruption types for sequential do- mains. We evaluate the classification error rate for each target domain after adaptation and prediction on target do- main data streams in online manner.
The total loss function CREM is expressed as a linear com- bination of LMCT, in Eq. 3 and LERI. in Eq. 4, as follows:
CREM = LMCL - X LERL,
(5)
where À is hyperparameter. We propose Ranked Entropy Minimization, which integrates the advantages of consis- tency regularization and entropy minimization within a ranked structure based on masking ratios.
4. Experiments
Comparison Methods. We compare various types of state- of-the-art CTTA approaches using the ViT-B/16 (Doso- vitskiy et al., 2021) pre-trained on the source domain. These include single model-based methods such as Pseudo- label (Lee, 2013), Tent (Wang al., 2021), VDP (Gan et al., 2023), and SAR (Niu et al., 2023), as well teacher- student frameworks, including CoTTA (Wang et al., 2022), PETAL (Brahma & Rai, 2023), ViDA (Liu et al., 2024b), and Continual-MAE (Liu et al., 2024a). Additionally, the supervised results, obtained by training with target labels using cross-entropy loss, are presented as an upper bound since target labels are unavailable in TTA.
In this section, we extensively explore the effectiveness of our REM on CTTA protocol (Wang al., 2022). The analy- sis includes comparisons with state-of-the-art baselines, ver- ification of its intended functionality through visualizations, and an understanding of its working mechanisms through ablation studics. Additionally, experiments on Online TTA and LanguaeLe Model are provided in Appendix C, D.
4.1. Experimental Setup
Benchmarks. We construct experiments on ImageNet-to- ImageNetC, CIFARR1--o-CIFARIII and CIFAR100-1
5

---

Ranked Entropy Minimization for Continual Test-Time Adaptation
Table Classification error (%) for CIFAR -0----CA-CC under CTTA scenario. Mcan (%) denotes the average error across
15 target domains. Gain (%) represents the relative performance improvement compared to the source model.
Time
Method
Gaussian
shot
impulee
deforus
glass
motion
zonm
SHOW
Frost
feg
trightness
contrast
elastic trans
pixclate
ineg
Mean]
Gain
Source (Dusovitskiy et al., 2021)
55.0
51.5
26.9
24.0
60,5
29.0
21.4
21.1
25.0
35.2
11.8
34.8
43.2
56.0
35,9
35.4
0.0
Pseudo--abee labell (Lee,, 2013)
53.8
48.9
25.4
23.0
58.7
27.3
19.6
20.6
23.4
31.3
11.8
28.4
39.6
52.3
33.9
33.2
+2.2
Tent (Wang
2021)
53.0
47.0
24.6
22.3
58.5
26.5
19.0
21.0
23.0
30.1
11.8
25.2
39.0
47.1
33.3
32.1
+3.3
CoTTA (Wang et al., 2022)
55.0
51.3
25.8
24.1
59.2
28.9
21.4
21.0
24.7
34.9
11.7
31.7
40.4
55.7
35.6
34.8
+0.6
VDP (Gan et 2023)
54.8
51.2
25.6
24.2
59.1
28.8
21.2
20.5
23.3
33.8
7.5
11.7
32.0
51.7
35.2
32.0
+3,4
SAR (Niu al., 2023)
31.0
19.8
20,9
22.6
19.1
20.3
24.3
11.8
22.3
35.2
32.1
30.1
26.2
+9.2
PETAL (Brahma & Rai, 2023)
49.2
38.7
24.1
26.3
38.2
25.4
19.4
21.0
19.3
26.6
15.4
31.8
28.3
26.6
29.5
28.0
+7,4
VIDA (Liu et al., 2024b)
50.1
40.7
22.0
21.2
45.2
21.6
16.5
17.9
16.6
25.6
11.5
29.0
29.6
34.7
27.1
27.3
+8.1
Continual-MAB (Liu et al., 2024u)
48.6
30.7
18.5
21.3
38.4
22.2
17.5
19.3
18.0
24,8
13.1
27.8
31.4
35.5
29.5
26.4
49.0
REM (Ours)
29.2
25.5
17.0
19.1
35.2
21.2
18.3
19.5
18.7
22.8
15.5
17.6
31.6
26.2
33.0
23.4
+12.0
Supervised
26.2
20.6
13.9
15.9
24.6
15.6
11.
13.1
12.1
13.6
8.5
9.7
20.2
13,5
21.5
16.1
+19.3
Table Forward transfer analysis on ImageNetC. Results (%) rep-
After Training on Corruptions
100
resent the error rates for unseen and seen domains, harmonic mean.
Directly test on unseen domains
Unseen]
Seen
Harmonic
Method
80
bri.
contrast elastic pixelate jpeg
Mean,
Mean]]
Mean
Error (%6
70
Source
26.4
91.4
$7.5
38.0
36.2
49.9
58.8
54.0
Tent
25,8
91.9
$7.0
37.2
35.7
49.5
$4.4
51,8
CoTTA
25.3
88.1
55.7
36.4
34.6
48.0
57.8
52.4
50
CoTTA
VIDA
24.6
68.2
49.8
34.7
34.1
42.3
45.9
44.0
VIDA
REM (I=:le-4)
23,9
66.3
47.6
35,9
33.1
41.4
45.4
43.1
30
REM(I=1e-3)
REM (Ir=le-3) 3)
24.8
66.9
53.5
40.0
39.4
44.9
42.1
43.5
REM(Ir=1e-4)
20
Centrold
Supervised
22.5
71.1
55.3
38.2
36.6
44.7
38.2
41.2
Adaptation Order
Implementation. We implemented the experiments on CI- FARIOC and CIFAR 100C using the open-source Continual- MAE code and the provided source model weights. For ImageNetC, the experiments were conducted using the open- source ViDA code with ImageNet pre-trained weights from timm (Wightman, 2019). Details for reproducibility and training regimes are provided Appendix B.
Figure 4, Backward transfer analysis on ImageNetC. We compare the performance of CTTA approaches on previous domains.
200
Accuracy (%)
Error
4.2. Quantitative Results
CRU ORW Learning Rate
Table presents the CTTA ex- perimental results for a source model pre-trained on Ima- geNet, using each corruption in ImageNetC as the target domain. The model sequentially adapts the target do- mains time, and compare the average error for each domain. Our method improves average performance by 16.6% over the source model and surpasses the previous -ate--of-the-art Continual-MAE by 3.3%. Notably, the per- formance gap to the supervised learning upper bound is only 3.5%, demonstrating the effectiveness of our approach.
Online Batch
Figure 5. Adaptability analysis on ImageNetC under Gaussian noise corruption.
Figure Robustness analysis with respect to learning rate on ImageNetC.
geNetC. These findings highlight the robustness and adapt- ability of our method across diverse datasets.
Forward and Backward Transfer Analysis. We analyze the performance on both future and past domains during the CTTA process to investigate the potential temporal effects of domain adaptation. Table 4 presents the performance on the 5 unseen domains after training on 10 domains. The investigation of the impact test-time adaptation at the cur- rent time on future performance reveals a trade-off between adaptability and generalization. Rapid adaptation driven by
and CIFARI00-tO-CIAAL Table 2 and 3 summarize the experimental results for mod- els trained on CIFAR10 10 and CIFAR100 source domains, with CIFARIOC and CIFAR 100C serving as the respective target domains. The results reveal consistent performance improvements on CIFAR1OC and CIFARI00C, which are widely recognized benchmarks for CTTA alongside Ima-
CIFARIO-1O-CIARRI
6

---

Ranked Entropy Minimization for Continual Test-Time Adaptation
Source
Tent
REM
Malamute
Prediction: Siberian husky & Entropy: 0.982
Prediction: Siberian husky Entropy: 1.020
Prediction: Malemute & Entropy: 0.891
Chickadee
Prediction: Dhole & Entropy: 4.585
Prediction: Coucal & Entropy: 1. 1.185
Prediction: Chickadee Entropy: 2,053
Koala
Prediction: Koala & Entropy: 2.033
Prediction: Indri Entropy: 1.281
Prediction: Koala & Entropy: 1.817
Figure 7. Masked image visualization. We compare the predictions and entropy of REM, Tent, and Source and visualize the results of our masking strategy. Each columa represents images with masking ratios 0, 10%, and 20% for each method, while each row shows the true label the left, along with correct and incorrect predictions and its corresponding entropy values.
Source
Tont
REM
Electric guitar
Table 5. Efficiency comparison, We provide number train- ing parameters, total time, number of the forward passes (FP) and models, and error rate (%).
Method
Parameters,
Total Timel
# FP
Total Models
Ertor
Mare
Tent
0.04M
8m335
Must
51.00
Soccer bail
CoTTA
86.4M
33m23
3or35
Must
Mouu Mar.
54.8
VIDA
Muss Mess
Continal MAE
86.5M
59m56s
Mexst Mare
42.5
REM
0.03M
17m22s
Must
39.2
Malinois
domain shifts effectively. Figure 5 presents a comparative analysis adaptability under varying learning rates, while Figure illustrates performance trends across a broad range of learning rate settings. The robustness of the proposed method across diverse learning rate boundaries underscores its practical utility, it enables flexible adaptation speed selection tailored to specific application requirements.
Figure 8. Grad CAM visualization. We compare attention maps to identify the pixels contributing to predictions.
4.3. Qualitative Results
high learning rates achieves the best performance of 42,1 1% Seen domains, while slower adaptation yields the best performance of 41.4% Unseen domains. Such a trend is also observed in Figure 4, which presents the performance across all domains for the model trained on sequential all domains. Based on the adaptation order, rapid adaptation demonstrates low error on domains learned later, while slow adaptation achieves low error on initial domains.
Masked Image Visualization. From Figure 7, we provide visualization of the masked images generated by the ex- plicit mask chaining, along with the predictions and entropy values for the corresponding original images. can be observed that masking is specifically applied to the pixels where objects are located. Moreover, compared to Tent, which tends be overly confident in uncertain predictions, this approach maintains higher entropy, allowing for im- provements in the incorrect predictions of the initial source model and effectively mitigating overconfidence.
The preceding experiments indicate that achiev- ing generalized performance across diverse domains does not necessarily guarantec optimal performance in CTTA. Distinctive advantage of TTA over domain generalization lies in its capability to perform domain speecifc adaptation through online learning, which critical for addressing
Discussion.
Class Attention Map Visualization. In Figure 8, we present the Grad-CAM (Selvaraju et al., 2017) visualiza- tion results to highlight the salient pixels influencing the

---

Ranked Entropy Minimization for Continual Test-Time Adaptation
Effect of each compoenet
Effect of each compoenet
Effect each compoenet
160
20
Model Collapsa
80
Model Collapes
20
Model Collapae
Enror (8a)
00
Emror (56)
60
Error 1961
REM
REM w/b ERL
20
Adaptation
Adaptation
Adapiation
(a) ImageNetC
(b) CIFAR 100C
(c) CIFAR 1OC
Figure 9. Effect of each component. We present the results of REM and compare them with variations where MCL and ERL are removed.
Effect of masking strategy
Effect of masking strategy
Effect of masking strategy
100
Model Collapse
Model Collapse
Erro (86)
Enar (56)
Error (98)
50
Foreground Masidng
Background Masking
Random Masking
Adopptation Order
Acsptation Order (-))
Adaptation Order
(a) ImageNetC
(b) CIFAR10OC
(c) CIFARIOC
Figure 10. Effect of masking strategy. We present the results of REM (foreground masking) in comparison with different masking strategics, including background masking, and random masking.
model's predictions and to gain insights into its decision- making process. Each row corresponds to the results for Gaussian noise, fog, and pixelation, respectively, demon- strating that Tent gradually highlights local regions over time. This phenomenon can be interpreted the model re- lying non-discriminative local features rather than global semantic context for predictions, resulting in the generation of uniform predictions irrespective of the input. In contrast, our method adopts consistency regularization for masked objects and effectively captures comprehensive information related context.
4.5. Ablation Studies
Effect of Each Component. We present the results of the ablation experiments for each component in Figure 9. Model collapse appeared early for CIFARC, which contains lower resolution and information compared to ImageNetC, when MCL. and REL were removed. Note that our method achieves stable performance without model collapse when both are applied, due to the organic design of each method.
Effect of Masking Strategy. In order to validate the appro- priateness of our masking strategy, Figure 10 illustrates a comparison between our foreground masking strategy and cases involving background masking or random masking. Our intuition behind defining an explicit ranking relation- ship for the predictions is satisfied when masking the fore- ground. In this case, the method performed as designed, and there is no model collapse for all test datasets.
4.4. Efficiency Comparison
We compare the computational efficiency of our method with Tent, an EM approach, and CR based state-of-the-an methods such as CoTTA, ViDA, and Continual-MAE in Ta- ble 5. Our method follows the strategy of existing EM approaches that update only the normalization layers of single test model (Mrest)) providing advantages in terms of training time, the number of trainable parameters, and the number of models that need to be stored. Recent CTTA approaches store the EMA model (Mema) used as teacher model and the source model (Merc) to regress the ini- tial source weights. In addition, it often requires numerous forward passes to model uncertainty. Compared to the re- cent srate-of-the-an Continual-MAB, we achieve a 3.3% performance improvement while requiring only 30% of the computation time and 0.03% of the training parameters.
5. Limitation
Our study is grounded in the assumption that explicitly masking objects used as the basis for predictions can lead to decrease in accuracy and an increase in entropy. While our method is simple and intuitive, it is not yet fully supported by rigorous theoretical proof. Despite efforts to address this, the counterexamples arising from the diversity of images still pose significant challenges. To mitigate this, we con- duct several experiments that provide empirical evidence demonstrating statistical significance.
8

---

Ranked Entropy Minimization for Continual Test-Time Adaptation
6. Conclusion
IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
In this paper, we introduce Ranked Entropy Minimization (REM) improve stability and efficiency in CTTA. Based on observation of model collapse, we propose a progressive masking strategy and dual complementary loss functions: masked consistency loss and ranked entropy loss. Conse- quently, REM captures the best of both worlds by integrating the stability of consistency regularization and the efficiency of entropy minimization. Through quantitative evaluations on various CTTA benchmarks, REM achieves state-of-the- art performance, demonstrating its effectiveness. Moreover, extensive qualitative experiments and ablation studies offer in-depth insights into the working principles. We hope that our work serves as foundation for valuable discussions computational cost in CTTA, paving the way for advances in efficiency and real-world applicability.
Cai, C., Ye, Z., Feng, L., Qi, J., and Liu, F. Sample-specine masks for visual reprogramming-bant prompting. In International Conference on Machine Learning. 2024.
Csaba, B., Zhang, W., Müller, M., Lim, S.-N., Torr, P., and Bíbi, A. Label delay online continual learning. Advances in Neural Information Processing Systems, 2024. Deng, J., Dong, W., Socher, R., Li, L..J.,, Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. Proceedings of the IEEE/CVF Conference on Com- puter Vision and Puttern Recognition, 2009.
Döbler, M., Marsden, R. A., and Yang, B. Robust mean teacher for continual and gradual test-time adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, 2023.
Impact Statement
This paper presents work whose goal is advance the field of Machine Learning. There are many potential societal consequences of our work. none which we feel must be specifically highlighted here.
Dosovitskiy. A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale. International Conference on Learning Representations, 2021.
Acknowledgements
This work supported by ITTP grant funded by the Korea goVe)) (RS-2025-02283048, Devel- oping the Nexe-Gennnation General AI with Reliability, Ethics, and Adaptability; RS-2023-00236245 Develop- ment of Perception/Plamning AI SW for Seamless Au- tonomous Driving in Adverse Weather/Unstructuredd En- vironment; RS-2021-1121268,, AI Innovation Hub) and NRF-2022RIAACII01140
Fang, C., Xu, Y., and Rockmore, D. N. Unbiased metric learning: On the utilization of multiple datasets and web images for softening bias. Proceedings of the IEEE International Conference on Computer Vision, 2013.
Gan, Y., Bai, Y., Lou, Y., Ma, X., Zhang, R., Shi, N., and Luo, L. Decorate the newcomers: Visual domain prompt for continual test time adaptation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 7595 7595-7603, 2023.
References
Alfarra, M., Itani, H., Pardo, A., shyma yaser alhuwaider, Ramazanova, M., Perez, J. C., zhipeng cai, Müller, M., and Ghanem, B. Evaluation of test-time adaptation under computational time constraints. In International Confer- ence on Machine Learning, 2024.
Gong, T., Jeong, J., Kim, T., Kim, Y., Shin, J., and Lee, S.-J. Note: Robust continual test-time adaptation against temporal correlation. Advances Neural Information Processing Systems, 2022.
Bolya, D., Fu, C.-Y., Dai, X., Zhang, P., Feichtenhofer, C., and Hoffman, J. Token merging: Your vit but faster. In International Conference on Learning Representations, 2023.
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. Generative adversarial nets. Advances in neural informa- tion processing systems, 2014.
Boudiaf, M., Mueller, R., Ben Ayed, I., and Bertinetto, L.. Parameter-fiee free online test-time adaptation. Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
Hakim, G. A. V., Osowiechi, D., Noori, M., Cheragha- likhani, M., Bahri, A., Yazdanpanah, M., Ayed, I. B., and Desrosiers, C. Clipartt: Leght adaptation clip to new domains at test time. arXiv preprint arXiv:2405.nm 20005440 2024.
Brahma, D. and Rai, A probabilistic framework for lifelong test-time adaptation. Proceedings of the

---

Ranked Entropy Minimization for Continual Test-Time Adaptation
Hendrycks, D. and Dietterich, T. Benchmarking neural network robustness common corruptions and perturba- tions. In International Conference on Learning Represen- tations, 2018.
adaptation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024a.
Liu, J., Yang, S., Jia, P., Lu, M., Guo, Y., Xue, W., and Zhang. S. Vida: Homeostatic visual domain adapter for continual test time adaptation. In International Confer- ence on Learning Representations, 2024b.
Hendrycks, D., Basart, S., Mu, N., Kadavath, S., Wang, F., Dorundo, E., Desai, R., Zhu, T., Parajuli, S., Guo, M., Song, D., Steinhardt, J., and Gilmer, J. The many faces of robustness: A critical analysis of out-of out-of-distribution generalization. ICCV, 2021.
Marsden, R. A., Döbler, M., and Yang, B. Universal test- time adaptation through weight ensembling, diversity weighting, and prior correction. In Proceedings of the IEEE/CVF Winter Conference on Applications of Com- puter Vision, 2024.
Hua, T., Wang. W., Xue, Z., Ren, S., Wang, Y., and Zhao, H. On feature decorrelation in supf-super learning. In Proceedings of the IEEF/CVF International Conference on Computer Vision, 2021.
Moon, J., Kim, J., Shin, Y., and Hwang, S. Confidence- aware learning for deep neural networks, In international conference on machine learning, 2020.
Huggett, N. Zeno's paradoxes. In Zalta, E. N. (ed.), The Stanford Encyclopedia of Philosophy. Meta- physics Research Lab, Stanford University, 2018 edi- tion, 2002. https: /ttc:/plaatt httpppatoooo. stanford. edu/ ENTRIES/ ENTRIES/paradox parpdox- zeno/.
Nacini, M. P., Cooper, G., and Hauskrecht, M. Obtaining well calibrated probabilities using bayesian binning. In Proceedings of the AAAI conference artificial intelli- gence, 2015.
Iwasawa, Y. and Matsuo, Y. Test-time classifier adjustment module for aadel-agnostic domain generalization. Ad- vances in Neural Information Processing Systems, 2021.
Niu, S., Wu, J., Zhang, Y., Chen, Y., Zheng, S., Zhao, P., and Tan, M. Efficient test-time model adaptation with- out forgetting. In International conference on machine learning. PMLR, 2022.
Kar, F., Yeo, T., Atanov, A., and Zamir, A. 3d common corruptions and data augmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, 2022.
Niu, S., Wu, J., Zhang, Y., Wen, Z., Chen, Y., Zhao, P., and Tan, M. Towards stable test-time adaptation in dynamic wild world. In Internetional Conference on Learning Representations, 2023.
Krizhevsky, A., Hinton, G., et al. Learning multiple layers of features from tiny images. 2009.
Niu, S., Miao, C., Chen, G., Wu, P., and Zhao, P. Test-time model adaptation with only forward passes. In Interna- tional Conference Machine Learning, 2024.
Lee, D.-H. Pseudo-label: The simple and efficient semi- supervised learning method for deep neural networks. In International conference on machine learning Workshop, 2013.
Noh, J., Park, H., Lee, J., and Ham, B. Rankmixup: Ranking- based mixup training for network calibration. In Proceedings of the IEEE//VE International Conference on Computer Vision, 2023.
Lee, J., Jung, D., Lee, S., Park, J., Shin, J., Hwang, U., and Yoon, S. Entropy is not enough for test-time adap- tation: From the perspective of disentangled factors. In International Conference on Learning Representations, 2024.
Osowiechi, D., Noori, M., Hakim, G. A. V.. Yazdanpanah, M., Bahri, A., Cheraghalikhani, M., Dastani, S., Beizaee, F., Ayed, I. B., and Desrosiers, C. WATT: Weight average test time adaptation of CLIP. In Advances neural information processing systems, 2024.
Lee, J. and Chang, J. Continual momentum filtering on parameter space for online test-time adaptation. In The Twelfth International Conference on Learning Repre- sentations, 2024.
Peng, X., Usman, B., Kaushik, N., Wang, D., Hoffman, J., and Saenko, K. Visda: A synthetic-to-reat benchmark for visual domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 2021 2021-2022 2026, 2018.
Li, D., Yang, Y., Song, Y.ZZ,, and Hospedales, T. M. Deeper, broader and artier domain generalization. In Proceedings of the IEEE international conference on computer vision, 2017.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, 2021.
Liu, J., Xu, R., Yang, S., Zhang, R., Zhang, Q., Chen, Z., Guo, Y., and Zhang, Continual maa: Adaptive distribution masked autoencoders for continual test-time
10

---

Ranked Entropy Minimization for Continual Test-Time Adaptation
Recht, B., Roelofs, R., Schmidt, L.,, and Shankar, V. Do ima- genet classifiers generalize imagenet? In International conference on machine learning. PMLR, 2019.
Zhang, M., Levine, S., and Finn, C. Memo: Test time robustness via adaptation and augmentation. Advances in neural information processing systems, 2022.
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., and Batra, D. Grad-cam: Visual explana- tions from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision, 2017.
Zhang, Q., Bian, Y., Kong, X., Zhao, P. and Zhang. COME: Test-time adaption by conservatively minimiz- ing entropy. In International Conference Learning Representations, 2025.
Shimodaira, H. Improving predictive inference under covari- ate shift by weighting the log-likeliine likklihhhodd function. Journal of statistical planning and inference, 2000.
Shu, M., Nie, W., Huang, D.-A., Yu, Z., Goldstein, T., Anandkumar, A., and Xiao, Test-time prompt tun- ing for zero-shot generalization in vision-language mod- els. Advances in Neural Information Processing Systems, 2022.
Son, S., Ryu, J., Lee. N., and Lee, J. The role of masking for efficient supervised knowledge distillation of vision trans- formers. European Conference on Computer Vision, 2024.
Sun, Y., Wang, X., Liu, Z.., Miller, J., Efros, A., and Hardt, M. Test-time training with self-supervision for generaliza- tion under distribution shifts. In International conference on machine learning. PMLR, 2020.
Tarvainen, A. and Valpola, Mean teachers are better role models: Weight- consistency targets improve semi-supervised deep learning results. Advances in neural information processing systems, 2017.
Venkateswara, H., Eusebio, J., Chakraborty, S., and Pan- chanathan. S. Deep hashing network for unsupervised domain adaptation. In Proceedings of the IEEE confer- ence on computer vision and pattern recognition, 2017.
Wang, D., Shelhamer. E., Liu, S., Olshausen, A., and Darrell, T. Tent: Fully test-time adaptation entropy minimization. In International Conference Learning Representations, 2021.
Wang, H., Ge, S., Lipton, Z., and Xing, E. P. Learning ro- bust global representations by penalizing local predictive power. Advances in Neural Information Processing Systems, 2019.
Wang, Q., Fink, O., Van Gool, L., and Dai, D. Continual test- time domain adaptation. In Proceedings of Conference on Computer Vision and Pattern Recognition, 2022.
Wightman, R. Pytorch image models. httpps:// /tt:::///tttiii huu.. com/ rwightman/ catpaaanppyttorccooo www.ooooimmeeeeooooooooopp 2019.

---

Ranked Entropy Minimization for Continual Test-Time Adaptation
Frocen Parurstars
Inering Parmmtaa
dith
Precktan Probubilty
Coreatenty Loss.
Frinpyy loo.
Updisle Layer
EMA Upddlte
1
Date Aug: MC Drepout
(a) Entropy Minimization
(b) Consistency Regularization
(c) Ranked Entropy Minimization
Figure 11. Conceptual illustration comparing CTTA frameworks. (a) Entropy minimization approach updates only the normalization layer for the target domain while utilizing single model. (h) Consistency regularization method employs both a student model, which updates all parameters, and teacher model, which is updated via exponential moving average (EMA). This approach enhances prediction diversity by numerous data augmentations or applying MC dropout to the model. (c) Our proposed method updates the normalization layer and adopts single model. It integrates entropy minimization and consistency regularization with only two additional forward passes.
A. Overall Framework and Comparison with Other Approaches
To improve clarity and facilitate comprehension, we present the overall frameworks of the Entropy Minimization (EM) approach, the Consistency Regularization (CR) approach, and our proposed method, REM in Figure 11. We maintain the training scheme of EM to preserve its computational efficiency. Additionally, replace the traditional data augmentations used in CR approaches with explicit masking, eliminating the reliance extensive image augmentations for enhancing predictive diversity. Consequently, our method preserves the computational efficiency of EM approaches while enhancing the robustness and generalization performance of CR approaches.
B. Implementation Details
Table 6 provides details on the implementation our experiments, including optimizer settings, learning rates, batch sizes, model architectures, and hyperparameters, For CTTA experiments, follow the Continual-MAE framework. Specifically, for CIFAR datasets, resize the input images to 384x384, while for all other experiments, the images resized to 224x224.
Table 6. Implementation details.
Task
CTTA
TTA
TTA-CLIP
Dataset
ImageNetC
CIFARC
ImageNetC ImageNet-R/V2/S
CIFAR
Other Datasets
Experimental Protocols
Reproducibility
Continual-MAi (Liu et al., 2024a)
DeYO (Lee et al., 2024)
WATT (Osowiechi et al., 2024)
Training Parameters
Optimizer
Adam
Adam
SGD
SGD
Adam
Adam
Optimizer momentum
(0.9, 0.999)
(0.9, 0.999)
0.9
(0.9, 0.999)
(0.9, 0.999)
Learning rate Batch size Model architecture
le-3
le-3
le-3
1e-3
le-3
le-4
50
20
or 64
1 or 64
128
128
ViT-B/16
ViT-B/16
VIT-B/16
ViT-B/16
CLIP- VTT-B16/
VLIP-VIT-/16
Algorithm Parameters
(Eq. 5) m (Fq. 4)
1.0
1.0
0.5
0.5
1.0
1.0
0
0
0
12

---

Ranked Entropy Minimization for Continual Test-Time Adaptation
Table Classification accuracy (%)
ImageNeiC. (severity level5) under online imhalanced label shifts (imbalance ratio co).
Label Shifts
Gaussian
shol
impulse
defocus
glass
mation
200m
SNOW
frost
feg
brighmess
contrast
etastic trans
piselate
jpet
Meant
Gain
Source (Dosovisskyy al., 2021)
9.4
6.7
8.3
29.1
23.4
34.0
27.0
15.8
26.3
47.4
$4.7
43.9
30.5
44.5
47.6
29,9
MEMO (Zhang al., 2022)
21.6
17.4
20.6
37.1
29.6
40.6
34.4
25.0
34.8
55.2
65.0
54.9
37.4
55.5
57.7
39.1
9.2
Tent (Wang al. 2021)
53.1
53.1
54.3
54.2
51.5
58.6
52.4
3.5
7.8
69.5
74.8
67.0
58.7
69.2
66.2
52.9
23.0
EATA (Niu et al., 2022)
45.5
47.2
44.1
45.4
41.5
52.0
47.4
54.8
46.7
57.1
70.4
29.2
55.9
62.2
60.6
50.7
20.8
SAR (Niu et al., 2023)
53.1
53.3
54.3
54,0
52.1
58.0
52.7
8.6
28.6
69.1
74.7
66.7
59.1
67.1
64.9
54.4
24.5
(Lee al., 2024)
52.9
54.8
55.4
54.1
55.6
62.1
34.4
64.6
63.7
71.1
77.1
64.2
67.2
72.4
68.2
61.2
31.3
REM (Ours)
57.0
57.2
58.1
58.6
56.3
63.2
58.4
67.3
67.5
74.4
78.9
70.5
65.6
73.3
70.0
65.1
35.2
Table Classification accuracy (%) ImageNetC (severity levelS) under batch size 1.
Batch Size
Gaussian
shot
impuise
defocus
glass
motion
zoom
BROW
frost
fog
brightness
contrast
elastic trans
pixelate
jpez
Meant Gain
Source (Dosoviskly
2021)
9.5
6.8
8.2
29.0
23.5
33.9
27.1
15.9
26.5
47.2
54.7
44.1
30.5
44.5
47.8
29.9
0.0
MEMO (Zhang al., 2022)
21.6
17.4
20.6
37.1
29.6
40.6
34.4
25.0
34.8
55.2
65.0
54.9
37.4
55.5
57.7
39.1
9.2
Tent (Wang et al., 2021)
52.1
51.8
53.2
52.4
48.7
56.5
49.5
8.5
15.2
67.3
73.4
66.7
52.6
64.9
64.3
51.8
21.9
EATA (Niu et al., 2022)
48.5
46.5
49.6
46.2
40.2
50.5
44.1
37.8
41.7
64.6
68.2
64.5
49.6
61.0
61.6
51.6
21.7
SAR (Niu et al., 2023)
52.0
51.7
53.1
51.7
48.9
56,8
50.6
54.8
67.2
74.7
66.1
55.3
66.8
65.2
55.5
DeYO (Lee al., 2024)
54.6
55.6
56.0
55.5
62.7
64.4
72.0
77.3
10,9
66.3
71.8
68.7
57.2
REM (Ours)
57.4
57.8
58.6
59.2
56.9
63.5
59.1
68.4
67.5
74.5
79.0
71.1
65.7
73.3
70.5
65.5
35.6
Table 9. Classification accuracy (%) on ImageNetC (severity
Table 10. Classification accuracy (%) ImaggNttRRVV2/kkee
level and level 3) under mixture of 15 corruption.
Mcan (%) denotes the average accuracy across target domains.
Mixed Shifts
Level 5 Level 3
Domain Shifts
R
V2
Sketch Mean
Source (Dosovitskiy al., 2021)
29.9
53.8
Source (Dosovitskiy al., 2021
59.5 75.4
44.9
59.9
Tent (Wang al., 2021)
70.2
Tent (Wang et al., 2021)
63.9 75.2
49.1
62.7
EATA (Niu el al., 2022)
56.4
69.6
CoTTA (Wang et al., 2022)
63.5 75.4
50.0
62.9
SAR (Niu et al., 2023)
57.1
70.7
SAR (Niu et al., 2023)
63.3 75.1
48.7
62.4
De (Lee al., 2024)
59.4
72.1
FOA (Niu al., 2024)
63.8 75.4
49.9
63,0
REM (Ours)
62.4
74.0
REM (Ours)
64.3 75.2
49.7
63.1
C. Experiments on Online Test-Time Adaptation Scenario
In addition to CTTA scenarios, our method readily applicable to wide range of TTA scenarios. To evaluate its effectiveness in a more challenging setting, we compare our approach against EM-based statc-of-the-as methods, including MEMO (Zhang et al., 2022), Tent (Wang et al., 2) EATA (Niu et al., 2022), SAR (Niu et al., 2023), and DeYO (Lee al., 2024), in the wild online TTA scenarios proposed in SAR.
Online Imbalanced Label Distribution Shifts. Table 7 presents the performance comparison of TTA methods under class-imbalanced distributions across different domains. Our method achieves best performance across all domains except for elastic transform, improving the average performance by 3.9% compared to the previous tftte-of-theaart method, DeYO.
Batch Size 1. Table 8 shows the results for TTA under a batch size of 1, demonstrating the robustness of our method in scenarios where batch statistics cannot be effectively leveraged. Similar to the label shift scenario, our method achieves the best performance across all domains except for elastic transform, resulting in an 8.3% performance improvement.
Mixed Distribution Shifts. Table 9 presents the results for the TTA scenario where domain boundaries ambiguous, leading to mixed domain distributions. Our method achieves performance improvements of 3.0% and 1.9% for corruption severity levels 5 and 3, respectively, indicating its potential for generalization across diverse domains.
Domain Shifts. Table 10 presents the TTA results for domain shifts from ImageNet to ImageNet-R (Hendrycks et al., 2021), ImageNet-Ve V2 (Recht et al., 2019), and ImageNet-Sken (Wang et al., 2019). We achieve a mean accuracy of 63.1% across all domains, surpassing the previous state-of-the-an FOA (Niu et al., 2024), which achieved 63.0%.
13

---

Ranked Entropy Minimization for Continual Test-Time Adaptation
Table 11. Classificationn accuracy (%) comparison vision-language model using CLIP ViT/B-16 across different datasets and domains.
Dataset
Domain
CL.IP
Tent
TPT
CLIPArTT
WATT
REM
REM++ATT
CIFAR-I0
89.25
92.75
89.80
92.61
91.97
91.76=0.06
93.19 10.14
CIFAR
CIFAR-100
64.76
71.73
67.15
71.34
72.85
69.15
72.87
Mean
77.01
82.24
78.48
81.98
82.41
80,46
83.03
3D (trainset)
87.16
87.57
84.04
87.58
87.72
87 97.45=0.cm
88.95 00.03
VisDA-C
YT (valsct)
86.61
86.81
85,90
86.60
86.75
86.89 00.01
86.84
Mcan
86.89
87.19
84.97
87.09
87.24
87.17
87.90
Art
79.30
79.26
81.97
79.34
80.43
80.17
80.29 11
Clipart
63.15
63.64
67.01
65.69
68.26
66. 16.96=sz 00.04
68.32 0.009
Office-Hore Home
Product
87.34
87.49
89.00
87.35
88.02
87. 87.77
87.99 #0.09
RealWorld
89.31
89.50
89.66
89.29
90.14
90.14 0..01
90.08 +:0.00 090.0
Mcan
80.28
80.47
81.91
80.42
81.71
81 31.26
81.67
Arl
97.44
97.54
95.10
97.64
97.66
97.71 =0.00
97.64
Cartoon
97.38
97.37
91.42
97.37
97.51
97.53
97.45+0000
PACS
Photo
99.58
99.58
98.56
99.58
99.58
99.58 10.00
99.58 10.00
Sketch
86.06
86.37
87.23
86.79
89.56
88. 00.07
90.19 :0.199:0.14 14
Mean
95.12
95.22
93.08
95.35
96.08
95.79
96.22
Caltech10f
99.43
99.43
97.62
99,43
99.36
99. 99.36=00.0
99.39
LabelMe
67.75
67.31
49.77
67.74
68.59
68 68.06.0.01
69.26
VLCS
SUN09
71.74
71.57
71.56
71.67
75.16
75 10.04
75.76 10.10
VOC2007
84.90
85.10
71.17
84.73
83.24
83.79=0..
83.89 2020
Mean
80.96
80.85
72.53
80.89
81.59
81.56
82.08
D. Experiments on Vision-Language Model
Our proposed REM can be applied in plug-and-play manner and is adaptable various modalities. We present experiments on the TTA setting with CLIP the target model in Table 11. We compare the performance of our method against CLIP (Radford et al., 2021), Tent (Wang et al., 2021), TPT (Shu et al., 2022), CLIPAITT (Hakim et al., 2024), and WATT (Osowiechi et al., 2024). Following WATT, we report the TTA results from the CLIP model to CIFAR (Krizhevsky et al., 2009) and various domain adaptation and generalization benchmarks, including VisDA (Peng et al., 2018), Office- Home (Venkateswara et al., 2017), PACS (Li et al., 2017). and VLCS (Fang et al., 2013). As a result, our method achieves competitive performance compared to the previous state-of-the-ar WATT, without requiring additional inner-loop training processes or ensemble methods. Furthermore, when WATT combined with REM, it surpasses the existing results.
E. Experiments on Practical TTA Scenarios
Computational Time Constraint TTA Scenario. Table 12 presents experimental results on ImageNet-3DCC (Kar et al., 2022) under the time-constrained protocol (Alfarra et al., 2024). We compare EATA (Niu et al., 2022) and our proposed method using ViT-B/16. EATA requires 2 the time relative to adaptation speed g for which C'(g) 1, while REM requires 5.10x the time. Therefore, in the episodic scenario. where the model re-initialized for each domain, REM shows lower performance than EATA due its relatively slower adaptation. However, in the continual scenario, where domains are learned sequentially without model re-initialization, method achieves higher performance due to the accumulation of leameed knowledge and demonstrates stable adaptation across domains.
Table 12. Classification error rate (%) on Image under time constraint scenario
Time
Depth field
Nosse
Lighting Flach
Weather
Video
Camera motion
Near focus
Far focus
Color quant
IS0
Low light
Fog 3D
Bis err
H H.265 ABR
H.265 CRF
XY-mot: blur
Z-mat. hlur
EATA- Eiiis-gi
27
35.58
42,14
46.25
33.61
55.64
$4,70
80.39
48.31
42,26
48.92
43,77
46.58
Rpisodi
41.18
Continual
40.66
43.94
35.05
57.21
$6.07
REM-Cooiiual
29.30
33.95
41.12
43.65
31.85
56.24
51.20
87.17
49.07
41.38
49.35
43.33
46.47
14

---

Ranked Entropy Minimization for Continual Test-Time Adaptation
F. Experiments on Different Network Architectures
The proposed REM leverages the self-attention mechanism of VIT, yet can be readily extended to other architectures as long as ranked structure of difficulty can be explicitly defined through chained masking. To demonstrate generality, we introduce two additional variants: one based on feature activation (FA), where the attention map computed as the average L.2norm of feature vectors across all spatial positions, and another based Grad-CAM (Selvaraju et al., 2017) to modulate the masked regions. As shown in Table 13, FA-based REM consistently improves performance across various Transformer architectures. Additionally, Table 14 demonstrates its applicability to CNNs, where our method achieves notable gains even without the use of self-attention, highlighting the broad utility of the proposed difficulty-awar masking strategy.
Table 13. Mean rate (%) ImageNetC using transformer architectures
Model
Source
Tent
CoTTA
ViDA
REM
Mobile-ViTE
75.28
75.61
75.72
75.27
74.28
Brinsfrrofomer-BBB
59.26
73.17
46.84
57.84
46.56
Table 14. Mean enror rate (%) on ImageNeiC using CNN architectures
Model
CoTTA
EATA
EcoTTA
BЕCOТТA
REM (FA)
REM (Grad-CAM)
WideResNet-22
16.2
18.6
16.8
16.9
16.5
WideRessNet-40
37.1
36.4
35.5
34.5
34.6
G. Calibration Error Analysis
We investigate the mitigation model collapse by analyzing the issue of overconfidence through model calibration error. We observe a consistent trend in which the Expected Calibration Error (ECE) tends to increase the model outperforms the initial source model (Naeini et al., 2015). Notably, method maintains low ECE while achieving low error rates, emphasizing the practical importance of calibration for reliable and robust adaptation under distribution shifts.
Table Comparison of ECE error rates on ImageNeiC.
ImageNet-C
Source
Tent
SAR
ViDA
REM
ECE (%) 4
5.3
12.6
10.3
14.6
8.7
Error (%)
55.8
51.0
45,2
43.4
39.2
H. Comparison with Augmentation-based EM approaches
Recent augmentation-based EM methods (Marsden et al., 2024; Lee & Chang, 2024) adopt a variety of stochastic augmenta- tions, such as color jitter and random affine transformations, similar to CoTTA and its variants. In contrast, our method introduces an interpretable and sample-specific augmentation scheme based on structured masking. By designing a ranked prediction distribution, we progressively refine the model's predictions while preserving the relative ranking, offering an intuitive and effective adaptation mechanism. ROID and CMT, like EATA, incorporate the Active Sample Criterion (ASC), which omits the backward pass for inaccurately predicted samples by setting their loss zero. When ASC is applied to method, it achieves a similar level of computational efficiency. As shown in Table 16, REM with ASC reduces inference time while maintaining competitive eиror rates. While ASC-based methods enable efficient adaptation by selectively updating on confident samples, our main objective is to leverage the entire set of test samples. Instead of discarding uncertain predictions, REM aims to reduce domain dependency at test time by enforcing a clear intra-image predictive structure, thereby enhancing robustness against unpredictable domain shifts.
Table 16. Comparison of total adaptation time and emrr rate on ImageNetC.
Method
ROID
CMT
REM (N=1)
REM (N=1, ASC)
Time
9m33s 41.4
9m38s 40.7
1m44s 39.5
9m22s 39.7
Error (%)
15

---

Ranked Entropy Minimization for Continual Test-Time Adaptation
I. Failure Case Analysis
We analyze the discrepancy between the outputs of original and masked images using Total Variation Distance (TVD) in Table 17, focusing on two domains where our method achieved significant performance improvements (Gaussian and Shot noise) and two domains where it showed relatively lower performance (Brightness and JPEG). Interestingly, the domains with successful performance gains, e.g., Gaussian and Shot noise, exhibited larger differences in the predicted probability distributions with and without masking. One possible interpretation is that, for relatively easier domains, small discrepancy between the predicted distributions of the original and masked images may lead to low loss, which in turn could reduce the adaptation speed. This observation suggests that dynamically adjusting the loss magnitude based on the estimated domain gap may further enhance the adaptation performance. In particular, incorporating an adaptive loss weighting scheme could help balance learning across domains of varying difficulty. We consider this a promising direction and leave its detailed exploration for future work.
Table 17. Total Variation Distance (TVD) for the first and last 50% samples under various corruptions CIFARIO0C.
CIFARI0OC
Gaussian
Shot
Brightness
Jpeg
TVD (first 50%)
5.54 = 1.36
3.44 + 1.38
1.69 + 0.45
2.90 0.95
TVD (last 50%)
5.03 = 1.38
3.82 + 0.69
1.55 + 0.41
2.59 = 0.78
J. Hyperparameter Sensitivity Analysis
REM incorporates three hyperparameters: the masking ratio MN in Equation (3), the margin in Equation (4), and the weighting coefficient À in Equation (5). We provide an ablation study on the masking ratio in Table 18. Although the best performance is achieved when N = 3, we adopt the combination My = {0, 5%, 10%} to strike a balance between computational complexity and accuracy. Moreover, Figure 12 presents the results of a grid search over all hyperparameters, reporting the mean error across all domains the that our method exhibits low sensitivity hyperparameter variations. Based these results, we set and m 0 for the CTTA experiments.
benchmark. The experimental results indicate
Table Mean error rate (%) for different masking ratios My on ImageNetC
60
57.5
MN
Mi
M2
M3
Enar
52.5
+5%
{0, 5%}
{0, 5%, 10%}
{0, 5%, 10%, 15%}
20
Error
40.6%
39.4%
38.9%
10
47.5
45.0
+10%
{0, 10%)
{0, 10%, 20%)
{0, 10%, 20%, 30%)
0.3
42.5
taanking Margin
9,
40.0
Error
39.7%
39.2%
39.4%
0.0
+15%
{0, 15%)
{0, 15%, 30%)
{0, 15%, 30%, 45%)
(ERL Loss Weight)
1.0
1.5
2.0
-01
a.
Error
39.5%
39.4%
40.0%
Figure 12, Hyperparameter sensitivity analysis
16


Reviews:
[{"review_id": "eAg3yDvzAz", "reviewer": "Reviewer_f4Hw", "summary": "This work introduces two novel loss functions for CTTA, which utilize different views of the samples to enforce consistency alignment while preserving their relative ranking. Experimental results demonstrate the effectiveness of the proposed approach.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your constructive comments. We address all the concerns and provide new experiments to support our contributions.\n>**1. Novelty**\n\nCompared with [1,2], our novelty is summarized;   \n- [1] proposed a contrastive learning method based on strong-weak augmentations, but our method addresses the problem within a ranked structure through progressive augmentation, and it differs in that ours does not require an additional momentum encoder and does not rely on random augmentation.\n- [2] analyzed that using the prediction from weak augmentation as a pseudo label to improve the prediction of strong augmentation is not effective. It aligns with the argument of CoTTA that data augmentation without considering sample characteristics is not suitable for handling abrupt distribution shifts. Ours applies masking on objects as a domain-invariant property and maintains a ranked structure, enabling stable adaptation by reducing discrepancies in prediction distributions.\n\nTherefore, ours has its own novelty compared with [1,2].\n\n>**2. Foreground masking**\n\nThe detailed descriptions on foreground masking are found at Sec 3.2. Briefly, for the final self-attention layer of the transformer, attention scores are computed based on the similarity between the class token's query and the image tokens' keys. Using these scores, we can efficiently and naturally mask the pixels where objects are located without any extra modules.\n\n>**3. Biased attention**\n\nDue to differences in object sizes across samples, a bias toward the background can occur when small objects occupy a relatively small portion of the pixels. Based on our observations in Fig. 3, to alleviate this concern, we apply low masking ratios of 10% and 20% where a linear relationship between entropy and error is maintained. Note that our goal is to find a simple yet efficient solution without additional computational cost.\n\n>**4. Additional comparison**\n\nWe provide the following table as requested. Note that TTE [3] and its code are not presented yet.\nMethod|EATA|SAR|PETAL|Roid|Ours\n---|---|---|---|---|---\nImageNetC|41.3|45.2|52.3|41.4|**39.2**\n\n[3] Test-Time Ensemble via Linear Mode Connectivity, ICLR2025.\n\n>**5. Ablation on mask**\n\nWe provide the ablation study regarding hyperparameters of the mask. Although the best accuracy was achieved when N=3, we used the combination of M_N ={0,5%,10%} to strike a balance between sensitivity to the masking ratio, increased computational complexity, and accuracy.\nM_N (N=1)|{0,5%}|{0,10%}|{0,15%}\n---|---|---|---\nError|40.6|39.7|39.5\nM_N (N=2)|{0,5%,10%}|{0,10%,20%}|{0,15%,30%}\nError|39.4|39.2|39.4\nM_N (N=3)|{0,5%,10%,15%}|{0,10%,20%,30%}|{0,15%,30%,45%}\nError|38.9|39.4|40.0\n\n>**6. Comparison with entropy-based method**\n\nSimilar to entropy-based methods, we use a single backpropagation per sample. Recent TTA approaches, including EATA, update parameters only using low-entropy samples, thereby improving learning efficiency. Furthermore, parameter-free methods such as T3A and LAME adapt to new domains by aligning logits using optimization algorithms or historical memory without backpropagation. Each of these methods is being studied independently and is based on a different underlying philosophy. We acknowledge that our method is slightly less computationally efficient compared to these approaches. Nevertheless, we believe that utilizing uncertain samples in alignment with the motto of continual adaptation is necessary for broader applicability in diverse environments. We think that these three works will converge by complementing each other’s strengths and weaknesses, and we believe our contribution can narrow the gap between them.\n\n>**7. Resolution**\n\nFor a fair comparison of SOTA, we followed the experimental settings of ViDA and Continual-MAE, using 384×384 for CIFAR, and ours was built on Continual-MAE. We agree that higher resolution can offer advantages for masking, and to clarify this point, we provide a 224x224 experiment.\nCIFAR10C|Source|Tent|CoTTA|Continual-MAE|Ours\n---|:---:|:---:|:---:|:---:|:---:\nViT-B/16-224|40.1|36.0|39.2|16.8|**7.9**\n\n>**8. Transformer backbone**\n\nWe provide experiments using different transformer backbones. \nImageNetC|Source|Tent|CoTTA|ViDA|Ours\n---|---|---|---|---|---\nMobile-ViT-S|75.28|75.61|75.72|75.27|**74.28**\nSwinTrans.-B|59.26|73.17|46.84|57.84|**46.56**\n\n>**9. Mask**\n\nThough our method employs a masking strategy, it is used solely to control the difficulty of prediction, and thus does not require a projector during the masking process or a decoder to reconstruct the masked regions, and it is relatively free from instability.\n\n>**10. Overconfidence**\n\nWe provide a table for calibration error analysis. \nImageNetC|Tent|ViDA|Ours\n---|---|---|---\nECE (low)|12.6|14.6|**8.7**\n\n>**11. MoCo-v3**\n\nThanks for your suggestion to utilize MoCo-V3, not trained with labels on source domain.\nImageNetC|Source|Tent|CoTTA|SAR|PETAL|Ours\n---|---|---|---|---|---|---\nMoco-v3|76.5|76.6|78.2|75.4|78.2|**65.7**", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Lack of novelty. the idea of consistency learning between strong and weak augmentation has been thoroughly explored by existing TTA works like [1,2]. Meanwhile, these methods are not discussed or compared.\n1. Contrastive Test-Time Adaptation\n2. Revisiting Realistic Test-Time Training: Sequential Inference and Adaptation by Anchored Clustering Regularized Self-Training\n\n\nLack of important details in method details like foreground masking in figure 10, where does the paper introduce foreground masking? \n\nThe propose method may lead to biased attention towards the background content, compared to source, as evidenced by the 3rd row of Figure 8.", "theoretical_claims": "NA", "experimental_designs_or_analyses": "Missing comparisons with strong baseline for CTTA:\n1. EATA 2. Roid 3. Test-Time Ensemble via Linear Mode Connectivity\n\nLack of important ablation and discussion about the key hyperparameters, N and M_N in Eqn. (3).\n\nInefficiency in computation compared to the entropy-based method, and potentially much higher memory consumption by using more bp per samples.\n\nLack of justification of resizing images to 384x384 on CIFAR, which significantly increases the computation burden but seems only beneficial to the proposed method that requires masking.\n\nLimited evaluations on only ViT-Base. Suggest to add more evaluations on architectures like MobileViT, SwinTransformer.\n\nRandom masking without a projector can be unstable, but the paper's results do not show the variance in performance.", "relation_to_broader_scientific_literature": "NA", "essential_references_not_discussed": "The work would benefit from further exploration and discussion of key references.", "other_strengths_and_weaknesses": "The paper claims to address overconfidence but provides no numerical experiment on metrics like false positive rate or ECE.", "other_comments_or_suggestions": "I would raise the score if the authors address all concerns", "questions_for_authors": "Questions: Is this fully TTA? More comparisons on ViT weights that is not pre-trained with MAE, like Moco-V3, to support this.\n\nOverall, the technical contribution is limited and some key technical details are unclear with insufficient evaluations and ablations.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1DFw93Mifs", "reviewer": "Reviewer_MEbz", "summary": "This paper proposes a masked consistency loss (MCL) and entropy ranked loss (ERL) based learning mechanism for continual test-time adaptation (CTTA).\nThe MCL involves incrementally masks the images for data augmentation, and the ERL contributes in ensuring that the entropy of predictions with a low masking ratio is lower than those with a high masking ratio.\nThe proposed approach utilizes the self-attention structure to cluster similar content in order to mask the content.\nExperiments on ImageNet-to-ImageNetC, CIFAR10-to-CIFAR10C, and CIFAR100-to-CIFAR100C benchmarks using ViT-B/16 architecture.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your positive reviews and valuable suggestions. We address your concerns and questions below in detail.\n>**1. Values of M and N involved in mask ratios and chains**\n\nA1. We provide the ablation test regarding hyperparameters of the mask. Although the best accuracy was achieved when N=3, we used the combination of M_N ={0,5%,10%} to strike a balance between sensitivity to the masking ratio, increased computational complexity, and accuracy.\nM_N (N=1)|{0,5%}|{0,10%}|{0,15%}\n---|---|---|---\nError|40.6|39.7|39.5\nM_N (N=2)|{0,5%,10%}|{0,10%,20%}|{0,15%,30%}\nError|39.4|39.2|39.4\nM_N (N=3)|{0,5%,10%,15%}|{0,10%,20%,30%}|{0,15%,30%,45%}\nError|38.9|39.4|40.0\n\n>**2. Details of incremental masking and its references**\n\nA2. Our research is inspired by [R2A], which demonstrates that the self-attention mechanism in ViTs naturally clusters tokens with similar contexts, merging semantically similar tokens. Building on this, we follow the implementation of [R2B], which improves ViT efficiency by omitting background tokens. Specifically, for TTA, we compute attention scores by averaging the similarity between the class token's query and the image tokens' keys across all heads in the final multi-head self-attention layer. Tokens with high attention scores are identified, and forward passes for the corresponding foreground tokens are selectively omitted. Based on the findings of [R2A] and [R2B], we confirm that efficient masking is achievable through the self-attention structure. By progressively masking foreground regions, we control the prediction difficulty and provide a structured approach to address the challenges of continual TTA.\n\n[R2A] Token Merging: Your ViT But Faster, ICLR2023.  \n[R2B] The Role of Masking for Efficient Supervised Knowledge Distillation of Vision Transformers, ECCV2024.\n\n>**3. Experiments on CNNs**\n\nA3. Thank you for suggesting experiments on CNNs. While our method leverages the powerful self-attention mechanism in ViTs, it can be easily extended to other networks as long as the difficulty can be structured through explicit mask chaining. To explore this, we provide two additional experiments: one that uses the activation of the final CNN feature map (FA), and another that adjusts masked pixels using Grad-CAM. For CIFAR10-C, we present results based on EcoTTA using WRN-28, and for CIFAR100-C, we report results from BECoTTA using WRN-40.\nDataset|EATA|EcoTTA|BECoTTA|Ours(FA)|Ours(Grad-CAM)\n---|---|---|---|:---:|:---:\nCIFAR10C|18.6|16.8|-|16.9|**16.5**\nCIFAR100C|37.1|36.4|35.5|**34.5**|34.6\n\n>**4. Hyperparameter $\\lambda$**\n\nA4. We also share the concern regarding tuning hyperparameters on the test set in TTA. In fact, there exist combinations of lambda and margin that yield higher performance, but we use 1 and 0 as default values for lambda and margin, respectively. Additionally, to verify the robustness of our method to hyperparameters, Fig.12 highlights that our method maintains consistent performance across a wide range of values rather than being sensitive to specific settings.\n\n>**5. Margin $m$**\n\nA5. The margin is related to whether entropy minimization is applied to each sample. A higher margin $m$ leads to entropy minimization for more samples, enabling faster adaptation, while a lower $m$ prioritizes stability. This corresponds to the stability-plasticity trade-off commonly observed in continual learning. In line with the goal of our proposed method, we adopt a margin value of 0 to mitigate overconfidence.\nm|0|0.1|0.2|0.3\n---|---|---|---|---\nError|39.2|38.8|38.8|38.9\nECE|8.7|10.2|10.8|11.0\n\n>**Weakness**\n\n- [Mask chaining mech.] Please refer to A2. \n- [CNN experiments] Please refer to A3. \n- [Hyperparameter M and N] Please refer to A1. \n- [Self-attention structure limitation for ViT] While our method leverages the self-attention mechanism, it is not limited to standard ViT architectures. As demonstrated in Response 8 to Reviewer f4Hw, our approach is applicable to various transformer backbones such as MobileViT and SwinTransformer. Furthermore, we also provide CNN-based experiments in Response A3.\n\n>**Minor**\n\n- Thanks for your suggestions on typos and references. We will revise it for better readability.", "claims_and_evidence": "Some of the claims are not well supported or lack evidence. Refer to Weaknesses and Questions.", "methods_and_evaluation_criteria": "The proposed approach shows improved performance for the compared ViT-B/16 architecture.\nHowever, the lack of experiments on CNNs, such as WideResNet-28,  ResNeXt-29 and ResNet-50 that have been widely used by the state-of-the-art continual TTA methods, hurts the applicability and utility of the proposed approach.\nMoreover, several state-of-the-art approaches, such as EcoTTA [1], EATA [2], BeCoTTA [3], and PETAL [4], are missing from experimental comparisons.\n\nReferences:\n1. Song, Junha, et al. \"Ecotta: Memory-efficient continual test-time adaptation via self-distilled regularization.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n2. Niu, Shuaicheng, et al. \"Efficient test-time model adaptation without forgetting.\" International conference on machine learning. PMLR, 2022.\n3. Lee, Daeun, Jaehong Yoon, and Sung Ju Hwang. \"BECoTTA: Input-dependent Online Blending of Experts for Continual Test-time Adaptation.\" International Conference on Machine Learning. PMLR, 2024.\n4. Brahma, Dhanajit, and Piyush Rai. \"A probabilistic framework for lifelong test-time adaptation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.", "theoretical_claims": "NA", "experimental_designs_or_analyses": "Refer to the weaknesses and questions.", "relation_to_broader_scientific_literature": "The key contribution of this paper is utilizing well established ranking loss using an incremental masking along with mask consistency loss which seems to improve CTTA.", "essential_references_not_discussed": "Recent state-of-the-art methods are not discussed, as well as the comparisons are missing.\n1. Song, Junha, et al. \"Ecotta: Memory-efficient continual test-time adaptation via self-distilled regularization.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n2. Lee, Daeun, Jaehong Yoon, and Sung Ju Hwang. \"BECoTTA: Input-dependent Online Blending of Experts for Continual Test-time Adaptation.\" International Conference on Machine Learning. PMLR, 2024.", "other_strengths_and_weaknesses": "**Strengths**\n* An interesting idea to improve data augmentation via incremental masking for Continual TTA\n* The experiments show improved performance on ViT\n\n**Weaknesses**\n* An elaborate detail about explicit mask chaining mechanisms seems to be lacking. This is a key element of the proposed approach.\n* Lack of experiments on CNNs, such as WideResNet-28, ResNeXt-29, and ResNet-50, that have been widely used by the state-of-the-art continual TTA methods, hurts the applicability/utility of the proposed approach.\n* Details about hyperparameters such as \"M and N involved in mask ratios and mask chains\" are missing\n* Exploiting the self-attention structure limits the applicability to ViT architectures.", "other_comments_or_suggestions": "1. Line 325: Section 4.2 is cited inside the same section.\n2. Include references to hyperparameter details and tuning in the main paper.\n3. This sentence is not grammatically correct: \"The principal idea is to explicitly enhance the prediction complexity of a sample by masking objects that domain invariant features.\"", "questions_for_authors": "1. What are the values of M and N involved in mask ratios and mask chains? Are these values tuned?\n2. How does incrementally masking content involve only the content containing the domain-invariant information? How is the self-attention structure utilized for doing this?\n3. Do they authors have any experiments on CNNs, such as WideResNet-28,  ResNeXt-29 and ResNet-50, that have been widely used by the state-of-the-art continual TTA methods?\n4. How is the hyperparameter λ in Equation 5 tuned? In Appendix E, is the error computed on the test split itself, on which the performance is reported?\n5. It is mentioned \"we set λ = 1 and m = 0\", and the Figure 12 shows that the margin does not matter. Why is it so, do the authors have any explanation or intuition about it?\n6. Is explicit mask chaining idea adopted from other existing literature? If so, a citation with better description will improve clarity.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iHQ0iBv0RZ", "reviewer": "Reviewer_sBwW", "summary": "This paper proposes a novel Ranked Entropy Minimization method for test-time adaptation. While leveraging entropy as a supervision signal may risk model prediction collapse, the authors address this challenge by first constructing an explicit masking chain with varying masking ratios on the original images. They introduce two critical constraints: enforcing consistency between predictions of highly masked images and those with lower masking ratios, and ensuring that the entropy of predictions for low-masking-ratio images remains consistently lower than that of high-masking-ratio images. These constraints effectively enhance the model’s cross-domain generalization capability while updating only a small number of parameters. The authors validate the effectiveness of their approach through comprehensive experiments across multiple benchmarks, demonstrating significant improvements in stability and adaptability under continual test-time adaptation scenarios.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your constructive comments and positive reviews. We address your questions below in detail.\n\n>**1. Sampling through image masking vs. sampling from a queue using active learning**\n\nA1. The active approach of capturing accurate samples and adapting using selected samples is meaningful in that it prevents the model from being degraded by uncertain samples. However, in this paper, we are concerned that sample selection based on active learning may heavily depend on the initial model and may be difficult to generalize due to the need to define an entropy threshold against varying training environments. Instead, we explicitly control the learning difficulty while training on all samples equally, aiming for stable operation and maintaining the goal of continual adaptation.\n\n>**2. Failure case analysis**\n\nA2. Thanks to your insightful suggestion, we analyzed the discrepancy between the output of original and masked images using Total Variation Distance (TVD) for two domains where our method achieved significant performance improvements (Gaussian and Shot noise) and two domains where it showed relatively lower performance (Brightness and JPEG). Interestingly, the domains with successful performance gains, e.g., Gaussian and Shot noise, exhibited larger differences in the predicted probability distributions with and without masking. One possible interpretation is that, for relatively easier domains, a small discrepancy between the predicted distributions of the original and masked images may lead to a low loss, which in turn could reduce the adaptation speed. This led us to the insight that adjusting the loss magnitude according to the domain gap may further aid adaptation. We sincerely appreciate your suggestion, which enabled a deeper understanding of both the strengths and limitations of our method.\nCIFAR100C|Gaussian|Shot|Brightness|Jpeg\n---|:---:|:---:|:---:|:---:\nTVD (first 50%)|5.54$\\pm$1.36|3.44$\\pm$1.38|1.69$\\pm$0.45|2.90$\\pm$0.95\nTVD (last 50%)|5.03$\\pm$1.38|3.82$\\pm$0.69|1.55$\\pm$0.41|2.59$\\pm$0.78", "claims_and_evidence": "The paper presents a clear and compelling argument. The authors observe that relying solely on entropy minimization as a supervisory signal can lead to model collapse in certain scenarios due to convergence to trivial solutions, where probability distributions collapse to a single point in polar coordinate representations. Inspired by Zeno’s paradox of Achilles and the tortoise, the authors propose a novel method based on entropy constraints for masked images to address this issue. The study is strengthened by extensive visualizations and insightful observations that illustrate the research motivation and theoretical foundations.", "methods_and_evaluation_criteria": "This paper tightly integrates observations with the motivation to propose an insightful and effective algorithm. By constructing two complementary constraints, the authors successfully address the challenge of models collapsing into trivial solutions when relying solely on entropy minimization. The introduction of masking on critical regions of images generating high-entropy data while imposing constraints, which ensures the preservation of global semantic comprehension capabilities. Furthermore, visualizations of the masking strategy demonstrate that obscuring object-related regions effectively establish a hierarchy of prediction difficulty, thereby preventing model collapse.", "theoretical_claims": "I have reviewed the authors' formulation of Explicit Mask Chaining and the two constraints, and they align with the theoretical descriptions provided by the authors.", "experimental_designs_or_analyses": "This paper thoroughly validates the effectiveness of Ranked Entropy Minimization in continuous test-time adaptation tasks through comprehensive experiments, demonstrating significant improvements over existing methods on benchmark datasets such as ImageNet-C, CIFAR10-C, and CIFAR100-C. The visualization analysis reveals REM’s core mechanism: the self-attention-based foreground masking strategy precisely localizes object regions, forcing the model to learn global semantics through mask consistency loss and thereby avoiding model collapse caused by over-reliance on local features.", "relation_to_broader_scientific_literature": "[1] published at ICML 2024, primarily investigates the application of active learning in test-time adaptation. Both this study and the current paper highlight the positive role of challenging samples in enhancing model domain generalization and propose insightful methods from different perspectives.\n\n[1]. Gui S, Li X, Ji S. Active Test-Time Adaptation: Theoretical Analyses and An Algorithm[C]//The Twelfth International Conference on Learning Representations.", "essential_references_not_discussed": "The references cited in this paper are sufficient.", "other_strengths_and_weaknesses": "This paper identifies a critical issue in test-time adaptation through a fascinating observational experiment: how to avoid model collapse caused solely by entropy constraints. The authors ingeniously propose two constraints: ensuring that the entropy of highly masked images is greater than that of lightly masked ones, and guiding the probability distribution of highly masked images to align with that of lightly masked images. These constraints effectively prevent prediction collapse. The motivation of the paper is clear, and the proposed method is highly effective.", "other_comments_or_suggestions": "None.", "questions_for_authors": "1. What are the advantages of constructing challenging samples through image masking in this paper compared to selecting challenging samples from a queue using active learning?\n\n2. In Table 3, some experimental results show significant differences compared to the current state-of-the-art (SOTA) methods (e.g., for brightness and JPEG). For certain types of image corruptions, if the masking ratio is too high, could it lead to a situation where the output probability distribution of high-masking-ratio samples consistently fails to align with that of low-masking-ratio samples? I believe this analysis is crucial, as it is directly related to the masking ratio set during Explicit Mask Chaining.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gA44EuClLI", "reviewer": "Reviewer_wsjg", "summary": "This paper addresses the model collapse issue in CTTA. Specifically, it aims to reconcile the trade-off between fast but unstable EM methods and stable yet computationally expensive CR methods. The authors propose REM, a novel EM-based approach incorporating a progressive masking strategy. This strategy gradually adjusts prediction difficulty by structuring entropy sequentially, thereby aiming to enhance both stability and adaptability.\n\n## update after rebuttal\nThe masking augmentation strategy proposed by the authors seems reasonable and well-motivated. That said, this approach relies on the assumption that ranking should be preserved, which can also be evaluated using more standard augmentation techniques such as varying rotation angles or crop levels. Given this, it is challenging to attribute the observed effects specifically to the proposed ranking structure. Furthermore, the performance gains over prior methods that combine entropy loss with data augmentation are relatively modest. For these reasons, I would like to retain my original score.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your constructive suggestions. We address your questions below.\n>**1. Model collapse problem**\n\nModel collapse in entropy minimization methods is a well-known issue, but we would like to emphasize that it remains an unresolved challenge in achieving stability during TTA process. Fig. 2 shows the sensitivity of entropy minimization to training strategy (learning rate), suggesting that even small variations can lead to model collapse in TTA. Fig. 6 also shows the model’s sensitivity to various learning rates. Since entropy minimization operates within a narrow range where collapse does not occur, it requires careful tuning. In contrast, our proposed method shows stable adaptation across different training strategies highlighting its relative robustness against model collapse, and we provide the comparison table with prior works.\nMethod|EATA|SAR|DeYO|Ours\n---|---|---|---|---\nImageNetC|41.3|45.2|42.9|**39.2**\n\n>**2. Explicit mask chaining augmentation**\n\nAugmentation-based EM methods[a,b], which employ diverse data augmentations, apply stochastic augmentations, e.g., color jitter and random affine, similar to CoTTA and its variants. In contrast, our method is distinguished by applying interpretable, sample-specific, masking-based data augmentation. By designing a ranked prediction distribution, we offer an intuitive and simple solution that progressively refines the prediction distribution while preserving the ranking order. We provide the comparison table with [a,b]. \nMethod|ROID[a]|CMF[b]|Ours\n---|---|---|---\nImageNetC|41.4|40.7|**39.2**\n\n>**3. Computational resource compared with TENT**\n\nIn Table5, our method requires approximately twice the training time (_not ten times_, tent: 8min vs ours: 17min) compared to TENT; however, it achieves higher performance while reducing the computational cost to 1/3 compared to Continual-MAE (59min). EM methods are efficient but raise concerns about model collapse, whereas mean teacher-based methods offer stability at the cost of high computational overhead. Our goal is to integrate the strengths of both approaches to achieve a balanced solution from the viewpoint of both computational efficiency and classification performance.\n\n>**4. Theoretical explanation**\n\nWhile our method provides a solution to mitigate model collapse, it does not theoretically guarantee complete prevention. Therefore, we conducted extensive experiments to empirically validate the effectiveness of our approach. In Fig. 3, the analysis of entropy and accuracy with respect to different masking ratios supports the soundness of our method’s intuition. The visualizations of masked images in Fig. 7 show that masking is applied as intended. Lastly, Fig. 8 shows class attention map visualizations, revealing that entropy minimization methods often rely on class-irrelevant local pixels as the basis for predictions, leading to suboptimal solutions. \n\n>**5. Model collapse**\n\nWe observed that entropy minimization can lead to predictions collapsing into a single class, regardless of the input. To mitigate this issue, it is important to focus on contextual information as the basis for prediction while addressing the influence of incorrect supervision signals. Our method alleviates model collapse by improving relative relationships rather than relying on direct supervision signals, through relational algorithms: our MCL, which focuses on contextual information, and our ERL, which preserves the ranking order of entropy based on prediction difficulty.\n\n>**6. Avoid overconfidence**\n\nIn Reviewer MEbz’s A5, we investigate the effect of the margin in the entropy ranking loss. As the margin increases, the influence on the ranking diminishes, leading to a rise in ECE. This suggests that as the weight on direct entropy minimization decreases, the overconfidence issue is alleviated.\n\n>**7.  Highly domain-shifted scenario**\n\nIn addition to the CTTA scenario of the main paper, we conducted experiments under standard protocols for various domain shifts in Appendix C and D, including Online TTA and Vision-Language Model(VLM)-based TTA scenarios. Our method shows successful adaptation not only on datasets where corruptions are applied to the source domain but also in cases involving substantial domain shifts, e.g., adapting from ImageNet to ImageNet-R, ImageNet-V2, and ImageNet-Sketch. We also validate ours using a realistic benchmark with ImageNet-3DCC (please, refer to Reviewer VSix’s A4).\n\n>**8. Failure case**\n\nWe establish a ranking structure using 10% and 20% masked images. However, when the object is entirely masked, there is a possibility that some predictions may become biased toward the background, as shown in the bottom-right visualization of Fig. 8. To alleviate this issue, we propose Entropy Ranking Loss in Sec. 3.4. Moreover, please refer to reviewer sBwW’s A2 regarding the failure case related to distributional discrepancy analysis.\n\n>**Minor**\n\nWe revised the arrow in Appendix C to point upward.", "claims_and_evidence": "**Strengths:**\n\n- Clearly visualizes the model collapse phenomenon and convincingly demonstrates the effectiveness of the proposed REM method through informative graphs and illustrations.\n\n**Weaknesses:**\n\n- The trivial solution-induced model collapse problem in EM-based methods has already been analyzed extensively in prior works such as EATA, SAR, and DeYO. This diminishes the novelty of the analytical contribution in Section 2.\n- The paper lacks a comparative analysis between the proposed masking-based augmentations and diverse existing augmentation techniques, thus insufficiently supporting the claimed novelty.", "methods_and_evaluation_criteria": "**Strengths:**\n\n- Employs standard benchmarks for CTTA evaluation and widely-adopted ViT models, thereby ensuring credibility and comparability of results.\n\n**Weaknesses:**\n\n- Although REM claims a novel mechanism to prevent model collapse, its fundamental idea of diverse augmentations for entropy minimization has already been explored in existing literature [a,b].\n\n> [a] Marsden et al., \"Universal test-time adaptation through weight ensembling, diversity weighting, and prior correction,\" WACV, 2024.\n> \n\n> [b] Lee & Chang, \"Continual momentum filtering on parameter space for online test-time adaptation,\" ICLR, 2024.\n> \n- Table 5 reveals that REM requires more than ten times the computational resources compared to TENT, questioning its practical viability.", "theoretical_claims": "**Weaknesses:**\n\n- The theoretical explanation supporting REM's capability to prevent model collapse, as mentioned in Section 5, is insufficiently developed.\n- The paper lacks a clear logical analysis explaining why the proposed progressive masking strategy prevents model collapse.\n- Although the authors assert that sequential entropy structuring avoids overconfidence in difficult samples, they do not rigorously analyze how sequential entropy ordering directly mitigates model collapse.", "experimental_designs_or_analyses": "**Strengths:**\n\n- The effectiveness of various masking strategies (Fig. 10) and the impact of removing specific loss terms (Fig. 9) were thoroughly analyzed, experimentally demonstrating that MCL and ERL significantly contribute to preventing model collapse.\n\n**Weaknesses:**\n\n- Lack of evaluation regarding the robustness of the proposed object-focused masking strategy in highly domain-shifted scenarios. Additional experiments or analyses are necessary to clearly establish its effectiveness under extreme domain shifts.\n- Inadequate analysis of failure cases. The manuscript does not provide an experimental investigation into situations where REM fails, which is essential for a comprehensive understanding of method limitations.\n- A comparison with EM approaches using varying intensities of augmentation is necessary to demonstrate the relative advantage of the proposed masking augmentation more explicitly.\n- Although Appendix C reports accuracy values, the use of downward arrows in the tables introduces confusion and could mislead readers regarding the interpretation of experimental results. Clarifying the representation method would improve the manuscript's readability.", "relation_to_broader_scientific_literature": "see above", "essential_references_not_discussed": "I keep up with the literature in this area.", "other_strengths_and_weaknesses": "see above", "other_comments_or_suggestions": "see above", "questions_for_authors": "see above", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yyVfAFm4gQ", "reviewer": "Reviewer_VSix", "summary": "This work studies continual test-time adaptation and proposes a method to tackle it. In particular, based on experimental motivation on how entropy minimization collapses under continual TTA to predicting the same class, they propose to optimize the cross entropy between the prediction of the model on two masked versions of the input. Experiments are conducted on ImageNet-C, CIFAR10/100-C showing consistent performance gain under the provided baselines.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive feedback and insightful suggestions. Below, we provide detailed responses to your questions.\n\n>**1. Computational intensive for mask**\n\nA1. The token-wise attention used to compute the mask is applied only to the final self-attention layer and accounts for a very small portion of the overall forward pass. We provide an analysis of the mask generation time  per batch with respect to $N$ in the table below:\nForward pass|Mask computation ($N=1$)|Mask computation ($N=2$)\n:---:|:---:|:---:\n7.012$\\pm$0.390ms|0.091$\\pm$0.008ms|0.134$\\pm$0.007ms\n\nAlthough our method relatively requires more total time than entropy minimization methods, it achieves higher accuracy with lower computational cost compared to recent state-of-the-art CTTA methods based on consistency regularization. Our goal is to find a compromise between the efficiency of entropy minimization and the stability of consistency regularization methods. Positioned in between these two paradigms, our method achieves the highest classification performance with balanced efficiency.\n\n>**2. Additional baseline**\n\nA2. Thanks to your suggestion, we provide the following table comparing our method with the additional baselines:\nMethod|EATA|SAR|Ours\n---|---|---|---\nImageNetC|41.3|45.2|**39.2**\n\n>**3. Ablation for mask parameter**\n\nA3. We added an ablation study on masking. As a trade-off between computational efficiency and accuracy, we utilize masks with ratios of {0, 10%, 20%}, and the results regarding the lambda parameter are provided in Appendix E.\nM_N (N=1)|{0,5%}|{0,10%}|{0,15%}\n---|---|---|---\nError|40.6|39.7|39.5\nM_N (N=2)|{0,5%,10%}|{0,10%,20%}|{0,15%,30%}\nError|39.4|39.2|39.4\nM_N (N=3)|{0,5%,10%,15%}|{0,10%,20%,30%}|{0,15%,30%,45%}\nError|38.9|39.4|40.0\n\n>**4. Experiment on realistic TTA**\n\nA4. Thank you for suggesting an evaluation under realistic scenarios to verify the practical applicability of our method. The table below presents the experimental results on the ImageNet-3DCC [A] dataset under the time-constrained protocol [B]. We compare EATA and our proposed method using ViT-B/16. EATA requires 2.41$\\times$ the time when $C(g)=1$, while REM requires 5.10$\\times$ the time. Therefore, in the *episodic* scenario, where the model is re-initialized for each domain, REM shows lower performance than EATA due to relatively slower adaptation. However, in the *continual* scenario, where domains are learned sequentially without model re-initialization, our method achieves higher performance due to the accumulation of learned knowledge and demonstrates stable adaptation across domains.\nImageNet-3DCC|Depth of field|Noise|Lighting|Weather|Video|Camera motion\n:---:|:---:|:---:|:---:|:---:|:---:|:---:\nEATA-Episodic|31.5|40.7|55.6|54.7|57.0|46.3\nOurs-Episodic|32.7|43.1|59.7|53.2|61.6|50.0\nEATA-Continual|30.0|39.9|57.2|56.3|62.2|53.0\nOurs-Continual|31.6|38.9|56.2|51.2|59.2|46.3\n\n[A] 3D Common Corruptions for Object Recognition, CVPR 2022.  \n[B] Evaluation of Test-Time Adaptation Under Computational Time Constraints, ICML2024.\n\n>**5. Practical TTA evaluation**\n\nA5. We also present experiments on CIFAR10C under practical TTA protocol [C] in the following table. These results confirm that our method operates robustly across various realistic scenarios as suggested.\nCIFAR10C|CoTTA|ViDA|Continual-MAE|Ours\n---|---|---|:---:|---\nError(%)|79.2|27.4|15.7|**14.2**\n\n[C] Robust Test-Time Adaptation in Dynamic Scenarios, CVPR 2023.\n\n>**6. Details in supervised learning**\n\nA6. The supervised result refers to the outcome obtained by training on the test set using cross-entropy loss with access to target labels in an online manner. Since target labels are not accessible during test time in TTA, this serves as an upper bound of the adaptation performance.\n\n>**Minor**\n\nThank you for pointing out the typo. We have corrected the reference from Sec 4.2 to Table 4 and fixed the typo accordingly.", "claims_and_evidence": "While the provided experiments generally support the claims of this work, the only claim that is not fully supported is the efficiency claim. The summation in Equation (3) and (4) requires 2*N forward passes. This makes the adaptation either very slow (under fixed computational budget), or very computational intensive. Further, it is Also not clear how costly it is to construct the N masks for each received input image at test time. While Table 5 shows that REM requires only 3 forward passes, extra discussion on the computational requirement of REM is necessary.", "methods_and_evaluation_criteria": "While the proposed method is evaluated on ImageNet-C and CIFAR10/100-C, it is also important to report the results on the more realistic benchmark ImageNet-3DCC [A]\n\n[A] 3D Common Corruptions for Object Recognition, CVPR 2022.", "theoretical_claims": "There are not theoretical claim in this work", "experimental_designs_or_analyses": "While I really like the experimental analysis presented in this work (especially the ones in Figure 3), certain experiments are missing and it would make the paper much stronger to include them:\n\n(1) Missing baselines: There are a couple of strong missing baselines in the comparisons tables such as EATA  and SAR. It is important to compare against both method in the main paper.\n\n(2) Missing ablation: There are two main missing ablations in this work: analyzing the impact $\\lambda$ and the impact of $M$.\n\n(3) Computational budgeted evaluation: Since the proposed method requires additional forward passes making it more computationally intensive, it is important to compare different methods (especially the efficient EATA) under computational time constraint settings [B].\n\n(4) Evaluation schemes: while this work mainly focuses on continual TTA, it is important to show experiments under different evaluation protocols; namely practical TTA [C].\n\n[B] 3D Common Corruptions for Object Recognition, ICML 2024.\n\n[C] Robust Test-Time Adaptation in Dynamic Scenarios, CVPR 2023.", "relation_to_broader_scientific_literature": "The findings of this work are related to the TTA literature", "essential_references_not_discussed": "- 3D Common Corruptions for Object Recognition, CVPR 2022.", "other_strengths_and_weaknesses": "Please refer to the earlier sections of my review.", "other_comments_or_suggestions": "There are a couple of typos in that paper:\n\n1) In line 325-326: \" Section 4.2 presents the performance on the 5 unseen domains after training on 10 domains.\" I think here you should refer to the Table, rather than referring to the section you are at.\n\n2) In line 418-419: \"Compared to the re- cent state-of-the-art, Continual-MAE,\", I believe the first comma should be replaced with ;.", "questions_for_authors": "In addition, I have the following question to be clarified:\n\n- How are the supervised learning results obtained? As far as I know, ImageNet-C does not have any training sets.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Five experts in the field reviewed this paper, and their recommendations are two Weak Rejects, one Weak Accept, and two Accepts after the rebuttal phase. Overall, the reviewers appreciated the paper because it provides a well-motivated solution to a challenge of model collapse under entropy minimization in continual TTA. Entropy ranking via progressive masking is intuitive and empirically validated. The proposed masked consistency and entropy ranking loss stabilize adaptation during deployment without requiring access to source data or complex regularization.

The reviewers raised some concerns regarding the following weaknesses:
- The proposed REM approach (e.g., strong/weak augmentation) has incremental contributions in TTA, although the structured ranking of entropy is an original component.
- REM requires multiple forward passes per sample, leading to higher computational cost versus TTA methods like TENT. Computational efficiency and scalability remain a concern for resource-constrained or real-time deployments.
- Improvements over standard augmentation strategies (e.g., random cropping, affine transformations) is unclear. How much of the gain is due to the specific design versus broader augmentation and entropy combinations.
The authors are encouraged to consider these comments when revising the paper.

I recommend accepting this paper based on the reviewers’ feedback and the authors’ detailed rebuttal. The rebuttal addresses most major concerns with extended experiments, analysis, and discussions of REM limitations. While the authors provided some intuitions, the work does not include strong theoretical justification to support the mitigation of model collapse. While some theoretical depth is still lacking, the method’s effectiveness and relevance to an active research area justify its inclusion. The paper can make valuable contribution to the TTA literature with minor revision. The proposed method is highly integrated with ViT models using self-attention-based foreground masking. The authors extended the approach to CNNs via Grad-CAM and attention-based feature maps. The empirical validation is convincing based on ImageNet-C, CIFAR10-C, CIFAR100-C, and ImageNet-3DCC in continual TTA under both TTA settings. The authors’ extended results include multiple ViT backbones and some CNN-based variants. Ablation studies show robustness to hyper-parameter values. Additionally, REM improves both classification accuracy and calibration, showing that it may be suitable for real-world deployment.
Decision: Accept (poster)