Entropy is not Enough for Test-Time Adaptation: From the Perspective of Disentangled Factors
Authors: Jonghyun Lee
Dahuin Jung
Saehyung Lee
Junsung Park
Juhyeon Shin
Uiwon Hwang
Sungroh Yoon
Venue: ICLR 2024 spotlight (2024)
Keywords: Test-time adaptation
Roustness
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: We first address the limitations of relying solely on entropy as a confidence metric for TTA. Based on the observation, we introduce a new TTA method called DeYO, which leverages our proposed confidence metric, PLPD.

Abstract: Test-time adaptation (TTA) fine-tunes pre-trained deep neural networks for unseen test data. The primary challenge of TTA is limited access to the entire test dataset during online updates, causing error accumulation. To mitigate it, TTA methods have utilized the model output's entropy as a confidence metric that aims to determine which samples have a lower likelihood of causing error. Through experimental studies, however, we observed the unreliability of entropy as a confidence metric for TTA under biased scenarios and theoretically revealed that it stems from the neglect of the influence of latent disentangled factors of data on predictions. Building upon these findings, we introduce a novel TTA method named Destroy Your Object (DeYO), which leverages a newly proposed confidence metric named Pseudo-Label Probability Difference (PLPD). PLPD quantifies the influence of the shape of an object on prediction by measuring the difference between predictions before and after applying an object-destructive transformation. DeYO consists of sample selection and sample weighting, which employ entropy and PLPD concurrently. For robust adaptation, DeYO prioritizes samples that dominantly incorporate shape information when making predictions. Our extensive experiments demonstrate the consistent superiority of DeYO over baseline methods across various scenarios, including biased and wild. Project page is publicly available at https://whitesnowdrop.github.io/DeYO/.

Published as a conference paper at ICLR 2024
ENTROPY IS NOT ENOUGH FOR TEST-TIME ADAPTA- TION: FROM THE PERSPECTIVE OF DISENTANGLED FACTORS
Jonghyun Leek* Dahuin Jung? Sachyung Lee-
Junsung Park
Juhyeon Shin Uiwon Hwang Sungroh Yoon 13355
Department of Electrical and Computer Engineering, Seoul National University
2 School of Computer Science and Engineering, Soongsil University
3 IInterdisciplinary Program in Artificial Intelligence, Seoul National University
DDvision of Digital Healthcare, Yonsei University
5 AIIS, ASRI, INMC, and ISRC, Seoul National University
lee jh 11961100mm0 @ kr, dahuin, dahuin.jungesse aessuacc.k, hallo2218esmuacckk
(jerryray, newjh12] Sanu. ac. kr, uiwon. hwwngnnhaangyccsc..... ac.kr, aryoon8smu.ac.k
ABSTRACT
Test-time adaptation (TTA) fine-tunes pre-trained deep neural networks for unseen test data. The primary challenge of TTA limited access the entire test dataset during online updates, causing error accumulation. To mitigate it, TTA methods have utilized the model output's entropy as confidence metric that aims de- termine which samples have lower likelihood of causing error. Through experi- mental studies, however, we observed the unreliability of entropy as a confidence metric for TTA under biased scenarios and theoretically revealed that it stems from the neglect the influence of latent disentangled factors of data on predic- tions. Building upon these findings, we introduce novel TTA method named De- stroy Your Object (DeYO), which leverages a newly proposed confidence metric named Pseudo-Label Probability Difference (PLPD)). PL.PD quantifies the influ- ence of the shape of an object on prediction by measuring the difference between predictions before and after applying an object-destructive transformation. De consists of sample selection and sample weighting, which employ entropy and PLPD concurrently. For robust adaptation, De prioritizes samples that dom- inantly incorporate shape information when making predictions. Our extensive experiments demonstrate the consistent superiority of De over baseline meth- ods across various scenarios, including biased and wild. Project page is publicly available at http://whittsnnwwrrp... wwtp:/ddrr.o....bioooooo github.io/DeYOO.
INTRODUCTION
Although deep neural networks (DNNs) demonstrate powerful performance across various domains, they lack robustness against distribution shifts under conventional training (He et al., 2016; Pan & Yang, 2009). Therefore, research areas such domain generalization (Blanchard al., 2011; Gul- rajani & Lopez-Paz, 2021), which involves training models to be robust against arbitrary distribution shifts, and unsupervised domain adaptation (UDA) (Ganin & Lempitsky, 2015; Park et al., 2020), which seeks domain-invarian information for label-absent target domains, have been extensively in- vestigated in the existing literature. Test-time adaptation (TTA) (Wang al., 2021a) has also gained significant attention as a means to address distribution shifts occurring during test time. TTA lever- ages each data point once for adaptation immediately after inference. Its minimal overhead compared to existing areas makes it particularly suitable for real-world applications (Azimi et al., 2022).
Because UDA assumes access the entire test samples before adaptation, it utilizes its informa- tion on task by analyzing the distribution of the entire test set (Kang et al., 2019). Oppositely,
"Equal Contribution
1 "Corrsponndig Authors

---

Published as a conference paper at ICLR 2024
due to limited access to the entire test data concurrently, TTA struggles to accurately estimate the entire test data distribution. It leads to inaccurate predictions, and incorporating them into model up- dates results in error accumulation within the model (Arazo et al., 2020). Hence, it is vital to utilize samples that are less prone to be incorrectly predicted in order to reduce error accumulation Prior researches (Geifman et al., 2019a; Lee et al., 2022) have adopted the concept of a confidence met- ric aiming to determine trustworthy samples. Currently, maximum softmax probability (Sohn et al., 2020) and entropy (Saito et al., 2020), utilizing the model's prediction, are the most employed confi- dence metrics in label-absent tasks. Several TTA methods have also proposed entropy-based sample selection approaches to identify trustworthy samples (Niu et al., 2022; 2023). Although entropy has advantages as confidence metric, natural question arises: Can it reliably identify trustworthy samples under various distribution shifts? Notably, we found out that entropy's 's reliability largely diminishes when a severe spurious correlation shift (Beery et al., 2018) 8) exists in the dataset. To jus- tify the observation, we draw inspiration from Wiles et al. (2022) and elucidate why the sole use of entropy is less reliable for adaptation by introducing latent disentangled factors of inputs.
Based on the observation, we introduce a theoretical proposition for identifying harmful samples, those that decrease the model's ' discriminability during adaptation: sample is harmful if its predic- tion is more influenced by TRAin-time only Positively correlated with label (TRAP) factors (e.g,, background, weather) rather than Commonly Positively-coRrelatedd with label (CPR) factors (e.g.. structure, shape), even its entropy is low. TRAP factors boost training performance but decrease inference performance, attributable a discrepancy the correlation sign with labels. If predictions mainly rely on TRAP factors, there is a high risk of wrong predictions under distribution shifts.
In this paper, we present new TTA method called Destroy Your Object (DeYO), which leverages our proposed confi- dence metric, Pseudo-Label Probability Difference (PLPD), to identify harmful samples that entropy cannot detect. De uses a single image transformation that distorts the shape of objects, which is a basic element of human visual percep-
70
DeYn (Durs)
Accurucy (%)
Pre-sdapiatin
TENT
60
SAR
EATA
50
tion (Geirhos et al., 2019) and is considered a representative CPR factor. PLPD measures the extent to which the probabil- ity of pseudo-label decreases after applying the transforma- tion. A high PLPD value indicates that the CPR factor has large impact on the prediction of the model. Ye involves two key processes: sample selection and sample weighting. By incorporating PLPD an additional selection criterion, the model selects samples that are more clearly rooted in CPR factors within the same entropy level for updates. As a result, to emphasize the effect of high-confidence samples on model updates, we assign a greater sample weight to those with low entropy and high PLPD values.
10 20 30 40 50 60 70 80 90 Iteration
Figure 1: The accuracy within the worst group of the Waterbirds benchmark.
We evaluated Yo on the ImageNet-C (Hendrycks & Dietterich, 2019) benchmark, not only in the mild scenario but also in wild scenarios, which encompass mixed shifts, label shifts, and batch size I scenarios. While baseline methods typically exhibit strength in either a mild or wild scenario, De consistently outperforms the baseline methods in all scenarios. Additionally, we tested Yo on the ColoredMNIST (Arjovsky et al., 2019) and Waterbirds (Sagawa et al., 2020) benchmarks, characterized by an extreme spurious correlations shift, where conventional TTA methods failed to perform adequately (Zhao et al., 2023). In the case of the Waterbirds, as illustrated in Figure 1, baseline methods show lower performance than the pre-adaptation model, whereas Ye succeeded in showing superiority over it. Furthermore, in the case of the ColoredMNIST. Ye is the only method that exceeds random guessing (50%) in terms of the worst group accuracy. We conducted diverse analyses to validate the rationale behind De'O's performance.
Contributions. 1) We show that using entropy alone measure of confidence is insufficient for TTA. Remarkably, we observed that even (extremely) low-entropy samples can potentially dimin- ish the TTA performance. Motivated by it, we establish a new proposition for identifying harmful samples based on disentangled factors. 2) We propose a novel TTA method called DeYO, which leverages both our proposed confidence metric called PLPD and entropy. PLPD examines the influ- ence of CPR factors, which is impossible to discern through entropy alone, via the change in model predictions resulting from an object-destructive transformation applied to objects. 3) We demon- strate that DeYO significantly outperforms existing TTA methods through extensive experiments. The improvement is pronounced in more challenging wild scenarios, and specifically, on the Col- oredMNIST, Yo is the first TTA method that exceeds random guessing.
2

---

Published as a conference paper at ICLR 2024
2 REVISITING TTA: FROM THE PERSPECTIVE OF DISENTANGLED FACTORS
Sec. shows that the entropy is not enough for TTA through the following subsections: in Sec. 2.1, present observations that examine the unreliability of entropy and elucidate its inherent charac- teristics. Inspired by these insights, Sec. 2.2 offers preliminary concepts and essential notations for further analysis. Subsequently, in Sec. 2.3, provide an in-depth exploration of why relying solely on entropy may not be considered reliable as confidence metric.
2.1 MOTIVATING OBSERVATIONS
Entropy-based sample selection methods can be found in diverse tasks (Saito et al., 2020; Wang et al., 2022). When model's probabilistic output closely resembles one-hot encoding, it suggests high likelihood of a sample being drawn from the training data distribution. These samples, charac- terized by low entropy, are commonly used trustworthy samples. In the TTA literature, Niu et al. (2022; 2023) provided empirical evidence of the efficacy of entropy- based sample selection on the ImageNet-C benchmark (Hendrycks & Dietterich, 2019).
DNNs easily leverage spurious features as well as semantically meaningful features, resulting in decreased performance when these spurious correlations are prominent (Beery al., 2018; Geirhos et al., 2020). For example, butterfly images often co-occur with flowers (Singla & Feizi, 2022), leading the model misclassify the image without a flower. As highlighted by Wiles et al. (2022), spurious correlation shift is one of the crucial real-world inspired distribution shifts and commonly exists in datasets with varying degrees. Hence, is necessary to confirm the reliability of entropy during TTA in the presence of a spurious correlation shift.
In order to furnish empirical results related to spurious correlation shift, we conducted an analysis of entropy on the Waterbirds (Sagawa et al., 2020). This benchmark enables the manipulation of the degree of spurious correlation shift between class categories and background, and is classified into four categories depending on the types of classes and backgrounds. More details for the Wa- terbirds are provided Appendix E.2. Remarkably, as shown Fig. 2(a), we observed that within the worst- pents-perfoming group, samples with entropy values below the first quartile exhibit lower pre- diction accuracy compared to other intervals. This observation demonstrates that the application of entropy-based sample selection to adaptation may, instead, result in performance decline. Indeed, when we applied entropy-base based adaptation on the Waterbirds benchmark, as shown in Fig. 1, it resulted in lower performance compared the pre-adaptation model.
To visually investigate the differences between correct and wrong samples with extremely high confidence, we employ Grad-CAM (Selvaraju et al., 2017) in Fig. 2(b), (c). It reveals that correct samples primarily focus on the birds (target object), while wrong samples relatively focus on the background (spurious feature). Theoretically. Zhou al. (2021) demonstrated that even in situations with only input distribution shifts (i.e., covariate shifts), deep models also learn spurious features present in the training data. Therefore, relying solely on entropy may not be consistently reliable under distribution shifts, as it cannot distinguish whether the model focuses on the spurious feature.
2.2 PRELIMINARIES
A with parameter = In TTA, we have a model Mg trained on Dirain {(x ( ]
(0,))%), where rdrain E Xttran and grain E V. The purpose of TTA is successfully adapting Me Artea where x E geest and yfest E Y. During adapta- tion, we cannot access yrest Instead, existing TTA methods update 0 in the direction of minimizing Enta(x test), the entropy of testt
using the test data Diest {(xxest y )
C Σ (x), log (o 1=1
Ente(x) = -po(x). log px(x) =
(I)
where po (x) = softmax(Mo(x))) = (p)...)......p.. on and C' is the number of classes.
E RD is the model's output probability
Motivated by Wiles et al. (2022), we assume that there is a disentangled latent vector v(x) = (vi (x), Va, (x)) E V corresponding to an input x, where vz(x) is called i-th factor of x.
3

---

Published as a conference paper at ICLR 2024
68-
Accimey
01
07
08
05
01 -0C
Waterbirds (100%)
Waterbirds
Watechirds (100%)
Waserbards
Landbads (100%)
Entropy
(100%)
(100%)
(100%)
(100%)
(100%)
(a) Entropy level vs accuracy
(b) Grad-CAM of correct samples
(c) Grad-CAM wrong samples
Figure 2: (a) A graph that represents accuracy by entropy levels. The lowest entropy interval O~Q1 exhibits the lowest accuracy. (b) and (c) display Grad-CAM visualization of samples with correct and incorrect predictions with extremely low entropy, respectively.
For convenience, will interchangeably use and v(x), assume v; [0, 1], and focus on bi- nary classification, where y E [-1,1]. We first define two values: corrtrain and corrlest where corr corr(ytrain verain) is the correlation between the train label ytrain and the i-th factor verain corresponding to x frainn and corr = corr(y test v!est) is the counterpart to x
test Then we
can divide into four partitions based on con and cont.
V pp = (vi|corr) > 0, corr. test
> 0}, pm {va corr train > 0, corr test S 0),
(2)
Vap = (vilcom train < 0, corr. test
> 0}, Vnn (ve corr train < 0, corp. test < 0).
2.3 ENTROPY IS NOT ENOUGH
As mentioned in See. 2.1, entropy cannot be considered a reliable confidence score in situations involving spurious correlation shift. In this subsection, we validate the inadequacy of the sole use of entropy as the confidence score from the perspective of disentangled factors. Let us assume that Me is linear classifier. Then, the parameter 0 also have four partitions {6 9 pm O Pnp: 6mn) } corresponding to {Vpp, Yaa Vnp: Vnn J. Then, we can express the logit ag. probabilistic scalar output Pg: and aseudo-label y as follows:
(x(x) -Mo(x)= 0.v(x)= = Ppp P Opm Vpn + Onn Vnp + Onn Vnns
(3)
exp (x)) y= 9-1-1 1
ag (x) > 0
Px(x) (x)(x))
otherwise
where (-) is the sigmoid function. Then, in the case TTA, the following proposition holds.
Proposition 1. Let us consider pre-trained linear classifier Mg that uses the latent disentangled factors vxx of sample as input. We define a harmful sample as one that reduces the difference in the mean logits between classes when used for adaptation. A sample x E teest is a harmful sample for adaptation using entropy minimization loss if it satisfies the following condition:
(v(()) (.e)))(Retnn (Pegteet xtea [v(xtcsst
(v (xx)st)) )) < 0,
(5)
where Xtest (x((x,)) Diest, y y}, and y (1.
A detailed proof is provided in Appendix A. In the rest of this section, with Proposition 1, we will explain why the samples with low entropy can be harmful.
According to the definition of the partition of disentangled factors, the partitions of optimal param- eters 0 for the training data satisfy pp.. 0r > O, p On. I O. Since 0g shares the same sign as 07 in the early stages of adaptation, x with high-confidence pseudo-label of ý satisfies
xx) 0pp Vpp 0 Vpn pm + 0 Vup P V 0,
1 Vpp + 0pm V >>> 10mp' VVp Onn.Vnll
In other words, the elements of Vnp and Vun tend to become zero, while ypp and Vpa become the dominant factors that compose We denote Vpp Commonly Dositively-coRRrlaiie (CPR) factors and Vnn as TRAin-time only Positively-correlate with label (TRAP) factors. The expected value of (xxest) follows the relationship by the defined CPR and TRAP factors:
with label
Extout (Vpp (xtest) bestt)
> Egtest Egtest ttrr pVp (xeest) }}, test
(6)
Ex (v (x [ ) < Extentt [V (x ).
4

---

Published as a conference paper at ICLR 2024
Original images
Pixel-shuffld images
Patch-shuffled images
Occluded images Xox
X
X pas
Figure 3: Examples of a transformed image x' created by different object-destructiv transformation methods. x is an example WaterBirds.
For with highly confident y 1, Eq. 5 can be approximated as follows:
fv(x)- (Bxtettt (v [x]] Xxtr [v(x )]
R ppp (x) (1E toss (oxt (V "x (x))s))) Extret x ( (pp(x" )))
(7aa
+ Vpn (x)(x) (Bxxoo pv tost
Extee Vv (x best ]])
< 0
(7.b)
In Eq. 7, as per Eq. 6, (7.a) related CPR factors becomes positive, while (7.b) related to TRAP factors becomes negative. Therefore, xC&T indicating x with [[7.a)]] < [I((.b)]l is harmful sample even if it shows high confidence in terms of entropy. This highlights that entropy, which relies solely on the compressed information expressed as 0-v, cannot distinguish harmful samples by its value, an adaptation is performed that does not take into account CPR and TRAP factors, in extreme cases, the relative order between two class logits may even change, leading to entirely incorrect predictions. Hence, we aimed to introduce a novel confidence metric that addresses various distribution shifts in the test dataset by avoiding TRAP factors and incorporating CPR factors.
METHODOLOGGY
In Sec. 2, we highlighted the issue of using entropy that ignores the influence of disentangled factors. In Sec. 3, we propose a novel TTA method named Destroy Your Object (DeYO) that incorporates the newly proposed Pseudo-Label Probability Difference (PLPD) score account for the influence of CPR factors on the model's predictions, particularly the shape information objects. By integrating the PLPD score that enforces the consideration of CPR factors while suppressing TRAP factors, alleviate the limitations tied to the exclusive reliance on entropy. Our De consists of sample selection (Sec. 3.1) and weighting (Sec. 3.2) based on the PL.D score.
3.1 SAMPLE SELECTION
For sample selection Niu et al. (2022; 2023), they employ a well-known confidence metric: en- tropy (Eq. 1). However, elaborated in Sec. 2, where there exists substantial distribution disparity between 2 train and Xrest such as a spurious correlations shift, the entropy becomes highly dependent on the TRAP factors, undermining its reliability. To be robust against distribution shifts, it is crucial to capture CPR factors. While capturing all CPR factors is challenging, we leverage the prominent and certain CPR factor: the shape information of objects (Geirhos et al., 2019).
Incorporating the shape information of objects can be achieved through image transformations such as pixel, patch-shuffling. or center occlusion. As illustrated Fig. 3, each object destruction tech- nique possesses distinct characteristics. With pixel-shuffing. the mean color of the image is main- tained, but it becomes difficult discern both the object and the background. Patch-shuffling dis- rupts the shape of the object but preserves local information through the patches. Center occlusion allows for the preservation of the background. However, when the object is not centered or excep- tionally large, it may not fully disrupt the object's shape. We conducted experiments with all three techniques, and as illustrated in Sec. 4.3, we observed the best performance with patch-shuffing which solely eliminates the shape information of objects, opposed affecting other components.
We propose a novel sample selection strategy based on entropy and PLPD to identify reliable sam- ples for model updates. Our method employs the following sample selection criteria:
Sx(x)= (x|Ento
Teni PLPDo(x,, x') TPLPD),
(8)
5

---

Published as a conference paper at ICLR 2024
Enta(X)
Inferance
Adoptation
Po(x)
DDvo = dg (x(x) 4umaaaiyy Ent,(
Areal
Area
Mo
DLroro
Tat
0
An
(x +
an
Area
Area 4
Sy(x)
TPLPD
PLPDg
Figure 4: The overview of De De comprises sample selection (Sec. 3.1) and sample weight- ing (Sec. 3.2) mechanisms. The areas within the green box are distinguished based entropy and PLPD intervals, with Area 4 corresponding to Sx(x) Sec. 3.1.
PLPDo(x, x') = (Po(x) - px(x)))s-
where an input and x' is patch-shuffled input. Teat and TPLPD are pre-defined thresholds for entropy and PLPD, respectively, and arg max (x is pseudo-label estimated by the pre- diction of x. PLPD can be expressed as (o(0.v) 000vv ppp)pp)) under the assumptions in Sec. 2. It represents that entropy-basee sample selection does not consider the influence of CPR factors on predictions because relies on a single prediction. In contrast, PLPD assesses the influ- ence of CPR factors by quantifying prediction difference contingent the presence absence of the shape information. Therefore, the use of PLPD supplements the entropy-based sample selection that may hold samples primarily influenced by TRAP factors. This selection process necessitates only single additional forward pass, incurring negligible overhead, as it does not require additional back bock-propagation.
3.2 SAMPLE WEIGHTING
Our approach enables identified samples to make varying contributions to the model updates through sample weighting. Specifically, samples belonging to Areas 1, 2, and in Fig. 4 are excluded, yet the contribution of samples in Area 4 to the model update varies based on their reliability. Formally, we express the weighting function (x) as follows:
ag (x) = exp( (Ento Ent,)) eXp(-PLPDg(x,x))
(10)
where Ento normalization factor. The first entropy-based weighting term has been employed in existing methods (Wang al., 2021b; Niu al., 2022) and is generally effective varying benchmarks. However, based on the observed unreliability. we introduce an additional PLPD-based weighting term. This term assigns larger weight the predictions are grounded in the ob- ject's shape. We confirm that each weighting term is effective, however combining them leads to improved performance, as presented in Sec. 4.3.
3.3 OVERALL PROCEDURE DEYO
DeYO performs sample selection by exploiting only the samples belonging to So(x) (Fq. 8) and calculates the sample-wise weights (x( (Eq. 10) to prioritize samples that particularly roots its prediction in the CPR factors. Then, the overall weipte- loss is given by:
Lnevo x; 0) (x(() [(xESo(x)) Exta(x),
(11)
which combines entropy-baa based and PLPD-based terms in both selection and weighting. The algo- rithmic representation of De can be found in Algorithm I of Appendix B.
4 EXPERIMENTS
We designed experiments to answer the following questions: 1) How does DeYO perform in comparison to baseline methods across various scenarios, including biased and wild scenarios that resemble real-world situations? 2) What role does the newly proposed confidence metric, PLPD, play within Ye?O? 3) How do the different components contribute to its performance, and to what degree is DeYO influenced by hyperparameters?

---

Published as a conference paper at ICLR 2024
Table I: Comparisons with baselines on ImageNet-C at severity level 5 under mild scenario re-
garding accuracy (%). The bold value signifies the top-performing result.
Noise
Blur
Weather
Digina
Mild
Gans.
Sloc
Ingul.
Defic.
Glass
Mution
Zoomi
Snow
Frost
Fog
Bri.
Conts:
Elatic
Facl
JPEG
Arg.
ResNet-S 50 BN
2.2
299
1.8
17.9
98
14.8
22.5
16.9
25.3
24.4
58.9
5.4
16.9
20.7
31.7
18.0
MEMO
7.5
8.3
3,9
19.8
13.0
20.7
27.7
25.3
28.7
32.2
61.0
11.0
13.8
13.0
37,6
23.9
Text
29.2
31.2
30,1
28.1
27.7
41.4
49.4
47.2
41.5
57.7
674
20.2
54.5
58.5
52.4
43.1
EATA
34.9
37.1
35.8
334
33.0
47.1
50.7
51.6
45.7
60.0
68.1
44.4
57.9
6016
55,
47.8
30.6
31.3
28.5
23,5
42.2
57.5
673
57.8
54.6
43.9
(ours)
35.6
37.9m
37.
338
M.l.r 485 52.8pa
5277ra
46.4m
60.6.11
68.0at 46.1m 584.11 61.5m 557.11 486
Table 2: Comparisons with baselines on Col-
Table 3: Comparisons with baselines on Water- Birds regarding accuracy (%).
oredMNIST regarding accuracy (%)
Blased
Avg Ace
Worst-Group Ace
Biased
Avg Acc
Worst-Group Acc
ResNet-18-ee 18-BN
63.40
20.05
NesNet-SO-BN
83.16
64.90
Tent
57.06
9.80
82.95
$4.14
MEMO
63.77
6.23
MEMO
82.34
50.47
SENTRY
63,23
15.78
SENTRY
85.77
60.90
EATA
60.81
17.98
EATA
82.38
$2.38
SAR
58.37
12.36
De (ours)
78.24
67.39
De (ours)
87.42
73.92
Benchmarks, Test Scenarios, and Models. For comparison, we conducted experiments on five benchmarks: 1) ImageNet-C (Hendrycks Dietterich, 2019), a well-known TTA benchmark cate- gorized into 15 corrppiion types encompassing 5 severity levels for each type. 2) ColoredMNIST and WaterBirds, two benchmarks assessing performance under an extreme spurious correlation shift presented in the dataset, and 3) ImageNet-R (Hendrycks et al., 2021) and VisDA-2021 (Bashkirova et al., 2022), two benchmarks encompassing diverse distribution shifts due to data collected from different style domains (c.g., cartoon, sketch, etc.) compared to ImageNet-C for more challenging wild test scenarios, For test scenarios, we followed the mild scenario proposed by Wang et al. (2021a) and three wild test scenarios suggested by Niu et al. (2023). Furthermore, we propose new biased scenarios with ColoredMNIST and WaterBirds, which encompass severe distribution shifts. Regarding the choice of models, we conducted experiments using ResNet-18-BN and ResNet-50-BA (batch normalization), ResNet-50-GN (group normalization), and VitBase-LN (layer normalization), taking into consideration TTA's utilization across various normalization lay- ers. More implementation and baseline details and hyperparameters can be found in Appendix E.
assess the efficacy
4.1
MAIN RESULTS
Comparison on Mild Scenario. For the mild scenario, the comparison results on ImageNet-C are reported in Tab. 1. DeYO consistently outperforms the baseline methods across all 15 corruption types in terms of accuracy, affirming the ef- fectiveness of our approach. Notably, even when compared to EATA (Niu et al., 2022), which demonstrates outstanding per- formance in the mild scenario. DeYO exhibited a 0.8% higher performance. We also confirmed that Ye yields comparable computational efficiency EATA and SAR (Niu et al., 2023) as summarized in Appendix C).
Table 4: Comparisons with base-
lines on ImageNet-C at severity level 3 and 5 under a mixture of 15
corruption regarding accuracy (%).
Mixed Shifts
Level
Level 3
ResNet-50-GN
30.6
54.0
MEMO
31.2
54.5
Tent
34.2
33.1
EATA
38.2
56.1
SAR
38.3
57.4
DeYO (ours)
38.611.
59.2..0.0
Comparison on Biased Scenario. Furthermore, DeYO show- cased clearly remarkable performance on test benchmarks that are susceptible to being influenced by TRAP factors. As shown in Tab. 2 and 3, we observed substantial performance improve- ments of 17.43% and 4.47% respectively on ColoredMNIST and WaterBirds, which inherently include a spurious correlation shift, compared to the second best results. For the worst group accuracy on ColoredMNIST, only DeYO surpasses random guessing.
VitBase-LN
29.9
53.8
MEMO
39.1
62.1
Tent
24.1
70,2
EATA
56.4
69.6
SAR
57.1
70.7
De (ours)
59.410.1
72.1 10.00
Comparison on Wild Scenario. TTA has been demonstrated to enhance the model's robustness against domain shifts. However, its outstanding performance is often achieved under certain mild test conditions. For instance, adapting with batch of test samples featuring the same type of distribution shift. In complex real-world scenarios, test data can arrive in a more unpredictable manner. Thus, SAR has proposed three more realistic test scenarios: dynamic shifts in the ground-truth test label distribution, which leads to imbalanced distributions at each corruption, ii) a single test sample, and iii) combination of multiple distribution shifts.
7

---

Published as a conference paper at ICLR 2024
Table 5: Comparisons with baselines on ImageNet-C at severity level 5 under online imbalanced label shifts (imbalance ratio = oo) or under batch size 1 regarding accuracy (%).
Naia:
Rlur
Wiather
Digital
Label Shifts
| Gauss.
Shoc
Impul.
Defoc.
Glass
Motion
Zoom
Snew
Frost
Fog
Brit
Contr.
Easte
Pisel
JPBG
Arg.
ResNett
17.9
19.7
24.9
36.3
18.7
28.4
52.2
30.6
MEMO
20.6
18.4
17.1
12.7
36.9
40.7
46.9
69.6
36.4
19.2
32.2
53.4
31.3
* Tent
3.6
4.2
4.4
16.5
599
26.9
28.4
17.9
26.2
23
72.2
46
7.3
52.3
56.2
24.7
EATA
25.7
28.6
24.8
18.5
19.6
24.1
28.4
35,3
53.0
41.2
65.2
33.3
28.0
42.4
43.1
32.7
* SAR
15.7
16.9
35.3
10.3
20.1
15.8
29.8
21,9
44.7
34-9
71.9
46-7
6.5
52.3
56.2
96.3
DeYO lours)
42.5
44.9122
438..11
22.3.r
16.3100
41.0
1322
S2.2.
$1.5.45
19.7.374
754
$2.6102
46.9
59.3
59.3.mn
43.9
VilBase-LN
5.3
29.1
23.4
27.8
25.4
44.0
30.5
47.6
29.9
MFMO
21.6
17.4
20.6
29.6
34.4
25.0
$4.8
65.0
54.9
17.4
55.5
57.7
19.1
Test
33.9
1.8
27.2
54.8
52.9
38.6
$4.3
12.4
11.7
69.7
76.3
66.3
59.6
09.7
66.6
47.3
EATA
36.2
34.7
35,5
43.4
44.3
49.3
48.5
53.2
$3.5
62.3
72.7
18.5
55.0
64.7
62.8
49.2
SAR
-2.3
34.9
44.1
50.0
$0.5
55.6
53.1
50.7
47.2
66.2
75.2
50.3
60
67.3
65.0
54.8
DeYO lours)
53.5
3600.a2
54.6.1
57.6.r1
58.7.03
63.7
46.2
67.6
66.0
73.2.r
77.9
66.7
69.0
73.5
20.3
62.3
Nois:
Blut Miction
Wentlae
Digial
Batch Sinee
Ganss.
Shot
Impal
Defoc.
Glass
Zoem
Stow
Frost
Fog
Brit.
Contr
Elastic
Pixel
JPE3
Arg.
RooNas-30-GIN
18.0
19.8
17.9
19.8
11.4
21.4
24.9
4014
47.3
33.6
69.3
36.3
18.6
28.4
52.3
30.6
* MEMO
18.5
20.5
18.4
17.1
12.6
21.8
26.9
-40.4
47.0
34.4
69.5
36.5
19.2
32.1
53.5
31.2
Tene
3.3
4.2
4.0
16.5
5.3
27.4
30.3
17.7
24.9
2.0
72.1
46.2
7.8
52.6
56.3
24.7
FATA
24.3
27.9
25.8
17.9
17.3
28.7
20.3
44.7
44.4
40.2
71.0
44.5
27.0
46.8
55.6
36.4
18.5
15.2
28.6
34.5
DeYO (ouns)
43,0am
22.5-a1
34.7aus
41.8.a1
24.4-sx
64.5
52.2
20.7-mm
73.5
44.4.15
Via Bosc-LLN
9.5
6.8
8.2
29.0
23.5
33.9
27.1
15.9
26.5
47.2
54.7
44.1
30.5
44.5
47.8
29.9
MEMO
21.6
17.3
20.6
37.1
29.6
40.4
34.4
24.9
34.7
55.1
64.8
54.9
37.4
55.4
57.6
39.1
52.8
48.8
55.9
51.3
66.9
75.1
48.9
FATA
53.7
50.5
57.4
68.9
66.8
DeYO (ouns)
54.1..77
52.1.33
58.1.0a
31.8.11
39.5
64.2.11
53.5
(82
66.4.c
737
78.3
68.2,.1
68.9
70.8::3 63.4.97
i) Online Imbalanced Label Distribution Shifts: We compared the performance of baseline methods and DeYO in situations where the class imbalance ratio is infinity, As shown in Tab. 5, DeYO exhibited significantly better performance than the baseline methods in both ResNet50-GN and VitBase-LN. For VitBase-LN, except for Zoom where Tent (Wang et al., 2021a) showed the best performance, Ye showcased its superiority across the 14 corruption types.
ti) Batch Size 1: As depicted in Tab. 5, DeYO demonstrated superior performance even in scenarios involving a single test sample. For VitBase-LLN, Yo showcased performance im- provement of 7.8% compared to SAR and notably outperforms the baseline methods on all other corruption types.
100
Risk (%%
Entropy
Pixul PLPD
50-
Potch PLPD
Occlusion PLPD
tii) Mixed Distribution Shifts: We evaluated the performance on mixture of 15 corruption types at severity levels 5 and 3, as shown in Tab, 4. DeYO still exhibited the most superior performance while the performance improvement may not be as pronounced as in the previous two wild test scenarios.
25
50
75
100
Coverage (%)
Figure 5: The Kisk-Cooerage curve of the worst group on Waterbirds.
The significant results are consistently observed on ImageNet-C at severity level (Appendix F) and more challenging benchmarks, ImageNet-R and Visda-2021, belonging to wild test scenarios (Appendix G). The overall results verify that De provides stronger robustness against various distribution shifts. The detailed analysis of PLPD in wild scenarios is provided in Appendix L.
4.2 ROLE AND EFFECT OF PLPD
Total 50,000 samples
Discussion on Spurious Correlation. As evident from Tab. 3, on the Waterbirds benchmark characterized by extreme spu- rious correlation shifts, DeYO outperformed other baselines. The primary distinction between De and the baselines lies in the inclusion of PLPD as an additional criterion for filtering. To analyze PL.D as a confidence metric, we employ the Risk- Coverage curve (Geifman et al., 2019b), where risk denotes the error rate, and coverage denotes the percentage of input. According to Ding et al. (2020), a reliable confidence metric should exhibit a low area under the risk-coverage curve (AURC). As demonstrated in Fig. 5, entropy, in the RC curve of the worst group, yields a large AURC compared to PLPD variations, indicating its limited reliability as a confidence metric. Detailed AURC values can be found in Appendix H. As result, Yo significantly outperformed the baselines with entropy filtering.
Area 1
Area 2
42,095 samples (84.2%)
1,513 samples (3.0%)
Ace 9. 9.9%
Acc 33.75%
Area. 3
Area 4
1,426 samples (2.9%)
4,966 samples (9.9%)
33.5%
Asc: 49.1%
Figure 6: Performance and per- centage of identified samples on each area on ImageNet-C.
8

---

Published as a conference paper at ICLR 2024
48.7
46.7
48.7
Aug. Ace (56)
48.2
48.2
47.7
47.7
47.7
47.2
47.2
47.2
Pixel
Patch
Occ
2X2
3X5
6X6
0.0
0.2
0.3
0.4
0.5
(a) Augmentation
(b) Number of patches
(c) Tyuno
Figure 7: Hyperparameter experiments of DeYO on ImageNet-C at severity level 5.
Performance of Filtered-out Samples. While we have established the unreliability of entropy on the Waterbirds benchmark, it essential investigate whether PLPD can leverage information that entropy might overlook even on ImageNet-C, which contains a relatively weak spurious correlation shift. This examination is presented in Fig. 6, which illustrates the accuracy of areas in Fig. 4 with the pre-adaptation model. When comparing the accuracy of areas with the same entropy level under Gaussian noise corruption on ImageNet-C severity level 5, it is evident that Area and Area 4, areas with high PLPD, exhibit higher accuracy compared to Area and Area 3, respectively. This result highlights that PLPD can identify more reliable samples by leveraging information that entropy cannot capture, even in benchmarks with less emphasis spurious correlation shift.
4.3 HYPERPARAMETER AND ABLATION STUDIES DEYO
Hyperparameter Sensitivity. We experimented with three transformations: pixel-shuflling, patch-shuflling, and center occlusion, to distort object shapes (=x'). Among these, as shown in Fig. 7(a), patch-shuffling. which eliminates only the shape information while pre- serving other details, yielded the best performance. When selecting patch-shuffling as transformation, we further tested different number of patches for an image size of 224 x 224. As depicted in Fig. 7(b), the number of patches of 4x4 exhibited the highest performance. while con- sistently surpassing EATA from the number of patches of 3x3 onwards. Lastly, DeYO requires the threshold, TPLPD, during the sampling selection process (Fig. 7(c)). A higher 7PLPD leads to the removal of more samples, po- tentially hindering sufficient adaptation knowledge acqui- sition. We observed that Yo achieves excellent perfor- (16) DeYO mance when TPLPD belongs [0.2, 0.3).
Table 6: Ablation study on the proposed components. Second best underline.
So(x)
(X)
Enta
PLPD,
Ent.
PLPD,
Avg.
(1) Tent
43.09
(2)
44.92
45 92.92
47.06
(5)
44.78
(7)
47.73
48.44
(9)
44.28
(10)
46 46.06
(11) Enta
47.35
(12)
47
(13)
(14)
46.38
/
48.60
Ablation Studies. To assess the significance of each component of De we conducted abla- tion studies on ImageNet-C at severity level Under otherwise identical conditions, using PLPD, yielded higher performance compared to Ente, and we achieved the best performance when both PLPD, and Ent, are used simultaneously. We also conducted experiments on the biased scenario, WaterBirds. We observed similar trends to ImageNet-C, however, in the case of WaterBirds, found that utilizing only PLPD, in Sg leads to better performance than utilizing both in Sg. This validates observation that entropy significantly decreases in reliability under severe distribution shifts. Further details and analyses of the results are provided in Appendix H.
5 CONCLUSION
In contrast to the common consensus, we have theoretically shown that even when utilizing ex- tremely low-entropy samples in adaptation, performance can be degraded we do not consider the influence of the disentangled factors. Expanding upon the theoretical evidence, a TTA method that combines our confidence metric PLPD, designed to account for the influence of the shape information of objects, with entropy. To the best of our knowledge, Yo is the first TTA method that demonstrates the best performance in both mild and wild scenarios, showing significant performance improvements across various distribution shifts.
propose DeYO,

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This work was supported by the National Research Foundation of Korea (NRF) grants funded by the Korea government (Ministry of Science and ICT, MSIT) 2002RIAA3B107722 and 2022R IAA AASA78330811), Institute Information & Communications Technology Planning Eval- uation (ITTP) grants funded by the Korea government (MSIT) (2021-0-01343 Artificial Intelligence Graduate School Program (Seoul National University) and 2022-0-00959), and the BK21 FOUR program of the Education and Research Program for Future ICT Pioneers, Scoul National Univer- sity in 2024.
REFERENCES
Eric Arazo, Diego Ortego, Paul Albert, Noel E O'Connor, and Kevin McGuinness. Pscudo-labeling and confirmation bias in deep semi-supervised learning. In 2020 International Joint Conference Neural Networks (IJCNN), pp. 1-8. IEEE, 2020. 2
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv: 2019. 2, 20
Fatemeh Azimi, Sebastian Palacio, Federico Raue, Jörn Hees, Luca Bertinetto, and Andreas Den- gel. Self-supervised test-time adaptation video data. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 3439-3448, 2022.
Yuval Bahat, Michal Irani, and Gregory Shakhnarovich. Natural and adversarial error detection using invariance to image transformations. arXiv preprint arXiv:1902.002.30 2019. 19
Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. Advances in neural information processing systems, 32, 2019. 20
Alexander Bartler, Andre Bühhle,, Felix Wiewel, Mario Döbler, and Bin Yang- MT3: meta test- time training for self-supervised test-time adaption. In International Conference on Artificial Intelligence and Statistics, pp. 3080-3090, 2022.
Dina Bashkirova, Dan Hendrycks, Donghyun Kim, Haojin Liao, Samarth Mishra, Chandramouli Rajagopalan, Kate Saenko, Kuniaki Saito, Burhan UI Tayyab, Piotr Teterwak, et al. Visda-2021 competition: Universal domain adaptation improve performance NewrIPS 2021 Competitions and Demonstrations Track, pp. 66-79, 2022. 7,20
cittof-ddisribution data. In
Sara Beery, Grant Van Hor,, and Pietro Perona. Recognition in terra incognita. In Proceedings of
the European conference on computer vision (ECCV), pp. 456-473, 2018. 2,
David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel. Mixmatch: A holistic approach to semi-supervised learning. Advances in neural informa-
tion processing systems, 32, 2019. 18, 19
Gilles Blanchard, Gyemin Lee, and Clayton Scott. Generalizing from several related classification tasks to a new unlabeled sample. Advances in neural information processing systems, 24, 2011.
Ricky TQ Chen, Xuechen Li, Roger B Grosse, and David K Duvenaud. Isolating sources of disen- tanglement in variational autoencoders. Advances in neural information processing systems, 31, 2018. 19
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learning, pp. 1597-1607. PMLR, 2020a.
Yining Chen, Colin Wei. Ananya Kumar, and Tengyu Ma. Self-training avoids using spurious fea- tures under domain shift. Advances in Neural Information Processing Systems, 13:21061-2101 2020b. 19
10

---

Published as a conference paper at ICLR 2024
Sungha Choi, Seunghan Yang, Scokeon Choi, and Sungrack Yun. Improving test-time adaptation via shift-agnostic weight regularization and nearest source prototypes. In European Conference Computer Vision, pp. 140-458. Springer, 2022. 18
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition workshops, pp. 702-703, 2020.. 2020. 19
Yukun Ding, Jinglan Liu, Jinjun Xiong, and Yiyu Shi. Revisiting the evaluation of uncertainty estimation and its application to explore model cuapleitty-uncertainty trade-off. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 4-5, 2020. 8
Andrea Dittadi, Frederik Träuble, Francesco Locatello, Manuel Wuthrich, Vaibhav Agrawal, Ole Winther, Stefan Bauer, and Bernhard Schölkopf. On the transfer of disentangled representations in realistic settings. In International Conference on Learning Representations, 2021. 19
Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In International conference on machine learning, pp. 1180-1189. PMLR, 2015. I
Yonatan Geifman, Guy Uziel, and Ran El-Yaniv. Bias-reduced uncertainty estimation for deep
neural classifiers. In International Conference Learning Representations, 2019a. 2
Yonatan Geifman, Guy Uziel, and Ran El- Yaniv. Bias-reduced uncertainty estimation for deep
neural classifiers. In International Conference on Learning Representations, 2019b.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Imagenet-trained cnns are biased towards texture; increasing shape bias im- proves accuracy and robustness. In International Conference on Learning Representations, 2019.
2.5
Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(11):665- 673, 2020.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. In International Conference on Learning Representations, 2018. 18
Yves Grandvalet and Yoshua Bengio. Semi-supervised learning by entropy minimization. Advances in neural information processing systems, 17, 2004. 19
Ishaan Guirajani and David Lopez-Paz. In search of lost domain generalization. In International Conference on Learning Representations, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pallern recognition, pp- 770-778, 2016.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common cor- ruptions and perturbations. In International Conference on Learning Representations, 2019. 2, 3, 7,20
Dan Hendrycks, Norman Mu, Ekin Dogus Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshmi- narayanan. Augmix: A simple data processing method to improve robustness and uncertainty, In International Conference Learning Representations, 2020. 19, 24
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang. Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of out-of-distributio generalization. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 8340-8349, 2021. 7, 20
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a strained variational framework. In International conference on learning representations, 2017.
19
11

---

Published as a conference paper at ICLR 2024
Dahuin Jung, Dongyoon Han, Jihwan Bang, and Hwanjun Song. Generating instance-level prompts for rehearsal- free continual learning. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 11847-11857, 2023a. 18
Dahuin Jung, Dongjin Lee. Sunwon Hong. Hyemi Jang, Ho Bae, and Sungroh Yoon. New in- sights for the stability-plasticiiy dilemmaa in online continual learning. In The Eleventh Interna- fional Conference on Learning Representations, 2023b. URL https:/ / http:///ppeeeiiieeeeeeeeeeeeeeee net/ forum? id= wed.d==fxxxxxiioc
18
Guoliang Kang, Lu Jiang, Yi Yang, and Alexander Hauptmann. Contrastive adaptation network for unsupervised domain adaptation. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 4893-4902, 2019.
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. In International Conference Machine Learning, pp. 2649-2658. PMLR, 2018. 19
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. Advances in neural information processing systems, 25, 2012. 20
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-22222 1998. 20
Jonghyun Lee, Dahuin Jung, Junho Yim, and Sungroh Yoon. Confidence score for source-free unsupervised domain adaptation. In International Conference on Machine Learning, pp. 12365- 12377. PMLR, 2022. 2
Jungsoo Lee, Eungyeup Kim, Juyoung Lee, Jihyeon Lee, and Jaegul Choo. Learning debiased representation via disentangled feature augmentation. Advances in Neural Information Processing Systems, 34:25123-251133 2021. 19
Jungsoo Lee, Jeonghoon Park, Dacyoung Kim, Juyoung Lee, Edward Choi, and Jaegul Choo. Re- visiting the importance amplifying bias for debiasing. In Proceedings of the AAAI Conference Artificial Intelligence, 2023. 19
Bo Li, Yezhen Wang. Tong Che, Shanghang Zhang, Sicheng Zhao, Pengfei Xu, Wei Zhou, Yoshua Bengio, and Kurt Keutzer. Rethinking distributional matching based domain adaptation. arXiv preprint arXiv:2006. 100006..3333 13352, 2020. 19
Jian Liang, Dapeng Hu, and Jiashi Feng. Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. International conference on machine learning, pp. 6028-608 6039. PMLR, 2020. 18
Hyesu Lim, Byeonggeun Kim, Jaegul Choo, and Sungha Choi. Tin: A domain-shift aware batch normalization in test-time adaptation. In The Eleventh International Conference Learning Representations, 2023. 18
Evan Z Liu, Behzad Haghgoo, Annie S Chen, Aditi Raghunathan, Pang Wei Koh, Shiori Sagawa, Percy Liang, and Chelsea Finn. Just train twice: Improving group robustness without training group information. In International Conference on Machine Learning, pp. 6781-6792. PMLR, 2021a. 19
Yuejiang Liu, Parth Kothari, Bastien Van Delft, Baptiste Bellot-Gurlet Taylor Mordan, and Alexan- dre Alahi, Tit++: When does self-supervised test-time training fail or thrive? Advances Neural Information Processing Systems, 34:21 808-21822, 2021b. 18
Mingsheng Long, Jianmin Wang, Guiguang Ding, Jiaguang Sun, and Philip S Yu. Transfer feature learning with joint distribution adaptation. In Proceedings of the IEEE international conference on computer vision, pp. 2200-2207, 2013. 19
Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Yaofo Chen, Shijian Zheng, Peilin Zhao, and Mingkui Tan. Efficient test-time model adaptation without forgetting. In International conference on machine learning, pp. 16988-16905. PMLR, 2022. 2, 3, 5, 7, 18, 20
12

---

Published as a conference paper at ICLR 2024
Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Zhiquan Wen, Yaofo Chen, Peilin Zhao, and Mingkui Tan. Towards stable test-time adaptation in dynamic wild world. In Internetional Conference on Learning Representations, 2023. 2, 3, 5, 7, 18, 20
Sinno Jialin Pan and Qiang Yang. A survey transfer learning. IEEE Transactions on knowledge and data engineering, 22110)1345-1359 2009.
Changhwa Park, Jonghyun Lee, Jaeyoon Yoo, Minhoe Hur, and Sungroh Yoon. Joint contrastive learning for unsupervised domain adaptation. arXiv preprint D.Xiv:2006.10977 2020.
Viraj Prabhu. Shivam Khare, Deeksha Kartik, and Judy Hoffman. Sentry: Selective entropy opti- mization via committee consistency for unsupervised domain adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8558 8567, 2021. 19, 20
Shiori Sagawa, Pang Wei Koh, Tatsunori Hashimoto, and Percy Liang. Distributionally robust neural networks. In International Conference Learning Representations, 2020. 2, 3, 20
Kuniaki Saito, Donghyun Kim, Stan Sclaroff, and Kate Saenko. Universal domain adaptation through self supervision. Advances in neural information processing systems, 13:16222-6 2020. 2,
Mehdi Sajjadi, Mehran Javanmardi, and Tolga Tasdizen. Regularization with stochastic transfor- mations and perturbations for deep semi-supervised learning. Advances in neural information processing systems, 29, 2016. 19
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based local- ization. In Proceedings of the IEEE international conference on computer vision, pp. 618-626, 3017.
Rui Shu, Hung H Bui, Hirokazu Narui, and Stefano Ermon. A dirt-t approach to unsupervised
domain adaptation. arXiv preprint 1rXiv:1802.000 08735, 2018. 18, 19
Sahil Singla and Soheil Feizi. Salient imagenet: How discover spurious features in deep learning? In International Conference on Learning Representations, 2022. 3
Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. Advances in neural information processing systems, 33:596-608, 2020. 2
Junha Song, Jungsoo Lee, In So Kweon, and Sungha Choi. Ecotta: Memory-efficient continual test- time adaptation via self-distilled regularization. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 11920-11929 2023. 18
Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei Efros, and Moritz Hardt. Test-time train- ing with self-supervision for generalization under distribution shifts. In International conference on machine learning, pp. 9229-9248. PMLR, 2020. 18
Shuhan Tan, Xingchao Peng, and Kate Saenko. Class-imbalanced domain adaptation: an empiri- cal odyssey. In Computer Vision-ECCV 2020 Workshops: Glasgow, UK, August 23-28, 2020, Proceedings, Part I 16, pp. 585 602. Springer, 2020. 19
C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The Caltech-UCSD 20rds-200-20111 dataset. Technical report, California Institute of Technology, 2011. 20
Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. In International Conference Learning Repre- sentations, 2021a. URL / net/ Eorum? idduxiilli 3bZLkr3C. 1, 7, 8, 18, 20
Xinshao Wang, Yang Hua, Elyor Kodirov, David A Clifton, and Neil M Robertson. Proselfic: Pro- gressive self label correction for training robust deep neural networks. In Proceedings of the IEFF/CVF conference on computer vision and pattern recognition, pp. 752-761, 202 6
13

---

Published as a conference paper at ICLR 2024
Yuchao Wang, Haochen Wang, Yujun Shen, Jingjing Fei, Wei Li, Guoqiang Jin, Liwei Wu, Rui Zhao, and Xinyi Le. Semi-supervised semantic segmentation using unreliable pseudo-labels. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp-
2248-4257, 2022. 3
Ross Wightman. Pytorch image models.
htttps///gichabb.cmmmm.gggg
mosarcc--maae-mddeeee 2019. 20
Olivia Wiles, Sven Gowal, Florian Stimberg, Sylvestre-Alvise Rebuffi, Ira Ktena, Krishnamurthy Dj Dvijotham, and Ali Taylan Cemgil. A fine-grained analysis distribution shift. In International Conference on Leurning Reprexentations, 2022. URL https: /ttp////ppe/eeeeeeeeeeeeeee net forum?
id-D14LetuLdy'S 2, 19
Qizhe Xie, Minh-Thang Thang Luong, Eduard Hovy, and Quoe V Le. Self-training with noisy student improves imagenet classification. In Proceedings the IEEE/CVF conference on computer vision the
and patlern recognition, pp. 10687-10698. 2020. 19
Marvin Zhang, Sergey Levine, and Chelsea Finn. Memo: Test time robustness via adaptation and augmentation. Advances in Neural Information Processing Systems, 35:38629 2022. 18, 20
Hao Zhao, Yuejiang Liu, Alexandre Alahi, and Tao Lin. On pitfalls of test-time adaptation. In International Conference Machine Learning (ICML), 2023. 2
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE transactions pattern analysis and machine intelligence, 4066):1452-1464 2017. 20
Chunting Zhou, Xuezhe Ma, Paul Michel, and Graham Neubig. Examining and combating spurious features under distribution shift. In International Conference Machine Learning, pp. 12857- 12867. PMLR, 2021. 3
14

---

Published as a conference paper at ICLR 2024
APPENDIX
CONTENTS
Proof of Proposition
16
Pseudo Code of DeYO
17
Efficiency
18
D Related work
18
D.I Test-time adaptation
18
D.2 Disentangled factors
19
D.3 Self-training
19
D.4 Consistency under augmentations
19
More Implementation Details
20
E.1
Baseline Methods
20
E.2 More Details on Benchmark
20
E.3 More Details on Experimental Protocols
20
F
Further Experiments ImageNet-C at Severity Level
21
F.1
Comparisons Mild Scenario
21
F2
Comparisons Wild Scenario
Further Experiments on ImageNet-R and VisDA-2021
22
Further Hyperparameter and Ablate Results
22
Further Hyperparameter Sensitivity Results
23
J
Efffect of multiple jjject-preeerving augmentations
24
Efffect of PL.PD during pre-training
25
Effeec of PLPD wild scenarios
25
M Limitations
26
15

---

Published as a conference paper at ICLR 2024
A
PROOF OF PROPOSITION
Proof. The gradient when adapting through entropy minimization loss is as follows:
(Ents(x)
(x) (x) (a)(x)
00;
(x) (x)(x)
00;
0( a(-plogy plogp - (1 - p) log (1-))) @o(a) v
Op
da
00,
(1 po(x))
log
(x(x) ((20( o(oo
sign log (1 - --P((x))
-C'vv;
(x)
(1 - (1-Po(x)) Px(()) po(x)) >0 0.
-y. where C== log
(x
where p, substitutes of Pa (x), as (x), respectively. Hence, the update of the model through entropy minimization loss is as follows:
EEtt (x) nýCvv = =SC'v Vi
AA = -1 00;
A = ýc' vxx,, vc'v(xx, where C' = nyC > 0.
The change in logit of test due to the model's update by x is as follows:
A(0-v(xtest) = ~0(x) = ýC' vxx))t(x(ttt))
The change in the gap between the mean logits of samples belonging to the two classes is as follows:
(Extent A 10 (0-v(x'est) Ex 10-v(x' sestt)]) )])
Excen x A V(0-v(x )}] E Extor X A A (x
Extctt v E8 Eyto A V
=C' c'yv(x)).(FF (Rxerewappe
(v(xles)) Xtou (xxxee))))
If the change is negative, it means that the distribution in the logit space of the two classes is getting closer, leading decrease in class discriminability. Since C' is positive, ultimately, when using adaptation that satisfies the following condition, discriminability decreases:
(v(x). (Begteet xrtan [v(x* test E t Ptent (v(x test I) <
16

---

Published as a conference paper at ICLR 2024
B PSEUDO CODE OF DEYO
We present the pseudo-code for DeYO, which encompasses several key steps. Firstly, we compute entropy for each sample x; according to Eq. 1. Next, employ PLPD on the test samples that satisfy the condition specified in lines 4-6 of Algorithm 1. That is, we compute PLPD in conjunction with entropy select reliable set denoted (x)x) (Eq. 8). Subsequently, we calculate the weights ag (x) for the samples within (x),, taking into account both entropy and PLPD as specified in Eq. 10. Lastly, optimize the parameter 0 by minimizing LDYO. combination of og (x) and entropy minimization loss as described in Eq. 11. Algorithm provides a detailed pseudo-code representation of these procedures.
Algorithm 1: Destroy Your Object (DeYO)
Input: Test samples Diest (x, j model MM(-) with trainable parameters 0 C 0,
object-destructive transformation A, step size rj > hyperparameters
Enlo: Tent: TPLPD > 0. Output: Predictions {9.)_1
Initialize = Og:
for x; Drest do
Compute entropy Exto(x;) and predict ÿN= arg max Mo(x;)) ;
// (Eq. 1)
(x,) > TEmt then
continue;
end
Obtain xx=A(xx); Compute PLPDo x';)
1/ (Eq. 9)
PLPDg(X,, x'i) < TPLPD then
10
continue; end
11
Compute sample weight (x:):
// (Eq. 10)
Compute the overall loss LDEYO and its gradient V : Update A t- 0 v/ WoDeroo
// (Eg. 11)
15
end
17

---

Published as a conference paper at ICLR 2024
C
EFFICIENCY
Table 7: The runtime of state-of-the-art methods. We assess the computational efficiency of various TTA methods using the ResNet-50-BN model on ImageNet-C. specifically examining the case of Gaussian noise at severity level 5, comprising a total of 50,000 images. The practical runtime is evaluated using a single A6000 GPU.
Methed
Need source data? Online update?
@FFoward
#Backward
computation
(50,000 images)
No adapt.
72 seconds
MEMO
x
X
50,000
50,000x64
AngMix n/a
30,818 seconds
Tent
X
50,000
50,000
86 seconds
EATA
50,000
19,085
regularizer
97 secomds
SAR
X
50,000
18,608
18,608 12,491
Additional model updates
123 seconds
DeYO (ours)
X
50,000
33,943
25,836
Eq. 9
secondss
Tab. 7 presents the results of measuring the time required for adaptation in our proposed DeYO and baselines under ImageNet-C. Gaussian noise, and severity level 5 environments. MEMO demands significantly more time due to the necessity of numerous additional augmentations compared to other methods. Generally, other methods show an increase computation time commensurate with the improvement in performance. De achieves the highest performance while requiring less time than SAR. SAR consumes more GPU time because of its two-step model updates although it has a smaller sum of forward and backward samples than DeYO.
D RELATED WORK
D.1 TEST-TIME ADAPTATION
Test-time adaptation (TTA) (Wang et al., 202 Sun et al., 2020; Choi al., 2022; Lim et al., 2023; Song et al., 2023) aims to enhance inference performance, balancing resource efficiency and effec- tiveness without access ground-truth labels. Whereas continual learning (Jung al., 2023b;a) aims to mitigate distribution shifts occurring while learning sequence of tasks, TTA aims to ef- ficiently adapt target benchmark at inference time by adjusting the pre-trained feature space. Because ground-truth label is inaccessible in TTA, it requires the use of unsupervised loss to adapt the model. Therefore, entropy minimization serves as a crucial unsupervised regularizer across var- ious tasks (Berthelot et al., 2019; Shu et al., 2018; Liang et al., 2020). By regularizing entropy, we can penalize decisions made in regions of high data density to enhance accuracy for distinct classes.
TTA can be broadly categorized into two groups based on its involvement in the training stage. The first category, known as Test-Time Training (TTT) (Sun et al., 2020), modifies the training loss in the training stage for enhanced test-time adaptation. Typically, TTT involves the simultaneous optimiza- tion the source model using both supervised and self-supe losses. In other words, TTT relies on proxy supervisdr)) task, and its loss is contingent upon the selection of proxy. Depend- ing on the choice of the proxy. TTT may incorporate objectives such as rotation prediction (Gidaris al., 2018) or contrastive- based losses (Liu al., 2021 bb; Bartler et al., 2022).
The second category, Fully Test-Time Adaptation (fully TTA) (Wang et al., 2021a), constitutes a research domain that refrains from intervening in the training stage and, thus, is applicable to all pre-trained models. Our proposed De falls within the realm of fully TTA as it can be applied any pre-trained model. While fully TTA exhibits distinct advantages compared to TTT, there are limitations in terms of computational cost and stability. Tent (Wang et al., 2021a utilizes an en- tropy minimization loss, focusing on updating batch normalization parameters to maintain efficiency while improving performance. Unlike Tent using all samples for adaptation, EATA (Niu et al., 2022) and SAR (Niu et al., 2023) achieved improved performance by filtering-out high entropy samples. However, EATA necessitates a relatively unrealistic assumption of class balance and a clean dataset, and SAR involves a longer computational time due to the additional backward passes. On the other hand, MEMO (Zhang et al., 2022) aims for stable single-sample adaptation by minimizing marginal entropy through multiple augmentations, but it suffers from a longer computational time due to the of multiple augmentations. DeYO, requiring no additional assumptions and avoiding extra backward passes, achieves significantly improved performance with minimal overhead by utilizing single augmentation.
18

---

Published as a conference paper at ICLR 2024
D.2 DISENTANGLED FACTORS
The disentanglement literature (Higgins et al., 2017; Kim & Mnih, 2018: Chen al., 2018; Dittadi et al., 2021; Lee et al., 2021; 2023) primarily centers around the task of decomposing images into disentangled factors, which represent distinct and independent attributes. This literature operates under the assumption that the data encompasses multiple (potentially numerous) factors and expects models to become invariant to these factors after being exposed various values of them. This, in turn, enables the models to generalize to novel instances and diverse factor distributions. For instance, in a dataset containing factors like shape and color, the aim is for the model to predict shapes even when confronted with previously unseen colors different color distributions. Wiles et al. (2022) discusses distribution shifts based on the joint distribution these disentangled factors. To enhance robustness against such shifts, various methods like weighted resampling (Liu et al., 2021a), data augmentation (Hendrycks et al., 2020), and representation learning (Higgins et al., 2017) are available. Our specific focus lies on weighted resampling, and, bolster its reliability, we incorporate single data augmentation step.
D.3 SELF-TRAINING
Self-training methods have demonstrated state-of-the-art performance in semi-supervised learn- ing (Xie al., 2020), adversarial robustness (Long et al., 2013), and unsupervised domain adapta- tion (Shu et al., 2018). Pscudo-labeling and conditional entropy minimization emerge as two promi- nent forms of self- eeff-training. This typically involves supervised training on confidently predicted tar- get pseudo-labels (Tan et al., 2020), or conditional entropy minimization on target instances (Grand- valet & Bengio, 2004). However, unconstrained self-training may result in the accumulation of er- rors. Consequently, in the field of semi-supervised learning, the error accumulation issues are ad- dressed through using a supervised loss as guidance. In unsupervised domain adaptation, entropy is also employed metric to identify reliable target instances (Prabhu et al., 2021).
Despite the success of self-training methods, there exists a limited understanding of the conditions and factors contributing to their effectiveness amid domain shifts. In the theoretical work by Chen et al. (2020b), it is demonstrated that in the particular setting, self-training can avoid using spurious features. While this work contributes ticular setting discussed, "the spurious feature which correlates with the label in the source domain does not exist in the target domain" and "the source model must be accurately trained", difficult to be considered as practical.
understanding of self- soff-training in domain shifts, the par-
D.4 CONSISTENCY UNDER AUGMENTATIONS
The utility of predictive consistency under augmentations has been recognized in various appli- cations, which can be broadly categorized into two main approaches. Firstly, it is employed regularizer (Berthelot et al., 2019; Sajjadi et al., 2016). In this context, consistency regularization is applied by introducing data augmentation, capitalizing on the principle that a classifier should yield identical class distributions for an unlabeled example, even postttuummetttion. This approach finds extensive applicability in supervised learning (Cubuk et al., 2020), self-supervised representation learning (Chen et al., 2020a), semi-supervised learning (Berthelot et al., 2019; Sajjadi et al., 2016), and unsupervised domain adaptation (Li et al., 2020).
The second approach is centered around the utilization of predictive consistency under augmenta- tions for detecting reliable instances (Prabhu et al., 2021). This approach is rooted the under- standing that consistency under image transformations serves dependable indicator of model errors (Bahat et al., 2019). Specifically, in the context of self-training, this approach employed to mitigate error accumulation. It leverages predictive consistency under committee of random trans- forms to identify instances deemed reliable for alignment. Subsequently, the model selectively optimized these identified instances.
19

---

Published as a conference paper at ICLR 2024
E MORE IMPLEMENTATION DETAILS
E. BASELINE METHODS
We compare DeYO with the following ofate-of-the-art TTA methods: MEMO (Zhang et al., 2022) enhances the consistency of predictions across various augmented copies for adaptation. SEN- TRY (Prabhu et al., 2021) is an UDA method which enhances target instance reliability by opti- mizing selective entropy based on consistency under varied image transformations and balances target class distributions using pseudo-labels. For use in the TTA, we remove the supervised loss of SENTRY. Tent (Wang et al., 202 guides model updates by reducing the entropy of test samples. EATA (Niu et al., 2022) introduced a novel approach that combines sample selection based on en- tropy and weighted adjustments to minimize entropy for selected samples. SAR (Niu et al., 2023) minimizes entropy with sharpness awareness for stable adaptation under wild test scenarios. Except for MEMO, the TTA methods including De make real-time (online) adjustments to the model during testing. Once adaptation for each dataset is complete, the model parameters are reset.
E.2 MORE DETAILS ON BENCHMARK
We assess DeYO's performance across five benchmarks: ImageNet-C (Hendryeks & Dietterich, 2019), ImageNet-R (Hendrycks et al., 2021), Visda-2021 (Bashkirova et al., 2022), ColoredM- NIST (Arjovsky et al., 2019), and Waterbirds (Sagawa et al., 2020). ImageNet-C, created by ap- plying 15 distinct corruptions to the ImageNet (Krizhevsky et al., 2012) benchmark, is widely em- ployed to conduct experiments observing robustness against shifts in distribution. For the evaluation of De under more challenging scenarios, we extend our analysis beyond the ImageNet-C bench- mark by including ImageNet-R R (Hendrycks et al., 2021) and Visda-2021 (Bashkirova et al., 2022) benchmarks. ImageNet-R consists of diverse array of artistic renditions (e.g., art, cartoons, graf- fiti, sculptures, etc.) representing 200 ImageNet classes. The Visda-2021 benchmark collects data from both ImageNet-R/C and ObjectNec (Barbu al., 2019). Unlike ImageNet-C. which primarily focuses on synthetic corruption, ImageNet-R, and Visda-2021 present more challenges due to the natural shifts inherent the data domain.
For the ColoredMNIST and Waterbirds benchmarks, the training data exhibits substantial spurious correlation, whereas the test data is characterized by minimal or no spurious correlation. The Col- oredMNIST benchmark is a derived version of the MNIST (LeCun et al., 1998). In this benchmark, images with digits ranging from 0 to are assigned to class O while those with digits from to 9 are assigned to class 1. The color ID is green when it is 0 and red when it is 1. The default color ID fol- lows the class ID and is flipped with a probability of 0.2 on the training data and 0.9 on the test data. The Waterbirds benchmark combines objects from the CUB (Wah et al., 2011) with backgrounds from the Places (Zhou et al., 2017). The correlations between the class and background are present in the training data but not in the test data. In the training set, 95% of waterbird images are positioned on water backgrounds, while the remaining 5% are placed on land backgrounds. Similarly, 95% of landbird images are on land backgrounds, with 5% positioned water backgrounds.
E.3 MORE DETAILS ON EXPERIMENTAL PROTOCOLS
For the ColoredMNIST benchmark, we opt for Gesseet-18BBy given its lower complexity compared to other benchmarks. For Waterbirds and ImageNet-C (mild scenario), we employ ResNet-50-BN, while for ImageNet-C (wild scenarios), ImageNet-R, and Visda-2021, we employ ResNet-50-GN and ViTbase-LN. Pre-training for ColoredMMIIS and Waterbirds involves 20 and 200 epochs, re- spectively, with a batch size of 64. As for ImageNet-C, ImageNet-R, and Visda-2021, we utilize publicly available pre-trained models from the torchvision and timm (Wightman, 2019) libraries. Our testing adaptation protocol basically adheres to the one proposed by Niu et al. (2023). We op- timize only normalization layer parameters. Under label shift scenarios of ImageNet-R and Visda- 2021, we assume imbalance ratio of infinity, randomizing the order of classes and samples within each class. Stochastic Gradient Descent (SGD) with a momentum of 0.9 serves as our optimizer. Typically, the batch size is set to 64, except for experiments where it is configured 1). The learning rate is fixed at 0.00025 for the ResNet model and 0.001 for the Vit model, except when the batch size equals! 1. In such cases, the learning rate is adjusted to 0.00025 divided by 16
I (batch size
20

---

Published as a conference paper at ICLR 2024
for the ResNet model and 0.001 divided by 32 for the Vit model. For the Waterbirds benchmark, YO filters out a substantial number of samples, so we increase the learning rate by a factor of 5.
The required hyperparameters for DeYO are TEn, TLLPD, and Ento. We set Ento and Tent to 0.4 x In C and 0.5 x In C, respectively. The analysis of two hyperparameters (Tem:: and Ento) is presented in Appendix H, and the analysis of TPLPD sensitivity can be found in Section 4.3 of the main paper. As shown in the sensitivity results, DeYO exhibits robustness with respect to hyperparameters. Re- garding TDLPD for the reported results, its value is configured at 0.3 for ImageNet-C (mild scenario), 0.5 for the biased scenario, and 0.2 for the rest of the experiments. ColoredMNIST and Waterbirds are binary benchmarks, and since they mostly produce one-hot predictions due to their simplicity, we use 0.5 for threshold of label flip. We do not use entropy filtering and set Ento = In C' for ColoredMNIST.
F FURTHER EXPERIMENTS ON IMAGENET-C AT SEVERITY LEVEL 3
The results at severity level 3 under the mild scenario and two wild scenarios (online imbalanced label distribution shift, and batch size 1) are reported in Tab. and 9, respectively. The results are consistent with those of severity level 5, demonstrating superior performance across the 15 corrup- tion types on average.
F.1 COMPARISONS ON MILD SCENARIO
Table 8: Comparisons with baselines on ImageNet-C at severity level 3 under mild scenario re- garding accuracy (%). The bold value signifies the top-performing result.
Noise Slot
Blur
Weather
Digina
Mild
Cama
Impul.
Defoc:
Motion
Zocm
Snow
Frost
Fog
Bra.
Conts:
Flacric
Pixel
JPEG
Avg:
ResNer-50. 100---50.33
25.0
25.2
37.9
35.2
35.2
32.1
46.7
69.6
46.0
55.6
39.7
51.7
49.3
48.6
65.8
72.1
59.2
EATA
57.0
56.8
56.2
52.4
50.2
61.0
58.6
51.3
67.1
72.3
68.2
69.9
61.0
SAR
51.5
$4.1
53.5
49.3
46.3
58.6
57.6
55.6
48.6
65.6
72.0
67.1
69.3
67.3
65.7
59.0
DeYO (ours)
58.3
$7.1.:
534..0
51.2,m
61.
29
59.611
51.Y
67._.:
72.0pz
68.5
68.6.ra
66.611
61.6gmm
F.2 COMPARISONS ON WILD SCENARIO
Table 9: Comparisons with baselines on ImageNet-C at severity level 3 under online imbalanced label shifts (imbalance ratio = oc) or under batch size 1 regarding accuracy (%).
Wenther
Digital
Shifts
Gauss,
Impul
Defoc.
Motion
Zoom
Fog
Deit.
Contr.
Avg.
ReNNe--St-GN
54.5
52.9
53.1
44.4
21.2
49.8
30.3
$4.9
$4.1
55.8
75.3
69.7
59.6
50.7
66.4
56.1
MEMO
35.9
54.3
54.1
40.1
23.1
49.5
41.4
54.8
54.1
57.6
75.7
70.2
60.2
61.5
66.7
51.6
Tent
59.1
58.6
58.3
39.0
21.9
54.7
41.1
51.3
41.4
62.0
75.2
70.1
62.3
63.7
66.4
55.1
EATA
52.3
$2.9
51.7
35.7
30.1
46.4
39.6
43.8
39.5
55.7
12.4
66.6
54.7
56.0
56.2
51.3
SAR
60.8
60.5
60.2
47.9
36.7
58.2
49.7
$7.9
$3.6
65.0
76.4
71.0
67.0
65.8
67.6
59.9
DeYO lours)
64.00
63.9
63.2
$4.0
62.2
61.2
$7.9
69.2.n
73.2.33
71.2
70.3
69.8
Viltsase LN
313
46.8
50.4
4K.7
37.1
58.7
41.6
35.1
33.3
6K.0
89.3
74,99
65.9
56.0
65.6
533
MEMO
62.1
57.9
61.5
$7.2
45.6
62.0
48.9
46.5
43.1
74.1
75.8
79.7
72.6
72.3
710.6
62.1
Tent
65.7
08.0
68.1
68.2
63.8
70.9
63.8
67.6
41.9
76..3
788.8
79.5
75.9
76.7
73.7
64.5
FATA
65.3
62.6
63.6
63.0
$7.1
66.3
50.3
64.5
61.0
75.3
9
75.0
74.2
74.8
73.1
67.4
SAR
68.8
68.2
68.4
68.3
64.7
71.0
64.2
68.1
66.0
76.4
79.0
79.6
76.2
77.1
74.1
71.3
DeYO touss) 71.7.g1
714-31
7006-g1
68.9ini
739--1
(9.2.n1
72.4.03
69.7.51
77.9..
301.2
79.5450
78.2.n
78.7.0
767.42
7400.1
Wenther
Tigital
Gauss.
Impul.
Defoc.
Glass
Motion
Fog
Delt.
Contr.
Avg.
KeNNe--SO-GN
$4.5
53.8
53.1
44.3
21.2
49.7
30.
$4.8
$4.0
55.8
75.4
69.5
69.6
59.7
66.
$4.0
MEMO
45.7
54.2
53.9
40.0
22.8
49.2
41.2
54.8
54.1
57.6
75.5
69.9
60.0
61.3
66.6
$4.5
Text
56.8
58.5
58.7
38.2
26.8
$4.9
42.6
$1.6
38.5
61.9
75.3
70.0
62.3
63.6
66.3
55.2
DATA
59.
58.7
58.3
45.7
32.6
55.5
45.9
56.4
52.7
63.6
75.9
21.1
66.7
61.5
67,8
5K22
SAR
60.3
99.6
59.5
46.6
33.0
57.5
47.8
$7.8
52.8
65.1
76.7
71.4
67.3
66.0
67.8
59.3
DeYO
64.4
64.5
63.7
55.9 (13
62.3
58.5
69.8
73.5
71.5
70.7
64.5
Villase LN
31.6
46.9
50.5
4K.7
37.2
56.7
41.6
35.1
35.5
67.5
89.3
74.8
65.8
56.0
65.7
553
MEMO
61.9
57.7
61.4
$7.0
45.4
61.8
49.8
46.6
43.1
73.99
75.7
79.6
72.6
72.1
71..5
61.9
Tent
67.1
66.2
66.3
66.3
80.9
69.1
61.4
65.2
60.4
75.2
78.
78.5
74.9
75.8
72.4
60.2
EATA
60.7
58.5
61.6
60.1
51.8
64.2
54.8
53.5
$2.6
72.5
73.6
77.9
71.3
71.5
69.7
63.6
SAR
68.5
67.8
68.0
67.8
63.1
70.7
63.5
66.9
62.8
75.8
71.7
78.4
74.7
75.7
72.7
70.3
DeYO (ours)
72.3.gn
72.1.o1
71.9
7L.1-gp
69.4.00
74.2.30
(9.3..
72.M.o1
70.1.51
7866.05
S0.7
SIL4.:0
786..6
79.2
77.2
74.5
21

---

Published as a conference paper at ICLR 2024
G FURTHER EXPERIMENTS ON IMAGENET-R AND VISDA-2021
We conducted additional experiments for wild scenarios using the ImageNet-R and Visda-2021 benchmarks with ResNet-50-GN and ViTBase-LN, Both benchmarks involve collecting data from various distribution shifts, including data with completely different styles aside from corruption. Therefore, for overall experiments, we consistently assumed mixed shift scenarios and then included label shifts batch size scenarios.
Tab. 10 and 11 report the results on online imbalanced label distribution shifts and batch size scenarios, respectively. Similar the results in the main for ImageNet-C, DeYO exhibited the best performance across scenarios and architectures on the ImageNet-R benchmark.
Table 10: Comparisons with baselines Table 11: Comparisons with baselines ImageNet-R under a wild scenario (single sam- ple adaptation (batch size 1)) regarding accuracy (%).
ImageNet-R under a wild scenario (online im- balanced lahel distribution shifts) regarding ac- curacy (%).
Method
ResNet-50-GN
VitBase-IN
Method
ResNet-500-GN
VitBase-LN
No Adapt.
40.8
43.1
No Adapt.
40.8
43.1
Tent
42.8
49.9
Tent
44.3
50.4
EATA
42.8
51.0
EATA
41.8
46.7
SAR
42.5
51.7
SAR
43.1
52.0
DeYO (ours)
47.0
59.3
DeYO (ours)
48.4
60.3
We also conducted similar experiments on the Visda-2021 benchmark. Tab. 12 and 13 report the re- sults online imbalanced label distribution shifts and batch size scenarios, respectively. Similarly, De demonstrated the best performance across scenarios and architectures on Visda-2021.
Table 12: Comparisons with baselines Visda- 2021 under wild scenario (online imbalanced label distribution shifts) regarding accuracy (%).
Comparisons with baselines Visda- 2021 under a wild scenario (single sample adap- tation (batch size 1)) regarding accuracy (%).
Table 13:
Method
ResNet-50-GN
VitBase-LN
Method
ResNet-50-GN
VitBasc-LN
No Adapt.
43.5
44.3
No Adapt.
43.5
44.3
Tent
43.8
50.2
Tent
43.9
50.4
EATA
43.2
51.1
EATA
43.9
47.9
SAR
43.7
50.4
SAR
43.8
51.2
DeYO (ours)
44.9
57.3
DeYO (ours)
45.9
58.7
An important observation in each of these tables is that, despite the inclusion of entropy-based sample selection in EATA and SAR, the performance improvement is minimal or even leads decrease in performance. This verifies that the effectiveness of entropy-based sample selection is
either negligible or, in some cases, negative for the datasets with various distribution shifts. DeYO demonstrated that significant performance gains not solely reliant entropy and that the use of PLPD with consideration for CPR factors contributes to improved adaptation.
H FURTHER HYPERPARAMETRA AND ABLATE RESULTS
55
Avg Acc (%)
48.5
45
48.6
48
35
.47
48.2
46.5
5
46
0.3InC
0.4hnC 0.5ImC
0.2InC
0.4hnC
(a) Tgnt
(b) Ent,
(c) PLPDj
Figure 8: Hyperparameter (TEmy Ento, and PLPDo) experiments ImageNet-C severity level 5.
22

---

Published as a conference paper at ICLR 2024
De relies on three essential hyperparameters: Teat TELPD. and Ento. As shown in the main paper, the results for TPLPD indicate that DeYO with TPLPD below 0.4 performs better than EATA, as long they do not overly restrict sample selection. For Tim: depicted in Fig. 8(a), there are quite numerous samples not selected by TPLPD, making the choice of 0.5 x In C more effective than the original 0.4 In C' used by EATA. However, notably, even with 0.4 x In C, DeYO outperformed EATA. Regarding Ento (Fig. 8(b)), it performed best when set to 0.4 x InC, identical to EATA. Lastly, PLPDo represents the normalization factor of the PLPD best performance was achieved when set to 0, we opted not to include the normalization factor when proposing the term, as shown in Fig. 8(c).
in sample weighting. As the
Tab. 14 presents the results of AURC measurement aimed at assessing the reliability of entropy and PLPD variants as a confidence score the worst group of the Waterbirds benchmark. Entropy exhibits the highest AURC value, making it the most unreliable confidence metric, while among the PLPD variants, the use of Xoee (Occlusion PLPD) demonstrates the lowest value which means the highest reliability. This observation is attributed to the fact that the Waterbirds inherently involve objects precisely centered the image. Furthermore, an ablation study was conducted to investigate the effects of DeYO's components Waterbirds, and the results documented in Tab. 15. While the overall trends align with the findings in Tab. 6, notably, the absence entropy filtering in case (8) yields higher performance than our proposed De It additionally illuminates the notion that entropy lacks reliability in the presence of spurious correlation shifts.
Table 14: The AURC value of Fig. 5.
Table 15:
Ablation study on the proposed components on Waterbirds in biased scenario (ResNet-50-BN), Second best underline.
Table 16: Ablation study on the proposed components on ImageNet-C label shifts scenario (ViTBase-LN), Sec- ond best underline.
AURC (%)
Entropy 4x AxAuMMix PLPD
56.30
35.58
Se50)
Ox
Se00
0600
Pixel PT.PD
20.71
Daats 193
Avg
Pais PLP PiNA
Avg
Patch PLPD
14.75
Occlusion PLPD
13.77
(3)
65 23
(3)
5411
(4)
59.07
(4)
54.05
(8)
13.05
61:44
(9)
5657
(9)
52.71
(10)
56.00
(10)
56.15
(11)
(1I)
50.00
(14)
70.33
(14)
51.58
(15)
63 90
(15)
(00) DAVE
(16)
I FURTHER HYPERPARAMETER SENSITIVITY RESULTS
Avg Acc (%)
55
55
55
50
50
50
Pixel
Patch
Occ
2X2
3X3
4X4
5X5
6X6
0.0
0.2
0.3
0.4
0.5
(a) A ugmentation type
(b) Number of patches
(c) Tpr.PD
Figure 9: Hyperparameter sensitivity experiments of DeYO on ImageNet-CC at severity level under online imbalanced label shifts (imbalance ratio oo).
As shown in Fig. 9 and Tab. 17, we experimented with the parameter sensitivity of DeYO under a different model architecture (ViTBase), dataset (WaterBirds), and test scenario (wild scenario: online imbalanced label shifts and biased scenario: spurious correlation). We observed consistent results with that of Fig. 7 (ResNet-50-BN) ImageNet-C, and mild scenario) in the main manuscript. For Waterbirds, it involves 2-class classification task. Therefore, the value of TeLeD should be larger than that of ImageNet-C, which has 1000 classes, to adequately capture the changes induced by transformations.
23

---

Published as a conference paper at ICLR 2024
Table 17: Hyperparameter sensitivity experiments of De on WaterBirds in terms of augmentation
type (left), number of patches (center), and TPLPD (right).
7PLPD
Avg (%)
Worst (%)
Patch
Avg (%)
Worst (%)
0.0
84.54
58.84
Type
Avg (%)
Worst (%)
2X2
86.98
74.96
0.2
87.31
71.67
Pixel
85.84
65.42
3X3
87.26
75.43
0.3
87.72
72.93
Patch
88.16
74.18
4X4
88.16
74.18
0.4
87.45
71.67
Occ
87.69
73.55
5X5
87.12
72.14
0.5
88.16
74.18
6X6
87.40
72.14
0.6
88.06
74.8
0.7
87.59
74.02
EFFECT OF MULTIPLE OBJECT-PRESERVING AUGMENTATIONS
We compared the average PLPD measured by augmenting multiple times through Aug- Mix (Hendrycks et al., 2020) with the PLPD of pixel-shuflling, patch-shuflling, and center occlusion. AugMix employs various prreerigres augmentations such as autocontrast, equalize, rotate, and solarize, randomly applying them to create multiple augmented instances. Then, the augmented instances are mixed with the original input. AugMix also utilized in MEMO.
When using the average PLPD for total of mixed inputs in Yo, it showed lower performance than three object-destructive transformation methods in both ColoredMNIST and Waterbirds as shown in the Tab. 18. The AURC, which evaluates the reliability of PLPDD, also exhibited worse performance compared to the object-destructive transformations as shown Tab. 14. Our proposed PLPD needs to disrupt the shape of the object to function effectively. Through the AURC values, empirically confirmed that AugMix cobject-preservin augmentation technique) cannot contribute to measuring the impact the CPR factor.
We conducted experiments EATA and Ye using average entropy over four AugMix augmen- tations for entropy filtering. As indicated in the Tab. 19, this modification resulted in only marginal performance improvements. The results show that the use of average entropy from augmented im- ages is not beneficial enhancing robustness against distribution shifts between training and infer- ence phases.
Additionally, in DeYO, we experimented using multiple AugMix(x) as x' for calculating PLPD. AugMix employs various object-preserving augmentations such as autocontrast, equalize, rotate, and solarize, randomly applying them to create multiple augmented instances. Then, the augmented instances are mixed with the original input. AugMix is also utilized in MEMO.
Table 18: The effect of multiple object-preserving augmentations on PLPD.
ColoredMNIST
Waterbirds
Avg Ace (%)
Worst Acc (%)
Avg Acc (%)
Worst Acc (%)
DeYO with 8x x AAgMix
53.53
4.19
82.94
60.75
DeYO with X
69,68
50.78
85.
64.17
De with x
74.56
62.06
86.78
74.96
DeYO with x
77.61
65.51
86.56
74.18
Table The effect multiple object-preserving augmentations on EATA and DeYO.
ColoredMNIST
Waterbirds
Avg Ace (%)
Worst Acc (%)
Avg Ace (%)
Worst Ace (%)
EATA
60.29
16.99
83.42
57.48
EATA with 4x AugMix(x)
60.29
17.00
83.60
57.94
DeYO
77.61
65.51
86.56
74.18
De with 4x AugMix(x)
77.83
65.58
86.81
74.45
24

---

Published as a conference paper at ICLR 2024
K EFFECT OF PLPD DURING PRE-TRAINING
We experimented with utilizing PLPD-based filtering during pre-training. For Waterbirds, during a total of 200 epochs, the initial 50 epochs were trained conventionally, followed by 150 epochs where only samples with PLPD greater than the threshold of 0.2 were used for pre-training. As shown in Tab. 20, after implementing PLPD filterin,, the performance 5.82% higher compared to the original result Tent. Similarly, the performance of Ye also improves by 3.48%.
Table 20: The effect of PLPD during pre-training stage. With PLPD filtering, the model shows performance improvements after TENT and DeYO.
Tent
DeYO
Avg Acc (%)
Worst Acc (%)
Avg Acc (%)
Worst Acc (%)
mre-training w/o PLPD
82.98
52.90 58.72
86.74
74.18
pre-training w/ PLPD
84,45
87.90
77.66
L EFFECT OF PLPD IN WILD SCENARIOS
Essentially, the enhancement in performance arising from excluding TRAP factors during adap- tation, thereby increasing robustness against covariate shifts and spurious correlations in the wild scenarios (label shifts and batch size 1), is the same as the mild scenario.
Label shifts and batch size scenarios have the following characteristics.
i) Label shifts:
An Input data stream is dependent on the ground truth class.
When i.i.d. samples are present within batch (a mild scenario), it's possible to discern which disentangled factors are emphasized for each class. However, in a label shifts sce- nario, since all samples in a batch belong to the same class, this cannot be determined.
ii) Batch size 1:
A model only uses single image to update its parameter.
When i.idd. samples are present within batch (a mild scenario), sample-specific noise is reduced through loss averaging. However, in a batch size scenario, this noise cannot be reduced, increasing sample dependency.
According to theoretical analysis, the increase in the weight of disentangled factors is propor- tional to the value of the corresponding factors, as described in Appendix A. In general, without PLPD filtering, during updates with batches where all samples have the ground truth label in label shifts scenario, the weight of disentangled factors common to samples the batch increases proportionally to the corresponding factors' values. In batch size scenario, the weight of all dis- entangled factors in the current sample increases proportionally to the values of the corresponding factors.
Using only entropy, weights for class e still increase regardless of its CPR and TRAP factors. Con- sequently, samples with different ground truth c' + c, the increased weights related class e result in larger logits of raising the likelihood of an incorrect prediction as class c. To maintain the prediction samples with ground truth c', the value of updated weights' corresponding factors should be small to reduce logit changes. Without prior knowledge of distribution shifts, we infer that CPR factors class hold smaller values in class c'. Thus, updating only the weights correspond- ing CPR factors of class can preserve the prediction. Utilizing PLPD filtering, which focuses on samples where the influence of CPR factors outweighs that of TRAP factors, allows for the selective update of weights corresponding to CPR factors of class while minimizing the change in weights for TRAP factors. Therefore, using PLPD filtering can preserve the original predictions of samples with different ground truths.
25

---

Published as a conference paper at ICLR 2024
Compared to mild scenario, wild scenarios suffer from lack of diversity in batch information of the distribution of disentangled factors or even proceed with updates using a single image. exacerbates optimization stability and provokes stochasticity issues. Consequently, the drawbacks of entropy are more pronounced in wild scenarios than in mild ones, leading to a significant per- formance decline of baseline methods. In contrast, information CPR/TRAP factors, which only De discriminates, is less affected by the batch size or class dependency of an input stream com- pared entropy, offering improved robustness.
M LIMITATIONS
Identifying all the CPR factors present in input image is practically impossible. In other words, achieving perfect PLPD is practically impossible. Therefore, we focused on one of the fundamental components of human visual perception the shape of objects for reliable PLPD. In cases where model provides accurate predictions based on specific local features, these predictions remain unaltered even after patch-shuffling augmentation, resulting in a low PI.PD) value. Therefore, might be less useful to identify beneficial samples for TTA through a high PLPD strategy. Through the ablation studies, it is evident that PLPD-based sample selection has an advantage over entropy. However, observing that entropy retains its strength in sample weighting, it appears that the effeets of PLPD in sample weighting can be explored. These limitations suggest moving beyond the current approach, which relies solely on the last layer (softmaxx. We think that exploring the intermediate feature maps makes it possible to obtain other CPR factors in different hierarchies.
26


Reviews:
[{"review_id": "LlXYQCwQPI", "reviewer": "Reviewer_M673", "summary": "This paper presents a new method for test-time adaptation (TTA) called Destroy Your Object (DeYO) that uses a novel confidence metric called Pseudo-Label Probability Difference (PLPD) to improve the adaptation performance and stability of test-time adaptation methods. The authors demonstrate the limitations of entropy as a confidence metric and compare the performance of DeYO with other TTA methods on the ImageNet-C and ImageNet-R benchmarks under various mild and wild test scenarios.", "questions": "Pls refer to Weakness.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The proposed DeYO method is simple yet effective for improving the stability and performance of entropy-based TTA.\n\nThe idea and motivation behind Pseudo-Label Probability Difference (PLPD) are novel and interesting, providing new insights for the community.\n\nThe paper is strong on the empirical side. Extensive experiments with various model architectures, datasets, and mild/wild test scenarios are thorough.", "weaknesses": "The proposed terms “TRAP” and “CRP” are a bit hard to understand. The authors could refine the name and give more high-level/easy-understanding explanations about them in the Introduction. \n\nFor parameter sensitivity analyses in Figure 7, could the authors report more results under different model architectures, datasets and test scenarios? This helps demonstrate the hyperparameters’ generality. \n\nAblation studies in Table 6 are also highly encouraged to be conducted on different models, datasets and test scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WuW1DZ4IGy", "reviewer": "Reviewer_xMUg", "summary": "The authors found that using entropy as metrics is not enough in some biased scenarios. To address this, the authors devise a new metric namely Pseudo-Label Probability Difference. The experimental results demonstrate the effectiveness of the proposed metric.", "questions": "It would be better if the authors could explain the motivations more clearly.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe authors empirically and theoretically analysis the entropy metric for TTA.\n2.\tThe authors devise a metric namely Pseudo-Label Probability Difference that further improves the entropy metric.\n3.\tThe proposed method is easy to understand and implement. I believe it can be applied to real-world applications. In addition, the proposed metric only requires negligible computational cost to compute, which would not introduce obvious latency compared with EATA or SAR.", "weaknesses": "1.\tCould the authors give simple explanation TRAP factors and CPR factors? With this, the readers may easily to capture the motivation of the proposed metric.\n2.\tCould the authors explain more about the motivation of the choice of patch shuffled input as x’?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "R16OniJLoe", "reviewer": "Reviewer_7TAy", "summary": "The paper works on test-time adaptation with entropy minimization. While the online updates with entropy minimization can lead to error accumulation, the paper proposed to do sample selection and weighting by the proposed DeYO, which combines entropy and pseudo-label probability difference. Experiments on several datasets with various distribution shifts show the effectiveness of the method.", "questions": "1. Did the authors try some other methods like data augmentation methods to replace the transformed one? Will they also work to incorporate CPR factors?\n\n2. How do the thresholds in eq. (9) defined?\n\n3. What are the numbers and sizes of the patches for patch shuffling? Will these also influence the adaptation?\n\n4. Why PLPD and entropy sample selections behave differently for different distribution shifts in Table 6? How to select different methods for different distribution shifts? Is there any theoretical support for this problem?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The observation and theoretical demonstration of \"entropy is not enough\" on the spurious correlation shifts is interesting and motivating to the method.\n\n2. The results of the proposed method on several datasets with different distribution shifts are good, demonstrating the effectiveness of the proposed method.", "weaknesses": "1. The motivating observations and theoretical support are conducted on spurious correlation shifts, which is mainly on the semantic level. Can this also be found and theoretically proof for the other distribution shifts like covariate shifts?\n\n2. In 2.3, the paper theoretically demonstrates that entropy is not enough and it is better to incorporate the CPR factors for sample selection and reweighting in test time adaptation, which is done by the proposed PLPD. However, PLPD is then combined with the common entropy method in the experiments and implementations. Then what role does entropy play in sample selection? and how it helps  PLPD to incorporate the CPR factors?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "I6CScj5bYF", "reviewer": "Reviewer_NrgV", "summary": "TTA approaches select good samples for TTA using entropy as a metric (some of the approaches, and not all as I point in my review). However, a sample can have low entropy but using spurious features for prediction. Including such samples for TTA can be hurtful as they use spurious features for low entropy. The authors propose patch shuffle augmentation to identify such samples and remove them. \nThe paper has missing important baseline for some core experiments (Table 2 and Table 3), and it is not clear why the approach helps in WILD distribution shift (label shift, batch size 1).", "questions": "See the weakness section for questions to answer during rebuttal", "limitations": "", "rating": 8, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "Strengths:\n- The problem is well motivated i.e. spurious features can cause entropy to be low.", "weaknesses": "Weakness:\n- Unclear what the authors mean by “disentangled latent vector” exactly for each image. No proper reference has been made as well to get details. It would be better to write more explicitly how does Equation 6 arise for fluency of the reading flow. Authors mention $x^{T>>C}$, please describe this notation (and might be not even required to introduce this as it is clear authors mean second term is more pronounced).\n- It is not a great practice to include the whole of related works in the appendix. Even the related works section on TTA in appendix is extremely sparse (a single paragraph!) and should be much more broader in it’s scope.\n- I understand that authors aim is to reject samples which use spurious features for prediction as doing TTA on them might be harmful. However, past theoretical work (https://proceedings.neurips.cc/paper/2020/file/f1298750ed09618717f9c10ea8d1d3b0-Paper.pdf) shows that self training avoids using spurious features. \n- There are many methods which assess augmentation + entropy (see MEMO, SENTRY https://arxiv.org/pdf/2012.11460.pdf). SENTRY does sample filtering based on consistency of predictions over various augmentations. It is a crucial and missing baseline in this work, especially in Table 2 and Table 3. I know SENTRY operates in UDA setting, but the consistency over augmentations part is pretty standard. \n- The authors mention existing works use entropy for selective minimization. There should be a baseline where we do selection based on average entropy over multiple augmentations. It is a trivial extension of previous works and should have been a baseline to asses the effectiveness of pseudo-label confidence drop proposed in this work.\n- Why is the MEMO baseline missing in Table 2 and Table 3, where I would guess MEMO to be most effective as it also does entropy average over multiple transformations. It is quite possible that the average entropy over augmentations is low for spurious correlations domainted prediction, making those samples contribute much less to the loss anyways.\n- Can the authors give any reason why their proposed approach should work better under label shift or for batch size 1 as shown in Table 5? I do not see any reason why augmentation based sample selection should help for label shift TTA (makes sense for spurious TTA). Similar is the question for Batch size 1 setting. Are the authors sure they did proper hyperparameter tuning for the baselines?\n- Further, the effect of such spurious correlations can be removed by incorporating such transforms (cutmix, random cropping, or the used patch shuffling) during pretraining itself. Will the proposed approach help in those cases as well?\n\n\nOverall, the paper does consider an important question. However, misses some very important baselines. Further, some of the results on some benchmarks have been left unexplained beyond simply stating the numbers. The writing also lacks heavily, with a complete miss on the related works section.\n\n\n_______post rebuttal_____\nAuthors addressed some of my major concerns with respect to the baselines. I am increasing my rating from 5 to 6.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)