Test-Time Adaptation Induces Stronger Accuracy and Agreement-on-the-Line
Authors: Eungyeup Kim
Mingjie Sun
Christina Baek
Aditi Raghunathan
J Zico Kolter
Venue: NeurIPS 2024 poster (2024)
Keywords: accuracy-on-the-line
agreement-on-the-line
test-time-adaptation
distribution-shift
Primary area: safety_in_machine_learning

Abstract: Recently, Miller et al. (2021) and Baek et al. (2022) empirically demonstrated strong linear correlations between in-distribution (ID) versus out-of-distribution (OOD) accuracy and agreement. These trends, coined accuracy-on-the-line (ACL) and agreement-on-the-line (AGL), enable OOD model selection and performance estimation without labeled data. However, these phenomena also break for certain shifts, such as CIFAR10-C Gaussian Noise, posing a critical bottleneck. In this paper, we make a key finding that recent test-time adaptation (TTA) methods not only improve OOD performance, but it drastically strengthen the ACL and AGL trends in models, even in shifts where models showed very weak correlations before. To analyze this, we revisit the theoretical conditions from Miller et al. (2021) that outline the types of distribution shifts needed for perfect ACL in linear models. Surprisingly, these conditions are satisfied after applying TTA to deep models in the penultimate feature embedding space. In particular, TTA causes the data distribution to collapse complex shifts into those can be expressed by a singular "scaling" variable in the feature space. Our results show that by combining TTA with AGL-based estimation methods, we can estimate the OOD performance of models with high precision for a broader set of distribution shifts. This lends us a simple system for selecting the best hyperparameters and adaptation strategy without any OOD labeled data. Code is available at https://github.com/EungyeupKim/TTALine.

Test-Time Adaptation Induces Stronger Accuracy and Agreement-on-the-Line
Eungyeup Kim
Mingjie Sun *Cannegie Mellon University "Bosch Center for Al feungyeuk, mingjies, kbaek, raditi, zkolter)0cs cmu. edu
Christina Baek' Aditi Raghunathan'
J. Zico Kolter
Abstract
Recently, Miller et al. [32 and Baek et al. [3] empirically demonstrated strong linear correlations between in-distribution (ID) versus out-of-distriutioon (OOD) accuracy and agreement. These trends, coined accuracy-on-the-
(ACL) and agreement-on-the-line (AGL), enable OOD model selection and performance es- timation without labeled data. However, these phenomena also break for certain shifts, such as CIFARIO-C Gaussian Noise, posing a critical bottleneck. In this pa- per, make key finding that recent test-time adaptation (TTA) methods not only improve OOD performance, but drastically strengthen the ACL and AGL trends in models, even in shifts where models showed very weak correlations before. To ana- lyze this, we revisit the theoretical conditions from Miller et al. [32 that outline the types of distribution shifts needed for perfect ACL in linear models. Surprisingly, these conditions are satisfied after applying TTA to deep models in the penultimate feature embedding space. In particular, TTA causes the data distribution collapse complex shifts into those can be expressed by a singular "scaling" variable in the feature space. Our results show that by combining TTA with AGL-based estimation methods, we can estimate the OOD performance of models with high precision for broader set of distribution shifts. This lends simple system for selecting the best hyperparameters and adaptation strategy without any OOD labeled data. Code is available hhttps::///// githu...t.... cow./Eunggyeupn///AAL....
1 Introduction
Neural networks often fail to generalize out-of-distribution (OOD) data that differs from the in- distribution (ID) data seen at train-time [ 41]. Thus, characterizing the behaviors of these models under distribution shift becomes crucial for reliable deployment. However, it is often extremely challenging to reliably estimate their performances because in many practical applications, OOD labeled data is scarce. Interestingly, recent studies [32 have found laws that describe the behavior of models across many distribution shift benchmarks. In particular, across numerous distribution shift benchmarks, the models' ID versus OOD accuracies, under probit scaling, tend to observe a strong linear correlation across numerous distribution shift benchmarks. Additionally, when accuracy is strongly correlated, the ID and OOD agrecment rates between pairs of these models are also strongly correlated with nearly identical slopes and biases. These phenomena, respectively referred as "accuracy-or -o-tec--o-t-e-ieeee (ACL) (32 and "agreement-on- coroonntteleee (AGL) 3. can be leveraged for precise OOD accuracy estimation without access to OOD labels: one could estimate the slope and bias of the ID vs OOD accuracy trend using agreement rates, then linearly transform ID accuracy using this approximate linear fi
of simple empirical
'Throughout this paper, we AGL comprehensive for indicate when models show strong linear trends both accuracy and agreement, and the slopes and biases these trends are identical.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Vanilla
BN Adapt
TENT
ETA
Gaussian Nosse 1.
Apperres, KFe.5
Agromment of 0.35 Aecuaiyy alla
Ady poroment,, 871
Accy ela
Aeculain, aff
Arsuniy Klg
8.5
45
85
03
05
CIFARIO test
CIFARIO CIFARIO-C Gaussian Noise
Vanilla
BN Adopt
TENT
ETA
Aureervenr Accuncyy MTALE
Agrnereent
Agrenvent,A
Shot Nolse
Accutny
Accurnyy Kf12
Arruniyy
A1
13
0.
65
03
ImageNeltest
(b) ImageNet vs. ImageNet-C Shot Noise
Vanilla
TENT
Varilla
TENT
Camelyon O0D
WitcAAM DOD
Camelyon117 test
WildCMM test
(c) Camelyon17 vs. Camelyon17-OOD
(d) iWildCAM vs. iWildCAM-OOD
Figure I: Linear trends in both accuracy and agreement hold substantially stronger degree after applying adaptation methods than before. Each blue and pink dot denotes the accuracy and agreement, followed by the linear fits for each, and R²2 is correlation coefficient.
56 48 have demonstrated that for distribution shifts benchmarks the linear However, studies 32
trends breaks down catastrophically. As shown in Fig. such as CIFARIO-C Gaussian Noise [4] or Camelyon II- 4T), models without TTA (i.e., Vanilla) with around 92 95% ID accuracy can have OOD accuracies that vary between 10 - 50%, so ID accuracy alone becomes extremely unreliable for understanding the OOD performance of models. Similarly, the correlation strength of ID and OOD agreement rates also weakens. Ideally. we would like to intervene in models in way that improves these linear correlations, such that we can reliably predict their performance under distribution shift. While theoretical works (32 31 51 25 provide insights for when these linear trends hold, there is little study on how to strengthen these trends in models.
In this paper, we empirically demonstrate that recent OOD test-time adaptation (TTA) strategies 44 (7 46 53 9 35| 55 36) not only improve OOD performance, but significantly restore the strong linear trends for broad range of distribution shifts where ACL and AGL are not initially observable. For instance, in Fig. the correlation coefficient (R²) of ID versus OOD agreement and accuracy of Vanilla models is 0.13 1 and 0.39, respectively, and both of these values improve to 1.0 after applying TENT [53] We observe such stronger linear trends after TTA throughout our extensive testbed consisting of 9 shifts, 7 adaptation methods, and over 40 network architectures. As test-time adapted models have OOD accuracies that degrade more predictably with respect to the ID aceuracy, we are also able to utilize AGI. based method ALine K to obtain precise OOD performance estimates without any OOD labels. Note that no OOD labels were utilized throughout this procedure, neither during TTA ALine. Our estimates of the models' OOD performances are drastically more precise after adaptation, e.g., estimation error of 11.99% in Vanilla models without TTA vs. 2.34% after applying TENT on CIFARIO-C Gaussian Noise.
Given that TTAS are designed to enhance OOD accuracy, the observation of such strong linear trends is unexpected and non-trivial. While some studies [29] 9 have explored how adaptations lead to improved OOD generalization, these efforts are orthogonal to those investigating the conditions under
which linear trends occur 32 31 To our knowledge, no studies have attempted to bridge these two
areas of research. This naturally raises the question: Why does adapting models at test-time to OOD data lead to stronger linear trends?
2

---

To answer this, we revisit the theoretical analysis in Miller et al. [32 of the sufficient conditions for observing highly correlated ID vs OOD accuracy in linear classifiers and Gaussian data. Theoretically, ACL holds exactly under distribution shifts where the direction of the class means and shape of the class covariances are fixed, and only the scale of the mean or covariance changes. Surprisingly, we find that TTA, in practice, seems to enforce exactly this condition: after applying TTA, the cosine similarity between the means and covariances of the penultimate-layer feature embedding of ID and OOD data tend to hover around 1, while their magnitude may change by some scaling constant. This implies that adaptations effectively collapse the complexity of the distribution shift to a singular "scaling" variable in the feature space. In addition to (empirically) justifying the use of TTA for strengthening ACI this discovery casts some insight into the nature of TTA in general, which has previously been largely heuristic approach.
Furthermore, models adapted with different TTA hyperparameters tend to lie on the same linear trend. Fig. rig.Billustrrate the strong correlation among models of various architectures first trained on ID data for various numbers of epochs, then adapted with different learning rates, batch sizes, adaptation steps. This critically allows us to tune the TTA hyperparameters and choice of TTA method without held-out OOD labeled set. Notably, correctly hyperparameter tuning without access to labels is an important challenge faced in practice that currently lacks principled approaches [2] 60 9 Using ACL and AGL, we are able to select models with accuracy less than 1% away from that of the best model OOD, e.g.. in CIFAR IO-C. This also allows to select the best TTA methods by comparing their estimated best OOD performances.
To summarize our contributions:
We observe after TTA, ACL and AGL hold across a wider set of distribution shifts and hyperparameter settings.
We explain our observation by showing TTA collapses the distribution shift to just constant scaling of the mean and covariance matrices in the feature space. This satisfies the theoretical conditions studied previously for observing strong linear trends.
Our findings provide a simple and effective strategy for finding the best TTA hyperparameters and the best TTA strategy without any OOD labels.
2 Related Work
Understanding accuracy and agreemeent-nn-hhe-inee Miller et al. [32 and Baek et al. 3 empirically observed coupled phenomena in deep models when evaluated on many standard distribution shift benchmarks: the ID vs. OOD accuracy and agreement are often strongly correlated
and the linear fits match almost exactly. Recent studies [32 31 51 25 24 attempt characterize what types of shifts lead to (or break) this phenomena. Miller et al. 32 prove that under a simple Gaussian data setup, ACL does not hold perfectly under distribution shifts that change the direction of the mean or transforms the covariance matrix. For example, they demonstrate that adding isotropic Gaussian noise to CIFARIO whose covariance is anisotropic, causes the linear trend to break. Mania and Sra (31] provided sufficient conditions directly over the outputs of trained models, in terms of their prediction similarity and distributional closeness. Tripuraneni et al. [51] and LeJeune et al. (25 show that ACL holds asymptotically under certain transformations to the covariance matrix. On the other hand, there has been comparatively little theoretical analysis of AGI. and why it appears together with ACL. Lee et al. (24] show that in random feature linear regression, AGL can break break partially, i.e., the slope of the agreement trend matches accuracy's, s, but the biases may be different. While further theoretical conditions are necessary to guarantee when these phenomena hold jointly, in our empirical findings, we see that the slopes and biases do always match across the wide variety of distribution shifts we test.
Extending upon such theoretical studies, we demonstrate that simple model intervention by test- time adaption allows these ID versus OOD trends to hold even stronger for more expansive set of distribution shifts. fact, we demonstrate that after adaptation, models actually satisfy the theoretical conditions necessary for ACI. as described in Miller et al. 332
2 AGL has not been similarly shown to hold theoretically for this case of scaled means, but empirically it often seems to hold when ACL holds 3
3

---

Learning
Adapt
Batch
Checkpoint
Gaussian Noise
Agrecmene, #P0.39
Agresmentt 61:2.00
Agresment, Accurocy, #f11.00
Agrissment, Accutacy #55.55
Acarsoy, R4-09>
Acour ocy, RP.100
37
CIFARIO test
(a) TENT tested on CIFARIO vs. CIFARIO-C Gaussian Noise
Learning Rate
Adapt Step
Batch Size
Checkpoint
Gaussian Noise
Agreament, 8h:087
Agrenment, A1-04
Agreement, A7:0.75
Agrnement, 4
0. 2.
21
02-
0.
0.
055
37
03
05
01
05
67
ImageNet test
(b) ETA tested on ImageNet vs. ImageNet-C Gaussian Noise
Learning
Adapt Step
Camelyon17 OOD
Agresment 67:2.95
Agresment, A7.5
Agroement 87.00
Accurocy, RP..37
Accuracy, A9-0.8
Accuracyy 89-052
8--
41
Cameylon17 test
(c) TENT tested on Camelyon17 vs. Camelyonl7-000
Figure 2: Strong AGL by adaptations with varying hyperparameters, including learning rates, adaptation steps, batch sizes, and early-stopped checkpoints of the ID-trained model. Each blue and pink dot denotes the accuracy and agreement, followed by the linear fits for each.
Adaptations under distribution shifts and their pittalls of reliability. Test-time adaptation aims to enhance model robustness by adapting models to unlabeled OOD test data. Test-time training methods [46] 29 7 involve learning shift-invariant features via solving self-supervision tasks. Other approaches include computing statistics in Batch Normalization (BN) layers using OOD data [20 44), instead of using ID statistics stored during training. Subsequent studies further adapt BN parameters by updating them using entropy minimization
53 35 36
Another popular approach is self-training
with pseudo-labels
40
9
One critical challenge in TTA is that, without OOD labels, it is prohibitively difficult to evaluate how effective the adaptation methods might be. As pointed
previous studies, adaptation methods may not succeed to address the full spectrum of distribution shifts, such as datasets reproductions [29] 60),
domain generalization benchmarks [2 61), and WILDS 40 Furthermore, these approaches
are known to be sensitive to different hyperparameter choices 4 61 36 23 Practitioners must
take a great care in optimizing such hyperparameters, but their tuning procedures lack clarity. They often follow the settings of the previous studies [35] (6), or rely some held-out labeled data [21 660 that unavailable practice. There exists a line studies on unsupervised model
validation 33 42 34 52 17.. Our study leverages ogreement-on-thellee phenomena within 77A to
offer a promising solution for these reliability issues.
Adaptations lead to stronger Agreementton--the-Linne
3.1 Experimental setup
Datasets and models. Our testbed includes diverse shifts, including common corruptions (15 failure shifts in CIFAR 10-C, CIFAR 100-C, and ImageNet-C [14), dataset reproductions (CIFAR10.1 [X
ImageNetV2 (39,, and real-world shifts (ImageNet-R [66, Gamellon/7-WIILDD iWildCAM-
WILDS, FMOL-WILDS [1).. Among them, CIFARIO-C Gaussian Noise, Camelyonl7

---

and WWWLDAM-WWLDS display the weakest correlations in both ID versus OOD accuracy and We test over 30 different architectures of convolutional neural networks (e.g..
agreement
32
3
56
VGG
45
ResNet 13
57
59 DenseNet [9]. MobileNet 43 and Vision-Transformens (ViTs [5].
DeiT
Swin'T
30
TTA methods. We investigate 7 recent state-of-the-art TTA methods, including SHOT [27). BN_Adaa [44) TTT [6), TENT [53] ConjPL [ ETA 35 and SAR [36]. This testbed in-
cludes different training strategies (e.g. self-supervision 46 PolyLoss (26 and updating certain layer parameters (BN layers
44
53 351, LayerNorm (LN) layers 2 36), entire feature extrac-
tors (27). We test all adaptation baselines, except SAR, on convolutional neural networks with BN layers. We apply SAR specifically to vision transformers [5] 49 30 because it prevents the model collapse that other udaptation methods exhibit in vision transformers with LN layers [6].
Calculating agreement. Given any pair of models (h, E H that are tested on distribution D, the expected accuracy and agreement of the models is defined as
Accuracy(h) (h) Eu (h(x) = v}], Agreement(h,, (h, h²) E (h(x) N(())3] (I)
where h(x) and h (x) are the normalized logits of models h and h' given datapoint x and y is the class label. Following (32 and [3], we apply probit scaling, which is the inverse of the cumulative
density function of the standard Gaussian distribution (Φ (0-1-_ [0, 1] [-o0, oo)), over accuracy and agreement for a better linear fit, specifically in Figs. and
Online test in ID and OOD during TTA. Unlike conventional offline inference at test-time, TTAs involve online learning. which dynamically updates the model's 's parameters while testing on OOD data. To evaluate this continuously updated model on both and OOD data, the data is fed into the model in minibatches and we collect the accuracy and agreement of the model over the i'th minibatch after the dynamic adaptation step i. Details are provided in Algorithm| ii the Appendix.
3.2 Main observation
We make a striking empirical observation in the models after applying TTA: the correlation strength between ID versus OOD accuracy and agreement increases by significant margin, especially in distribution shifts where models without TTA show wildly varying trends. In Fig. | we demonstrate this on the four distribution shifts that show weakest ACL and AGL trends Vanilla models: CIFAR10-4 Gaussian Noise, ImageNet-C: Shot Noise, Camelon 1-- WILDS, and WiildCAM-WILJD In partieular, the correlation coefficients also been noted by previous studies 32 3 56). Surprisingly, after applying TTA methods, such as TENT, the strength of these linear trends measured by 18² increase dramatically, (e.g., 0.15 0.97 and 0.33 + 0.97 in agreement and accuracy in Camelyon WI-WID Our observations hold consistently across our entire testbed of shifts and adaptation methods, as shown in Figs. 6 9 1O
most R 0.4 before TTA. These failure shifts have
and [2] Moreover, in shifts where models without TTA already exhibit strong linear trends, such as ImageNet- V2 or FMOW-WILDS, these linear trends persist even after TTA, irrespective of whether TTA improves or degrades the OOD accuracy (Fig. 4).
Furthermore, models adapted using the same TTA method but with different adaptation hyperparame- ters lie on the same accuracy and agreement linear trend. Specifically, we vary learning rates, the number of adaption steps, batch sizes, and the number of epochs the model is initially trained on ID data. In Fig. 2 we see that on CIFARIO-C and ImageNet-C Gaussian Noise, and Camelyon 17-WLDSS models adapted with different hyperparameters exhibit correlation strength Ri2 close to in both ID versus OOD accuracy and agreement. We show that this occurs across distribution shifts in Figs. and 13
4
Why does adaptations lead to strong linear trends?
In this section, we focus on explaining why TTA leads to restoration of these strong linear trends. Although do not provide complete theoretical justification, we identify key pattern in how TTA modifies models that provides a strong clue as to why these methods substantially strengthen ACL and AGL..
We begin by revisiting the theoretically sufficient conditions for ACL from Miller et al. 32 over a
simple Gaussian data and linear classifier setup. In this setting, ACL holds perfectly (R² = 1) only
5

---

when the distribution shifts by a simple scaling factor to the norm of the mean and covariance. On the other hand, if the actual direction of the mean or shape of the covariance matrix changes, the linearity breaks. We then demonstrate that even in CIFARIO-C shifts and deep neural networks, models after applying TTA meet these theoretical conditions better in the penultimate-las feature space.
4.1 Theoretical conditions for linear trends in Gaussian data
We briefly introduce the theoretical conditions for ACL in Gaussian data and linear models, from Miller et al. [3), restated here for ease of presentation. Consider the binary classification over Gaussian data setup with label E { - 1.111. 1.. The OOD distribution Q differs from the ID distribution P by just some scaling constants a, 7 >> O,
P(æ v) Nyy pG 2), Q(x 1v) -N (y ap;))))
Theorem [Miller et al. 132y, simplified] Under the Gaussian data setup in Equation 2 across all linear classifiers Ja x + sign(0 x). the probit-scaled accuracies over and Q observes perfect linear correlation with bias of zero and a slope of n
7
The proof follows immediately from the fact that the test accuracy of a linear classifier on this Gaussian data P is given by 2(0 p/ E0)): applying same result to Q immediately gives the desired linear relationship. The main implication is that if the distribution shifts only in the scale of the mean and covariance, while preserving their direction, ACL is guaranteed to occur across any set of linear classifiers. Note that we focus on the conditions for ACL in particular. Additional theoretical conditions are usually required to observe AGL together with ACL [24]. but they have not been setup. Thus,
well-chhracterizee especially for this for AGL,, if any, in our work. Still, we find that AGL is always tightly coupled with ACL across an extensive range of benchmarks and TTA methods, similar to Back et al. B
forgo analyzing assumptions
4.2 Empirical analysis under adaptation to CIFAR1O-C
We now investigate how well the above conditions are met before and after TTA for real-world data with deep models. Here, evaluate models trained on CIFARIO on CIFARIO-C Gaussian Noise. We adapt models using BN__dapi [44] or TENT [53] at test-time. To match the theoretical setup, analyze the class-wise feature embeddings from the penultimate-
of these models that directly precedes the final linear classifier. Namely, we can measure the "alignment" or the cosine similarity between the ID and OOD class means and covariances of these embeddings. Similar to CIFAR10-C Gaussian Noise results in Figs. [ and 2 we evaluate the embedding alignment of models in each hyperparameter setup: we adapt models at test-time varying a single hyperparameter, e.g.. architectures, learning rates, batch sizes, and early-stopped checkpoints, while keeping the remaining hyperparameters fixed.
Shifts become aligned in their mean and covariance after adaptation. Table showss the mean and standard deviation of the cosine similarity measured across different setups. We first notice that, without adaptation (Vanilla), both mean and covariance of CIFARIO and CIFAR10-C Gaussian Noise features are misaligned with cosine similarity much smaller than 1. Surprisingly, after adaptations, the similarity substantially increases and becomes very close to 1, across every architecture we test (with standard deviation close 0). In other words, in the feature embedding space, the two distributions have means and covariances that (roughly) point in the same directions after adaptation. This also implies that in adapted models, the distribution shift represented as a simple change in the scale of the features alone.
Consider CIFARIO- Gaussian Noise which corrupts CIFARIO images by additive isotropic Gaussian Noise. In the input image space, the direction of the covariance matrix clearly changes with the addition of this noise. Yet TTA has the non-trivial ability of being able to "simplify" complex drifts to scale shifts within the features. These observations support the theoretical conditions in Theorem and gives us loose insight into why TTA strengthens ACL and AGL in models in practice.
Theoretical slope matches the empirical slope. Finally, under scale shifts our theoretical setup, the slope of the ACL trend is exactly A = az/7. Assuming the direction of the mean and covariance remains fixed, we may estimate a and y as simply the average change in magnitude of the class means and covariances, and use this to approximate A. In Table I we compare how this theoretical

---

Cosine Similarity
Slope
Setup
Mean
Covariance
Theoretical
Empirical
Vanilla (Archs.)
0.691 + 0.175
0,750 0.109
BN Adapt (Archs.)
0,988 + 0.007
0.972
0.011
0.751 +: 0.075
0.758
TENT (Archs.)
0.990 + 0.005
0.974 + 0.011
0.753 + 0.072
0.778
Learning rates
0.993 + 0.003
0.977 +: 0.006
0.759 + 0.041
0.76
Batch Sizes
0,995 + 0.003
0.982 0.010
0,831 + 0.101
0.809
Check Points
0.992 +0.003
0.976 + 0.008
0.782 + 0.033
0.838
Table 1: Cosine similarity between mean direction and covariance shape of class-wise penultimate- layer features, followed by the comparison between theoretical and empirical slope. They are evaluated on CIFARIO vs. CIFAR 10-C Gaussian Noise, measured across architectures and hyperpa- rameters. We report their means and standard deviations.
slope compares to the true slope of ID versus OOD accuracy fit by linear regression. These slopes tend to closely match each other, highlighting that the theoretical results of TTA seem to provide correct intuition about why ACL holds in practice, especially after TTA.
TTA improves linear trends irrespective of whether it improves OOD generalization. We make the subtle distiction between TTA's ability to strengthen ACL and AGL phenomena from its original purpose of improving OOD accuracy. One could imagine a trivial reason why ACL happens is because the ID and OOD gap diminishes after TTA, thereby bringing the ID versus OOD accuracy of all models close to the y = line. However, this explanation does not reflect our empirical findings. While TTA seems to reduce the complexity of the shift to scale shifts, the magnitude of this scaling factor may remain very large, i.e., a < 1, 7 >> This results in a near-perfect linear trend that, nonetheless, lies arbitrarily far from the y x. Consider the shift ImageNet-C: Shot Noise in Fig. I After adaptation, there is still approximately a 40% gap in and OOD accuracy, but ACL holds strongly. On the other hand, consider the counterfactual TTA method which improves OOD performance by decreasing the covariance scale y in the feature embedding space, but the means/covariances remain misaligned. Then the linear trend moves closer I, but there is guarantee about the strength of the linear trend.
Experiments
Our finding that TTA induces strong ACL and AGL has two key practical applications: (i) estimation of OOD accuracy, and (ii) unsupervised validation for TTA without access to any OOD labels.
5.1 OOD Accuracy estimation after adaptation
Experimental Setup. We employ AGL-based estimation method, namely ALine (3]. This method first estimates the slope and bias of the linear trend in ID versus OOD accuracy, via agreements (which requires no labels), and uses them to linearly transform ID accuracy to get an estimate of OOD accuracy. The full details of ALine are illustrated in Algorithm/2ja iiin Appendix. We apply ALine to models after TTA, and compare these results with (i) those of ALine applied models before adaptation, and (ii) estimates using existing estimation baselines. These baselines include average thresholded confidence (ATC) S difference of confidence (DOC)-feat [10, average confidence [15]
and agreement 22 We evaluate baselines on CIFAR10-C, CIFAR 100-C, ImageNet-C [14] and CamelyonI7-WILDS and SELLCAM--WLDDS 41]
Results. ALine shows accurate estimation under shifts where models demonstrate strong AGL (e.g., in CIFARI0-C snow, mean absolute error (MAE) of estimation 0.93% in ALine-D), but critically fails when shifts do not have such linear trends (e.g.. in CIFAR10-C Gaussian Noise, MAE is 10.76% and in Camelyon (eeelloon7--WLDDD WILDDS MAE is 12.85 12.88%). As result, as shown in Table [ estimates using ALine-S/D on Vanilla models are sometimes error-pront Next, after applying adaptation methods, such as SHOT, BN_ Adapt, and TENT, the estimation performance ALine-S/D improves, showing substantially lower MAE compared to that of Vanilla models (e.g., MAE decreases 10.76% + 2.34% in CIFARIO- Gaussian Noise and 12.88% - 1.42% in Camelyon
ALine on top of TTA also outperforms other bascline methods most of the distribution shifts tested.
7

---

Dataset
Method
Error
ATC
DOC-feat
AC
Agreement
ALine-S
ALine-D
Vanilla
31.38
8.31
15.03
17.42
5,45
6.02
5.87
SHOT
15.40
1.63
4.63
7.63
1.78
0.77
BN_ Adapt
16.87
3.69
4.79
7.53
1.93
1.12
0.91
CIFARIO-C
TENT
15.43
4.25
4.65
7.66
1.79
0.97
0.77
ConjPL
16.62
1.80
6.16
11.46
2.02
1.18
1.01
ETA
15.14
4.58
4.50
7.68
1.76
0.92
0.72
Vanilla
59.04
5.05
12.82
18.34
6,96
7.49
7.22
SHOT
40.79
2.21
5.44
14.36
2.52
1.64
0,90
CIFARI00-C
BN Adapt
42.69
2.89
4.42
11.81
2.33
1.13
TENT
41.11
6.60
14.85
2.65
0.88
ConiPL
42.79
1.09
6.55
23.73
2.40
1.67
1.18
ETA
44.27
7.15
4.92
16.49
4,96
1.44
0.81
Vanilla
80.41
3.95
13.72
17.34
9,06
6.00
5,95
BN Adapt
69.05
7.37
2.63
2.86
3,91
6.16
6.09
ImageNet-C
TENT
56.58
5.98
6.54
12.70
7.48
4,62
4.57
ETA
56,56
10.21
7.91
34.38
8.02
3,66
3.72
SAR
43.30
5.39
8.61
13.68
5.51
4.17
Camelyon17
Vanilla
34.07
14,91
17.31
21.69
11.95
12.88
13.46
-WILDS
TENT
14.37
3.00
3,43
6,94
6,49
2,29
2.27
ETA
16.43
3.05
4.38
6.85
5.33
2.24
1.42
IWilCAMM
Vanilla
50.27
7.12
2,73
3.00
3.53
2.82
TENT
47.39
5,44
3.20
28.03
3.55
2.59
2.96
WILDS
ETA
46.49
6.61
3,40
29.34
4.62
2.14
2.82
Table The results of OOD accuracy estimation, measured by MAE (%) between estimated and actual OOD accuracy. The gray shades denote the results calculated after applying adaptations, and bold texts indicate the smallest estimation error among estimators. We also report the classification error (%) in both Vanilla and adaptation methods for each dataset.
5.2 Unsupervised validation for TTA
In this section, we demonstrate that strong AGL allows us to validate the best hyperparameter for the OOD accuracy estimates to choose the best
TTA without any labels. Furthermore, TTA strategy overall.
Experimental setup. First, as we have shown, models adapted using particular method with varying hyperparameters tend to show strong ACL behavior. When ACL holds, we can simply select the best OOD model by selecting the one with best ID accuracy [32]. Thus, we first collect candidates by systematically sweeping over hyperparameter values, and select the model with the best accuracy. We focus TENT, optimizing its various hyperparameters including learning rates, adaptation steps, architectures, batch sizes, and early-stopped checkpoints of training. See our total pool of models in Appendix Table 6 We compare this naive strategy to existing unsupervised validation methods including MixVas [7], ENT (3), IM [34), Corr-C [52). and SND [42]. We evaluate them on four shifts, CIFARIO-C. ImageNet-C, ImageNet-R, and CamelyonI7-WILDDS
Results. Table reports the absolute difference (MAE (%) between the OOD accuracy of the selected model and the best OOD accuracy, within a set of models where we vary a particular TTA hyperparameter. We find that using ID accuracy for model selection consistently achieves competitive unsupervised validation results, often outperforming other baselines. We noticed that current state-of-the-arr unsupervised model selection methods, i.e., Val or IM, perform well on CIFAR10 ImageNet-C, and ImageNet-R, but they critically fail in Camelyon17-WILDD e.g., validation error of 7. 7.98% in Mix Val and 23.52% in IM. Such failures of existing baselines might come from their assumptions, e.g., low-density separation, that do not generalize to such distribution shifts. contrast, our method shows low MAE across all distribution shifts, including those where other baselines fail, e.g., 0.62% in Camelyon 1 7-WILDS. We note that there are cases, specifically models adapted with various learning rates for ImageNet-CC where MAE large (9.70%). We see that models optimized with very small learning rates, e.g., 10-5 deviate from the linear trend. Results on other TTA baselines' unsupervised validation are Table [
Extension to selecting best TTA strategy. After hyperparameter tuning for each TTA strategy using the unsupervised strategy from the previous section, we could further leverage ALine to then select for the best TTA method overall. Different TTA methods exhibit different slopes and biases in their ACL

---

HyperParumeter
CIFARIO-C
ImageNet-CC
MixVal
ENT
IM
Corr-C
SND
Ours
MixVal
ENT
IM
Corr-C
SND
Ours
Architecture
2.31
1,06
21.71
2.77
0.03
6.22
0.96
0.47
26.32
20.60
0.75
Learning Rate
6.97
8.88
2.24
11.56
1.87
0.72
12.75
20.49
1.49
20.18
12.61
9.70
Checkpoints
3.21
0.0
0.0
5.53
3.46
0.05
Batch Size
7.85
3,32
0.96
32.37
5.68
0.77
14.29
42.31
0.99
42.31
42.31
5.61
Adapt Step
0.85
0.0
1.02
0.23
188
3.09
2.17
0.30
Average
4.23
2.65
0.85
14.43
2.75
0.36
8.77
16.42
1.05
14.43
22.97
4.0
HyperParameter
ImageNet-R
Camelyon1-WWI.D
MixVal
ENT
IM
Corr-C
SND
Ours
Mix
ENT
IM
Corr-C
SND
Ours
Aschitecture
1.75
0.62
0.62
22.17
22.17
0.85
28.87
1.03
1.03
28.87
28.87
0.85
Learning Rate
3.12
10.16
4.73
19.16
19.16
2.8
0.91
48.37
46.41
48.37
48.37
1.14
Batch Size
1.83
35.88
0.08
35.88
35.88
1.74
46.67
46.67
40.45
40.45
1.37
Adapt Step
1.07
1.07
1.07
1.07
1.07
0.0
33.12
0.0
33.12
33.12
0.0
Average
1.94
14.18
1.62
19.57
19.57
1.34
7.98
32.29
23.52
37.70
37.70
0.62
Table 3: Results of unsupervised validation, measured by MAE (%) between OOD accuracy of model selected by best ID model and actual best OOD model. Our results are highlighted in shade.
Architecture
Learning Rate
Check Points
Batch Size
Adapt Step
Estimated Accuracy
0475
0253
0169
0.5
A7 Accuracy
Accuracy
Accuracy
Accuracy
Accuracy
shat
bn
tent
canjpl
eta
Figure 3: 2-D visualizations each TTA baseline's ground truth best OOD accuracy (x-axis) and estimated OOD accuracy of best-ID models (y-axis) marked in cross (x). Each color denotes different TTA baselines. Circle dot (o) represents estimates and ground truth accuracies averaged over all hyperparameter values.
trends, so use AGL-based estimators to estimate the best OOD accuracy across hyperparameters for each method separately, then compare these estimates to choose the best method. Specifically, we select between five TTA baselines BN Adapt, TENT, SHOT, ConjPL, and ETA on CIFARIO-C across all corruptions. For ease of presentation, we consider selecting the best TTA method after tuning a single hyperparameter for each method, with other parameters fixed. In Fig. 3 we plot using "x" the selected models' estimated OOD accuracy for each method versus each method's actual best OOD performances. Our method almost always selects the best OOD model for each method but also accurately estimates their performances, as shown by points lying close to the Furthermore, the ranking of TTA methods by OOD accuracy stays the same when using the estimates. We also plot in "o" the average ground truth and estimated accuracy across hyperparameters, This demonstrates that our approach effectively selects the best TTA strategy without OOD labels.
line.
6 Ablation Study: When does TTA not improve linear trends?
So far, we focused on numerous TTA methods that tend to dramatically improve the strength of ACL and AGL phenomena. We then connected this behavior models representing the distribution shift in its feature space as a scale shift, satisfying the theoretical conditions of ACL from Miller et al. (32). This behavior was especially prominent in methods that adjust the batch normalization layer statistics, egg. AN_Adapt, modify the feature extractor, e.g. TENT. This section presents an ablation study on the impact of normalization layers (e.g., BN, LN) and the components adapted by TTA (e.g., feature extractor, final classifier) for observing ACL and AGL.
Normalization layers.
As shown in our results for BN _dapt Section 4.1 the strength of the ACI and AGL trends tends to be sensitive to the statistics stored in the batch normalization layers. Taking the same set of models

---

from Section 4.1 we test two architectural variants models with no normalization layers (denoted as Vanilla w/o N) and layer normalization (Vanilla w/ LN) and report the cosine similarity of the class means and covariance matrices of the ID and OOD feature embeddings.
We also report the correlation coefficient (R²) of the ACL and AGL trends in Table 4 Inter- estingly, the mean and covariance alignment as well as correlation strength tends to be larger than that of Vanilla with BN-layers, but still falls short of models with BN-layers adapted with BN Adapt which is very close
Cosine Similarity
Correlations
Setup
Mean
Covariance
Acc.
Agr:
Vanilla w/o N
0.794
0.29
0.71
Vanilla w/ LN
0.935 0.042
0.854 = 0.063
0.56
0.86
Vanilla w/ BN
0.691 0.175
0.750 = 0.109
0.18
0.39
Bd_t
0.998 0.007
0,972 0.011
0.99
0.99
Table 4: Cosine similarity of mean direction / co- variance shape and correlation coefficients (R²)) in CIFAR10-C Gaussian Noise, measured across architectures. Last two rows are from Table [
Methods such as SAR which substitutes BN- layers with LN part of their adaptation strat- egy, also shows stronger linear trends in Fig. 12 than Vanilla counterparts in Fig. I1, but not as consistently as those of BN-layer-adappation methods in Fig. I We conjecture that this stems from how different architectures encode distribution shifts: BN-layers result in severe covariate shift, but most of the shift is encoded in the BN statistics, making it easy to align shifts via adaptation. In contrast, non- BN models encode shifts across the entire model parameters, which may dampen the covariate shift in the representations, but also make it harder to remove by adaptation. Schneider et al. 44 suggests a similar conclusion.
In this section, we consider the TTA method T3A (1], which leverages feature extractor without BN layers and only adapts the last linear classifier. This is in contrast to the other TTA methods we study in our work, which largely modifies the feature extractor. We adapt the same of models in Section 4.1 and observe that the 1t2 for accuracy and agreement in T3A are 0.80 and 0.46, which is substantially lower than models adapted using BN_Adapt. This may be because the features remain misaligned after T3A. Overall, our ablation studies clarify what TTA designs ideally lead to strong linear trends. Since Con-BN-TIRE are often adopted in numerous practical circumstances, e.g., single batch, it remains an important open question how to develop alternate TTA strategies that achieves strong ACL and AGL trends in such settings, thus improving the reliability of models.
Featurizer or classifier.
Conclusion and Limitations
In this paper, we provide a key observation that recent TTAs lead to stronger AGL across a wide range of distribution shifts, encompassing those with weak correlations before adaptation. We explain this phenomena by the complexity of distribution shifts being substantially reduced to those where the direction/shape of the mean and covariance remains identical, satisfying the theoretical conditions for linear trends. This naturally leads to enhanced estimation of OOD performances across a wider range of shifts than before TTA. We can leverage this to perform unsupervised hyperparameter validation and select for the best TTA method without any labels.
While the method we propose does not require OOD labels, AGL still requires sufficient ID data to accurately estimate agreement rate. As a result, our strategy may not be complementary with TTA methods that do not require any ID data. Moreover, could make our strategy computationally expensive. In Section D we conduct an ablation study to minimize the required amount of ID data for observing AGL for accuracy estimation. The results show that even with only 5% of the original ID data, OOD accuracy estimation performance remains nearly the same as with full access, outperforming other estimators. We believe that overcoming dependency on ID data and exploring a fully test-time approach for observing AGL remains a promising direction for future research.
Acknowledgement
We acknowledge the anonymous reviewers for their valuable feedbacks. Eungyeup Kim, Mingjie Sun, and Christina Baek are supported by funding from the Bosch Center for Artificial Intelligence. Aditi Raghunathan gratefully acknowledges support from Open Philanthropy, Google, Apple and Schmidt AI2050 Early Career Fellowship.
10

---

References
[I] M. Arjovsky, L. Bottou, I. Gulrajani, and D. Lopez-Paz. Invariant risk minimization, 2020.
[2] L.. Ba, J. R. Kiros, and G. E. Hinton. Layer normalization, arXiv preprint Xii:1600.006500 2016.
[3] C. Back, Y. Jiang, A. Raghunathan, and J. Z. Kolter. Agreement-on-the-lmnee Predicting the performance of neural networks under distribution shift. In Advances in Neural Information Processing Systems, volume 35, pages 19274-19288 2022.
[4] M. Boudiaf, R. Mueller, 1. Ben Ayed, and L. Bertinetto. Parameter-free online test-time adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8344 8353, June 2022.
[5] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenbor,, X. Zhai, T. Unterthiner, M. Dehghann, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. ICLR, 2021.
[6] P. Foret, A. Kleiner, H. Mobahi, and B. Neyshabur. Sharpness- minimization for efficiently improving generalization. In International Conference on Learning Representations, 2021.
[7] Y. Gandelsaman, Y. Sun, X. Chen, and A. A. Efros. Test-time training with masked autoencoders. In Advances in Neural Information Processing Systems, 2022.
[8] S. Garg. S. Balakrishnan, Z. C. Lipton, B. Neyshabur, and H. Sedghi. Leveraging unlabeled data to predict dutto--distiibution performance. NeurlPS 202/ Workshop on Distribution Shifts:
Connecting Methods and Applications, 2021.
[9] S. Goyal, M. Sun, A. Raghunanthan, and 7.. Kolter. Test-time adaptation via conjugate pseudo- labels. Advances in Neural Information Processing Systems, 2022.
[10] D. Guillory, V. Shankar, S. Ebrahimi, T. Darrell, and L. Schmidt. Predicting with confidence on unseen distributions. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 1 1134-1145 1144, October 2021.
[11] I. Gulrajani and D. Lopez-Paz. In search of lost domain generalization. In International Conference on Learning Representations, 2021.
[12] K. He, X. Zhang, S. Ren, and J. Sun. Identity mappings in deep residual networks. In European Conference on Computer Vision, 2016. URL httpss//api. http://///p..smmmniicchhhooooooo semanticscholar org/Corpus 64472777
[13] K. He. X. Zhang, S. Ren, and Sun. Deep Residual Learning for Image Recognition. In Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR '16, pages 770-778. IEEE, June 2016. doi: 10.1100/CVR..2016..90
[14] D. Hendrycks and T. Dietterich. Benchmarking neural network robustness to common corrup- tions and perturbations. Proceedings of the International Conference on Learning Representa- tions, 2019.
[15] D. Hendrycks and K. Gimpel. baseline for detecting misclassified and distrr-idittiittionn examples in neural networks. Proceedings of International Conference on Learning Represen- lations, 2017.
[16] D. Hendrycks, Basart, N. Mu, S. Kadavath, Wang, E. Dorundo, Desai, Zhu, S. Parajuli, M. Guo, D. Song, J. Steinhardt, and Gilmer. The many faces of robustness: A critical analysis of outtof-distributtion generalization. ICCV, 2021.
[17] D. Hu, J. Liang, Liew, C. Xue, Bai, and X. Wang. Mixed samples as probes for unsupervised model selection in domain adaptation. In seveth Conference on Neural Information Processing Systems, 2023.
[18] Hu, I.. Shen, and G. Sun. Squeeze-and-excialee networks. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), June 2018.

---

[19] G. Huang, Z. Liu, L. van der Maaten, and K. Q. Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017.
[20] S. Ioffe and Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 448-456, Lille, France, 07-09 Jul 2015. PMLR.
[21] Y. Iwasawa and Y. Matsuo. Test-time classifier adjustment module for model-agnostic domain generalization. In Advances in Neural Information Processing Systems, volume 34, pages 2427-2440. Curran Associates, Inc., 2021.
[22] Y. Jiang, V. Nagarajan, C. Baek, and J. Z. Kolter. Assessing generalization of SGD via disagreement. In International Conference on Learning Representations, 2022.
[23] A. Khurana, S. Paul, P. Rai, S. Biswas, and G. Aggarwal. Sita: Single image test-time adaptation, 2022.
[24] D. Lee, B. Moniri, X. Huang, E. Dobriban, and H. Hassani. Demystifying disagreement- on-the-line in high dimensions. In International Conference on Machine Learning (ICML., 2023.
[25] D. Leleune, J. Liu, and R. Heckel. Monotonic risk relationships under distribution shifts for regularized risk minimization. Journal of Machine Learning Research, 25(54):1-37, 2024. URL http://jmlr.. org/ wtt.pppers/v22222../
pwpprs/v25/22--aeeeeepp 22- 1197 .ttll
[26] Z. Leng, M. Tan, C. Liu, E. D. Cubuk, J. Shi, S. Cheng, and D. Anguelov. Polyloss: A polynomial expansion perspective of classification loss functions. In International Conference on Learning Representations, 2022.
[27] Liang, D. Hu, and Feng. Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. In International Conference on Machine Learning (ICML), pages 6028-6039, 2020.
[28] C. Liu, B. Zoph, M. Neumann, J. Shlens, W. Hua, L. Li, L. Fei-Fei, A. L. Yuille, J. Huang, and K. Murphy. Progressive neural architecture search. European Conference on Computer Vision, 2018
[29] Y. Liu, Kothari, B. G. van Delft, B. Bellot-Gurlet T. Mordan, and A. Alahi. Tu++: When does self-supervised test-time training fail or thrive? In Thirty-Fifth Conference on Neural Information Processing Systems, 2021.
[30] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEF/CVF International Conference on Computer Vision (ICCV), 2021.
[31] H. Mania and S. Sra. Why do classifier accuracies show linear trends under distribution shift?, 2021.
[32] Miller, R. Taori, A. Raghunathan, Sagawa, P. W. Koh, V. Shankar, P. Liang, Carmon, and L. Schmidt. Accuracy on the line: on the strong correlation between out-of- and in-distribution generalization. Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 7721-7735. PMIR, 18-24 Jul 2021.
[33] P. Morerio, J. Cavazza, and V. Murino. Minimal-entropy correlation alignment for unsupervised deep domain adaptation. International Conference on Learning Representations, 2018. URI. https://openreviezz feeee//orrmmiddrrreeee..
[34] K. Musgrave, S. Belongie, and S.NN Lim. Three new validators and a large-scale benchmark ranking for unsupervised domain adaptation, 2023.
12

---

[35] S. Niu, J. Wu, Y. Zhang. Y. Chen, S. Zheng, P. Zhao, and M. Tan. Efficient test-time model adaptation without forgetting. In The Internetional Conference on Machine Learning. 2022.
[36] S. Niu, J. Wu, Y. Zhang, Z. Wen, Y. Chen, P. Zhao, and M. Tan. Towards stable test-time adaptation in dynamic wild world. Internetional Conference on Learning Representations. 2023.
[37] Radosavovic, R. Kosaraju, R. Girshick, K. He, and P. Dollar. Designing network de- sign spaces. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
[38] B. Recht, R. Roelofs, L. Schimidt, and V. Shankar. Do cifar-10 classifiers generalize to cifar-10? arXiv preprint arXiv: 1806.00451, 2018
[39] B. Recht, R. Roelofs, L. Schmidt, and V. Shankar. Do ImageNet classifiers generalize to Ima- geNet? In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 5389-5400. PMLR, 09-15 Jun 2019.
[40] E. Rusak, S. Schneider, G. Pachitariu, L. Eck, P. Gehler, O. Bringmann, W. Brendel, and M. Bethge. If your data distribution shifts, use self-learning. Transactions of Machine Learning Research, 2022.
[41] S. Sagawa, P. W. Koh, T. Lee, I. Gao, S. M. Xie, K. Shen. A. Kumar, W. Hu, M. Yasunaga, H. Marklund, S. Beery, E. David, Stavness, W. Guo, Leskovee, K. Saenko, Hashimoto, S. Levine, C. Finn, and P. Liang. Extending the wilds benchmark for unsupervised adaptation. In International Conference on Learning Representations (ICLR), 2022.
[42] K. Saito, D. Kim, P. Teterwak, S. Sclaroff, T. Darrell, and K. Saenko. Tune it the right way: Unsupervised validation of domain adaptation via soft neighborhood density. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 818 9193, October 2021.
[43] M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018
[44] S. Schneider, E. Rusak, L. Eck, O. Bringmann, W. Brendel, and M. Bethge. Improving robustness against common corruptions by covariate shift adaptation. In Advances in Neural Information Prucessing Systems, volume 33, pages 11539-11551 11551. Curran Associates, Inc., 2020.
[45] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference Leurning Representations, 2015. 2015.
[46] Y. Sun, X. Wang, Z. Liu, Miller, A. A. Efros, and M. Hardt. Test-time training with self- supervision for generalization under distribution shifts. In International Conference on Machine Learning (ICML), 2020.
[47] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. Going deeper with convolutions, 2014.
[48] D. Teney, Y. Lin, S. J. Oh, and E. Abbasnejad. Id and ood performance are sometimes inversely correlated on weal-world datasets, 2023.
[49] H. Touvron, M. Cord, M. Douze, F. Massa. A. Sablayrolles, and H. Jegou. Training data-efficient image transformers & distillation through attention. In Proceedings of the 38th International Conference Machine Learning, volume 139 of Proceedings of Machine Learning Research. PMLR, 2021.
[50] H. Touvron, M. Cord, and H. J'egou. Deit iii: Revenge of the vit. In European Confer- ence on Computer Vision 2022. URL. https::/api semanticscholar wwwg/oorpusseeo. 248178188
13

---

[51] N. Tripuraneni, B. Adlam, and J. Pennington. Overparameterization improves robustness covariate shift in high dimensions. In A. Beygelzimer, Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems, 2021. URL https
[52] w. Tu, W. Deng. T. Gedeon, and L. Zheng. Assessing model ont-of-distributiion generalization with softmax prediction probability baselines and correlation method, 2023. URL https / /ooenreview..o aet/forumm id= 111mxxEyyeqxxx
[53] D. Wang, E. Shelhamer, S. Liu, B. Olshausen, and T. Darrell. Tent: Fully test-time adaptation by entropy minimization. In International Conference Learning Representations, 2021.
[54] J.-K. Wang and A. Wibisono. Towards understanding gd with hard and conjugate pseudo-labels for test-time adaptation. In Internetional Conference on Learning Representations, 2023.
[55] Q Wang, O. Fink, L.. Van Gool, and D. Dai. Continual test-time domain adaptation. In Proceedings of Conference on Computer Vision and Pattern Recognition, 2022.
[56] Wenzel, A. Dittadi, P. V. Gehler, C.-J. Simon Simon-Gabriel, M. Horn, D. Zietlow, D. Kernert,
C. Russell, T. Brox, B. Schiele, B. Schölkopf. and F. Locatello. Assaying outto--distributtin generalization in transfer learning. In Neural Information Processing Systems, 2022.
[57] S. Xie, R. Girshick, P. Dollár, Z. Tu, and K. He. Aggregated residual transformations for deep neural networks. arXiv preprini arXiv:161 1.05431, 2016.
[58] Yu, Wang, E. Shelhamer, and T. Darrell. Deep layer aggregation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
[59] S. Zagoruyko and N. Komodakis. Wide residual networks, 2017.
[60] M. Zhang, S. Levine, and C. Finn. MEMO: Test time robustness via adaptation and augmentation.
In Advances in Neural Information Processing Systems, 2022.
[61] H. Zhao, Y. Liu, A. Alahi, and T. Lin. On pitfalls of test-time adaptation. In International Conference Machine Learning (ICML), 2023.
14

---

A Details on Experimental Setup
A.1 Model Architectures
We list the types of architectures for each testbed of dataset.
ImageNet Architectures
CIFARIO, 100
Architectures
ResNet18 [3
ResNet34
13
ResNet18
ResNet50
13
ResNet26 13
ResNet101
13
ResNet34
13
WILDS
ResNet152
13]
ResNet50
13
Architectures
ResNeXT32x44
ResNet101
[3]
DenseNet121
19
ResNet18
[3
WideResNer288
[59
WideResNer50
55
ResNet34
PreActResNNet18
WideResNNet01
[59
ResNet50
PreActResNer34
DLA34 58
ResNet101
[3]
PreActResNNet50
DLA46C
58
ResNet152
[3]
PreActResNNtt001 12
DLA60
58
ResNeXT32x44
[57
RegNet X200
37
DLA102
58
DenseNet121
19
RegNet X400
37
DLA169
58
WideResNet50
[59
RegNet Y400
VIT-B/16
WideResNer101
159
VGG11 45
ViT-L/16
VGG11
45
VGG13 45
ViT-B/32
VGG13 45
VGG16 45
SwinTTSS 30
VGG16 45
VGG19 45
SwinT-B
VGG19
MobileNetV2
43
S-L
30
PNASNet-A
28
DeiT-S/16
49
PNASNet-B
[28
DeiT-B/16
49
SENet18 [8
DeiT3-B/16
GoogLeNet 47
DeiT3-L/16
50
DeiT3-H/14
50
Table 5: The list of architecture types for each testhed of datasets, including CIFARIO, CIFAR100, ImageNet, Camelyon/7-WILDDS and WILDS.
For CIFARIO, CIFAR100, and WILDS datasets, we train the models from the scratch, while for ImageNet, we use the pretrained model weights from torchvision and timm package. In addition, since TTT requires specific network composition required for the cotation-predictio task during pretraining, we train them using ResNet- 34,26,32,50,140 and 152, which are available in the original implementation
For Camelyon17 dataset, we also test the neural networks with their feature extractor (all parameters before the final linear classifier) randomly initialized, following Miller et al. [32 that tested the low-accuracy models on the dataset. We notice that these models, even with their most of parameters being randomized, still achieve high accuracy in both ID and OOD after training the last linear classifier. They also exhibit improved OOD accuracy after adaptations. We apply this randomization to ResNer18, 8, ResNet34, ResNet50, VGG16, VGG13, and VGGI1.
We trained and tested all models and datasets in NVIDIA RTX 6000.
A.2 Distribution Shifts
We test the models 9 different distribution shifts that include synthetic corruptions and real-world shifts. Here are details of tested benchmarks:
htaps//githbbbcoommyuuuprrrarare release
15

---

Synthetic corruptions. CIFARIO-C, CIFARIOO-C, and ImageNet-C [14] are designed to apply the 15 different types of coruptions, such as Gaussian Noise, on their original dataset counterparts. We use the most severe corruptions, which have severity of 5, in all experiments. These corruptions datasets are most commonly evaluated distribution shifts in a wide range of TTA papers
29
27
53
9
35
36 61].
Dataset reproduction. CIFARIO.1 and ImageNetV2 are reproduced version of their original counterparts, CIFAR 10 and ImageNet, respectively. They are collected by carefully following the dataset creation procedure of the original dataset, yet resultnng in non non-trivial performance degradations to models trained on original counterparts.
Style shifts. ImageNet-R R is variant of ImageNet which contains the images with renditions of various styles. Such styles include paintings, cartoons, sketches and others, which induces significant distribution shifts even when the semantics invariant.
Real-world shifts. Camelyon WIWIIDSS 41 contains the medical images of tissue collected from different hospitals, where distribution shifts are designed by different hospitals the model is trained and tested. The task is to predict whether it has tumor tissue or not (binary classification). We followed the same evaluation protocol of Miller et al. [32], collecting 30 whole-slide images (WSI) from hospitals as ID (total 302,436 for train and 33,560 for test). while 10 WSI from a different hospital as OOD (total 85, 85,054). W-WLDS [4]] contains the apatio-temporal satellite imagery of 62 different use land or building categories, where distribution shifts originate from the years that the imagery is taken. Specifically, following Miller al. [32.
IID set consists of images taken from 2002 to 2013 (training data of 76,863 and test of 11 1,,27), and OOD set taken between 2013 and 2016 (test data of 22, 108). 182 total species, which are taken with different camera traps. Following Miller et al. [32], we use 299 train images and 8, 154 test images taken by the same 243 camera traps ID. OOD data consists of total 42,791 images, which are taken with different camera traps, which involve different camera angle, background, and others.
MAMdCAM-WILDDD 41 contains the images of animals of
A.3 Adaptation hyperparameters
Table/ojshows the hyperparameter pools for each setup when used for observing AGL across different hyperparameters. We use SGD optimizer with momentum of 0.9 for all adaptation baselines except for SAR, which uses sharpness-aware mininization (SAM) optimizer 6
Setup
CIFARIO, CIEARI00
ImageNNt
5,5 3.1
4,5
1.10 5.2 5,
4,2
Learning Rates
5-10-3 3,100,1--00 1,
2,5 10 2,1 10-1, 1,5-0-1-11
5- 10
10 9,5 9,5 10-3
Batch Sizes
1,2,4,8, 16, 32, 64, 128, 256
4.8, 16, 32, 64, 128, 256, 512
100, 110, 120, 130, 1400, 130, 160, 170, 180, 190
Early-stopped Checkpoints
(out rotal epochs)
Adaptation Steps
1, 2, 3, 4,5
1, 1,4,,,4,5
(a) CIFAR10, 10, CIFARI00, ImageNet
Setup
Camelyon17-WIn
WWIdCAM-WILLS
2. 10 9,5 10 5,1 10 4,2 10 4,5 -. 10 4 10
2 9,5. 10 5,1 1. 10 4,2 10 5-10-4.1-1 10-
Learning Rates
2 9-10-3.5.
10-
10
10--10--22000 10-2
10 10-
Batch Sizes
4,8, 16, 32, 61, 128,
256, 512
1,2,4,8, 32, 61, 128
Early-stopped Checkpoints
Adaptation Steps
1,2,3, 4,5
1,2,3,455
(b) C 7-WIIDS, WildCAM-WILLDS
Table 6: The hyperparameter pools utilized observing AGL across hyperparameters CIFARIO, CIFARI00.
ImageNet, Camelyonl7-Wi 7-WILDS, and iWildCAM-WII dataset.
16

---

A.4 Online test in ID and OOD during TTA
Algorithm I provides the details of how ID and OOD performances are tested during TTA. TTA updates the model parameters at each iteration using the different batch of unlabeled OOD data (FOOD), described L6 of Algorithm I Note that during TTA. we also provide batch of ID data (ITD). After adaptation, we inference both ID and OOD data to the updated model to obtain its predictions (1.7-8), stored for performance evaluation at the end of test. Stored predictions are used for calculating not just accuracy, but agreements which require additional model.
Such online different from traditional testing. Traditional testing its pre-trained parameters, while the online "test" is testing a model that is continuously updated its parameters during TTA. In other words, the model tested iteration of é is different from that iteration t. Specifically in Algorithm I L6 makes the only difference between traditional and online testing, where L6 updates the model parameters every iteration before testing in L7 and L8.
the model that is fixed in
Algorithm 1 Online Test in and OOD during TTA
1: Inputs: pair data Ain, Vio Xoon, Voon model he-
2: Algorithms: Adaptation objective CrTA(.).
3:
4: Prediction sets Pm 0, Poon 1
5: for batch (Zw, Vm)), (ToOD, YoOD) in Xí,, Viu Koov, Voow do
+- 1, Lrv (2o(2)))))
>> Apply TTA
7: PP = Pw U he (Zw)
8: POOD Poon U he (TOOD)
end for
return Acc(Pi,. VID), Acc(Poo;
17

---

Vanilla
SHOT
TENT
ETA
CIFAR10.1
$1-000
97.100
Accenso
0.7
67
67
39
6.7
6
CIFARIO test
69
3.7
65
CIFARIO vs. CIFARIO.1
ImageNet-V2
$7..0
$13.00
Acartio
23
0.5
0.5
37
85
82
0.7
03
ImageNet test
07
33
15
0.7
0.9
(b) ImageNet vs. ImageNet-V2 V2
ImageNet-R
33-
Acaens ATA
22
22
0.1
23
ImageNet
63
(c) ImageNet vs, ImageNet-R
FMoW-OOD
Somen prace
65
65
FMoW
(d) FMoW vs. FMoW-OOD
Figure 4: The linear trends visualization of shifts that already exhibit strong AGL in Vanilla. Notice that adaptations do not break the linear trends, even when they lead to accuracy drops. We test SHOT, TENT, and ETA shifts such CIFAR1O 10 ImageNetV2, ImageNet-R, and FMoW-WILDS. Each blue and pink dot denotes the accuracy and agreement, followed by the linear fits for each. The axes probit scaled. scaled.
B Analysis on distribution shifts that have AGL without adaptations
We also examine the distribution shifts that exhibit strong AGL even before TTA, which are dataset reproductions (CIFARIO.1 (38), ImageNetV2 (39]), and other real-world shifts (ImageNet-R (16), FMoW-WILDS [4]), as shown in Fig. 4 The results show that adaptation baselines, such as SHOT, TENT, and ETA, do not improve OOD generalizations or even degrade them, which were also evidenced by previous studies (53) 61). However, they persist to have the strong linear trends, which implies that TTA does not break the linearity in shifts that show the strong correlations. This suggests that we can reliably predict if TTA may succeed or falter in wide range of distribution shifts.
C OOD accuracy estimation baselines
C.1 ALine-S and ALine-D
Back et al. 3 propose ALine-S and ALine-D, which assess the models' OOD accuracy without to labels by leveraging the agreement-on-the-le among models. We provide the detailed algorithm of ALine-S and ALine-D in Algorithm 2
C.2 Average thresholded confidence (ATC)
Garg et al. S introduce OOD accuracy estimation method, ATC, which learns the confidence threshold and predicts the OOD accuracy by using the fraction of unlabeled OOD samples for which model's negative entropy is less that threshold. Specifically, h(xx) e IR denote the softmax output of model h given data from XooD for classifying among classes. The method can be written as
18

---

Algorithm 2 ALine-S and ALine-D
I: Input: ID predictions Pw and labels Vis, OOD predictions POOD.
2: Function: Probit transform + (-), Linear regression F(-).
3:
4: à, b= F ((((a>((A((((AA(((( (Agr(Pii)), + ((Agr(Poon)))
Estimate slope and bias of linear fit b ALine-S
5: AccoOD Φ Acc(Pin ym) + b) m(m-l) b I2 -
m-8
6: Initialize R
8: for (py.D. Pk.m). (p,,000. Pk.000) E Pw. PooD do
9:
Ag = 2. Aa = 2 Au = 04l $ j,k
10:
b= (Agr(psos00
+ a. (3" (A (Aco(py in m.-bb)))
④ -- (A..)
2
11: i=i+1
12: end for
w"
arg wer" ||Aw - b112
14: Accoon (w Jvi E [n]
ALine-D
--S
15: return AccooD, Accood
below:
AcCoOD [[((s(((x)) < <t],,
(3)
where is the negative entropy, i.. s(((x)) E. ha(ar) log(he and satisfies
E [[{{s(((>)) < 01 = E arg max ho(x) + y}].
C.3 Difference of confidence (DOC)-feat
Guillory et al. [00 observe that the shift of distributions is encoded in the difference of model's confidences between them. Based on this observation, they leverage such differences in confidences as the accuracy gap under distribution shifts for calculating the final OOD accuracy. Specifically,
[15] [ max ho (rm) IB max he (2OOD))
AcCOOD = Accıp
(5)
C.4 Average confidence (AC)
Hendrycks and Gimpel 15 estimate the OOD accuracy based model's averaged confidence, which can be written as
AccoOD
EE] max h(rroo)]
C.5 Agreement
Jiang et al. (22 observe that disagreement between the models that trained with different setups closely tracks the error models in ID. We adopt this as the baseline for assessing generalization under distribution shifts, where we can estimate ACcOOD Agr(PooD) where Poop denotes the set of predictions the models on OOD data XoOD-
19

---

CIFAR10-C
CIFAR100-C
ALine-S
Aline-D
MAE (%)
ATC
MAE (%6
DoC-feat
Agreement
20%
2%
50%
Ratio of ID Data
Ratio of ID Data
ImageNet-C
Homelyon17-WiLe.
10
MAE (%)
MAE (96)
5%
10%
50%
2%
20%
50%
Ratio of ID Data
Ratio of ID Data
WHIldAM-WILLS
MAE (%)
106
2%
10%
20%
50%
Ratio ID Data
Figure 5: OOD accuracy estimation results with limited amount ID data, decreasing from 50% to 1% of entire data pool. We randomly sampled 10 different subsets for each ratios, and visualize the distribution of MAE (%) results. The results of baseline including ATC, DoC-feat, and Agreement included for comparison.
D Ablation study the number of ID data
Even if access to labeled ID data is often easily available in practice more than obtaining OOD data, ID data could be limited certain applications due to privacy issue or computational complexity. Therefore, this section further explores how sensitive our accuracy estimations are under conditions where the number of ID samples is critically limited.
Specifically, for each dataset, we gradually decrease the ratio of ID data for calculating both accuracy and agreement in ID, ranging from 50% 1% of entire data. To report the confidence interval of different random subsets, we iterate 10 independent sampling of data for each ratio. Box plots in Fig. Slsow that for CIFARIO-C and CIFARI00-6 with even 5% begins to become close to precision of using 10x more, i.e., 50%, achieving the state-of-the-an estimation performances outperforming other baselines. Also, in more complex dataset such as ImageNet-C, C WI-DSD and WWI.dCAM-IILDS, even using 1% of entire ID data achieves the best estimation performances among baselines. These results indicate that our framework practically feasible in such practical applications where access to ID is highly limited.
20

---

HyperParameter
CIFARIO-C
CIFARIO-C
BN_ Adupt
SHOT
TENT
ConjPL
BN_Adapt
ConjPl.
Leurning Rate
3.71
0.65
0.72
0.72
0.42
0,90
1.05
0.68
Adapt Step
0.04
0.23
0.23
0.24
0.12
0.63
0.0
0.55
0.63
Architecture
0.21
0.49
0.03
0.03
0.01
0.04
0.61
1.13
0.69
0.38
Batch Size
0.73
0.77
0.77
0.18
0.02
0.57
0.50
0.75
Checkpoints
0.0
0.48
0.07
0.05
0.01
0.11
0.01
0.00
0.24
0.0
0.0
Average
0.07
1,18
0.34
0.36
0.35
0.17
0.21
0.48
0.49
0.53
0.48
HyperParametter
Net-eNet-C
ImageNet-R
Camelyon17
IWBIСAM
BN Adapt
TENT
ETA
SAR
BN. Adapt
TENT
ETA
BN_ Adapt
TENT
BN Adapt
TENT
Leaming
Adapt Step
0.30
0.0
3.07
0.0
0.0
0.0
1.97
Architecture
2.23
0.39
0.24
0.75
0.06
0.0
0.0
4.08
Batch Size
0.00
5.61
5.70
7.21
0.0
1.74
4.02
0.0
1.37
0.15
2.04
Average
1.15
4.0
3.15
4.31
0.03
1.34
2.83
0.61
1.25
0.07
Table 7: Results of unsupervised validation of TTA methods other than TENT, measured by MAE (%) between OOD accuracy of model selected by best ID model and actual best OOD model. Since BN_ LdNat do not involve parameter updates, we exclude learning rate and the adaptation steps. In addition, TTT uses single batch, and exclude the batch size.
E Unsupervised validation results on TTA baselines
This section supplements Table by the unsupervised validation results of various TTA methods other than TENT, which include those of BN Ad_Adapt, TTT, SHOT, ETA, ConjPL, and SAR. Table reports the difference OOD accuracy (MAE (%)) between the model with best ID accuracy and the actual best OOD model, tested across different adaptation hyperparameters. The OOD model selection via best ID model achieves less than 1% MAE over almost every hyperparameter setup and adaptations, in CIFAR I0-C and CIFAR every corruptions in average. It also demonstrates that the hyperparameter optimization in ImageNet-C, , R and WILDS benchmark datasets have reasonably low MAE across diverse setups and adaptation baselines. These indicate that the strong linear trends across hyperparameters enable the selection near-optimal OOD adaptations across various shifts and TTA strategies.
21

---

F Additional results on CIEARIO-C, CIFAR100-C, and ImageNet-C
22

---

200m blur mation blur glass blur defocus blumpuusse noseehhtt polsegaussiaa noise
Vanilla
BN Adapt
SHOT
TENT
ETA
Aicenty M1
Panertin 41
0.7
6.9
0.7
6.
033
65
83
65
87
6a
1-
EEC
2.3
2 209
Accanie af
Accrtt 410
Acarty 41030
Acteroy a33
Actentyy aa3
0.7
0.9
0.7
0.9
0.7
0.9
0.5
87
0,9
05
87
G.G.
Acuacc afo
2.
077
07
07
0.7
2lasr
show 1
Frost
9.5
03
87
fog
C S
C S
S
C
G C
brightness
Actemmnt aPail
Agumet
jpeg compressiorreee celierrpeelllee latelt lastic www.ssarrma.ntttt... commmnnntaast
3.2
slose
2.
2/5
033
Agueet deo
224
07
87
87
CIFARIO test
Figure 6: Results on CIFARIO-C corruptions (different architectures), BN XN__dppt
SHOT 27
TENT 53 and ETA 35
23

---

mocton blur glass blur defocus blummpuls. nolseshor nols nois-gaussian no Se
glass blu defocus bluuppuuuuss nolse shot pols-grussaa hoGe
Vanilla (PolyLess)
ConjPL
Jonntyy shaae
04
06
2010
zoom blur motton blur
200m. blur
snow
snow
Aneoa 2i AM
Neaee pleat
Frost
frost
Fag
tog
brig brighnness
brigtnes
transformoontra
65
Jpgg compressoomeeeeec.c
Jpeg. compr
Aewenn Vrass
CIFAR1O
CIFAR10
Figure 7: Results on CIFAR10-C corruptions (different architectures), TTT
46 ConjPl. 9
24

---

nolsegauus naise
Learning
Adapt Step
Batch
Checkpoint
Z00m blur motion blur glasss, blur defecus www.pppluuustee compuuss,, naiseshot cecsshtt_nnissggeeeeeeee
Agumns
2 2000
Smaryy prast
CG2
81 11
D5
Aumay $11.5
COPE
201.00
Accatt $11.5
Satnsto $1..00
63 35
22
05
67
31
63
67 66
055
SHIDN 2 0000
6.10
63
21
Frost
Acalcc 17
Acrinc $11.0
61 03 03 67
05 87
31
03
65
0.5
20130
1ag
brightness
C CO0
jpeg comprrpeeee compressioriikel aceeel asticc Transformeenttras
1.10
Aucutto 211000
Jpmeres Hass
63 35 O3
32
C6
33
11
CIFAR10 test
Figure 8: Results on CIFAR10-C corruptions (adaptation hyperparameters of TENT
53
25

---

zoom blur motion blur glass blur deforus w...uuummppp pulse, naiseshat noisegaussia noise
Vanilla
BN Adapd
SHOT
TENT
ETA
Arneryy 111100
Artoryy 6715
Amaonyy
Scara Rlow
2.2
6.3
07
07
21
07
31
07
02
Anany $10.0
Asara R1.41
Asara R1Ga
Amarte $1
Asuenta dt
SPace
Agemer
3.7
63
07
07
33
07
33
07
33
07
Jaaeyy $1..0
Annnyy $16.00
Anary $1G81
Amertyy S11
Amearty 21.
22
Go
Snow
ADHEROLL APAie
frost
3.3
0.7
07
33
07
33
07
33
07
Tog
Animyy 616
Arany $10.0
Arany srods
Amarry S1A
Amerny phant
brightness
20046
2014
Aamp $11.0
Agtrmen
Agtrmnn PALES
joeg comprree compresstoppise wwwpppiellaaeee ateelastic transformeontras
2,
G
G
3.3
02
07
07
n7
47
3.7
$1..0
Anery S1631
Anery $71.5
Ameryy S7-011
Amernyy 27.
3.3
$10.0
Aeauy kroat
3.7
67
67
67
3.5
6
65
33
63
07
07
07
CIFAR100 test
Figure 9: Results on CIFAR100-C corruptions (different architectures). BN BN_Adapt 44
SHOT 27
TENT [53 and ETA 35
26

---

zoom, blur metion bur alos Blur defocus bluimpu be noseesher polsegausiin nol S2
Vanilla Lossy
Con
Umimay
65
41
25
02-
Agorral L8110
2.5
29
03-
Insen B1c
Lecrnn Sicee: V1c.
snow
Acrwnn Biices
Knwwny Ric
Frost
Fag
33
65
brightness
65
wwwppclllteeeetteecooo ceellstic transforeentraas
0.-
02-
13
09
65
Jesg cemppresiooriiclltee
02-
25
Vicat
vico
2.3.
CIFAR100
Figure 10: Results on CIFAR100-C corruptions (different architectures), ConjPL 9
27

---

zoom blur motion blur glass blur deforus bluimpulse, naiseshot noisegaussian noise
Vanilla
BN Adapt
TENT
ETA
Jan $1-047
Amortcy 2713
Apturtit d..
21
25
07
65
63 65
6
63
CCC
12
Aprerti d.000
2CC
07
07
CGC
SnOW
Frast
0-35 15
Apereer 21031
Aprentet R-:30
222
07
24
64
15
1eg
brightness
Jpeg. comprree cmppressiorpiia pplxeelateell lastic. grandormenntrass
Apermer P6610
2.1
Aptertr 4.0
03
ImageNet test
Figure 11: Results on ImageNet-C corruptions (different architectures), AN__ddat TENT and ETA (35]
28

---

aloo blur defocus bwwimpuuuueee nolsesher noisaguuususss no 10.32
Vanilla (Vision Wansformers)
SAR
motion blur
zoom blur
282
snow 222
Frost
222 2
Fog
65
13 15
brightness
04
25
transformcntrras
222
03
10
15
03
Jong www.crooooooop.coooo
ImageNet
Figure 12: Results on ImageNet-C corruptions (different architectures). SAR 36
29

---

nolsegauss naise 24.
Learning
Adapt Step
Batch
Checkpoint
200m blur motion blur glasss, blur defecus www.pppluuustce compuulse, naiseshot cecenot__niseegeeeee
E 2000
511000
Jenorry prant
81
01
CO0 2
CDC
COPC
Sereenin..0o0 2000
$1.00
05
41 63 25 06
31 63 6/
SHIDN 2 0000
Frost
Acalco 670
CCC
61 03 33 E7
01 65 25
05
31
65
03
67
20130
Tag L0000
C CRC
brightness
C COOC
jpeg compprpeeeee cmmpressccrpieel acteel astiic transformeantrass
63 35 63
33
21
C6
G5
CCCC
33
611
0.5
11
ImageNet test
Figure 13: Results on ImageNet-C corruptions comppins (adaptation hyperparameters of ETA
30

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Both abstract and introduction includes our main observations, analysis on them, followed by experimental results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
we discuss about our paper's limitation that AGL requires ID data during test-time, and we conduct ablation study on limited number of ID data in Section
Justification: In Section 7
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
31

---

Answer: [Yes]
Justification: In Section 4.1 we provide exact notations, setups, theorems and proofs with proper numbers and references.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide the detailed algorithm for online test in ID and OOD in Algorithm which is very easy to follow. Also, the estimation algorithm we employ in other paper already has the public code, but we provide detailed algorithm in Algorithm 2
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
32

---

reproducible, they 5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: Our work mainly focuses on existing test-time adaptation methods as well as publicly available datasets, which are already highly accessible.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/nips. wttpcc/pcccccccccccooooooop httiic//giidees/ddeSubbbiiiii wwwtanssssonnpo. ssinnolicyy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide detailed experimental setups for our observation Section 3.1 as well our experiments in Sections [.11 5.2 and Section
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification: We conducted iterative experiments on measuring mean and covariance simi- larity in Table well as subsampling ID data in Fig. 5
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
33

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide the details about the type of GPU machines for our entire experi- ments in Section A
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: We have reviewed the NeurIPS Code of Ethics, and confirmed that our paper confirms them.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: In Section 7 we discuss the positive socictal impacts of our work, the signifi- cant improvements of reliability of neural networks under distribution shift.
Guidelines:
34

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymentt of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: This work does not include such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We properly cite all the test-time adaptation basclines and datasets used in our paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
35

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
If assets are released, the license, copyright information, and terms use in the datasets.paperssiii paperswi thcode. www.com//aaaa datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
This work does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
This work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
36

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
37


Reviews:
[{"review_id": "VoqtGplPP6", "reviewer": "Reviewer_puv1", "summary": "This paper presents observations that TTA models exhibit strong agreement-on-the-line (AGL) and accuracy-on-the-line (ACL) phenomenon, which persists across a wide range of distribution shifts and models. Leveraging this observation, the authors apply methods to estimate OOD accuracy without labeled data and perform a hyperparameter selection task for a TTA model. The proposed methods are evaluated extensively, demonstrating their effectiveness in TTA settings.", "questions": "Kindly see the weaknesses part.", "limitations": "The authors partially discuss the limitations in the paper.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The identification of the AGL phenomenon in TTA models is interesting and novel. This observation is leveraged effectively to address significant challenges in the field, including performance evaluation and hyperparameter tuning.\n\n2. The authors provide extensive experimental results to support their claims. The evaluation covers various TTA methods and datasets, demonstrating the robustness and generality of the proposed approach.", "weaknesses": "1. The paper lacks a comprehensive comparison with the extensive body of literature on model selection methods under distribution shifts (e.g., i-iv below) that share a same goal of predicting OOD performances and thereby achieving a better model selection performance.\n\n2. While the paper examines the hyperparameter selection task for individual TTA methods, a more interesting and practical task would be to choose the best model among various TTA methods. The suggested metric might have a slope unique to each TTA method, potentially complicating the model selection task among different TTA methods. Addressing this issue would enhance the practical applicability of the proposed approach. \n\n3. The paper is missing a core theoretical explanation of how TTA methods can result in a feature space that satisfies the sufficient condition in Theorem 1. Without this theoretical foundation, I have an impression that this paper could be interpreted as a simple application of Baek et al. and Miller et al.'s findings to the TTA setting. Providing a deeper theoretical insight would strengthen the contribution of the paper. \n\ni. Hu, D., Liang, J., Liew, J. H., Xue, C., Bai, S., & Wang, X. (2024). Mixed Samples as Probes for Unsupervised Model Selection in Domain Adaptation. In NeurIPS.  \n\nii. Musgrave, K., Belongie, S., & Lim, S. N. (2022). Benchmarking validation methods for unsupervised domain adaptation. arXiv preprint arXiv:2208.07360, 2(6), 12. \n\niii. Yang, J., Qian, H., Xu, Y., & Xie, L. (2023). Can we evaluate domain adaptation models without target-domain labels? a metric for unsupervised evaluation of domain adaptation. arXiv preprint arXiv:2305.18712. \n\niv. Saito, K., Kim, D., Teterwak, P., Sclaroff, S., Darrell, T., & Saenko, K. (2021). Tune it the right way: Unsupervised validation of domain adaptation via soft neighborhood density. In ICCV.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer puv1, \n\nWe greatly appreciate your thoughtful reviews on our paper. To address the concerns you raised, we included additional results and clarifications as following:\n+ Comparisons with five existing model selection baselines,\n+ Best TTA methods selection results using our method,\n+ Clarified our paper’s novelty and insight provided by Section 4 analysis.\n\nWe address the individual concern as below:\n\n**1. The paper lacks a comprehensive comparison with the extensive body of literature on model selection methods under distribution shifts.** \n\nTo address the concern we add comparisons in Table 1 of the rebuttal pdf as below: \n+ Hyperparameter selection with MixVal[1], Entropy[2], IM[3], Corr-C[4], and SND[5] on CIFAR10-C over all corruptions, ImageNet-R, and Camelyon17-WILDS.\n\nWe evaluate methods across various hyperparameters including architecture, learning rates, early-stopped checkpoints, batch sizes, and adaptation steps. Our method consistently outperforms or is competitive against other baselines across datasets and hyperparameters, resulting in state-of-the-art performances on average. We noticed that current state-of-the-art UDA model selection methods, i.e., MixVal[1] or IM[3], perform well on CIFAR10-C and ImageNet-R, but they critically fail in Camelyon17-WILDS, e.g., validation error of 7.98\\% in MixVal and 23.52\\% in IM.\n\nIn contrast, our method is consistently reliable across shifts, including those where other baselines fail, e.g., 0.62\\% in Camelyon17-WILDS. The failure modes of existing baselines might come from their assumptions, e.g., low-density separation, that do not generalize to such distribution shifts. Overall, our method of inducing AGL using TTA is most reliable for model selection.\n\n**2. A more interesting and practical task would be to choose the best model among various TTA methods.** \n\nGreat suggestions! Our method can be easily applied to select the best TTA strategy. Our unsupervised hyperparameter selection strategy not only effectively chooses the optimal hyperparameters for each TTA method, but _using AGL-based estimators, can also closely predict the OOD accuracy of each method + hyperparameter choice pair_. These estimates can be used to select the overall best strategy.\n\nIn Fig.1 of the rebuttal pdf, we present a\n+ Comparison of our OOD accuracy estimates in SHOT, BN, TENT, ConjPL, and ETA on CIFAR10-C over all corruptions\n\nFor each TTA method, we plot the true (GT) OOD accuracy on the x-axis, and our estimates on the y-axis. Plotted as “x” marks are TTA strategies paired with the optimal hyperparameters selected using our method. For each TTA strategy, we also report its OOD accuracy averaged over all hyperparameter choices and our average estimates of these accuracies, marked as “o”.\n\nYou can see that our method precisely estimates both the best and the averaged OOD performances of each TTA strategy (i.e., very close to $y=x$ line). Notably, our estimates preserve the ranking of the TTA methods by their OOD accuracy, i.e., almost no reversed order in $y$-axis compared to $x$-axis. This indicates that our methods can be easily adapted for selecting the best TTA models overall.\n\n**3. Without this theoretical foundation, I have an impression that this paper could be interpreted as a simple application of Baek et al. and Miller et al.'s findings to the TTA setting.** \n\nThank you for the feedback! Beyond our methodological contributions, we identify an interesting behavior where TTA methods reduce distribution shift in each class distribution to just a scale shift in the representation space. This happens to be the theoretically sufficient condition for ACL as described in Miller et al [6] and our Theorem 1. We take this even further in Table 1 of our original submission, where we show that if we measure the scale shift in the representation space and compute the theoretical slope, it closely matches the actual empirical slope of the ACL trend. We are the first to characterize this behavior of TTA methods and this is a novel contribution, on its own, unexplored by Miller et al. [6] or Baek et al. [7]. \n\nWe provide rough intuition for why this may happen in BN_Adapt. In BN_Adapt, the data is standardized in each BN layer using the OOD 1st and 2nd moments calculated at test time instead of the ID statistics saved during training. Imagine that the final-layer features are the output of a BN layer. Before TTA, due to frozen BN stats, the ID features are roughly standardized to mean 0 and unit variance, while the OOD features have shifted mean and covariance. Then after TTA, the test-time BN stats also standardizes the OOD features closer to mean 0 and unit variance. However, while BN_Adapt standardizes the _overall_ OOD distribution, BN may not be able to deal with any scale shifts within each _class-conditioned_ distribution. We can follow-up during the discussion period with a more precisely constructed argument if interested. \n\nWe hope these responses address the concerns, and let us know if there is any further feedback.\n\n\n>**References** \\\n> [1] Hu et al., Mixed Samples as Probes for Unsupervised Model Selection in Domain Adaptation, NeurIPS 2023 \\\n[2] Morerio et al., Minimal-entropy correlation alignment for unsupervised deep domain adaptation, ICLR 2018 \\\n[3] Musgrave et al., Benchmarking validation methods for unsupervised domain adaptation, arXiv 2022 \\\n[4] Tu et al., Assessing model out-of-distribution generalization with softmax prediction probability baselines and a correlation method, 2023 \\\n[5] Saito et al., Tune it the right way: Unsupervised validation of domain adaptation via soft neighborhood density,  ICCV 2021 \\\n[6] Miller et al., Accuracy on the Line: On the Strong Correlation between Out-of-Distribution and In-Distribution Generalization, ICML 2021 \\\n[7] Baek et al., Agreement-on-the-Line: Predicting the Performance of Neural Networks under Distribution Shift, NeurIPS 2022", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HI82s2coak", "reviewer": "Reviewer_JQG9", "summary": "The paper presents a study on how test-time adaptation influences accuracy-on-the-line (ACL) and agreement-on-the-line (AGL). The authors empirically find that TTA methods significantly enhance the ACL and AGL, enabling better OOD performance estimation. Extensive experiments support their findings.", "questions": "1. Visualizing the change of correlation coefficient as the model adapts to the test data flow could be more beneficial for understanding the contribution of the paper.", "limitations": "N/A", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. This paper investigates an under-explored but important observation of TTA.  TTA can boost AGL and ACL phenomenon on out-of-distribution datasets.\n2. The observed phenomenon is verified under diverse settings, including learning rates, the number of adapting steps, batch sizes, and the early-stopped epoch for pretraining, etc.", "weaknesses": "1. The scope should be revised. TTA encompasses more than just feature extractor adaptation, but also classifier adaptation such as T3A [1]. Additionally, it appears that the author does not employ TTA methods with a memory bank [1-4]. Therefore, the paper should clearly specify which types of TTA methods support the observations.\n2. The experimental setting is idealized. It seems that the paper only focuses that how TTA always improves accuracy. However, TTA might fail under continuously changing distributions, especially in cases of small batch sizes (e.g.,1), or significant distribution shifts. The author should provide a discussion of these scenarios.\n3. The analysis in section 4.2 is not supervising, since BN_Adapt and TENT only modify the $\\gamma$ and $\\beta$ in BN layers to make the output features similar to the target domain. It would be better to discuss whether other non-BN methods align with these results.\n4. One key advantage of TTA is that it does not require source domain data. However, the AGL-based method requires it, which limits the contribution of this paper.\n\n[1] Iwasawa Y, Matsuo Y. Test-time classifier adjustment module for model-agnostic domain generalization. In NeurIPS. 2021.\n\n[2] Jang M, Chung S Y, Chung H W. Test-time adaptation via self-training with nearest neighbor information. In ICLR. 2023.\n\n[3] Yuan L, Xie B, Li S. Robust test-time adaptation in dynamic scenarios. In CVPR. 2023.\n\n[4] Wang S, Zhang D, Yan Z, et al. Feature alignment and uniformity for test time adaptation. IN CVPR. 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer JQG9, \n\nWe greatly appreciate your thoughtful reviews on our paper. To address the concerns you raised, we included additional results and clarifications as following:\n+ Added T3A as non-BN method and its AGL visualizations and feature alignment analysis,\n+ Clarified our paper’s results on when TTA does not improve accuracy,\n+ Clarified our method’s robustness under ID-data limited setup,\n+ Added visualizations of coefficient correlations during the progress of TTA.\n\nDetailed responses for each individual concern are as below:\n\n**1, 3. TTA encompasses more than just feature extractor adaptation, but also classifier adaptation such as T3A. Also, it would be better to discuss whether other non-BN methods align with these results.** \n\nWe appreciate the Reviewer's feedback! To address the concern, in rebuttal Figure 2  and Table 2, we include\n+ T3A[1]’s AGL visualizations as well as feature alignment analysis on CIFAR10-C Gaussian Noise. \n\nSince T3A[1] is applied to non-BN networks, we train the networks without BN layers and apply T3A. In Fig.2, we similarly observe that, using T3A (left) to adapt the last classifier only results in stronger ACL compared to models before adaptation (labeled as “Vanilla w/o BN”) (right). Specifically, the correlation coefficient of ACL increases from 0.29 to 0.80. Still, T3A’s correlation coefficients in both accuracy and agreement are relatively weak compared to BN-based methods (e.g., 0.80 and 0.46 in T3A vs. 1.0 and 1.0 in TENT). We also make note that T3A’s accuracy and agreement lines have different biases.\n\nHowever, while T3A does improve ACL trends, it does not seem to satisfy the same sufficient theoretical condition of ACL that BN-adaptation methods satisfy. As T3A only modifies the final classifier, the distribution shift from ID to OOD in the penultimate representation space remains more complicated than a simple scale shift. In Table 2 we reported the cosine similarity between the ID and OOD mean and covariance of Vanilla w/o BN, which is the same as that of T3A, since no updates in feature extractor. The results in Table 2 show that their cosine similarity are approximately 0.78 (averaged over different architectures), with standard deviation exceeding 0.1. The cosine similarities are _much lower compared to those of BN-based TTA methods_ (e.g, TENT has 0.990$\\pm$0.005 and 0.974$\\pm$0.011 for mean and covariance).\n\nOverall, our method may also apply to non-BN adaptation methods such as T3A, but the theory in our work does not explain why stronger ACL trends appear in this setting. This is an important open question we hope to understand further in the future. Due to the limited timeline of rebuttal, we could not examine other baselines mentioned by the Reviewer.\n\n**2. The paper only focuses that how TTA always improves accuracy, but TTA might fail under continuously changing distributions, especially in cases of small batch sizes (e.g.,1), or significant distribution shifts.**\n\nWe would like to clarify that our paper already explores scenarios where the TTA degrades OOD performance, including:\n\n * Poor hyperparameter choices that lead to low ID and OOD accuracy, such as extremely small batch sizes (e.g., 1 on CIFAR10-C). However, these models still  conform to the same linear ACL/AGL trends (Figs. 2 and 7 of original submission). In fact, our theoretical explanation generalizes to such settings. In Table 2 of original submission, we show that the features after TENT with batch size of 1 also demonstrate near perfect alignment (shown by near-zero standard deviation).\n\n* Real-world distribution shifts, e.g., CIFAR10.1, ImageNet-V2, FMoW-WILDS, where TTA hurts performances, but maintains linear trends (Fig. 3  of original submission).\n\nImportantly, _AGL is restored regardless of whether TTA succeeds_ for any one hyperparameter choice $-$ this is precisely why we can employ our method for hyperparameter optimization.\n\n**4. One key advantage of TTA is that it does not require source domain data. However, the AGL-based method requires it, which limits the contribution of this paper.** \n\nOur method works well at estimating OOD accuracy given just a small number of ID data (i.e., 1-5% of total), as demonstrated in Section D of our submission. In fact, even with limited data, our methods outperform other baselines, e.g., ATC and DoC-feat, which also require the access to ID data for temperature scaling.\n\nThe fact of the matter is even recent TTA methods are highly sensitive to hyperparameters and their OOD is often unpredictable. Zhao et al.[2] recently pointed out this issue. Using a reasonably small amount of ID data, which is relatively easy-to-obtain than labeled OOD data, our method can greatly enhance the reliability of TTA methods in their practical usage.\n\n**5. Visualizing the change of correlation coefficient as the model adapts to the test data flow could be more beneficial for understanding the contribution of the paper.**\n\nGreat suggestions! In Figure 3 of our rebuttal PDF we’ve added a\n* Visualization of BN_Adapt, TENT, and ETA's correlation coefficient ($R^2$) of ACL as TTA progresses on CIFAR10-C Gaussian Noise and CIFAR100-C Glass Blur\n\nFor comparison, we also added the correlation coefficient of the Vanilla model at iteration=0.\n\nWe observed that the strong correlation rapidly appears (e.g., 0.4 to 1.0 in TENT on CIFAR10-C Gaussian Noise) at the very beginning of adaptation (i.e., iteration=1) and remains high until the end of adaptation. Our finding shows that strong AGL induced by TTA appears in the very early stage of adaptation, and remains strong over time.\n\n\\\nWe hope these responses address the concerns, and let us know if there is any further feedback.\n\n\n>**References** \\\n> [1] Iwasawa et al., Test-time classifier adjustment module for model-agnostic domain generalization. In NeurIPS 2021 \\\n[2] Zhao et al., On pitfalls of test-time adaptation, ICML 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nQeWFVeMiM", "reviewer": "Reviewer_ZucK", "summary": "This papers shows that using test-time-adaptation  improve the accuracy of the algorithms used for OOD performance estimation algorithms based on agreement-on-the-line and accuracy-on-the-line. This is shown through a series of experiments. Some justification has also been provided based on prior work,.", "questions": "See weaknesses.", "limitations": "The authors should discuss the limitations I mentioned in the weaknesses section.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "- OOD performance estimation using unlabeled data is a very important problem and methods based on agreement-on-the-line and accuracy-on-the-line are the state-of-art methods.\n\n- The experimental details are very well presented and the paper is very well written and clear.", "weaknesses": "- Intuitively, if we assume that TTA is *perfect*, then we expect ID and OOD accuracy to be the same (i.e., y = x); the model has been fully adapted to the OOD setting and there is no difference between ID and OOD. Also the ID and OOD agreements should intuitively become the same (i.e., y = x). Is this understanding correct? It seems that there is a very basic reason behind the main observation of the paper. ‌Because of this, I think table Table 2 does not show a fair comparison; TTA is making models more similar and shift all the lines towards the y = x line. Is the message of the paper as simple as this? I think this answers the question \"What specific mechanisms within these adaptation methods cause the observed phenomena? \" asked in the conclusion section.\n\n\n- In a practice, we use OOD performance estimation with unlabeled data in settings where compute, budget, etc. is limited; thus we cannot collect new labeled data. Given this, is a method that is computationally expensive (that involves test time adaptation) feasible? Also, note that this method does not work if we only have black box access to a model (which is again, typical in potential applications of AGL and ACL).\n\n\n- What is the condition of theorem 1? It is written that \"... correlation if with a bias of zero and a slope ...\". Also, the condition in Eq. (2) is very strong. See my comments below for other theoretical results on AGL and ACL that study the problem in other (I believe more realistic) settings where the covariance and mean can shift. \n\n\n- The literature review is missing many theoretical results on agreement-on-the-line and accuracy-on-the-line. In the past 1-2 years, there has been a lot of theory results on these topics that show that these phenomenon are features of the high-dimensionality.\n\nFor example:\n\n[1] N Tripuraneni, B Adlam, J Pennington, Overparameterization improves robustness to covariate shift in high dimensions, NeurIPS 2021.\n\n[2] D Lee, B Moniri, X Huang, E Dobriban, H Hassani, Demystifying Disagreement-on-the-Line in High Dimensions, ICML 2023.\n\n[3] D LeJeune, J Liu, R Heckel, Monotonic Risk Relationships under Distribution Shifts for Regularized Risk Minimization, JMLR, 2024.\n\n[4] A Sanyal, Y Hu, Y Yu, Y Ma, Y Wang, B Schölkopf, Accuracy on the wrong line: On the pitfalls of noisy data for out-of-distribution generalisation, Arxiv 2024.\n\n- The papers above show that these phenomena are fragile; these papers show clear settings where Accuracy-on-the-line and agreement-on-the-line break.\n\n- More important than the previous comment: Lee et al (2023) show that the line for agreement and the line for accuracy can have different intercepts (at least in their toy model). Although the liens have the same slope. This is seen when looking back at the plots of the original real world experiments in Baek et al. This issue needs to be addressed before any real application of AGL for practice. There might be some fundamental issue with the AGL method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer ZucK, \n\nWe greatly appreciate your thoughtful reviews on our paper. To address the concerns you raised, we made clarifications on:\n+ Intuition of linear trend observations, \n+ Our study’s contribution under cost-limited scenarios,\n+ Condition 1 of the Eq. 2 of our original manuscript,\n+ Intuition of condition for ACL guarantee and potential failure of AGL\n\nDetailed responses for each individual concern are as below:\n\n**1. TTA is making models more similar and shift all the lines towards the y = x line. Is the message of the paper as simple as this?**\n\nWe would like to clarify that these two phenomena are independent. A substantial gap between ID and OOD accuracies can remain after TTA, but models’ performances may still be strongly correlated. For example, after applying TENT on ImageNet-C Shot Noise, there is still approximately a 40\\% gap in ID and OOD accuracy, but strong AGL holds (see submission Fig.1). In other words, strong AGL is not simply explained by the decreased gap between ID and OOD by TTA. In Section 4, we build a theoretical intuition of why TTA induces these strong ACL/AGL trends.\n\n**2. We use OOD performance estimation with unlabeled data in settings where compute, budget, etc. is limited; thus we cannot collect new labeled data. Given this, is a method that is computationally expensive feasible? Also, this method does not work if we only have black box access to a model.** \n\nGreat question! TTA is generally very computationally cheap, and arguably much less resource-intensive than collecting labels, which usually requires expensive manual labor. Oftentimes only a very small number of parameters are adapted, making TTA far cheaper than methods like test-time training[5,6] or unsupervised domain adaptation[1-3]] that require a separate pre-training procedure and heavy training. With a small amount of resources, our paper shows that TTA can significantly restore AGL, allowing TTA models to be safely deployed across various distribution shifts. While our method is not feasible with black box access, non-black box settings cover a wide variety of practical scenarios where our method is useful. Moreover, our method often significantly outperforms baselines that do work on black box models with estimation error smaller by a whole factor. \n\n**3. What is the condition of theorem 1?**\n\nWe apologize for the confusion. The conditions for Theorem 1 are detailed in Eq. 2. In particular, the class-conditioned distributions must be normally distributed and the distribution shift must only be a simple scale shift, i.e., the mean and covariance can change in magnitude but not their directions. Also, the classifiers $f_{\\theta}$ must be linear with no bias. Under these conditions, ID versus OOD accuracy is perfectly linear correlated with a slope of $\\alpha / \\gamma$.\n\n**3. The condition in Eq. (2) is very strong. Other theoretical results on AGL and ACL that study the problem in other settings where the covariance and mean can shift.**\n\nThank you for the feedback! We emphasize that, while this assumption seems strong, our empirical study in Section 4 demonstrates that the “scale-shift” condition in Theorem 1 is almost perfectly satisfied via TTA. Indeed, this is precisely why our method induces strong linear trends in practice. Furthermore, the theoretical studies [7,8] that do allow the covariance shape to shift are asymptotic results that rely heavily on the dimensionality of the data and classifiers to go to infinity. They explicitly note that their theory does not guarantee linear trends in finite dimensions (e.g., Figure 2 of [8]). Our paper also include covariance shape shifts such as CIFAR10 vs. CIFAR10-C Gaussian Noise, where classifiers do not exhibit linear trends in finite dimensions.\n\n**4. The literature review is missing many theoretical results on AGL and ACL. The papers above show that these phenomena are fragile. This issue needs to be addressed before any real application of AGL for practice.**\n\nThank you again for the feedback! We will make sure to include these works in our literature review. We do note that failure modes observed theoretically may not be consistent with common real-world data. For example, Lee et al.[9] show that in random feature linear regression, AGL does not hold in CIFAR10-C. However, in neural networks trained with logistic regression, both our paper and Baek et al. show strong AGL on CIFAR10-C. On the other hand, Sanyal et al.[10] points to AGL breaking under heavy label noise which we don’t observe in many practical settings. Generally, we agree that AGL could be fragile. In fact, our paper directly addresses this problem, expanding, via TTA, the set of distribution shifts where AGL holds. We observe repeatedly across a broad range of distribution shifts that TTA improves ACL and AGL trends, and in all these scenarios the shifts reduce to “scale shifts”  thus satisfying the theoretically sufficient condition for ACL.\n\\\nWe hope these responses address the concerns, and let us know if there is any further feedback.\n\n>**References** \\\n> [1] Ganin et al., Domain Adversarial Training of Neural Networks, JMLR 2016 \\\n[2] Sun and Saenko, Deep CORAL: Correlation alignment for deep domain adaptation, ECCV 2016 \\\n[3] Li et al., Domain generalization with adversarial feature learning. CVPR 2018 \\\n[5] Sun et al., Test-Time Training with Self-Supervision for Generalization under Distribution Shifts, ICML 2020 \\\n[6] Gandelsman et al., Test-Time Training with Masked Autoencoders, NeurIPS 2022 \\\n[7] Tripuraneni et al., Overparameterization improves robustness to covariate shift in high dimensions, NeurIPS 2021 \\\n[8] LeJeune et al., Monotonic Risk Relationships under Distribution Shifts for Regularized Risk Minimization, JMLR 2024 \\\n[9] Lee et al., Demystifying Disagreement-on-the-Line in High Dimensions, ICML 2023 \\\n[10] Sanyal et al., Accuracy on the wrong line: On the pitfalls of noisy data for out-of-distribution generalisation, Arxiv 2024", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BDSE6A1eeH", "reviewer": "Reviewer_1nU3", "summary": "This paper try to achieve reliable TTA by tackling three bottlenecks, including performance evaluation without labeled data, some distribution shifts, and hyperparameter selection for TTA methods.", "questions": "Overall, the work is an important study of TTA, however there are still some questions.\n\n1.\tAbout Algorithm 1: In TTA, does updating the model parameters require multiple iteration steps? If it does, according to the basic idea of online algorithms, Algorithm 1 does not seem to reflect the model parameters being trained with different batches of data at different time steps. Could you provide a detailed explanation of the difference between the online algorithm you provided and the traditional offline algorithm?\n\n2.\tWhy the inverse of the cumulative density function of the standard Gaussian distribution is applied on the axes of accuracy and agreement? Is there any intuitions and theoretical explanations?", "limitations": "No potential negative societal impact.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tTTA is an important topic, and this work is a relevant and timely contribution.\n\n2.\tThe experiments are extensive, covering various tasks comprehensively and convincingly.\n\n3.\tThe presentation is clear and the motivation is well-presented.", "weaknesses": "1.\tThe novelty is relatively limited; it is an application of Baek et al. (2022) to the TTA setup. And the limited theoretical conclusion is from Miller et al. (2019).\n\n2.\tIt might be necessary to evaluate on more complex tasks, such as object detection and semantic segmentation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 1nU3, \n\nWe greatly appreciate your thoughtful reviews on our paper.\nTo address the concerns you raised, we made changes and clarifications as following:\n+ Clarified our paper’s novelty,\n+ Clarified notations on Algorithm 1,\n+ Added theoretical intuition of using inverse of cumulative density function of standard Gaussian distribution.\n\nFor each concern, we attempted to address it with individual responses:\n\n**1. The novelty is relatively limited; it is an application of Baek et al.[2] to the TTA setup. And the limited theoretical conclusion is from Miller et al.[1].**\n\nWe would like to note that our paper not just applies AGL to TTA setup, but first finds the significant restoration of AGL in various distribution shifts, including those where the linear trends did not hold previously. Previous studies[1-4], including Miller et al.[1] and Baek et al.[2], did not include such restorations, limiting the broader applicability of AGL.\n\nIn addition, while Miller et al.[1] only considered the very simple Gaussian setup for the sufficient condition for Theorem 1, our paper presents that TTA involving more complex data with non-linear deep networks satisfy the condition.\n\n**2. It might be necessary to evaluate on more complex tasks, such as object detection and semantic segmentation.**\n\nWe appreciate the Reviewer’s suggestion. Since it remains ambiguous to define the agreement between models’ outputs in more than a single label, e.g., object detection and semantic segmentation, our paper mainly focuses on image classification, following Baek et al.[2]. However, extension to such tasks could further extend the understanding of the potential of TTA in stronger linear trends. We will discuss this in our final manuscript.\n\n**3. Algorithm 1 does not seem to reflect the model parameters being trained with different batches of data at different time steps. Could you provide a detailed explanation of the difference between the online algorithm you provided and the traditional offline algorithm?**\n \nWe apologize for the confusion regarding Algorithm 1. As mentioned by the reviewer, TTA updates the model parameters in multiple iterations, i.e., at each iteration using the different batch of unlabeled OOD data, as described in L6 of Algorithm 1. We use the notation of x_OOD that means the current batch of OOD test data, which is different at every iteration. We will clarify this by utilizing the different notations for data in different time steps in the final version.  \n\nThe online algorithm in Algorithm 1 describes the online _test_, which is different from traditional testing. Traditional testing tests the model that is fixed in its pre-trained parameters, while the online \"test\" is testing a model that is continuously updated its parameters during TTA. In other words, the model tested at iteration of $t-1$ is different from that at iteration $t$. Specifically in Algorithm 1, L6 makes the only difference between traditional and online testing, where L6 updates the model parameters every iteration before testing in L7 and L8.\n\n\n**4. Why the inverse of the cumulative density function of the standard Gaussian distribution is applied on the axes of accuracy and agreement? Is there any intuitions and theoretical explanations?** \n\nGreat question! Let us revisit the simple Gaussian distribution setup as described in Eq.2 of our original manuscript, where $Q$ distribution shifts from $P$ only in scale of mean direction and covariance shape: \\\n$P(x\\mid y)=\\mathcal{N}(y\\cdot \\mu; \\Sigma)$, $Q(x\\mid y)=\\mathcal{N}(y\\cdot \\alpha\\mu;\\gamma^2\\Sigma)$, \\\nwhere the label $y \\in \\\\{-1,1\\\\}$ and $\\alpha,\\gamma$ are constant scalars.\n\nWhen we consider a linear classifier $f_{\\theta}:x \\mapsto \\theta^{\\top}x$, its accuracy on distribution $P$ and $Q$ are defined as \\\n$\\text{acc}_P(\\theta)=\\Phi(\\frac{\\theta^{\\top} \\mu}{\\sqrt{\\theta^{\\top} \\Sigma \\theta}})$ ,\\\n$\\text{acc}_Q(\\theta)=\\Phi(\\frac{\\theta^{\\top} (\\alpha\\mu)}{\\sqrt{\\theta^{\\top} \\gamma^2\\Sigma \\theta}})$, \\\nwhere $\\Phi$ is the cumulative density function (CDF) of standard Gaussian distribution.\n\nAfter applying the inverse of $\\Phi$ for both accuracies, the linearity between two is calculated by\\\n$\\frac{\\Phi^{-1}(\\text{acc}_Q(\\theta))}{\\Phi^{-1}(\\text{acc}_P(\\theta))} =  \\frac{\\theta^{\\top} (\\alpha\\mu)}{\\sqrt{\\theta^{\\top} \\gamma^2\\Sigma \\theta}} / \\frac{\\theta^{\\top} \\mu}{\\sqrt{\\theta^{\\top} \\Sigma \\theta}} =\\alpha / \\gamma$, \\\nwhich is a constant independent of $\\theta$. This means the linear relationship between $\\Phi^{-1}(\\text{acc}_P(\\theta))$ and $\\Phi^{-1}(\\text{acc}_Q(\\theta))$ holds across linear classifiers.\nTherefore, in theory with simple Gaussian setup, such inverse of CDF is required to exactly derive the linear relationship between ID and OOD. \n\nThe application of the inverse of the CDF surprisingly improves linear fit beyond simple Gaussian setups, even in more complex real-world distribution shifts, as extensively observed in Miller et al.[1], Baek et al.[2], and our paper.\n\n\\\nWe hope these responses address the concerns, and let us know if there is any further feedback.\n\n>**References** \\\n> [1] Miller et al., Accuracy on the Line: On the Strong Correlation between Out-of-Distribution and In-Distribution Generalization, ICML 2021 \\\n[2] Baek et al., Agreement-on-the-Line: Predicting the Performance of Neural Networks under Distribution Shift, NeurIPS 2022 \\\n[3] Wenzel et al., Assaying Out-Of-Distribution Generalization in Transfer Learning, NeurIPS 2022 \\\n[4] Teney et al., ID and OOD Performance are Sometimes Inversely Correlated on Real-world Datasets, NeurIPS 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We greatly appreciate all four reviewers' valuable feedback and thoughtful suggestions.

The reviewers highlighted the following strengths of our paper:

* Our paper investigates the under-explored but novel and important observation of TTA inducing AGL phenomenon (Reviewer puv1, Reviewer JQG9).
* Our paper addresses significant and important challenges, such as OOD performance estimation and hyperparameter tuning, and demonstrates its effectiveness (Reviewer puv1, Reviewer ZucK, Reviewer 1nU3).
* Our paper includes extensive experimental results to support our claims (Reviewer puv1, Reviewer JQG9, Reviewer 1nU3).

Overall summary of additional experiments, clarifications, and discussions are as below:
* We added clarifications on our paper's novelty compared to existing TTA studies and Miller et al.[1] and Baek et al.[2] (Reviewer puv1, Reviewer 1nU3)
* We additionally tested our method for comparison with other model selection baselines and best TTA methods selection application (Reviewer puv1)
* We discussed our theoretical insight on how TTA achieves features that satisfy the theoretical guarantee of ACL (Reviewer puv1)
* We included additional analysis on how TTA induces strong AGL, by adding T3A as new TTA baseline and tracking correlation coefficient during TTA. (Reviewer JQG9)
* We added clarifications on our observations on circumstances including those TTA fails or access to ID data is limited (Reviewer JQG9)
* We added clarifications on why we observe stronger AGL in TTA is interesting, and TTA's feasibility under compute-limited scenario for better OOD estimation (Reviewer ZucK)
* We discussed on our paper compared to existing theoretical studies on ACL and AGL (Reviewer ZucK)
* We added clarification on lack of details, including condition for theoretical setup in Theorem 1, concept of online test in Algorithm 1, and theoretical insight in using inverse of CDF of standard Gaussian distribution for better linear fit (Reviewer ZucK, Reviewer 1nU3)

To address the concerns and suggestions raised by the reviewers, we uploaded additional pdf that includes figures and tables.

Meta-review: This paper empirically observes that test-time adaptation methods strengthen the "accuracy-on-the-line" and "agreement-on-the-line" phenomena. All reviewers were in agreement that the paper is worth publishing. However, they also raised some concerns whose resolutions  should be in in the final version (e.g., adding the experiments on model selection).
Decision: Accept (poster)