Test-Time Adaptation Induces Stronger Accuracy and Agreement-on-the-Line

Authors: eungyeup  kim
mingjie  sun
christina  baek
aditi  raghunathan
j  zico kolter
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 02:30:38

Abstract: Recently, Miller et al. (2021) and Baek et al. (2022) empirically demonstrated strong linear correlations between in-distribution (ID) versus out-of-distribution (OOD) accuracy and agreement. These trends, coined accuracy-on-the-line (ACL) and agreement-on-the-line (AGL), enables OOD model selection and performance estimation without labeled data. However, these phenomena also break for certain shifts, such as CIFAR10-C Gaussian Noise, posing a critical bottleneck. In this paper, we make a key finding that recent test-time adaptation (TTA) methods not only improve OOD performance, but drastically strengthens the ACL and AGL trends in models, even in shifts where models showed very weak correlations before. To analyze this, we revisit the theoretical conditions established by Miller et al. (2021), which demonstrate that ACL appears if the distributions only shift in mean and covariance scale in Gaussian data. We find that these theoretical conditions hold when deep networks are adapted to OOD, e.g., CIFAR10-C --- models embed the initial data distribution, with complex shifts, into those only with a singular ``scaling'' variable in the feature space. Building on these stronger linear trends, we demonstrate that combining TTA and AGL-based methods can predict the OOD performance with high precision for a broader set of distribution shifts. Furthermore, we can leverage ACL and AGL to perform hyperparameter search and select the best adaptation strategy without any OOD labeled data.

Main paper: 
 introduction: 
 Neural networks often fail to generalize to out-of-distribution (OOD) data that differs from the indistribution (ID) data seen at train-time [1, 11, 41] . Thus, characterizing the behaviors of these models under distribution shift becomes crucial for reliable deployment. However, it is often extremely challenging to reliably estimate their performances because in many practical applications, OOD labeled data is scarce. Interestingly, recent studies [32, 3] have found a set of simple empirical laws that describe the behavior of models across many distribution shift benchmarks. In particular, across numerous distribution shift benchmarks, the models' ID versus OOD accuracies, under probit scaling, tend to observe a strong linear correlation across numerous distribution shift benchmarks. Additionally, when accuracy is strongly correlated, the ID and OOD agreement rates between pairs of these models are also strongly correlated with nearly identical slopes and biases. These phenomena, respectively referred to as "accuracy-on-the-line" (ACL) [32] and "agreement-on-the-line" (AGL) [3] , can be leveraged for precise OOD accuracy estimation without access to OOD labels: one could estimate the slope and bias of the ID vs OOD accuracy trend using agreement rates, then linearly transform ID accuracy using this approximate linear fit. 1 .Figure 1 : Linear trends in both accuracy and agreement hold to a substantially stronger degree after applying adaptation methods than before. Each blue and pink dot denotes the accuracy and agreement, followed by the linear fits for each, and R 2 is correlation coefficient.However, studies [32, 3, 56, 48] have demonstrated that for distribution shifts benchmarks where the linear trends breaks down catastrophically. As shown in Fig. 1 , such as CIFAR10-C Gaussian Noise [14] or Camelyon17-WILDS [41] , trained models (i.e., Vanilla) with around 92 -95% ID accuracy can have OOD accuracies that vary between 10 -50%, so ID accuracy alone becomes extremely unreliable for understanding the OOD performance of models. Similarly, the correlation strength of ID and OOD agreement rates also weakens. Ideally, we would like to intervene in models in a way that improves these linear correlations, such that we can reliably predict their performance under distribution shift. While theoretical works [32, 31, 51, 25] provide insights for when these linear trends hold, there is little study on how to strengthen these trends in models.In this paper, we empirically demonstrate that recent OOD test-time adaptation (TTA) strategies [44, 27, 46, 53, 9, 35, 55, 36] not only improve OOD performance, but significantly restore the strong linear trends for a broad range of distribution shifts where ACL and AGL are not initially observable. For instance, in Fig. 1 , the correlation coefficient (R 2 ) of ID versus OOD agreement and accuracy of Vanilla models is 0.18 and 0.39, respectively, and both of these values improve to 1.0 after applying TENT [53] . We observe such stronger linear trends after TTA throughout our extensive testbed consisting of 9 shifts, 7 adaptation methods, and over 40 network architectures. As test-time adapted models have OOD accuracies that degrade more predictably with respect to the ID accuracy, we are also able to utilize AGL based method ALine [3] to obtain precise OOD performance estimates without any OOD labels. Note that no OOD labels were utilized throughout this procedure, neither during TTA or ALine. Our estimates of the models' OOD performances are drastically more precise after adaptation, e.g., estimation error of 11.99% in Vanilla vs. 2.34% after applying TENT on CIFAR10-C Gaussian Noise.Given that TTAs are designed to enhance OOD accuracy, the observation of such strong linear trends is unexpected and non-trivial. While some studies [29, 9] have explored how adaptations lead to improved OOD generalization, these efforts are orthogonal to those investigating the conditions under which linear trends occur [32, 31] . To our knowledge, no studies have attempted to bridge these two areas of research. This naturally raises the question: Why does adapting models at test-time to OOD data lead to stronger linear trends?To answer this, we revisit the theoretical analysis in Miller et al. [32] of the sufficient conditions for observing highly correlated ID vs OOD accuracy in linear classifiers and Gaussian data. Theoretically, ACL holds exactly under distribution shifts where the direction of the class means and shape of the class covariances are fixed, and only the scale of the mean or covariance changes. Surprisingly, we find that TTA, in practice, seems to enforce exactly this condition: after applying TTA, the cosine similarity between the means and covariances of the penultimate-layer feature embedding of ID and OOD data tend to hover around 1, while their magnitude may change by some scaling constant. This implies that adaptations effectively collapse the complexity of the distribution shift to a singular "scaling" variable in the feature space. In addition to (empirically) justifying the use of TTA for strengthening ACL 2 , this discovery casts some insight into the nature of TTA in general, which has previously been a largely heuristic approach.Furthermore, models adapted with different TTA hyperparameters tend to lie on the same linear trend. Fig. 2 illustrates the strong correlation among models first trained on ID data over different architectures and different early-stopping thresholds, then adapted with varying learning rates, batch sizes, adaptation steps. This critically allows us to tune the TTA hyperparameters and choice of TTA method without a held-out OOD labeled set, which is a long-standing challenging unsolved by existing TTA studies [21, 60, 9] . Using ACL and AGL, we are able to select models with accuracy less than 1% away from that of the best model OOD. This also allows to select the best TTA methods by comparing their estimated best OOD performances.To summarize our contributions:• We observe after TTA leads, ACL and AGL hold across a wider set of distribution shifts and hyperparameter settings. • We explain our observation by showing TTA collapses the distribution shift to just a constant scaling of the mean and covariance matrices in the feature space. This satisfies the theoretical conditions studied previously for observing strong linear trends. • Our findings provide a simple and effective strategy for finding the best TTA hyperparameters and the best TTA strategy without any OOD labels. 
related work: 
 Understanding accuracy and agreement-on-the-line. Miller et al. [32] and Baek et al. [3] empirically observed a coupled phenomena in deep models evaluated on a wide variety of standard distribution shift benchmarks: the ID vs. OOD accuracy and agreement are often strongly correlated and the linear fits match almost exactly. Recent studies [32, 31, 51, 25, 24] attempt to build specific characterization of the shifts that lead to (or break) this phenomena. Miller et al. [32] theoretically explained that under a simple Gaussian data setup, ACL does not hold perfectly under distribution shifts that change the direction of the mean or transforms the covariance matrix. For example, they demonstrate that adding isotropic Gaussian noise to CIFAR10, which does not have an isotropic covariance matrix, causes the linear trend to break. Theoretical works have also tried to characterize when ACL holds more broadly. Mania and Sra [31] provided sufficient conditions directly over the outputs of trained models, in terms of their prediction similarity and distributional closeness. Tripuraneni et al. [51] and LeJeune et al. [25] show that ACL holds asymptotically under certain transformations to the covariance matrix. On the other hand, there has been comparatively little theoretical analysis of AGL and why it appears together with ACL. Lee et al. [24] show that in random feature linear regression, AGL can break break partially, i.e., the slope of the agreement trend matches accuracy's, but the biases may be different. While further theoretical conditions are necessary to guarantee when these phenomena hold jointly, in our empirical findings, we see that the slopes and biases do always match across the wide variety of distribution shifts we test.Extending upon such studies that analyze when ACL and AGL may hold (or break) naturally, we demonstrate that a simple model intervention by test-time adaption allows these ID versus OOD trends to hold even stronger for a more expansive set of distribution shifts. In fact, we demonstrate that after adaptation, models actually satisfy the theoretical conditions necessary for ACL as described in Miller et al. [32] . (c) TENT tested on Camelyon17 vs. Camelyon17-OOD Figure 2 : Strong AGL by adaptations with varying hyperparameters, including learning rates, adaptation steps, batch sizes, and (early-stopped) checkpoints of the ID-trained model. Each blue and pink dot denotes the accuracy and agreement, followed by the linear fits for each.Adaptations under distribution shifts and their pitfalls of reliability. Test-time adaptation aims to enhance model robustness by adapting models to unlabeled OOD test data. Test-time training methods [46, 29, 7] involve learning shift-invariant features via solving self-supervision tasks. Other approaches include computing statistics in Batch Normalization (BN) layers using OOD data [20, 44] , instead of using ID statistics stored during training. Subsequent studies further adapt BN parameters by updating them using entropy minimization [53, 35, 36] . Another popular approach is self-training with pseudo-labels [40, 54, 9] .One critical challenge in TTA is that, without OOD labels, it is prohibitively difficult to evaluate how effective the adaptation methods might be. As pointed out in previous studies, adaptation methods may not succeed to address the full spectrum of distribution shifts, such as datasets reproductions [29, 60] , domain generalization benchmarks [21, 61] , and WILDS [40] . Furthermore, these approaches are known to be critically sensitive to different hyperparameter choices [4, 61, 36, 23] . Practitioners must take a great care in optimizing such hyperparameters, but their tuning procedures lack clarity. They often follow the settings of the previous studies [35, 36] , or rely on some held-out labeled data [21, 60, 9] that is unavailable in practice. There exists a line of studies on unsupervised model validation [33, 42, 34, 52, 17] . Our study leverages agreement-on-the-line phenomena within TTA to offer a promising solution for these reliability issues.3 Adaptations lead to stronger Agreement-on-the-Line 
experimental setup: 
 Datasets and models. Our testbed includes diverse shifts, including common corruptions (15 failure shifts in CIFAR10-C, CIFAR100-C, and ImageNet-C [14] ), dataset reproductions (CIFAR10.1 [38] , ImageNetV2 [39] ), and real-world shifts (ImageNet-R [16] , Camelyon17-WILDS, iWildCAM-WILDS, FMoW-WILDS [41] ). Among them, CIFAR10-C Gaussian Noise, Camelyon17-WILDS, and iWildCAM-WILDS display the weakest correlations in accuracy and agreement [32, 3, 56] . We test over 30 different architectures of convolutional neural networks (e.g., VGG [45] , ResNet [13, 57, 59] , DenseNet [19] , MobileNet [43] ) and Vision-Transformers (ViTs [5] , DeiT [49] , SwinT [30] ).TTA methods. We investigate 7 recent state-of-the-art TTA methods, including SHOT [27] , BN_Adapt [44] , TTT [46] , TENT [53] , ConjPL [9] , ETA [35] , and SAR [36] . This testbed includes different "pretraining" (e.g., self-supervision [46] , PolyLoss [26] ), updating certain layer parameters (BN layers [44, 53, 35] , LayerNorm (LN) layers [2, 36] , entire feature extractors [27] ). We test all adaptation baselines, except SAR, on convolutional neural networks with BN layers. We apply SAR specifically to vision transformers [5, 49, 30] because it prevents the model collapse that other adaptation methods exhibit in vision transformers with LN layers [36] .Calculating agreement. Given any pair of models (h, h ′ ) ∈ H that are tested on distribution D, the expected accuracy and agreement of the models is defined asAccuracy(h) = E x,y∼D 1{h(x) = y} , Agreement(h, h ′ ) = E x,y∼D 1{h(x) = h ′ (x)} (1)where h(x) and h ′ (x) are the normalized logits of models h and h ′ given datapoint x and y is the class label. Following [32] and [3] , we apply probit scaling, which is the inverse of the cumulative density function of the standard Gaussian distribution (Φ-1 : [0, 1] → [-∞, ∞]), over accuracy and agreement for a better linear fit, specifically in Figs. 1 and 2 .Online test in ID and OOD during TTA. Unlike conventional offline inference at test-time, TTAs involve online learning, which dynamically updates the model's parameters while testing on OOD data. To evaluate this continuously updated model on both ID and OOD data, the data is fed into the model in minibatches and we average over the accuracy and agreement of the model over the i'th minibatch after the dynamic adaptation step i. Details are provided in Algorithm 1 in the Appendix.We empirically observe that after adaptation, models show stronger linear trends in their ID versus OOD accuracy and agreements, even in distribution shifts where Vanilla models show wildly varying trends. In Fig. 1 , we demonstrate this on the four distribution shifts with weakest ACL and AGL trends in Vanilla models. These include CIFAR10-C Gaussian Noise, ImageNet-C Shot Noise, Camelon17-WILDS, and iWildCAM-WILDS, which all have R 2 values in their accuracy and agreement lower than 0.4 before TTA. These failure shifts have also been identified by previous studies [32, 3, 56] . Surprisingly, after applying several adaptation methods, such as TENT, the strength of these linear trends increase dramatically, (e.g., 0.15 → 0.97 and 0.33 → 0.97 in agreement and accuracy, in Camelyon17-WILDS). Our observations hold consistently across our entire testbed of shifts and adaptation methods, as shown in Figures 6, 7, 9, 10, 11, and 12 . Moreover, in shifts where Vanilla models already exhibit strong linear trends, such as ImageNet-V2 or FMoW-WILDS, these linear trends interestingly persist even after TTA including when TTA degrades OOD accuracy instead of improving (Fig. 4 ).Furthermore, we find that for each TTA method, models adapted with different adaptation hyperparameters follow the same linear trend in both accuracy and agreement. Specifically, we test learning rates, adaption steps, batch sizes, and the early-stopped epoch for pretraining. In Fig. 2 we see that on CIFAR10-C and ImageNet-C Gaussian Noise, and Camelyon17-WILDS, models adapted with different hyperparameters exhibit correlation strength R 2 close to 1 in both ID versus OOD accuracy and agreement. We also show that this occurs across other distribution shifts in Figs. 8 and 13.4 Why does adaptations lead to strong linear trends?In this section, we focus on explaining why TTA leads to restoration of these strong linear trends. Although we do not provide a full theoretical justification, we do identify a key pattern in how TTA modifies models that provides a strong clue as to why these methods substantially strengthen the ACL phenomenon in particular.We begin by revisiting the theoretically sufficient conditions for ACL from Miller et al. [32] over a simple Gaussian data and linear classifier setup. They showed that ACL holds perfectly only when distribution shifts by a simple scaling factor to the norm of the mean and covariance. On the other hand, if the actual direction of the mean or shape of the covariance matrix changes, the linearity breaks. We then demonstrate that even in CIFAR10 shifts and non-linear neural networks, adaptation causes models to meet these theoretical conditions better in the penultimate-layer feature space, thus leading to stronger linear trends.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: We provide the details about the type of GPU machines for our entire experiments in Section A.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
theoretical conditions for linear trends in gaussian data: 
 We introduce the theoretical conditions for having ACL in Gaussian data setup, from Miller et al. [32] , restated briefly here for ease of presentation. Consider the binary classification over Gaussian data setup with label y ∈ {-1, 1}. The OOD distribution Q differs from the ID distribution P by just some scaling constants α, γ > 0EQUATIONTheorem 1 (Miller et al. [32] , simplified) Under the Gaussian data setup in Equation 2, for linear classifiers f θ : x → sign(θ ⊤ x), the probit-scaled accuracies over P and Q observes perfect linear correlation with a bias of zero and a slope of α γ .The proof follows immediately from the fact that the accuracy of a linear classifier on this Gaussian data P is given by Φ(θ ⊤ µ/ √ θ ⊤ Σθ)); applying same result to Q immediately gives the desired linear relationship. The main implication is that if the data distribution where the linear classifiers are applied have the shifts that have the same mean directions and covariance shapes, ACL is guaranteed across linear classifiers.We now investigate how well the above conditions that are required for the linear trends are met before and after TTA, for real-world data with deep models. Here, we consider CIFAR10-C Gaussian Noise and and models pretrained on CIFAR10 then adapted using BN_Adapt [44] or TENT [53] at test-time. Since we are interested in distributions where the linear classifiers are applied onto, we take the class-wise feature embeddings from the penultimate-layer of these models. Then we analyze how these features are distributed by measuring their mean and covariance alignment, using cosine similarity of their normalized ones. We evaluate such alignment across architectures as well as adaptation hyperparameters such as learning rates, batch sizes, and early-stopped checkpoints, which are the same setups we used in Figs 1 and 2 CIFAR10-C Gaussian Noise results.Shifts become aligned in their mean and covariance after adaptation. Table 1 shows the mean and standard deviation of the cosine similarity measured across different setups. We first notice that, without adaptation (Vanilla), both mean and covariance of CIFAR10 and CIFAR10-C Gaussian Noise features have less than 1 cosine similarity, showing their are misaligned. Surprisingly, after applying adaptations, the similarity substantially increases and becomes very close to 1, across every architecture we test (with standard deviation close to 0). This means that ID and OOD, in feature space, have (roughly) the same means and covariances in their shapes, after adaptations. This implies that adapted models empirically seem to represent the shift in distributions largely as a shift in the scale of the features alone. Considering that CIFAR10-C Gaussian Noise is created with additive isotropic Gaussian Noise to CIFAR10, which is a non-isotropic covariance shift, this represents a non-trivial ability of TTA to "simplify" the nature of complex drifts within the features. This observations, in turn, approximately satisfies the conditions above and thus at least gives a partial insight into why we observe the strong linear trends such a setting after adaptations.In addition, we have the similar results when testing with varying hyperparameters, as shown in Table 1 . We apply TENT with different learning rates, batch sizes, and checkpoints, and they consistently show the similarity close to 1, similar to architectures. This shows that adapting with different hyperparameter setups does not affect the alignments between shifts, maintaining such simplicity. This explains why we also observe the strong linear trends across different hyperparameters.Theoretical slope for linear trends matches the empirical slope. Finally, the simple theoretical setting suggests not only that there is a linear correlation where distribution shifts involved only a scaling of mean and covariance, but also specifies the actual slope as α γ . We thus investigate whether this slope matches that produced empirically, when applying the same class of different adaptations as highlighted above. Table 1 shows the slopes predicted by the simple Gaussian setting, along with the slopes estimated via the empirical mean and variance scaling. These are generally in strong 1 : Cosine similarity between mean direction and covariance shape of class-wise penultimatelayer features, followed by the comparison between theoretical and empirical slope. They are evaluated on CIFAR10 vs. CIFAR10-C Gaussian Noise, measured across architectures and hyperparameters. We report their means and standard deviations.agreement, highlighting that the empirical results of TTA seem to provide a great deal of intuition about why how the ACL phenomenon holds in practice.Finally, we note that while these observations help explain ACL in adaptations, they do not apply to AGL, as AGL involves more than one linear classifier for calculating the agreement, and there is no similar closed-form estimate of the agreement even for Gaussian data. Still, we find that AGL is tightly coupled with ACL in an extensive range of benchmarking shifts and adaptations we examine on, similar to Baek et al. [3] .TTA aligns shifts that cause linear trends, correlating with improved OOD generalization. One might interpret the observed linear trends as evidence that TTA improves OOD generalizations by reducing the performance gap between ID and OOD, thereby bringing it closer to the y = x and showing a strong linear trend. However, our analysis clarifies that shift alignments by TTA is the true causal factor for stronger linear trends, and the closer gap between ID and OOD is correlated. While TTA satisfies the condition for ACL by aligning the direction/shape of the ID/OOD mean and covariance, the scaling factor might still be far off, i.e., α ≪ 1, γ ≫ 1. This results in a near-perfect linear trend that, nonetheless, can lie arbitrarily far from the y = x which represents perfect robustness. One empirical evidence is ImageNet-C Shot Noise in Fig. 1 , where there is still approximately a 40% gap in ID and OOD accuracy, but strong ACL holds. In addition, we could think of imaginary TTA which improves OOD performance by reducing scales shifts in covariance in the feature space, but the shape of the covariance matrices remain misaligned. This can be synthetically simulated over toy Gaussian data, where the cosine similarity between the shape of ID and OOD covariance matrices remain less than 1, and γ 2 decreases after TTA. Then the linear trend moves closer to y = x, but the strength of the linear trend remains weak. 
experiments: 
 Our observations, stronger linear trends after adaptations, provide substantial improvements, particularly in the context of adaptation, in two key practical applications: (i) accuracy prediction of the OOD accuracy, and (ii) unsupervised validation for TTA -without access to OOD labels.Experimental Setup. We employ AGL-based estimation method, namely ALine [3] . This method first estimates the linear fit of the accuracy, i.e. slope and bias, via agreements (which requires no labels), and linearly transforms the ID accuracy with them for estimating OOD accuracy. The detail of ALine is illustrated in Algorithm 2 in Appendix. We apply this to models adapted with TTA methods, and compare these results with (i) those of ALine applied to models before adaptation, and (ii) estimates using existing estimation baselines. These baselines include average thresholded confidence (ATC) [8] , difference of confidence (DOC)-feat [10] , average confidence [15] , and agreement [22] . We evaluate them on widely used benchmarking distribution shifts in TTA literature, CIFAR10-C, CIFAR100-C, ImageNet-C [14] , and Camelyon17-WILDS and iWildCAM-WILDS [41] . in Camelyon17-WILDS, MAE is 12.88%). As a result, as shown in Table 2 , applying ALine-S/D on Vanilla models outperforms other estimators with marginal gap, or sometimes shows higher errors. After applying adaptation methods, such as SHOT, BN_Adapt, and TENT, the estimation performance of ALine-S/D improves, showing substantially lower MAE compared to that of vanilla models across shifts (e.g., MAE decreases 10.76% → 2.34% in CIFAR10-C Gaussian Noise and 12.88% → 1.42% in Camelyon17-WILDS). It also outperforms the estimating baselines when applied to adapted models. Notably, baseline estimators also show improved estimation performance on adapted models compared to vanilla, which is unexpected. Nonetheless, ALine consistently performs better.In this section, we demonstrate that strong AGL allows practical applications in unsupervised validations, choosing the best hyperparameter for TTA. Furthermore, we extend our application of such unsupervised validation to the task of choosing best TTA strategy among baselines.Experimental setup. We select the best OOD model by selecting the one with best ID accuracy, as ACL holds across hyperparameters and best ID-performing hyperparameter is likely the best OOD-performing one [32] . Specifically, we first test the candidates by systematically sweeping over hyperparameter values, and select the one with the best ID accuracy. We focus on TENT, optimizing its various hyperparameters including learning rates, adaptation steps, architectures, batch sizes, and early-stopped checkpoints of pretraining. We select the candidates among the reasonably wide range of pools, as described in Table 6 in Appendix. To evaluate our method, we compare with other existing unsupervised validation methods including MixVal [17] , ENT [33] , IM [34] , Corr-C [52] , and SND [42] . We evaluate them on four shifts, CIFAR10-C, ImageNet-C, ImageNet-R, and Camelyon17-WILDS.Results. MixVal and 23.52% in IM. Such failures of existing baselines might come from their assumptions, e.g., low-density separation, that do not generalize to such distribution shifts. In contrast, our method shows consistently low MAE across shifts, including those where other baselines fail, e.g., 0.62% in Camelyon17-WILDS. We also observe that in some cases, e.g., learning rate tested in ImageNet-C, MAE is relatively large (9.70%), which stems from the models adapted with very small learning rates, e.g., 10 -5 that deviate from the correlation line.Application of selecting best TTA strategy. We could easily extend our reliable unsupervised validation to select the best TTA methods from the baselines. Different TTA methods exhibit unique slopes and biases in their linear functions, so we use AGL-based estimators to predict and compare each method's OOD accuracy across hyperparameter settings. Specifically, we evaluated five TTA baselines -BN_Adapt, TENT, SHOT, ConjPL, and ETA -on CIFAR10-C across all corruptions. Let us assume we are comparing TTA methods by controlling one hyperparameter at a time while keeping others fixed, e.g., comparing which TTA performs best with each method's optimal architecture or on average. For each hyperparameter, we identified the best OOD model for each TTA method using unsupervised validation. Fig. 3 illustrates the estimated versus true OOD performances. Optimal hyperparameter settings for each TTA method (i.e., best OOD) are marked with "x", and average performances across hyperparameters are marked with "o". Our method accurately estimates both the best and average OOD performances, aligning closely with the y=x line and maintaining the ranking of TTA methods by OOD accuracy. This demonstrates that our approach effectively selects the best TTA strategy without requiring labels in OOD.6 Ablation studies: What factors in TTA lead to strong linear trends?Despite intriguing observations of TTA that induces strong linear trends, the specific mechanisms driving this trend remain unclear. To investigate, this section presents an ablation study on two main adaptation components: normalization layers (e.g., BN, LN) and where it updates (e.g., feature extractor, final classifier).Normalization layers. We begin by ablating BN-layers, as BN_Adapt that normalizes features with OOD test data in BN layers lead to strong linear trends. Specifically, we test two variants -models with no normalization layers (denoted as Vanilla w/o N) and LN-layers (Vanilla w/ LN) -and report their cosine similarity of mean direction / covariance shape between shifts and correlation coefficient (R 2 ) in Table 4 . Interestingly, their similarities as well as correlations are larger than that of Vanilla with BN-layers, but still fall short of BN_Adapt's which is very close to 1. The results of SAR, which utilize LN instead of BN, in Fig. 12 also show that its vanilla shows relatively stronger linear trends than those of Vanilla in Fig. 11 , but not strongly and consistently as those of BN-layer-based methods in Fig. 11 . We conjecture that this stems from how differently distribution shifts are encoded in network: Such shifts absorbed into BN-layers result in severe covariate shift, but at the same time make it easy to align shifts via adaptation. In contrast, non-BN models encode shifts across the entire model parameters, suffering relatively less shift misalignment (in feature space), but remain worse than adapted BN-layer models. Schneider et al. [44] suggested the similar discussions.  
conclusion and limitations: 
 In this paper, we provide a key observation that recent TTAs lead to stronger AGL across a wide range of distribution shifts, encompassing those with weak correlations before adaptation. We explain this phenomena by the complexity of distribution shifts being substantially reduced to those with almost identical mean direction and covariance shape, satisfying the theoretical conditions for linear trends. This naturally leads to enhanced estimation of OOD performances across a wider range of shifts than before TTA, and also enables unsupervised hyperparameters as well as TTA method selection.While not requiring OOD labels, AGL intrinsically relies on ID data, which may undermine the advantage of TTA that doesn't require ID data for adaptations. This reliance can raise privacy concerns due to potential inclusion of sensitive information and increase computational demands. In Section D, we conduct an ablation study to minimize the required amount of ID data for observing AGL for accuracy estimation. The results show that even with only 5% of the original ID data, OOD accuracy estimation performance remains nearly the same as with full access, outperforming other estimators. We believe that overcoming dependency on ID data and exploring a fully test-time approach for observing AGL remains a promising direction for future research. 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: In Section 7, we discuss the positive societal impacts of our work, the significant improvements of reliability of neural networks under distribution shift.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: This work does not include such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: We properly cite all the test-time adaptation baselines and datasets we used in our paper.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification: This work does not release new assets.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: This work does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: This work does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: We list the types of architectures for each testbed of dataset.ResNet18 [13] ResNet26 [13] ResNet34 [13] ResNet50 [13] ResNet101 [13] WideResNet28 [59] PreActResNet18 [12] PreActResNet34 [12] PreActResNet50 [12] PreActResNet101 [12] RegNet X200 [37] RegNet X400 [37] RegNet Y400 [37] VGG11 [45] VGG13 [45] VGG16 [45] VGG19 [45] MobileNetV2 [43] PNASNet-A [28] PNASNet-B [28] SENet18 [18] GoogLeNet [47] ImageNet Architectures ResNet18 [13] ResNet34 [13] ResNet50 [13] ResNet101 [13] ResNet152 [13] ResNeXT32×4d [57] DenseNet121 [19] WideResNet50 [59] WideResNet101 [59] DLA34 [58] DLA46C [58] DLA60 [58] DLA102 [58] DLA169 [58] ViT-B/16 [5] ViT-L/16 [5] ViT-B/32 [5] SwinT-S [30] SwinT-B [30] SwinT-L [30] DeiT-S/16 [49] DeiT-B/16 [49] DeiT3-B/16 [50] DeiT3-L/16 [50] DeiT3-H/14 [50] WILDS Architectures ResNet18 [13] ResNet34 [13] ResNet50 [13] ResNet101 [13] ResNet152 [13] ResNeXT32×4d [57] DenseNet121 [19] WideResNet50 [59] WideResNet101 [59] VGG11 [45] VGG13 [45] VGG16 [45] VGG19 [45] Table 5 : The list of architecture types for each testbed of datasets, including CIFAR10, CIFAR100, ImageNet, Camelyon17-WILDS, and iWildCAM-WILDS.For CIFAR10, CIFAR100, and WILDS datasets, we train the models from the scratch, while for ImageNet, we use the pretrained model weights from torchvision and timm package. In addition, since TTT requires specific network composition required for the rotation-prediction task during pretraining, we train them using ResNet-14,26,32,50,104, and 152, which are available in the original implementation 3 .For Camelyon17 dataset, we also test the neural networks with their feature extractor (all parameters before the final linear classifier) randomly initialized, following Miller et al. [32] that tested the low-accuracy models on the dataset. We notice that these models, even with their most of parameters being randomized, still achieve high accuracy in both ID and OOD after training the last linear classifier. They also exhibit improved OOD accuracy after adaptations. We apply this randomization to ResNet18, ResNet34, ResNet50, VGG16, VGG13, and VGG11.We trained and tested all models and datasets in NVIDIA RTX 6000.We test the models on 9 different distribution shifts that include synthetic corruptions and real-world shifts. Synthetic corruptions datasets, CIFAR10-C, CIFAR100-C, and ImageNet-C [14] are designed to apply the 15 different types of corruptions, such as Gaussian Noise, on their original dataset counterparts. We use the most severe corruptions, which have severity of 5, in all experiments. These corruptions datasets are most commonly evalutated distribution shifts in a wide range of TTA papers [44, 46, 29, 27, 53, 9, 35, 36, 61] .We also test on real-world shifts, which include CIFAR10.1 [38] , ImageNetV2 [39] , ImageNet-R [16] , Camelyon17-WILDS, iWildCAM-WILDS, and FMoW-WILDS [41] . CIFAR10.1, and ImageNetV2 are the reproduction of datasets by following the original dataset creation procedures. ImageNet-R is the variant of ImageNet which contains the images with renditions of various styles, such as paintings or cartoons. FMoW-WILDS [41] contains the spatio-temporal satellite imagery of 62 different use of land or building categories, where distribution shifts originate from the years that the imagery is taken. Specifically, following Miller et al. [32] , we use ID set consists of images taken from 2002 to 2013, and OOD set taken between 2013 and 2016.Table 6 shows the hyperparameter pools for each setup when used for observing AGL across different hyperparameters. We use SGD optimizer with momentum of 0.9 for all adaptation baselines except for SAR, which uses sharpness-aware minimization (SAM) optimizer [6] .Learning Rates1 • 10 -5 , 5 • 10 -5 , 1 • 10 -4 , 5 • 10 -4 , 1 • 10 -3 , 5 • 10 -3 , 1 • 10 -2 , 5 • 10 -2 , 1 • 10 -1 , 5 • 10 -1 1 • 10 -5 , 2 • 10 -5 , 5 • 10 -5 , 1 • 10 -4 , 2 • 10 -4 , 5 • 10 -4 , 1 • 10 -3 , 2 • 10 -3 , 5 • 10 -3Batch Sizes (a) CIFAR10, CIFAR100, ImageNet Setup Camelyon17-WILDS iWildCAM-WILDS Learning Rates 2 • 10 -5 , 5 • 10 -5 , 1 • 10 -4 , 2 • 10 -4 , 5 • 10 -4 , 1 • 10 -3 , 2 • 10 -3 , 5 • 10 -3 , 1 • 10 -2 , 2 • 10 -2 , 5 • 10 -2 2 • 10 -5 , 5 • 10 -5 , 1 • 10 -4 , 2 • 10 -4 , 5 • 10 -4 , 1 • 10 -3 , 2 • 10 -3 ,We also examine the shifts that exhibit strong AGL without adaptations, including dataset reproductions (CIFAR10.1 [38] , ImageNetV2 [39] ), and other real-world shifts (ImageNet-R [16] , FMoW-WILDS [41] ). As seen in Fig. 4 , applying adaptation baselines, such as SHOT, TENT, and ETA, does not have improvements in OOD generalizations or even results in degradation, as evidenced by previous studies [53, 61] . However, they persist to have the strong linear trends, and this implies that once the distribution shifts show the correlations, adaptations do not affect on the trends. This highlights that we could potentially predict whether adaptations may succeed or falter in these shifts.C.1 ALine-S and ALine-D Baek et al. [3] propose ALine-S and ALine-D, which assess the models' OOD accuracy without access to labels by leveraging the agreement-on-the-line among models. We provide the detailed algorithm of ALine-S and ALine-D in Algorithm 2.Garg et al. [8] introduce OOD accuracy estimation method, ATC, which learns the confidence threshold and predicts the OOD accuracy by using the fraction of unlabeled OOD samples for which model's negative entropy is less that threshold. Specifically, let h(x) ∈ R c denote the softmax output of model h given data x from X OOD for classifying among c classes. The method can be written as EQUATIONwhere s is the negative entropy, i.e., s(h(x)) = c h c (x) log(h c (x)), and t satisfiesEQUATIONGuillory et al. [10] observe that the shift of distributions is encoded in the difference of model's confidences between them. Based on this observation, they leverage such differences in confidences as the accuracy gap under distribution shifts for calculating the final OOD accuracy. Specifically,EQUATIONC.4 Average confidence (AC)Hendrycks et al. [15] estimate the OOD accuracy based on model's averaged confidence, which can be written asEQUATIONJiang et al. [22] observe that disagreement between the models that are trained with different setups closely tracks the error of models in ID. We adopt this as the baseline for assessing generalization under distribution shifts, where we can estimate Acc OOD = Agr(P OOD ), where P OOD denotes the set of predictions of the models on OOD data X OOD . Ratio of ID Data Observing AGL requires the ID data during test time. Even if the access to labeled ID data is often much available in practice than obtaining OOD data, ID data might be limited in particular applications due to privacy issue. To further mitigate this concern, we conduct an ablation study on how the number of ID samples affects on the precision of accuracy estimation. Specifically, for each dataset, we use the subset of original ID data, such as 5% of total, for calculating both accuracy and agreemenet in ID. By randomly iterating 10 different subsets for each ratio, we test how the estimation performance vary according to different subsets. Fig. 5 shows that for CIFAR10-C and CIFAR100-C, only 5% reliably achieves the state-of-the-art estimation performances outperforming other baselines. Also, in more complex dataset such as ImageNet-C, Camelyon17-WILDS, and iWildCAM-WILDS, they show that even using just 1% of original ID data achieves the best estimation performances among baselines. These results indicate that our framework is practically feasible in such practical applications where access to ID is highly limited. Agreement, R 2 :0.99 Accuracy, R 2 :0.98 0.5 0.7 0.9 0.3 0.5 0.7 0.9 Agreement, R 2 :0.99 Accuracy, R 2 :0.99 0.5 0.7 0.9 0.3 0.5 0.7 0.9 Agreement, R 2 :0.99 Accuracy, R 2 :0.98 0.5 0.7 0.9 0.3 0.5 0.7 0.9 Agreement, R 2 :0.99 Accuracy, R 2 :0.99 0.5 0.7 0.9 0.3 0.5 0.7 0.9 fog Agreement, R 2 :0.96 Accuracy, R 2 :0.87 0.5 0.7 0.9 0.3 0.5 0.7 0.9 Agreement, R 2 :0.94 Accuracy, R 2 :0.88 0.5 0.7 0.9 0.3 0.5 0.7 0.9 Agreement, R 2 :0.91 Accuracy, R 2 :0.83 0.5 0.7 0.9 0.3 0.5 0.7 0.9 Agreement, R 2 :0.91 Accuracy, R 2 :0.82 0.5 0.7 0.9 0.3 0.5 0.7 0.9 Agreement, R 2 :0.92 Accuracy, R 2 :0.84 0.5 0.7 0.9 0.5 0.7 0.9 brightness Agreement, R 2 :0.99 Accuracy, R 2 :0.97 0.5 0.7 0.9 0.5 0.7 0.9 Agreement, R 2 :0.99 Accuracy, R 2 :0.99 0.5 0.7 0.9 0.5 0.7 0.9 Agreement, R 2 :0.99 Accuracy, R 2 :0.99 0.5 0.7 0.9 0.5 0.7 0.9 Agreement, R 2 :0.99 Accuracy, R 2 :0.99 0.5 0.7 0.9 0.5 0.7 0.9 Agreement, R 2 :0.99 Accuracy, R 2 :0.99 0.5 0.7 0.9 0.3 0.5 0.7 0.9 contrast Agreement, R 2 :0.90 Accuracy, R 2 :0.82 0.5 0.7 0.9 0.3 0.5 0.7 0.9 Agreement, R 2 :0.97 Accuracy, R 2 :0.96 0.5 0.7 0.9 0.3 0.5 0.7 0.9 Agreement, R 2 :0.97 Accuracy, R 2 :0.93 0.5 0.7 0.9 0.3 0.5 0.7 0.9 Agreement, R 2 :0.97 Accuracy, R 2 :0.94 0.5 0.7 0.9 0.3 0.5 0.7 0.9 Agreement, R 2 :0.97 Accuracy, R 2 :0.94 0.5 0.7 0.9 0.5 0.7 0.9 elastic_transform Agreement, R 2 :0.92 Accuracy, R 2 :0.74 0.5 0.7 0.9 0.5 0.7 0.9 Agreement, R 2 :0.97 Accuracy, R 2 :0.92 0.5 0.7 0.9 0.5 0.7 0.9 Agreement, R 2 :0.98 Accuracy, R 2 :0.93 0.5 0.7 0.9 0.5 0.7 0.9 Agreement, R 2 :0.98 Accuracy, R 2 :0.94 0.5 0.7 0.9 0.5 0.7 0.9 Agreement, R 2 :0.98 Accuracy, R 2 :0.94 0.5 0.7 0.9 0.3 0.5 0.7 0.9 pixelate Agreement, R 2 :0.87 Accuracy, R 2 :0.49 0.5 0.7 0.9 0.3 0.5 0.7 0.9 Agreement, R 2 :0.99 Accuracy, R 2 :0.98 0.5 0.7 0.9 0.3 0.5 0.7 0.9 Agreement, R 2 :0.99 Accuracy, R 2 :0.99 0.5 0.7 0.9 0.3 0.5 0.7 0.9 Agreement, R 2 :0.99 Accuracy, R 2 :0.99 0.5 0.7 0.9 0.3 0.5 0.7 0.9 Agreement, R 2 :0.99 Accuracy, R 2 :0.98 0.5 0.7 0.9 0.5 0.7 0.9 jpeg_compression Agreement, R 2 :0.90 Accuracy, R 2 :0.49 0.5 0.7 0.9 0.5 0.7 0.9 Agreement, R 2 :0.94 Accuracy, R 2 :0.75 0.5 0.7 0.9 0.5 0.7 0.9 Agreement, R 2 :0.94 Accuracy, R 2 :0.88 0.5 0.7 0.9 0.5 0.7 0.9 Agreement, R 2 :0.94 Accuracy, R 2 :0.88 0.5 0.7 0.9 0.5 0.7 0.9 Agreement, R 2 :0.94 Accuracy, R 2 :0.88Figure 6 : Results on CIFAR10-C corruptions (different architectures), BN_Adapt [44] , SHOT [27] , TENT [53] , and ETA [35] . Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.Justification: We have reviewed the NeurIPS Code of Ethics, and confirmed that our paper confirms them.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).Throughout this paper, we use AGL as comprehensive term for indicate when models show strong linear trends in both accuracy and agreement, and the slopes and biases of these trends are identical.AGL has not been similarly shown to hold theoretically for this case of scaled means, but empirically it often seems to hold when ACL holds[3]https://github.com/yueatsprograms/ttt_cifar_release 


Decision: This paper empirically observes that test-time adaptation methods strengthen the "accuracy-on-the-line" and "agreement-on-the-line" phenomena. All reviewers were in agreement that the paper is worth publishing. However, they also raised some concerns whose resolutions  should be in in the final version (e.g., adding the experiments on model selection).

Reviews: review: paper_summary: paper_summary: This paper presents observations that TTA models exhibit strong agreement-on-the-line (AGL) and accuracy-on-the-line (ACL) phenomenon, which persists across a wide range of distribution shifts and models. Leveraging this observation, the authors apply methods to estimate OOD accuracy without labeled data and perform a hyperparameter selection task for a TTA model. The proposed methods are evaluated extensively, demonstrating their effectiveness in TTA settings.
strength_weakness: strength_weakness: 1. The paper lacks a comprehensive comparison with the extensive body of literature on model selection methods under distribution shifts (e.g., i-iv below) that share a same goal of predicting OOD performances and thereby achieving a better model selection performance.

2. While the paper examines the hyperparameter selection task for individual TTA methods, a more interesting and practical task would be to choose the best model among various TTA methods. The suggested metric might have a slope unique to each TTA method, potentially complicating the model selection task among different TTA methods. Addressing this issue would enhance the practical applicability of the proposed approach. 

3. The paper is missing a core theoretical explanation of how TTA methods can result in a feature space that satisfies the sufficient condition in Theorem 1. Without this theoretical foundation, I have an impression that this paper could be interpreted as a simple application of Baek et al. and Miller et al.'s findings to the TTA setting. Providing a deeper theoretical insight would strengthen the contribution of the paper. 

i. Hu, D., Liang, J., Liew, J. H., Xue, C., Bai, S., & Wang, X. (2024). Mixed Samples as Probes for Unsupervised Model Selection in Domain Adaptation. In NeurIPS.  

ii. Musgrave, K., Belongie, S., & Lim, S. N. (2022). Benchmarking validation methods for unsupervised domain adaptation. arXiv preprint arXiv:2208.07360, 2(6), 12. 

iii. Yang, J., Qian, H., Xu, Y., & Xie, L. (2023). Can we evaluate domain adaptation models without target-domain labels? a metric for unsupervised evaluation of domain adaptation. arXiv preprint arXiv:2305.18712. 

iv. Saito, K., Kim, D., Teterwak, P., Sclaroff, S., Darrell, T., & Saenko, K. (2021). Tune it the right way: Unsupervised validation of domain adaptation via soft neighborhood density. In ICCV.
questions: questions: Kindly see the weaknesses part.
limitations: limitations: The authors partially discuss the limitations in the paper.
paper_summary: paper_summary: The paper presents a study on how test-time adaptation influences accuracy-on-the-line (ACL) and agreement-on-the-line (AGL). The authors empirically find that TTA methods significantly enhance the ACL and AGL, enabling better OOD performance estimation. Extensive experiments support their findings.
strength_weakness: strength_weakness: 1. The scope should be revised. TTA encompasses more than just feature extractor adaptation, but also classifier adaptation such as T3A [1]. Additionally, it appears that the author does not employ TTA methods with a memory bank [1-4]. Therefore, the paper should clearly specify which types of TTA methods support the observations.
2. The experimental setting is idealized. It seems that the paper only focuses that how TTA always improves accuracy. However, TTA might fail under continuously changing distributions, especially in cases of small batch sizes (e.g.,1), or significant distribution shifts. The author should provide a discussion of these scenarios.
3. The analysis in section 4.2 is not supervising, since BN_Adapt and TENT only modify the $\gamma$ and $\beta$ in BN layers to make the output features similar to the target domain. It would be better to discuss whether other non-BN methods align with these results.
4. One key advantage of TTA is that it does not require source domain data. However, the AGL-based method requires it, which limits the contribution of this paper.

[1] Iwasawa Y, Matsuo Y. Test-time classifier adjustment module for model-agnostic domain generalization. In NeurIPS. 2021.

[2] Jang M, Chung S Y, Chung H W. Test-time adaptation via self-training with nearest neighbor information. In ICLR. 2023.

[3] Yuan L, Xie B, Li S. Robust test-time adaptation in dynamic scenarios. In CVPR. 2023.

[4] Wang S, Zhang D, Yan Z, et al. Feature alignment and uniformity for test time adaptation. IN CVPR. 2023.
questions: questions: 1. Visualizing the change of correlation coefficient as the model adapts to the test data flow could be more beneficial for understanding the contribution of the paper.
limitations: limitations: N/A
paper_summary: paper_summary: This papers shows that using test-time-adaptation  improve the accuracy of the algorithms used for OOD performance estimation algorithms based on agreement-on-the-line and accuracy-on-the-line. This is shown through a series of experiments. Some justification has also been provided based on prior work,.
strength_weakness: strength_weakness: - Intuitively, if we assume that TTA is *perfect*, then we expect ID and OOD accuracy to be the same (i.e., y = x); the model has been fully adapted to the OOD setting and there is no difference between ID and OOD. Also the ID and OOD agreements should intuitively become the same (i.e., y = x). Is this understanding correct? It seems that there is a very basic reason behind the main observation of the paper. ‌Because of this, I think table Table 2 does not show a fair comparison; TTA is making models more similar and shift all the lines towards the y = x line. Is the message of the paper as simple as this? I think this answers the question "What specific mechanisms within these adaptation methods cause the observed phenomena? " asked in the conclusion section.


- In a practice, we use OOD performance estimation with unlabeled data in settings where compute, budget, etc. is limited; thus we cannot collect new labeled data. Given this, is a method that is computationally expensive (that involves test time adaptation) feasible? Also, note that this method does not work if we only have black box access to a model (which is again, typical in potential applications of AGL and ACL).


- What is the condition of theorem 1? It is written that "... correlation if with a bias of zero and a slope ...". Also, the condition in Eq. (2) is very strong. See my comments below for other theoretical results on AGL and ACL that study the problem in other (I believe more realistic) settings where the covariance and mean can shift. 


- The literature review is missing many theoretical results on agreement-on-the-line and accuracy-on-the-line. In the past 1-2 years, there has been a lot of theory results on these topics that show that these phenomenon are features of the high-dimensionality.

For example:

[1] N Tripuraneni, B Adlam, J Pennington, Overparameterization improves robustness to covariate shift in high dimensions, NeurIPS 2021.

[2] D Lee, B Moniri, X Huang, E Dobriban, H Hassani, Demystifying Disagreement-on-the-Line in High Dimensions, ICML 2023.

[3] D LeJeune, J Liu, R Heckel, Monotonic Risk Relationships under Distribution Shifts for Regularized Risk Minimization, JMLR, 2024.

[4] A Sanyal, Y Hu, Y Yu, Y Ma, Y Wang, B Schölkopf, Accuracy on the wrong line: On the pitfalls of noisy data for out-of-distribution generalisation, Arxiv 2024.

- The papers above show that these phenomena are fragile; these papers show clear settings where Accuracy-on-the-line and agreement-on-the-line break.

- More important than the previous comment: Lee et al (2023) show that the line for agreement and the line for accuracy can have different intercepts (at least in their toy model). Although the liens have the same slope. This is seen when looking back at the plots of the original real world experiments in Baek et al. This issue needs to be addressed before any real application of AGL for practice. There might be some fundamental issue with the AGL method.
questions: questions: See weaknesses.
limitations: limitations: The authors should discuss the limitations I mentioned in the weaknesses section.
paper_summary: paper_summary: This paper try to achieve reliable TTA by tackling three bottlenecks, including performance evaluation without labeled data, some distribution shifts, and hyperparameter selection for TTA methods.
strength_weakness: strength_weakness: 1.	The novelty is relatively limited; it is an application of Baek et al. (2022) to the TTA setup. And the limited theoretical conclusion is from Miller et al. (2019).

2.	It might be necessary to evaluate on more complex tasks, such as object detection and semantic segmentation.
questions: questions: Overall, the work is an important study of TTA, however there are still some questions.

1.	About Algorithm 1: In TTA, does updating the model parameters require multiple iteration steps? If it does, according to the basic idea of online algorithms, Algorithm 1 does not seem to reflect the model parameters being trained with different batches of data at different time steps. Could you provide a detailed explanation of the difference between the online algorithm you provided and the traditional offline algorithm?

2.	Why the inverse of the cumulative density function of the standard Gaussian distribution is applied on the axes of accuracy and agreement? Is there any intuitions and theoretical explanations?
limitations: limitations: No potential negative societal impact.

Comments: title: Further response to Reviewer ZucK's comments
comment: Dear Reviewer puv1,

Thank you for updating your score, and we are glad that the clarifications were helpful. We will make sure to include such results and clarifications in our final manuscript.
I appreciate the authors' efforts in addressing my questions and conducting additional experiments. I believe the newly added experiments enhance the practical value of the findings presented in the paper. Considering the rebuttal and the comments from other reviewers, I have increased my score to 5.
Dear Reviewer puv1,

Thank you once again for your thoughtful and detailed review. We kindly remind you of our rebuttal that include:
* (**W1**) Adding a comparison to existing model selection baselines,
* (**W2**) Demonstrating best TTA method selection,
* (**W3**) Providing intuition on how TTA satisfies the condition in Theorem 1, and clarified novelty against Miller et al. and Baek et al.,

to address your concerns. We are happy to discuss with you further if you have any other questions or feedback!
Dear Reviewer 1nU3,

Thank you for your positive score, and we are glad that the clarifications were helpful. We will make sure to include such clarifications in our final manuscript.
Dear Authors,

Thank you for your response. I maintain my initial positive rating.
Dear Reviewer JQG9,

Thank you for your positive score, and we are glad that the clarifications were helpful. We will make sure to include such clarifications in our final manuscript.
Dear Reviewer ZucK,

Thank you for your updated score and we are glad that the clarifications were helpful. We will make sure to include the discussion in our final manuscript.
Dear Authors,

Thank you for your response. I maintain my positive rating.

Best,

Reviewer JQG9
I thank the authors for their response. My concerns about the paper are now resolved. However, I believe this discussion is necessary and adding it will significantly increase the quality of the paper. Under the condition that these discussion will be added to the paper, I increase my score to 7 and recommend acceptance.
Dear Reviewer Zuck,

We really appreciate your quick response to our rebuttal, and we will definitely include further theoretical discussion on ACL/AGL theory and the computational complexity of our method in our final manuscript. 

You were still concerned that the strong linear trend, which appears after TTA, is caused by model performances approaching the $y=x$ line. We clarify the subtle point below as to **why the closer ID and OOD gap is correlated, but not causal, to the strong linear trend**.  

As stated in Theorem 1, under distribution shifts that simply “scale” up or down the _norm_ of the mean and covariance, i.e.,  $\mu_{OOD} = \alpha \mu_{ID}$, $\Sigma_{OOD}  = \gamma^2 \Sigma_{ID}$, classifiers can observe perfect linear trends with slope $\alpha / \gamma$. In Section 4, we showed that TTA satisfies this condition by aligning the “direction/shape” of the ID/OOD mean and covariance, while the scaling factor might still be far off, i.e., $\alpha$ << $1$, $\gamma$ >> $1$. Note that the degree of “scale shift” can be large, and in fact, _the perfect trend can lie arbitrarily far from $y=x$_, i.e., $\alpha / \gamma$ << $1$. We see examples of this in our paper, such as TENT on ImageNet-C Shot Noise. 

On the other hand, any shift, however small, that moves the “direction / shape” of the mean or the covariance matrix breaks the perfect linear trend _in finite dimensions_ (Miller et al.[1]). As an intervention, let us imagine a “Scale TTA” which improves OOD performance by reducing “scales shifts” in covariance in the feature space, but the “shape” of the covariance matrices remain misaligned. We can synthetically simulate the effect of such TTA over toy Gaussian data, where the cosine similarity between the shape of ID and OOD covariance matrices remain less than 1, and $\gamma^2$ decreases from 1 to 0.2 after Scale TTA. Empirically, we can see that the linear trend moves closer to $y=x$ (slope 0.66$\rightarrow$ 0.8), but the strength of the linear trend remains weak ($R^2$ 0.75$\rightarrow$ 0.63). This empirical evidence proves that improving OOD performance does not necessarily make the linear trend stronger. 

Overall, we see that ACL improves after TTA because TTA aligns the “shape” of the means/covariances, which is the true causal factor for _both_ stronger ACL and the linear trend moving closer to $y = x$. On the other hand, had TTA only reduced “scale shifts” and the “shapes” were unaligned, models would observe weak linear trends even though the slope moves closer to $y=x$. 

We hope this addresses your concern, and thus makes our paper’s observation and analysis more interesting to you. We will make sure to add the clarification in our final manuscript. Let us know if you have any questions!


>**Reference**  \
> [1] Miller et al., Accuracy on the Line: On the Strong Correlation between Out-of-Distribution and In-Distribution Generalization, ICML 2021
I thank the authors for their detailed response. Below please find my response:

1. I'm still not convinced that these phenomena are independent. In the same experiment that you mention in the rebuttal, there is a significant improvement in the OOD errors after applying TENT. The OOD and ID accuracies are becoming closer. How do you argue that these phenomena are independent?  There needs to be a systematic experiment that demonstrates that the observed phenomena cannot be simply explained by the decreased gap between ID and OOD by TTA. 

2. The rebuttal resolves my concerns about the computational complexity. Although I believe this that this potential limitation should be further discussed in the paper.

3. Thanks for the clarification.

4. Thanks for the clarification. However, this response further strengths my doubt that most probably, the observed phenomena can be simply explained by the decreased gap between ID and OOD by TTA. As you discuss here, after TTA, the covariances and means align very well.

5. Thanks. I believe such discussion will strengthen the quality of the paper.