Improving self-training under distribution shifts via anchored confidence with theoretical guarantees

Authors: taejong  joo
diego  klabjan
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 22:45:50

Abstract: Self-training often falls short under distribution shifts due to an increased discrepancy between prediction confidence and actual accuracy. This typically necessitates computationally demanding methods such as neighborhood or ensemble-based label corrections. Drawing inspiration from insights on early learning regularization, we develop a principled method to improve self-training under distribution shifts based on temporal consistency. Specifically, we build an uncertainty-aware temporal ensemble with a simple relative thresholding. Then, this ensemble smooths noisy pseudo labels to promote selective temporal consistency. We show that our temporal ensemble is asymptotically correct and our label smoothing technique can reduce the optimality gap of self-training. Our extensive experiments validate that our approach consistently improves self-training performances by 8% to 16% across diverse distribution shift scenarios without a computational overhead. Besides, our method exhibits attractive properties, such as improved calibration performance and robustness to different hyperparameter choices.

Main paper: 
 introduction: 
 In this work, we address the challenge of adapting pre-trained neural networks at test time under distribution shifts, a problem known as test-time adaptation (TTA) or source-free domain adaptation (SFDA). Distribution shifts-where a model trained on one distribution is then tested on a different one-are ubiquitous in many practical scenarios due to demographic subpopulation shift [1] and changes in data collection environments [2] [3] [4] . Despite the robust performance of neural networks under independent and identically distributed (i.i.d.) settings, they often suffer from substantial performance degradation under such shifts [5, 6] . Recently, TTA and SFDA have proven their effectiveness in resolving these critical issues by effectively leveraging information about the distribution shifts contained in unlabeled samples given at the test time.Self-training is the basis of many state-of-the-art methods in TTA and SFDA [7] [8] [9] , utilizing pseudo labels generated from the model's own predictions to train a model on unlabeled samples [10] . Since pseudo labels are regarded as true labels in self-training, the success of self-training methods highly depends on how to filter incorrect pseudo labels to prevent self-confirmation bias [11] . This issue has been effectively handled via simple confidence-based thresholding in i.i.d. settings [12] [13] [14] . However, the distribution shifts make it hard to filter incorrect pseudo labels due to high noise rates even under high threshold [15] . Thus, sophisticated methods filter incorrect pseudo labels based on a neighborhood structure of the data [16] [17] [18] and consistency of multiple predictions under different models [19] or augmentations [20] (cf. Section 5) , which are computationally intensive by nature. previous methods. The temporal consistency regularizer, so called early learning regularization (ELR) [21] , was originally developed to address neural networks' tendency to learn clean information first and then gradually memorize noisy labels [22, 23] ; this setting is naturally connected to self-training scenarios when we regard the pseudo labels as random noisy labels. However, the impacts of ELR on self-training have not been fully understood. Also, since ELR does not consider unique characteristics of distribution shifts, we aim to answer the following question: Is there any principled way to improve the way of memorizing all past predictions tailored self-training under distribution shifts?In this work, we show that the answer is affirmative by proposing Anchored Confidence (AnCon) that uses confident predictions to support a generalized notion of temporal consistency. Specifically, we construct a generalized temporal ensemble, which weighs predictions based on predictive uncertainty, and then use the ensemble as a smoothing vector in label smoothing [24] . Then, through rigorous theoretical analyses, we show that our simple heuristic for the generalized temporal ensemble is asymptotically correct and that the label smoothing formulation can reduce the optimality gap. As a result, AnCon can correct wrong pseudo labels without expensive computations and can be easily applied to self-training methods by replacing one-hot pseudo labels with smooth pseudo labels, unlike neighborhood-based or centroid methods. Through extensive experiments, we show that AnCon improves self-training under diverse distribution shift scenarios and posses many attractive properties.Our contribution can be summarized as follows: 1) We develop AnCon, which is the first algorithm that attempts to improve self-training under distribution shifts by generalizing a notion of temporal consistency with theoretical guarantees; 2) Without any additional forward passes or neighborhood search, AnCon improves self-training performances 8% and 16% under domain shifts and image corruptions, respectively; 3) Remarkably, we also show that AnCon significantly improves calibration performance and is robust with respect to model selection methods and hyperparameter choices. 
background: 
 Notation and setup For an input space X ⊆ R d and a label space Y = [K] := {1, 2, • • • , K}, we define X and Y to be random variables of input and output with probability densities p X and p Y , respectively. We also define a neural network f (•; θ) : X → △ K-1 parameterized by a parameter θ ∈ R p where △ K-1 is the probability simplex with K elements. Our goal is to minimize the cross-entropy loss min θ∈R p l(θ) := E XY [H(f (X; θ), p Y |X )] where H(f (x; θ), p Y |X=x ) := -k∈[K] p(Y = k|x) log f k (x; θ). In the SFDA setting, we are given an initial parameter θ 0 that is trained on a different data generating distribution (X ′ , Y ′ ), e.g., θ 0 ∈ arg minθ∈R p E X ′ Y ′ [H(f (X ′ ; θ), p Y ′ |X ′ )].We can think about this setting as either (1) (X ′ , Y ′ ) being pre-training data and θ 0 the foundation model with the task of fine-tuning the model on unlabeled X or (2) a transfer learning problem with (X ′ , Y ′ ) being the source domain data and X the target domain. Here, we assume only covariate shift without concept shift; that is, p X ̸ = D p X ′ but p Y |X = D p Y ′ |X ′ . Even in this case, we note that suboptimality under the distribution shift, i.e., min θ∈R p l(θ) -l(θ 0 ), can be large.In this work, we tackle the distribution shifts by using the self-training method that replaces the true label Y (x) by the pseudo label, i.e., min θnew l(θ new ; θ) = E X [-log f Ŷ (x;θ) (x; θ new )] where Ŷ (x; θ) := arg max k∈[K] f k (x; θ) is a pseudo label under θ. Specifically, the algorithmic framework of self-training is as follows: given θ 0 , we iteratively find model parameter θ m+1 for m = 0, 1, • • • with θ m+1 ∈ arg min θnew l(θ new ; θ m ). For later use, we also define a prediction confidence c(x; θ):= max k∈[K] f k (x; θ) and θ 0:m := (θ 0 , • • • , θ m ).Early learning regularization In the learning from noisy labels (LFN) scenario, [21] identified an "early-learning phenomenon" where neural networks initially learn information contained in clean labels before gradually memorizing noisy labels, leading to a performance deterioration as training progresses. To mitigate this issue, ELR penalizes predictions that deviate from earlier predictions by defining a target network from the past predictions: fELR (x;θ 0:m ) := m j=0 (1 -γ) • γ m-j f (x; θ j ).Then, adding an auxiliary loss of L ELR (θ; θ 0:m ) = E X [log(1 -f (X; θ) T fELR (X; θ 0:m ))] to l(θ; θ m ) can prevent memorization of noisy labels while preserving correct patterns.Notably, this insight has recently been confirmed to be applicable in the SFDA setting by [8] . This observation is appealing because ELR can be efficiently implemented by reusing past predictions without additional forward passes or neighborhood searching, unlike dominant methods in SFDA [16] [17] [18] [19] [20] . Nevertheless, given that ELR stems from a general property of the neural network training in the i.i.d. setting, herein we aim to step towards a more principled approach to encourage temporal consistency tailored for distribution shift scenarios. 
anchored confidence: 
 In this section, we introduce AnCon, which promotes the temporal consistency on selectively chosen predictions via label smoothing. In Section 3.1, we first explain the idea of promoting selective temporal consistency based on confident predictions via label smoothing [24] with a temporal ensemble. Then, in Section 3.2, we explain how to effectively construct temporal ensemble for improving self-training under distribution shifts. Finally, we theoretically analyze the efficacy of AnCon by drawing connection between our method and knowledge distillation in Section 3.3.In this work, we utilize label smoothing [24] to promote the selective temporal consistency instead of using an auxiliary loss function like ELR. Specifically, given a generalized temporal ensemble f (x; θ 0:m , w 0:m ) with w 0:m := (w 0 , • • • , w m ) which will be specified in Section 3.2, we construct a regularized pseudo label Ỹ (X; θ 0:m , w 0:m ) by using f (x; θ 0:m , w 0:m ) as a smoothing vector for the pseudo label Ŷ (x; θ m ). That is, we perform self-training bymin θ E X [H(f (X; θ), Ỹ (X; θ 0:m , w 0:m ))], Ỹ (X; θ 0:m , w 0:m ) = (1-λ)E 1 ( Ŷ (x; θ m ))+λ f (x; θ 0:m , w 0:m ) (1) where λ ∈ [0, 1] is a coefficient, E 1 (•) is the one-hot encoding, and f (x) := ( f1 (x), • • • , fK (x)) is the K-dimensional output of the generalized temporal ensemble (cf. (2)).Thus, AnCon can control the usage of potentially noisy information in Ŷ (x; θ m ) based on its consistency with f (x; θ 0:m , w 0:m ). Not only can this approach preserve the early learning phenomenon as in ELR (cf. Section 2), but the label smoothing formulation also significantly stabilizes the selftraining performance under different hyperparameter choices due to the fact that the optimal values of hyperparameters are less problem dependent compared to its equivalent auxiliary regularization [25] . Beyond removing the burden of hyperparameter search, this robustness is a particularly intriguing property under distribution shift scenarios where the model selection becomes a challenging task.Further, encouraging the temporal consistency through label smoothing enables us to connect our method with knowledge distillation (KD) [26] , which can provide a wide range of principled techniques and theoretical results developed in KD. As a concrete example, we will show when and how AnCon can reduce the optimality gap of self-training, i.e., a case with λ = 0, in Section 3.3.Next, we construct the generalized temporal ensemble that makes the selective temporal consistency in (1) work effectively in self-training under distribution shifts. Specifically, given θ 0:m and weights w 0:m (x) ∈ △ m for each x ∈ X , the prediction by the generalized temporal ensemble isEQUATIONwhere p(y|x, θ i ) is the prediction made by f (x; θ i ), which can be either soft (p(y = j|x, θ i ) = f j (x; θ i )) or hard (p(y = j|x, θ i ) = 1 if j = arg max k∈[K] f k (x; θ i ) and p(y = j|x, θ i ) = 0 otherwise).In this work, we use hard prediction because soft prediction puts more weights on recent predictions since self-training tends to keep increasing the prediction confidence during training.Surprisingly, we will show that the following simple relative thresholding for determining w 0:m gives the asymptotic optimal weights achieving the minimum worst-case optimality gap of self-training:EQUATIONwhere δ(β)m is an exponential moving average (EMA) of prediction confidence with hyperparameter β and Ê[c(X; θ i )] is a Monte-Carlo approximation of E[c(X; θ i )] with mini-batch samples.Intuitively, our weighting mechanism aggregates only relatively confident predictions with a uniform weight. Given the observation that relative ordering of confidence is highly correlated with accuracy even under distribution shifts [27, 28] , our thresholding rule would tend to put non-zero weights on correct predictions. Besides, by employing the relative criterion, the thresholding does not suffer from the problems that neglect predictions obtained in the early stage of training.We also remark that AnCon has almost the same computational cost as ELR. Specifically, (2) can be implemented by fk (x; θ 0:m , w 0:m ) = fk (x; θ 0:m-1 , w 0:m-1 ) + w m (x)p(y = k|x, θ m ) that requires to store the weighted sum of previous predictions without additional forward passes or storing previous parameters, which is the same as storing the previous logit vector in ELR. Similarly, (3) can be efficiently implemented by δ(β) m = βδ (β) m-1 +(1-β) ÊX [c(X; θ m )]that requires constant additional computational costs compared to vanilla self-training with the constant being small. Therefore, AnCon shares the same computational benefits as ELR compared to other state-of-the-art methods in SFDA.On optimality of the relative thresholding in (3) From the optimization perspective, the optimal weights w † 0:m correspond to the weights under which self-training with Ỹ (X; θ 0:m , w 0:m ) can minimize the expected loss:EQUATIONUnfortunately, l(θ † w0:m ), or its empirical counterpart, is not available in self-training due to the absence of labels. Further, even if labels are given, solving (4) is intractable due to non-smoothness of l(θ † w0:m ) with respect to w 0:m and the cost of finding θ † w0:m . To circumvent this issue, we show in Section 3.3 that (4) can be relaxed to the problem of finding ensemble weights that give a maximum likelihood estimation (MLE) solution under certain conditions. As a result, instead of solving the intractable optimization in (4), we find the optimal weights byEQUATIONIn the following theorem which is proven in Appendix B.2, we show that the simple relative thresholding in (3) can make f (x; θ 0:m , w 0:m ) asymptotically correct for samples where the neural network tends to be relatively confident during self-training and thus our simple weighting mechanism in (3) to be the solution of (5) in the asymptotic region. Theorem 3.1.Let A i (c) := {x ∈ X |c(x; θ i ) > c}, Q(x; c 0:m ) := m i=0 1(x ∈ A i (c i )), and p(x; c 0:m ) = 1 Q(x;c0:m) m i=0 E Y |X=x [1(Y (x) = Ŷ (x; θ i ))]1(x ∈ A i (c i )) for x such that Q(x; c 0:m ) >0. Let us assume that random events 1(Y (x) = Ŷ (x; θ i )) and 1(Y (x) = Ŷ (x; θ j )) are conditionally independent given X ∈ A i (c) for j ∈ {0, • • • , i -1}, x ∈ X , c ∈ [0, 1). If x ∈ X such that p(x; c 0:m ) > 1/2,then for the generalized temporal ensemble in (3), it holds that p(arg maxEQUATIONwhere ξ(z):= 2z -1 -log(2z) is a positive increasing function in z ∈ [0.5, 1].The result states that as long as the average accuracy for relatively confident predictions over iterations exceeds 50%, the error rate of the generalized temporal ensemble monotonically decreases as Q(x; c 0:m ) increases. Furthermore, f (x; θ 0:m , w 0:m ) is asymptotically correct on x such that Q(x; c 0:m ) → ∞ as m → ∞. AnCon aims to achieve these desirable properties through the uncertainty-aware temporal consistency that helps to satisfy the condition p(x; c 0:m ) > 0.5. Specifically, as shown in Figure 4a in Appendix, our generalized temporal ensemble's accuracy tends to significantly increase as the number of confident samples increases, being consistent with our theory. We note that this monotonic improvement would not be the case for the temporal ensemble without uncertainty-awareness and vanilla self-training (cf. Figure 4a ).Finally, we emphasize that the assumption p(x; c 0:m ) > 0.5 applies only to relatively confident predictions which are averaged over iterations. This is significantly weaker than requiring a lower bound of an expected accuracy of each sample for every iteration, which is the case when LFN methods are directly applied to the self-training scenario. Also, due to its dependency on the choice of the confidence thresholds c 0:m , the assumption can hold by controlling c 0:m at the expense of loosening the upper bound in (6) (e.g., selecting 90th-quantile as in Figure 4b in Appendix). Specifically, increasing the thresholds can improve ξ(p(x; c 0:m )) and enhance the chance of satisfying p(x; c 0:m ) > 1/2 but reducing Q(x; c 0:m ). While this trade-off necessitates a proper choice of c 0:m , our extensive experiments show that setting the threshold c m by the EMA of prediction confidence, i.e., δm in (3), works effectively.In this section, we present a novel connection between AnCon and KD for addressing intractability of (4). KD is a framework for training a small student network f , e.g., ResNet-50 [29] , with an additional supervision from a large teacher network f (t) , e.g., ResNet-152. Specifically, a cross-entropy under KD isl KD (θ) = E XY H(f (X; θ), (1 -λ KD )E 1 (Y (X)) + λ KD f (t) (X)) with λ KD ∈ [0, 1],which bears a significant similarity with AnCon (cf. ( 1)). Indeed, AnCon can be understood as a special case of KD called self-distillation when f and f (t) have the same architecture, where the generalized temporal ensemble f corresponds to the teacher network f (t) with a notable difference that the pseudo label Ŷ is used instead of the true label Y . Based on this connection, we perform a convergence analysis of AnCon by modifying the partial variance reduction theory [30] , as given below. We note that the usage of Ỹ and f results in an inherently biased gradient estimator, which requires special treatments for the convergence analysis unlike the typical self-distillation setting.Setup Following [30] , we assume a linear modelf k (x; θ) = exp(Θ T k x)/ i∈[K] exp(Θ T i x) with Θ i ∈ R d for i ∈ [K], θ := Concat(Θ 1 , • • • , Θ K ) ∈ R dK ,and Concat(•) is the concatenation operation. Also, we assume a bounded support for X; that is, ∥ x ∥≤ C for all x where p X (x) > 0. Under this setting, we repeat the following steps starting from a given θ 0 (i.e., m = 0):1. Outer temporal ensemble update: Update w 0:m by ( 3) to obtain f (x; θ 0:m , w 0:m ) (cf. ( 2)).2. Inner parameter update: With θ m,0 := θ m and (θ 0:m , w 0:m ), solve (1) with stochastic gradient descent θ m,t+1 = θ m,t -γg(m,t) ξ for t ∈ {0, 1, • • • , T -1} and set θ m+1 = θ m,T . 3. Iteration number update: Set m = m + 1 and terminate if m = T .Here, γ is the learning rate, ξ contains b random samples from p X , and the stochastic gradient under AnCon is defined as g(m,t) ξ := ∇ θm,t 1 b Xi∈ξ H(f (X i ; θ m,t ), Ỹ (X i ; θ 0:m , w 0:m )).For the linear model, we note that g(m,t) ξ = ∇ lξ (θ m,t ) -λĝ ξ (θ 0:m , w 0:m )where lξ (θ m,t ) =1 b Xi∈ξ H(f (X i ; θ m,t ), Ŷ (X i ; θ m )) and ĝξ (θ 0:m , w 0:m ) := Concat(ĝ 1,ξ (θ 0:m , w 0:m ), • • • , ĝK,ξ (θ 0:m , w 0:m ))with ĝk,ξ (θ 0:m , w 0:m ) := 1 b Xi∈ξ [( fk (X i ; θ 0:m , w 0:m ) -Ŷk (X i ; θ m ))X i ] for k ∈ [K]. In Theorem 3.γ ≤ µ 4L•L , a carefully chosen λ (cf. λ = λ † m := E ξ [⟨∇ lξ (θ * ),ĝ ξ (θ0:m,w0:m)⟩] E ξ ∥ĝ ξ (θ0:m,w0:m)∥ 2 + 2Lγ ∥ĝ(θ0:m,w0:m)∥ 2 in Lemma B.6 in Appendix B.5), and any realization of θ m , it holds thatE[l(θ m,t ) -l(θ * )|θ m ] Optimality gap ≤ (1 -γµ) t (l(θ m ) -l(θ * ))Improvement over the initial model+ 8C 2 µ g E (θ m )Bias of the pseudo label+ 2 µ N (λ † m ; θ 0:m , w 0:m ) Neighborhood size (7) where l * := l(θ * ) with θ * ∈ arg min θ∈Θ l(θ), g E (θ) := E[1( Ŷ (X; θ) ̸ = Y (X))], N (λ; θ 0:m , w 0:m ) = λ 2 ∥ ĝ(θ 0:m , w 0:m ) ∥ 2 + Lγ 2 E ξ ∥ ∇ lξ (θ * ) -λĝ ξ (θ 0:m , w 0:m ) ∥ 2 , and N (λ † m ; θ 0:m , w 0:m ) ≤ N (0)where N (0) := N (0; θ 0:m , w 0:m ) for any (θ 0:m , w 0:m ). Further, when w 0:m is such that E ξ [⟨∇ lξ (θ * ), ĝξ (θ 0:m , w 0:m )⟩] ≥ 0, i.e., the teacher has a sufficiently good performance, it holds thatEQUATIONwhereσ 2 * := E ξ ∥ ∇ lξ (θ * ) ∥ 2 , σ 2 (θ0:m,w0:m) := E ξ ∥ ĝξ (θ 0:m , w 0:m ) ∥ 2 , g KL (θ 0:m , w 0:m ) = E X [D KL (f (X; θ * ) ∥ f (X; θ 0:m , w 0:m ))] with D KL (p ∥ q)is the KL-divergence between p and q, and gC (θ 0:m , w 0:m ) =∥ E X [ f (X; θ 0:m , w 0:m )] -E X [ Ŷ (X; θ m )] ∥ 2 .In Theorem 3.2, (7) characterizes the optimality gap in the inner loop optimization. Specifically, the first term is about reducing the initial optimality gap over iterations and motivates why we need adaptation, e.g., by self-training, if the performance deteriorates under severe distribution shifts. The second term is about the bias of the pseudo label and motivates the challenges of self-training under poorly performing pseudo labels in the case of severe distribution shifts. Crucially, these two terms can be fully characterized by the quality of initial model θ m and do not depend on w 0:m . Thus, we concentrate on the impacts of w 0:m designed in ( 1)-(3) on N (λ † m ; θ 0:m , w 0:m ) to show that AnCon's effectiveness on improving self-training performance under distribution shifts.First, Theorem 3.2 shows the effectiveness of our label smoothing formulation in (1) under properly chosen λ † m . Specifically, compared to vanilla self-training, AnCon results in the smaller neighborhood size of the stochastic gradient descent; N (λ † m ; θ 0:m , w 0:m ) ≤ N (0). That is, the result suggests that AnCon is at least better than vanilla self-training under the mild regularity conditions. Further, under the additional assumption of a sufficiently good performance temporal ensemble, (8) motivates AnCon's weighting mechanism as a relaxed solution of the intractable optimization problem in (4). Specifically, if the marginal distribution of the pseudo labels does not change quickly over outer iterations (which is the case especially for the later training stages as shown in Figure 5 in Appendix), changing w 0:m would have only a marginal impact on g C (θ 0:m , w 0:m ). Thus, the weighting mechanism that minimizes g KL (θ 0:m , w 0:m ) would minimize the worst-case optimality gap, which justifies our approach of circumventing intractability of ( 4) with (5) . In this regard, AnCon's weighting mechanism in (3) could be thought of as a relaxed solution of (4) as it minimizes g KL (θ 0:m , w 0:m ) in the asymptotic region (cf. Theorem 3.1).We conclude this section by analyzing the three iterative steps where the pseudo labels and the temporal ensembles keep updating. Corollary 3.2.1. Let us assume l(θ) satisfies L-smoothness, L-expected smoothness, and µ-PL condition. For γ ≤ µ 4L•L , λ carefully adjusted at each outer temporal ensemble update (cf. λ = λ † j in Lemma B.6 for each outer iteration j ∈ {0, • • • , T -1}), and any initial parameter θ 0 , it holds thatEQUATIONwhere p( 4 and gives a whole picture of the optimality gap under AnCon. We first remark the trade-off associated with T on the suboptimality E[l(θ T ) -l * ], which characterize the early-learning phenomenon observed in the biased gradient settings (e.g., self-training [31] and LFN [32] ). Specifically, in (9), increasing T reduces the first term (1-µγ) T •( T -1) (l(θ 0 )-l * ) but increases the coefficient of the second term ζ T . Therefore, a longer training with a large T may not enhance the self-training performance especially under a large second term due to inaccurate pseudo labels or the temporal ensemble.I ( T ) = j) ∝ (1 -µγ) T •( T -1-j) for j ∈ {0, • • • , T -1} and ζ T = T -1 i=0 (1 -µγ) T •i . Corollary 3.2.1 is proved in Appendix B.Nevertheless, for each outer loop iteration, g E (θ j ) would be smaller under AnCon than its value under vanilla self-training as E[l(θ j ) -l * ] has a tighter upper bound under AnCon due to Theorem 3.2. Therefore, with the guarantee N (λ † j ; θ 0:j , w 0:j ) ≤ N (0), AnCon would achieve a tighter upper bound of (9) than the vanilla self-training method, enabling longer training with smaller value of the second term as observed in Figure 1b . Finally, we remark that this theoretical superiority of AnCon can be extended to the self-training methods with other weighting mechanisms in the asymptotic region when E ξ [⟨∇ lξ (θ * ), ĝξ (θ 0:m , w 0:m )⟩] ≥ 0 (cf. Theorem 3.2). 
experiments: 
 Goal and baselines Part of the experiments shows that AnCon surpasses the vanilla self-training method that uses Ŷ (x; θ) as a pseudo label, which serves as a strong baseline, under different types of distribution shifts (Sections 4.1-4.2). We also show that AnCon achieves a better performance than ELR with λ * ELR ∈ {1, 3, 7, 12, 25}, which is the coefficient multiplied to the auxiliary loss, to show effectiveness of uncertainty aware temporal ensemble. Finally, we assess the integration of AnCon with generalized cross-entropy (GCE) [33, 31] and neighborhood reciprocity clustering (NRC) [17] , which are frequently cited as state-of-the-art methods [20, 34, 31] . We note that AnCon can be seamlessly applied to GCE and NRC by replacing one-hot pseudo labels by AnCon's regularized pseudo labels Ỹ (X; θ 0:m , w 0:m ). For descriptions of the training configurations which we adopt from literature, see Appendix D.Evaluation In our self-training settings (SFDA and TTA), we should determine the best checkpoint without labeled samples, i.e. to select a model θ 0 , θ 1 , • • • , θ I which is used for evaluation on test. We use information maximization [35] ,IM (θ) = H E (E X [f (X; θ)]) -E X [H E (f (X; θ))]when H E is the entropy, which is proven to be effective in unsupervised domain adaptation (UDA) model selection literature [36, 37] . Specifically, we evaluate IM (θ m ) on the hold-out unlabeled samples at the end of each epoch m for I number of training epochs and select the checkpoint with θ * ∈ arg max m∈[I] IM (θ m ) as the best one. This θ * is used for final evaluation on the test data.Implementation details of AnCon For each (outer) iteration m, we setw j (x) = 1(c(x; θ j ) > δ (β) j ) for j ∈ {0, • • • , m}, instead of w j (x) = 1(c(x;θj )>δ (β) j ) m i=0 1(c(x;θi)>δ (β) i ). Note that the former with λ can be thought of as the latter with instance-dependent coefficient λ • Q(x; c 0:m ), which assigns more weight on the generalized temporal ensemble for x with large Q(x; c 0:m ). In addition, we consider the online update scenario of pseudo labels, i.e., T = 1. Finally, given the challenging nature of hyperparameter selection in self-training under distribution shifts, we perform all experiments by using a single configuration of hyperparameters of AnCon (λ = 0.3 and β = 0.9). These hyperparameters result in the maximum value of max m∈[ T ] IM (θ m ) on the hold-out unlabeled samples on Office-31.We first consider SFDA that adapts a model trained in one domain by performing self-training in the distribution shifted domain. For the network architecture, we use the modified ResNet used in [16] , which includes batch normalization [38] after the bottleneck layer and weight normalization [39] in the last linear layer, which is used in [17, 8] for stabilizing the learning process in SFDA.Datasets We evaluate AnCon on the following datasets: Office-31 [40] with 4,000 images of 31 categories from three domains (amazon, dslr, webcam); OfficeHome [41] with 15,000 images of 65 categories from four domains (art, clipart, product, real-world); VisDa [42] with 280,000 images of 31 categories from two domains (synthetic, real).Results AnCon consistently improves self-training in diverse domain pairs across the three datasets (Table 1 ). Specifically, it reduces the average self-training test error by 5% in Office-31, 6% in OfficeHome, and 13% in VisDa. In addition, compared to ELR with its dataset-dependent optimal hyperparameter value, AnCon shows comparable performance to ELR (2%, 3%, and -3% performance differences in Office-31, OfficeHome, and VisDa, respectively). Despite its slightly inferior performance in VisDa, we note that AnCon achieves significantly better accuracy for low-performing classes: for "Skateboard" and "Truck" classes, AnCon achieves 50% and 16% of accuracies, while ELR achieves 39% and 0.43% in these classes. Thus, AnCon can be as effective as ELR for improving the self-training performances under distribution shifts without needing to adjust hyperparameter values for each dataset unlike ELR. Notably, AnCon significantly improves performances of both GCE and NRC via fundamentally different mechanisms for handling noisy pseudo labels (e.g., reducing test accuracy of 3% and 8% on average in OfficeHome, respectively). Specifically, NRC filters incorrect predictions based on local consistency, while AnCon uses temporal consistency. Combining NRC and AnCon leverages pseudo labels that are both locally and temporally consistent, resulting in significant performance improvements over NRC or Self-Training + AnCon (cf. Table 1 ). In addition, GCE reduces the impact of wrong pseudo labels rather than finding them. Applying GCE to AnCon minimizes the effects of potentially wrong but temporally consistent pseudo labels, which can be implied by the performance of GCE + AnCon compared to GCE or Self-Training + AnCon (cf. Table 1 ). Thus, the impressive performance gains from AnCon, which would be orthogonal to the gains from state-of-the-art methods in SFDA, show its significant practical implications.While we have considered the domain shift, e.g., adaptation of a model trained on synthetic images to real images, in Section 4.1, this section examines the self-training's ability to adapt to distribution shifts by synthetic image corruptions. This setting has been used to measure the robustness of neural networks with respect to a general out-of-distribution setting. To this end, we consider ImageNet-C [6] , which consists of 50,000 images drawn from a validation set of ImageNet [43] where each image is corrupted by 15 types of synthetic corruptions related to noise, blur, weather and digital.Result Consistent with the findings under the domain shift, AnCon outperforms the average performances of self-training and ELR under varying levels of corruption intensities (cf. Figure 1a and Tables 7-11 in Appendix), improving the self-training method's accuracy by 16% on average. Further, the gains from AnCon is significant when the distribution shifts are intense (e.g., improving accuracies by 20% and 52% on average in intensities of 4 and 5) where the initial model trained on the source domain significantly deteriorates. Specifically, for Shot, Impulse, and Gaussian corruptions with the most extreme shift intensity of 5, where the initial model achieves accuracies of (3.04%, 1.76%, 2.12%), AnCon achieves (22.56%, 26.56%, 25.85%) (cf. Table 11 ). This striking improvement compared to vanilla self-training with performances (0.26%, 1.72%, 1.04%) and ELR with performances (8.00%, 14.12%, 16.00%), underscores the importance of the AnCon's uncertainty-aware temporal consistency scheme, as shown in Corollary 3.2.1. We note that this impressive result is also explained by AnCon's ability to prevent the gradual performance degradation during the course of training with the extremely noisy pseudo labels (cf. Figure 1b ). Combined with previous results in domain shift scenarios, we expect that AnCon would work effectively in various out-of-distribution settings.In previous sections, we have shown the universality of AnCon by evaluating it on diverse distribution shift scenarios. In this section, we show versatility of AnCon by analyzing its attractive properties in robustness and uncertainty representation.There is no universally agreed model selection criterion, such as cross-validation in the i.i.d. setting, in self-training under distribution shifts. This is partly due to the variety of distribution shift scenarios, where an effective criterion in one may be ineffective or inapplicable in another; for instance, a principled criteria called importance-weighted cross validation [44] in UDA cannot be applied to SFDA. In this regard, it would be an important characteristic of a self-training method under distribution shift to be robust with respect to different choices of model selection criteria. Therefore, we evaluate robustness with respect to the following different model selection criteria: InfoMax [35] , Corr-C [36] , and Ent [45] (see Appendix D.3 for the description). Figure 1c shows that AnCon's maximum performance change due to different model selection methods is much lower than that of other methods, especially under severe distribution shifts. This valuable advantage can be contributed to the property of AnCon that can prevent performance degeneration (cf. Figure 1b ). Given that, in practice, we barely know when the model collapse happens and which model selection criteria are the best, the results highlight a significant practical value of AnCon.Throughout this paper, we have shown that our single configuration of parameters (λ = 0.3, β = 0.9) work well across a wide range of benchmark problems. In this section, we aim to show our findings can be preserved when the hyperparameter values deviate from the default setting by performing a sensitivity analysis for values λ ∈ {0.1, 0.3, 0.5, 0.7, 0.9} and β ∈ {0.1, 0.3, 0.5, 0.7, 0.9}. We also test two frequently used annealing schedules that λ m = m/I and λ m = min(1, 2m/I), called full and half, respectively. Figure 2 shows that AnCon is stable even under extreme values of hyperparameters. Specifically, for both hyperparameters, the maximum average performance change is less than 1%, and β barely impacts the performance of AnCon. Indeed, our analysis suggests to increase λ from our default setting; that is, to put a higher weight on the general temporal ensemble's prediction. Here, we note that our suboptimal choices of hyperparameters are due to our rigorous and practical hyperparameter choice. Given the challenging nature of hyperparameter optimization under distribution shifts, the stable performances of AnCon under arbitrary choices of hyperparameters would enable AnCon to be seamlessly applied to diverse practical settings. We have shown that all self-training methods significantly improve the performance of the baseline method after the adaptation period. However, it is widely known that these noticeable improvements come with the price of sacrificing an uncertainty representation ability which is critical in real-world decision-making scenarios [46, 47] . Specifically, the calibration performance, which is the gap between the prediction confidence and accuracy, usually monotonically increases as self-training keeps reducing the uncertainty for all predictions during the course of training.In this regard, we analyze the calibration performance with respect to the expected calibration error (ECE; see Appendix for definition). Here, a lower ECE means a lower gap between confidence and accuracy.As shown in Figure 3a and Table 5 in Appendix, AnCon gives much lower ECE compared to other methods. Considering ELR and GCE both have regularization effects, we conjecture that this phenomenon is due to selective regularization in AnCon that increases prediction confidences of samples only if the past confident predictions are consistent with the current prediction. Especially, in Figure 3b which confirms the accuracy-calibration dilemma in VisDa, AnCon is shown to limit the ECE increases during training compared to all other methods. That is, AnCon helps to significantly reduce the price of the calibration performance we need to pay for improving accuracy, which are both important measures in practice.Recall that we define f (x; θ 0:m , w 0:m ) = m i=0 w i (x) • p(y|x, θ i ) with our simple design choices: the relative thresholding for weighting scheme w i (x) ∝ 1(c(x; θ i ) > δ (β) i ) and hard prediction for p(y|x, θ i ). In Appendix C.3, we found that our simple design choices are more appropriate for the distribution shift settings than several more sophisticated alternatives, which can be summarized as follows.• More sophisticated weighting schemes (e.g., Entropy (wi (x) ∝ exp {-H E [f (x; θ i )]}))reduce the self-training performance, despite being a more accurate measure of prediction uncertainty. We conjecture that the poor calibration performance of the neural network in self-training under distribution shifts prevents the sophisticated weighting schemes from accurately reflecting the goodness of the prediction. • Various soft prediction schemes, which can give more information about the non-leading entry values, leads to performance reductions. We conjecture that the continuously increasing confidence in the later stage of self-training would make soft prediction ignore early-stage predictions which may be valuable to memorize. 
related work: 
 Filtering incorrect pseudo labels Popular confidence-based thresholding methods [13, 14, 9 ] fall short under distribution shifts since even high confident predictions can be highly incorrect. Therefore, recent advances in SFDA and TTA utilize higher order information to filter incorrect pseudo labels.For instance, based on the intuition that true labels of adjacent samples would be same, centroids for each predicted class can be maintained in the feature space and then the pseudo label for each input is corrected by the adjacent centroid [16] . The idea of using per-class centroids has been extended to incorporate more general clustering structures [18, 17] . However, the neighborhood structure-based methods are computationally demanding due to storage of memory banks in the feature space and nearest neighbors search. Such computational complexity persists in other approaches, which are based on the consistency of multiple predictions from different augmentations [20] and models trained with different loss functions [19] . Compared to these solutions, AnCon can efficiently estimate correct labels with only limited extra memory overhead of storing past predictions.Learning from noisy labels Treating pseudo labels as inherently noisy, techniques from the LFN literature have been integrated to self-training. For instance, the LFN literature has proposed robust loss functions that reduce impacts of random noisy labels [48, 33] , and a recent large-scale experimental study shows the applicability of the generalized cross-entropy in the SFDA setting [31] .The effectiveness of ELR on SFDA [8] bears a similar idea because ELR was developed to regularize the neural networks' tendencies to memorize incorrect labels [21] . Despite their effectiveness, by nature, these approaches do not consider important characteristics of the unbounded and instancedependent noise rates inherent in self-training under distribution shifts, which results in significant suboptimality in both theory and practice. However, by considering the unique characteristics of self-training under distribution shift, AnCon relaxes the conditions required to achieve optimality as well as boosts the self-training performance in diverse scenarios. 
conclusion: 
 This paper introduces AnCon, which effectively improves self-training performances under diverse distribution shift scenarios by promoting selective temporal consistency based on confident predictions. As a result, AnCon effectively mitigates the detrimental effects of noisy pseudo labels without much computational overhead, unlike the previous methods. We show that AnCon not only advances our theoretical understanding of a generalized notion of temporal consistency in self-training but also can be a practical asset as a simple and effective self-training method with attractive properties. In Appendix C.4, we present limitations and future directions, such as adaptive determination of λ, combining local and temporal consistency, and extending the selective temporal consistency in the sequential decision making problems.  
 
 Appendix: (θ) = E XY [H(f (X; θ), p Y |X )] and l ξ (θ) = 1 b Xi∈ξ H(f (X i ; θ), p Y |Xi )where ξ contains b number of random samples from p X . We also denote θ * ∈ arg min θ∈R p l(θ). In this work, we assume the following three regularity conditions on l(θ) which is differentiable with respect to θ, which are mild but essential for most theoretical studies with convergence analyses.EQUATION)Assumption B.2 (L-expected smoothness [30] ). l(•) is L-smooth in expectation with respect to ξ ∼ D; that is,EQUATION)Assumption B.3 (µ-Polyak-Lojasiewicz (PL) condition [49] ). l(•) satisfies the µ-PL condition for some constant µ > 0; that is,EQUATION)We remark that Assumptions B.1 and B.2 can trivially hold under bounded parameter values that can be guaranteed by optimizing neural networks with finite iterations under a gradient or weight clipping. Assumption B.3 holds for infinite-width neural networks, i.e., the neural tangent kernel (NTK) regime [50] . Given that the gradient descent training dynamics of neural networks can be well approximated by NTK [51] , the PL condition can be generally regarded as a mild assumption.B.2 Proof of Theorem 3.1Proof. For x such that Q(x; c 0:m ) > 0, let us consider subsequence of index l such that x ∈ A l (c l ); that is, (j 1 , • • • , j Q(x;c0:m) ) where j l = min{l ≥ j l-1 |x ∈ A l (c l )} and j 0 = 0. Let S Q(x;c0:m) = Q(x;c0:m) i=1 1(Y (x) = Ŷ (x; θ ji )). Then, we have the following inequalityEQUATIONwhere the first inequality holds because arg max k fk (x; θ 0:m , w 0:m ) = Y (x) if S Q(x;c0:m) > Q(x;EQUATION)Then, from the definition N (λ;θ 0:m , w 0:m ) = λ 2 ∥ ĝ(θ 0:m , w 0:m ) ∥ 2 + Lγ 2 E ξ ∥ ∇ lξ (θ * ) -λĝ ξ (θ 0:m , w 0:m ) ∥ 2, we get (7) . To show N (λ † m ; θ 0:m , w 0:m ) ≤ N (0), we apply Lemma B.6 asEQUATION)To show ( 8), let us assume E ξ [⟨∇ lξ (θ * ), ĝξ (θ 0:m , w 0:m )⟩] ≥ 0. Then, we get the following inequality:EQUATIONEQUATIONwhere (18) holds due toa 2 + b 2 ≥ 2ab; (19) holds due to the assumption ∥ x ∥≤ C, ∥ • ∥ 2 ≤∥ • ∥ 1 ,and the Pinsker's inequality applied asEQUATIONThus, we get our desired result by combining the following inequality with (15) :EQUATION≤ 2 min 1,EQUATIONEQUATIONwhere ( 24) holds due to 1EQUATIONThus, by recursively applying (27) to θ 0 , we get our desired result asEQUATIONB.5 Supporting lemmas Lemma B.4.Let A i ∼ Bern(p i ) with p i ∈ [0, 1] and S o = i∈[o] A i, where A i and A j are independent for i ̸ = j. Then for any q < p := 1 o i∈[o] p i , we have the tail probability boundEQUATIONProof. For any t, the moment generating function of S m is given byEQUATIONwhere the first equality comes from the independence and the inequality is from log x ≤ x -1.Then, for t < 0 and q < p, the Chernoff bound [52] givesEQUATIONFinally, setting t = log (q/p) < 0 gives our desired result.Lemma B.5 (Modification from [30] ). Let us assume that l(θ) satisfies the L-smoothness, the L-expected smoothness, and the µ-PL condition. Also, we assume a linear modelf k (x; θ) = {exp(θ T x)} k / i∈[K] {exp(θ T x)} i with θ ∈ R d×K .Then, for the stochastic gradient descentEQUATION)Proof. For any realization of θ m,t , we have the following bound:EQUATIONEQUATIONEQUATIONEQUATIONEQUATIONEQUATIONEQUATIONwhere (33) holds due to L-smoothness; (34) holds due to g(m,t) ξ = ∇l ξ (θ m,t )-b (m,t) ξ = ∇ lξ (θ m,t )- λĝ ξ (θ 0:m , w 0:m ); (36) holds due to ⟨a, b⟩ ≤ 1 4 ∥ a ∥ 2 + ∥ b ∥ 2 ;(37) holds due to the inequality ∥ x + y ∥ 2 ≤ 2 ∥ x ∥ 2 +2 ∥ y ∥ 2 ; (38) holds due to the expected smoothness assumption; (39) holds due to the condition γ ≤ 1 4L µ L ; (41) holds due to the law of total expectation with the random event 1(Y (X) = Ŷ (X; θ m )) and then applying the bounded support assumption.By recursively applying the above bound to any realization of θ m , we get our desired result as follows:EQUATION)Lemma B.6 (Lemma 1 in [30] ). Let N (λ; θ 0:m , w 0:m ) := λ 2 ∥ ĝ(θ 0:m , w 0:m ) ∥ 2+ Lγ 2 E ξ ∥ ∇ lξ (θ * ) -λĝ ξ (θ 0:m , w 0:m ) ∥ 2 . Then, for λ † m = E ξ [⟨∇ lξ (θ * ),ĝ ξ (θ0:m,w0:m)⟩] E ξ ∥ĝ ξ (θ0:m,w0:m)∥ 2 + 2Lγ ∥ĝ(θ0:m,w0:m)∥ 2 , we haveEQUATION)C Additional results and discussions Here, we examine effectiveness of our design choices-the relative thresholding for weighting scheme w i (x) and hard prediction for p(y|x, θ i ). To this end, we replace our design choices with several alternatives, keeping other features of AnCon the same. Then, we analyze the changes in the performance compared to the default setting of AnCon in four domain pairs of OfficeHome.Sophisticated weighting scheme for w i (x) We compare our relative thresholding with sophisticated weighting schemes called Entropy (w i (x) ∝ exp {-H E [f (x; θ i )]}) and Maxprob (w i (x) ∝ max k∈[K] f k (x; θ i )). These weighting schemes aim to more precisely weight the predictions based on the uncertainty of the prediction. However, in Figure 6a , the simple relative thresholding works better than these sophisticated weighting schemes. We conjecture that the poor calibration performance of the neural network in self-training under distribution shifts prevents the estimated uncertainty of a prediction from accurately reflecting the goodness of the prediction. Further, the ineffectiveness of these sophisticated weighting schemes persists even when we improve the calibration performance with last-layer Dirichlet [53] and Monte-Carlo Dropout [54] (see Appendix D.3 for detail). Therefore, our relative thresholding is simple yet effective in self-training under distribution shift.Soft prediction for p(y|x, θ i ) We compare hard prediction vs soft prediction with a softmax temperature parameter T ∈ {0.5, 0.75, 1.0, 1.25, 1.5} (cf. Appendix D.3 for the definition). By design, soft prediction is more informative, giving values in non-leading entries. However, in Figure 6b , soft prediction turns out to underperform hard prediction for various values of T . We conjecture that the continuously increasing confidence in the later stage of self-training would make soft prediction ignore early-stage predictions which may be valuable to memorize. Thus, we believe that hard prediction would be more appropriate for AnCon.Albeit AnCon proving its effectiveness with a fixed value of λ, Theorem 3.1 suggests that its optimal value depends on the temporal ensemble performance. Thus, future works could aim for adaptively determining λ. In addition, we observe that all self-training based methods fall short without sophisticated model designs tailored for distribution shifts [16, 31] (cf. Appendix E). In this regard, it would be valuable to rigorously understand properties of function classes that enable successful self-training under distribution shifts. Also, while AnCon significantly relaxes the conditions required to achieve optimality compared to LFN techniques, the on-average correct prediction condition still can be violated in challenging self-training scenarios. Given the compatibility of AnCon and NRC (cf. Table 1 ), efficiently combining local and temporal consistencies could be a step toward further relaxing the optimality condition. Finally, we remark that the idea of selective temporal consistency in sequential decision-making scenarios could be a notable future direction of research. Extending AnCon to sequential decision-making scenarios presents significant challenges as they involve the fundamentally different mechanisms. In sequential decision-making, leveraging observed rewards and balancing exploitation and exploration are core aspects (e.g., constructing the upper confidence bound of the reward in the bandit problem), unlike in self-training. Therefore, the main challenges for applying AnCon to this setting would be defining rewards and incorporating exploration strategies into pseudo label generation.Our training configuration is based on [16] . Specifically, we train ResNet-50 [29] as a source classifier, which is used as an initial point for SFDA, for 50 epochs, with minibatch stochastic gradient descent with Nesterov momentum by using the initial learning rate of 0.01, the momentum parameter of 0.9, and the minibatch size of 64 in the Office Home dataset. The learning rate is decayed by (1 + 10 • current iteration number / maximum number of iterations) -0.75 , and for the pre-trained layers we use a 10 times smaller learning rate. For Office-31, we change the number of epochs by 100 and other configurations are the same as those used in the setting in Office Home. Similarly, we only change the number of epochs to 10, the learning rate to 0.001, and the architecture to ResNet-101 for VisDa. We split dataset into 90% of the training set and 10% of the validation set, and the best model is chosen based on the lowest error rate on the validation set. For adapting the pre-trained model in the target domain, we use the exactly same configurations, except reducing the number of epochs to 30 for Office-31 and OfficeHome and to 15 for VisDa.For applying self-training, we follow the training configuration used in [31] ; specifically, given ResNet-50 pretrained on ImageNet, we perform self-training without threshold by running the stochastic gradient descent optimizer with the learning rate of 0.001 and the minibatch size of 100 for 20 epochs. We remark that other techniques such as momentum and learning rate scheduling are not used.Definition of ECEEQUATIONwhereG θ g := {x ∈ D| g G ≤ c(x; θ) < g+1 G }, Acc(G θ g ) is the average accuracy in G θ g , and Conf (G θ g ) is the average confidence in G θ g .Configuration of the sophisticated uncertainty representation methods We used the regularization coefficient of 0.001 for the belief matching loss [53] and 20 forward passes to compute the posterior predictive distribution for MC-dropout with dropout probabilities 0.2 and 0.5 [54] .• InfoMax (higher is better)EQUATIONwhere H(p) = -k∈[K] p i log p i for p ∈ △ K-1 . Note that we abuse the notation H(•) with the definition of the cross-entropy; that is, H(q, p) = -k∈[K] p i log q i and H(p) = -k∈[K] p i log p i .• Corr-C (lower is better)EQUATIONwhereEQUATIONSoftmax with a temperature parameterEQUATIONwhere f -ϕ (x; θ) are the logits of the neural network such that fi (x; θ) = exp(f -ϕ i (x;θ)) k∈[K] exp(f -ϕ k (x;θ)) for i ∈ [K] and ϕ(x) is the softmax function.In this work, we use multiple servers which consist of multiple GPUs including TITAN XP (12GB), RTX 8000 (50GB), and A100 (40GB). AnCon takes 1.16 seconds on average for each iteration.While SFDA does not assume any information, unlabeled target domain samples as domain shift information can be available during the source domain training process in some practical scenarios, e.g., when labeling target domain samples is costly. In this case, UDA methods are used to minimize the potential impact of distribution shifts by matching input marginal probability distributions of source and target domains. Given the prevalence of UDA scenarios in practice, we aim to show whether AnCon and other self-training methods can be effective for the UDA methods (conditional domain adversarial network (CDAN) [55] and maximum mean discrepancy (MDD) [56] ) in the adaptation stage. However, as shown in Table 2 , all methods decrease the performance of both CDAN and MDD for most cases, albeit the reduction rates in AnCon are smaller than self-training and ELR. We conjecture that this is because the base methods (CDAN and MDD) do not contain the sophisticated tricks, such as adding weight normalization to the final linear layer, which are used in the SFDA literature [16] .  


Decision: The paper introduces a novel self-training method (AnCon) to improve test-time adaptation (TTA) and source-free domain adaptation (SFDA) under distribution shifts. It proposes using uncertainty-aware temporal ensembles and label smoothing to enhance pseudo-label quality during self-training. Theoretical guarantees are provided for the method's asymptotic correctness and reduced optimality gap. Extensive experiments demonstrate AnCon's effectiveness in various distribution shift scenarios.

The paper provides a well-founded and efficient approach to improving self-training under distribution shifts, with solid theoretical support and strong empirical results. Although its novelty is incremental, the method's practicality and demonstrated effectiveness across diverse settings make it a valuable contribution to the field. The thorough rebuttals addressed reviewers' concerns. As such, the AC recommends acceptance to NeurIPS.

Reviews: review: paper_summary: paper_summary: the authors propose a novel approach to enhance self-training for test-time adaptation (TTA) or source-free domain adaptation (SFDA) in neural networks facing distribution shifts. The core idea revolves around a method called Anchored Confidence (AnCon), which uses temporal ensembles and label smoothing to improve the accuracy and robustness of pseudo labels generated during self-training. This method aims to address the challenge of filtering incorrect pseudo labels, a common issue under distribution shifts, without incurring significant computational overhead. Theoretical guarantees and extensive experiments validate the efficacy of AnCon in diverse distribution shift scenarios.
strength_weakness: strength_weakness: * The novelty of the paper is limited! (*The integration of temporal consistency and ensembles for self-training under distribution shifts)
* Although the method shows good empirical results on small datasets, there may be scenarios or datasets where the performance gains are less pronounced, which are not extensively discussed.
* The success of the method is somewhat dependent on the quality of the initial model parameter $\theta_0$. If the initial model is significantly biased or underperforming, the improvements might be limited.
questions: questions: * The paper claims that the generalized temporal ensemble with prediction confidences is asymptotically correct. Can you provide a detailed proof or further elaboration on the conditions under which this asymptotic correctness holds? Specifically, what assumptions about the data distribution or model convergence are necessary to guarantee this property?

* The theoretical analysis suggests that label smoothing can reduce the optimality gap in self-training. Could you explain the underlying mechanism of how label smoothing achieves this reduction? Additionally, what are the theoretical bounds on the optimality gap with and without label smoothing, and how do these bounds depend on the hyperparameters of the smoothing technique?

* How does AnCon perform on highly imbalanced datasets or with extreme distribution shifts?
* What are the specific hyperparameter settings used in the experiments, and how sensitive is the method to these settings?
* Can the proposed method be integrated with other advanced TTA or SFDA techniques, and what would be the potential benefits or drawbacks?
* How does the method handle concept shifts, where p(Y|X) changes between the training and test distributions?
* are there any specific types of neural network architectures or tasks (e.g., vision, NLP) where AnCon performs exceptionally well or poorly?
limitations: limitations: Please see the questions and the weaknesses.  
My current decision on this paper is borderline (reject/accept). I look forward to the authors' rebuttal to address the following questions and concerns, which will help in making a final decision.
paper_summary: paper_summary: The paper aims to improve self-training, a common technique used for learning on unlabeled data (with iteratively generated pseudo labels), when it is applied to scenarios with distribution shifts. The proposed approach, Anchored Confidence (AnCon), essentially applies label smoothing on the pseudo labels with the ensemble prediction of the models obtained from previous iterations (temporal ensemble) to achieve better temporal consistency and improve the quality of the pseudo labels. Theoretical analysis is provided on the benefits of using temporal ensemble and the connection to ensemble knowledge distillation in the linear case. Empirically, when AnCon is shown both **effective** and **stable** when combined with a few orthogonal baselines such as Self-training and GCE under settings such as unsupervised domain adaptation (Sec. 4.1), robustness against label corruption (Sec. 4.2), calibration, etc. (Sec. 4.3).
strength_weakness: strength_weakness: 1. Empirically, the approach improves over various baselines when being added, but what about the existing SoTA baselines for unsupervised domain adaptations, which seem relevant to this work. Besides, it seems rather odd to only report the results in Sec. 4 with non-optimal $\lambda=0.3$, as the sensitivity analysis in Figure 2(a) clearly prefers much larger $\lambda$. 
2. A critical assumption in the theorems is that the teacher (temporal ensemble) has a sufficiently good performance. It will be helpful to justify this with more empirical evidence. 
3. Non-trivial typos: In line 149, $x$ is already used in the main theorem, so may want to change $x$ to $z$ to avoid confusion, and this issue also occurred on 643 with $m$, which might have already caused *major* problems in proof (see below in Question 3)
4. Minor typos: Line 667, Proof of Lemma A.4, Eq. (31), $k$ should be written as $mq$; Some references from NeurIPS 2024 should be 2023. 
5. There might be some mistakes in the theorem proofs. See below in Questions 3 and 4.
questions: questions: 1. The pseudo label $\tilde{Y}$ is actually also part of the smoothing vector from $\bar{f}$. I wonder how necessary it is to separate $\tilde{Y}$ out as its purpose is to increase the weight of the most recent model, or whether there is a more unified notion to describe the proposed approach with just $\bar{f}$. 
2. In Theorem 3.1, the random events of model predictions from different iterations $i, j$ are assumed to be independent. How realistic is the assumption given that the model is trained based on the labels given by the previous iterations?
3. Throughout the paper, $m$ is already defined as the number of models, but the proof of Theorem 3.1 assigns $m = Q(x;c_{0;m})/2$, which is the number of sufficiently confident predictions for $x$ made by the $m$ models. I don’t think it’s appropriate to assign $m$ something else here as it is already defined. If you fix the notation of $m$ (of $q \cdot m$) in Eq. (29) and Eq. (31), Lemma A.4, the bound may look a bit more complex, because the $m$ in Eq. (30) cannot be assigned to a different value. Please clarify. 
4. Moreover, in line 152, the paper claims $\bar{f}$ is asymptotically correct, but it is unclear to me if the probability on the L.H.S. of Eq. (6) still goes to 0 when $m \rightarrow \infty$ after fixing the aforementioned problem with $m$. Please clarify. 
5. Theorem 3.1 studies the asymptotic correctness of the temporal ensemble with large $m$. The experiments mostly show the AnCon under 20 epochs. How does the performance change when more epochs are added? Is it still consistent with the theoretical implications?
limitations: limitations: The authors have discussed the limitations in Section 6.
paper_summary: paper_summary: This paper presents Anchored confidence (AnCon), a novel self-training algorithm to improve test-time accuracy under distribution shifts. AnCon modifies the standard self-training algorithm by adding a temporal ensemble regularization. This regularizer is constructed by the consistency of temporal ensembles weighting with their predictive confidences. This intuitively helps the model avoid the "early-learning phenomenon", i.e., the model exploiting noisy labels after initially learning clean labels. Contributions include:
1. A new algorithm for memorizing past ensemble predictions to avoid the "early-learning phenomenon", enhancing self-training performance under sequential distribution shifts.

2. In terms of theory, the authors provide a rigorous theoretical analysis for the upper bound of the test-time error of the proposed temporal ensembles by concentration inequality in Theorem 3.1. Inspired by Knowledge Distillation (KD), they also design an upper bound of the expected cross-entropy loss between the temporal ensembles and the optimal value in Theorem 3.2.

3. Extensive experiments confirm the hypothesis that the proposed algorithm can improve self-training performance regarding test-time accuracy and calibration under distribution shifts across different hyperparameter choices.
strength_weakness: strength_weakness: - Regarding the proposed algorithm: compared with the standard self-training, AnCon requires pre-defining the regularizer $\lambda$ and the discount factor for the temporal ensembles $\beta$. 
 - Regarding the theory: Theorem 3.1 requires $\bar{p}(x;\mathbf{c}_{0:m}) \geq 1/2$, i.e., at least 50% accuracy on average for the temporal ensembles. This is a quite strong assumption.
 - Regarding the experiments: (1) The quantitative results in all tables are reported without the standard deviation and significant tests; (2) The improvement in calibration is shown without any theoretical evidence. Fig. 3 (a) only shows ECE without the reliability diagram, making it hard to assess the correlation between the accuracy and uncertainty of softmax’s probability; (3) Lacks of comparison with other baselines, e.g., many baselines [1] in UDA could be added to compare in Appendix C.
  - Others that are mentioned by the authors: the challenge of self-training methods under distribution shifts and the violation of average correct predictions in self-training.
- Miscellaneous: (1) Although assumptions A.1, A.2, and A.3 are mild and have been cited, I believe it is still worth discussing how they are realistic with the paper's setting; (2) Some mathematical notations are vague and perhaps need to be polished (e.g., $f(x;\theta)$ or $f_k(x;\theta)$ in L-78, how $\bar{f}(\cdot)$ in Eq. 1 connect to the Def. $\bar{f_k}(\cdot)$ in Eq. 2, $\hat{\mathbb{E}}$ in Eq. 3 seems not to be defined, the same notation $\sigma$ for different $\sigma_*^2$ in Theorem 3.2 and perhaps the softmax $\sigma_k(x;\theta_i)$ in L-339, more in Question 2); (3) I find it quite hard to follow acronyms without links, I would recommend using \usepackage[acronym]{glossaries} so readers can smoothly follow the paper; (4) Some Figure’s x-y-titles are tiny, e.g., Fig. 3 (b).
questions: questions: 1. The connection of AnCon with KD is not clear to me. The pseudo label $\hat{Y}(x;\theta)$ in self-training with AnCon and the true label  $Y(X)$ in L-163 ($Y(x)$ in L-74) are completely different because $Y(X)$ is unobservable while $\hat{Y}(x;\theta)$ is model’s prediction. So in Section 3.3, the statement in L-165-166 to connect AnCon with KD is still wrapping my head. In particular, why AnCon's pseudo-label can be considered as $Y(x)$ (perhaps the true label $Y(X)$ in L-163)? Also, I am curious about why Eq. (40) can lead to Eq. (41), i.e., how we can bound the norm of gradient loss $||\nabla l(\theta_{m,t}) - \nabla \tilde{l}(\theta_m)||^2$ by the $Err(\hat{Y};\theta_{m})$ while this upper bound related the true label $Y(X)$?
2. Some questions for clarification: is $\delta$ in L-83 $\in [0,1]$? $\sum_0^m$ or $\sum_0^{m-1}$ in Eq. 3? $\theta_k$ or $\theta_m$ in $Err_{XY}(\cdot)$ in L-184? $\mathbf{w}\_{1:m}$ or $\mathbf{w}_{0:m}$ in RHS Eq. in L-186?
3. Compared with ELR which computes the temporal ensembles by the logit vector $f(x;\theta_i)$ only in L-83, the proposed algorithm requires computing $\mathbb{E}\_{x\in X}\max_{k\in [K]}f_k(x;\theta_i)$ in Eq. 3, does this causes a higher computational demand?
4. Can you generate a reliability diagram [2] to show the overconfidence and underconfidence of methods? I think this could be easier to assess how your method can improve the model's uncertainty quality.
6. In the testing of different weighting schemes in Fig. 4 at Section 4.4, have you considered the setting of a uniform weighting for $\mathbf{w}$, i.e., set Eq. 2 to $\bar{f_k}(x;\theta_{0:m},\mathbb{w}_{0:m}):= \sum_i^m p(y=k|x;\theta_i)$? This is a simple setting but no longer temporal ensembles by discount factors, and also not standard (vanilla) self-training. I am quite curious about this result.
5. Do the authors think your proposed method can work in other sequential decision-making, e.g., bandits, RL, bayesian optimization, active learning, etc.? If yes, could you please give some comments? Otherwise, could you please raise some challenges?

References:

[1] Fang et al., Source-Free Unsupervised Domain Adaptation: A Survey, arXiv, 2023.

[2] Guo et al., On Calibration of Modern Neural Networks, ICML, 2017.
limitations: limitations: Please see my comments on the Weaknesses.

Comments: title: Official Comment by Reviewer fUqB
Reviewer Response
Further clarification
Reviewer Response
Responses to questions
Detailed changes as per the response to W2
Responses to Q5 and Q6
comment: Thank you for your careful review and for taking the time to consider our rebuttal and the feedback from other reviewers. We greatly appreciate your acknowledgement of our efforts to clarify W1 and address your questions. We will ensure that the insights from these valuable discussions are thoroughly incorporated into the final version of the paper.

Thank you once again for your thoughtful evaluation and support!
I have carefully reviewed the feedback from other reviewers, considered the author’s rebuttal, and followed the ensuing discussion. I appreciate the authors' thorough responses, particularly their clarifications on W1 and answering my questions. 

Assuming that the insights from these discussions will be included in the final paper, I recommend the paper for acceptance as it provides interesting insights and has the potential to contribute to the ML community and I will raise my score from 5 to 7.
Thank you for your thoughtful review and for acknowledging the clarifications and new results we provided. We fully understand the importance of clear notation, which we have significantly improved thanks to the reviewers’ suggestions. We will carefully revise the notation and correct any typos in the final version of the paper. Thank you again for your insightful comments throughout the review process!
I thank the authors for the detailed rebuttal, especially for clarification for Q1 and new results in Q4-5. I keep my original rating for this paper. Since other reviewers also need clarification on the typos, I hope the authors can carefully revise the notation in the final version of the paper. Good luck!
Thank you for taking the time to revisit our explanation. We're glad that the clarification resolved the initial confusion regarding the notation. Also, we appreciate your positive evaluation of our work and your consideration of the theoretical analyses and empirical results. Your insights have been so much helpful in improving the clarity and rigor of our paper!
Thanks for the clarification. I apologize for mistakenly thinking $m$ for $S_m$ should not be updated to $o$. Now everything is clear! The other two reviewers are also generally positive about the contribution of this work, with minor similar concerns about the assumption of $\bar{p}(x;c_{0:m}) \geq 1/2$. Considering the soundness of theoretical analysis and the empirical advantages, I believe this paper will greatly contribute to the research on self-training and test-time adaptation. I have raised my score to 6 for acceptance :)
Thanks for reading the authors' rebuttal and asking a further clarifying question. 

The confusion of Reviewer qtUp is noted. In our changed notation, all $ m $ in Lemma A.4. are changed to $ o $, which is the number of random Bernoulli samples. We remark that Lemma A.4 is a supporting lemma, which means that the semantics of notations are independent of other parts of the paper (i.e., none of the quantities in Lemma A.4, including Eq. (30), is interpreted as the number of models). 


Therefore, all $ m $ notations in Lemma A.4 are changed to $ o $. Then, the right-hand side of Eq. (31) is $ o q - o \bar{p} - o q \log \frac{q}{\bar{p}} $, which is obtained by $M(t) = exp( o \bar{p} (exp(t) - 1))$ and $ t = log \frac{q}{\bar{p}} $. 


From the above, Eq. (29) is $ p(S_o \leq q o) \leq exp(o (q - \bar{p} - q \log \frac{q}{\bar{p}}) ) $. Therefore, applying this inequality to Theorem 3.1 with $ p_i  = p(Y(x) = \hat{Y}(x; \theta_i)) $, $ o = Q(x; c_{0:m}) $, and $ q = 1 / 2 $ proves the last inequality in Eq. (11). 

We hope this clarifies why there is no mistake in the proof and that the confusion arose from the same notation used in the supporting lemma and other parts of the paper. We sincerely appreciate Reviewer qtUp's thorough review and attention to detail.
Thank you for the responses for addressing my concerns including the asymptotic convergence w.r.t. $Q$ and the choice of hyperparameters. 

For W3 and Q3: In Eq. (30), $m$ is the number of models. In Eq. (31), you set $t = \log \frac{k}{m\bar{p}}$, is this $m$ the number of model or the new $o$? Let’s suppose $m$ is changed to $o$, such that $t=\log \frac{q}{\bar{p}}$ here, then the bound in Eq. (31) may look like $\exp(oq-m\bar{p}-oq\log\frac{q}{\bar{p}})$ with both $m$ and $o$ inside. I think this should affect the bound in Theorem 3.1 as the number of models $m$ would appear. It’s better for the authors to show a more complete derivation after fixing the notations.
**Q1-1:** We inadvertently caused confusion to Reviewer hP9c on the connection between AnCon and knowledge distillation (KD). To be clear, we do not intend to treat the pseudo label $ \hat{Y} $ as the true label $ Y $ since this reduces the theoretical rigor of our work. Indeed, we clearly state the bias coming from pseudo labels in the optimality gap in Theorem 3.2. Also, we state that “the gradient is biased due to the usage of pseudo labels and the generalized temporal ensemble” in Line 168-169. The purpose of this connection is to show how AnCon can reduce the variance of stochastic gradients in the self-training scenario through the partial variance reduction theory by interpreting our generalized temporal ensemble as the teacher network. In order to avoid any confusion, we have changed Lines 165-169 as follows:

“””
… , where the generalized temporal ensemble is the teacher network $ f^{(t)} $ with a notable difference that the pseudo label $ \hat{Y} $ is used instead of the true label $ Y $ which requires careful analysis for studying the optimality. The purpose of this connection is to perform a convergence analysis of AnCon by modifying the partial variance reduction theory [31], as given below. We note that the usage of pseudo labels, instead of the true label, and the generalized temporal ensemble result in an inherently biased gradient estimator. Therefore, the convergence analysis requires special treatments for handling the biased gradient, unlike typical supervised learning settings in self-distillation literature.   
“””


**Q1-2:** It can be derived by using the law of total expectation with the random event $ 1(Y(X) = \hat{Y}(X)) $ and then applying the bounded support assumption, which we have added to Line 678. 


**Q2-1:** If this refers to Line 125 in Eq 3, yes due to the recursion. 

**Q2-2:** It is \sum_{0}^{m}, which means the current average confidence is taken into account because the exponential moving average is introduced to avoid the computation of the expectation with respect to all samples. 

**Q2-3:** Thank you so much for your careful review and pointing out the type. It is $\theta_m$ because $Err_{XY}(\hat{Y}; \theta_{m,0})$ is an error rate of pseudo label under parameter $\theta_m$. 

**Q2-4:** Thanks again for pointing out the typo. This is $ w_{0:m} $ because the generalized temporal ensemble includes the initial model prediction. (After very careful review, we have found other typos in Line 178, Line 179, Line 674, which we corrected).



**Q3:** AnCon has almost the same computational complexity as ELR, and the confusion of Reviewer hP9c would be due to our undefined notation of $ \hat{E}[c(X; \theta_i)] $ which is a Monte-Carlo approximation of $ E[c(X; \theta_i)] $ with mini-batch samples $ \xi $. Specifically, Eq (2) for storing previous predictions is the same as with storing the previous logit vector in ELR ($ K \times N $ numbers). Also, Eq (3) can be efficiently implemented by using softmax output values during the training process which are already computed for computing the self-training loss. Thus, Eq. (3) requires storing only a single number with the computational cost of adding $ B $ numbers. In order to prevent the confusion, we have added the above discussion in Line 131. 


**Q4:** Kindly see the response to the weakness. 


**Q5:** As per Reviewer’s insightful comment, we have tested the undiscounted vanilla temporal ensemble (UVTE) method, which results in 4.2\% of reduction of accuracy on average. Intuitively, UVTE does not work well because memorizing all predictions, which involve highly inaccurate ones, can result in a low-quality temporal ensemble. Indeed, this is well illustrated in Figure R3 where the UVTE would not take advantage of gathering more samples due to inclusion of more incorrect predictions compared to AnCon. We believe this new result further highlights the importance of uncertainty awareness for the temporal ensemble in self-training, which is consistent with our theoretical results in Theorems 3.1 and 3.2.


**Q6:** Thank you for the insightful question that made us think about the non-trivial extension of AnCon. Extending AnCon to sequential decision-making scenarios presents significant challenges as they involve the fundamentally different mechanisms. In sequential decision-making, leveraging observed rewards and balancing exploitation and exploration are core aspects (e.g., constructing the upper confidence bound of the reward in the bandit problem), unlike in self-training. Therefore, the main challenges for applying AnCon to this setting would be defining rewards and incorporating exploration strategies into pseudo label generation. As it holds great significance, we mark this extension as an important future direction.
“””
Assume $ l(\theta) $ satisfies $ L $-smoothness, $ \mathcal{L} $-expected smoothness, and $\mu$-Polyak-Lojasiewicz (PL) condition (cf. Assumptions A.1, A.2, and A.3). 
For $\gamma \leq \tfrac{\mu}{4 \mathcal{L} \cdot L}$ and a carefully chosen $\lambda$ (cf. Lemma A.6), it holds that 
			$$ Eq. (12) $$ 

Further, if $\mathbf{w}_{0:m}$ such that {...}, i.e., the teacher has a sufficiently good performance, the optimality gap becomes

$$ Eq. (7) $$

where $l^*$ {...} . 

“”” (here we could only refer equation number and use {...} to represent the repetitive equations since the MathJax cannot display the equations properly.)


Modified discussion in Lines 187-190:
“””
In Theorem 3.2, we remark that the vanilla self-training ($ \lambda=0 $) results in the maximum value of the last term in Eq. (12), which is the neighborhood size of the stochastic gradient descent, while having the same first two terms. In other words, the result suggests that AnCon is at least better than vanilla self-training under the mild regularity conditions. Furthermore, under the assumption of a sufficiently good performance temporal ensemble, we gain valuable insights on the design of $ w_{0:m} $: aiming for …
“””
**Q5:** We kindly refer to “Section 4.1” for the integration of AnCon with state-of-the-art SFDA technique (NRC) and TTA technique (GCE). We also kindly refer to [G5 in the global response; GLOBAL SOTA] for conceptualizing the state-of-the-art performance levels of considered baselines. 
Regarding potential benefits, AnCon differs from state-of-the-art methods in handling noisy pseudo labels under distribution shifts. Specifically, NRC filters incorrect predictions based on local consistency, while AnCon uses temporal consistency. Combining NRC and AnCon leverages pseudo labels that are both locally and temporally consistent, resulting in significant performance improvements over NRC or Self-Training + AnCon (cf. Table 1). 
In addition, GCE reduces the impact of wrong pseudo labels rather than finding them. Applying GCE to AnCon minimizes the effects of potentially wrong but temporally consistent pseudo labels, which can be implied by the performance of GCE + AnCon compared to GCE or Self-Training + AnCon (see Table 1). 
In summary, AnCon can complement existing state-of-the-art methods by handling noisy pseudo labels fundamentally differently.


**Q6:** We kindly remind Reviewer fUqB that the concept shift is out of the scope of our work as stated in Line 71. Also, we note that addressing concept shifts in the self-training scenario without labels is questionable, as even detecting such shifts without feedback on predictions through shifted labels in a principled manner is impossible (e.g., Lu et al., 2018). 

- Lu et al. (2018). Learning under concept drift: A review. IEEE T-KDE.

**Q7:** We appreciate the curiosity of Reviewer fUqB about extending AnCon to various neural network architectures and tasks. Indeed, our theory and algorithm for AnCon are broadly applicable as AnCon is independent of specific neural network architectures or dataset structures, unlike methods that rely on batch normalization statistics or random image augmentations. At the same time, being agnostic to architectures and data structures make it inherently hard to make rigorous theoretical statements on the extensibility. Therefore, affirmative conclusions about the extension of AnCon would require large-scale comprehensive experiments with careful design of explorative strategies, which is beyond the scope of this paper.
Moreover, even if AnCon performs suboptimally in certain architectures or tasks, this does not invalidate (1) our theoretical contribution of showing how uncertainty-aware temporal consistency improves self-training under distribution shifts and (2) our algorithmic contribution of designing an effective temporal ensemble mechanism without necessitating computationally heavy processes such as additional forward passes or neighborhood searches. 
In summary, while we acknowledge the potential benefits of further explorations, we believe our current focus provides a solid foundation for understanding AnCon's impact and effectiveness.