Title: Improving self-training under distribution shifts via anchored confidence with theoretical guarantees
Venue: NeurIPS 2024
Accepted: yes
Keywords: self-training

distribution shifts

source-free domain adaptation

test-time adaption
Abstract: Self-training often falls short under distribution shifts due to an increased discrepancy between prediction confidence and actual accuracy. This typically necessitates computationally demanding methods such as neighborhood or ensemble-based label corrections. Drawing inspiration from insights on early learning regularization, we develop a principled method to improve self-training under distribution shifts based on temporal consistency. Specifically, we build an uncertainty-aware temporal ensemble with a simple relative thresholding. Then, this ensemble smooths noisy pseudo labels to promote selective temporal consistency. We show that our temporal ensemble is asymptotically correct and our label smoothing technique can reduce the optimality gap of self-training. Our extensive experiments validate that our approach consistently improves self-training performances by 8% to 16% across diverse distribution shift scenarios without a computational overhead. Besides, our method exhibits attractive properties, such as improved calibration performance and robustness to different hyperparameter choices.

Improving self-training under distribution shifts via anchored confidence with theoretical guarantees

Abstract

Self-training often falls short under distribution shifts due to an increased discrepancy between prediction confidence and actual accuracy. This typically necessitates computationally demanding methods such as neighborhood or ensemble-based label corrections. Drawing inspiration from insights on early learning regularization, we develop a principled method to improve self-training under distribution shifts based on temporal consistency. Specifically, we build an uncertainty-aware temporal ensemble with a simple relative thresholding. Then, this ensemble smooths noisy pseudo labels to promote selective temporal consistency. We show that our temporal ensemble is asymptotically correct and our label smoothing technique can reduce the optimality gap of self-training. Our extensive experiments validate that our approach consistently improves self-training performances by 8% to 16% across diverse distribution shift scenarios without a computational overhead. Besides, our method exhibits attractive properties, such as improved calibration performance and robustness to different hyperparameter choices.

1 Introduction

In this work, we address the challenge of adapting pre-trained neural networks at test time under distribution shifts, a problem known as test-time adaptation (TTA) or source-free domain adaptation (SFDA). Distribution shifts-where a model trained on one distribution is then tested on a different one-are ubiquitous in many practical scenarios due to demographic subpopulation shift [1] and changes in data collection environments [2-4]. Despite the robust performance of neural networks under independent and identically distributed (i.i.d.) settings, they often suffer from substantial performance degradation under such shifts [5, 6]. Recently, TTA and SFDA have proven their effectiveness in resolving these critical issues by effectively leveraging information about the distribution shifts contained in unlabeled samples given at the test time.

Self-training is the basis of many state-of-the-art methods in TTA and SFDA [7-9], utilizing pseudo labels generated from the model's own predictions to train a model on unlabeled samples [10]. Since pseudo labels are regarded as true labels in self-training, the success of self-training methods highly depends on how to filter incorrect pseudo labels to prevent self-confirmation bias [11]. This issue has been effectively handled via simple confidence-based thresholding in i.i.d. settings [12-14]. However, the distribution shifts make it hard to filter incorrect pseudo labels due to high noise rates even under high threshold [15]. Thus, sophisticated methods filter incorrect pseudo labels based on a neighborhood structure of the data [16-18] and consistency of multiple predictions under different models [19] or augmentations [20](cf. Section 5), which are computationally intensive by nature.

Recent insights in [8] suggest an alternative strategy can be also effective: promoting temporal consistency can enhance self-training performance in SFDA without the computational burden of

previous methods. The temporal consistency regularizer, so called early learning regularization (ELR) [21], was originally developed to address neural networks' tendency to learn clean information first and then gradually memorize noisy labels [22, 23]; this setting is naturally connected to self-training scenarios when we regard the pseudo labels as random noisy labels. However, the impacts of ELR on self-training have not been fully understood. Also, since ELR does not consider unique characteristics of distribution shifts, we aim to answer the following question: Is there any principled way to improve the way of memorizing all past predictions tailored for self-training under distribution shifts?

In this work, we show that the answer is affirmative by proposing Anchored Confidence ( AnCon ) that uses confident predictions to support a generalized notion of temporal consistency . Specifically, we construct a generalized temporal ensemble, which weighs predictions based on predictive uncertainty, and then use the ensemble as a smoothing vector in label smoothing [24]. Then, through rigorous theoretical analyses, we show that our simple heuristic for the generalized temporal ensemble is asymptotically correct and that the label smoothing formulation can reduce the optimality gap. As a result, AnCon can correct wrong pseudo labels without expensive computations and can be easily applied to self-training methods by replacing one-hot pseudo labels with smooth pseudo labels, unlike neighborhood-based or centroid methods. Through extensive experiments, we show that AnCon improves self-training under diverse distribution shift scenarios and posses many attractive properties.

Our contribution can be summarized as follows: 1) We develop AnCon , which is the first algorithm that attempts to improve self-training under distribution shifts by generalizing a notion of temporal consistency with theoretical guarantees; 2) Without any additional forward passes or neighborhood search, AnCon improves self-training performances 8% and 16% under domain shifts and image corruptions, respectively; 3) Remarkably, we also show that AnCon significantly improves calibration performance and is robust with respect to model selection methods and hyperparameter choices.

2 Background

Notation and setup For an input space X ⊆ R d and a label space Y = [ K ] := { 1 , 2 , · · · , K } , we define X and Y to be random variables of input and output with probability densities p X and p Y , respectively. We also define a neural network f ( · ; θ ) : X → △ K -1 parameterized by a parameter θ ∈ R p where △ K -1 is the probability simplex with K elements. Our goal is to minimize the cross-entropy loss min θ ∈ R p l ( θ ) := E XY [ H ( f ( X ; θ ) , p Y | X )] where H ( f ( x ; θ ) , p Y | X = x ) := -∑ k ∈ [ K ] p ( Y = k | x ) log f k ( x ; θ ) . In the SFDA setting, we are given an initial parameter θ 0 that is trained on a different data generating distribution ( X ′ , Y ′ ) , e.g., θ 0 ∈ arg min θ ∈ R p E X ′ Y ′ [ H ( f ( X ′ ; θ ) , p Y ′ | X ′ )] . We can think about this setting as either (1) ( X ′ , Y ′ ) being pre-training data and θ 0 the foundation model with the task of fine-tuning the model on unlabeled X or (2) a transfer learning problem with ( X ′ , Y ′ ) being the source domain data and X the target domain. Here, we assume only covariate shift without concept shift; that is, p X = D p X ′ but p Y | X = D p Y ′ | X ′ . Even in this case, we note that suboptimality under the distribution shift, i.e., min θ ∈ R p l ( θ ) -l ( θ 0 ) , can be large.

̸

Self-training In this work, we tackle the distribution shifts by using the self-training method that replaces the true label Y ( x ) by the pseudo label, i.e., min θ new ˆ l ( θ new ; θ ) = E X [ -log f ˆ Y ( x ; θ ) ( x ; θ new )] where ˆ Y ( x ; θ ) := arg max k ∈ [ K ] f k ( x ; θ ) is a pseudo label under θ . Specifically, the algorithmic framework of self-training is as follows: given θ 0 , we iteratively find model parameter θ m +1 for m = 0 , 1 , · · · with θ m +1 ∈ arg min θ new ˆ l ( θ new ; θ m ) . For later use, we also define a prediction confidence c ( x ; θ ) := max k ∈ [ K ] f k ( x ; θ ) and θ 0: m := ( θ 0 , · · · , θ m ) .

Early learning regularization In the learning from noisy labels (LFN) scenario, [21] identified an "early-learning phenomenon" where neural networks initially learn information contained in clean labels before gradually memorizing noisy labels, leading to a performance deterioration as training progresses. To mitigate this issue, ELR penalizes predictions that deviate from earlier predictions by defining a target network from the past predictions: ¯ f ELR ( x ; θ 0: m ) := ∑ m j =0 (1 -γ ) · γ m -j f ( x ; θ j ) . Then, adding an auxiliary loss of L ELR ( θ ; θ 0: m ) = E X [log(1 -f ( X ; θ ) T ¯ f ELR ( X ; θ 0: m ))] to ˆ l ( θ ; θ m ) can prevent memorization of noisy labels while preserving correct patterns.

Notably, this insight has recently been confirmed to be applicable in the SFDA setting by [8]. This observation is appealing because ELR can be efficiently implemented by reusing past predictions without additional forward passes or neighborhood searching, unlike dominant methods in SFDA

[16-20]. Nevertheless, given that ELR stems from a general property of the neural network training in the i.i.d. setting, herein we aim to step towards a more principled approach to encourage temporal consistency tailored for distribution shift scenarios.

3 Anchored confidence

In this section, we introduce AnCon , which promotes the temporal consistency on selectively chosen predictions via label smoothing. In Section 3.1, we first explain the idea of promoting selective temporal consistency based on confident predictions via label smoothing [24] with a temporal ensemble. Then, in Section 3.2, we explain how to effectively construct temporal ensemble for improving self-training under distribution shifts. Finally, we theoretically analyze the efficacy of AnCon by drawing connection between our method and knowledge distillation in Section 3.3.

3.1 Selective temporal consistency via label smoothing

In this work, we utilize label smoothing [24] to promote the selective temporal consistency instead of using an auxiliary loss function like ELR. Specifically, given a generalized temporal ensemble ¯ f ( x ; θ 0: m , w 0: m ) with w 0: m := ( w 0 , · · · , w m ) which will be specified in Section 3.2, we construct a regularized pseudo label ˜ Y ( X ; θ 0: m , w 0: m ) by using ¯ f ( x ; θ 0: m , w 0: m ) as a smoothing vector for the pseudo label ˆ Y ( x ; θ m ) . That is, we perform self-training by

(1)

where λ ∈ [0 , 1] is a coefficient, E 1 ( · ) is the one-hot encoding, and ¯ f ( x ) := ( ¯ f 1 ( x ) , · · · , ¯ f K ( x )) is the K -dimensional output of the generalized temporal ensemble (cf. (2)).

Thus, AnCon can control the usage of potentially noisy information in ˆ Y ( x ; θ m ) based on its consistency with ¯ f ( x ; θ 0: m , w 0: m ) . Not only can this approach preserve the early learning phenomenon as in ELR (cf. Section 2), but the label smoothing formulation also significantly stabilizes the selftraining performance under different hyperparameter choices due to the fact that the optimal values of hyperparameters are less problem dependent compared to its equivalent auxiliary regularization [25]. Beyond removing the burden of hyperparameter search, this robustness is a particularly intriguing property under distribution shift scenarios where the model selection becomes a challenging task.

Further, encouraging the temporal consistency through label smoothing enables us to connect our method with knowledge distillation (KD) [26], which can provide a wide range of principled techniques and theoretical results developed in KD. As a concrete example, we will show when and how AnCon can reduce the optimality gap of self-training, i.e., a case with λ = 0 , in Section 3.3.

3.2 Constructing an effective generalized temporal ensemble with prediction confidences

Next, we construct the generalized temporal ensemble that makes the selective temporal consistency in (1) work effectively in self-training under distribution shifts. Specifically, given θ 0: m and weights w 0: m ( x ) ∈ △ m for each x ∈ X , the prediction by the generalized temporal ensemble is

where p ( y | x, θ i ) is the prediction made by f ( x ; θ i ) , which can be either soft ( p ( y = j | x, θ i ) = f j ( x ; θ i ) ) or hard ( p ( y = j | x, θ i ) = 1 if j = arg max k ∈ [ K ] f k ( x ; θ i ) and p ( y = j | x, θ i ) = 0 otherwise). In this work, we use hard prediction because soft prediction puts more weights on recent predictions since self-training tends to keep increasing the prediction confidence during training.

Surprisingly, we will show that the following simple relative thresholding for determining w 0: m gives the asymptotic optimal weights achieving the minimum worst-case optimality gap of self-training:

where δ ( β ) m is an exponential moving average (EMA) of prediction confidence with hyperparameter β and ˆ E [ c ( X ; θ i )] is a Monte-Carlo approximation of E [ c ( X ; θ i )] with mini-batch samples.

Intuitively, our weighting mechanism aggregates only relatively confident predictions with a uniform weight. Given the observation that relative ordering of confidence is highly correlated with accuracy even under distribution shifts [27, 28], our thresholding rule would tend to put non-zero weights on correct predictions. Besides, by employing the relative criterion, the thresholding does not suffer from the problems that neglect predictions obtained in the early stage of training.

We also remark that AnCon has almost the same computational cost as ELR. Specifically, (2) can be implemented by ¯ f k ( x ; θ 0: m , w 0: m ) = ¯ f k ( x ; θ 0: m -1 , w 0: m -1 ) + w m ( x ) p ( y = k | x, θ m ) that requires to store the weighted sum of previous predictions without additional forward passes or storing previous parameters, which is the same as storing the previous logit vector in ELR. Similarly, (3) can be efficiently implemented by δ ( β ) m = βδ ( β ) m -1 +(1 -β ) ˆ E X [ c ( X ; θ m )] that requires constant additional computational costs compared to vanilla self-training with the constant being small. Therefore, AnCon shares the same computational benefits as ELR compared to other state-of-the-art methods in SFDA.

On optimality of the relative thresholding in (3) From the optimization perspective, the optimal weights w † 0: m correspond to the weights under which self-training with ˜ Y ( X ; θ 0: m , w 0: m ) can minimize the expected loss:

Unfortunately, l ( θ † w 0: m ) , or its empirical counterpart, is not available in self-training due to the absence of labels. Further, even if labels are given, solving (4) is intractable due to non-smoothness of l ( θ † w 0: m ) with respect to w 0: m and the cost of finding θ † w 0: m .

To circumvent this issue, we show in Section 3.3 that (4) can be relaxed to the problem of finding ensemble weights that give a maximum likelihood estimation (MLE) solution under certain conditions. As a result, instead of solving the intractable optimization in (4), we find the optimal weights by

In the following theorem which is proven in Appendix B.2, we show that the simple relative thresholding in (3) can make ¯ f ( x ; θ 0: m , w 0: m ) asymptotically correct for samples where the neural network tends to be relatively confident during self-training and thus our simple weighting mechanism in (3) to be the solution of (5) in the asymptotic region.

Theorem 3.1. Let A i ( c ) := { x ∈ X| c ( x ; θ i ) > c } , Q ( x ; c 0: m ) := ∑ m i =0 1 ( x ∈ A i ( c i )) , and ¯ p ( x ; c 0: m ) = 1 Q ( x ; c 0: m ) ∑ m i =0 E Y | X = x [ 1 ( Y ( x ) = ˆ Y ( x ; θ i ))] 1 ( x ∈ A i ( c i )) for x such that Q ( x ; c 0: m ) >0. Let us assume that random events 1 ( Y ( x ) = ˆ Y ( x ; θ i )) and 1 ( Y ( x ) = ˆ Y ( x ; θ j )) are conditionally independent given X ∈ A i ( c ) for j ∈ { 0 , · · · , i -1 } , x ∈ X , c ∈ [0 , 1) . If x ∈ X such that ¯ p ( x ; c 0: m ) > 1 / 2 , then for the generalized temporal ensemble in (3), it holds that

̸

where ξ ( z ) := 2 z -1 -log(2 z ) is a positive increasing function in z ∈ [0 . 5 , 1] .

The result states that as long as the average accuracy for relatively confident predictions over iterations exceeds 50%, the error rate of the generalized temporal ensemble monotonically decreases as Q ( x ; c 0: m ) increases. Furthermore, ¯ f ( x ; θ 0: m , w 0: m ) is asymptotically correct on x such that Q ( x ; c 0: m ) → ∞ as m → ∞ . AnCon aims to achieve these desirable properties through the uncertainty-aware temporal consistency that helps to satisfy the condition ¯ p ( x ; c 0: m ) > 0 . 5 . Specifically, as shown in Figure 4a in Appendix, our generalized temporal ensemble's accuracy tends to significantly increase as the number of confident samples increases, being consistent with our theory. We note that this monotonic improvement would not be the case for the temporal ensemble without uncertainty-awareness and vanilla self-training (cf. Figure 4a).

Finally, we emphasize that the assumption ¯ p ( x ; c 0: m ) > 0 . 5 applies only to relatively confident predictions which are averaged over iterations . This is significantly weaker than requiring a lower bound of an expected accuracy of each sample for every iteration , which is the case when LFN methods are directly applied to the self-training scenario. Also, due to its dependency on the choice of the confidence thresholds c 0: m , the assumption can hold by controlling c 0: m at the expense

of loosening the upper bound in (6) (e.g., selecting 90th-quantile as in Figure 4b in Appendix). Specifically, increasing the thresholds can improve ξ (¯ p ( x ; c 0: m )) and enhance the chance of satisfying ¯ p ( x ; c 0: m ) > 1 / 2 but reducing Q ( x ; c 0: m ) . While this trade-off necessitates a proper choice of c 0: m , our extensive experiments show that setting the threshold c m by the EMA of prediction confidence, i.e., δ ( β ) m in (3), works effectively.

3.3 Theoretical insights from knowledge distillation

In this section, we present a novel connection between AnCon and KD for addressing intractability of (4). KD is a framework for training a small student network f , e.g., ResNet-50 [29], with an additional supervision from a large teacher network f ( t ) , e.g., ResNet-152. Specifically, a cross-entropy under KD is l KD ( θ ) = E XY [ H ( f ( X ; θ ) , (1 -λ KD ) E 1 ( Y ( X )) + λ KD f ( t ) ( X )) ] with λ KD ∈ [0 , 1] , which bears a significant similarity with AnCon (cf. (1)). Indeed, AnCon can be understood as a special case of KD called self-distillation when f and f ( t ) have the same architecture, where the generalized temporal ensemble ¯ f corresponds to the teacher network f ( t ) with a notable difference that the pseudo label ˆ Y is used instead of the true label Y . Based on this connection, we perform a convergence analysis of AnCon by modifying the partial variance reduction theory [30], as given below. We note that the usage of ˜ Y and ¯ f results in an inherently biased gradient estimator, which requires special treatments for the convergence analysis unlike the typical self-distillation setting.

Setup Following [30], we assume a linear model f k ( x ; θ ) = exp(Θ T k x ) / ∑ i ∈ [ K ] exp(Θ T i x ) with Θ i ∈ R d for i ∈ [ K ] , θ := Concat (Θ 1 , · · · , Θ K ) ∈ R dK , and Concat ( · ) is the concatenation operation. Also, we assume a bounded support for X ; that is, ∥ x ∥≤ C for all x where p X ( x ) > 0 . Under this setting, we repeat the following steps starting from a given θ 0 (i.e., m = 0 ):

Outer temporal ensemble update: Update w 0: m by (3) to obtain ¯ f ( x ; θ 0: m , w 0: m ) (cf. (2)).

Inner parameter update: With θ m, 0 := θ m and ( θ 0: m , w 0: m ) , solve (1) with stochastic gradient descent θ m,t +1 = θ m,t -γg ( m,t ) ξ for t ∈ { 0 , 1 , · · · , T -1 } and set θ m +1 = θ m,T .

Iteration number update: Set m = m +1 and terminate if m = ˆ T .

Here, γ is the learning rate, ξ contains b random samples from p X , and the stochastic gradient under AnCon is defined as g ( m,t ) ξ := ∇ θ m,t 1 b ∑ X i ∈ ξ H ( f ( X i ; θ m,t ) , ˜ Y ( X i ; θ 0: m , w 0: m )) . For the linear model, we note that g ( m,t ) ξ = ∇ ˜ l ξ ( θ m,t ) -λ ˆ g ξ ( θ 0: m , w 0: m ) where ˜ l ξ ( θ m,t ) = 1 b ∑ X i ∈ ξ H ( f ( X i ; θ m,t ) , ˆ Y ( X i ; θ m )) and ˆ g ξ ( θ 0: m , w 0: m ) := Concat (ˆ g 1 ,ξ ( θ 0: m , w 0: m ) , · · · , ˆ g K,ξ ( θ 0: m , w 0: m )) with ˆ g k,ξ ( θ 0: m , w 0: m ) := 1 b ∑ X i ∈ ξ [( ¯ f k ( X i ; θ 0: m , w 0: m ) -ˆ Y k ( X i ; θ m )) X i ] for k ∈ [ K ] .

In Theorem 3.2 which is proven in Appendix B.3, we analyze the convergence of the inner parameter update step under AnCon and vanilla self-training.

Theorem 3.2. Let us assume l ( θ ) satisfies L -smoothness, L -expected smoothness, and µ -PolyakLojasiewicz (PL) condition (cf. Assumptions B.1, B.2, and B.3 in Appendix B.1). For γ ≤ µ 4 L· L , a carefully chosen λ (cf. λ = λ † m := E ξ [ ⟨∇ ˜ l ξ ( θ ∗ ) , ˆ g ξ ( θ 0: m , w 0: m ) ⟩ ] E ξ ∥ ˆ g ξ ( θ 0: m , w 0: m ) ∥ 2 + 2 Lγ ∥ ˆ g ( θ 0: m , w 0: m ) ∥ 2 in Lemma B.6 in Appendix B.5), and any realization of θ m , it holds that

̸

where l ∗ := l ( θ ∗ ) with θ ∗ ∈ arg min θ ∈ Θ l ( θ ) , g E ( θ ) := E [ 1 ( ˆ Y ( X ; θ ) = Y ( X ))] , N ( λ ; θ 0: m , w 0: m ) = λ 2 ∥ ˆ g ( θ 0: m , w 0: m ) ∥ 2 + Lγ 2 E ξ [ ∥ ∇ ˜ l ξ ( θ ∗ ) -λ ˆ g ξ ( θ 0: m , w 0: m ) ∥ 2 ] , and N ( λ † m ; θ 0: m , w 0: m ) ≤ N (0) where N (0) := N (0; θ 0: m , w 0: m ) for any ( θ 0: m , w 0: m ) .

Further, when w 0: m is such that E ξ [ ⟨∇ ˜ l ξ ( θ ∗ ) , ˆ g ξ ( θ 0: m , w 0: m ) ⟩ ] ≥ 0 , i.e., the teacher has a sufficiently good performance, it holds that

where σ 2 ∗ := E ξ ∥ ∇ ˜ l ξ ( θ ∗ ) ∥ 2 , σ 2 ( θ 0: m , w 0: m ) := E ξ ∥ ˆ g ξ ( θ 0: m , w 0: m ) ∥ 2 , g KL ( θ 0: m , w 0: m ) = E X [ D KL ( f ( X ; θ ∗ ) ∥ ¯ f ( X ; θ 0: m , w 0: m ))] with D KL ( p ∥ q ) is the KL-divergence between p and q , and g C ( θ 0: m , w 0: m ) = ∥ E X [ ¯ f ( X ; θ 0: m , w 0: m )] -E X [ ˆ Y ( X ; θ m )] ∥ 2 .

In Theorem 3.2, (7) characterizes the optimality gap in the inner loop optimization. Specifically, the first term is about reducing the initial optimality gap over iterations and motivates why we need adaptation , e.g., by self-training, if the performance deteriorates under severe distribution shifts. The second term is about the bias of the pseudo label and motivates the challenges of self-training under poorly performing pseudo labels in the case of severe distribution shifts. Crucially, these two terms can be fully characterized by the quality of initial model θ m and do not depend on w 0: m . Thus, we concentrate on the impacts of w 0: m designed in (1)-(3) on N ( λ † m ; θ 0: m , w 0: m ) to show that AnCon 's effectiveness on improving self-training performance under distribution shifts.

First, Theorem 3.2 shows the effectiveness of our label smoothing formulation in (1) under properly chosen λ † m . Specifically, compared to vanilla self-training, AnCon results in the smaller neighborhood size of the stochastic gradient descent; N ( λ † m ; θ 0: m , w 0: m ) ≤ N (0) . That is, the result suggests that AnCon is at least better than vanilla self-training under the mild regularity conditions.

Further, under the additional assumption of a sufficiently good performance temporal ensemble, (8) motivates AnCon 's weighting mechanism as a relaxed solution of the intractable optimization problem in (4). Specifically, if the marginal distribution of the pseudo labels does not change quickly over outer iterations (which is the case especially for the later training stages as shown in Figure 5 in Appendix), changing w 0: m would have only a marginal impact on g C ( θ 0: m , w 0: m ) . Thus, the weighting mechanism that minimizes g KL ( θ 0: m , w 0: m ) would minimize the worst-case optimality gap, which justifies our approach of circumventing intractability of (4) with (5). In this regard, AnCon 's weighting mechanism in (3) could be thought of as a relaxed solution of (4) as it minimizes g KL ( θ 0: m , w 0: m ) in the asymptotic region (cf. Theorem 3.1).

We conclude this section by analyzing the three iterative steps where the pseudo labels and the temporal ensembles keep updating.

Corollary 3.2.1. Let us assume l ( θ ) satisfies L -smoothness, L -expected smoothness, and µ -PL condition. For γ ≤ µ 4 L· L , λ carefully adjusted at each outer temporal ensemble update (cf. λ = λ † j in Lemma B.6 for each outer iteration j ∈ { 0 , · · · , ˆ T -1 } ), and any initial parameter θ 0 , it holds that

Corollary 3.2.1 is proved in Appendix B.4 and gives a whole picture of the optimality gap under AnCon . We first remark the trade-off associated with ˆ T on the suboptimality E [ l ( θ ˆ T ) -l ∗ ] , which characterize the early-learning phenomenon observed in the biased gradient settings (e.g., self-training [31] and LFN [32]). Specifically, in (9), increasing ˆ T reduces the first term (1 -µγ ) T · ( ˆ T -1) ( l ( θ 0 ) -l ∗ ) but increases the coefficient of the second term ζ ˆ T . Therefore, a longer training with a large ˆ T may not enhance the self-training performance especially under a large second term due to inaccurate pseudo labels or the temporal ensemble.

E

Nevertheless, for each outer loop iteration, g ( θ j ) would be smaller under AnCon than its value under vanilla self-training as E [ l ( θ j ) -l ∗ ] has a tighter upper bound under AnCon due to Theorem 3.2. Therefore, with the guarantee N ( λ † j ; θ 0: j , w 0: j ) ≤ N (0) , AnCon would achieve a tighter upper bound of (9) than the vanilla self-training method, enabling longer training with smaller value of the second term as observed in Figure 1b. Finally, we remark that this theoretical superiority of AnCon can be extended to the self-training methods with other weighting mechanisms in the asymptotic region when E ξ [ ⟨∇ ˜ l ξ ( θ ∗ ) , ˆ g ξ ( θ 0: m , w 0: m ) ⟩ ] ≥ 0 (cf. Theorem 3.2).

4 Experiments

Goal and baselines Part of the experiments shows that AnCon surpasses the vanilla self-training method that uses ˆ Y ( x ; θ ) as a pseudo label, which serves as a strong baseline, under different types

of distribution shifts (Sections 4.1-4.2). We also show that AnCon achieves a better performance than ELR with λ ∗ ELR ∈ { 1 , 3 , 7 , 12 , 25 } , which is the coefficient multiplied to the auxiliary loss, to show effectiveness of uncertainty aware temporal ensemble. Finally, we assess the integration of AnCon with generalized cross-entropy (GCE) [33, 31] and neighborhood reciprocity clustering (NRC) [17], which are frequently cited as state-of-the-art methods [20, 34, 31]. We note that AnCon can be seamlessly applied to GCE and NRC by replacing one-hot pseudo labels by AnCon 's regularized pseudo labels ˜ Y ( X ; θ 0: m , w 0: m ) . For descriptions of the training configurations which we adopt from literature, see Appendix D.

Evaluation In our self-training settings (SFDA and TTA), we should determine the best checkpoint without labeled samples, i.e. to select a model θ 0 , θ 1 , · · · , θ I which is used for evaluation on test. We use information maximization [35], IM ( θ ) = H E ( E X [ f ( X ; θ )]) -E X [ H E ( f ( X ; θ ))] when H E is the entropy, which is proven to be effective in unsupervised domain adaptation (UDA) model selection literature [36, 37]. Specifically, we evaluate IM ( θ m ) on the hold-out unlabeled samples at the end of each epoch m for I number of training epochs and select the checkpoint with θ ∗ ∈ arg max m ∈ [ I ] IM ( θ m ) as the best one. This θ ∗ is used for final evaluation on the test data.

Implementation details of AnCon For each (outer) iteration m , we set w j ( x ) = 1 ( c ( x ; θ j ) > δ ( β ) j )

for j ∈ { 0 , · · · , m } , instead of w j ( x ) = 1 ( c ( x ; θ j ) >δ ( β ) j ) ∑ i m =0 1 ( c ( x ; θ i ) >δ ( β ) i ) . Note that the former with λ can be thought of as the latter with instance-dependent coefficient λ · Q ( x ; c 0: m ) , which assigns more weight on the generalized temporal ensemble for x with large Q ( x ; c 0: m ) . In addition, we consider the online update scenario of pseudo labels, i.e., T = 1 . Finally, given the challenging nature of hyperparameter selection in self-training under distribution shifts, we perform all experiments by using a single configuration of hyperparameters of AnCon ( λ = 0 . 3 and β = 0 . 9 ). These hyperparameters result in the maximum value of max m ∈ [ ˆ T ] IM ( θ m ) on the hold-out unlabeled samples on Office-31.

4.1 Self-training under domain shifts

We first consider SFDA that adapts a model trained in one domain by performing self-training in the distribution shifted domain. For the network architecture, we use the modified ResNet used in [16], which includes batch normalization [38] after the bottleneck layer and weight normalization [39] in the last linear layer, which is used in [17, 8] for stabilizing the learning process in SFDA.

Datasets We evaluate AnCon on the following datasets: Office-31 [40] with 4,000 images of 31 categories from three domains (amazon, dslr, webcam); OfficeHome [41] with 15,000 images of 65 categories from four domains (art, clipart, product, real-world); VisDa [42] with 280,000 images of 31 categories from two domains (synthetic, real).

Results AnCon consistently improves self-training in diverse domain pairs across the three datasets (Table 1). Specifically, it reduces the average self-training test error by 5% in Office-31, 6% in OfficeHome, and 13% in VisDa. In addition, compared to ELR with its dataset-dependent optimal hyperparameter value, AnCon shows comparable performance to ELR (2%, 3%, and -3% performance differences in Office-31, OfficeHome, and VisDa, respectively). Despite its slightly inferior performance in VisDa, we note that AnCon achieves significantly better accuracy for low-performing classes: for 'Skateboard' and 'Truck' classes, AnCon achieves 50% and 16% of accuracies, while ELR achieves 39% and 0.43% in these classes. Thus, AnCon can be as effective as ELR for improving the self-training performances under distribution shifts without needing to adjust hyperparameter values for each dataset unlike ELR.

(a)

(b)

(c)

70-870870.6

40

15

Self-training

Test Accuracy (%)

60

70

50

工H

109

63.062.6

53.3

56.556.8

50.351.3

Test Accuracy (%)

30

Kperfchange

10

AnCon

ELR

Self-training

40

ELR

33.3

20

0E

AnCon

10

Self-training

5

20

ELR (A=3.0)

10

0

Anchored Confidence

0

0

Level 1

Level 2

Corruption Intensity

Level 3

Level 4

Level 5

0

5

Epochs

10

15

20

1

Corruptionintensity

2

3

4

5

Notably, AnCon significantly improves performances of both GCE and NRC via fundamentally different mechanisms for handling noisy pseudo labels (e.g., reducing test accuracy of 3% and 8% on average in OfficeHome, respectively). Specifically, NRC filters incorrect predictions based on local consistency, while AnCon uses temporal consistency. Combining NRC and AnCon leverages pseudo labels that are both locally and temporally consistent, resulting in significant performance improvements over NRC or Self-Training + AnCon (cf. Table 1). In addition, GCE reduces the impact of wrong pseudo labels rather than finding them. Applying GCE to AnCon minimizes the effects of potentially wrong but temporally consistent pseudo labels, which can be implied by the performance of GCE + AnCon compared to GCE or Self-Training + AnCon (cf. Table 1). Thus, the impressive performance gains from AnCon , which would be orthogonal to the gains from state-of-the-art methods in SFDA, show its significant practical implications.

4.2 Self-training under synthetic corruption operations

While we have considered the domain shift, e.g., adaptation of a model trained on synthetic images to real images, in Section 4.1, this section examines the self-training's ability to adapt to distribution shifts by synthetic image corruptions. This setting has been used to measure the robustness of neural networks with respect to a general out-of-distribution setting. To this end, we consider ImageNet-C [6], which consists of 50,000 images drawn from a validation set of ImageNet [43] where each image is corrupted by 15 types of synthetic corruptions related to noise, blur, weather and digital.

Result Consistent with the findings under the domain shift, AnCon outperforms the average performances of self-training and ELR under varying levels of corruption intensities (cf. Figure 1a and Tables 7-11 in Appendix), improving the self-training method's accuracy by 16% on average. Further, the gains from AnCon is significant when the distribution shifts are intense (e.g., improving accuracies by 20% and 52% on average in intensities of 4 and 5) where the initial model trained on the source domain significantly deteriorates. Specifically, for Shot, Impulse, and Gaussian corruptions with the most extreme shift intensity of 5, where the initial model achieves accuracies of (3.04%, 1.76%, 2.12%), AnCon achieves (22.56%, 26.56%, 25.85%) (cf. Table 11). This striking improvement compared to vanilla self-training with performances (0.26%, 1.72%, 1.04%) and ELR with performances (8.00%, 14.12%, 16.00%), underscores the importance of the AnCon's uncertainty-aware temporal consistency scheme, as shown in Corollary 3.2.1. We note that this impressive result is also explained by AnCon 's ability to prevent the gradual performance degradation during the course of training with the extremely noisy pseudo labels (cf. Figure 1b). Combined with previous results in domain shift scenarios, we expect that AnCon would work effectively in various out-of-distribution settings.

4.3 Versatility of AnCon

In previous sections, we have shown the universality of AnCon by evaluating it on diverse distribution shift scenarios. In this section, we show versatility of AnCon by analyzing its attractive properties in robustness and uncertainty representation.

4.3.1 Robustness to model selection

There is no universally agreed model selection criterion, such as cross-validation in the i.i.d. setting, in self-training under distribution shifts. This is partly due to the variety of distribution shift scenarios, where an effective criterion in one may be ineffective or inapplicable in another; for instance, a

principled criteria called importance-weighted cross validation [44] in UDA cannot be applied to SFDA. In this regard, it would be an important characteristic of a self-training method under distribution shift to be robust with respect to different choices of model selection criteria. Therefore, we evaluate robustness with respect to the following different model selection criteria: InfoMax [35], Corr-C [36], and Ent [45] (see Appendix D.3 for the description).

Figure 1c shows that AnCon 's maximum performance change due to different model selection methods is much lower than that of other methods, especially under severe distribution shifts. This valuable advantage can be contributed to the property of AnCon that can prevent performance degeneration (cf. Figure 1b). Given that, in practice, we barely know when the model collapse happens and which model selection criteria are the best, the results highlight a significant practical value of AnCon .

4.3.2 Robustness to the choice of hyperparameters

Throughout this paper, we have shown that our single configuration of parameters ( λ = 0 . 3 , β = 0 . 9 ) work well across a wide range of benchmark problems. In this section, we aim to show our findings can be preserved when the hyperparameter values deviate from the default setting by performing a sensitivity analysis for values λ ∈ { 0 . 1 , 0 . 3 , 0 . 5 , 0 . 7 , 0 . 9 } and β ∈ { 0 . 1 , 0 . 3 , 0 . 5 , 0 . 7 , 0 . 9 } . We also test two frequently used annealing schedules that λ m = m/I and λ m = min(1 , 2 m/I ) , called full and half, respectively. Figure 2 shows that AnCon is stable even under extreme values of hyperparameters. Specifically, for both hyperparameters, the maximum average performance change is less than 1%, and β barely impacts the performance of AnCon . Indeed, our analysis suggests to increase λ from our default setting; that is, to put a higher weight on the general temporal ensemble's prediction. Here, we note that our suboptimal choices of hyperparameters are due to our rigorous and practical hyperparameter choice. Given the challenging nature of hyperparameter optimization under distribution shifts, the stable performances of AnCon seamlessly applied to diverse practical settings.

(%)

3

Relative performancechange

2

1

0

2

-3

4

0.1

0.30.50.7

0.90.99full

half

Smoothing coefficient(A)

(a)

(b)

(%)

3

change

2

Relativeperformance

1

0

-1

0.1

0.3

0.5

0.7

0.9

Confidence EMA coefficient(β)

under arbitrary choices of hyperparameters would enable AnCon to be

4.3.3 Improved calibration performance

We have shown that all self-training methods significantly improve the performance of the baseline method after the adaptation period. However, it is widely known that these noticeable improvements come with the price of sacrificing an uncertainty representation ability which is critical in real-world decision-making scenarios [46, 47]. Specifically, the calibration performance, which is the gap between the prediction confidence and accuracy, usually monotonically increases as self-training keeps reducing the uncertainty for all predictions during the course of training. In this regard, we analyze the calibration performance with respect to the expected calibration error (ECE; see Appendix for definition). Here, a lower ECE means a lower gap between confidence and accuracy.

As shown in Figure 3a and Table 5 in Appendix, AnCon gives much lower ECE compared to other methods. Considering ELR and GCE both have regularization effects, we conjecture that this phenomenon is due to selective regularization in AnCon that increases prediction confidences of samples only if the past confident predictions are consistent with the current prediction. Especially, in

100

Self-training

ELR

08

AnCon

60

ECE(

51.4

40

27.1

20

19.119.1

15.9

18.6

9114

E:OT

4.21

Level 1

Level 2

Level 3

Level 4

Level 5

Corruption Intensity

(a)

(b)

28

Self-training

Self-training

55

ELR

ELR

GCE

26

GCE

50

AnCon

AnCon

Test ECE (%)

24

45

2

20

35

OE

18

0

Epochs

10

15

20

0

Epochs

10

15

Figure 3b which confirms the accuracy-calibration dilemma in VisDa, AnCon is shown to limit the ECE increases during training compared to all other methods. That is, AnCon helps to significantly reduce the price of the calibration performance we need to pay for improving accuracy, which are both important measures in practice.

4.4 Algorithmic design choices

Recall that we define ¯ f ( x ; θ 0: m , w 0: m ) = ∑ m i =0 w i ( x ) · p ( y | x, θ i ) with our simple design choices: the relative thresholding for weighting scheme w i ( x ) ∝ 1 ( c ( x ; θ i ) > δ ( β ) i ) and hard prediction for p ( y | x, θ i ) . In Appendix C.3, we found that our simple design choices are more appropriate for the distribution shift settings than several more sophisticated alternatives, which can be summarized as follows.

More sophisticated weighting schemes (e.g., Entropy ( w i ( x ) ∝ exp {-H E [ f ( x ; θ i )] } )) reduce the self-training performance, despite being a more accurate measure of prediction uncertainty. We conjecture that the poor calibration performance of the neural network in self-training under distribution shifts prevents the sophisticated weighting schemes from accurately reflecting the goodness of the prediction.

Various soft prediction schemes, which can give more information about the non-leading entry values, leads to performance reductions. We conjecture that the continuously increasing confidence in the later stage of self-training would make soft prediction ignore early-stage predictions which may be valuable to memorize.

5 Related work

Filtering incorrect pseudo labels Popular confidence-based thresholding methods [13, 14, 9] fall short under distribution shifts since even high confident predictions can be highly incorrect. Therefore, recent advances in SFDA and TTA utilize higher order information to filter incorrect pseudo labels. For instance, based on the intuition that true labels of adjacent samples would be same, centroids for each predicted class can be maintained in the feature space and then the pseudo label for each input is corrected by the adjacent centroid [16]. The idea of using per-class centroids has been extended to incorporate more general clustering structures [18, 17]. However, the neighborhood structure-based methods are computationally demanding due to storage of memory banks in the feature space and nearest neighbors search. Such computational complexity persists in other approaches, which are based on the consistency of multiple predictions from different augmentations [20] and models trained with different loss functions [19]. Compared to these solutions, AnCon can efficiently estimate correct labels with only limited extra memory overhead of storing past predictions.

Learning from noisy labels Treating pseudo labels as inherently noisy, techniques from the LFN literature have been integrated to self-training. For instance, the LFN literature has proposed robust loss functions that reduce impacts of random noisy labels [48, 33], and a recent large-scale experimental study shows the applicability of the generalized cross-entropy in the SFDA setting [31]. The effectiveness of ELR on SFDA [8] bears a similar idea because ELR was developed to regularize the neural networks' tendencies to memorize incorrect labels [21]. Despite their effectiveness, by nature, these approaches do not consider important characteristics of the unbounded and instancedependent noise rates inherent in self-training under distribution shifts, which results in significant suboptimality in both theory and practice. However, by considering the unique characteristics of self-training under distribution shift, AnCon relaxes the conditions required to achieve optimality as well as boosts the self-training performance in diverse scenarios.

6 Conclusion

This paper introduces AnCon , which effectively improves self-training performances under diverse distribution shift scenarios by promoting selective temporal consistency based on confident predictions. As a result, AnCon effectively mitigates the detrimental effects of noisy pseudo labels without much computational overhead, unlike the previous methods. We show that AnCon not only advances our theoretical understanding of a generalized notion of temporal consistency in self-training but also can be a practical asset as a simple and effective self-training method with attractive properties. In Appendix C.4, we present limitations and future directions, such as adaptive determination of λ , combining local and temporal consistency, and extending the selective temporal consistency in the sequential decision making problems.

Acknowledgments and Disclosure of Funding

We would like to thank Jihyeon Hyeong, Yuchen Lou, Jiezhong Wu, and anonymous reviewers for their valuable discussions and constructive suggestions during the preparation of this manuscript. We declare that there was no funding received for this work.

References

Joy Buolamwini and Timnit Gebru. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Conference on Fairness, Accountability and Transparency , 2018.

Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do Imagenet classifiers generalize to Imagenet? In International Conference on Machine Learning , 2019.

Neal Jean, Marshall Burke, Michael Xie, W Matthew Davis, David B Lobell, and Stefano Ermon. Combining satellite imagery and machine learning to predict poverty. Science , 353 (6301):790-794, 2016.

Sara Beery, Grant Van Horn, and Pietro Perona. Recognition in terra incognita. In European Conference on Computer Vision , 2018.

Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht, and Ludwig Schmidt. Measuring robustness to natural distribution shifts in image classification. In Advances in Neural Information Processing Systems , 2020.

Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In International Conference on Learning Representations , 2019.

Hong Liu, Jianmin Wang, and Mingsheng Long. Cycle self-training for domain adaptation. In Advances in Neural Information Processing Systems , 2021.

Li Yi, Gezheng Xu, Pengcheng Xu, Jiaqi Li, Ruizhi Pu, Charles Ling, Ian McLeod, and Boyu Wang. When source-free domain adaptation meets learning with noisy labels. In International Conference on Learning Representations , 2023.

David Berthelot, Rebecca Roelofs, Kihyuk Sohn, Nicholas Carlini, and Alex Kurakin. Adamatch: A unified approach to semi-supervised learning and domain adaptation. arXiv preprint arXiv:2106.04732 , 2021.

Dong-Hyun Lee. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on Challenges in Representation Learning, ICML , 2013.

Eric Arazo, Diego Ortego, Paul Albert, Noel E O'Connor, and Kevin McGuinness. Pseudolabeling and confirmation bias in deep semi-supervised learning. In International Joint Conference on Neural Networks , 2020.

Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semisupervised learning with consistency and confidence. In Advances in Neural Information Processing Systems , 2020.

Bowen Zhang, Yidong Wang, Wenxin Hou, Hao Wu, Jindong Wang, Manabu Okumura, and Takahiro Shinozaki. Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling. In Advances in Neural Information Processing Systems , 2021.

Yidong Wang, Hao Chen, Qiang Heng, Wenxin Hou, Yue Fan, Zhen Wu, Jindong Wang, Marios Savvides, Takahiro Shinozaki, Bhiksha Raj, et al. Freematch: Self-adaptive thresholding for semi-supervised learning. arXiv preprint arXiv:2205.07246 , 2022.

Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, David Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. In Advances in Neural Information Processing Systems , 2019.

Jian Liang, Dapeng Hu, and Jiashi Feng. Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. In International Conference on Machine Learning , 2020.

Shiqi Yang, Joost van de Weijer, Luis Herranz, Shangling Jui, et al. Exploiting the intrinsic neighborhood structure for source-free domain adaptation. In Advances in Neural Information Processing Systems , 2021.

Shiqi Yang, Yaxing Wang, Joost Van De Weijer, Luis Herranz, and Shangling Jui. Generalized source-free domain adaptation. In IEEE/CVF International Conference on Computer Vision , 2021.

Hao-Wei Yeh, Thomas Westfechtel, Jia-Bin Huang, and Tatsuya Harada. Boosting source-free domain adaptation via confidence-based subsets feature alignment. In International Conference on Pattern Recognition , 2022.

Nazmul Karim, Niluthpol Chowdhury Mithun, Abhinav Rajvanshi, Han-pang Chiu, Supun Samarasekera, and Nazanin Rahnavard. C-sfda: A curriculum learning aided self-training framework for efficient source free domain adaptation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023.

Sheng Liu, Jonathan Niles-Weed, Narges Razavian, and Carlos Fernandez-Granda. Earlylearning regularization prevents memorization of noisy labels. In Advances in Neural Information Processing Systems , 2020.

Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM , 64(3): 107-115, 2021.

Devansh Arpit, Stanisław Jastrz˛ ebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, et al. A closer look at memorization in deep networks. In International Conference on Machine Learning , 2017.

Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In IEEE Conference on Computer Vision and Pattern Recognition , 2016.

Gabriel Pereyra, George Tucker, Jan Chorowski, Łukasz Kaiser, and Geoffrey Hinton. Regularizing neural networks by penalizing confident output distributions. arXiv preprint arXiv:1701.06548 , 2017.

Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 , 2015.

Yaodong Yu, Stephen Bates, Yi Ma, and Michael Jordan. Robust calibration with multi-domain temperature scaling. In Advances in Neural Information Processing Systems , 2022.

Taejong Joo and Diego Klabjan. IW-GAE: Importance weighted group accuracy estimation for improved calibration and model selection in unsupervised domain adaptation. In International Conference on Machine Learning , 2024.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition , 2016.

Mher Safaryan, Alexandra Peste, and Dan Alistarh. Knowledge distillation performs partial variance reduction. In Advances in Neural Information Processing Systems , 2023.

Evgenia Rusak, Steffen Schneider, George Pachitariu, Luisa Eck, Peter Vincent Gehler, Oliver Bringmann, Wieland Brendel, and Matthias Bethge. If your data distribution shifts, use selflearning. Transactions on Machine Learning Research , 2022.

Mingchen Li, Mahdi Soltanolkotabi, and Samet Oymak. Gradient descent with early stopping is provably robust to label noise for overparameterized neural networks. In International Conference on Artificial Intelligence and Statistics , 2020.

Aritra Ghosh, Himanshu Kumar, and P Shanti Sastry. Robust loss functions under label noise for deep neural networks. In AAAI Conference on Artificial Intelligence , 2017.

Ori Press, Steffen Schneider, Matthias Kümmerer, and Matthias Bethge. Rdumb: A simple approach that questions our progress in continual test-time adaptation. In Advances in Neural Information Processing Systems , 2023.

Yuan Shi and Fei Sha. Information-theoretical learning of discriminative clusters for unsupervised domain adaptation. In International Coference on International Conference on Machine Learning , 2012.

Weijie Tu, Weijian Deng, Tom Gedeon, and Liang Zheng. Assessing model out-of-distribution generalization with softmax prediction probability baselines and a correlation method, 2023. URL https://openreview.net/forum?id=1maXoEyeqx .

Dapeng Hu, Jian Liang, Jun Hao Liew, Chuhui Xue, Song Bai, and Xinchao Wang. Mixed samples as probes for unsupervised model selection in domain adaptation. In Advances in Neural Information Processing Systems , 2023.

Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning , 2015.

Tim Salimans and Durk P Kingma. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. In Advances in Neural Information Processing Systems , 2016.

Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell. Adapting visual category models to new domains. In European Conference on Computer Vision , 2010.

Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In IEEE Conference on Computer Vision and Pattern Recognition , 2017.

Xingchao Peng, Ben Usman, Neela Kaushik, Judy Hoffman, Dequan Wang, and Kate Saenko. Visda: The visual domain adaptation challenge. arXiv preprint arXiv:1710.06924 , 2017.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In IEEE Conference on Computer Vision and Pattern Recognition , 2009.

Masashi Sugiyama, Matthias Krauledat, and Klaus-Robert Müller. Covariate shift adaptation by importance weighted cross validation. Journal of Machine Learning Research , 8(5), 2007.

Pietro Morerio, Jacopo Cavazza, and Vittorio Murino. Minimal-entropy correlation alignment for unsupervised deep domain adaptation. arXiv preprint arXiv:1711.10288 , 2017.

Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in AI safety. arXiv preprint arXiv:1606.06565 , 2016.

John D Lee and Katrina A See. Trust in automation: Designing for appropriate reliance. Human Factors , 46(1):50-80, 2004.

Yang Liu and Hongyi Guo. Peer loss functions: Learning from noisy labels without knowing noise rates. In International Conference on Machine Learning , 2020.

Boris Teodorovich Polyak. Gradient methods for minimizing functionals. Zhurnal Vychislitel'noi Matematiki i Matematicheskoi Fiziki , 3(4):643-653, 1963.

Zachary Charles and Dimitris Papailiopoulos. Stability and generalization of learning algorithms that converge to global optima. In International Conference on Machine Learning , 2018.

Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. In Advances in Neural Information Processing Systems , 2018.

Herman Chernoff. A measure of asymptotic efficiency for tests of a hypothesis based on the sum of observations. The Annals of Mathematical Statistics , pages 493-507, 1952.

Taejong Joo, Uijung Chung, and Min-Gwan Seo. Being Bayesian about categorical probability. In International Conference on Machine Learning , 2020.

Yarin Gal and Zoubin Ghahramani. Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. In International Conference on Machine Learning , 2016.

Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael Jordan. Conditional adversarial domain adaptation. In Advances in Neural Information Processing Systems , 2018.

Yuchen Zhang, Tianle Liu, Mingsheng Long, and Michael Jordan. Bridging theory and algorithm for domain adaptation. In International Conference on Machine Learning , 2019.

A List of acronyms

TTA Test time adaptation

i.i.d. independent and identically distributed

ELR Early learning regularization

LFN

Learning from noisy labels

KD Knowledge distillation

EMA Exponential moving average

GCE Generalized cross-entropy

NRC Neighborhood reciprocity clustering

PL Polyak-Lojasiewicz

B Proof of claims

B.1 Assumptions

Let us recall that l ( θ ) = E XY [ H ( f ( X ; θ ) , p Y | X )] and l ξ ( θ ) = 1 b ∑ X i ∈ ξ H ( f ( X i ; θ ) , p Y | X i ) where ξ contains b number of random samples from p X . We also denote θ ∗ ∈ arg min θ ∈ R p l ( θ ) . In this work, we assume the following three regularity conditions on l ( θ ) which is differentiable with respect to θ , which are mild but essential for most theoretical studies with convergence analyses.

Assumption B.1 ( L -smoothness) . l ( · ) is L -smooth for some constant L > 0 ; that is,

Assumption B.2 ( L -expected smoothness [30]) . l ( · ) is L -smooth in expectation with respect to ξ ∼ D ; that is,

Assumption B.3 ( µ -Polyak-Lojasiewicz (PL) condition [49]) . l ( · ) satisfies the µ -PL condition for some constant µ > 0 ; that is,

We remark that Assumptions B.1 and B.2 can trivially hold under bounded parameter values that can be guaranteed by optimizing neural networks with finite iterations under a gradient or weight clipping. Assumption B.3 holds for infinite-width neural networks, i.e., the neural tangent kernel (NTK) regime [50]. Given that the gradient descent training dynamics of neural networks can be well approximated by NTK [51], the PL condition can be generally regarded as a mild assumption.

B.2 Proof of Theorem 3.1

Proof. For x such that Q ( x ; c 0: m ) > 0 , let us consider subsequence of index l such that x ∈ A l ( c l ) ; that is, ( j 1 , · · · , j Q ( x ; c 0: m ) ) where j l = min { l ≥ j l -1 | x ∈ A l ( c l ) } and j 0 = 0 . Let S Q ( x ; c 0: m ) = ∑ Q ( x ; c 0: m ) i =1 1 ( Y ( x ) = ˆ Y ( x ; θ j i )) . Then, we have the following inequality

̸

where the first inequality holds because arg max k ¯ f k ( x ; θ 0: m , w 0: m ) = Y ( x ) if S Q ( x ; c 0: m ) > Q ( x ; c 0: m ) / 2 and the second inequality holds due to Lemma B.4 given later in Appendix B.5 with p i = p ( Y ( x ) = ˆ Y ( x ; θ i )) , o = Q ( x ; c 0: m ) , and q = 1 / 2 .

B.3 Proof of Theorem 3.2

Proof. By Lemma B.5, any realization of θ m satisfies

Then, from the definition N ( λ ; θ 0: m , w 0: m ) = λ 2 ∥ ˆ g ( θ 0: m , w 0: m ) ∥ 2 + Lγ 2 E ξ [ ∥ ∇ ˜ l ξ ( θ ∗ ) -λ ˆ g ξ ( θ 0: m , w 0: m ) ∥ 2 ] , we get (7). To show N ( λ † m ; θ 0: m , w 0: m ) ≤ N (0) , we apply Lemma B.6 as

To show (8), let us assume E ξ [ ⟨∇ ˜ l ξ ( θ ∗ ) , ˆ g ξ ( θ 0: m , w 0: m ) ⟩ ] ≥ 0 . Then, we get the following inequality:

where (18) holds due to a 2 + b 2 ≥ 2 ab ; (19) holds due to the assumption ∥ x ∥≤ C , ∥ · ∥ 2 ≤∥ · ∥ 1 , and the Pinsker's inequality applied as

Thus, we get our desired result by combining the following inequality with (15):

where (24) holds due to 1 -(1 -u ) 2 1+ v ≤ 2 u + v [30].

B.4 Proof of Corollary 3.2.1

Proof. For any realization of θ m for m ∈ { 0 , 1 , · · · , ˆ T -1 } , applying Theorem 3.2 with λ = λ † m (cf. Lemma B.6) gives

Thus, by recursively applying (27) to θ 0 , we get our desired result as

B.5 Supporting lemmas

̸

Lemma B.4. Let A i ∼ Bern ( p i ) with p i ∈ [0 , 1] and S o = ∑ i ∈ [ o ] A i , where A i and A j are independent for i = j . Then for any q < ¯ p := 1 o ∑ i ∈ [ o ] p i , we have the tail probability bound

Proof. For any t , the moment generating function of S m is given by

where the first equality comes from the independence and the inequality is from log x ≤ x -1 . Then, for t < 0 and q < ¯ p , the Chernoff bound [52] gives

Finally, setting t = log ( q/ ¯ p ) < 0 gives our desired result.

Lemma B.5 (Modification from [30]) . Let us assume that l ( θ ) satisfies the L -smoothness, the L -expected smoothness, and the µ -PL condition. Also, we assume a linear model f k ( x ; θ ) = { exp( θ T x ) } k / ∑ i ∈ [ K ] { exp( θ T x ) } i with θ ∈ R d × K . Then, for the stochastic gradient descent θ m,t +1 = θ m,t -γg ( m,t ) ξ with g ( m,t ) ξ = ∇ ˆ E X i ∈ ξ [ H ( f ( X i ; θ m,t ) , ˜ Y ( X i ; θ 0: m , w 0: m )] , θ m, 0 := θ m , and a constant learning rate γ ≤ µ 4 L· L , any realization of θ m satisfies

Proof. For any realization of θ m,t , we have the following bound:

where (33) holds due to L -smoothness; (34) holds due to g ( m,t ) ξ = ∇ l ξ ( θ m,t ) -b ( m,t ) ξ = ∇ ˜ l ξ ( θ m,t ) -λ ˆ g ξ ( θ 0: m , w 0: m ) ; (36) holds due to ⟨ a, b ⟩ ≤ 1 4 ∥ a ∥ 2 + ∥ b ∥ 2 ; (37) holds due to the inequality ∥ x + y ∥ 2 ≤ 2 ∥ x ∥ 2 +2 ∥ y ∥ 2 ; (38) holds due to the expected smoothness assumption; (39) holds due to the condition γ ≤ 1 4 L µ L ; (41) holds due to the law of total expectation with the random event 1 ( Y ( X ) = ˆ Y ( X ; θ m )) and then applying the bounded support assumption.

By recursively applying the above bound to any realization of θ m , we get our desired result as follows:

C Additional results and discussions

C.1 Empirical evidence for Theorem 3.1

(a)

(b)

1.0

1.0

1.0

0.9

Temporal ensemble

Generalized temporal ensemble

0.9

Generalized temporalensemble

Temporal ensemble

0.9

0.8

Pseudo label

0.8

Pseudo label

0.8

0.7

0.7

0.7

ccuracy

0.6

ccura

0.6

ccurac

0.6

B

0.5

B

0.5

AC

0.5

0.4

0.4

0.4

Mean (default)

0.3

0.3

0.3

Pseudo label

0.2

0.2

0.2

Quantile 0.7

Quantile 0.9

0.1

0

20

40

60

80

100

0.1

0

20

40

60

80

100

0.1

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

The number of confident samples

The number of confident samples

The number of confident samples

C.2 Marginal distribution of pseudo labels over epochs

C.3 Algorithmic design choices

Here, we examine effectiveness of our design choices-the relative thresholding for weighting scheme w i ( x ) and hard prediction for p ( y | x, θ i ) . To this end, we replace our design choices with several alternatives, keeping other features of AnCon the same. Then, we analyze the changes in the performance compared to the default setting of AnCon in four domain pairs of OfficeHome.

Sophisticated weighting scheme for w i ( x ) We compare our relative thresholding with sophisticated weighting schemes called Entropy ( w i ( x ) ∝ exp {-H E [ f ( x ; θ i )] } ) and Maxprob ( w i ( x ) ∝ max k ∈ [ K ] f k ( x ; θ i ) ). These weighting schemes aim to more precisely weight the predictions based on the uncertainty of the prediction. However, in Figure 6a, the simple relative thresholding works better than these sophisticated weighting schemes. We conjecture that the poor calibration performance of

(a)

(b)

1000

5000

0.070

800

0.065

4000

0.060

Total Variation

600

Count

3000

Class

0.055

0.050

2000

400

0.045

1000

200

0.040

0.035

0

0

10

20

30

40

50

60

70

80

90

0

20

40

60

80

100

Epochs

Epochs

(a)

(b)

3

3

（%)

2

(%)

2

Change

Change

1

RelativePerformance

RelativePerformance

D

-1

-1

-2

2

-3

-3

-4

-4

-5

-5

AnCon

Entropy

Maxprob

Dirichlet MC (p=0.2)MC (p=0.5)

AnCon(Hard)T=0.5

T=0.75

T=1.0

T=1.25

T=1.5

Methods

Methods

the neural network in self-training under distribution shifts prevents the estimated uncertainty of a prediction from accurately reflecting the goodness of the prediction. Further, the ineffectiveness of these sophisticated weighting schemes persists even when we improve the calibration performance with last-layer Dirichlet [53] and Monte-Carlo Dropout [54] (see Appendix D.3 for detail). Therefore, our relative thresholding is simple yet effective in self-training under distribution shift.

Soft prediction for p ( y | x, θ i ) We compare hard prediction vs soft prediction with a softmax temperature parameter T ∈ { 0 . 5 , 0 . 75 , 1 . 0 , 1 . 25 , 1 . 5 } (cf. Appendix D.3 for the definition). By design, soft prediction is more informative, giving values in non-leading entries. However, in Figure 6b, soft prediction turns out to underperform hard prediction for various values of T . We conjecture that the continuously increasing confidence in the later stage of self-training would make soft prediction ignore early-stage predictions which may be valuable to memorize. Thus, we believe that hard prediction would be more appropriate for AnCon .

C.4 Limitations and future directions

Albeit AnCon proving its effectiveness with a fixed value of λ , Theorem 3.1 suggests that its optimal value depends on the temporal ensemble performance. Thus, future works could aim for adaptively determining λ . In addition, we observe that all self-training based methods fall short without sophisticated model designs tailored for distribution shifts [16, 31] (cf. Appendix E). In this regard, it would be valuable to rigorously understand properties of function classes that enable successful self-training under distribution shifts. Also, while AnCon significantly relaxes the conditions required to achieve optimality compared to LFN techniques, the on-average correct prediction condition still can be violated in challenging self-training scenarios. Given the compatibility of AnCon and NRC

(cf. Table 1), efficiently combining local and temporal consistencies could be a step toward further relaxing the optimality condition. Finally, we remark that the idea of selective temporal consistency in sequential decision-making scenarios could be a notable future direction of research. Extending AnCon to sequential decision-making scenarios presents significant challenges as they involve the fundamentally different mechanisms. In sequential decision-making, leveraging observed rewards and balancing exploitation and exploration are core aspects (e.g., constructing the upper confidence bound of the reward in the bandit problem), unlike in self-training. Therefore, the main challenges for applying AnCon to this setting would be defining rewards and incorporating exploration strategies into pseudo label generation.

D Additional details

D.1 SFDA training configurations

Our training configuration is based on [16]. Specifically, we train ResNet-50 [29] as a source classifier, which is used as an initial point for SFDA, for 50 epochs, with minibatch stochastic gradient descent with Nesterov momentum by using the initial learning rate of 0.01, the momentum parameter of 0.9, and the minibatch size of 64 in the Office Home dataset. The learning rate is decayed by (1 + 10 · current iteration number / maximum number of iterations ) -0 . 75 , and for the pre-trained layers we use a 10 times smaller learning rate. For Office-31, we change the number of epochs by 100 and other configurations are the same as those used in the setting in Office Home. Similarly, we only change the number of epochs to 10, the learning rate to 0.001, and the architecture to ResNet-101 for VisDa. We split dataset into 90% of the training set and 10% of the validation set, and the best model is chosen based on the lowest error rate on the validation set. For adapting the pre-trained model in the target domain, we use the exactly same configurations, except reducing the number of epochs to 30 for Office-31 and OfficeHome and to 15 for VisDa.

D.2 ImageNet-C training configuration

For applying self-training, we follow the training configuration used in [31]; specifically, given ResNet-50 pretrained on ImageNet, we perform self-training without threshold by running the stochastic gradient descent optimizer with the learning rate of 0.001 and the minibatch size of 100 for 20 epochs. We remark that other techniques such as momentum and learning rate scheduling are not used.

D.3 Further experimental details in analyses

Definition of ECE

where G θ g := { x ∈ D| g G ≤ c ( x ; θ ) < g +1 G } , Acc ( G θ g ) is the average accuracy in G θ g , and Conf ( G θ g ) is the average confidence in G θ g .

Configuration of the sophisticated uncertainty representation methods We used the regularization coefficient of 0.001 for the belief matching loss [53] and 20 forward passes to compute the posterior predictive distribution for MC-dropout with dropout probabilities 0.2 and 0.5 [54].

Definitions of model selection criteria

InfoMax (higher is better)

where H ( p ) = -∑ k ∈ [ K ] p i log p i for p ∈ △ K -1 . Note that we abuse the notation H ( · ) with the definition of the cross-entropy; that is, H ( q, p ) = -∑ k ∈ [ K ] p i log q i and H ( p ) = -∑ k ∈ [ K ] p i log p i .

Corr-C (lower is better)

where C ∈ R K × K and ∥ · ∥ F is the Frobenius norm.

Ent (lower is better)

Softmax with a temperature parameter

where f -ϕ ( x ; θ ) are the logits of the neural network such that f i ( x ; θ ) = exp( f -ϕ i ( x ; θ )) ∑ k ∈ [ K ] exp( f -ϕ k ( x ; θ )) for i ∈ [ K ] and ϕ ( x ) is the softmax function.

D.4 Computational resources for experiments

In this work, we use multiple servers which consist of multiple GPUs including TITAN XP (12GB), RTX 8000 (50GB), and A100 (40GB). AnCon takes 1.16 seconds on average for each iteration.

E Result in UDA

While SFDA does not assume any information, unlabeled target domain samples as domain shift information can be available during the source domain training process in some practical scenarios, e.g., when labeling target domain samples is costly. In this case, UDA methods are used to minimize the potential impact of distribution shifts by matching input marginal probability distributions of source and target domains. Given the prevalence of UDA scenarios in practice, we aim to show whether AnCon and other self-training methods can be effective for the UDA methods (conditional domain adversarial network (CDAN) [55] and maximum mean discrepancy (MDD) [56]) in the adaptation stage. However, as shown in Table 2, all methods decrease the performance of both CDAN and MDD for most cases, albeit the reduction rates in AnCon are smaller than self-training and ELR. We conjecture that this is because the base methods (CDAN and MDD) do not contain the sophisticated tricks, such as adding weight normalization to the final linear layer, which are used in the SFDA literature [16].

F Additional tables

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: In Section 1, we clearly state this work's scope and contributions, which are shown in Theorem 3.1, Theorem 3.2, and extensive experiments in Section 4.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]

Justification: We include "Limitations and future directions" in Section 6 and discuss implications of the assumptions for theoretical results.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We have added all assumptions in the statements of Theorems 3.1 and 3.2 with proofs in Appendix.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We include training configurations of all experiments in Appendix D.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We release our code at https://github.com/tjoo512/ancon .

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We include training configurations of all experiments in Appendix D.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We have included error bars with box plots for experiments that have high variance across different random seeds.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provide computational resources used in the research in the Appendix.

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We thoroughly read the NeurIPS Code of Ethics and have confirmed that our research does not violate any of them.

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [NA]

Justification: This work considers general principle of promoting temporal consistency for improving self-training under distribution shifts. Therefore, it would not have direct societal impacts, which are contributed particularly by our work.

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our experiments concern only standard models with publicly available benchmarks.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We cite papers related to code, data, and models used in our experiments.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: This paper does not include the new assets.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not include crowdsourcing or research with human subjects.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not include crowdsourcing or research with human subjects.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents Anchored confidence (AnCon), a novel self-training algorithm to improve test-time accuracy under distribution shifts. AnCon modifies the standard self-training algorithm by adding a temporal ensemble regularization. This regularizer is constructed by the consistency of temporal ensembles weighting with their predictive confidences. This intuitively helps the model avoid the \"early-learning phenomenon\", i.e., the model exploiting noisy labels after initially learning clean labels. Contributions include:\n1. A new algorithm for memorizing past ensemble predictions to avoid the \"early-learning phenomenon\", enhancing self-training performance under sequential distribution shifts.\n\n2. In terms of theory, the authors provide a rigorous theoretical analysis for the upper bound of the test-time error of the proposed temporal ensembles by concentration inequality in Theorem 3.1. Inspired by Knowledge Distillation (KD), they also design an upper bound of the expected cross-entropy loss between the temporal ensembles and the optimal value in Theorem 3.2.\n\n3. Extensive experiments confirm the hypothesis that the proposed algorithm can improve self-training performance regarding test-time accuracy and calibration under distribution shifts across different hyperparameter choices.\n\nSTRENGTHS:\n- This paper is very well-written and clear to understand the important aspects of the algorithm.\n- Although the temporal ensemble weighing technique reminds me of similarities with Early learning regularization (ERL) in discount factor from past predictions, I still like the AnCon in general because of a **novel** of high-quality uncertainty estimation behavior in neural network self-training.\n- Similar to ERL, AnCon is also more computationally efficient than non-parametric techniques and standard sampling-based Deep Ensembles.\n- The theoretical contribution of this paper is good, yielding a solid AnCon algorithm. Specifically, under mild assumptions, Theorem 3.1 and 3.2 formally explain why AnCon works and how temporal ensemble regularizers can contribute to improving self-training under covariate shifts (details in Contribution 2).\n- Experiments show a significant improvement of the standard self-training algorithm across different distribution shifts like artificial corruptions and real-world domain shifts.\n- The robustness of the proposed model across different settings is also confirmed across different hyperparameter choices of coefficient regularizer $\\lambda$ in Eq. 1 and discount factor $\\beta$ in Eq. 3, different proportional weighting schemes for $w_m(x)$ in Eq. 3, and different soft-hard predictions for $p(y|x,\\theta_i)$ in Eq. 2.\n\nWEAKNESSES:\n- Regarding the proposed algorithm: compared with the standard self-training, AnCon requires pre-defining the regularizer $\\lambda$ and the discount factor for the temporal ensembles $\\beta$. \n - Regarding the theory: Theorem 3.1 requires $\\bar{p}(x;\\mathbf{c}_{0:m}) \\geq 1/2$, i.e., at least 50% accuracy on average for the temporal ensembles. This is a quite strong assumption.\n - Regarding the experiments: (1) The quantitative results in all tables are reported without the standard deviation and significant tests; (2) The improvement in calibration is shown without any theoretical evidence. Fig. 3 (a) only shows ECE without the reliability diagram, making it hard to assess the correlation between the accuracy and uncertainty of softmax’s probability; (3) Lacks of comparison with other baselines, e.g., many baselines [1] in UDA could be added to compare in Appendix C.\n  - Others that are mentioned by the authors: the challenge of self-training methods under distribution shifts and the violation of average correct predictions in self-training.\n- Miscellaneous: (1) Although assumptions A.1, A.2, and A.3 are mild and have been cited, I believe it is still worth discussing how they are realistic with the paper's setting; (2) Some mathematical notations are vague and perhaps need to be polished (e.g., $f(x;\\theta)$ or $f_k(x;\\theta)$ in L-78, how $\\bar{f}(\\cdot)$ in Eq. 1 connect to the Def. $\\bar{f_k}(\\cdot)$ in Eq. 2, $\\hat{\\mathbb{E}}$ in Eq. 3 seems not to be defined, the same notation $\\sigma$ for different $\\sigma_*^2$ in Theorem 3.2 and perhaps the softmax $\\sigma_k(x;\\theta_i)$ in L-339, more in Question 2); (3) I find it quite hard to follow acronyms without links, I would recommend using \\usepackage[acronym]{glossaries} so readers can smoothly follow the paper; (4) Some Figure’s x-y-titles are tiny, e.g., Fig. 3 (b).\n\nQUESTIONS:\n1. The connection of AnCon with KD is not clear to me. The pseudo label $\\hat{Y}(x;\\theta)$ in self-training with AnCon and the true label  $Y(X)$ in L-163 ($Y(x)$ in L-74) are completely different because $Y(X)$ is unobservable while $\\hat{Y}(x;\\theta)$ is model’s prediction. So in Section 3.3, the statement in L-165-166 to connect AnCon with KD is still wrapping my head. In particular, why AnCon's pseudo-label can be considered as $Y(x)$ (perhaps the true label $Y(X)$ in L-163)? Also, I am curious about why Eq. (40) can lead to Eq. (41), i.e., how we can bound the norm of gradient loss $||\\nabla l(\\theta_{m,t}) - \\nabla \\tilde{l}(\\theta_m)||^2$ by the $Err(\\hat{Y};\\theta_{m})$ while this upper bound related the true label $Y(X)$?\n2. Some questions for clarification: is $\\delta$ in L-83 $\\in [0,1]$? $\\sum_0^m$ or $\\sum_0^{m-1}$ in Eq. 3? $\\theta_k$ or $\\theta_m$ in $Err_{XY}(\\cdot)$ in L-184? $\\mathbf{w}\\_{1:m}$ or $\\mathbf{w}_{0:m}$ in RHS Eq. in L-186?\n3. Compared with ELR which computes the temporal ensembles by the logit vector $f(x;\\theta_i)$ only in L-83, the proposed algorithm requires computing $\\mathbb{E}\\_{x\\in X}\\max_{k\\in [K]}f_k(x;\\theta_i)$ in Eq. 3, does this causes a higher computational demand?\n4. Can you generate a reliability diagram [2] to show the overconfidence and underconfidence of methods? I think this could be easier to assess how your method can improve the model's uncertainty quality.\n6. In the testing of different weighting schemes in Fig. 4 at Section 4.4, have you considered the setting of a uniform weighting for $\\mathbf{w}$, i.e., set Eq. 2 to $\\bar{f_k}(x;\\theta_{0:m},\\mathbb{w}_{0:m}):= \\sum_i^m p(y=k|x;\\theta_i)$? This is a simple setting but no longer temporal ensembles by discount factors, and also not standard (vanilla) self-training. I am quite curious about this result.\n5. Do the authors think your proposed method can work in other sequential decision-making, e.g., bandits, RL, bayesian optimization, active learning, etc.? If yes, could you please give some comments? Otherwise, could you please raise some challenges?\n\nReferences:\n\n[1] Fang et al., Source-Free Unsupervised Domain Adaptation: A Survey, arXiv, 2023.\n\n[2] Guo et al., On Calibration of Modern Neural Networks, ICML, 2017.\n\nLIMITATIONS:\nPlease see my comments on the Weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper aims to improve self-training, a common technique used for learning on unlabeled data (with iteratively generated pseudo labels), when it is applied to scenarios with distribution shifts. The proposed approach, Anchored Confidence (AnCon), essentially applies label smoothing on the pseudo labels with the ensemble prediction of the models obtained from previous iterations (temporal ensemble) to achieve better temporal consistency and improve the quality of the pseudo labels. Theoretical analysis is provided on the benefits of using temporal ensemble and the connection to ensemble knowledge distillation in the linear case. Empirically, when AnCon is shown both **effective** and **stable** when combined with a few orthogonal baselines such as Self-training and GCE under settings such as unsupervised domain adaptation (Sec. 4.1), robustness against label corruption (Sec. 4.2), calibration, etc. (Sec. 4.3).\n\nSTRENGTHS:\n1. The main paper is well-written and pleasant to read. \n2. The paper effectively modifies and adapts Early Learning Regularization (ELR) to the setting of self-training under distribution shifts with both theoretical justification and empirical evidence. The theoretical derivations are generally rigorous with assumptions clearly stated. \n3. The connection to self-distillation helps to better understand the approach. \n4. The empirical and ablation studies are comprehensive and clearly discuss a few critical criteria such as model selection, hyperparameter, and calibration.\n\nWEAKNESSES:\n1. Empirically, the approach improves over various baselines when being added, but what about the existing SoTA baselines for unsupervised domain adaptations, which seem relevant to this work. Besides, it seems rather odd to only report the results in Sec. 4 with non-optimal $\\lambda=0.3$, as the sensitivity analysis in Figure 2(a) clearly prefers much larger $\\lambda$. \n2. A critical assumption in the theorems is that the teacher (temporal ensemble) has a sufficiently good performance. It will be helpful to justify this with more empirical evidence. \n3. Non-trivial typos: In line 149, $x$ is already used in the main theorem, so may want to change $x$ to $z$ to avoid confusion, and this issue also occurred on 643 with $m$, which might have already caused *major* problems in proof (see below in Question 3)\n4. Minor typos: Line 667, Proof of Lemma A.4, Eq. (31), $k$ should be written as $mq$; Some references from NeurIPS 2024 should be 2023. \n5. There might be some mistakes in the theorem proofs. See below in Questions 3 and 4.\n\nQUESTIONS:\n1. The pseudo label $\\tilde{Y}$ is actually also part of the smoothing vector from $\\bar{f}$. I wonder how necessary it is to separate $\\tilde{Y}$ out as its purpose is to increase the weight of the most recent model, or whether there is a more unified notion to describe the proposed approach with just $\\bar{f}$. \n2. In Theorem 3.1, the random events of model predictions from different iterations $i, j$ are assumed to be independent. How realistic is the assumption given that the model is trained based on the labels given by the previous iterations?\n3. Throughout the paper, $m$ is already defined as the number of models, but the proof of Theorem 3.1 assigns $m = Q(x;c_{0;m})/2$, which is the number of sufficiently confident predictions for $x$ made by the $m$ models. I don’t think it’s appropriate to assign $m$ something else here as it is already defined. If you fix the notation of $m$ (of $q \\cdot m$) in Eq. (29) and Eq. (31), Lemma A.4, the bound may look a bit more complex, because the $m$ in Eq. (30) cannot be assigned to a different value. Please clarify. \n4. Moreover, in line 152, the paper claims $\\bar{f}$ is asymptotically correct, but it is unclear to me if the probability on the L.H.S. of Eq. (6) still goes to 0 when $m \\rightarrow \\infty$ after fixing the aforementioned problem with $m$. Please clarify. \n5. Theorem 3.1 studies the asymptotic correctness of the temporal ensemble with large $m$. The experiments mostly show the AnCon under 20 epochs. How does the performance change when more epochs are added? Is it still consistent with the theoretical implications?\n\nLIMITATIONS:\nThe authors have discussed the limitations in Section 6.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nthe authors propose a novel approach to enhance self-training for test-time adaptation (TTA) or source-free domain adaptation (SFDA) in neural networks facing distribution shifts. The core idea revolves around a method called Anchored Confidence (AnCon), which uses temporal ensembles and label smoothing to improve the accuracy and robustness of pseudo labels generated during self-training. This method aims to address the challenge of filtering incorrect pseudo labels, a common issue under distribution shifts, without incurring significant computational overhead. Theoretical guarantees and extensive experiments validate the efficacy of AnCon in diverse distribution shift scenarios.\n\nSTRENGTHS:\n* the paper provides rigorous theoretical analyses to support the proposed method. It shows that the generalized temporal ensemble with prediction confidences is asymptotically correct and that label smoothing can reduce the optimality gap.\n* The method does not require additional forward passes or neighborhood searches, making it computationally efficient compared to existing techniques.\n* experimental results demonstrate improvements in self-training performance across distribution shift scenarios.\n\nWEAKNESSES:\n* The novelty of the paper is limited! (*The integration of temporal consistency and ensembles for self-training under distribution shifts)\n* Although the method shows good empirical results on small datasets, there may be scenarios or datasets where the performance gains are less pronounced, which are not extensively discussed.\n* The success of the method is somewhat dependent on the quality of the initial model parameter $\\theta_0$. If the initial model is significantly biased or underperforming, the improvements might be limited.\n\nQUESTIONS:\n* The paper claims that the generalized temporal ensemble with prediction confidences is asymptotically correct. Can you provide a detailed proof or further elaboration on the conditions under which this asymptotic correctness holds? Specifically, what assumptions about the data distribution or model convergence are necessary to guarantee this property?\n\n* The theoretical analysis suggests that label smoothing can reduce the optimality gap in self-training. Could you explain the underlying mechanism of how label smoothing achieves this reduction? Additionally, what are the theoretical bounds on the optimality gap with and without label smoothing, and how do these bounds depend on the hyperparameters of the smoothing technique?\n\n* How does AnCon perform on highly imbalanced datasets or with extreme distribution shifts?\n* What are the specific hyperparameter settings used in the experiments, and how sensitive is the method to these settings?\n* Can the proposed method be integrated with other advanced TTA or SFDA techniques, and what would be the potential benefits or drawbacks?\n* How does the method handle concept shifts, where p(Y|X) changes between the training and test distributions?\n* are there any specific types of neural network architectures or tasks (e.g., vision, NLP) where AnCon performs exceptionally well or poorly?\n\nLIMITATIONS:\nPlease see the questions and the weaknesses.  \nMy current decision on this paper is borderline (reject/accept). I look forward to the authors' rebuttal to address the following questions and concerns, which will help in making a final decision.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}