Cycle Self-Training for Domain Adaptation

1 Introduction . Transferring knowledge from a source domain with rich supervision to an unlabeled target domain is an important yet challenging problem . Since deep neural networks are known to be sensitive to subtle change in underlying distributions [ 70 ] , models trained on one labeled dataset often fail to generalize to another unlabeled dataset [ 58 , 1 ] . Unsupervised domain adaptation ( UDA ) addresses the challenge of distributional shift by adapting the source model to the unlabeled target data [ 50 , 43 ] . The mainstream paradigm for UDA is feature adaptation , a.k.a . domain alignment . By reducing the distance of the source and target feature distributions , these methods learn invariant representations to facilitate knowledge transfer between domains [ 34 , 22 , 36 , 54 , 37 , 73 ] , with successful applications in various areas such as computer vision [ 63 , 27 , 77 ] and natural language processing [ 75 , 49 ] . Despite their popularity , the impossibility theories [ 6 ] uncovered intrinsic limitations of learning invariant representations when it comes to label shift [ 74 , 32 ] and shift in the support of domains [ 29 ] . Recently , self-training ( a.k.a . pseudo-labeling ) [ 21 , 78 , 30 , 32 , 47 , 68 ] has been gaining momentum as a promising alternative to feature adaptation . Originally tailored to semi-supervised learning , self-training generates pseudo-labels of unlabeled data , and jointly trains the model with source labels and target pseudo-labels [ 31 , 39 , 30 ] . However , the distributional shift in UDA makes pseudo-labeling more difficult . Directly using all pseudo-labels is risky due to accumulated error and even trivial solution [ 14 ] . Thus previous works tailor self-training to UDA by selecting trustworthy pseudo-labels . Using confidence threshold or reweighting , recent works try to alleviate the negative effect of domain ⇤Corresponding author : Mingsheng Long ( mingsheng @ tsinghua.edu.cn ) 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . shift in standard self-training [ 78 , 47 ] , but they can be brittle and require expensive tweaking of the threshold or weight for different tasks , and their performance gain is still inconsistent . In this work , we first analyze the quality of pseudo-labels with or without domain shift to delve deeper into the difficulty of standard self-training in UDA . On popular benchmark datasets , when the source and target are the same , our analysis indicates that the pseudo-label distribution is almost identical to the ground-truth distribution . However , with distributional shift , their discrepancy can be very large with examples of several classes mostly misclassified into other classes . We also study the difficulty of selecting correct pseudo-labels with popular criteria under domain shift . Although entropy and confidence are reasonable selection criteria for correct pseudo-labels without domain shift , the domain shift makes their accuracy decrease sharply . Our analysis shows that domain shift makes pseudo-labels unreliable and that self-training on selected target instances with accurate pseudo-labels is less successful . Thereby , more principled improvement of standard self-training should be tailored to UDA and address the domain shift explicitly . In this work , we propose Cycle Self-Training ( CST ) , a principled self-training approach to UDA , which overcomes the limitations of standard self-training ( see Figure 1 ) . Different from previous works to select target pseudo-labels with hard-to-tweak protocols , CST learns to generalize the pseudo-labels across domains . Specifically , CST cycles between the use of target pseudo-labels to train a target classifier , and the update of shared representations to make the target classifier perform well on the source data . In contrast to the standard Gibbs entropy that makes the target predictions over-confident , we propose a confidence-friendly uncertainty measure based on the Tsallis entropy in information theory , which adaptively minimizes the uncertainty without manually tuning or setting thresholds . Our method is simple and generally applicable to vision and language tasks with various backbones . We empirically evaluate our method on a series of standard UDA benchmarks . Results indicate that CST outperforms previous state-of-the-art methods in 21 out of 25 tasks for object recognition and sentiment classification . Theoretically , we prove that the minimizer of CST objective is endowed with general guarantees of target performance . We also study hard cases on specific distributions , showing that CST recovers target ground-truths while both feature adaptation and standard self-training fail . 2 Preliminaries . We study unsupervised domain adaptation ( UDA ) . Consider a source distribution P and a target distribution Q over the input-label space X ⇥ Y . We have access to ns labeled i.i.d . samples bP = { xsi , ysi } ns i=1 from P and nt unlabeled i.i.d . samples bQ = { xti } nt i=1 from Q . The model f comprises a feature extractor h parametrized by and a head ( linear classifier ) g✓ parametrized by ✓ , i.e . f✓ , ( x ) = g✓ ( h ( x ) ) . The loss function is ` ( · , · ) . Denote by LP ( ✓ , ) : = E ( x , y ) ⇠P ` ( f✓ , ( x ) , y ) the expected error on P . Similarly , we use L bP ( ✓ , ) to denote the empirical error on dataset bP . We discuss two mainstream UDA methods and their formulations : feature adaptation and self-training . Feature Adaptation trains the model f on the source dataset bP , and simultaneously matches the source and target distributions in the representation space Z = h ( X ) : min ✓ , L bP ( ✓ , ) + d ( h ] bP , h ] bQ ) . ( 1 ) Here , h ] bP denotes the pushforward distribution of bP , and d ( · , · ) is some distribution distance . For instance , Long et al . [ 34 ] used maximum mean discrepancy dMMD , and Ganin et al . [ 22 ] approximated the H H-distance dH H [ 7 ] with adversarial training . Despite its pervasiveness , recent works have shown the intrinsic limitations of feature adaptation under real-world situations [ 6 , 74 , 33 , 32 , 29 ] . Self-Training is considered a promising alternative to feature adaptation . In this work we mainly focus on pseudo-labeling [ 31 , 30 ] . Stemming from semi-supervised learning , standard self-training trains a source model fs on the source dataset bP : min✓s , s L bP ( ✓s , s ) . The target pseudo-labels are then generated by fs on the target dataset bQ . To leverage unlabeled target data , self-training trains the model on the source and target datasets together with source ground-truths and target pseudo-labels : min ✓ , L bP ( ✓ , ) + Ex⇠ bQ ` ( f✓ , ( x ) , argmax i { f✓s , s ( x ) [ i ] } ) . ( 2 ) Self-training also uses label-sharpening as a standard protocol [ 31 , 57 ] . Another popular variant of pseudo-labeling is the teacher-student model [ 4 , 61 ] , which iteratively improves the quality of pseudo-labels via alternatively replacing ✓s and s with ✓ and of the previous iteration . 2.1 Limitations of Standard Self-Training . Standard self-training with pseudo-labels uses unlabeled data efficiently for semi-supervised learning [ 31 , 39 , 57 ] . Here we carry out exploratory studies on the popular VisDA-2017 [ 45 ] dataset using ResNet-50 backbones . We find that domain shift makes the pseudo-labels biased towards several classes and thereby unreliable in UDA . See Appendix C.1 for details and results on more datasets . Pseudo-label distributions with or without domain shift . We resample the original VisDA-2017 to simulate different relationship between source and target domains : 1 ) i.i.d. , 2 ) covariate shift , and 3 ) label shift . We train the model on the three variants of source dataset and use it to generate target pseudo-labels . We show the distributions of target ground-truths and pseudo-labels in Figure 2 ( Left ) . When the source and target distributions are identical , the distribution of pseudo-labels is almost the same as ground-truths , indicating the reliability of pseudo-labels . In contrast , when exposed to label shift or covariate shift , the distribution of pseudo-labels is significantly different from target ground-truths . Note that classes 2 , 7 , 8 and 12 appear rarely in the target pseudo-labels in the covariate shift setting , indicating that the pseudo-labels are biased towards several classes due to domain shift . Self-training with these pseudo-labels is risky since it may lead to misalignment of distributions and misclassify many examples of classes 2 , 7 , 8 and 12 . Change of pseudo-label distributions throughout training . To further study the change of pseudolabels in standard self-training , we compute the total variation ( TV ) distance between target groundtruths and target pseudo-labels : dTV ( c , c0 ) = 12 P i kci c 0 ik , where ci is the ratio of class i . We plot its change during training in Figure 2 ( Middle ) . Although the error rate of pseudo-labels continues to decrease , dTV remains almost unchanged at 0.26 throughout training . Note that dTV is the lower bound of the error rate of the pseudo-labels ( shown in Appendix C.1 ) . If dTV converges to 0.26 , then the accuracy of pseudo-labels is upper-bounded by 0.74 . This indicates that the important denoising ability [ 66 ] of pseudo-labels in standard self-training is hindered by domain shift . Difficulty of selecting reliable pseudo-labels under domain shift . To mitigate the negative effect of false pseudo-labels , recent works proposed to select correct pseudo-labels based on thresholding the entropy or confidence criteria [ 35 , 21 , 37 , 57 ] . However , it remains unclear whether these strategies are still effective under domain shift . Here we compare the quality of pseudo-labels selected by different strategies with or without domain shift . For each strategy , we compute False Positive Rate and True Positive Rate for different thresholds and plot its ROC curve in Figure 2 ( Right ) . When the source and target distributions are identical , both entropy and confidence are reasonable strategies for selecting correct pseudo-labels ( AUC=0.89 ) . However , when the target pseudo-labels are generated by the source model , the quality of pseudo-labels decreases sharply under domain shift ( AUC=0.78 ) . 3 Approach . We present Cycle Self-Training ( CST ) to improve pseudo-labels under domain shift . An overview of our method is given in Figure 1 . Cycle Self-Training iterates between a forward step and a reverse step to make self-trained classifiers generalize well on both target and source domains . 3.1 Cycle Self-Training . Forward Step . Similar to standard self-training , we have a source classifier ✓s trained on top of the shared representations on the labeled source domain , and use it to generate target pseudo-labels as y 0 = argmax i { f✓s , ( x ) [ i ] } , ( 3 ) for each x in the target dataset bQ . Traditional self-training methods use confidence thresholding or reweighting to select reliable pseudo-labels . For example , Sohn et al . [ 57 ] select pseudo-labels with softmax value and Long et al . [ 37 ] add entropy reweighting to rely on examples with more confidence prediction . However , the output of deep networks is usually miscalibrated [ 25 ] , and is not necessarily related to the ground-truth confidence even on the same distribution . In domain adaptation , as shown in Section 2.1 , the discrepancy between the source and target domains makes pseudo-labels even more unreliable , and the performance of commonly used selection strategies is also unsatisfactory . Another drawback is the expensive tweaking in order to find the optimal confidence threshold for new tasks . To better apply self-training to domain adaptation , we expect that the model can gradually refine the pseudo-labels by itself without the cumbersome selection or thresholding . Reverse Step . We design a complementary step with the following insights to improve self-training . Intuitively , the labels on the source domain contain both useful information that can transfer to the target domain and harmful information that can make pseudo-labels incorrect . Similarly , reliable pseudo-labels on the target domain can transfer to the source domain in turn , while models trained with incorrect pseudo-labels on the target domain can not transfer to the source domain . In this sense , if we explicitly train the model to make target pseudo-labels informative of the source domain , we can gradually make the pseudo-labels more accurate and learn to generalize to the target domain . Specifically , with the pseudo-labels y0 generated by the source classifier ✓s at hand as in equation 3 , we train a target head ✓̂t ( ) on top of the representation with pseudo-labels on the target domain bQ , ✓̂t ( ) = argmin ✓ Ex⇠ bQ ` ( f✓ , ( x ) , y 0 ) . ( 4 ) We wish to make the target pseudo-labels informative of the source domain and gradually refine them . To this end , we update the shared feature extractor to predict accurately on the source domain and jointly enforce the target classifier ✓̂t ( ) to perform well on the source domain . This naturally leads to the objective of Cycle Self-Training : minimize ✓s , LCycle ( ✓s , ) : = L bP ( ✓s , ) + L bP ( ✓̂t ( ) , ) . ( 5 ) Bi-level Optimization . The objective in equation 5 relies on the solution ✓̂t ( ) to the objective in equation 4 . Thus , CST formulates a bi-level optimization problem . In the inner loop we generate target pseudo-labels with the source classifier ( equation 3 ) , and train a target classifier with target pseudo-labels ( equation 4 ) . After each inner loop , we update the feature extractor for one step in the outer loop ( equation 5 ) , and start a new inner loop again . However , since the inner loop of the optimization in equation 4 only involves the light-weight linear head ✓t , we propose to calculate the analytical form of ✓̂t ( ) and directly back-propagate to the feature extractor instead of calculating the second-order derivatives as in MAML [ 18 ] . The resulting framework is as fast as training two heads jointly . Also note that the solution ✓̂t ( ) relies on ✓s implicitly through y0 . However , both standard self-training and our implementation use label sharpening , making y0 not differentiable . Thus we follow vanilla self-training and do not consider the gradient of ✓̂t ( ) w.r.t . y0 in the outer loop optimization . We defer the derivation and implementation of bi-level optimization to Appendix B.2 .

Summary: This paper proposes an unsupervised domain adaptation method based on cyclical self-training. There are two main components to the method: 1. a cyclical self-training algorithm that solves a bilevel optimization problem based on an in the inner loop which trains a target classifier with target pseudo-labels  and an outer loop that makes the target classifier perform well on the source domain by updating the shared representations in the outer loop. 2. an uncertainty measure based on the Tsallis entropy, which adaptively minimizes the uncertainty in target pseudo-labels, which replaces the standard Gibbs entropy used by self-training methods. Experiments on computer vision and NLP datasets show that proposed method outperforms baseline methods by a decent margin. Ablation studies show that both components of the algorithm---cyclical self-training and Tsallis entropy---meaningfully contribute to performance improvement. Qualitative and quantitative studies show that target pseudo-label quality is improved by the proposed approach.