Cycle Self-Training for Domain Adaptation

1 Introduction . Transferring knowledge from a source domain with rich supervision to an unlabeled target domain is an important yet challenging problem . Since deep neural networks are known to be sensitive to subtle change in underlying distributions [ 70 ] , models trained on one labeled dataset often fail to generalize to another unlabeled dataset [ 58 , 1 ] . Unsupervised domain adaptation ( UDA ) addresses the challenge of distributional shift by adapting the source model to the unlabeled target data [ 50 , 43 ] . The mainstream paradigm for UDA is feature adaptation , a.k.a . domain alignment . By reducing the distance of the source and target feature distributions , these methods learn invariant representations to facilitate knowledge transfer between domains [ 34 , 22 , 36 , 54 , 37 , 73 ] , with successful applications in various areas such as computer vision [ 63 , 27 , 77 ] and natural language processing [ 75 , 49 ] . Despite their popularity , the impossibility theories [ 6 ] uncovered intrinsic limitations of learning invariant representations when it comes to label shift [ 74 , 32 ] and shift in the support of domains [ 29 ] . Recently , self-training ( a.k.a . pseudo-labeling ) [ 21 , 78 , 30 , 32 , 47 , 68 ] has been gaining momentum as a promising alternative to feature adaptation . Originally tailored to semi-supervised learning , self-training generates pseudo-labels of unlabeled data , and jointly trains the model with source labels and target pseudo-labels [ 31 , 39 , 30 ] . However , the distributional shift in UDA makes pseudo-labeling more difficult . Directly using all pseudo-labels is risky due to accumulated error and even trivial solution [ 14 ] . Thus previous works tailor self-training to UDA by selecting trustworthy pseudo-labels . Using confidence threshold or reweighting , recent works try to alleviate the negative effect of domain ⇤Corresponding author : Mingsheng Long ( mingsheng @ tsinghua.edu.cn ) 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . shift in standard self-training [ 78 , 47 ] , but they can be brittle and require expensive tweaking of the threshold or weight for different tasks , and their performance gain is still inconsistent . In this work , we first analyze the quality of pseudo-labels with or without domain shift to delve deeper into the difficulty of standard self-training in UDA . On popular benchmark datasets , when the source and target are the same , our analysis indicates that the pseudo-label distribution is almost identical to the ground-truth distribution . However , with distributional shift , their discrepancy can be very large with examples of several classes mostly misclassified into other classes . We also study the difficulty of selecting correct pseudo-labels with popular criteria under domain shift . Although entropy and confidence are reasonable selection criteria for correct pseudo-labels without domain shift , the domain shift makes their accuracy decrease sharply . Our analysis shows that domain shift makes pseudo-labels unreliable and that self-training on selected target instances with accurate pseudo-labels is less successful . Thereby , more principled improvement of standard self-training should be tailored to UDA and address the domain shift explicitly . In this work , we propose Cycle Self-Training ( CST ) , a principled self-training approach to UDA , which overcomes the limitations of standard self-training ( see Figure 1 ) . Different from previous works to select target pseudo-labels with hard-to-tweak protocols , CST learns to generalize the pseudo-labels across domains . Specifically , CST cycles between the use of target pseudo-labels to train a target classifier , and the update of shared representations to make the target classifier perform well on the source data . In contrast to the standard Gibbs entropy that makes the target predictions over-confident , we propose a confidence-friendly uncertainty measure based on the Tsallis entropy in information theory , which adaptively minimizes the uncertainty without manually tuning or setting thresholds . Our method is simple and generally applicable to vision and language tasks with various backbones . We empirically evaluate our method on a series of standard UDA benchmarks . Results indicate that CST outperforms previous state-of-the-art methods in 21 out of 25 tasks for object recognition and sentiment classification . Theoretically , we prove that the minimizer of CST objective is endowed with general guarantees of target performance . We also study hard cases on specific distributions , showing that CST recovers target ground-truths while both feature adaptation and standard self-training fail . 2 Preliminaries . We study unsupervised domain adaptation ( UDA ) . Consider a source distribution P and a target distribution Q over the input-label space X ⇥ Y . We have access to ns labeled i.i.d . samples bP = { xsi , ysi } ns i=1 from P and nt unlabeled i.i.d . samples bQ = { xti } nt i=1 from Q . The model f comprises a feature extractor h parametrized by and a head ( linear classifier ) g✓ parametrized by ✓ , i.e . f✓ , ( x ) = g✓ ( h ( x ) ) . The loss function is ` ( · , · ) . Denote by LP ( ✓ , ) : = E ( x , y ) ⇠P ` ( f✓ , ( x ) , y ) the expected error on P . Similarly , we use L bP ( ✓ , ) to denote the empirical error on dataset bP . We discuss two mainstream UDA methods and their formulations : feature adaptation and self-training . Feature Adaptation trains the model f on the source dataset bP , and simultaneously matches the source and target distributions in the representation space Z = h ( X ) : min ✓ , L bP ( ✓ , ) + d ( h ] bP , h ] bQ ) . ( 1 ) Here , h ] bP denotes the pushforward distribution of bP , and d ( · , · ) is some distribution distance . For instance , Long et al . [ 34 ] used maximum mean discrepancy dMMD , and Ganin et al . [ 22 ] approximated the H H-distance dH H [ 7 ] with adversarial training . Despite its pervasiveness , recent works have shown the intrinsic limitations of feature adaptation under real-world situations [ 6 , 74 , 33 , 32 , 29 ] . Self-Training is considered a promising alternative to feature adaptation . In this work we mainly focus on pseudo-labeling [ 31 , 30 ] . Stemming from semi-supervised learning , standard self-training trains a source model fs on the source dataset bP : min✓s , s L bP ( ✓s , s ) . The target pseudo-labels are then generated by fs on the target dataset bQ . To leverage unlabeled target data , self-training trains the model on the source and target datasets together with source ground-truths and target pseudo-labels : min ✓ , L bP ( ✓ , ) + Ex⇠ bQ ` ( f✓ , ( x ) , argmax i { f✓s , s ( x ) [ i ] } ) . ( 2 ) Self-training also uses label-sharpening as a standard protocol [ 31 , 57 ] . Another popular variant of pseudo-labeling is the teacher-student model [ 4 , 61 ] , which iteratively improves the quality of pseudo-labels via alternatively replacing ✓s and s with ✓ and of the previous iteration . 2.1 Limitations of Standard Self-Training . Standard self-training with pseudo-labels uses unlabeled data efficiently for semi-supervised learning [ 31 , 39 , 57 ] . Here we carry out exploratory studies on the popular VisDA-2017 [ 45 ] dataset using ResNet-50 backbones . We find that domain shift makes the pseudo-labels biased towards several classes and thereby unreliable in UDA . See Appendix C.1 for details and results on more datasets . Pseudo-label distributions with or without domain shift . We resample the original VisDA-2017 to simulate different relationship between source and target domains : 1 ) i.i.d. , 2 ) covariate shift , and 3 ) label shift . We train the model on the three variants of source dataset and use it to generate target pseudo-labels . We show the distributions of target ground-truths and pseudo-labels in Figure 2 ( Left ) . When the source and target distributions are identical , the distribution of pseudo-labels is almost the same as ground-truths , indicating the reliability of pseudo-labels . In contrast , when exposed to label shift or covariate shift , the distribution of pseudo-labels is significantly different from target ground-truths . Note that classes 2 , 7 , 8 and 12 appear rarely in the target pseudo-labels in the covariate shift setting , indicating that the pseudo-labels are biased towards several classes due to domain shift . Self-training with these pseudo-labels is risky since it may lead to misalignment of distributions and misclassify many examples of classes 2 , 7 , 8 and 12 . Change of pseudo-label distributions throughout training . To further study the change of pseudolabels in standard self-training , we compute the total variation ( TV ) distance between target groundtruths and target pseudo-labels : dTV ( c , c0 ) = 12 P i kci c 0 ik , where ci is the ratio of class i . We plot its change during training in Figure 2 ( Middle ) . Although the error rate of pseudo-labels continues to decrease , dTV remains almost unchanged at 0.26 throughout training . Note that dTV is the lower bound of the error rate of the pseudo-labels ( shown in Appendix C.1 ) . If dTV converges to 0.26 , then the accuracy of pseudo-labels is upper-bounded by 0.74 . This indicates that the important denoising ability [ 66 ] of pseudo-labels in standard self-training is hindered by domain shift . Difficulty of selecting reliable pseudo-labels under domain shift . To mitigate the negative effect of false pseudo-labels , recent works proposed to select correct pseudo-labels based on thresholding the entropy or confidence criteria [ 35 , 21 , 37 , 57 ] . However , it remains unclear whether these strategies are still effective under domain shift . Here we compare the quality of pseudo-labels selected by different strategies with or without domain shift . For each strategy , we compute False Positive Rate and True Positive Rate for different thresholds and plot its ROC curve in Figure 2 ( Right ) . When the source and target distributions are identical , both entropy and confidence are reasonable strategies for selecting correct pseudo-labels ( AUC=0.89 ) . However , when the target pseudo-labels are generated by the source model , the quality of pseudo-labels decreases sharply under domain shift ( AUC=0.78 ) . 3 Approach . We present Cycle Self-Training ( CST ) to improve pseudo-labels under domain shift . An overview of our method is given in Figure 1 . Cycle Self-Training iterates between a forward step and a reverse step to make self-trained classifiers generalize well on both target and source domains . 3.1 Cycle Self-Training . Forward Step . Similar to standard self-training , we have a source classifier ✓s trained on top of the shared representations on the labeled source domain , and use it to generate target pseudo-labels as y 0 = argmax i { f✓s , ( x ) [ i ] } , ( 3 ) for each x in the target dataset bQ . Traditional self-training methods use confidence thresholding or reweighting to select reliable pseudo-labels . For example , Sohn et al . [ 57 ] select pseudo-labels with softmax value and Long et al . [ 37 ] add entropy reweighting to rely on examples with more confidence prediction . However , the output of deep networks is usually miscalibrated [ 25 ] , and is not necessarily related to the ground-truth confidence even on the same distribution . In domain adaptation , as shown in Section 2.1 , the discrepancy between the source and target domains makes pseudo-labels even more unreliable , and the performance of commonly used selection strategies is also unsatisfactory . Another drawback is the expensive tweaking in order to find the optimal confidence threshold for new tasks . To better apply self-training to domain adaptation , we expect that the model can gradually refine the pseudo-labels by itself without the cumbersome selection or thresholding . Reverse Step . We design a complementary step with the following insights to improve self-training . Intuitively , the labels on the source domain contain both useful information that can transfer to the target domain and harmful information that can make pseudo-labels incorrect . Similarly , reliable pseudo-labels on the target domain can transfer to the source domain in turn , while models trained with incorrect pseudo-labels on the target domain can not transfer to the source domain . In this sense , if we explicitly train the model to make target pseudo-labels informative of the source domain , we can gradually make the pseudo-labels more accurate and learn to generalize to the target domain . Specifically , with the pseudo-labels y0 generated by the source classifier ✓s at hand as in equation 3 , we train a target head ✓̂t ( ) on top of the representation with pseudo-labels on the target domain bQ , ✓̂t ( ) = argmin ✓ Ex⇠ bQ ` ( f✓ , ( x ) , y 0 ) . ( 4 ) We wish to make the target pseudo-labels informative of the source domain and gradually refine them . To this end , we update the shared feature extractor to predict accurately on the source domain and jointly enforce the target classifier ✓̂t ( ) to perform well on the source domain . This naturally leads to the objective of Cycle Self-Training : minimize ✓s , LCycle ( ✓s , ) : = L bP ( ✓s , ) + L bP ( ✓̂t ( ) , ) . ( 5 ) Bi-level Optimization . The objective in equation 5 relies on the solution ✓̂t ( ) to the objective in equation 4 . Thus , CST formulates a bi-level optimization problem . In the inner loop we generate target pseudo-labels with the source classifier ( equation 3 ) , and train a target classifier with target pseudo-labels ( equation 4 ) . After each inner loop , we update the feature extractor for one step in the outer loop ( equation 5 ) , and start a new inner loop again . However , since the inner loop of the optimization in equation 4 only involves the light-weight linear head ✓t , we propose to calculate the analytical form of ✓̂t ( ) and directly back-propagate to the feature extractor instead of calculating the second-order derivatives as in MAML [ 18 ] . The resulting framework is as fast as training two heads jointly . Also note that the solution ✓̂t ( ) relies on ✓s implicitly through y0 . However , both standard self-training and our implementation use label sharpening , making y0 not differentiable . Thus we follow vanilla self-training and do not consider the gradient of ✓̂t ( ) w.r.t . y0 in the outer loop optimization . We defer the derivation and implementation of bi-level optimization to Appendix B.2 .

Summary: The paper tackles the problem of domain adaptation with a newly proposed cycle self-training algorithm. Given observed that the pseudo labels are noisy and existing de-noising methods require ad-hoc hyper-parameters for specific tasks, the authors attempt to progressively refine the pseudo labels by the capability of the network itself. With the intuition of transferring the source-domain knowledge to the target domain, the authors propose to regularize the pseudo labels with a cyclic training pipeline. Tsallis Entropy is further introduced to improve the label quality. Experiments on both visual classification and linguistic sentiment classification indicate the effectiveness of the introduced method.