Distributionally Robust Learning for Unsupervised Domain Adaptation

1 INTRODUCTION . In many real-world applications , the target domain for deployment of a machine-learning ( ML ) model can significantly differ from the source training domain . Furthermore , labels in the target domain are often more expensive to obtain compared to the source domain . An example is synthetic training where the source domain has complete supervision while the target domain of real images may not be labeled . Unsupervised domain adaptation ( UDA ) aims to maximize performance on the target domain , and it utilizes both the labeled source data and the unlabeled target data . A popular framework for UDA involves obtaining proxy labels in the target domain through selftraining ( Zou et al. , 2019 ) . Self-training starts with a classifier trained on the labeled source data . It then iteratively obtains pseudo-labels in the target domain using predictions from the current ML model . However , this process is brittle , since wrong pseudo-labels in the target domain can lead to catastrophic failure in early iterations ( Kumar et al. , 2020 ) . To avoid this , self training needs to be conservative and select only pseudo-labels with sufficiently high confidence level . This entails accurate knowledge of the confidence levels . Accurate confidence estimation is a challenge for current deep learning models . Deep learning models tend to produce over-confident and misleading probabilities , even when predicting on the same distribution ( Guo et al. , 2017a ; Gal & Ghahramani , 2016 ) . Some attempts to remedy this issue include temperature scaling ( Platt et al. , 1999 ) , Monte-Carlo sampling ( Gal & Ghahramani , 2016 ) and Bayesian inference ( Blundell et al. , 2015 ; Riquelme et al. , 2018 ) . However , Snoek et al . ( 2019 ) has shown that the uncertainty estimation from these models can not be trusted under domain shifts . In this paper , we instead consider the distributionally robust learning ( DRL ) framework ( Liu & Ziebart , 2014 ; 2017 ) which provides a principled approach for uncertainty quantification under domain shifts . DRL can be formulated as a two-player adversarial risk minimization game , as depicted in Figure 1 ( a ) . Recall that the standard framework of empirical risk minimization ( ERM ) directly learns a predictor P̂ ( Y |X ) from training data . In contrast , DRL also includes an adversaryQ ( Y |X ) that is allowed to perturb the labels , subject to certain feature-matching constraints to ensure datacompatibility . Formally , the minimax game for DRL is : min P̂ ( Y |X ) max Q ( Y |X ) lossPt ( X ) ( P̂ ( Y |X ) , Q ( Y |X ) ) , ( 1 ) where the adversary Q ( Y |X ) is constrained to match the evaluation of a set of features Φ ( x , y ) to that of the source distribution ( see Section 2 for details ) . Note that the loss in ( 1 ) is evaluated under the target input distribution Pt ( X ) , and the predictor does not have direct access to the source data { Xs , Ys } . Instead , the predictor optimizes the target loss by playing a game with an adversary constrained by source data . A special case of UDA is the covariate shift setting , where the label-generating distribution P ( Y |X ) is assumed to be the same in both source and target domains . Under this assumption , with log-loss and a linear predictor parameterized by θ and features Φ ( x , y ) , ( 1 ) reduces to : P̂ ( y|x ) ∝ exp ( Ps ( x ) Pt ( x ) θ · Φ ( x , y ) ) . ( 2 ) Intuitively , the density ratio Ps ( x ) /Pt ( x ) prevents the model from being overconfident on target inputs far away from the source domain . Thus , the DRL framework is a principled approach for conservative confidence estimation . Previous works have shown that DRL is highly effective in safety-critical applications such as safe exploration in control systems ( Liu et al. , 2020 ) and safe trajectory planning ( Nakka et al. , 2020 ) . However , these works only consider estimating the density ratio in low dimensions ( e.g . control inputs ) using standard kernel density estimator ( KDE ) and extending it to high-dimensional inputs such as images remains an open challenge . Moreover , it is not clear if the covariate-shift assumption holds for common high-dimensional settings such as images — which we investigate in this paper . In this paper , we propose a novel deep-learning method based on the DRL framework for accurate uncertainties that scales to modern domain-adaptation tasks in computer vision . Summary of Contributions : 1 . We develop differentiable density-ratio estimation as part of the DRL framework to enable efficient end-to-end training . See Figure 1 ( b ) . 2 . We employ DRL ’ s confidence estimation in the self-training framework for domain adaptation and term it as distributionally robust self-training ( DRST ) . See Figure 2 . 3 . We further combine it with automated synthetic to real generalization ( ASG ) framework of ( Chen et al. , 2020b ) to improve generalization in the real target domain when the source domain consists of synthetic images . 4 . We demonstrate that DRST generates more calibrated probabilities . DRST-ASG achieves competitive accuracy on the VisDA2017 dataset ( Peng et al. , 2017 ) with 1 % improvement over the baseline class-regularized self-training ( CRST ) using the standard soft-max confidence measure . 5 . We analyze the reason for the effectiveness of DRST through a careful ablation study . One challenge for training DRL is that the training loss can not be directly evaluated under the UDA framework . However , we show that the gradients of the target loss can indeed be evaluated . By deriving gradients for both neural networks and proposing a joint training algorithm ( Alg . 1 ) , we show the network can be trained efficiently . We also directly incorporate class regularization in the minimax game under our DRL framework . This is a principled approach in contrast to standard label smoothing incorporated on top of a given learning method . In our ablation studies , we observe that the covariate-shift assumption progressively holds to a greater extent as the iterations in self-training proceed . This is also correlated with the greater ability to capture shape features through self training , as seen in the Grad-CAM visualization ( Selvaraju et al. , 2017 ) . 2 PROPOSED FORMULATION AND ALGORITHMS . In this section , we first introduce the class regularized DRL framework ( 2.1 ) and then propose differentiable density ratio estimation to enable end-to-end learning of DRL using neural networks . We provide training details , especially the gradient computation for training both networks ( 2.2 ) . This is unique for our setting since the actual training loss on target can not be evaluated due to lack of target labels . Finally , we propose our self-training algorithm DRST in 2.3 . 2.1 DISTRIBUTIONALLY ROBUST LEARNING WITH CLASS REGULARIZATION . We are interested in robustly minimizing the classification loss on the target domain with confidence of adversary ’ s prediction regularized . We use a weighted logloss term to penalize high confidence in the adversary ’ s label prediction as the regularization . We make the same covariate shift assumption as in ( Liu & Ziebart , 2014 ) that only the marginal input distribution changes and P ( y|x ) is shared between source and target : Ps ( x ) 6= Pt ( x ) and Ps ( y|x ) = Pt ( y|x ) . We aim to solve the following : min P̂ ( Y |X ) max Q ( Y |X ) ∈Ξ loglossPt ( X ) ( P̂ ( Y |X ) , Q ( Y |X ) ) − rEPt ( x ) Q ( y|x ) [ Y log P̂ ( Y |X ) ] , ( 3 ) where Y is a one-hot encoding of the class and r ∈ [ 0 , 1 ] is a hyper-parameter that controls the level of regularization . In this formulation , the estimator player P̂ ( Y |X ) first chooses a conditional label distribution to minimize the regularized logloss on the target domain and then an adversarial playerQ ( Y |X ) chooses a conditional label distribution from the set ( Ξ ) to maximize the regularized logloss . The constraint set Ξ defines how much flexibility we want to give to the adversary . Usually , we design feature functions on both X and Y and restrict the adversary to match statistics of the expectation of the features . We have the following lemma : Lemma 1 . If we choose feature map Φ ( X , Y ) as the statistics to constrainQ ( Y |X ) , then equation 3 can be reduced to a reguralized maximum entropy problem with the estimator constrained : max P̂ ( Y |X ) EPt ( x ) P̂ ( y|x ) [ −log P̂ ( Y |X ) ] − rEPt ( x ) P̂ ( y|x ) [ Y log P̂ ( Y |X ) ] ( 4 ) such that : P̂ ( Y |X ) ∈∆ and |EPs ( x ) P̂ ( y|x ) [ Φ ( X , Y ) ] − EPs ( x , y ) [ Φ ( X , Y ) ] | ≤ λ , where ∆ defines the conditional probability simplex that P̂ ( y|x ) must reside within , Φ is a vectorvalued feature function that is evaluated on input x , and EPs ( x , y ) [ Φ ( X , Y ) ] is a vector of the expected feature values that corresponds with the feature function . λ is the slack term of constraints . The proof of this lemma involves strong duality of the convex-concave function , such that the min and max player can switch the order . We refer more details to the appendix . The following theorem states the solution of the problem : Theorem 1 . The parametric solution of ( 4 ) for P̂ ( y|x ) takes the form : P̂ ( y|x ) ∝ exp Ps ( x ) Pt ( x ) θ · Φ ( x , y ) + ry ry + 1 , ( 5 ) where the parameter θ can be optimized by maximizing the log-likelihood on the target distribution . The gradients take the form : ∇θEPt ( x ) P ( y|x ) [ − log P̂θ ( Y |X ) ] = EP̃s ( x ) P̂ ( y|x ) [ Φ ( X , Y ) ] − c̃ , ( 6 ) where c̃ , EP̃s ( x ) P̃ ( y|x ) [ Φ ( X , Y ) ] , as the empirical evaluation of the feature expectations . Here P̃s ( x ) and P̃ ( y|x ) are the empirical distribution . In principle , P ( y|x ) is the ground truth conditional label distribution shared between source and target domains . We call EPt ( x ) P ( y|x ) [ − log P̂θ ( Y |X ) ] the expected target loss in the paper . Even though it is not available in practice , we can approximate the gradients ( 6 ) using the source data in training . The norm of the approximated gradient converges to the true gradient in the rate of O ( 1/m ) , where m is the amount of source data . The proof involves application of the Lagrangian multiplier , setting the derivative of each specific P̂ ( y|x ) to 0 and utilizing the KKT condition . We refer details to the appendix . Algorithm 1 End-to-end Training for DRL Input : Source data , Target data , DNN φ ) , DNN τ , SGD optimizerOpt1 andOpt2 , learning rate γ1 and γ2 , epoch number T . Initialization : φ , τ ← random initialization , epoch← 0 While epoch < T For each mini-batch Compute ( 10 ) back propagate Back propagate Ld Optimizer Opt1 ( γ1 ) updates β Compute p̂ following ( 5 ) . Compute gradients using ( 9 ) Back-propagate through∇φLc w = w − γ2 · ∇wLc b = b− γ2 · ∇bLc Optimizer Opt2 ( γ2 ) updates α epoch← epoch +1 Output : Trained α , β , w , b We use this form to illustrate the property of representation-level conservativeness and the class-level regularization of our formulation . Representation-level conservativeness : The prediction has higher certainty for inputs closer to the source domain , when magnitude of Ps ( x ) /Pt ( x ) is large . On the contrary , if the inputs are farther away from the source , which means Ps ( x ) /Pt ( x ) is small , the prediction is uncertain . Class-level regularization : Hyper-parameter r adjusts the smoothness of theQ ( Y |X ) ’ s label prediction in ( 3 ) . It translates to the ry terms in the parametric form . In training , we compute the gradients using source labels where y is the one-hot encoding of the class . In testing , we can set y to be all one vector to obtain smoothed confidence . In machine learning methods using density ratios , such as transfer learning ( Pan & Yang , 2009 ) , or off-policy evaluation ( Dudı́k et al. , 2011 ) , a plugin estimator for the density ratio Ps ( x ) /Pt ( x ) is used . However , density ratio estimation ( Sugiyama et al. , 2012 ) , especially in the high-dimensional data , is rather different . It is also not the case that more accurate density ratio estimation would lead to the better downstream task performance . We have a synthetic example shown in Appendix E. To scale up the method for modern domain adaptation tasks , we ask the question : can we train the density ratio estimation and the learning tasks that use the ratios together such that they share the common goal–the target domain predictive performance ?

Summary: I find the paper to be well motivated. Self-labeling has proven to be a useful approach for unsupervised domain adaptation. And since wrong pseudo-labels in the target domain result catastrophic failure in early iterations, it makes sense to calibrate the production of pseudo-labels through the use of uncertainty estimation. This is done through the framework of distributionally robust learning.