Distributionally Robust Learning for Unsupervised Domain Adaptation

1 INTRODUCTION . In many real-world applications , the target domain for deployment of a machine-learning ( ML ) model can significantly differ from the source training domain . Furthermore , labels in the target domain are often more expensive to obtain compared to the source domain . An example is synthetic training where the source domain has complete supervision while the target domain of real images may not be labeled . Unsupervised domain adaptation ( UDA ) aims to maximize performance on the target domain , and it utilizes both the labeled source data and the unlabeled target data . A popular framework for UDA involves obtaining proxy labels in the target domain through selftraining ( Zou et al. , 2019 ) . Self-training starts with a classifier trained on the labeled source data . It then iteratively obtains pseudo-labels in the target domain using predictions from the current ML model . However , this process is brittle , since wrong pseudo-labels in the target domain can lead to catastrophic failure in early iterations ( Kumar et al. , 2020 ) . To avoid this , self training needs to be conservative and select only pseudo-labels with sufficiently high confidence level . This entails accurate knowledge of the confidence levels . Accurate confidence estimation is a challenge for current deep learning models . Deep learning models tend to produce over-confident and misleading probabilities , even when predicting on the same distribution ( Guo et al. , 2017a ; Gal & Ghahramani , 2016 ) . Some attempts to remedy this issue include temperature scaling ( Platt et al. , 1999 ) , Monte-Carlo sampling ( Gal & Ghahramani , 2016 ) and Bayesian inference ( Blundell et al. , 2015 ; Riquelme et al. , 2018 ) . However , Snoek et al . ( 2019 ) has shown that the uncertainty estimation from these models can not be trusted under domain shifts . In this paper , we instead consider the distributionally robust learning ( DRL ) framework ( Liu & Ziebart , 2014 ; 2017 ) which provides a principled approach for uncertainty quantification under domain shifts . DRL can be formulated as a two-player adversarial risk minimization game , as depicted in Figure 1 ( a ) . Recall that the standard framework of empirical risk minimization ( ERM ) directly learns a predictor P̂ ( Y |X ) from training data . In contrast , DRL also includes an adversaryQ ( Y |X ) that is allowed to perturb the labels , subject to certain feature-matching constraints to ensure datacompatibility . Formally , the minimax game for DRL is : min P̂ ( Y |X ) max Q ( Y |X ) lossPt ( X ) ( P̂ ( Y |X ) , Q ( Y |X ) ) , ( 1 ) where the adversary Q ( Y |X ) is constrained to match the evaluation of a set of features Φ ( x , y ) to that of the source distribution ( see Section 2 for details ) . Note that the loss in ( 1 ) is evaluated under the target input distribution Pt ( X ) , and the predictor does not have direct access to the source data { Xs , Ys } . Instead , the predictor optimizes the target loss by playing a game with an adversary constrained by source data . A special case of UDA is the covariate shift setting , where the label-generating distribution P ( Y |X ) is assumed to be the same in both source and target domains . Under this assumption , with log-loss and a linear predictor parameterized by θ and features Φ ( x , y ) , ( 1 ) reduces to : P̂ ( y|x ) ∝ exp ( Ps ( x ) Pt ( x ) θ · Φ ( x , y ) ) . ( 2 ) Intuitively , the density ratio Ps ( x ) /Pt ( x ) prevents the model from being overconfident on target inputs far away from the source domain . Thus , the DRL framework is a principled approach for conservative confidence estimation . Previous works have shown that DRL is highly effective in safety-critical applications such as safe exploration in control systems ( Liu et al. , 2020 ) and safe trajectory planning ( Nakka et al. , 2020 ) . However , these works only consider estimating the density ratio in low dimensions ( e.g . control inputs ) using standard kernel density estimator ( KDE ) and extending it to high-dimensional inputs such as images remains an open challenge . Moreover , it is not clear if the covariate-shift assumption holds for common high-dimensional settings such as images — which we investigate in this paper . In this paper , we propose a novel deep-learning method based on the DRL framework for accurate uncertainties that scales to modern domain-adaptation tasks in computer vision . Summary of Contributions : 1 . We develop differentiable density-ratio estimation as part of the DRL framework to enable efficient end-to-end training . See Figure 1 ( b ) . 2 . We employ DRL ’ s confidence estimation in the self-training framework for domain adaptation and term it as distributionally robust self-training ( DRST ) . See Figure 2 . 3 . We further combine it with automated synthetic to real generalization ( ASG ) framework of ( Chen et al. , 2020b ) to improve generalization in the real target domain when the source domain consists of synthetic images . 4 . We demonstrate that DRST generates more calibrated probabilities . DRST-ASG achieves competitive accuracy on the VisDA2017 dataset ( Peng et al. , 2017 ) with 1 % improvement over the baseline class-regularized self-training ( CRST ) using the standard soft-max confidence measure . 5 . We analyze the reason for the effectiveness of DRST through a careful ablation study . One challenge for training DRL is that the training loss can not be directly evaluated under the UDA framework . However , we show that the gradients of the target loss can indeed be evaluated . By deriving gradients for both neural networks and proposing a joint training algorithm ( Alg . 1 ) , we show the network can be trained efficiently . We also directly incorporate class regularization in the minimax game under our DRL framework . This is a principled approach in contrast to standard label smoothing incorporated on top of a given learning method . In our ablation studies , we observe that the covariate-shift assumption progressively holds to a greater extent as the iterations in self-training proceed . This is also correlated with the greater ability to capture shape features through self training , as seen in the Grad-CAM visualization ( Selvaraju et al. , 2017 ) . 2 PROPOSED FORMULATION AND ALGORITHMS . In this section , we first introduce the class regularized DRL framework ( 2.1 ) and then propose differentiable density ratio estimation to enable end-to-end learning of DRL using neural networks . We provide training details , especially the gradient computation for training both networks ( 2.2 ) . This is unique for our setting since the actual training loss on target can not be evaluated due to lack of target labels . Finally , we propose our self-training algorithm DRST in 2.3 . 2.1 DISTRIBUTIONALLY ROBUST LEARNING WITH CLASS REGULARIZATION . We are interested in robustly minimizing the classification loss on the target domain with confidence of adversary ’ s prediction regularized . We use a weighted logloss term to penalize high confidence in the adversary ’ s label prediction as the regularization . We make the same covariate shift assumption as in ( Liu & Ziebart , 2014 ) that only the marginal input distribution changes and P ( y|x ) is shared between source and target : Ps ( x ) 6= Pt ( x ) and Ps ( y|x ) = Pt ( y|x ) . We aim to solve the following : min P̂ ( Y |X ) max Q ( Y |X ) ∈Ξ loglossPt ( X ) ( P̂ ( Y |X ) , Q ( Y |X ) ) − rEPt ( x ) Q ( y|x ) [ Y log P̂ ( Y |X ) ] , ( 3 ) where Y is a one-hot encoding of the class and r ∈ [ 0 , 1 ] is a hyper-parameter that controls the level of regularization . In this formulation , the estimator player P̂ ( Y |X ) first chooses a conditional label distribution to minimize the regularized logloss on the target domain and then an adversarial playerQ ( Y |X ) chooses a conditional label distribution from the set ( Ξ ) to maximize the regularized logloss . The constraint set Ξ defines how much flexibility we want to give to the adversary . Usually , we design feature functions on both X and Y and restrict the adversary to match statistics of the expectation of the features . We have the following lemma : Lemma 1 . If we choose feature map Φ ( X , Y ) as the statistics to constrainQ ( Y |X ) , then equation 3 can be reduced to a reguralized maximum entropy problem with the estimator constrained : max P̂ ( Y |X ) EPt ( x ) P̂ ( y|x ) [ −log P̂ ( Y |X ) ] − rEPt ( x ) P̂ ( y|x ) [ Y log P̂ ( Y |X ) ] ( 4 ) such that : P̂ ( Y |X ) ∈∆ and |EPs ( x ) P̂ ( y|x ) [ Φ ( X , Y ) ] − EPs ( x , y ) [ Φ ( X , Y ) ] | ≤ λ , where ∆ defines the conditional probability simplex that P̂ ( y|x ) must reside within , Φ is a vectorvalued feature function that is evaluated on input x , and EPs ( x , y ) [ Φ ( X , Y ) ] is a vector of the expected feature values that corresponds with the feature function . λ is the slack term of constraints . The proof of this lemma involves strong duality of the convex-concave function , such that the min and max player can switch the order . We refer more details to the appendix . The following theorem states the solution of the problem : Theorem 1 . The parametric solution of ( 4 ) for P̂ ( y|x ) takes the form : P̂ ( y|x ) ∝ exp Ps ( x ) Pt ( x ) θ · Φ ( x , y ) + ry ry + 1 , ( 5 ) where the parameter θ can be optimized by maximizing the log-likelihood on the target distribution . The gradients take the form : ∇θEPt ( x ) P ( y|x ) [ − log P̂θ ( Y |X ) ] = EP̃s ( x ) P̂ ( y|x ) [ Φ ( X , Y ) ] − c̃ , ( 6 ) where c̃ , EP̃s ( x ) P̃ ( y|x ) [ Φ ( X , Y ) ] , as the empirical evaluation of the feature expectations . Here P̃s ( x ) and P̃ ( y|x ) are the empirical distribution . In principle , P ( y|x ) is the ground truth conditional label distribution shared between source and target domains . We call EPt ( x ) P ( y|x ) [ − log P̂θ ( Y |X ) ] the expected target loss in the paper . Even though it is not available in practice , we can approximate the gradients ( 6 ) using the source data in training . The norm of the approximated gradient converges to the true gradient in the rate of O ( 1/m ) , where m is the amount of source data . The proof involves application of the Lagrangian multiplier , setting the derivative of each specific P̂ ( y|x ) to 0 and utilizing the KKT condition . We refer details to the appendix . Algorithm 1 End-to-end Training for DRL Input : Source data , Target data , DNN φ ) , DNN τ , SGD optimizerOpt1 andOpt2 , learning rate γ1 and γ2 , epoch number T . Initialization : φ , τ ← random initialization , epoch← 0 While epoch < T For each mini-batch Compute ( 10 ) back propagate Back propagate Ld Optimizer Opt1 ( γ1 ) updates β Compute p̂ following ( 5 ) . Compute gradients using ( 9 ) Back-propagate through∇φLc w = w − γ2 · ∇wLc b = b− γ2 · ∇bLc Optimizer Opt2 ( γ2 ) updates α epoch← epoch +1 Output : Trained α , β , w , b We use this form to illustrate the property of representation-level conservativeness and the class-level regularization of our formulation . Representation-level conservativeness : The prediction has higher certainty for inputs closer to the source domain , when magnitude of Ps ( x ) /Pt ( x ) is large . On the contrary , if the inputs are farther away from the source , which means Ps ( x ) /Pt ( x ) is small , the prediction is uncertain . Class-level regularization : Hyper-parameter r adjusts the smoothness of theQ ( Y |X ) ’ s label prediction in ( 3 ) . It translates to the ry terms in the parametric form . In training , we compute the gradients using source labels where y is the one-hot encoding of the class . In testing , we can set y to be all one vector to obtain smoothed confidence . In machine learning methods using density ratios , such as transfer learning ( Pan & Yang , 2009 ) , or off-policy evaluation ( Dudı́k et al. , 2011 ) , a plugin estimator for the density ratio Ps ( x ) /Pt ( x ) is used . However , density ratio estimation ( Sugiyama et al. , 2012 ) , especially in the high-dimensional data , is rather different . It is also not the case that more accurate density ratio estimation would lead to the better downstream task performance . We have a synthetic example shown in Appendix E. To scale up the method for modern domain adaptation tasks , we ask the question : can we train the density ratio estimation and the learning tasks that use the ratios together such that they share the common goal–the target domain predictive performance ?

Summary: The paper proposes to use the distributionally robust learning (DRL) for unsupervised domain adaptation. First, the authors demonstrate how differentiable density ratio estimation can be done for source and target domains in an end-to-end manner. Following this, the authors demonstrate how confidence estimation (reliant on DRL) can be utilized for self-training based approaches for unsupervised domain adaptation. In terms of results, the authors show that the proposed approach — Distributionally Robust Self-Training (DRSL) — can provide competitive performance on t he VisDA 2017 benchmark. Furthermore, DRSL is able to achieve more calibrated probabilities (useful for more calibrated uncertainty measures) with competitive predictive performance. Finally, the authors provide some intuition as to why DRSL results in increasingly aligned conditional source and target distributions due to increased focus on learning shape features.