Distributionally Robust Learning for Unsupervised Domain Adaptation
Authors: Haoxuan Wang
Anqi Liu
Zhiding Yu
Yisong Yue
Anima Anandkumar
Venue:  (2021)
Keywords: Distributionally Robust Learning
Domain Adaptation
Self-training
Density Ratio Estimation

Abstract: We propose a  distributionally robust learning (DRL) method for unsupervised domain adaptation (UDA)  that scales to modern computer-vision benchmarks.  DRL can be naturally formulated as a competitive two-player game between a predictor and an adversary that is allowed to   corrupt the labels, subject to certain constraints, and reduces to incorporating  a density ratio between the source and target domains (under the standard log loss).  This formulation motivates the use of two neural networks that are jointly trained --- a discriminative network between the source and target domains  for density-ratio estimation, in addition to the standard classification network. The use of a density ratio in DRL prevents the model from being overconfident on target inputs far away from the source domain. Thus,  DRL   provides conservative confidence estimation in the target domain, even when the target labels are not available. This conservatism motivates the use of DRL in  self-training for sample selection, and we term the approach distributionally robust self-training (DRST). In our experiments, DRST generates more calibrated  probabilities and achieves state-of-the-art self-training accuracy on benchmark datasets. We demonstrate that DRST captures  shape features more effectively, and reduces the extent of distributional shift during self-training. 

Under review as a conference paper at ICLR 2021
DISTRIBUTIONALLY ROBUST LEARNING FOR UNSU- PERVISED DOMAIN ADAPTATION
Anonymous authors Paper under double-blind review
ABSTRACT
We propose distributionally robust learning (DRL) method for unsupervised do- main adaptation ( (UDA) that scales to modern computer-vision benchmarks. DRL can be naturally formulated tor and an adversary that is allowed to corrupt the labels, subject to certain con- straints, and reduces to incorporating density ratio between the source and target domains (under the standard log loss). This formulation motivates the use of two neural networks that are jointly trained source and target domains for density-ratio estimation, in addition to the stan- dard classification network. The use of density ratio in DRL prevents the model from being overconfident on target inputs far away from the source domain. Thus, DRL provides conservative confidence estimation the target domain, even when the target labels are not available. This conservatism motivates the use of DRL. in self-training for sample selection, and we term the approach distributionally robust self-training (DRST). In our experiments, DRST generates more calibrated proba- bilities and achieves state-of-the-an self-training accuracy on benchmark datasets. We demonstrate that DRST captures shape features more effectively, and reduces the extent of distributional shift during self-training.
competitive two-player game between a predic-
discriminative network between the
INTRODUCTION
In many real-world applications, the target domain for deployment of a machine-learning (ML) model can significantly differ from the source training domain. Furthermore, labels in the target domain are often more expensive to obtain compared to the source domain. An example is synthetic training where the source domain has complete supervision while the target domain of real images may not be labeled. Unsupervised domain adaptation (UDA) aims to maximize performance on the target domain, and it utilizes both the labeled source data and the unlabeled target data.
A popular framework for UDA involves obtaining proxy labels in the target domain through self- training (Zou et al.) 2019). Self-training starts with classifier trained on the labeled source data. It then iteratively obtains pseudo-labels in the target domain using predictions from the current ML model. However, this process is brittle, since wrong pseudo-labels in the target domain can lead to catastrophic failure in early iterations (Kumar et al. (020). To avoid this, self training needs to be conservative and select only pseudo-labels with sufficiently high confidence level. This entails accurate knowledge of the confidence levels.
Accurate confidence estimation is challenge for current deep learning models. Deep learning models tend to produce confccnnident and misleading probabilities, even when predicting on the same distribution (Guo et al., 2017a] / Gal Ghahramani] (216)). Some attempts to remedy this issue include temperature scaling (Platt et al.) 1999), Monte-Carlo sampling (Gal & Ghahramani] 2016) and Bayesian inference (Blundell et al. 2015) Riquelme et al.) 2018)). However, [Snoek et al. 2019) has shown that the uncertainty estimation from these models cannot be trusted under domain shifts.
In this paper, we instead consider the distributionally robust learning (DRL) framework (Liu & Ziebart 2014 2017) which provides principled approach for uncertainty quantification under do- main shifts. DRL can be formulated as two-player adversarial risk minimization game, as depicted in Figure Ia). Recall that the standard framework of empirical risk minimization (ERM) directly learns predictor ((Y]X) from training data. In contrast, DRL also includes adversary Q(

---

Under review as a conference paper at ICLR 2021
Pidt Plakk] x
Equilibrium
Constraints Festure Matching on X. Y.
X, Y
Adversary
Ciazifianion
Target Predictor
Network
Spred
Hiske
X
Dsorimaatio
Predictor
Network
Figure 1: (a) Intuition of DRL under domain shift, where X, and Y, represent the source labeled data, and X₁ represents the target unlabeled data, Upeed is the predictor's 's probabilistic labels and the Stake is the adversary's proposed probabilistic labels. (b) Architecture for end-to-end training of the DRL framework without class regularization. It is an instantiation of (a) using neural networks. The expected target loss cannot evaluated due lack of target labels in the UDA setting. Instead, we need to compute the gradients directly for training the networks. We present the details in Sec. 2.2
that is allowed to perturb the labels, subject to certain feature-matching constraints to ensure data- compatibility. Formally, the minimax game for DRL is:
min max (OS) (P(YIX) P(YI)) Q(()))
where the adversary Q(Y[X) is constrained to match the evaluation of a set of features $ v) to that of the source distribution (see Section 2 for details). Note that the loss in (I) is evaluated under the target input distribution P.(X), and the predictor does not have direct access to the source data (Xa, Y.). Instead, the predictor optimizes the target loss by playing a game with an adversary constrained by source data.
A special case of UDA is the covariate shift setting, where the tabel-generating distribution P(Y is assumed to be the same in both source and target domains. Under this assumption, with log-loss and a linear predictor parameterized by and features + ((x, v). ① reduces to:
Pa(x))
P(y(z) x exp
0. ((1 )() P((x)
(2)
Intuitively, the density ratio P,(x)/((()) prevents the model from being overconfident target inputs far away from the source domain. Thus, the DRL framework is a principled approach for conservative confidence estimation.
Previous works have shown that DRL is highly effective in safety-critical applications such as safe exploration in control systems (Liu al. 2020) and safe trajectory planning (Nakka et al.. 2020). However, these works only consider estimating the density ratio in low dimensions (e.g. control inputs) using standard kernel density estimator (KDE) and extending it to high-dimensional inputs such as images remains an open challenge. Moreover, it is not clear if the covariate -hiff assumption holds for common -igh-dimensional settings such as images which we investigate in this paper.
In this paper, we propose a novel deep-learning method based on the DRL framework for accurate uncertainties that scales to modern domain- adaptaaiooo tasks in computer vision.
Summary of Contributions:
We develop differentiable density-ratin estimation as part of the DRL framework enable effi- cient end-to-end training. See Figure lb).
We employ DRL's confidence estimation in the self-training framework for domain adaptation and term it as distributionally robust self-training (DRST). See Figure
We further combine with automated synthetic to real generalization (ASG) framework (Chen et al.) 2020b) to improve generalization in the real target domain when the source domain consists of synthetic images.
4. We demonstrate that DRST generates calibrated probabilities. DRST-ASG achieves com- petitive accuracy on the VisDA2017 dataset (Peng et al.) (017)) with 1% improvement over the baseline class-regularized self-training (CRST) using the standard soft-max confidence measure.
2

---

Under review as a conference paper at ICLR 2021
5. We analyze the reason for the effectiveness of DRST through careful ablation study.
One challenge for training DRL is that the training loss cannot be directly evaluated under the UDA framework. However, show that the gradients of the target loss indeed be evaluated. By deriving gradients for both neural networks and proposing a joint training algorithm (Alg. ! we show the network can be trained efficiently.
We also directly incorporate class regularization in the minimax game under our DRL framework. This is principled approach in contrast to standard label smoothing incorporated top of given learning method. In our ablation studies, we observe that the covariate-shift assumption progres- sively holds greater extent as the iterations in self-training proceed. This is also correlated with the greater ability to capture shape features through self training, as seen in the Grad-CAM visualization (Selvaraju et al. 2017).
PROPOSED FORMULATION AND ALGORIHTM
In this section, first introduce the class regularized DRL framework (.. and then propose differentiable density ratio estimation to enable end-to-end learning of DRL using neural networks. We provide training details, especially the gradient computation for training both networks (2.2) This is unique for our setting since the actual training loss on target cannot be evaluated due to lack of target labels. Finally, we propose our self-training algorithm DRST 2.3
2.1 DISTRIBUTIONALLY ROBUST LEARNING WITH CLASS REGULARIZATION
We are interested in robustly minimizing the classification loss on the target domain with confidence of adversary's 's prediction regularized. We use a weighted logloss term to penalize high confidence in the adversary's 's label prediction as the regularization. We make the same covariate shift assumption as in (Liu & Ziebart, 2014) that only the marginal input distribution changes and P(yla (z) is shared between source and target: (a(x:) + Pi(x) and P.(y)=r) Pi(yl=)). We aim to solve the following:
min max 1oglossp((x) (P(YIX), P(Y Q(YIX)ES
-rE (EBr()Q()l))) (Y log P(Y]))), (3)
where Y is a one-hot encoding of the class and E [0, 1] is a hyper-parameter that controls the level of regularization. In this formulation, the estimator player P(Y|X) first chooses a conditional label distribution to minimize the regularized logloss on the target domain and then an adversarial player Q(Y X) chooses a conditional label distribution from the set (E) to maximize the regularized logloss. The constraint set E defines how much flexibility
want to give to the adversary. Usually, we design feature functions on both X and Y and restrict the adversary to match statistics of the expectation of the features. We have the following lemma:
Lemma 1. If we choose feature map o (XX, Y) as the stafistics to constrain Q( then equation can be reduced to a reguralized maximum entropy problem with the estimator constrained:
max E P((ce)((ylaa ((Y)X)) - rE ((pce)) [Y log P(Y[X)) P(Y())
(4)
such that: P(YI)) and [ [ [0(X, (8
p (D(X S
where A defines the conditional probability simplex that P(y]z) must reside within T is vector- valued feature Sunction that is evaluated on inpul and Ep,(z.s) [x Y), is a vector of the ex- pected feature values that corresponds with the feature function. X is the slack term of constraints.
The proof of this lemma involves strong duality of the convex-concave function, such that the min and max player can switch the order. We refer more details to the appendix. The following theorem states the solution of the problem:
Theorem 1. The parametric solution of (4) for P(y]a) takes the form:
Prte P(x)
0
4( v) + ry
P(yla)) x exp
(5)
ry +
3

---

Under review as a conference paper at ICLR 2021
where the parameter 0 can be optimized by maximizing the log- lillekeiihoodd on the target distribution. The gradients take the form:
log Po(Y] P(Dll
(
(6)
where ē - (P(())) ((XXY)) as the empirical evaluation of the feature expectations.
Here (x(x) and P(ylæ) are the empirical distribution. In principle, P(y]z) is the ground truth conditional label distribution shared between source and target domains. We call BR(s))(yla)) log (x(Y(X) the expected target loss in the paper. Even though it is not available in practice. we can approximate the gradients (6) using the source data training. The norm of the approximated gradient converges to the true gradient in the rate of O (1)mm), where m is the amount of source data, The proof involves application of the Lagrangian multiplier, setting the derivative of each specific P(y)(z) to 0 and utilizing the KKT condition. We refer details to the appendix.
We use this form to illustrate the property of representation-levè conservativeness and the class-level regularization of our formulation.
Algorithm 1
End-to-end Training for DRL
Input: Source data Target data DNN Φ). DNN T. SGD optimizer Opt, and Optz, learn- ing rate 'n and *22, epoch number T'. Initialization: Φ, + random initialization,
Representation-Ee level conservativeness: The pre- diction has higher certainty for inputs closer to the source domain, when magnitude of Pz(x)/ P((x)) large. On the contrary, if the inputs are farther away from the source, which means P.(x)//((x) small, the prediction uncertain.
epnch 4 -
While epoch < T
For each mini-hatch
Compute (TO) back propagate
Class-level regularization: Hyper-parameer parameter r adjusts the smoothness of the QY'|X)'s label pre- diction in (3. It translates to the ry terms in the parametric form. In training, we compute the gra- dients using source labels where y is the one-hot encoding of the class. In testing, we can set y to be all one vector to obtain smoothed confidence.
Back propagate La
Optimizer Opl1 updates B
Compute P following 5
Compute gradients using 9
propkpro through Vo
W = w 7/2- WwC. b = 7/2 VbC
Optimizer Opt2(12) updates epoch <- epoch +1 Output: Trained a. 3, w. b
In machine learning methods using density ratios,
such transfer learning (Pan & Yang] 2009), or off-poll off-policy evaluation (Dudík et al. 2011),1 a plug- in estimator for the density ratio P.()))((()) is used. However, density ratio estimation (Sugiyama eet al. 2012), especially in the high- digh-dimensionll data, rather different. It is also not the case that more accurate density ratio estimation would lead the better downstream task performance. We have a synthetic example shown in Appendix E. To scale up the method for modern domain adapta- tion tasks, we ask the question: can we train the density ratio estimation and the learning tasks that use the ratios together such that they share the common goal-the the target domain predictive performance?
2.2 DIFFERENTAALEY DENSITY RATIO ESTIMATION AND END-TO-END TRAINING FOR DRL
We propose an end-to-end training procedure for DRL such that density ratio estimation is trained together with the target classification. We use two neural networks for classification and the density ratio estimation, respectively. See figure (b))
Differentiable density ratio estimation:
We make the observation that the density ratio in the para- metric form (5) can be a trainable weight for each example, which can receive gradients from the objective function. On the other hand, the density ratio can estimated using a binary classification using unlabeled source and target data. Therefore, work to differentiate the two domains for the density ratio estimation, which receive training signals from both the target classification loss and the binary classification loss.
propose to train discriminative neural net-
Expected target loss as a training objective: According to Theorem I even though the expected target loss cannot be evaluated using data, we can approximate the gradient (6) using source samples. We can directly apply the gradients on the last layer of the classification network and back prok-prr to the other layers. We next present the overall training objective and derive the gradients.
4

---

Under review as a conference paper at ICLR 2021
The overall training objective: Assume (( a, b) is the classification neural net with param- eters w and b in the linear last layer and parameters a in the other layers. The input is the source data. Assume T(x, (3) is the discriminative neural net with parameters B, with input both unla- beled source and target data. The last layer of the classification network is: w Φ(x o)) + b. We the following training objective that accounts for the interactions between training density ratios and classification performance for the target domain:
C L. + La PBP log P(Y + P(Po) log P(CIX)),
where L. is the expected target loss P((le) log x()[ which cannot be evaluated but has gradients available for training from the DRL framework, P(Y]X) takes the following form with neural networks as feature functions: P(y[æ) x exp
w-bo(s,a)+bbrgg
We use La to
ry+1
represent the cross-entropy loss on the domain discriminator that produces P(CIX), with C e {s as the source domain class and target domain class. Here we use P(z) and P(c|z) to represent the overall unlabeled data distribution and the empirical domain class in the data. Then the predictions
from the classifier are P(c(z) x expfcr(x, 6)}. Based on the Bayes' rule, the density ratio P((a) can be computed as:
P,(z)
P(x)) P(xltt P(tl))
P(t()) P(s)
(8)
Pa(x) P(z]s) P(s|z) P (x)) P(s)z)
Therefore, we can use a discriminator for estimating P(s
P((z) from unlabeled source and target data
(Bickel et al. 2007).
Gradients for the classification network: With density ratios, given by the discriminative network,
we compute the gradient of the classification network following (6) on w, o0x,, a),, and b.
V VolEp( log [X) = -P-
VwEp(t))(lnn log ((] = (P - y)
V V.LEE(())()aa)l log P(Y]X)) = ()-y);
where P and y are vectors of predicted conditional label probabilities using the current parameters as in Eq. (..2) and the one-hot encoding of the true labels in the source data, respectively. Note that V P(En log P(Y( corresponds with the gradient for learning in Eq. (6. with (xx y) w (V((,, a) + b.
Gradients of the discriminative network from expected target loss: We denote d, P(s]æ) and di = P(t|z) as the two weight scalars for each input æ, and therefore have P(y]z) o exp (w yo(x, a) b)]. Since vec(d,, d,) is exactly the output of the density estimator, treat (dg, de) as trainable variables and derive gradients from the expected target loss L. In this way, the parameter B of the density ratio estimation network has additional learning signals from both losses. We derive the gradients as:
de Va C. - E (Ms))/M))) ÿō(zz a) bb, Co - - (olle)/ a) + b), (10) de
which is dependent on unlabeled the target inputs P((z), but does not rely on the target labels. We refer to the appendix for the detailed derivation. Then we concatenate (10) into a gradient vector and back-propagate the discriminative network. We summarize the procedure in Algorithm [
2.3 DISTRIBUTIONALL! ROBUST SELF-TRE TRAININGNN
Algorithm: We propose Algorithm to combine the DRL model with self-training. The idea is to regard each training epoch new distribution shift problem in DRL. After each training epoch, we make predictions on the target domain and select the top confident portion with the proxy labels merge into the source training data. Both the pseudo labels and the model confidence are from the DRL model 5). Then the labeled source data and newly pseudo-labeled target portion become the new source set for the next learning epoch for DRL, as shown in Figure 2 As studied in the previous self- self-training work (Zou et 2018), the data proportion p does not have significant impact on the results. After validating this in practice, we set p = 0.5 for DRST for simplicity.
5

---

Under review as a conference paper at ICLR 2021
DRL model
Retraining
DRL model
Retraining
DRL model olAgg
Training
(pre
Preudo
Source
Source
Labels
Libes
Y
Nui Xa
X.
Y
Y,
Targes
Targer
Target
Confideenc--iion bisoed Selection (in Alg2)
Confidence based Selection (in Aig2)
Figure 2: The self-training process of DRST, where X11 and X₂2 represent the portion of target data that is pseudo-labeled with Yn1 and 1/2- We DRL. model's prediction pseudo-labels and its model confidence as the criterion for choosing pseudo-labeled data from the target domain.
Algorithm Algorithm of DRST
Initialization with ASG: The quality of source model has impact on the final self-training performance. Recently, (Chen et al. 2020b) proposed automated synthetic-to-reat generalization (ASG) where ImageNet pre-trained knowledge is used to improve synthetic training through distillation loss. It was shown that initializing with ASG trained model leads to significant improvement in self-training. As result, the ASG model as our model initialization, and show solid improvements from DRST on top of competitive baselines.
Input: Source data Target data, epoch number Z'. pseudo-label portion p%. Load pretrained model
While epoch T
Predict target data using 5 and rank the confidence
Label the top p% of target data
Merge pseudo-labeled data into train-
also take
ing data
Conduct Algorithm for one epoch epoch f epoch +
EXPERIMENTS
In this section, we evaluate the performance of our method on benchmark domain adaptation bench- mark datasets. We evaluate DRST as effective unsupervised domain adaptation method (Sec. 3.1 and DRL domain generalization method providing more calibrated uncertainties (Sec. 3.2..
We adopt three datasets in our experiments: Office31 (Saenko et al. 2010). Office-Home (Venkateswara et al.] 2017) and VisDA2017 (Peng et al. 2017). In particular, we use the largest data Vi5DA2017 for the evaluation of DRST, for which we compare with (1) traditional UDA base- lines: MMD (Long et al. (2015). MCD (Saito et al. (0188) and ADR (Saito et al. 2018a); (2) recent self-training UDA baselines: CBST (Zu et al. (020) and CRST (Zou al. 20020) (3) other uncer- tainty quantification or UDA methods + self-training baselines: AVH (Chen et al. (0200) CBST and DeepCORAL((Su) & Saenko, CCBST. In addition, we use Office31 and Office-Homm for evaluating DRL's performance. We compare DRL with source training only and temperature scaling for demonstrating the calibration of the uncertainties used in DRST.
Apart from accuracy, we also use Brier score (Brier 1950> and the realiability plot (Guo er al. 2017a) evaluate the performance of our proposed method and the baselines. Brier score measures the mean squared difference between the predicted probability assigned to the possible outcome
and actual outcome y: BS
1 1 3
(p 21 The lower the Brier score is for set of
n
predictions, the better the predictions are calibrated.
3.1 UNSUPERVISED DOMAIN ADAPTATION USING DRST
Accuracy and calibration: Figure 3 a) and table D demonstrates that DRST outperforms all the
baselines in accuracy. Our vanilla version of DRST outperforms CRST by more than 5% and with ASG initialization, we improve the SOTA self-training accuracy on VisDA by more than 1 1%. Note that in the rest of the results, we directly use DRST represent DRST-ASG. Figure 3b) demon- strates that DRST also achieves more calibrated confidence.
Ablation study: Figure (()(b) also include two ablation methods. In the first ablation, set to O so that there is no class regularization in DRL ("r = 0*). The prediction then follows the form in Eq. (2).
In the second ablation, we set density ratio to 1 such that there is representation level

---

Under review as a conference paper at ICLR 2021
Table I: Accuracy comparison with different methods on VisDA2017.
Method
Azto
Bike
Blus
Car
Hone
Knite
Mosoc
Person
Plant
Skatebourd
Train
Track
Miaan
Sturce
2013
35.1
33.3
61.9
35.1
80.6
17.5
19.7
31.2
51.0
25.5
73.5
8.5
52.4
76.5
90.3
36.3
83.88
88.9
83.0
1100
87.2
0910
78.9
DeepCORA
Ail Se0tlo
2016 CCDST
92.1
78.9
83.0
73.8
95.2
94.7
89.0
$3.0
$9.8
81.2
83.5
44.9
82.4
DRST (poposs))
93.47
56.30
65.74
68.05
93.99
95.08
87:34
83.30
92.97
88.65
83.66
66.42
83.75
ASGi (Cien el Hl. 2020h
19.81
68.55
66.31
73.066
95.78
9.31
84.35
29.58
82.13
33.76
86.00
12.04
61.17
CBST-ASIT ALTT-ASSLLLEE (Lis 2ini
95.12
86.53
79.83
76.01
94.61
92. 34
85.04
75.08
80.23
82.
73.42
56.49
82.23
CRST ASG
92.38
$1.30
74.63
84.40
90.90
92.43
83.78
94.92
88.12
74.88
84,21
Test accuracy compartson
Test orter score comparison
Compartson distributionooo gap
0.333
sccure
04050
DAST with
with Rua
(WRST
CAST
DRST B=1
CRST
0401
1ea (RST with mn
DIST
Epoch
Epoch
Epoch
(c)
Figure 3: (a)- (b) Accuracy and Brier score of self-training on VisDA2017 with 5 random seeds. We demonstrate the 95% standard error to show that DRST is outperforming the baselines signifi- cantly. (c) Distribution gap in P(y|z): (z) - Pa(æ, y)/ (a y), between source and target domains. Comparing with DRL, without self-training, DRST helps reduce the gap.
consertiveness ("R = 1"), which means we mute the differentiable density ratio estimation in our method. We can see that DRST achieves the best performance when both components are present.
Input image
CBST
CRST
DRST
Bus x
Truck v
Figure 4:
Model attention visualized using Grad-Cam (Selvaraju et al. 2017). We also show the predicted labels by different methods. Our method captures the shape features of the image better.
Covariate shift: In figure c)., we demonstrate the ratio difference between (x( P,(z,y)/ Pr y), where I is the last layer representations from each of the method. We esti- mate this ratio using discriminative density ratio estimators (per class). When Pv(v (o(y)rr), the ratio difference is close to O. We can see that the P(y(z) gap between source and target is de- creasing along self- self-training. This due to the shape features captured by the method. We can also observe it from figure 4 where we visualize the last layer model attention of our model and the baselines. Therefore, even though the covariate shift assumption from DRI. may not be satisfied at the beginning of self-training, self-training helps align P(y(x) by promoting the learning of shape representations, such that P. (y(lz) and Pz(y) converges along the training.
We show pair of examples in the target domain with relatively high and low density ratios. In [ggure[[a) a more typical "train" image obtains higher density ratio than train busy platform, whose shape of the train is not obvious. DRL is able to give higher confidence to images that are better- in the synthetic training domain. More examples given in the appendix.
Density ratio estimation:

---

Under review as a conference paper at ICLR 2021
PACIFIC
213 222
(a) Left: (.(x)/P((z) 1.004 Right: (x) 2.132
(b)
Figure 5: (a) Density ratios and example target images for category "Train" in VisDA, Larger density ratio (Pa (x) (x)))) indicates more certain prediction. DRL give more cerntain prediction on "train"
that is better represented in the source domain, which on the right hand side. (b) Accuracy and brier score using DRL on VisDA. DRL significantly improves the source model along the training.
acro
50
125
150
05
20
bookle
Epocta
(a)
(b)
Figure 6: Accuracy and Brier Score of DRL on Office31 and Office-Home. DRL improves source- only by large margin.
3.2 PREDICTIVE PERFORMANCE AND UNCERTAINTY QUANTIFICATION OF DRL
We compare DRL with source models on VisDA2017. Office-31, and Office- Home datasets to show the DRL model's s accuracy and calibration. Note that DRL can be regarded as a lightweight model generalization method since the target unlabeled data is only used for the density ratio estimation.
DRL's improvement on source training: Figure S (b) and Figure 6 demonstrate the significant improvement over the source synthetic training model in all the data sets. Note that the only differ- ence between the DRL and the source model DRL's predictive form and the differentiable density ratio estimation. In these plots, all the models are trained with the same number of total training epochs. The accuracy and brier score at epoch 5 means we incorporate DRL to improve source training mode from epoch 5, Therefore, we can observe DRL's consistent significant improvement over the source training model even when starting late the learning process. The lower brier score provides evidence on more conservative model confidence that benefits self-training.
DRL's conservativeness in confidence: Figure 7 /emonssrrtess DRL's more calibrated uncertainty measure. We adopt the same definition of confidence and accuracy (Guo et al., (017a) and the same diagram plotting protocol as (Maddox et al., 2019). The closer the curve is to the dash line, the more calibrated the uncertainty is. DRL tends to be little underconfident but also stays closer to the calibration line (dash line).
Retibility on offoc31
Rollasiity plat OfficHome -)
Relacility plation
Fellability
Condbrce Accuees
Corfiace AcQureE
Cortderce Accuses
DAST
Contodence tman prini
Cant denice craec prcb)
Cantidance OTHEne prcb)
EMU Connidecc proto
(a)
(b)
(d)
Figure 7: Reliability diagrams with respect to source-only, temperature scaling (TS) and DRL. The data is separated into 20 bins with different interval lengths. The closer the curve to the dash line, the more calibrated the uncertainty is. DRL is able to achieve more calibrated results, usually conservative confidence, while the other methods always tend to generate over-confident predictions.
8

---

Under review as a conference paper at ICLR 2021
4 RELATED WORK
Uncertainty Quantification with Deep Models:
Bayesian methods largely contribute to current ef-
fort for complementing deep learning with uncertainty quantification. MCMC (Gal 2016 Welling
& Teh
2011: Chen et al. 2014), (stochastic) variational inference for deep learning (Kingma &
Welling 2013 Louizos & Welling 2017 Blundell et al.] [2015] Riquelme et al. 2018), dropout
variational inference (Gal & Ghahramani 2015: Gal et al., 2017 Kendall & Gal 2017), and SGD-
based approximations (Mandt et al. 2017) aim to approximate the Bayesian inference from different perspectives the deep learning setting. However, with a large number of parameters the neu- ral network, Bayesian deep learning may suffer from computational inefficiency. Other calibration methods such as temperature scaling (Platt et al. 1999 Guo et al.] 2017b), deep ensemble (Lakshmi- narayanan al. 2017), calibration regression (Kuleshov et al. (2018), quantile regression (Romano et at 2019) and trainable calibration measure (Kumar et al. 2018). However, these methods are not applicable to our domain shift setting.
Unsupervised Domain Adaptation: One line of work in unsupervised domain adaptation methods focus on (feature) representation alignment between the source and target domains (Fernando al.,
2014
Pan ct al. 2010; Sun et al. 2016 Wang et al., 2018 Baktashmottagh et al., 2013 Gretton 2012). In deep learning, is more prevalent use a discriminator network differentiate
al.
Iwo learned representations (adversarially) to locate such a feature space (Tzeng et al., 2017 Ajakan
ct al. 2014| Long et al., 2018 Sankaranarayanan et al. 2017). However, recent study show that this
approach lacks theoretical grounding (Wu et al. (2019) and suffers from unstable training. Another line of work focus on more specific shift assumptions between two domains, like covariate shift (Shimodaira] 2000) and label shift (Lipton et al. 2018; Azizzadenesheli et al.) 2019). Importance weighting on source samples is the common strategy to correct the shift. For the covariate shift case, even though various density ratio estimation has been investigated before (Sugiyama et al., 2012). only few works explore the possibility to apply them in high-dimensional data (Khan et al. 2019
Moustakides & Basioti 2019
Park et al. 2020).
Self-training for UDA: Multiple self- trafniraaning methods have been proposed to do unsupervised do- main adaptation. Class-balance self-training (Zou al., 20188) propose class-balancing technique to avoid the gradual dominance of large classes on pseudo-label generation. Smoothing the model It is also helpful use query selection strategy based on angular visual hardness (Chen et al. 2019). Recently. more theoretical
confidence further help improve the performance (Zou al.
(019)).
understanding of self-training are developed (Kumar et al.) 2020 Chen et al,, 2020c). In this paper, we incorporate ideas from the CRST to smooth the DRL's model uncertainty and demonstrate that
DRL's model confidence helps choose better pseudo-labeled data from the target domain.
Distributionally robust learning: Distributionally robust learning (Liu & Ziebart, 2014
2017
Chen et al., 2016 Fathony et al. 2016: 2018) for domain shift developed based on pessimistic perspective of the finite samples existing in the source domain. However, the limitation of this line of work is that it usually only works for low-dimensional data. Another inconvenience lies in the requirement for the density ratio (z) (xx)) to be done beforchand. Instead of having the adversary as the conditional labeling functioo, an alternative is to have perturbing the covariate variable
(Ben-Tal et al.) 2013) Shhfieezaadh Abadeh et al.) 2015 Sinha et al.) 2017: Hu et al.) 2018 Najafi et al. 2019). This line of work has been mostly focusing on model robustness against adversarial perturbations. Instead, in this paper, we focus on the benefit of DRL's uncertainty estimation for self-training in UDA. We apply neural networks and propose end-to-end learning method to handle high-dimensional image data.
5 CONCLUSION
In this paper, we develop a learning method under the distributionally robust learning framework for modern domain adaptation. We propose differentiable density ratio estimation and class regulariza- tion in the framework. We develop end-to-end training techniques for our proposed method. Using DRL's model confidence, stay calibrated. We also demonstrate that self-training helps reduce the distribution gap between source and target domains, facilitating DRL to be effective.
self-training algorithm achieves SOTA predictive performance while
9

---

Under review as a conference paper at ICLR 2021
REFERENCES
Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, and Mario Marchand. Domain-adversarria neural networks. ArXiv, abs/1412.4446, 2014.
Kamyar Azizzadenesheli, Anqi Liu, Fanny Yang, and Animashree Anandkumar, Regularized learn- ing for domain adaptation under label shifts. arXiv preprint arXiv: /903. 09734, 2019.
Mahsa Baktashmotlagh,, Mehrtash T Harandi, Brian C Lovell, and Mathicu Salzmann. Unsupervised domain adaptation by domain invariant projection. In Proceedings of the IEEE International Conference on Computer Vision, 769-776, 2013.
Aharon Ben-Tal, Dick Den Hertog, Anja De Waegenaere, Bertrand Melenberg, and Gijs Rennen. Robust solutions of optimization problems affected by uncertain probabilities. Management Sci- ence, 59(2):341-357 2013.
Steffen Bickel, Michael Brückner, and Tobias Scheffer. Discriminative learning for differing training and distributions. In Proceedings of the 24th international conference on Machine learning. pp. 81-88, 2007.
Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural networks. arXiv preprint arXiv:1505.05424, 2015.
Glenn W Brier. Verification of forecasts expressed in terms of probability. Monthly weather review, 78(1):1-3, 1950.
Beidi Chen, Weiyang Liu Animesh Garg, Zhiding Yu, Anshumali Shrivastava, Jan Kautz, and Anima Anandkumar. Angular visual hardness. arXiv preprint arXiv: 2019.
Beidi Chen, Weiyang Liu, Zhiding Yu, Jan Kautz, Anshumali Shrivastava, Animesh Garg, and An- ima Anandkumar. Angular visual hardness, 2020a.
Tianqi Chen, Emily Fox, and Carlos Guestrin. Stochastic gradient hamiltonian monte carlo. In International conference on machine learning, pp. 1683-1691, 2014.
Wuyang Chen, Zhiding Yu, Zhangyang Wang, and Anima Anandkumar. Automated synthetic-to- real generalization. arXiv preprint arXiv:2007.06665, 2020b.
Xiangli Chen, Mathew Monfort, Anqi Liu, and Brian D Ziebart. Robust covariate shift regression. In Artificial Intelligence and Statistics, pp. 1270-1279, 2016.
Yining Chen, Colin Wei, Ananya Kumar, and Tengyu Ma. Self-training avoids using spurious features under domain shift, 2020c.
Miroslav Dudik, John Langford, and Lihong Li. Doubly robust policy evaluation and learning. arXiv preprint arXiv: 2011.
Rizal Fathony, Anqi Liu, Kaiser Asif, and Brian Ziebart. Adversarial multiclass classification: A risk minimization perspective. In Advances in Neural Information Processing Systems, pp. 559-567, 2016.
Rizal Fathony, Kaiser Asif, Anqi Liu, Mohammad Ali Bashiri, Wei Xing, Sima Behpour, Xinhua Zhang, and Brian D Ziebart. Consistent robust adversarial prediction for general multiclass clas- sification. arXiv preprint arXiv:1812.07552 2018.
Basura Fernando, Amaury Habrard, Mare Sebban, and Tinne Tuytelaars. Subspace alignment for domain adaptation. CoRR, Abs/1449,5241, 2014. URL
5241
Yarin Gal. Uncertainty in deep learning. 2016.
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. 2015.
10

---

Under review as a conference paper at ICLR 2021
Yarin Gal and Zoubin Ghahramani. Dropout as bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pp. 1050-1059, 2016.
Yarin Gal, Jiri Hron, and Alex Kendall. Concrete dropout. In Advances in neural information processing systems, pp. 3581-3590, 2017.
Arthur Gretton, Karsten M. Borgwardt, Malte Rasch, Bernhard Schölkopf, and Alexander Smola.
A kernel two-sample test. 13:723 -73,, 2012. ISSN 1532-4435..
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Srarning-Volumee 70, pp. 1321-1330. JMLR. org, 2017a.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. CoRR, abs/1706.04599, 2017b.
Weihua Hu, Gang Niu, Issei Sato, and Masashi Sugiyama. Does distributionally robust supervised learning give robust classifiers? In International Conference on Machine Learning, pp. 2029- 2037. PMLR, 2018.
Alex Kendall and Yarin Gal. What uncertainties do we need in bayesian deep learning for computer vision? In Advances in neural information processing systems, pp. 5574-5584, 2017.
Haidar Khan, Lara Marcuse, and Bülent Yener. Deep density ratio estimation for change point detection. arXiv preprint (rXXiv1905.09870 2019.
Diederik P Kingma and Max Welling.
Auto-encoding variational bayes.
arXiv preprint
arXiv:1312.6114. 2013.
Volodymyr Kuleshov, Nathan Fenner, and Stefano Ermon. Accurate uncertainties for deep learning using calibrated regression. arXiv preprint arXiv:1807.00226, 2018.
Ananya Kumar, Tengyu Ma, and Percy Liang. Understanding self-training for gradual domain adaptation. arXiv preprint arXiv:2002. / /361, 2020.
Aviral Kumar, Sunita Sarawagi, and Ujjwal Jain. Trainable calibration measures for neural networks from kernel mean embeddings. In International Conference on Machine Learning, pp. 2805- 2814, 2018
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predic- tive uncertainty estimation using deep ensembles. Advances in neural information processing systems, pp. 6402-6415 24226613,,40 2017.
Zachary C Lipton, Yu- Xiang Wang, and Alex Smola. Detecting and correcting for label shift with black box predictors. arXiv preprint 80X.v:180..03916, 2018.
Anqi Liu and Brian Ziebart. Robust classification under sample selection bias. In Advances in neural information processing systems, pp. 37-45, 2014.
Anqi Liu and Brian D Ziebart. Robust covariate shift prediction with general losses and feature views. arXiv preprint arXiv: 10043, 2017.
Anqi Liu, Guanya Shi, Soon-Jo Chung, Anima Anandkumar, and Yisong Yue. Robust regression for safe exploration in control. In Learning for Dynumics and Control, pp. 608-619, 2020.
Mingsheng Long, Yue Cao, Jianmin Wang, and Michael I. Jordan. Learning transferable features with deep adaptation networks, 2015.
Mingsheng Long, Zhangjic Cao, Jianmin Wang, and Michael Jordan. Conditional adversarial domain adaptation. In Advances in Neural Information Processing Systems, pp. 1640-1650, 2018.
Christos Louizos and Max Welling. Multiplicative normalizing flows for variational bayesian neural networks. In Proceedings of the 34th International Conference on Machine Learning Volume 70, pp. 2218-2227. JMLR. org, 2017.
11

---

Under review as a conference paper at ICLR 2021
Wesley Maddox, Timur Garipov, Pavel Izmailov, Dmitry P. Vetrov, and Andrew Gordon Wilson. A simple baseline for bayesian uncertainty in deep learning. CoRR, abx/1902.02476, 2019.
Stephan Mandt, Matthew D Hoffman, and David M Blei. Stochastic gradient descent as approximate bayesian inference. The Journal of Machine Learning Research, 18(1):487344977, 2017.
George V Moustakides and Kalliopi Basioti, Training neural networks for likelihood/density ratio estimation. arXiv preprint arXiv: wetXiv:/91..00 1.11000110.000 2019.
Amir Najafi, Shin-ichi Maeda, Masanori Koyama, and Takeru Miyato. Robustness to adversarial perturbations in learning from incomplete data. In Advances in Neural Information Processing Systems, pp. 5541-5551, 2019.
Yashwanth Kumar Nakka, Anqi Liu, Guanya Shi, Anima Anandkumar, Yisong Yue, and Soon-Jo Chung. Chance-conse conesraind trajectory optimization for safe exploration and learning of nonlinear systems. arXiv preprint arXiv:2005. 2002:20005003374. 2020.
Sinno Jialin Pan and Qiang Yang. survey on transfer learning. IEEE Transactions on knowledge and data engineering, 22110:1345-11355 2009.
Sinno Jialin Pan, Ivor W Tsang, James T Kwok, and Qiang Yang. Domain adaptation via transfer component analysis. IEEE Transactions on Neural Networks, 22(2):199-210, 2010.
Sangdon Park, Osbert Bastani, James Weimer, and Insup Lee. Calibrated prediction with covariate shift via unsupervised domain adaptation. arXiv preprint urXiv:2003.00344 2020.
Xingchao Peng, Ben Usman, Neela Kaushik, Judy Hoffman, Dequan Wang, and Kate Saenko. Visda: The visual domain adaptation challenge, 2017.
John Platt al. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in large margin classifiers, 10(3):61-74, 1999.
Carlos Riquelme, George Tucker, and Jasper Snoek. Deep bayesian bandits showdown: An empirical comparison of bayesian deep networks for thompson sampling. arXiv preprint arXiv:1802.09117. 2018.
Yaniv Romano, Evan Patterson, and Emmanuel Candes. Conformalized quantile regression. In Advances in Neural Information Processing Systems, pp. 3538-3548, 2019.
Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell. Adapting visual category models to new domains. In Kostas Daniilidis, Petros Maragos, and Nikos Paragios (eds.), Computer Vision ECCV 2010, pp. 213-226, Berlin, Heidelberg, 2010. Springer Berlin Heidelberg. ISBN 978-3- 642-15561-1.
Kuniaki Saito, Yoshitaka Ushiku, Tatsuya Harada, and Kate Saenko. Adversarial dropout regular- ization, 2018a.
Kuniaki Saito, Kohei Watanabe, Yoshitaka Ushiku, and Tatsuya Harada. Maximum classifier dis- crepancy for unsupervised domain adaptation, 2018b.
Swami Sankaranarayanan, Yogesh Balaji, Carlos D. Castillo, and Rama Chellappa. Generate to adapt: Aligning domains using generative adversarial networks. CoRR, abs/1704.01105, 2017.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based local- ization. In Proceedings of the IEEE international conference on computer vision, pp. 618-626, 2017.
Soroosh Shafieezadeh Abadeh, Peyman M Mohajerin Esfahani, and Daniel Kuhn. Distributionally robust logistic regression. Advances in Neural Information Processing Systems, 28:1576-1584, 2015.
Hidetoshi Shimodaira. Improving predictive inference under covariate shift by weighting the log- likelihood function. Journal of statistical planning and inference, 90(2):227-244, 2000.
12

---

Under review as a conference paper at ICLR 2021
Aman Sinha, Hongseok Namkoong, Riccardo Volpi, and John Duchi. Certifying some distributional robustness with principled adversarial training. arXiv preprint arXiv:1710.10571, 2017.
Jasper Snoek, Yaniv Ovadia, Emily Fertig, Balaji Lakshminarayanan, Sebastian Nowozin, D Scul- ley, Joshua Dillon, Jie Ren, and Zachary Nado. Can you trust your model's uncertainty? eval- uating predictive uncertainty under dataset shift. In Advances in Neural Information Processing Systems, pp. 13969-13988 2019.
Masashi Sugiyama, Taiji Suzuki, and Takafumi Kanamori. Density ratio estimation in machine learning. Cambridge University Press, 2012.
Baochen Sun and Kate Saenko. Deep coral: Correlation alignment for deep domain adaptation. In ECCV 2016 Workshops, 2016.
Baochen Sun, Jiashi Feng, and Kate Saenko. Return of frustratingly easy domain adaptation. In AAAI, 2016.
Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. CoRR, abs/1702.05464, 2017.
Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. CoRR, abs/ 706.07752, 2017.
Jindong Wang, Wenjie Feng, Yiqiang Chen, Han Yu, Meiyu Huang. and Philip S. Yu. Visual domain adaptation with manifold embedded distribution alignment. CoRR, abs/1807.07258, 2018.
Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th international conference machine learning (ICML-I/), pp. 681-688, 2011.
Yifan Wu, Ezra Winston, Divyansh Kaushik, and Zachary Lipton. Domain adaptation with
asymmetrically-rrlxxdd distribution alignment. arXiv preprint arXiv: /901999930..9.. 2019.
Yang Zou, Zhiding Yu, BVK Kumar, and Jinsong Wang. Domain adaptation for semantic segmen- tation via class-balances self self-trainin training. arXiv preprint arXiv: 2018.
Yang Zou, Zhiding Yu, Xiaofeng Liu, BVK Kumar, and Jinsong Wang. Confidence regularized self-training. In Proceedings of the IEEE International Conference on Computer Vision, pp. 5982-5991, 2019.
Yang Zou, Zhiding Yu, Xiaofeng Liu, B. V. K. Vijaya Kumar, and Jinsong Wang. Confidence regularized self- self-training, 2020.
13

---

Under review as a conference paper at ICLR 2021
A
PROOF OF LEMMA1
Proof. The two-player game in (3) can be written as:
min
DIa Q01X)E
Σ P((x) Σ (ogl - Σ Pi(x) Σ Qu1 max Q(v((x)) y) log P(ylæ).
P(Y]X)
V
According to strong Lagrangian duality, we can switch the order the two players and is equiva-
lent to:
P.(x)) Σ ((glz))og y
Σ Pt(x)
Σ Q(1l=) max Q(1(==) y) log P(y]a)).
max min
QYIX) P(Y)))
y
Solving the minimizing problem first assuming that know Q(Y[X), get the result that P(Y Q(YIX). Plugging it into the maximizing problem, the whole problem reduces to 4
B PROOF OF THEOREM
Proof. The generalized constrained optimization problem in (4) can be written as:
max - -E Pi(x) P(YIX)
Σ
(1 + rI(arg max P(v'lz) = y)) log P(ylx)
y
TEA
VEY
such that:
Σ
P. ((()))((())))]]((XXX & Ak E {l...., K}
FE VEy
Vx E X:
Σ P(ylx)=1 =1
Ve E X, E y: P(yl=) 2 0.
Note that the final constraint is superfluous since the domain of the objective function is the positive real numbers. The Lagrangian associated with this problem is:
((0,))= =- Σ P((x))
Σ
((yla)(1 + rarg max P(y/(x)) (x) = y)) log P(ylz)+
(11)
ZEN
VEY
Σ 02 Σ
Σ p. P( & + E x(x)
P(yl))
VEY
DEY
where 0 and A(x) are the Lagrangian multipliers. Taking the derivative with respect P(y]x),
((0,)) -(0,A)= (1) + rI rI(agg max P(y'(x) v)) log P(y)z) P((x) Σ (T) +(x),,
0P(y)z)
yf'
setting equal to zero, af(v)=))
9 [(0, A) 0, and solving, get:
P. (x) P((x)
X(x) P(xx)
(1 + rr(arrrggaa P()|x) v))
(1 rI(rg max P(y/()) v)) log P(ylæ) 1 Σ Oafa(a,))
of
P((x)
Therefore, if we instead set y as a one-hot encoding of the prediction, we conclude:
(2) Ei Ata) (g.w)
P(y|æ) xc e
The derivation of gradient then resembles the derivation in Theorem in (Lu & Ziebart 2014).
14

---

Under review as a conference paper at ICLR 2021
Source:
Target:
ASG
ASG+CBST
ASG+CRST
ASG+DRST
Figure 8: TSNE visualization of the learned classifier using different methods. Using DRST, the
classes are well-separated.
C DETAILED DERIVATION OF GRADIENTS
Gradient of the expected target loss over source densities d., where we use R. for the density ratio B(x)
or da: R(w)) d.
OL(O)
(- (L (XX))) + (()))) log Zx(X))) = E -B)()) P()lm)) (
Od,
0d,
1 (0)
di 0R
(12)
and gradient of loss target densities de:
AT(()) 0
d,
d, OL(0)
E (ogg(o(X)))
de (P 1
dZ OR (13)
0d. ad,
where:
(L(()
exp(R20,0) (T(X,Y) DEY Zo(X)
E.)())
E.
(14)
0R
D ADDITIONAL DRST EXPERIMENTAL RESULTS
We demonstrate the TSNE plot of the learned decision boundaries for CBST, CRST and DRST in figure 8 shows the misclassification entropy comparison between source model and DRL model. Misclassification entropy is calculated S, 1 Σr Σ Pij log where is the number of samples and m is the number of categories in the dataset, and Pij indicates the prediction probability of the ith sample on the jth category. The larger misclassification entropy is, the more uncertain the model prediction result is the wrong predictions. This means the model would fail more gently. Figure 10 demonstrates additional model attention visualization. Figure IT shows additional target examples with high and low density ratios. Our model is able to find noisy and less-repred resreseerded image from the target data by estimating the density ratios. DRL would be more uncertain those data.
15

---

Under review as a conference paper at ICLR 2021
Spachs
Epocts
Epodhs
(b)
(c)
Figure 9: Comparison of DRL and DRST misclassification entropy on different datasets.
Input image
DRST
Horse
Horse
Car
V
Motorcycle
Motorcydle
V
Horse
Horse
Bicycle
Motorcycle
Motorcydl
Skateboard
Model attention visualized using Grad-Cam Seevaraju et al. 2017). Our method can also capture the domain knowledge well. For example, the first row input image contains giraffe and a car. However, giraffe is not existing label for VisDA2017. While CBST and CRST captures the wrong information, DRST is able correctly capture the domain information.
Figure 10:
E SIMULATION ON PLUG-IN ESTIMATOR
DRL for domain shift requires (x(x)/(((x) to adjust the representation-level conservativeness Like many other machine learning methods using importance weights, like in transfer learning (Pan & Yang, 2009). off- off-policy evaluation (Dudik al., 2011), we can use plug-in estimator for the density ratio P.(x)/P((()) However, density ratio estimation (Sugiyama et al. 2012). especially in high-dimensional data, rather different. Here, we ask the question: whether a more accurate density ratio estimation leads to greater predictive performance in the downstream tasks?
We show a two-dimensional binary classification example in figure 12 to demonstrate relation be- tween the performance of the density (ratio) estimation and the performance of the ultimate target learning tasks. We use the RBA method (Liu & Ziebart 2014) as an example. We conduct kernll density estimation (KDE) and evaluate the average log-like likelihood of the source and target domain. We take the ratio of the density from KDE and plug in the RBA method. We can see that the case with higher log-likelihood actually fail to give informative predictions. One of the reason is the
16

---

Under review as a conference paper at ICLR 2021
1.536
2-1.13
r=1.014
r= 1.078
Truck
Figure 11:
Additional examples for density ratio estimation for different categories. can observe that data less well-represented in the source data has much lower density ratio. This shows that learned density ratio is a good measure of the level representation data in source and target domains.
density (ratio) estimation task, as an independent learning task, does not share information with the downstream prediction tasks that use the ratios.
Likelikeld
Loc liceliiklddodd 1441
135
150
405 423 -30
1MNN 3004 1e0
(a)
(b)
(d)
Figure 12: (a) Source and target data points are drawn from two Gaussian distributions. Solid line: source and dashed line: target. The underlying true decision boundary for the binary classes the same between the two domains. (b)-(c) Prediction with density ratios from low and high density estimation likelihoods. With more accurate density estimation, the RBA predictor gives overly conservative predictions on the target domain. The colormap is the confidence P(*1"(z)) (ar). (d) With larger likelihood in density ratio estimation, the target log loss becomes worse
Arch secr beie SOMEERE OIRIOT DI
Acca ond by ScorE OfficET
Accenscy and Offectt MO
1000
Accuracy
Rgg Onter store
Can Scrtee
Epochss
123
250
228
300
353 2TA
10.2
250 13.
25.
Figure 13: Additional Office31 results on DRL compared with source-only, which is a complement for Figure 6.
17

---

Under review as a conference paper at ICLR 2021
Delaatallyy Off (W 02)
Rabatally Oll 02
Uee Madiebilit slart Officesi 10
Co-teencc Accuracy
Lunditeee Rasumasy
DT1TT
DRST
Confidance (max prob)
Confidence
(mix prob)
Confidence (tia proz)
Figure 14: Additional 0ffice3 1 results reliability plots, which is complement for Figure 7. DRL
compared with source-only and temperature scaling.
F ADDITIONAL RESULTS FOR DRL ON OFFICE-31
We include additional result for distributionally robust learning on office-31 Figure 13 and Figure
14
G
CODE REPOSITORY
Code
be
found at:
18


Reviews:
[{"review_id": "XlE85-w7eVX", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summary**\nThe paper proposes to use the distributionally robust learning (DRL) for unsupervised domain adaptation. First, the authors demonstrate how differentiable density ratio estimation can be done for source and target domains in an end-to-end manner. Following this, the authors demonstrate how confidence estimation (reliant on DRL) can be utilized for self-training based approaches for unsupervised domain adaptation. In terms of results, the authors show that the proposed approach — Distributionally Robust Self-Training (DRSL) — can provide competitive performance on t he VisDA 2017 benchmark. Furthermore, DRSL is able to achieve more calibrated probabilities (useful for more calibrated uncertainty measures) with competitive predictive performance. Finally, the authors provide some intuition as to why DRSL results in increasingly aligned conditional source and target distributions due to increased focus on learning shape features.\n\n\n**Strengths**\n- With the exception of a few points raised in the weaknesses section, the paper is generally well-written and easy to follow. \n- When initialized with ASG, the proposed approach shows roughly ~1% improvement compared to baselines and prior approaches on the VisDA benchmark. This improvement is coupled with increased sample-efficiency (significant within 95% standard error), more calibrated probabilities and monotonic reductions in the measure distribution gap. This is perhaps one of the strongest positive points in support of the proposed approach.\n- The presented analysis about the kind of features DRST increasingly focuses on (although not fleshed out more extensively) is interesting and can likely provide more insights into how to improve on DRST in future work. \n\n**Weakesses**\n- One of the minor weaknesses with the current draft is readability. I found it slightly hard to digest the proposed approach in the form it is presented in the current draft. For instance, it was unclear as to how gradient expressions for the classification network from the expected target loss for DRL from equation 8 were aligned with the proposed reduction in equation (6) — specifically, what data are the gradients estimated on? In context of DRST, is this on labeled source + pseudo labeled target per-epoch from an earlier DRST epoch? While sections 2.1 and 2.2 in themselves are more or less straightforward to follow, putting them together in context of DRST seems slightly hard for a reader. Improving this in terms of clarity would definitely benefit the paper.\n- While the experiments attempting to understand the kind of features DRST focuses compared to baselines and other approaches using Grad-CAM are useful and definitely provide some insight, the paper would definitely benefit if this claim were backed by quantitative metrics of some kind. Namely, (1) to understand if DRST indeed focuses on shape features, what is the mean IoU of region highlighted by grad-cam (w.r.t. the predicted class) with (say) the segmentation mask of concerned object in the image and (2) since Grad-CAM is being relied on as a tool to obtain saliency maps, observed saliency maps should also be provided with measures of fidelity — how reliant are Grad-CAM maps in terms of providing feature importance visualizations in context of the models and dataset involved (see [A] for adopted benchmarks and metrics, namely the Remove and Re-train (ROAR) metric). Since focus on shape features if proposed as a cause for declining source and target conditional distributions, including measures from the aforementioned points would strengthen this claim.\n\n[A] - A Benchmark for Interpretability Methods in Deep Neural Networks", "extra_scores": {}, "extra_text": {}}, {"review_id": "-nMDSrTbZtl", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "summary:\nThis paper introduces distributional robust learning (DRL) for the unsupervised domain adaptation task. The proposed DRL based approach is claimed to better measure model prediction during self-training with an auxiliary discriminator. This paper also recognizes an optimization issue of conventional DRL and proposes an end-to-end solution via density ratio estimation. \n \nHowever, I have the following concerns,\n\n1.The generalization error (both theoretically and empirically) of the gradient approximation is unclear. It is necessary to analyze how effective and under what conditions the proposed approximation can work for the expected target loss optimization.\n\n2.It needs elaboration why the density ratios can be directly replaced as discriminator predictions, which seems not straight-forward and is the main difference to the conventional DRL. \n\n3.There are some related work not covered in the paper. It is quite surprising that Ben-Tal et al. (2013) is not discussed, also a few later application work (Shafieezadeh-Abadeh et al. 2015; Sinha, Namkoong and Duchi 2018; Hu et al. 2018). More importantly, some existing study already touched high-dimension data and self-training (19NIPS). A thorough discussion of how this work makes unique contributions is required.\n\n4.The baseline results on Office31 and Office-Home datasets are missing. Besides, since it claims to better measure prediction uncertainty, a direct comparison of prediction confidence with the baselines could better justify the superiority of the proposed approach.\n\n\nBen-Tal et al. \"Robust solutions of optimization problems affected by uncertain probabilities,\" Management Science, 2013\nShafieezadeh-Abadeh et al., “Distributionally robust logistic regression,” in NeurIPS, 2015\nSinha, Namkoong and Duchi, “Certifiable distributional robustness with principled adversarial training,” in ICLR, 2018.\nHu et al., “Does distributionally robust supervised learning give\nrobust classifiers?” in ICML, 2018\nNajaf et al., \"Robustness to Adversarial Perturbations in Learning from Incomplete Data,\" NeurIPS, 2019\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "-R3HZVynElR", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I find the paper to be well motivated. Self-labeling has proven to be a useful approach for unsupervised domain adaptation. And since wrong pseudo-labels in the target domain result catastrophic failure in early iterations, it makes sense to calibrate the production of pseudo-labels through the use of uncertainty estimation. This is done through the framework of distributionally robust learning. \n\nI find three central concepts in the method: First, the formulation of the problem as a min-max game between the adversary and the predictor (appears in Eq.1 and Eq.3). Second, the relaxation of the predictor into a formula involving a ratio of densities (Eq.2, Eq.4 and Eq.5) and third the estimation of the gradients of the target loss through the use of source data. While the individual components seem well motivated and clear, I struggle to see the connection between them. For instance, how does figure 1(a) related to figure 1(b)? What is the intuition of using a ratio of densities as a good uncertainty estimation? How can source data be used to approximate gradients on the target? A clearer explanation of the connection could be useful or at least a sketch of the proofs be provided in the main text. \n\nIn addition, while Alg.1 is shown to be a relaxation of the original formulation, Alg.2 seems to be an adhoc attempt to combine the DRL framework with self-training. Why is this the correct formulation to use? At the very least, an experimental comparison to other possible approaches can be used - for instance, one can apply a standard UDA approach instead of Alg.1 and choose, based on standard confidence measures, the top confident labels for a subsequent iterations of Alg.2.\n\nThe use of three different datasets though a variety of experiments demonstrates the usefulness of the approach.  However, some experimental evaluation is missing in my view. First, an ablation study for the different components of both Alg.1. and Alg.2 would be usefull. For instance, in Alg.1, viewing the density ratios as an uncertainty estimation, what is the effect of using other approximations for the uncertainty, either by the adjusting this ratio, or using existing uncertainty estimation methods. How does a different choice for the flexibility of the adversary affect UDA results (Eq.3)  For Alg.2 what is the effect of using different pseudo-label portion p? \n\nFurther, the paper claims to better captures the shape features, but this is not shown.  One could use the evaluation shown in [1], for instance, to demonstrate this. \n\nOverall, on the positive side,  I found the problem well motivated and important. The individual components of Alg. 1 are well explained and shown theoretically. The overall approach well evaluated on 3 datasets and some of the components evaluated.\nOn the negative side, clarity of the paper could be improved, particularly with regards the the connection between the main components. Additional experiments, and an in particular an ablation study regarding Alg1. and Alg.2 would provide a much better understanding of the method. \n\n[1] ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. ICLR 2019. \n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a rather complex algorithm for unsupervised doamin adaptation.
While the paper provides detailed explanation, some motivation and some experimental resulst,
it does not provide any theoretical guarantees for its performance. More concerning, since domain adaptation
can only succeed when there is a close relationship between the source and target tasks, and only with algorithms 
that take that relationship into account, any scientific proposal for domain adaptation should include a clear
discussion of the assumptions driving the proposed algorithms and of the circumstances under which the proposed approach 
may or may not work. This is missing in the current submission. 

More specifically, a similar ocncern was voiced by Reviewer 3 
Namely ".The generalization error (both theoretically and empirically) of the gradient approximation is unclear. It is necessary to analyze how effective and under what conditions the proposed approximation can work for the expected target loss optimization." Thsi point was not addressed in teh authors' rebuttal.

Anotehr key concerning point that was also brought up by reviewer 3 read:
"It needs elaboration why the density ratios can be directly replaced as discriminator predictions, which seems not straight-forward and is the main difference to the conventional DRL." In response the authors cite the paper by Bickel et al 2007 but it falls short of addressing the well know fact that density ratio cannot be reliably estimated from samples of bounded size. The authors should have explained specific assumptions that can make this step of their algorithm og through.
Decision: Reject