Implicit Task-Driven Probability Discrepancy Measure for Unsupervised Domain Adaptation
Authors: Mao Li
Kaiqi Jiang
Xinhua Zhang
Venue: NeurIPS 2021 Poster (2021)
Keywords: probability discrepancy measure
unsupervised domain adaptation

Abstract: Probability discrepancy measure is a fundamental construct for numerous machine learning models such as weakly supervised learning and generative modeling.  However, most measures overlook the fact that the distributions are not the end-product of learning, but are the basis of downstream predictor.  Therefore it is important to warp the probability discrepancy measure towards the end tasks, and we hence propose a new bi-level optimization based approach so that the two distributions are compared not uniformly against the entire hypothesis space, but only with respect to the optimal predictor for the downstream end task.  When applied to margin disparity discrepancy and contrastive domain discrepancy, our method significantly improves the performance in unsupervised domain adaptation, and enjoys a much more principled training process.

Implicit Task-Driven Probability Discrepancy Measure for Unsupervised Domain Adaptation
Mao Li, Kaiqi Jiang,
Xinhua Zhang
Department of Computer Science, University of Illinois at Chicago
Chicago, IL 60607
(ali206,kjjaane zhangx)Quic.
Abstract
Probability discrepancy measure is fundamental construct for numerous machine learning models such as weakly supervised learning and generative modeling. However, most measures overlook the fact that the distributions product of learning, but are the input downstream predictor. Therefore, it is important warp the probability discrepancy measure towards the end tasks, and towards this goal, we propose a new bi-level optimization based approach so that the distributions
not the end-
compared not uniformly against the entire hypothesis space, but only with respect to the optimal predictor for the downstream end task. When applied to margin disparity discrepancy and contrastive domain discrepancy, our method significantly improves the performance in unsupervised domain adaptation, and enjoys much more principled training process.
Introduction
Discrepancy measures on two distributions underpin large variety of machine learning tasks, and have been studied extensively since the dawn of modern probability [1]. For example, in generative models, such measure is applied to align the generated distribution with the empirical one, and prevalent examples include 1) the f-divergence that admits a convenient variational form hence can be effectively evaluated via sample-based adversarial optimization [2, 3]; 2) integral probability metric [IPM, 4] that seeks the largest discrepancy in function expectation over reproducing kernel Hilbert space (RKHS) [MMD GAN, 5-7], I-Lipschitz continuous functions [Wasserstein GAN, 8, 9], or unit La norm functions [Fisher GAN, 10). etc.
In domain adaptation [DA, 11, 12], probability discrepancy is also the key construct in the feature adaptation approach, where a feature extractor & is sought to align the source and target distributions transformed by & [13, 14]. The aforementioned measures can be applied directly in this context.
has been long noted that the discrepancy should be tailored to the function class of interest, e.g., those for which would like to compute expectations. This principle has been applied to density estimation [15] amongst others, where the RKHS is selected to match the downstream task such as image categorization based the compressed pixel distribution. Naturally this motivation can be easily implemented in IPMs by customizing the generating function space.
However such tailoring remains oblivious to the loss and available labels of the end task. Intuitively, the latent features in DA are to be used for classification, then whether the loss is AUC or F-score should idcally influence the probability discrepancy. The seminal H.A -diverreeee is designed for classification accuracy [16], with few extensions to Bayesian and other losses [17-20]. Despite being data-dependent, however, they are unsupervised without accounting for the available labels. Likewise, if a generative model is used to augment data so as to improve segmentation accuracy [21], then the adversarial network in GANs should not only be able distinguish between real and synthetic, but also "align", an appropriate sense, with the segmentation labels at hand.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Warping probability discrepancies towards a task has been lightly touched in unsupervised DA (UDA). [22] trains two classifiers that not only boost the source-domain accuracy, but also maximally disagree on the target domain. Unfortunately, it is only formulated as a procedure, not a probability discrepancy. Most relevant to our work the margin disparity discrepancy IMDD, 23], which is based on the HLA-H-divergence where two fictitious classifiers h and h are jointly optimized to maximally reveal the two distributions' difference. [23] took the key insight that /t can be tied with the source-domain predictor, and can thus be optimized to simultaneously reduce the source-domain risk and the -iveeriveg,, However, in spite of its effectiveness both theory and practice, we discover that the specific formulation conflicts with the HAH H-divergence maximize over h so as to promote the divergence, while MDD tries to minimize it (Section 3). This undermines the power of MDD in distinguishing two distributions illustrated in Figure 1. Flipping the sign and min/max cannot resolve the issue.
the latter tries to
Our first contribution, therefore, is to develop new task-driven discrepancy framework that over- comes this obstacle. The key inspiration that MDD relies on the pseudo-label in the target domain (i.e., speculation of their labels based, e.g., on the source-domain head), and this is also the case for some other measures such as the contrastive domain discrepancy [CDD, 24], which promotes the proximity between the class mean of the two domains for each class, and pushes apart the mean of different classes. Such a commonality motivates us to generate the target-domain pseudo- based on the optimal -ource-domin classifier h". In MDD, this provides natural substitute for the fictitious classifier h (Section 3.2) which no longer needs to be optimized over, thereby solving the aforementioned problem. As our second contribution, we extend this strategy to CDD in Section 4. The overall formulation becomes a bi-level optimization solvable by implicit differentiation (hence the modifier "implicit" in the method's 's name).
We note in passing that pseudo-label is commonly used in self-training for UDA [25-28]. However, most methods require various refinements of it in order to mitigate its inaccuracy due to distributional shift [29]. Examples include label sharpening [30], entropy reweighting [31], cycle training [29]. We instead directly use the output of I as the pseudo-label in probability discrepancy, outperforming state of the art on a range of datasets (Section 6).
UDA has recently received considerable interest, and most algorithms rely on ad-hoc heuristics; we will mention a few below. Many of them require perusing the code and configuration script. As such, our main goal is not to develop yet another highly engineered model that performs better, but to present a principled formulation solvable by off-the-shelf optimizers. Although our implicit task-driven discrepancy can be straightforwardly applied to generative models, we deem it better use of space to fully demonstrate its power in UDA. Such probability discrepancy can also be easily extended to measure (conditional) independence. which has witnessed immediate application in fair and disentangled representation learning [32-35].
2 Preliminaries
In UDA, there is source domain and target domain, and they are respectively represented a joint distribution S and T on an input-output space x V. We will denote their marginal distributions via subscripts, e.g., S5 and Ty. The y domain can be multiclass with labels [C] == {1, 2, ..., C). We are provided with labeled examples in the source domain, denoted as an empirical distribution S. On the target domain, however, we can only access unlabeled examples, i.e., an empirical distribution T2 which only encompasses the input part of an empirical distribution T. In short, let the empirical distributions consist of {aq, vt M2 and {a 17 for the source and target domains respectively.
The goal of UDA is to find classifier that predicts well on the target domain T'. This is often referred to as inductive learning, while, in contrast, transductive learning is only concerned with the prediction on the empirical distribution T, whose feature component T, is available at training time.
The classification model, shared by both domains, consists of feature extractor (e.g., ResNet) parameterized by ó and a head ho parameterized by 0. Letting l be the loss over the ground-truth label and the prediction ho ()(x)), we seek the @ and @ that minimize the target-domain risk
E E( f(y, ho()())))
or its empirical counterpart
E(E.n)) Ely, ho()))))
(1)
In order to leverage the labeled data from the source domain and the unlabeled torget- data, the feature adaption approach enforces low empirical risk the source domain (thanks to the availability

---

of labels there) and encourages that the source domain distribution, after being transformed by the feature extractor 6, "aligns" well with that of the target domain [13, 14, 36, 37]. This is achieved by
min E (x.))~ 0,0
Ely, ha ha(o(z))) + d(o#$2, oTT2).
(2)
where d#5S, is the pushforward distribution of 8z, and denotes some discrepancy measure between two distributions. The intuition is that by "mixing" the latent distributions across the two domains through & the favorable accuracy of hg on the source domain can be transferred to the target domain. For simplicity, we will denote P == ##5x and P := p#S.. and explicitize its dependency on ø by writing Ps whenever necessary. With o(z), we can derive conditional distribution of given based on S, and we denote it Syls- Similarly, let o# Q = ## and define Ta analogously.
3 Implicit Task-Driven Margin Disparity Discrepancy
There has been a plethora of research on sample-basse based discrepancy measure between two distributions. Examples include maximum mean discrepancy [MMD, 38], and (neural) variational optimization [39] which effectively subsumes number of adversarial learning based measures [2, 14).
However, these methods are oblivious to the subsequent tasks that are based on and Q. For example, UDA can be aimed to classify well on these distributions. In Nomain-adversariaa neural networks [DANN, 14], the discrepancy between P and Q is measured via the -ensen-Shannon divergence, reformulated in an adversarial objective in the generative adversarial network [GAN, 40J. Moreover, MMD simply measures the largest possible difference the function expectation over P and Q:
MMD(P,Q) () = sup SEM:ISSI
- E k(x,)) x~P -
E amP
f(x)
f(0)
E k(x, 2~2
111; (3)
a~e
where H. is the reproducing kernel Hilbert space (RKHS) induced by kernel k. Obviously, it does not take into account whether f is used for classification or regression. The celebrated H.A H.A--divergence addresses this problem by focusing on binary classification [16, Lemma
dHAN (P. Q) = max D(h, P,Q),
MCH K'E
where D(h, K', P,Q) == Ep[sign oh' * sign oh] - Eq Isign oh' A sign oh]].
(5)
Here sign oh applics the sign function on the output of h. H. is a hypothesis space (not necessarily an RKHS), and I-1 is the Iverson bracket that evaluates to 1 if is true, and 0 otherwise. However, it still does not concern the label of the data (i.e., align only in an unsupervised fashion). To warp the measure to the end-task in data-dependent fashion, [23] proposed the margin disparity discrepancy (MDD), which improved upon [22] by formulating a principled objective function instead a heuristic procedure. According Equation 24 of [23], MDD employs
dMDD (P. Q) min R(h; P) + max D(h, D(h. ..O0). hew
(6)
W'HH
where R(; P) : P. ((h(2),)) v) reg(h) is the regularized risk,
and the 0-1 loss in D can be replaced by smooth surrogates such as hinge loss or cross-entropy loss. Here reg is any standard regularizer applied in regularized risk minimization, e.g., f2 norm. The underlying insight is that when comparing P and Q, one only needs to consider those h that predict well on the (labeled) source domain, while leaving h' to reveal the maximum discrepancy between P and Q. Similar ideas have been leveraged in [22, 41].
3.1 Conflict between MDD and H.AH-divvegeence
Unfortunately, dMDD turns out conflicting with the spirit of -diverrieee in an important way. Note that h is maximized in D as (4), while it minimized dMDD as in (6). This raises a natural question: can the distribution discrepancy be sufficiently revealed when max, is replaced by min, in the definition of D. i.e..
d
min HAH
(P. Q)
min max D(h, P.Q).
(8)
hEH MEH
3

---

h
+
h
+
(a)
(b)
Figure I: An example showing that changing max; into min; undercuts the power of discriminating two distributions. Here the source distribution P has two blue clusters, and target distribution Q consists of two red clusters. The location of h in (a) makes max Dhh, h P. Q) O. meaning that the new discrepancy min H (P. Q) cannot distinguish the two distributions. contrast, the h in (b) makes max D(h, h Q) L, implying that the original dHAH (P,, Q) can distinguish.
turns out such a change does undermine the discriminative power, and an example is illustrated in Figure 1. Here both the source and target domains have two separate clusters, and the hypothesis space the horizontal or vertical half spaces (i.e., decision stumps). Sub-figure (a) shows that the minimum h in d HAA is attained at the horizontal line, and it is easy to check that no matter where h' placed, D(h, R², P. Q) = O. contrast, the h and / shown (b) attain D(h, h², P. Q) = So changing maximization of h into minimization caused significant loss in the discrimination power. A more detailed discussion conjunction with as in (6) is available in Appendix
3.2 A new implicit task-driven MDD
Flipping back the optimization of h turns out far more involved that it appears. It cannot be achieved by simply changing ming into max,, in (6) with the source domain risk negated:
- R(; P) + max D(h, P.Q)).
max KEH
WEH
This is because P and Q indeed depend on the feature extractor Φ. If we next minimize dMDD (P, Q) over é, then it implicitly promotes the source domain risk. If dMuDD Q) is instead maximized over é, then 6 would attempt to increase the discrepancy D. With a few trials, becomes clear that the same issue persists in other combinations of flipping sign or min/max.
Our first contribution, hence, is to resolve this issue by turning dWDD into constrained formulation:
max max D(h, P,Q), (ERER(EEP)SA
(10)
where À is some pre-specified cap of loss. Constraining the performance of classifier is quite commonly used in, e.g., gradient episodic memory to combat catastrophic forgetting [GEM, 42, 43]. However, GEM only solves a linear approximation instead of the exact problem, and it is arguably difficult to differentiate through for optimizing ő in (10). Therefore, we finally develop a bi-level optimization by setting f to the optimal one for the source domain, and then using it in the discrepancy measure. We call it i-MDD because it will rely on implicit differentiation for training. The overall training objective can be written as:
min do (Po Qo) + RR(h";
where
diMDD(Po 2+) = max D(h", H', Po. Q0), (11) K'EH
h" == arg min R Pg).
(12)
Here a is a tradeoff parameter. If we do not include R(; Pa) the objective, then the feature would receive no incentive to reduce the source-domain risk. This term in the objective function does not necessitate new implicit differentiation, because ht" is exactly the minimizer of R(h;; Pa). The architecture of i-MDD shown in Figure 2, comparison with MDD.
'One might wonder why the max over in (10) turned into min over & (12). This is because À set to ming Rh; P). Analogously, maximizing f f(, y) over (x 1)² < 0 is equivalent to evaluating f(1, v). because 0 is the minimum of (x - 1)² attained at 1. Or to bear more resemblance to (11) and (12), it is equal f(x", y) where x² argmin, (xr 1)

---

Source
Risk
Feeture Extrecto
Mirmi
-zation
urg min (M) be
Source
Target
dMDD
de MDD
(a) i-MDD
(b) MDD
Figure 2: Illustration of i-MDD and MDD. The h", fed into dR-DDD in i-MDD, is the minimizer of R.
3.3 Practical discussions: differentiable surrogates
Since the 0-1 loss in D is not amenable to differentiable training, we follow [23] to morph it into the cross-entropy loss (CE). In particular, suppose h outputs C dimensional logit vector, and
p softmax(h), Similarly, y' softmax(h'). Then the standard CE(p', p) = -E log 2 0. To combat exploding or vanishing gradient, [3] proposed modified CE: MCE(p', p) =;,, Pi log(1
pt) I 0. Then [23] adopts the approximation
D(h, h1, Po. Q+) R Eg. MCE(y', ind p, CE(P', inddiindo. o), (7
(13)
where ind RC - IC is the indicator function mapping a vector v to the i" -"-th canonical vector with i* arg max; Ug. In practice, the formulation has two issues. First, the right-hand side of (13) is unbounded from below, making it possible for & (the minimizing variable) to push it to the negative infinity when solved by stochastic saddle-point optimization. As result, the implementation of [23] tuned the step size delicately. Secondly, the indicator function ind blocks the backpropagation through the branch of h. jeopardizing the proper optimization. We tried removing the indicator function but observed negative infinity even after finely tuning the step size.
In contrast,
i-MDD is immune to these issues, where (13) is used without including the indicator function. In our experiment, we observed that the head hg only needs to be linear in order achieve state-of-the-at performance. This provided considerable convenience because the optimization for h" in (12) can be accomplished very efficiently with high precision by convex solvers such as LIBLINEAR [44]. Similarly, it is clear that he does not depend on h', but on Ps only (i.e., Φ). Therefore, we can first solve h² in (12), and then fix it when solving h in (11), which results in another convex problem thanks to the linearity of lt. These conveniences significantly benefit computation and convergence properties.
Although MDD can forgo the stochastic muddle-point optimization and also evaluate dMDD exactly, the inner joint maximization over h and h' leads to non-concave function, hence impairing the precision of backpropagation. Even if the indicator function is imposed and optimization only over h', we found linear h was insufficient to deliver accurate predictions.
3.4 Bi-level optimization
Bi-level optimization has recently received intensive study [45-48], and they can be easily applied to i-MDD. Thanks to the linearity of h and h', the backpropagation can be performed a closed form. Denote the ultimate objective value in (11) as J. Letting 2f <(x;)) and 2 o(x)). only need to derive new strategies to compute 0J/azş and 0J/Ozf. based on which backpropagation through the feature extractor will be standard. Towards this end, most of the implicit differentiation approaches rely on multiplying a given vector the Hessian of the loss in (12) with respect [45]. Interestingly, for linear multi-class classifiers with cross-entropy loss, the formula has already been derived by [49, Appendix DJ, and we quote their results in Appendix for completeness, along with the detailed analysis of computational complexity.
To summarize, the of i-MDD is replace the h in the -ivverrieeeee by the optimal source domain classifier A* under the current Φ This is in line with the pseudo-label approach and λ* be applied the target domain to provide soft label. Indeed this principle can be applied to other class-aware discrepancy measures, and our next contribution is to warp the contrastive domain discrepancy [CDD, 24] towards the end task.
5

---

Task-driven Contrastive Domain Discrepancy
Underpinning CDD is the hard pseudo-label tj E [C] assigned to each target domain example aj. [24] adopted clustering on zj, where each class corresponds to cluster, and its center is initialized by the mean of the source domain 27. Naturally, 9/ is set to the cluster that 2j belongs to at convergence.
Then the discrepancy between and Q is defined as (distilled from Equations and [24])
dcDD = 2 Σ F -
Tπट- E
l - cFc
(14)
cE|C]
intra-class discrepancy
inter-class discrepancy
where
Me =
mean (aaan((k(2)) [b[(] ; [na] and yf = cc, EcE pe = mean (A(2, : E [me] and 0j = =c, Ece
(15)
(16)
Here 3 > is tradeoff coeffient. The underlying motivation is align the class-wise center between source and target domains (the intra-class discrepancy), and push apart the centers of different classes (the inter-class discrepancy). Although the source-domain label is used to initialize clustering, the prediction head h is not involved in dCDD, hence not sufficiently driven by the end task.
In addition, a number of heuristics are required for CDD to perform well. Firstly, after clustering, only the target-domain examples that are close to the center are included to compute the mean M.. This introduces one hyperparameter to tune. Secondly, domain specific required. Finally, the bandwidth of the RBF kernel needs be learned for each pair of (c, d') in the implementation. To remove all these nuisances and formulate principled optimization, we next warp CDD towards tasks based on bi-level optimization.
is
4.1 Implicit task-driven CDD
Our key insight is that the head h" in (12) constitutes a natural source of pseudo-label that is superior clustering. Firstly, h' is uniquely determined thanks to the convexity originating from the linearity of h. Moreover, clustering is a "procedure" which is not amenable to differentiation despite some recent progress in reversible learning [46]. In contrast, differentiation through h is straightforward as discussed above.
This intuition can be directly implemented by redefining the class centers in the target domain based on the ht -inducdd soft pseudo-label for each example z/. Recall h (2²)) produces the C-dimenssioall logit (unnormalized score) for the C classes, and the softmax of it yields C-dimensional probability vector pýj. whose e-th element encodes the probability of belonging to class c. Accordingly. we can morph the target-domain center p.: into
/11- Edstoo
= = 1=1
(r)) (())==>())+ 10-
where p = (of)max(((E))) E
(17)
Note the kernel i removed and we directly used 2/. We also added a small smoothing factor 10-6 in case all examples are unlikely to belong to class To summarize, our training objective is
min d4CDDo( (Pos Qo) + R((";; Pb)
(18)
where di Po. ②+)
Σ CE|C]
2 p'((h*) I| B C(C - 1)
Σ exce
llke
11
(19)
C
h" := arg min R(h; Pa). KER
(20)
It is clearly identical to i-MDD in (11) except that the dDDDD is replaced by d:-CDD- Compared with deup (14), we slightly changed the inter- term from between source and target domains (n - v()) into within source domain only (n² - -.)). This simplifies optimization because the centers the source domain do not depend on hi* Meanwhile, different classes are still pushed apart in both domains because 1) it is enforced on the source domain, and 2) the source domain centers μ are aligned with those of the target domain (h((h)) Backpropagation and bi-level optimization are similar to i- i-DDD, with even reduced complexity optimization (over h') is involved in d.-cD-

---

4.2 Cache-augmented training
was noted in [24] that the limited size of mini batch may leave only a small number of examples for each class (or even none), especially when there many classes. This hampers the computation of class means. They thus resorted to class-aware sampling strategy where only a subset of classes picked at each iteration, and samples are drawn only for these classes. This again relies the result of clustering for the target domain, exacerbating the fallout of not backpropagating through it.
To address this issue, we followed [50, 51] by caching the latent representations z in the most recent iterations via a circular queue for each class. This allows the class means to be computed more accurately, and the backpropagation is still conducted only on the current mini-batch examples.
We emphasize that our overall optimization remains principled even with cache augmentation, an observation that has not been made in literature to the best of our knowledge. Since & is updated with a small step size and only a small number of latest iterations are cached, the continuity of the algorithm ensures that the 2 computed from stale ø is still close to the value if it were computed with the latest Φb As a result, the bias of the gradient can be bounded linearly by the step size times the staleness (i.e., the length of the queue / mini-batch size). We relegate the details to Appendix C.
Related Works in Unsupervised Domain Adaptation via Feature Adaptation
Although our motivation is to develop a task-driven probability discrepancy measure while UDA is used only as an example application, we would also like to place our approach the context of UDA literature. A detailed and recent survey is available in [52], and we will only focus of methods that are most related to our approach, namely feature adaptation based methods. These methods seek feature extractors that the source and target domains are aligned in the feature space, hence called domain-invariant feature representations [53]. Although methods may differ in whether different domains share the feature extractors in part, in whole, or none, the most prominent variation lies in the alignment metric.
category
Conventional probability discrepancy measures include Jensen-Shannon divergence used by GAN and DANN, and Wasserstein distance [54, 55]. The MMD in (3) has multiple variants such as multiple kernels [13] and joint MMD [56]. When the kernel is not universal, e.g. polynomial, MMD essentially compares the statistics such as the variance ssecond-orders and various comparison metrics have been studied [c.g.. 57, 58]. CDD further accounts for the source-domain label information (but not the target-domain head) via the intra-class and inter-class distances.
Several adversarial methods try to align the domains by demoting the features' discriminative power in identifying the domain. The idea can be traced back least GAN, and example variants include DANN and [31, 59-61].
Experimental Results
We finally validate the implicit task-driven discrepancy by comparing i-MDD and i-CDD against state-of-the-ar methods for unsupervised domain adaptation, especially MDD and CDD. Ablation studies will also be carried out to examine the influence of various components. More details on the experiment setup and results are available in Appendix D.
6.1 Comparison of target-domain accuracy
Datasets. We adopted three public domain datasets for UDA benchmarking.
Office-31 [62] is standard dataset for real-world domain adaptation. It consists of 4,652 images belonging to 31 unbalanced classes. These images are collected from three distinct domains:
Amazon (from Amazon website), Webcam (from web camera) and DSLR (by digital SLR camera).
Office-Home [63] is more challenging dataset for visual domain adaptation. It contains 15,500 images of daily objects office or home environment, belonging to 65 categories. The images are sampled from four domains: Artistic images, Clip Art, Product images, and Real-world images.
7

---

ImageCLEF-DD [64] consists of images from three domains: Caltech-256, ImageNet ILSVRC 2012 and Pascal VOC 2012. Each domain has 12 categories and each class contains 50 images.
Baselines. We compared our i-MDD and i-CDD with the following state-of-the-art UDA methods: Deep Adaptation Networks (DAN) [13], Domain Adversarial Neural Network (DANN) [14], Residual Transfer Network (RTN) [65], Joint Adaptation Networks (JAN) [64], the Entropy Conditioning Variant of Conditional Domain Adversarial Network (CDAN+E) [31], Multi-Adversaria Domain Adaptation (MADA) [66], Conditional Domain Adversarial Network with Batch Spectral Penalization (BSP+CDAN) [67], CDD [24] (which named it Contrastive Adaptation Network), Cluster Aligmment with a Teacher with Robust Gradient Reversal (rRevGrad+CAT [68], MDD [23], MDD with Implicit Alignment (MDD+IA) [69], and Adversarial Spectral Adaptation Network (ASAN) [70].
We also considered a variant of CDD (named vCDD) where μº u is replaced by μ - - in source domain only, and the class-aware sampling in [24] is replaced by cache augmentation. This allows us to compare i-CDD with the exact counterpart that does not use bi-level optimization.
Additional comparisons with some state-of sarcc-of-hh--ate ft-th--art methods that are not based on feature adaptation are available in Appendix D.3.
Implementation details. We followed the commonly used experimental protocol for unsupervised domain adaptation from [14]. We report the average accuracy and standard deviation of five inde- pendent runs. For i-MDD we mainly used the hyper-paramettn
from [23], i.c., the margin factor y in (13) was chosen from {2, 3, 4} and was kept the same for all tasks the same dataset. For i-CDD, the trade-off coefficient 3 between intra-class loss and inter-class loss in (14) is chosen from (0.1, 0.001}. The cache size for each class is 30.
We implemented our methods in PyTorch. The head classifier (in both i-CDD and i-MDD) and the auxiliary classifier (h' in i-MDD) are both I-layer neural network with width 1024. We did not restrict MDD and CDD to single-layer h or it..
For optimization, we used mini-batch SGD with Nesterov momentum 0.9. The initial learning rate was 0.004, which was adjusted according [14]. The mini-batch size is 150 for each domain. More detailed explanation of thper-parameter selection is presented in the supplementary materials, along with the sensitivity analysis of them. ResNet-50 pretrained on ImageNet was used as the feature extractor in all methods. Since our aim is to improve the probability discrepancy measure for UDA, we employed the standard backbone ResNet-50 instead of integrating weight feature extractors, ad-hoc engineering heuristics, or generic feature improvements [e.g., 71].
The accuracy of farget-domain prediction is presented in Table 1 for Office-31, 31, Table 2 for Office-Home, and Table 3 for ImageCLEF. Clearly i-CDD achieves the highest average accuracy among all methods over all datasets. As zoom into each pair of domain, it is also either the best performer or close to the best. Secondly, by comparing vCDD with i-CDD and MDD with i-MDD, it is clear that the implicit (i.e., bi-level) formulation can significantly boost the performance upon the standard joint optimization, except i-MDD on ImageCLEF where it is a tie. This validates our original motivation. Thirdly, vCDD outperforms CDD two datasets and ties on Office-31, implying that computing the inter-class discrepancy based solely on the source domain is superior to that based on both source and target domains. This makes sense because ground-truth labels are only available for the source, and the pseudo-labels for the target domain can be noisy and detrimental.
Results.
Overall, i-CDD is superior to i-MDD. This makes sense because i-CDD not only matches the center of each class between source and target, but also promotes the inter-class discrepancy, i.c., pushing apart the center of different classes. The latter "contrastive" component appears quite beneficial.
Finally, MDD+IA can often outperform MDD, and although i-MDD achieves significantly higher accuracy than MDD+IA on Office-31, it is less competitive on the other two datasets. This does not invalidate our implicit task-driven principle, and we can implicitize MDD+IA for future work.
6.2 Ablation study
We next examine the influence of several important components of i-DD and i-CDD, including the cache size (queue length) i-CDD, the dimensionality of hidden representation, i-CDD equipped with the class-aware sampling [24]. All the ablation studies were conducted on Ar:Cl in Office-Home
8

---

Table 1: Accuracy (%) on Office-31 for unsupervised domain adaptation (based ResNet-50)
Method
A
W
D - W
W -
A
D A
W
A
Avg
ResNet-50
68.4
96.7=0.
99.3
68.9 +.2
62.5 =3
60.7 0.3
76.1
DAN
80.5 104
97.1 =.2
9.
63.6 13
62.8 0.2
80.4
DANN
82.0
96.9 002
99.1
79.7
68.2 u4
6.
82.2
RTN
84.
96.8=00
99.4
77.5
66.2
u22
64.8
81.6
JAN
85.4
97.4 0.2
99.8
84.7 1.33
68.6
70.0
84.3
CDAN+E
9..1
98.6 0.1
100.0 1:00
92.
71.0
69.3
87.7
MADA
90.0
97.4 0.1
99.6
87.8
70.3 103
66.4
85.2
BSP+CDAN
93.3
98.2 0.2
100.0
932
73.6
72.6
88.5
CDD
93.
99.1 0.2
99.8
95.0
78.0 =0
77.0
90.6
IRevGrad+CAT
94.4
98.0 0.2
100.0 000
90.8
72.2
70.2
87.6
MDD
93
98.4 0.11
100.0
93.5
74
72.2
88.9
MDD+IA
90.3
98.7 0.1
99.8
92.1
75.3 =2
74.9
88.8
ASAN
95.6
98.8 0.2
100.0 0.
94.4
74.7
74.0 09
90.0
vCDD
95.1 0.7
98.4 0.3
99.5 0.3
94,8
76.2 0.
76.9 0.6
90.6
iCCD
0.
98.5 0.2
100.0 + 0.0
96.3
77.2 03
78.3 - 0.2
90.9
i-DDD
94.8 0.5
98.4 0.3
100.0 = 0.0
94.2 +
75.1 05
74.1 = 0.7
89.4
Table 2: Accuracy (%) on Office-Home for unsupervised domain adaptation (based on ResNet-50)
Method
Ar:CI Ar:Pr Ar:Rw Cl:Ar Cl:Pr Cl:Rw Pr:Ar Pr:CI
Pr:Rw Rw:Ar Rw:CI Rw:Pr
Avg
ResNet-50
34.9
50.0
58.0
37.4
41.9
46.2
38.5
31.2
60.4
53.9
41.2
59.9
46.1
DAN
43.6
57.0
67.9
45.8
56.5
60.4
44.0
43.6
67.7
63.1
51.5
74.3
56.3
DANN
45.6
59.3
70.1
47.0
58.5
60.9
46.1
43.7
68.5
63.2
51.8
76.8
57.6
JAN
45.9
61.2
68.9
50.4
59.7
61.0
45.8
43.4
70.3
63.9
52.4
76.8
58.3
CDAN+E
50.7
70.6
76.0
57.6
70.0
70.0
57.4
50.9
77.3
70.9
56.7
81.6
65.8
BSP+CDAN
52.0
68.6
76.1
58.0
70.3
70.2
58.6
50.2
77.6
72.2
59.3
81.9
66.3
CDD
51.6
71.2
76.7
59.8
70.8
70.8
59.8
49.9
77.4
70.6
58.8
80.5
66.5
MDD
54.9
73.7
77.8
60.0
71.4
71.8
61.2
53.6
78.1
72.5
60.2
82.3
68.1
MDD+IA
56.2
77.9
79.2
64.4
73.1
74.4
64.2
54.2
79.9
71.2
58.1
83.1
69.5
ASAN
53.6
73.0
77.0
62.1
73.9
72.6
61.6
52.8
79.8
73.3
60.2
83.6
68.6
vCDD
56.2
74.2
77.0
62.4
72.3
71.4
61.7
61.4
78.7
71.3
60.6
81.7
69.3
iCDD
60.8
77.5
78.8
64.3
74.3
73.4
65.3
61.9
78.7
72.1
61.8
81.8
70.8
i-DD
56.5
74.7
78.3
61.9
72.4
72.3
63.2
55.6
78.4
71.4
59.7
81.7
68.8
Impact of cache size in i-CDD and vCDD. Figure 3 shows the fluctuation of prediction accuracy for vCDD and i-CDD. The accuracy first grows when the length of the queue for each class increases from 1 to 30, corroborating the benefit of cache in improving the accuracy of center means, But then it starts to decay, suggesting that the stale samples accrued start to hurt.
Since there is large number of class compared with the mini- size, the cluster mean cannot be estimated accurately. For example, Ar:Cl of Office-Home has 65 classes while the GPU memory limited our mini-batch size to 150. The cache augments the pool of latent feature values, hence improving the mean estimation. However, an overly large queue size may leave the stored values stale, i.e., inconsistent with the true value if it were computed from the current ResNet é.
Empirically, found it generally effective set the cache size around 50% of the data size (number of images) of each domain. For example, there are about 2000 images in the Amazon website domain of Office-31, and we set the queue length 30 for each of the 31 classes. This amounted to cache size of 30 31 = 930 images. which is about half of 2000. It well balanced the sample size with staleness, and cost only small amount of memory and computation thanks the low dimensionality of the latent feature space.
Impact of latent dimensionality. Figure demonstrates the prediction accuracy of vCDD and i-CDD, when the dimensionality of latent feature (zf and 25) is varied in {128, 256, 512, 1024).

---

Table 3: Accuracy (%) on ImageCLEF for unsupervised domain adaptation (based on ResNet-50)
Method
1 P
p-1
I-C
C+1
C
P C
Avg
ResNet-50
74.8 0.3
83.9 #01
=2
003
91.2
80.7
DAN
74.5 u.4
82.2
92.8
86.3 104
89.8 4.44
82.5
DANN
7..
86.0
96.2
87.
74.3
91.5
85.0
RTN
86.8 =01
95.3
863
92
84.9
JAN
76.8 0.44
88.0
94.7
89.5
74.2
91.7
85.8
CDAN+E
77.7 0.03
90.7
97.7
91.
74.2
94
87.7
MADA
75.0 0.3
87.9
96.0
888
75.2
92.2
85.8
CDD
77.0
89
97.2
91.5
76.2
95.6
87.8
RevvGra++CAT
77.2 0.2
91.0
93
91.3
75.3
93.6
87.3
MDD
0.2
91.1
92
92.1
77.6
93.8
88.4
MDD+IA
0.2
91.8
96.7
93.0
79.0
94.2
88.8
ASAN
78.9
9. 92.3 05
92.1 13
76.4 0.7
94.4
88.6
vCDD
78.8 0.4
92.1 0.1
97.0
91.3 0.3
78.2 + 0.3
96.2 0.4
88.9
iCDD
79.8 0.4
92.6 0.3
97.2 0.4
92.0 -03
78.6
= 0.3
96.5 0.2
89.4
i-DD
78.5 0.6
91.6
96.5 0.4
91.4 0.3
76.8 + 0.6
95.4 0.3
88.4
60.0
60.0
60.0
VED0
VCDD
0.00
57.5
100
57.5
A-D00
57.5
25.0
Accunccy 189
55.0
55.0
52.5
52.5
52.5
50.0
30.0
50.0
47.5
47.5
47.5
3000
1024
10
Cache
Figure Accuracy v.s. cache Figure 4: Accuracy v.s. latent Figure 5: Class-aware sampling size for each class
dimensionality
v.S. cache augmentation
Evidently, increasing the dimensionality tends to improve the accuracy for both methods, but at the cost of more computation.
Class aware sampling cache augmentation. The problem of low sample for each class in mini-batch (ref Section 4.2) was addressed by [24] via class-aware sampling (CAS), where a small number of classes (e.g.. 10) are randomly selected, and mini-batch only draws samples from these classes. Essentially, each iteration is based only on subset of classes, while our i-CDD and vCDD still allow all classes participate via cache augmentation. is therefore of interest compare CAS with cache. As shown in Figure 5, vCDD using CAS enjoys monotonic growth of accuracy as more and more classes are involved in each iteration. When all the 65 classes are used, CAS gets close to cache augmentation. Without cache or CAS, the performance is lower (green line). This partly explains the success of vCDD, which later improved further by i-CDD via the bi-level formulation.
Additional ablations studies are available in Appendix D.4, including the impact of batch size and standard deviations.
Conclusion
In this paper, we proposed warping probability discrepancy measures towards the end tasks by leveraging the pseudo-labels produced by the optimal predictor. Application to unsupervised domain adaptation significantly outperformed the state the art in prediction accuracy, and the training formulated principled optimization problem solvable by standard optimizers. For future work, it will be interesting to extend this technique to warping (conditional) independence measures, and to apply to structured and dynamic settings.
10

---

Acknowledgements
We thank the reviewers for their constructive comments. This work is supported by NSF grant RI:1910146 and NIH grant RO15888827
References
[1] S. T. Rachev. Probability metrics and the stability of stochastic models. Wiley, Chichester, 1991.
[2] S. Nowozin, B. Cseke, and R. Tomioka. f-GAN: training generative neural samplers using variational divergence minimization. In Advances Neural Information Processing Systems (NeurlPS). 2016.
[3] Goodfellow, Touget-Abadic Mirza, Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems (NeurIPS). 2014.
[4] A. Müller. Integral probability metrics and their generating classes of functions. Advances in Applied Probability, 29(2)4429-443, 1997.
[5] C.-L. Li, W.-C. Chang, Y. Cheng, Y. Yang, and B. P6czos. MMD GAN: towards deeper understanding of moment matching network. In Advances in Neural Information Processing Systems (NeurIPS). 2017.
[6] Y. Li, K. Swersky, and R. Zemel. Generative moment matching networks. In International Conference on Machine Learning (ICML). 2015.
[7] G. K. Dziugaite, D. M. Roy, and Z. Ghahramani. Training generative neural networks via maxi- mum mean discrepancy optimization. In Conference on Uncertainty Artificial Intelligence (UAI). 2015.
[8] M. Arjovsky, S. Chintala, and L. Bottou. Wasserstein generative adversarial networks. In D. Precup and Y. W. Teh, eds., Proceedings of the 34th international conference on machine learning. vol. 70 of Proceedings of machine learning research, pp. 214 223.223.222 PMLR, Sydney, Australia, Aug 2017.
[9] I. Gulrajani, Ahmed, M. Arjovsky, V. Dumoulin, and Courville. Improved training of Wasserstein GANs. In Advances in Neural Information Processing Systems (NeurIPS). 2017.
[10] Y. Mroueh and T. Sercu. Fisher GAN. In Advances in Neural Information Processing Systems (NeurIPS). 2017.
[11] J. Quinoonero-Candel M. Sugiyama, A. Schwaighofer, and N. Lawrence, eds. Dataset Shift in Machine Learning. MIT Press, Cambridge, MA, 2008.
[12] S. J. Pan, I. W. Tsang, J. T. Kwok, and Q Yang. A survey on transfer learning. IEEE Transactions on Neural Networks, 22(2):199-21) 210, 2011.
[13] M. Long, Y. Cao, J. Wang, and M. I. Jordan. Learning transferable features with deep adaptation networks. International Conference Machine Learning (ICML). 2015.
[14] Y. Ganin, E. Ustinova, H. Ajakan, P. Germain, H. Larochelle, F. Laviolette, M. Marchand, and training of neural networks. Journal of Machine Learning
V. Lempitsky. Research, 17559)11-35, 2016.
[15] L. Song, X. Zhang, A. Smola, A. Gretton, and B. Schölkopf. Tailoring density estimation via reproducing kernel moment matching. In International Conference on Machine Learning (ICML). 2008.
[16] S. Ben-David, J. Blitzer, Crammer, A. Kulesza, F. Pereira, and J. Wortman Vaughan. theory of learning from different domains. Machine Learning Journal, 12(1-2):151-175, 2010.
11

---

[17] Y. Mansour, M. Mohri, and A. Rostamizadeh. Domain adaptation: Learning bounds and algorithms. Conference on Computational Learning Theory (COLT). 2009.
[18] M. Mohri and A. M. Medina. New analysis and algorithm for learning with drifting distributions. In Conference on Computational Learning Theory (COLT). 2012.
[19] Germain, A. Habrard, Laviolette, and E. Morvant. A PAC-Bayesian approach for domain adaptation with specialization to linear classifiers. In International Conference on Machine Learning (ICML). 2013.
[20] C. Cortes, M. Mohri, and M. Medina. Adaptation algorithm and theory based on generalized discrepancy. In ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD). 2015.
[21] V. Sandfort, K. Yan, P. Pickhardt, and R. Summers. Data augmentation using generative adver- sarial networks (CycleGAN) to improve generalizability in CT segmentation tasks. Scientific Reports, 9, 2019.
[22] K. Saito, K. Watanabe, Y. Ushiku, and T. Harada. Maximum classifier discrepancy for unsuper- vised domain adaptation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2018.
[23] Y. Zhang, T. Liu, M. Long, and M. Jordan. Bridging theory and algorithm for domain adaptation. In International Conference on Machine Learning (ICML). 2019.
[24] G. Kang, L. Jiang, Y. Yang. and A. G. Hauptmann. Contrastive adaptation network for unsuper- vised domain adaptation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2019.
[25] D. hyun Lee. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on challenges in representation learning, ICML. 2013.
[26] A. Kumar, T. Ma, and Liang. Understanding self-training for gradual domain adaptation. In International Conference on Machine Learning (ICML). 2020.
[27] V. Prabhu, S. Khare, D. Kartik, and J. Hoffman. SENTRY: Selective entropy optimization via committee consistency for unsupervised domain adaptation. arXiv:2012. 1460, 2020.
[28] A. Mey and M. Loog. A soft-labeled self-training approach. In Proc. Intl. Conf. Pattern Recognition. 2016.
[29] H. Liu, J. Wang, and M. Long. Cycle self-training for domain adaptation. arXiv:2103.03571, 2021.
[30] K. Sohn, D. Berthelot, C.-L. Li, Z. Zhang, N. Carlini, E. D. Cubuk, A. Kurakin, H. Zhang, and C. Raffel. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. In Advances Neurul Information Processing Systems (NeurIPS). 2020.
[31] M. Long, Z. Cao, J. Wang, and M. I. Jordan. Conditional adversarial domain adaptation. In Advances in Neural Information Processing Systems (NeurIPS). 2018.
[32] N. Quadrianto, V. Sharmanska, and Thomas. Discovering fair representations in the data domain. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2019.
[33] E. Adeli, Q. Zhao, A. Piefferbaum, E. V. Sullivan, L. Fei-Fei, J. C. Niebles, and K. M. Pohl. Rep- resentation learning with statistical independence to mitigate bias. In IEEE Winter Applications of Computer Visions (WACV). 2021.
[34] Locatello, G. Abbati, Rainforth, S. Bauer, Schoelkopf, and O. Bachem. On the fairness of disentangled representations. In Advances in Neural Information Processing Systems (NeurIPS). 2019.
[35] Y. Atzmon, Kreuk, U. Shalit, and Chechik. causal view of compositional zero-shot recognition. In Advances in Neural Information Processing Systems (NeurIPS). 2020.
12

---

[36] B. Li, Y. Wang. T. Che, S. Zhang, S. Zhao, P. Xu, W. Zhou, Y. Bengio, and K. Keutzer. Rethinking distributional matching based domain adaptation. arXiv:2006.13352. 2020.
[37] F. D. Johansson, D. Sontag, and R. Ranganath. Support and invertibility in domain-invariant representations. In International Conference on Artificial Intelligence and Statistics (AISTATS). 2019.
[38] A. Gretton, K. M. Borgwardt, M. Rasch, B. Schoelkopf, and A. Smola. A kernel two-sample test. Journal of Machine Learning Research, 13:723-773, 2012.
[39] N. Wan, D. Li, and N. Hovakimyan. f-divergence variational inference. In Advances in Neural Information Processing Systems (NeurIPS). 2020.
[40] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Bengio. Generative adversarial networks. Advances in Neural Information Processing Systems (NeurIPS). 2014.
[41] B. Gholami, P. Sahu, M. Kim, and V. Pavlovic. disk-diseriminative domain alignment for unsupervised domain adaptation. In 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW). 2019.
[42] D. Lopez-Paz and M. Ranzato. Gradient episodic memory for continual learning, In Advances in Neural Information Processing Systems (NeurIPS). 2017.
[43] A. Chaudhry, M. Ranzato, M. Rohrbach, and M. Elhoseiny. Efficient lifelong learning with A-GEM. In International Conference on Learning Representations (ICLR). 2019.
[44] R.-E. Fan, J.-W. Chang, C.-J. Hsieh, X.-R. Wang, and C.-J. Lin. LIBLINEAR: A library for large linear classification. Journal of Machine Learning Research, 9:1871-1874, Aug 2008.
[45] Lorraine, Vicol, and D. Duvenaud. Optimizing millions of hyperparameters by implicit differentiation. In International Conference on Artificial Intelligence and Statistics (AISTATS). 2020.
[46] L. Franceschi, M. Donini, P. Frasconi, and M. Pontil. Forward and reverse gradient-based hyperparameter optimization. In International Conference on Machine Learning (ICML). 2017.
[47] S. Jenni and Favaro. Deep bilevel learning. In European Conference on Computer Vision (ECCV). 2018.
[48] A. Rajeswaran, C. Finn, S. Kakade, and Levine. Meta-lcarning with implicit gradients. In Advances in Neural Information Processing Systems (NeurIPS). 2019.
[49] Y. Yu, X. Zhang, and D. Schuurmans. Generalized conditional gradient for sparse estimation. arXiv: 1810.488,, 2014.
[50] K. He, H. Fan, Y. Wu, S. Xie, and Girshick. Momentum contrast for unsupervised visual representation learning. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2020.
[51] T. Xiao, S. Li, B. Wang. L. Lin, and X. Wang. Joint detection and identification feature learning for person search. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2017.
[52] G. Wilson and D. J. Cook. A survey of unsupervised deep domain adaptation. ACM Trans. Intell. Syst. Technol., 11 (1(5):1-46 2020.
[53] H. Zhao, R. T. Des Combes, K. Zhang, and G. Gordon. On learning invariant representations for domain adaptation. In International Conference on Machine Learning (ICML). 2019.
[54] N. Courty, R. Flamary, A. Habrard, and A. Rakotomamonjy. Joint distribution optimal trans- portation for domain adaptation. In Advances Neural Information Processing Systems (NeurIPS), 2017.
13

---

[55] B. B. Damodaran, B. Kellenberger, R. Flamary, D. Tuia, and N. Courty. Deepjdot: Deep joint distribution optimal transport for unsupervised domain adaptation. European Conference on Computer Vision (ECCV). 2018.
[56] M. Long, H. Zhu, J. Wang, and M. I. Jordan. Deep transfer learning with joint adaptation networks. In International Conference on Machine Learning (ICML). 2017.
[57] P. Morerio, J. Cavazza, and V. Murino. Minimal-entropy correlation alignment for unsupervised deep domain adaptation. In International Conference on Learning Representations (ICLR). 2018.
[58] C. Chen, Z. Fu, Z. Chen, S. Jin, Z. Cheng, X. Jin, and X.-S. Hua. Homm: Higher-order moment matching for unsupervised domain adaptation. In National Conference of Artificial Intelligence (AAAT). 2020.
[59] S. Purushotham, W. Carvalho, T. Nilanon, and Y. Liu. Variational adversarial deep domain adaptation for health care time series analysis. In International Conference on Learning Representations (ICLR). 2017.
[60] E. Tzeng, J. Hoffman, T. Darrell, and K. Saenko. Simultancous deep transfer across domains and tasks. In International Conference on Computer Vision (ICCV). 2015.
[61] E. Tzeng, J. Hoffman, K. Saenko, and T. Darrell. Adversarial discriminative domain adaptation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2017.
[62] K. Saenko, B. Kulis, M. Fritz, and T. Darrell. Adapting visual category models to new domains. In Proceedings of the 1 11th European Conference on Computer Vision: Part IV, I0, p. 213-226. Springer-Veriae Verlag, Berlin, Heidelberg, 2010. ISBN 364215360X.
[63] H. Venkateswara, Eusebio. Chakraborty, and S. Panchanathan. Deep hashing network for unsupervised domain adaptation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2017.
[64] M. Long, H. Zhu, Wang, and M. I. Jordan. Deep transfer learning with joint adaptation networks. In Proceedings of the 34th International Conference on Machine Learning. vol. 70 of Proceedings of Machine Learning Research, pp. 2208 2217. PMLR, 06-11 1 Aug 2017.
[65] M. Long, H. Zhu, J. Wang, and M. I. Jordan. Unsupervised domain adaptation with residual transfer networks. In Advances in Neural Information Processing Systems, vol. 29, Curran Associates, Inc., 2016.
[66] Z. Pei, Z. Cao, M. Long, and J. Wang. Multi-adversaril domain adaptation. Proceedings of the AAAI Conference on Artificial Intelligence, 32, Apr 2018.
[67] X. Chen, S. Wang, M. Long, and J. Wang. Transferability vs. discriminability: Batch spectral penalization for adversarial domain adaptation. In Proceedings of the 36th International Conference on Machine Learning, vol. 97 of Proceedings of Machine Learning Research, pp- 1081-1090. PMLR, 09-15 Jun 2019.
[68] Z. Deng, Y. Luo, and J. Zhu. Cluster alignment with a teacher for unsupervised domain adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). October 2019.
[69] X. Jiang, Q. Lao, S. Matwin, and M. Havaei. Implicit class-conditioned domain alignment for unsupervised domain adaptation. In Proceedings of the 37th International Conference on Machine Learning, vol. 119 of Proceedings of Machine Learning Research, pp. 481 4826-4887. PMLR, 13-18 Jul 2020.
[70] C. Raab, P. Vath, P. Meier, and F.-M. Schleif. Bridging adversarial and statistical domain transfer via spectral adaptation networks. In Proceedings of the Asian Conference on Computer Vision (ACCV). November 2020.
[71] X.
Wang, Jin, M. Long, J. Wang, and M. Jordan. Transferable normalization: Towards im- proving transferability of deep neural networks. Advances in Neural Information Processing Systems (NeurIPS). 2019.
14

---

[72] J. Liang, D. Hu, and J. Feng. Do we really need to access the source data? Source hypothesis transfer for unsupervised domain adaptation. In International Conference on Machine Learning (ICML). 2020.
[73] Q. Wang and T. P. Breckon. Unsupervised domain adaptation via structured prediction based selective pseudo-labeling. In National Conference of Artificial Intelligence (AAAI). 2020.
[74] J. Wang. Chen, J. Lin, L. Sigal, and C. W. de Silva. Discriminative feature alignment: Im- proving transferability of unsupervised domain adaptation by Gaussian-guided latent alignment. Puttern Recognition, p. 107943, 2021.
15


Reviews:
[{"review_id": "migJuOZngyR", "reviewer": "Reviewer_iSrE", "summary": "The paper proposes warping probability discrepancy measures towards the end tasks. To achieve this, the authors leveraged the pseudo-labels from the optimal predictor with a new bi-level optimization-based approach.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Originality\n\nThe proposed framework is designed upon maximum mean discrepancy (MMD) and contrastive domain discrepancy (CDD). However, the modification is technically sound and novel.\n\n### Quality\n\nIn Table 2, ASAN is not the state-of-the-art method. [ref1] achieves 71.8% on Office-Home dataset.\n\nIn Table 3, ASAN is not state-of-the-art method. It is encouraged to include recent state-of-the-art methods, such as [ref2] and [ref3]. [ref2] achieves 90.3% and [ref3] achieves 90.2%.\n\n\n### Clarity\n\nThe paper is well written and easy to follow. The highlighted words were also helpful for understanding equations.\n\n### Significance\n\nThe experimental results of the proposed method are not significant. For example, [ref1] outperforms the proposed method by 1.0% on Office-Home and [ref2] outperforms the proposed method by 0.9%.\n\n[ref1] Do We Really Need to Access the Source Data? Source Hypothesis Transfer for Unsupervised Domain Adaptation, ICML 2020.\n\n[ref2] Unsupervised Domain Adaptation via Structured Prediction Based Selective Pseudo-Labeling, AAAI 2020.\n\n[ref3] Discriminative Feature Alignment: Improving Transferability of Unsupervised Domain Adaptation by Gaussian-guided Latent Alignment, Pattern Recognition 2021.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "UtlMZUEI1bq", "reviewer": "Reviewer_dRho", "summary": "The paper \"Implicit Task-Driven Probability Discrepancy Measure for Unsupervised Domain Adaptation\" proposes a probability discrepancy measure in the context of unsupervised domain adaptation while keeping the end goal, namely, minimizing the target domain risk, as part of the measure. The proposed measure, i-MDD, is based on a modification of the $\\mathcal{H} \\Delta \\mathcal{H}$-divergence of Ben-David et al. (2010) and Marginal Disparity Discrepancy (MDD) of Zhang et al. (2019). The paper then discusses practical considerations like how to actually minimize the i-MDD, which is noted to be simplified due to the structure of i-MDD, and then finally discusses experiments.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is a mixed bag, it starts great by clearly introducing the problem and the preliminaries, but then completely loses its momentum by muddling its presentation on the new proposed discrepancy measure, arguably the part without which I cannot recommend accepting this paper. Let me expound on this.\n\nFor someone well-versed with the jargon of domain adaptation, reading the first two sections, introduction and preliminaries, and some parts of other sections leaves the reader with a clear picture of the problem and the literature. I am not an expert in domain adaptation, and some phrases like \"margin disparity discrepancy\" (MDD) or \"contrastive domain discrepancy\" were new to me. However, I could still get a reasonably complete picture by reading the aforementioned parts of the paper or by going through some of the references cited. But this is where the good parts end, and the problems start.\n\nThe paper fails to motivate or justify its development of the proposed discrepancy measure, namely equations (11) and (12). After introducing two other relevant measures of discrepancy, $\\mathcal{H} \\Delta \\mathcal{H}$-divergence and MDD, the paper tries to reconcile these two measures by changing the MDD to make it look like $\\mathcal{H} \\Delta \\mathcal{H}$-divergence. I don't see the point of this exercise at all. Why is this \"conflict of spirit\", as the authors put it, undesirable? The whole point of minimisation over $h$ in MDD is to select for those $h \\in \\mathcal{H}$ that have low source domain risk. I understand that optimizing for MDD or $\\mathcal{H} \\Delta \\mathcal{H}$-divergence isn't easy and this is where i-MDD shines, but to me this analysis of reconciliation seems forced. Next, the lines 136-143 feel disconnected. A formulation is proposed in equation (10) which intuitively seems to strike a balance between $\\mathcal{H} \\Delta \\mathcal{H}$-divergence and MDD, but then it is changed to something very different just on the grounds that GEM doesn't work here? Finally, there's the issue of the formulation of MDD in equation (6). It doesn't seem to correspond to the one discussed in the referenced paper by Zhang et al. (2019). I would appreciate if the authors could help me see how their formulation in equation (6) relates to Definition 3.2 of Zhang et al. where they propose the MDD. Relatedly, the formulation of i-MDD seems exactly the Disparity Discrepancy (DD) discussed in Definition 3.1 of Zhang et al. except with an extra $\\alpha \\mathcal{R}(h^*;\\tilde{P}_\\phi)$ term and the fact that a specific $h$ is being used, namely $h^*$. This reinforces my point that all the reconciliation seems superfluous.\n\nThere are other issues of notation as well:\n\n1. One line 94, it is stated that $P := \\phi \\\\# S$ and $\\tilde{P} := \\phi \\\\# \\tilde{S}$. Did the authors intend $P := \\phi \\\\# S_x$ and $\\tilde{P} := \\phi \\\\# \\tilde{S}_x$ because otherwise it doesn't make sense. The feature extractor $\\phi$ maps $\\mathcal{X}$ to some latent space and $S$ is a distribution on $\\mathcal{X} \\times \\mathcal{Y}$, and so $S \\circ \\phi^{-1}$ does not make sense.\n2. What is $Q$ on line 100? Is it $\\phi \\\\# T_x$?\n3. On line 83, instead of $\\tilde{T}$ did you intend $\\tilde{T}_x$?\n\nI did not check the experimental results.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10 hours"}, "extra_text": {}}, {"review_id": "BDTKsMOxwv", "reviewer": "Reviewer_PZK7", "summary": "In this paper, the authors propose a new bi-level optimization-based approach to handle the issue of [23], i.e., the proposed margin disparity discrepancy (MDD) measure conflicts with the H\\DeltaH-divergence. The theoretical results are then applied to MDD and CDD, which gives the implicit task-driven discrepancy method, i-MDD and i-CDD. Experimental studies are done to verify the effectiveness of the proposed i-MDD and i-CDD.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Here are some comments:\n\nPros:\n(1)\tThe paper is well-written and easy to follow. \n\n(2)\tThe theoretical analyses are given to motivate the implicit task-driven disparity discrepancy.\n\n(3)\tNew implicit task-driven MDD and CDD, i.e., i-MDD and i-CDD, are given.\n\nCons:\n\n(1)\tThe paper needs a thorough related work section to elaborate the existing domain adaptation learning theory as well as the corresponding domain adaptation algorithms. Discussions on how the current work differentiates from existing works are needed. \n\n(2)\tRegarding Eq. (4), it is necessary to provide more details on how this equation relates to what equation of [16].\n\n(3)\tThe proposed two method i-CDD and i-MDD are not well compared, e.g., pros and cons. Moreover, is there any guide on which one should be used in what scenario?\n\n(4)\tThe empirical results of i-CDD and i-MDD are only comparable with the state-of-the-art methods (in each dataset, we can find some comparable baselines), which makes the methods less convincing.\n\n(5)\tFigure 3 shows that the cache size is an important factor affecting the final performance. Is there any rule to set such a hyper-parameter when facing a new UDA task? Note that different settings are used in the office 31 and office-home dataset.\n\nOverall, this is an interesting paper with strong theoretical analyses but weak empirical results. \n\nAfter rebuttal.\n\nThanks for the authors' response. After reading it, I would like to keep my score.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "An explicit limitation section is needed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "04CQF0uHw7", "reviewer": "Reviewer_Fjy7", "summary": "This paper introduces an algorithm for invariant representation learning for domain adaptation, which builds on the existing work by MDD in order to effectively calculate the $D_{\\mathcal{H}\\Delta \\mathcal{H}}$ divergence using the source domain classifier. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is clearly presented and plenty of relevant related work is cited. The paper provides a meaningful discussion on the drawbacks of MDD and addresses the problem by introducing a bi-level optimization approach. Furthermore, the paper provides training strategies to improve performance (such as taking advantage of a linear $h$, how to make use of pseudo-labels, and cache-augmented training). The empirical experiments are comprehensive and seem convincing. \n\nOne thing that would improve the paper would be a discussion on how DANN [Ganin et al., 2016] estimate the $D_{\\mathcal{H}\\Delta \\mathcal{H}}$ divergence using adversarial training, and why it is necessary to have equation (13) when the divergence was initially minimized using adversarial training. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No limitations are discussed, and any inclusion of them in the paper would be welcome. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper originally had originally some borderline negative reviews where the reviewers acknowledged a novel DA approach and  theoretical results but  noted some lack of positioning with respect to other DA methods such as DANN and some missing comparisons and justifications for the method. The authors did a very good job at answering those concerns in short and efficient replies that clarified all the major concerns from the reviewers. 

The final consensus is that the paper should be accepted. Note that the authors have to take into account the comments from the reviewed and integrate their very pertinent answers to those comments in the final version of the paper.
Decision: Accept (Poster)