A General Upper Bound for Unsupervised Domain Adaptation
Authors: Dexuan Zhang
Tatsuya Harada
Venue:  (2020)
Keywords: unsupervised domain adaptation
upper bound
joint error
hypothesis space constraint
cross margin discrepancy

Abstract: In this work, we present a novel upper bound of target error to address the problem for unsupervised domain adaptation. Recent studies reveal that a deep neural network can learn transferable features which generalize well to novel tasks. Furthermore,  Ben-David et al. (2010) provide an upper bound for target error when transferring the knowledge, which can be summarized as minimizing the source error and  distance between marginal distributions simultaneously. However, common methods based on the theory usually ignore the joint error such that samples from different classes might be mixed together when matching marginal distribution. And in such case, no matter how we minimize the marginal discrepancy, the target error is not bounded due to an increasing joint error. To address this problem, we propose a general upper bound taking joint error into account, such that the undesirable case can be properly penalized. In addition, we utilize constrained hypothesis space to further formalize a tighter bound as well as a novel cross margin discrepancy to measure the dissimilarity between hypotheses which alleviates instability during adversarial learning. Extensive empirical evidence shows that our proposal outperforms related approaches in image classification error rates on standard domain adaptation benchmarks.

Under review as a conference paper at ICLR 2020
A GENERAL UPPER BOUND FOR UNSUPERVISED DOMAIN ADAPTATION
Anonymous authors Paper under double-blind review
ABSTRACT
In this work, we present a novel upper bound of target error to address major problem faced by many unsupervised domain adaptation methods. Recent studies reveal that a deep neural network can learn transferable features which generalize well to novel tasks. Furthermore, a theory proposed by (Ben-David et al. (2010) provides an upper bound for target error when transferring the knowledge, which can be summarized as minimizing the source error and distance between marginal distributions simultancously. However, common methods based the theory usually ignore the joint error such that samples from different classes might be mixed together when matching marginal distribution. And in such case, no matter how we minimize the marginal discrepancy, the target error is not bounded due to an increasing joint error. To tackles this problem, we propose general up- per bound taking joint error into account, such that the undesirable case can be properly penalized. In addition, formalize a tighter bound as well as a novel cross margin discrepancy to measure the dissimilarity between hypotheses which alleviates instability during adversar- ial learning. Extensive empirical evidence shows that our proposal outperforms related approaches in image classification error rates on standard domain adapta- tion benchmarks.
utilize constrained hypothesis space to further
1
Introduction
The advent of deep convolutional neural networks (Krizhevsky et al. 2012) brings visual learning into a new era. However, the performance heavily relies on the abundance of data annotated with ground-truth labels. Since traditional machine learning assumes a model is trained and verified in fixed distribution (single domain), where generalization performance is guaranteed by VC theory (N. Vapnik 2000), thus it cannot always be applied to real-world problem directly. Take image classification task as example, number of factors. such the change of light, noise, angle in which the image is pictured, and different types of sensors, can lead to a domain-shift thus harm the performance when predicting on test data.
Therefore, in many practical cases, we wish that a model trained in one or more source domains also applicable another domain. As a solution, domain adaptation (DA) aims transfer the knowledge learned from a source distribution, which is typically fully labeled into a different (but related) target distribution. This work focus on the most challenging case, i.e, unsupervised domain adaptation (UDA), where no target label is available.
Ben-David 2210) suggests that target error can be minimized by bounding the error of a model on the source data, the discrepancy between distributions of the two domains, and a small optimal joint error. Owing to the strong representation power of deep neural nets, many researchers focus on learning domain-invarian features such that the discrepancy of two feature spaces can be minimized. For aligning feature distributions across domains, mainly two strategies have been substantially explored. The first one is bridging the distributions by matching all their statistics (Long et al. 2013 2017: Pan et al., 2009). The second strategy is using adversarial learning (Goodfellow et al, 2014 to build minimax game between domain discriminator and feature extractor, where domain discriminator is trained to distinguish the source from the target while the feature extractor is learned to confuse simultaneously (Ganin & Lempitsky 2015 Ganin et al. 2016 Tzeng et al. 2017).

---

Under review as a conference paper at ICLR 2020
In spite of the remarkable empirical results accomplished by feature distribution matching schemes, they still suffer from a major limitation: the joint distributions of feature spaces and categories are not well aligned across data domains. As is reported in Ganin et al. (2016), such methods fail to generalize in certain closely related source/target pairs, e.g., digit classification adaptation from MNIST to SVHN. One potential reason is when matching marginal distributions of source and target domains, samples from different classes can be mixed together, where the joint error becomes non- negligible since no hypothesis can classify source and target at the same time.
This work aims to address the above problem by incorporating joint error to formalize an optimiz- able upper bound such that the undesired overlap due to a wrong match can be properly penalized. We evaluate our proposal on several different classification tasks. In some experimental settings, our method outperforms other methods by a large margin. The contributions of this work can be summarized as follows:
We propose novel upper bound taking joint error into account and theoretically prove that our proposal can reduce to several other methods under certain simplifications.
We construct a constrained hypothesis space such that much tighter bound can be obtained during optimization.
We adopt a novel measurement, namely cross margin discrepancy, for the dissimilarity of two hypotheses on certain domain alleviate the instability during adversarial learning and provide reliable performance.
2
Related Work
The upper bound proposed by Ben-David et al. (2010) invokes numerous approaches focusing on reducing the gap between source and target domains by learning domain-invariant features, which can be achieved through statistical moment matching. Long et al. 20155) (017) use maximum mean discrepancy (MMD) to match the hidden representations of certain layers in deep neural network. Transfer Component Analysis (TCA) (Pan et al. [201T) tries to learn a subspace across domains in a Reproducing Kernel Hilbert Space (RKHS) using MMD that dramatically minimize the distance between domain distributions. Adaptive batch normalization (AdaBN) (Li et al. (0018) modulates the statistics from source to target on batch normalization layers across the network in a parameter- free way.
Another way learn fomain-invarian features is by leveraging generative adversarial network to produce target features that exactly match the source. Ganin & Lempitsky (2015) relax divergence measurement the upper bound by a worst case which equivalent to the maximum accuracy that discriminator possibly achieve when distinguishing source from target. [Teng al. (2017)
follow this idea but separate the training procedure into classification stage and adversarial learn- ing stage where an independent feature extractor is used for target. Saito et al.) (007h)) explore a tighter bound by explicitly utilizing task-specific classifiers as discriminators such that features nearby the support of source samples will be favored by extractor. Zhhnng et al. (2019) introduce margin disparity discrepancy, a novel measurement with rigorous generalization bounds, tailored to the distribution comparison with the asymmetric margin loss bridge the gap between theory and algorithm. Methods perform distribution alignment on pixel- lexel in raw input, which is known as taage-to-image translation, are also proposed (Liu & Tuzel 2016 Bousmalis al. 2017 Sankara- narayanan et al. 2017, Shrivastava et al. 2016 Hoffman et al.) 2018 Murez et al. 2017).
Distribution matching may not only bring the source and target domains closer, but also mix samples with different class labels together. Therefore, (Saito et al. (2017a): /Snner et al. (2016): Zhang et al.) 2018 aim to use paeudo-labels to learn target discriminative representations encouraging a low- density separation between classes in the target domain (Lee) 2013). However, this usually requires auxiliary data-dependent hyper-paramete to set a threshold for a reliable prediction. Long et al. (2018) present conditional adversarial domain adaptation, principled framework that conditions the adversarial adaptation models on discriminative information conveyed in the classifier predictions, where the back paok-propagttion of training objective is highly dependent pseudo-labels.
2

---

Under review as a conference paper at ICLR 2020
3 Proposed Method
3.1 A General Upper Bound
We consider the unsupervised domain adaptation as a binary classification task (our proposal holds for multi-class case) where the learning algorithm has access a of labeled points { v() (X x Y) sampled i.i.d. from the source domain S and set of m unlabeled points {xx) E
| sampled i.i.d. from the target domain T. Let Js X + {0, 1} and fT X {0. 1] be the optimal labeling functions the source and target domains, respectively. Let (usually 0-1 loss) denotes distance metric between two functions over a distribution that satisfies symmetry and triangle inequality. As commonly used notation, the source risk of hypothesis h X {0.1) is the error w.r.t. the true labeling function fs under domain S, i.e., (s(h) == ch, fs). Similarly, (r) to represent the risk of the target domain. With these notations, the following bound
holds:
(1) eT(h, fT)
= eT(h, - - (es(h, es(h, Ss) [s)) + (hr eh
css -e + - eshh
+ fT) +es(fs (r(h (sh,
=s((h) + (f.r(fs..r.h))
(1)
For simplicity, we Cs.r fr, h) denote er(ss, fr) +eg(Js, + (t) -s)--sshhhhh fr).
The above upper bound is minimized when h = fs. and it is equivalent to er (fs. fr) owing to the triangle inequality:
(s(h) + +cs(ss, = cs(h, ((( cc((s,f))
2 (s(h, ST)
Furthermore, we demonstrate such case, our proposal is equivalent to an upper bound of optimal joint error À because:
fr(fs.fr) er(ss. + fss(ss.fs) = (S(ss +es(ss) min(er(h)) + es(h))
Fig. 1b illustrates case where common methods fail to penalize the undesirable situation when samples from different classes are mixed together during distribution matching, while our proposal is capable to do (for simplicity assume fs takes specific form, then er (fs, fr) measures the overlapping area and 5, which is equivalent to the optimal joint error A).
Juid
Pugee Alc
fs
Clews
fs
Source
fr
f
Fr
Tagett
(a) legend
(b) joint error
(c) location of fr
Figure 1: (a) Legend used in entire paper. (b) Joint error (area 2 and 5) is penalized such that extractor must try to separate the overlap. (c) fi does not necessarily classify source samples.
3

---

Under review as a conference paper at ICLR 2020
3.2 Hypothesis Space Constraint
Since optimal labeling functions fs, fr are not available during training, we shall further relax the upper bound by taking supreme w.r.t fs, fo within hypothesis space H:
(T - Eh(h) + (r:
(h) + sup f2,
(4)
fu.1EH
Then minimizing target risk (r)h) becomes optimizing a minimax game and since the max max-playee taking two parameters f1, f2 is too strong, we introduce a feature extractor g to make the min-player stronger. Applying to the source and target distributions, the overall optimization problem can be written as:
min( a.h
-
max JibfEH
(f.,
(5)
However, if we leave H unconstrained, the supreme term can be arbitrary large. In order to obtain a tight bound, we need to restrict the size of hypothesis space as well as maintain the upper bound. For fs E H1 S H and fr E H2 I H, the following holds:
Co h)
sup J2HH
Co(s).otz f2,11 S sup (f f2,1) (6)
fIJEEH
The constrained subspace for H1 trivial as according to its definition, fs must belong to the space consisting of all classifiers for source domain, namely H.c- However, the constrained subspace for H2 a little problematic since we have no access to the true labels the target domain, thus is hard locate fr. Therefore, the only thing we can do is to construct hypothesis space for H2 that most likely contains fr. As is illustrated in Fig. [c] when matching distributions of source and target domain, if the ideal case achieved where the conditional distributions of source and target are perfectly aligned, then it is fare to assume fr E H.c- However, if the worst case is reached where samples from different class are mixed together, then we tend to believe fr & H Considering this, we present two proposals in the following sections based on different constraints.
3.2.1 Original Proposal
We assume H2 is space where the hypothesis can classify the samples from the source domain with an accuracy of y [o, 1]. namely HJ., such that we can avoid the worst case by choosing small value for the hyper-parameter when huge domain shift exists. In practice, it is difficult to actually build such space and sample from due to a huge computational cost. Instead, we use weighted source risk to constrain the behavior of f2 approximation to the sample from Hy.. which leads to the final training objective:
(ming.4(( (h) + max + (f1, (h,) f1) Co(s) f2))) s.t. ming.h..z ((( +((()
3.2.2 Alternative Proposal
Firstly, we build space consisting of all classifiers for approximate target domain {( h(x))) e
X a based on pseudo labels which can be obtained by the prediction of h during training procedure, namely H Here, we assume H, is an intersection between two hypothesis spaces, i... Hy. n H.! !!!" where the hypothesis can classify the samples from source domain with accuracy of n E j0, 1] and classify the samples from approximate target domain with accuracy of 1 - n- Given enough reliable pscudo labels, we can be confident about fr E H2. Analogously, the training objective is given by:
4

---

The reason we make such assumption for H2 can be intuitively explained by Fig. If H2 H then fe must perfectly classify the source samples, and it is possible that f2 does not pass through some target samples (shadow are in [a). especially when two domains differ lot. In such case, the feature extractor can move those samples into either side the decision boundary to reduce the Under review as a conference paper at ICLR 2020
mming,blCtte (h) + max,, f2) + s.t. ming.fi (allafnnf((((s((iii)++++ (oa(s) ((r)((()))
coss)
(o(T)(h, fi) Ca(s) f2)))
(8)
3.2.3 Intuition
training objective (shadow area) which is not desired behavior. With appropriate constraint (2b). as for the extractor, the only way to reduce the objective (shadow area) is to move those samples (orange) inside of f2.
f
f
6
6)
<r(6.6)
c)(6.6)
(a) H₂ H.c
(b) H₂ H M
Figure (a) Failure due to improper constraint, (b) Proper constraint helps for huge domain shift.
3.3 Cross Margin Discrepancy
Following the above notations, consider score function s(z, y) for multi-class classification where the output indicates the confidence of the prediction on class y. Thus an induced labeling function named from X - Y is given by:
1. x - arg max s(x,,) v)
As well-established theory, the margin between data points and the classification surface plays a significant role in achieving strong generalization performance. order to quantify into dif- ferentiable measurement surrogate of (-- loss, introduce the margin theory developed by Koltchinskii & Panchenko (002)., where a typical form of margin loss can be interpreted as:
((z.g)e (0ax(0, max s 3/) s(z, v))]
(10)
y'²y
We aim to utilize this concept to further improve the reliability of our proposed method by leveraging this margin loss to define a novel measurement the discrepancy between two hypotheses fi. fa (e.g. softmax) over distribution D. namely cross margin discrepancy:
(n(i, f2) = Even S2,
(11)
Before further discussion, we firstly construct two distributions D Dy, induced by fi, J2 respec- tively, where D (x,,, (x)){æ D} and D = {xx 4, (z)) D). Then we consider the case where two hypotheses fi and fa disagree, i.c. L (x) # [z(x)) = and the primitive loss is defined as:
d( 2) = log fi (z. y1) log f2 91) + log fz(x. y2) log fi (x. 92) llog fi (x, yn) log (xr v2) + log fz y2) log f2 v1)
(12)

---

Under review as a conference paper at ICLR 2020
Then the cross margin discrepancy can be viewed as: cD(fi. f2) = D,, [max log fi (z. y/) -log fi (2, (a.w)), Imaxloge (2 y²) -oog fa v) y'#y
y'
(13)
which is a sum of the margin loss for fi on D2, and the margin loss for f2 on D we use the logarithm of softmax the score function.
Thanks to the trick introduced by Goodfellow et al. (2014) to mitigate the burden exploding or vanishing gradients when performing adversarial learning, further define dual form
d(I1: (2, = log /1 (x, v1) + log(1 f1 (x, 12)) + log f2(2, (2) + log(1 [ - f2(2,,1))
(14)
This dual loss resembles the objective of the generative adversarial network, where two hypothe- ses try to increase the probability of their own prediction and simultancously decrease the prob- ability of their opponents; whereas the feature extractor is trained to increase the probability of their opponents, such that the discrepancy can be minimized without unnecessary oscillation. How- ever, a big difference here when training extractor, GANs usually maximize an alternative term log fi (ar, y2) + log (2 y1) instead of directly minimizing log(1 fi (z, y/2) + log(1 f2 wi) since the original term is close to zero if the discriminator achieves optimum. hypothesis can hardly beat the extractor thus the original form can be more smoothly optimized.
our case, the
During the training procedure, the two hypotheses will eventually agree on some points (l (œ) = l, (z) y) such that need define new form of discrepancy measurement, Analogously, the primitive loss and its dual form are given by:
f2, x) log max (f (3r, y), (2 v)) log fan
(15)
d2,1) = log max (x, f), [2 v)) + log f1 v), f2(2, w))
(16)
Another reason why we propose such a discrepancy measurement that it helps alleviate instability for adversarial learning. As is illustrated in Fig. 36 during optimization of a minimax game, when two hypotheses try maximize the discrepancy (shadow area), one moves too fast around the decision boundary such that the discrepancy is actually maximized w.r.f some samples, then these samples can be aligned on either side to decrease the discrepancy by tuning the feature extractor, which is not desired behavior. From Fig. 3a we
that our proposed cross margin discrepancy flat for the points around original, i.c. the gradient w.r.t those points nearby the decision boundary will be relatively small, which helps to prevent such failure.
3.5
h
3.0
Logistic
2.5
2.0
Hinge
cs(Si. 22)
f
f
FINETEE opaiion
0.5
0.0
-0.5
0.0 0.5 1.0 1.5 2.0
(7(1.6)
(a) loss curve
(b) instability
Figure 3: (a) Comparisons for binary classification case. (b) Failure due to steep gradient nearby the decision boundary.
3.4
Comparisons with Other Methods
3.4.1 Margin Disparity Discrepancy
Zhang al. (019 propose a novel margin- generalization bound based on scoring functions and new divergence MDD. The training objective used in MDD can be alternativeyy interpreted as (here ehh, f) denotes the margin disparity):
min(C,(s)()) + 1 mmxx( (f, - cst) (f.t)) (f))))
(17)
g.

---

Under review as a conference paper at ICLR 2020
Recall 7 if we set fi f2 f and free the constraint of f to any E H, our proposal degrades exactly to MDD. As is discussed above, when matching distribution, if and only the ideal case is achieved, where the conditional distributions of induced feature spaces for source and target perfectly match (which is not always possible), can we assume two optimal labeling functions fs Jr to be identical. Besides, unconstrained hypothesis space for f is definitely not helpful to construct tight bound.
3.4.2 Maximum Classifier Discrepancy
Saito et al. (2017b) propose two task-specific classifiers fi, f2 that are used to separate the decision boundary source domain, such that the extractor is encouraged to produce features nearby the support of the source samples. The objective used in MCD can be alternatively interpreted as (here c(fi, f2) is quantified by Ln):
ming.h (f)(s)()) + ma s.t. ming.f (fi( + (f))
(18)
Again, recall Eq 7 if we set and h = fI, MCD is equivalent to our proposal. As is proved in section [.1] the upper bound optimized when fs. However, it no longer holds since the upper bound is relaxed by taking supreme to form an optimizabel objective, i.e. setting h = f1 does not necessarily minimize the objective. Besides, as we discuss above, fixed y 1, i. H2 H,e lacks generality since we have no idea about where fr might be, such that it is not likely to be applicable to those cases where huge domain shift exists.
4 Evaluation
4.1
Experiment Digit Dataset
In this experiment, our proposal is assessed in four types of adaptation scenarios by adopting com- monly used digits datasets (Fig. in ippendix),.ee MNIST (LeCun et al. 1998), Street View House Numbers (SVHN) (Netzer al. 2011). and USPS (Hull) 1994) such that the result could be easily compared with other popular methods. All experiments are performed in an unsupervised fashion without any kinds of data augmentation.
4.1.1 Experimental Setting
Details are omitted due to the limit of space (see A.1
4.1.2
Result
We report the accuracy of different methods in Tab. I Our proposal outperforms the competitors in almost all settings except a single result compared with GPDA (Kim al. 2019). However, their solution requires sampling that increases data size and is equivalent to adding Gaussian noise to the last layer of a classifier, which considered as a type of augmentation. Our success partially owes to combining the upper bound with the joint error, especially when optimal label functions differ from each other ge.g. MNIST SVHNN. Moreover, as most scenarios are relatively easy for adaptation thus can be more confident about the hypothesis space constraint owing to reliable pseudo-labela which leads to a tighter bond during optimization. The results demonstrate our proposal can improve generalization performance by adopting both of these advantages.
Fig. 4a shows that our original proposal quite sensitive to the hyper- parameter 7. In short, setting here yields the best performance in most situations, since fs. fr be quite close after aligning distributions, especially these easily adapted scenarios. However, in MNIST + SVHN, setting 7 0.1 gives the optimum which means that fs, fr far away due to huge domain

---

Under review as a conference paper at ICLR 2020
Table Results of the adaptation experiment on the digits datasets (note that means different set- ting which use labeled target samples for validation; MNIST" and USPS* denote the whole training set; ours: original proposal and L1 (y 1); ours": original proposal and cross margin discrepancy
(7 1); ours": alternative proposal and cross margin discrepancy (1) = 0)).
SVHN to MNIST
MNIST to SVHN
MNIST
MNIST to USPS" 79.7
USPS to MNIST
METHOD
to USPS
Source Only
67.1
21.3
76.7
63.4
MDD
Long ct al.
2015
71.1
81.1
DANNI
Gamin ct al
2016
71.1
25.1
77.3
85.1
73.2
DRCN
Gintary el al 2016
82.0+0.11
40.1+0..1
91.8++0.1
0.
ADDA
Izeng of al. 2017
76.0=11.8
89.4+00.2
90.1+-0.8
MCD
Sallo
96.2+0.4
11.24111
94.2++0.7
96.5++0.3
94.1+00.3
GPDA
2019
98.210011
96.5+10.2
98.11:0.1
96.410.11
96.810.2
30.4115
94.5.10.
96.810.3
95.2.10.2
ours" ours'
97,510.2
31,5118
95.3.10.3
97.2.10.2
95.6.10.2
98.6:10.1
50.34113
96.8.10.2
97.910.1
96.9100.1
shift that extractor is capable of introducing identical conditional distribution in feature space. The improvement is not that much, but at least we outperform the directly comparable MCD and show the importance of hypothesis space constraint. Furthermore, Fig. [d]emiiiaallyy proves simply minimizing the discrepancy between the marginal distribution does not necessarily lead to reliable adaptation, which demonstrates the importance of joint error. In addition, Fig. 4b Fig. 4c show the superiority of the cross margin discrepancy which accelerates the convergence and provides slightly better result.
0.9
E0:4 0.32
0.8
-.7
g07
5-
MCD
MCD
0.3
Durs
Ours
0.2
065
000
0.0g
0.00
epach
(a) domain shift affects y
(b) S-
(c) M-U
(d) M-+ S
Figure 4: (a) Sensitivity w.r.t y. (b)-(c) Comparisons for convergence rate. (d) Comparisons for marginal discrepancy.
4.2 Experiment on VisDA Dataset
We further evaluate our method on object classification. The VisDA dataset (Peng et al. 2017 is used here, which is designed for 12-class adaptation task from synthetic object real object images. Source domain contains 152,397 synthetic images (Fig. 7a in Appendix), which are generated by rendering 3D CAD models. Data of the target domain is collected from MSCOCO (Lin et al.) 2014) consisting of 55,388 real images (Fig. 7 in Appendix). Since the 3D models are generated without the background and color diversity, the synthetic domain is quite different from the real domain, which makes it much more difficult problem than digits adaptation. Again, this experiment is per- formed in unsupervised fashion and no data augmentation technique excluding horizontal flipping is allowed.
4.2.1 Experimental Setting
Details omitted due the limit of space (see A.2
8

---

Under review as a conference paper at ICLR 2020
Table 2: The accuracy of ResNet-101 model fine-tuned on the VisDA dataset within 10 epoch up-
dates (ours: original proposal and L1 (y 1); ours": original proposal and cross margin discrepancy
( = 1); ours": alternative proposal and cross margin discrepancy (r 0.9)).
METHOD
plane
beyel
bus
car
horse
kniíe
meyel
person
plant
skibed
trann
truck
ave
Source Only
53.3
59.1
31.2
73.5
52.4
Long al. 2015
87.1
63.0
76.5
42.0
90.3
42.9
85.9
53.1
49.7
36.3
85.8
20.7
61.1
DANN Gamin ef nl 2016
81.9
77.7
82.8
44.3
81.2
29.5
65.1
28.6
51.9
54.6
82.8
7.8
57.4
MCD
87.0
60.9
83.7
64.0
79.6
84.7
76.9
83.0
25.8
71.9
GPDAT (Kim a.. 2019
83.0
74.3
80.4
66.0
87.6
75.3
83.8
73.1
90.1
$7.3
80.2
31.9
73.3
OUTS
86.3
82.7
83.7
68.7
72.7
61.5
73.4
ours
88.4
83.3
74.8
78.0
88.1
43.2
88.2
68.9
87.6
65.5
92.6
58.5
76.4
OurS
91.5
80.3
75.5
66.1
91.4
87.6
85.2
78.7
91.2
77.2
82.8
48.9
79.7
4.2.2 Result
We report the accuracy of different methods in Tab. 2 and find that proposal outperforms the competitors in all settings. The image structure of this dataset is more complex than that of digits, yet method provides reliable performance key observation is that some competing methods (e.g.. DANN, MCD), which can be categorized distribution matching based on adversarial learning, perform worse than MDD which simply matches statistics, in classes such as plane and horse, while our methods perform better across all classes, which clearly demonstrates the importance taking the joint error into account.
under such challenging condition. Another
As for the original proposal (Fig. (c), performance drops when relaxing the constraint which actually confuses us. Because we expect an improvement here since is unbelievable that fs, fr eventually lie in a similar space judging from the relatively low prediction accuracy. As for the alternative proposal (Fig. [d)], we test the adaptation performance for different and the prediction accuracy drastically drops when goes beyond 0.2. One possible cause is that f2 and h might almost agree on target domain, such that the prediction of h could not provide more accurate information for the target domain without introducing noisy pseudo labels. Fig. Sal Fig. 5b] again demonstrate the superiority cross margin discrepancy and the importance of joint error.
0.70
0.80
: 2.72
0.74
20.77
8000
9.0
MCD
[ 20.74
0.76
0.78
Eonc
0.78
3.04
0.6
0.43
0.6
OMD
0.00
0.01
0.68
0.6
VIDDA
872
VIDA
10
0.0
10
03
33
04
08
18
0.70
6.
04
epach
(a) faster convergence
(b) higher discrepancy
(c) optimum
(d) optimum 0.1
Figure 5: (a) Comparisons for convergence rate. Sensitivity w.r.t 7. (d) Sensitivity w.r.t 17.
(b) Comparisons marginal discrepancy. (c)
5 Conclusion
In this work, we propose general upper bound that takes the joint error into account. Then we further pursuit a tighter bound with reasonable constraint on the hypothesis space. Additionally, we adopt a novel cross domain discrepancy for dissimilarity measurement which alleviates the in- stability during adversarial learning. Extensive empirical evidence shows that learning an invariant representation is not enough to guarantee good generalization in the target domain, as the joint error matters especially when the domain shift is huge. We believe our results take an important step towards understanding unsupervised domain adaptation, and also stimulate future work on the design of stronger adaptation algorithms that manage to align conditional distributions without using pseudo-labels from the target domain.

---

Under review as a conference paper at ICLR 2020
Acknowledgments
This work was partially supported by JST CREST Grant Number JPMJCRI400 and partially sup- ported by JSPS KAKENHI Grant Number JP19H01115. We would like to thank Yusuke Mukuta, Toshihiko Matsuura, Akihiro Nakamura, Wataru Kawai and Hao-Wa Yeh for their thoughtful com- ments and suggestions.
References
Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Vaughan. A theory of learning from different domains. Machine Learning. 79:151-175, 2010.
Konstantinos Bousmalis, Nathan Silberman, David Dohan, Dumitru Erhan, and Dilip Krishnan. Unsupervised pixel-level domain adaptation with generative adversarial networks, In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15-104, 2017.
Jun Deng, Wei Dong, Richard Socher, Li-Jia Li, Kuntai Li, and Li Fei-Fei. Imagenet: large-scale hierarchical image database. IEEE Conference on Computer Vision and Pattern Recognition, pp. 248 255, 2009.
Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In Proceedings of the 32nd International Conference Machine Learning, volume 37, pp. 1180- 1189. JMLR.org. 2015.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural net- works. Journal of Machine Learning Research, 17(1):2096 20000-2000,, 2016.
Muhammad Ghifary, W. Bastiaan Kleijn, Mengjie Zhang, David Balduzzi, and Wen Li. Deep eeconstrucioon-caasifccaaai networks for unsupervised domain adaptation. In European Con- ference on Computer Vision, volume 9908, pp. 597 613. Springer, 2016.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Infor- mation Processing Systems, pp. 2672-2680. Curran Associates, Inc., 2014.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. IEEE Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2015.
Judy Hoffman, Eric Tzeng, Taesung Park, Jun-Yan Zhu, Phillip Isola, Kate Saenko, Alexei Efros, and Trevor Darrell. CyCADA: Cycle-consistent adversarial domain adaptation. In Proceedings of the 35th International Conference on Machine Learning, volume 80, 1989-1998. PMLR, 2018
Jonathan Hull. Database for handwritten text recognition research. IEEE Transactions on Patlern Analysis and Machine Intelligence, 16:550 554, 1994.
Minyoung Kim, Pritish Sahu, Behnam Gholami, and Vladimir Pavlovic. Unsupervised visual do- main adaptation: A deep max-margin gaussian process approach. In IEEE Conference on Com- puter Vision and Pattern Recognition, 2019.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CORR, abs/1412,6990, 2014.
V. Koltchinskii and D. Panchenko. Empirical margin distributions and bounding the generalization error of combined classifiers. Ann. Statist., 30( 1050, 2002.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. In Advances in Neural Information Processing Systems, pp. 1097-1105. Curran Associates, Inc., 2012.
10

---

Under review as a conference paper at ICLR 2020
Yann LeCun, Lon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient- based learning applied to document recognition. In Proceedings of the IEEE, volume 86, pp. 2278-2324, 1998.
Dong-Hyun Lee. Pseudo-label: The simple and efficient semi-supervised learning method for deep
neural networks. ICML 2013 Workshop on Challenges in Representation Learning, 2013.
Yanghao Li, Naiyan Wang, Jianping Shi, Xiaodi Hou, and Jiaying Liu. Adaptive batch normalization for practical domain adaptation. Pattern Recognition, 80: 109-117, 2018.
Tsung-Yi Yi Lin, Michael Maire, Serge J. Belongie, Lubomir Bourdev, Ross B. Girshick, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. In European Conference on Computer Vision, 2014.
Ming-Yu Yu Liu and Oncel Tuzel. Coupled generative adversarial networks. In Advances in Neural Information Processing Systems, pp. 469-477. Curran Associates, Inc., 2016.
Mingsheng Long. Jianmin Wang, Guiguang Ding, Jia-Guang Sun, and Philip S. Yu. Transfer feature learning with joint distribution adaptation. IEEE International Conference on Computer Vision, pp. 2200-2207, 2013.
Mingsheng Long, Yue Cao, Jianmin Wang, and Michael L. Jordan. Learning transferable features with deep adaptation networks. In Proceedings of the 32nd International Conference on Interna- fional Conference on Machine Learning, volume 37, pp. 97-105. JMLR.org. 2015.
Mingsheng Long. Han Zhu, Jianmin Wang, and Michael I. Jordan. Deep transfer learning with joint adaptation networks. In Proceedings of the 34th International Conference on Machine Learning, volume 70, pp. 2208-2217. JMI..orrg 2017.
Mingsheng Long, ZHANGJIE CAO, Jianmin Wang, and Michael I Jordan. Conditional adversar- ial domain adaptation. In Advances in Neural Information Processing Systems, pp. 1640-1650. Curran Associates, Inc., 2018.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. ArXiv, abs/1802.059577 2018.
Zak Murez, Soheil Kolouri, David J. Kriegman, Ravi Ramamoorthi, and Kyungnam Kim. Image to image translation for domain adaptation. IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 455004509, 2017.
Vladimir N. Vapnik. The Nature of Statistical Learning Theory, volume 8, pp. 1-15. Springer- Verlag New York, Inc., 2000.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2011.
S J Pan, WW Tsang, J T Kwok, and Q Yang. Domain adaptation via transfer component analysis. IEEE Transactions on Neural Networks, (2(2):199-2100 2011.
Sinno Jialin Pan, Ivor Wai-Hung Tsang, James Kwok, and Qiang Yang. Domain adaptation via transfer component analysis. IEEE Transactions Neural Networks, 22:199-210, 2009.
Xingchao Peng, Ben Usman, Neela Kaushik, Judy Hoffman, Dequan Wang, and Kate Saenko. Visda: The visual domain adaptation challenge. ArXiv, abs/1710.06924, 2017.
Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell. Adapting visual category models to new domains. In European Conference Computer Vision, pp. 216-226. Springer- Verlag, 2010.
Kuniaki Saito. Yoshitaka Ushiku, and Tatsuya Harada. Asymmetric tri tri-training for unsupervised domain adaptation. Proceedings of the 34th International Conference on Machine Learning, volume 70, pp.
PMLR, 2017a.
Kuniaki Saito, Kohei Watanabe, Yoshitaka Ushiku, and Tatsuya Harada. Maximum classifier dis- crepancy for unsupervised domain adaptation. JEEE/CVF Conference on Computer Vision and Patlern Recognition, pp. 3723-3732, 2017b.
11

---

Under review as a conference paper at ICLR 2020
Swami Sankaranarayanan, Yogesh Balaji, Carlos D. Castillo, and Rama Chellappa. Generate to adapt: Aligning domains using generative adversarial networks. CoRR, abs/1704.01705. 2017.
Ozan Sener, Hyun Oh Song, Ashutosh Saxena, and Silvio Savarese. Learning transferrable repre- sentations for unsupervised domain adaptation. In Advances in Neural Information Processing Systems, pp. 21 10-2118. Curran Associates, Inc., 2016.
Ashish Shrivastava, Tomas Pfister, Oncel Tuzel, Josh Susskind, Wenda Wang, and Russell Webb. Learning from simulated and unsupervised images through adversarial training. IEEE Conference on Computer Vision and Pattern Recognition, pp. 2242-2251, 2016.
Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. IEEE Conference on Computer Vision and Pattern Recognition, pp. 2962-2971, 2017.
Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. IEEE Conference on Computer Vision and Pallern Recognition, pp. 5385-5394, 2017.
Weichen Zhang, Wanli Ouyang, Wen Li, and Dong Xu. Collaborative and adversarial network for unsupervised domain adaptation. IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 3801-3809, 2018.
Yuchen Zhang, Tianle Liu, Mingsheng Long, and Michael Jordan. Bridging theory and algorithm for domain adaptation. In Proceedings of the 36th International Conference on Machine Learning, volume 97, pp. 7404-7411 PMLR, 2019.
A Appendix
A.1 Experimental Setting for Digits Dataset
Throughout the experiment, we employ the CNN architecture used in (Sato et al. (2017h), where batch normalization is applied to each layer and a 0.5 rate of dropout is used between fully connected layers. Besides, spectral normalization (Miyato et al. 2018) deployed for the classifiers in all of the following experiments to stabilize adversarial learning. Adam (Kingma & Ba 2014) is used for optimization with a mini-batch size of 128 and a learning rate of 10 10- As for the original proposal, we verify the sensitivity of our model on various value for y = 10.1, 0.5, 0.9, 1.0). As for the alternative proposal, setting 7 = usually provides a reliable performance and the adaptation result changes subtly for different 77.
SVHN ++ MNIST We firstly examine the adaptation from SVHN (Fig. 6ah to MNIST (Fig. 6b). We use the standard training set and testing set for both source and target domains. The feature extractor contains three convolutional layers with stride two x 3 max pooling placed after the first two convolutional layers and single fully-connected layer. For classifiers, use 2-layer fully-connected networks.
MNIST ++ USPS As for the adaptation between MNIST and USPS (Fig, 6c follow the training protocol established in Long et al. (2013) by sampling 2000 images from MNIST and 1800 from USPS. As for the test samples, the standard version is used for both source and target domains. The feature generator contains two 5 x 5 convolutional layers with stride two 2 x 2 max pooling placed after each convolutional layer and single fully-connected layer. For classifiers, we use 2-layer fully-connected networks.
A.2 Experimental Setting for VisDA Dataset
Following the protocol in Saito et al. 2017b). we evaluate our method by fine-tuning ResNet-101 (He et al. 2015) model pretrained ImageNet (Deng al., 2009). The model except the last layer combined with a lingle-layer bottleneck is used as feature extractor and a randomly initialized 2- layer fully-connected network is used classifier, where batch normalization is applied to each
12

---

Under review as a conference paper at ICLR 2020
992
7
(a) SVHN
(b) MNIST
(c) USPS
Figure 6: Random samples from each dataset.
layer and a 0.5 rate of dropout is conducted. Nesterov accelerated gradient is used for optimization with a mini-batch size of 32 and an initial learning rate of 10 -0 which decays exponentially, As for the hyper-parameter, comparison, we report the accuracy after 10 epochs.
test for y {0.1, 0.5, 0.9, 1} and "i 0.5, 0.8, 0.9). For direct
(a) synthetic images
(b) real images
Figure 7: (a) Samples from source domain. (b) Samples from target domain.
A.3 Additional Experiment on Office-Home Dataset
Office-Home (Venkateswara from four significantly different domains: Art (paintings, sketches and/or artistic depictions). Clipart (clip art images), Product (images without background), and Real-world (regular images captured with a camera). In this experiment, following the protocol from Zhang et al. (2019), we evaluate our method by fine-tuning ResNet-50 (He et al. 2015) model pretrained on ImageNet (Deng al.) 2009). The model except the last layer combined with single-layer bottleneck is used as feature extractor and a randomly initialized 2-layer funne-coneected network with width 1024 is used classifier, where batch normalization is applied to each layer and a 0.5 rate of dropout is conducted. For optimization, we use SGD with the Nesterov momentum term fixed to 0.9, where the batch size is 32 and learning rate is adjusted according to Ganin et al. (2016).
2017) is complex dataset (Fig- 8 containing 15,500 images
From Tab. 3 we can see the adaptation accuracy of the source-only method is rather low, which means huge domain shift is quite likely exist. In such case, simply minimizing the discrepancy between source and target might not work as the joint error can be increased when aligning distribu- tions, thus the assumption of the basic theory (Ben- David et al., (010)) does not hold anymore. On the other hand, our proposal incorporates the joint error into the target error upper bound which can boost the performance especially when there is large domain shift.
Roal World Product Clipurt
Spos
Alaxs
Desk
Figure 8: Sample images from the Office-Home dataset /Venkateswara al. 2017).
13

---

Under review as a conference paper at ICLR 2020
Table The accuracy ResNet-50 model fine-tuned on the Office- dataset (ours": alterna- tive proposal and cross margin discrepancy (1) = 0.9)).
METHOO
Ar-xil
Ar-4F
Ar-KK
CI-AAr
C--h -h
CT
Pr-sAr
Pr C1
Pr Kw
Rw
Kw K1
Rw-FFr
Ang
31.9
38.5
60.4
59.5
46.1
76.3
CDANLEe
507
77.3
56.7
MDD 7
2019
54.9
73.7
77.8
718
61.
53.6
72.5
82.3
68.1
563
758
79.3
654
70.7
708
633
35.7
754
733
39.3
821
69.4
Table 4: The accuracy of ResNet-50 model fine-tuned on the Office-31 dataset (ours": alternative proposal and cross margin discrepancy (7) = 0.9)).
METHOD
A-W
D-*W
WDD
A-D
D-+A
W-+A
Avg
Source Only
68.4+0.2
96.7+0.1
99.3+0.1
68.9+0.2
62.5+003
60.740.3
76.1
DANNiGanin et al.
2016
82.0+0.4
96.9=0.2
99.1+0.1
79.7++0.4
68.2+00.4
67.4+0.5
82.2
ADDA Tzeng el al
2017
86.2.1..5
96.2-1..3
98.410.3
77.810.3
69.5.10.4
68.910.55
82.9
MCD
Saito et al.
2017b
88.6+0.2
98.5=0.1
100.0
92.2+0.2
69.5=0.1
69.7#0..3
86.5
CDAN Long
2018
94.1#0.1
98.6=0.1
100.0
92.9:1.2
71.020.3
69.340.3
87.7
MDD
Zhang
2019
94.5+00.
98.4-0.11
100.0+.0
93.5++0.2
74.6+0..
72.2+0.1
88.9
ours'
91.4+0.5
99.2+0.1
100.0
94.6+F0.5
76,2+0.2
78.0+F.2
89.9
A.4 Additional Experiment on Office-31 Dataset
Office-31 (Saenko et al. 2010) (Fig. 99 is a popular dataset to verify the effectiveness of a domain adaptation algorithm, which contains three diverse domains, Amazon from Amazon website, Web- cam by web camera and DSLR by digital SLR camera with 4,652 images in 31 unbalanced classes. In this experiment, following the protocol from Zhang et al. (2019). we evaluate our method by fine-tuning ResNet-50 (He et al. 2015) model pretrained on ImageNet (Deng al. 22099). The model used here is almost identical to the one in Office-Home experiment except different width 2048 for classifiers. For optimization, 0.9, where the batch size is 32 and learning rate is adjusted according to Ganin et al. (2016).
the SGD with the Nesterov momentum term fixed to
The results Office-31 reported in Tab. 4 As for the tasks D~ and W->,, judging from the adaptation accuracy of those previous methods that do not consider the joint error, it is quite likely that samples from different classes are mixed together when matching distributions, Our method shows an advantage in such case which demonstrates that the proposal manage penalize the undesired matching between source and target. As for the tasks A-W and A-D, our proposal shows relatively high variance and poor performance especially in W. One possible reason is that our method depends on building reliable classifiers for source domain satisfy the constraint. However, the Amazon dataset contains lot of noise (Fig. (O) such that the decision boundary of the source classifiers varies drastically in each itcration during training procedure, which can definitely harm the convergence.
AMAZON
DSLR
WEBCAM
monitor
bottle
pen
speaker
phone
scissors
mug
mouse
ruler
projector printer
Figure 9: Sample images from the Office-31 dataset.
14

---

Under review as a conference paper at ICLR 2020
Scolch
Panasonic.
ruler
trash_com tape dispenser pen
bottle
scissors
hin__binder
phone
punchers
computer
Figure 10: Noisy images from the Office-31 dataset.
15


Reviews:
[{"review_id": "S1xBSNsZKH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces a new upper bound of unsupervised domain adaptation, which takes the adaptability term lambda into consideration. The new theory can be expanded into a novel algorithm. Experiments on domain adaptation datasets demonstrate improvement over previous state-of-the-art methods.\nThe authors propose to incorporate lambda into adversarial feature learning. Specifically, the authors assume that f_s and f_t are from some hypothesis space H. Then relaxing f_s and f_t to f_1 and f_2, we can turn the problem into a minimax game between f_1, f_2 and feature extractor g. To further implement their method, the authors propose to constrain f_1 and f_2 with source accuracy and target pseudo label accuracy. Based on the margin theory, the authors also introduce the cross margin discrepancy, which increase the reliability of adversarial adaptation.\nThe paper is well-written and the contributions are stated clearly. The attempt to incorporate lambda into feature learning is really interesting.\n\nHowever, I have several concerns:\n*The proposed theory of equation (4), (5), and (6) is problematic. h is the hypothesis which belongs to a hypothesis class H. f_s and f_t are true labeling functions, and do not necessarily belong to the hypothesis space H. In this sense, the inequality of equation (4) does not hold. Problems of equation (5) and (6) are similar. The authors do realize that the supremum term can be arbitrarily large and put constraints to f_1 and f_2. But no matter what hypothesis class we are using, it generally does not contain the true labeling functions, and what we can do is only approximating them. Thus, in spite of the good performance of the proposed method, the proposed upper bound is not reliable.\n*Lack of experimental results on the role of f_1 and f_2. The proposed method demonstrates good performance, but the manuscript does not provide some experimental results on the source of performance gain. In particular, how is f_1 and f_2 changed during training? Are they substantially different from the h’in MCD and MDD? Besides, how does each part contribute to the performance gain? Is it from the novel loss function or just the new adversarial adaptation method itself? A proper ablation study would be helpful. \n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "HyeTcvDaKr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose an approach based on an upper bound on target domain error for the task\nof unsupervised domain adaptation (where one does not have access to\nany labels in the target domain). The upper bound makes possible the\npenalization of mixing samples from different classes together during\ndistribution matching.  Empirical results on image classification in a\nfew benchmarks show the significant promise of the approach.\n\n\nThe paper makes a good attempt at explaining the ideas/concepts, but\nstill remains very  unpolished (including English usage issues), and it is\nhard to follow in a number of places. A few such, a sampling, are mentioned below with\nsome suggestions. In my opinion, \nnot ready for publication.\n\n* abstract: '.. to address the problem for unsupervised domain\n  adaptation.' >> 'to address a major problem facing many unsupervised\n  domain adaptation techniques'.\n\n* page 2, several rewordings in: \"The reason is obvious, as marginal\n  distributions being matched for source and target, it is possible\n  that samples from different classes are aligned together, where the\n  joint error becomes non-negligible since no hypothesis can classify\n  source and target at the same time.\" For instance, a partial\n  rewording: '.. when an attempt is made to match marginal\n  distributions of source and target domains, samples from different\n  classes can be mixed together'. Also, I wouldn't use \"The reason is\n  obvious\"...\n\nFinally, here it is a good place to refer to Figure 1.\n\n\n\n* page 2: 'our proposal can degrade to some other methods' >> '.. can reduce\n  to several other methods ..'\n\n\n\n* in lines 1 and 2 on pg 3, the simplification from line 1 to 2,\n  explain the major reasoning (perhaps in the appendix if you don't\n  have space): there are number of terms added and subtracted (which\n  is understandable), but hard to keep track of what simplifications\n  are being carried and where the terms move (too many terms)...\n\n* 'the following theorem holds' >> 'the following bound holds' (or\n  inequality, etc.)\n\n* Is derivation 3 from triangle inequality?  (add that explanation to\n  line 3)\n\n* hard to parse (missing pronoun): 'the above upper bound in minimized when h=f_s thus\n  equivalent to ..'\n\n* 'is capable of' >> 'is capable to do so' (and at this point, is not\n  yet clear how the bound helps avoid the problems with distribution\n  matching.. )\n\n* Figure 1 (and subsequent figures): suggest put A, B and A' and B'\n  for the two classes and domains, inside the circles, so it's easier to see what\n  the source and target domain classes are (dotted boundary is hard to discern).\n\n\n* on top of pg 4: couldn't the feature extractor make the max-player\n  stronger too? ( In \"... since the max-player taking two parameters\n  f1 , f2 is too strong, we introduce a feature extractor g to make\n  the min-player stronger.. \" ..   )\n\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "H1eCGeC6tS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n-------\nThis paper presents a novel theoretical analysis for unsupervised domain adaptation by revisiting the \\lambda joint error term charactering adaptability in the seminal analysis of Ben-David et al. They propose to replace it by considering discrepancy between information on constrained class hypothesis and a possible discrepancy term that related the learned model with the class A cross-margin discrepancy is proposed in the multiclass context. They extend their approach by proposing to reweight differently some errors with an expected performance of target models on source and an additional one where they distinguish the performance with respect to accuracy on pseudo-labeled target data. Their approach leads to an adversarial-based loss function to optimize which is evaluated on two visual domain adaptation tasks. \n\nEvaluation\n--------\nThe idea is novel and I find the discussion on the considered restriction on the hypothesis class interesting. The experimental evaluation is interesting with good results reported on the two problems considered. I think that some parts could be improved in terms of presentation, in particular The paper contains many typos that make sometimes the reading difficult.\n\nOther comments\n------------\n\n-The comparison with other existing bounds is interesting. I think the authors should expand this by also taking into consideration the bound of Mansour et al., COLT'09 which has some links with the proposed approach (check their comparison to Ben-David's bound). \nThe links with this bound and the one of Ben-David could also be summarized in an appendix, I would like to see a better characterization of the cases where the proposed bound is better and worse. \n\n-About original proposal. I am wondering if the authors could discuss the relationship between the value of \\gamma and the expressiveness of the considered model. If the model is powerful enough, \\gamma should certainly be large. In a context of a training with \"Learning without forgetting\" strategy, the performance on source could maintained in a high standard. \nNote that in this context, the classic joint error of Ben-David et al. can be considered as rather small.\n\n-About the alternative proposal: I am a bit skeptical on the ration \\eta and (1-\\eta) for accuracy on source and pseudo-labeled data respectively. Indeed, since the pseudo-labels are obtained from a classifier that make use a lot of source information, one may think that their performance is rather related. So using to correlated ratios would probably be more relevant here, but maybe the authors can bring some arguments against.\n\n-In the experimental evaluation, the tuning of the different parameters, in particular \\gamma and \\mu, is not particularly discussed and there is probably an issue. I tend to think that these values are rather difficult to assess. A discussion on this point would be welcomed.\n\n-I am not sure to understand the optimization problem (8), (9) and (18), in particular the term after the \"s.t.\": I would expect an inequality somewhere, otherwise everything can be added to the general objective function. If there is an alternate optimization scheme, this should be mentioned explicitly.\n\n-Table 1 and Table 2: use a third identifier different from the second for the last version of your method.\n\n-Please check the typos.\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Given two distributions, source and target, the paper presents an upper bound on the target risk of a classifier in terms of its source risk and other terms comparing the risk under the source/target input distribution and target/source labeling function. In the end, the bound is shown to be minimized by the true labeling function for the source, and at this minimum, the value of the bound is shown to also control the "joint error", i.e., the best achievable risk on both target and source by a single classifier. 

The point of the analysis is to go beyond the target risk bound presented by Ben-David et al. 2010 that is in terms of the discrepancy between the source and target and the performance of the source labeling function on the target or vice versa, whichever is smaller. Apparently, concrete domain adaptation methods "based on" the Ben-David et al. bound do not end up controlling the joint error. After various heuristic arguments, the authors develop an algorithm for unsupervised domain adaptation based on their bound in terms of a two-player game.

Only one reviewer ended up engaging with the authors in a nontrivial way. This review also argued for (weak) acceptance. Another reviewer mostly raised minor issues about grammar/style and got confused by the derivation of the "general" bound, which I've checked is ok. The third reviewer raised some issues around the realizability assumption and also asked for better understanding as to what aspects of the new proposal are responsible for the improved performance, e.g., via an ablation study.

I'm sympathetic to reviewer 1, even though I wish they had engaged with the rebuttal. I don't believe the revision included any ablation study. I think this would improve the paper. I don't think the issues raised by reviewer 3 rise to the level of rejection, especially since their main technical concern is due to their own confusion. Reviewer 2 argues for weak acceptance. However, if there was support for this paper, it wasn't enough for reviewers to engage with each other, despite my encouragement, which was disappointing.
Decision: Reject