Unsupervised Domain Adaptation via Minimized Joint Error
Authors: Dexuan Zhang
Thomas Westfechtel
Tatsuya Harada
Venue: Accepted by TMLR ()

Abstract: Unsupervised domain adaptation transfers knowledge from a fully labeled source domain to a different target domain, where no labeled data are available. Some researchers have proposed upper bounds for the target error when transferring knowledge. For example, Ben-David et al. (2010) established a theory based on minimizing the source error and distance between marginal distributions simultaneously. However, in most research, the joint error is ignored because of its intractability. In this research, we argue that joint errors are essential for domain adaptation problems, particularly when the domain gap is large. To address this problem, we propose a novel objective related to the upper bound of the joint error. Moreover, we adopt a source/pseudo-target label-induced hypothesis space that can reduce the search space to further tighten this bound. To measure the dissimilarity between hypotheses, we define a novel cross-margin discrepancy to alleviate instability during adversarial learning. In addition, we present extensive empirical evidence showing that the proposed method boosts the performance of image classification accuracy on standard domain adaptation benchmarks.

Published in Transactions on Machine Learning Research (06/2023)
Unsupervised Domain Adaptation via Minimized Joint Error
Dexuan Zhang The University of Tokyo
deruan.bbon@@mi..uttbto.....pppp
Thomas Westfechtel
thomas@mii..o-ookgoo......
The University of Tokyo
Tatsuya Harada
hamada@@mit..ttookoo..o..
The University of Tokyo RIKEN
Reviewed OpenReview: https: openreview.
httpe://openneeveeu
net/ forum? wwt.c//aaaacct////ototteeee-aaa
Abstract
Unsupervised domain adaptation transfers knowledge from fully labeled source domain to different target domain, where labeled data available. Some researchers have proposed upper bounds for the target error when transferring knowledge. For example, Ben--Dvvid et al. (2010) established a theory based on minimizing the source error and distance between marginal distributions simultaneously. However, most research, the joint error is ignored because of its intractability. In this rescarch, we argue that joint errors are essential for domain adaptation problems, particularly when the domain gap is large. To address this problem, we propose novel objective related to the upper bound of the joint error. Moreover, we adopt a source/ cource/pseudo-tarre target label-induced hypothesis space that can reduce the search space to further tighten this bound. To measure the dissimilarity between hypotheses, we define a novel cross-margin discrepancy to alleviate instability during adversarial learning. In addition, we present extensive empirical evidence showing that the proposed mothod boosts the performance of image classification accuracy on standard domain adaptation benchmarks.
1 Introduction
Traditional machine-learning theorics generally assume that training and test data drawn from identical distribution, that is, the same domain. However, this assumption does not necessarily hold in real-world settings. Considering image classification as an example, several factors, such as the change in light, noise, angle at which the image is captured, and different types of semsors, can lead to domain gap that harms the performance when predicting the test data. In many practical cases, a model trained in one domain is expected to be applied another. As a solution, domain adaptation (DA) aims to transfer the knowledge learned from a source domain, which is typically fully labeled, into different (although related) target domain. This research focused the most. challenging case, unsupervised domain adaplation (UDA), in which target label is available.
Con-David et al. (2010) suggested that the target error can be minimized by bounding the error source data, the discrepancy between domains, and a small optimal joint error. The optimal joint error is quantified by the lowest error rate that a hypothesis can achieve both domains. Many researchers have focused on learning domain-invariant features such that the discropancy between domains can be minimized. Two strategies were explored for aligning the domains. The first method bridges domains by matching their statistics (Long Pan et al. 2009). The second method utilizes adversarial learning (Goodfellow et. al. 2014
et al. 2015 2017

---

Published in Transactions on Machine Learning Research (06/2023)
to develop minimax game in which domain discriminator is trained to distinguish the source from the target domain while the feature extractor is trained to confuse it simultaneously (Ganin & Lempitsky 2015 Ganin et al. 2016: Tzeng et al. 2017). Despite the remarkable performance achieved by domain-matching schemes, they still suffer from a major limitation: the joint distributions of feature spaces and categorics are not well-aligned across domains. As reported in Ganin el al. (2016), such methods fail to generalize to closely related source-targe pairs, for example, the adaptation from MNIST (LeCun et al., 1998) to SVHN (Netzer eet al. 2011). One potential reason is that when matching the marginal distributions of the two domains, samples from different classes can be mixed up, where the joint error becomes nonnegligible because no hypothesis can jointly classify the source and target data with high accuracy.
This research aims to address the aforementioned problem by incorporating the joint error to formalize an optimizable upper bound such that the mismatch of the joint distributions can be properly penalized. Adversarial learning is effective for aligning distributions [Ganin et al. 2016 Tzong et al. 2017: Saito et al., 2017b; Zhang et al., 2019b;; however, it can suffer from instability owing rapid changes in the discriminator. We propose a specially designed distance measurement based on margin theory that has a smoother gradient around the decision boundary to alleviate this instability, namely, cross-margin discrepancy (CMD). In addition, when we apply CMD to the proposed target error bound, we can prove that part of the objective can be transformed into a CGAN (Mirza & Osindero 2014h objective + which is effective in aligning conditional distributions. We evaluate the proposed method using several classification tasks. Our method improves performance by a large margin, particularly when the domain gap is large. The contributions this research can be summarized as follows:
We propose a novel objective that relates to an upper bound of the joint error and show that our proposal can reduce the chance of misalignment through the distribution matching,
We create a specific hypothesis space induced by pattceesseuoottae objective and avoid optimizing a loose bound within an immense searching space.
labels to stiffen the proposed
We adopt novel measurement, namely CMD, which measures the dissimilarity between hypotheses. We demonstrate that using this measurement, we can alleviate the instability during adversarial learning.
We provide extensive empirical evidence showing that our proposal outperforms other upper bound related methods in image classification on several DA benchmarks, particularly when the domain gap is large.
2
Related Work
The upper bound proposed by Ben-David et al. (2010) invoked numerous approaches that focused on reducing the discrepancy between the source and target domains by learning domain-invariant features. Long et al. (2015 2017) utilized maximum mcan discrepancy (MMD) to match the hidden representations of certain layers in deep neural network. Pan et. al. (2011) proposed transfer component analysis to learn subspace across domains in a reproducing kernel Hilbert space using MMD, which dramatically minimizes the discrepancy between domains. Li et al. (0018) applied adaptive batch normalization to modulate statistics from the source to the target domain in the batch normalization layers across the network in a parameter-Fe manner.
Another way to learn domain-invariant fcatures is to leverage generative adversarial networks. Ganin & Lempitsky (0155) relaxed the discrepancy measurement in the upper bound from BennDavid al. (011)), where a domain classifier is trained to distinguish different domains while feature extractor is trained to confuse it. Tzeng et al. (2017) followed this idea. but divided the training procedure into a classification stage and an adversarial learning stage. [aaio et al. (2017b) explored a tighter bound by explicitly utilizing task-specific classifiers as discriminators such that features near the support of source samples will be favored by the extractor. Zhang et al. (2019b) introduced margin disparity discrepancy, a novel measurement with rigorous generalization bounds, to bridge the gap between the theory and algorithm. Methods that perform
In Eq we show that our proposal includes a CGAN objective that a aligns three pairs of hypotheses induced distributions
2

---

Published in Transactions on Machine Learning Research (06/2023)
distribution alignment at the pixel level in raw images, known as image-to-image translation, have been explored by Liu & Tuzel (2016); Bousmalis et al. (2017); Sankaranarayanan et al. (2017); Shrivastava et. al. (2016); Hoffman et al. (2018); Murez et al. (2017)
Distribution matching can not only bring domains closer, but also mix samples from different classes. Therefore, Sener et al. (2016); /Zhang et al. (2018) used pseudo-labels to learn discriminative target representations. However, this typically requires a complex labeling procedure and data-dependent hyperparameters to set threshold for reliable prediction. Long al. (2018) proposed framework that conditions adversarial adaptation models the discriminative information conveyed in classifier predictions. However, it matches the marginal distributions of joint distributions for features and pscudo-predictions which require a balanced label assumption to achieve the aligmment of conditional distributions. Wu et al. (2019) raised attention to the increasing joint error caused by distribution matching and proposed a relaxed match by restricting the power of domaín classifier to handle the label shifting problem. However, this requires an overlap between the source and target domains the input space and does not necessarily reduce the joint crror. Saito et al. (2017a) claimed that their proposal can reduce the joint error along with the discrepancy between domains by progressively creating a pseudo-labeled target set. However, it reforms the joint error as the sum of the error on the source and pscudo-labeled target domains and a false label rate that cannot be optimized. In this research, we proposed a theoretical approach that constructs an optimizable upper bound on the joint error for UDA tasks.
In summaary, the upper bound proposed by BennDavid et al. (2010) and its extensions (Ganin et al. 2016 Tzeng et al., 2017 Saito et al. 2017b Zhang al. 2019b) continue improve the alignment of the marginal distributions of the source and target domains. However, the problem of ignoring joint errors remains unsolved. In the following sections, we theoretically explain the ways by which it can harm performance in DA tasks and the ways our proposal can handle this problem.
3 Proposed Method
In this section, present details of the proposed method. First, in Sec we propose target error bound and show its relation to the joint error. Then, we thcoretically explain the importance of joint errors in DA. Second, in S.2 we introduce approximated labeling functions inside constrained hypothesis space to formalize an objective that can be optimized. Finally, in Sec 3.3 we propose a novel measurement for the dissimilarity between hypotheses based on the margin theory and show its utility in adversarial learning.
3.1 Upper Bound Incorporating Joint Error
We consider the UDA as multiclass classification task in which the learning algorithm has access to set of n labeled points {(r), vi) E (X E E x x =1, K})}= sampled i.i.d. from the source domain S and set of m unlabeled points ((x]) E X E R sampled ii.d. from target domain T. Let fs :x: E RD R and fr: X E R + R be the true labeling functions in the source and target domains, respectively, and their outputs are one-hot vectors denoting the corresponding classes of inputs. Let eD(J denote distance metric that measures the expectation ion of disagreement between two functions J, over distribution D. To refer to the source error of a hypothesis & E H X E RD + RRKK we use the shorthand es(h) ;= (h( fs), which measures the disagreement w.r.t. the true labeling function fs over the domain S. Similarly, we use (r(h) to denote the target error (the second argument of e is omitted when we refer to the source and target errors according to Ben--avid et al. (0110)). With these notations, we propose the following upper bound for target error"
(r)h) <h) + er(fs, + te((fs,f)) fr) + er - =e)(h) + Cs. hr
ee, For simplicity, we use (fs, fr, h) to denote eT(fs. fs..s.fri fr) (es(fs, fr) + (h, fs)-fss)sse (h, fr), which indicates a discrepancy between the domains. The aformentioned upper bound is minimized when fs, and the
28ee proof in A.6
3

---

Published in Transactions on Machine Learning Research (06/2023)
minimum is equal to fr) given the triangle incquality?. Furthermore, demonstrate that, in such case, our proposal becomes the upper bound of the optimal joint error A.
The reason we focus on the joint error is that Ben-David et al. (2010) state the target error can be upper bounded by the sum of the source error and the marginal discrepancy between domains and the joint error:
et(h) S Es(h) + max h.JJEH
(es(f, (f1,
However, many works (Ganin et al. 2016 Tzeng et al. 2017: Saito et al, 2017b b Zhang al. 2019b
Kim et al. 2019) choose to ignore X and mainly focus on minimizing the marginal discrepancy within S).Tg = {,(x))<< ~ T} introduced by a feature
a transformed input space Sg = {(g(x,) (..(Ir)).))(( (T (.).))))))))))))))))))(.) E
extractor X RDD - IRF and h, f2 E HP :RF SRK
(ss,()) + max min kEHF
(,,, (/1, (2) - (T,, +
SIJZEHF
where joint error is Xy enr cs, (h²) + er, (h"). Without the feature extractor, the joint error is independent lerm. However, when we introduce the feature extractor to minimize the marginal discrepancy, we also violate the independence. The joint error varies with the change in g and can become nonnegligible samples from different classes are mixed, particularly when a large domain gap exists (Zhao et al. 2019). In this case, regardless of the way we minimize the marginal discrepancy, the target error is unbounded because not a single h can jointly classify both domains.
In Fig. [b we illustrate case in which common methods fail to penalize mismatches. Without supervision, samples from different classes can be aligned during marginal distribution matching, which introduces large joint errors. This can be observed on the right-hand side of Fig. [b whereas the marginal discrepancy (areas 1, 3, 4, and 6) is relatively small, a large joint error (areas 2 and 5) exists. In this case, reducing the marginal discrepancy (areas 1, 3, 4, and 6) increases the joint error (areas 2 and 5). Because the two areas represent. the overlaps different classes, no hypothesis can correctly classify the samples in these areas, leading to a nonnegligible joint error. In the case illustrated in Fig. 1b fs assumed to have a specific shape for simplicity. Then, eT(Ss .T) (the minimum of our upper bound) exactly measures the overlapping areas 2 and 5, which are equivalent to the optimal joint error. Our proposal is closely related the optimal joint error; thus, it adequately reduce the size of the incorrect overlapping during distribution matching. In addition, according to Mansour et al. 2209), we provide the Rademacher complexity bound in A.3.
Small Juen Enor Dioopprcy
Huge Ente
Sinul Decrapacy
fs
fs
Clase A
Class B
fs
2
ft
fT
naiding
fT
h
5
(a) legend
(b) joint error
(c) location of fr
Figure 1: Left side a classifier (arrow) is class A and the right side class B; (a) Legend; (b) Joint error (areas 2 and 5) is unbounded by simple distribution matching; (c) even with the marginal distribution aligned, Is and fr may be significantly different.
3.2 Hypothesis Space Constraint
Because the true labeling functions fs. fo are not available, relax the upper bound Eq 1 by considering maximum w.r.t. approximated labeling functions fi, within hypothesis space 3
er (h) < (h) + OS. (fss fr, S <es(h) + max (1,r(11122, fiffEH
3Qur proposal holds fs, fo H, which is proved in A.4
4

---

Published in Transactions on Machine Learning Research (06/2023)
However, if we leave H unconstrained, the maximum term can be arbitrarily large. To obtain tight bound, the size of the hepothesis space must be restricted and an upper bound must be maintained. We assume that it is possible to create two subspaces H1, H2 C H such that the following holds:
Cs.(ffs: fr, S
CS. (f1, h2, S 2.
max
max Ju.f2EH
(f,,
(2)
The right-hand side of the aforementioned inequality always holds because of the subspace relationship. A sufficient condition for the left-hand inequality is fs E H1, H2. The condition for H1 can be easily achieved because we have source labels to create space consisting of all the classifiers for the source domain, namely Hee that must. contain fs- However, the condition for H2 is slightly problematic because we have no access to the true labels of the target domain; therefore, locating fr is difficult. Consequently, we construct a constrained hypothesis space for H2 that is likely to contain fr. Generally, it is difficult to demonstrate the validity of the left side inequality of Eq theoretically. Alternatively, we present the experimental results of the real-world problem to support our assumption
As illustrated in Fig. lc if the conditional distributions are well-aligned (for example, the left-hand side of Fig. Lc], after matching the source and target domains, it is reasonable to assume that fr E Hęc- However, good aligmment cannot always be achieved, particularly when the domain gap is large (for example, the right-hand side of Fig. lch. Considering this, in the following sections, we propose a hypothosis space based on the constraints for H2, which aims alleviale the worst case caused by the unknown location of fr.
3.2.1 Target-driven Hypothesis Space (THS)
We first create a space in which the hypothesis can classify all the samples from the source domain, namely Ho. Then, we create space His consisting of all classifiers for the pseudo-labeled target domain {(r). R))) E x Y (D) is provided by the prediction of h), where components f E Hi, can minimize (f) (f, We further define H2 as an intersection between two hypothesis spaces, i.e., HH.HH- where the hypothesis can classify samples from the source domain with an accuracy of E [o, 1] and classify samples from the approximated target domain with an accuracy of 1 -7
7 E [0.1] (gencrally, we set. 7 = 1) for simplicity; however, in some experiments we tune the two parameters to show the influence when restricting the belief toward the source predictions and leveraging the escudo-prediction its high compulational cost. Alternatively, we use weighted average to constrain the behavior of f2 as an approximation of the sample from H3 n H!-", which enables us to use to balance the confidence in the source and pseudo-target domains. This leads to the following constraints:
In practice, it is difficult to create such space and sample from it owing to
H1 {fi arg (esss
(3)
H₂ (fzl arg men + (1-
3.2.2 Intuition to the Difference between Hypothesis Spaces
The reason for placing the aforementioned constraints the hypothesis space H2 can be intuitively explained by Fig. 2 If we set the hypothesis space H2 to Hsc (a set of classifiers for all source samples), classifier f2 E H2 is forced to correctly classify all source samples. This increases the probability that all target samples are outside the decision boundary of f2, particularly when the domain gap is large. In this case, these samples can be moved to either side of the decision boundary of fo to reduce objective eT (fi, f2) (shaded area in Fig. which may cause incorrect aligmment. By constructing H₂ as an intersection of two hypothesis spaces H%. (a set of classifiers for some of the source samples), H::- (a set of classifiers for some of the labeled target samples) in Fig, 2b we alleviate the restrictions on fa such that the decision boundary can pass through at least few target samples where the objective (shaded area) can be properly reduced by
We show that our proposal remains a valid upper bound, even the domain gap large as we are unlikely to construct hesot space that contains fr A.5
5

---

Published in Transactions on Machine Learning Research (06/2023)
moving the samples inside the decision boundary of fz. In addition, given these hypothesis space constraints, we demonstrate that under certain conditions, our proposal reduces to other UDA method
ss(S. f2)
f
c7(h. 6)
cr(h,6)
(a) H2 Hsc
(b) H2 H63 H
Figure 2: Improper hypothesis space constraint can cause wrong alignment. (a) If f2 is forced to only classify all source samples, it may completely misclassity all target samples when the domain gap large; (b) If fz classify part of source samples and pseudo-labeled target samples, S, T can be aligned in desired way by minimizing the upper bound (shadow area).
3.3 Margin Discrepancy (CMD)
In this section, inspired by the margin theory, we propose a novel discrepancy measurement called the CMD for c instead of the commonly used loss functions (e.g., logistic, hinge, L1) stabilize the training of adversarial learning. Following the aforementioned notation, define the measurement of the discrepancy between any two hypotheses f2 E H X RRD - R (e.g., multilayer perceptron with output layer of softmax function) over distribution D:
(4)
We then consider the probability function [wla=,, which indicates the y-th element of the out of ʃ e H : X e IR + RK evaluated on input Thus, an induced labeling function, named ly from X - Y is expressed by
1 arg max f(ya yEY
When two hypotheses (f1, disagree on æ, i..,, y1 = (x) + (x) = y/2 the loss is defined as follows:
cmd f2; log fi (v1 x)) log (122==)) + log f2 (3)2 + log(1 f2(11(=))
(5)
When two hypotheses (f1, f2) agree on x, i.c., y = 4 (x) (x), the loss is defined as follows:
cmd(f1, f2;ffffzz;) log max (yll) fz (1)1)) log max fi (y)æ)), fz(1l=))
We demonstrate that CMD can be viewed as: sum of two well-established margin losse: which play significant role in achieving strong generalization performance (Koltchinskii & Panchenko, 2002 an objective of CGAN which is efficient method to aligu conditional distributions {Mirza & Osindero 2014). Another reason to propose such a discrepancy measurement is that it textbf helps alleviate instability in adversarial learning. As illustrated in Fig. 3b) during the optimization of a minimax game, when two hypotheses attempt to maximize the discrepancy (shaded area), if one hypothesis moves exceedingly fast around the decision boundary such that the discrepancy is maximized w.r.t. some samples, then these
58cc the proof A.2
6See proof in A.8
7 7ee proof in

---

Published in Transactions on Machine Learning Research (06/2023)
samples can be moved to either side to decrease the discrepancy by tuning the feature extractor. As shown in Fig- Ba the CMD is flat around the original, that is, the gradient with respect to the points near the decision boundary is relatively small. This can help prevent the aforementioned failure because each update of the hypotheses will be subtle during training.
CMD (Ours)
3.0
Logistic
f
f
Hinge
2.0
0-1
c f2)
f2
1.0
apoilon
0.0
cT( f2)
0..5
2.0
-15
-1.0
00.5
0.0
0.5
1.0
1.5
2.0
(a) loss curve
(b) instability
Figure (a) Losses of binary classification, where the proposed method shows small gradient around the
original; (b) Steep gradients near the original may lead to an extreme move of the decision boundary.
3.4 Training Procedure
We introduce a feature extractor X E RDD - R and the transformed feature space Sg {(((x,) (x,), y,) Sy.T, {x ~ T} well hypotheses h, fi, E H R First, we define the source error of h based on CMD:
R k
Lee = ts, (h) = es, (h,, [s) = Er (en,, [si E,,,y. log fs (v. log(1 hyvlg(xx)))
E Es. log h(,. gxx.))
where the source error can expressed as a cross-entropy loss because the true labeling functions map the inputs to one-hot vectors, denoting their corresponding labels. The source errors of f1: f2 denoted by cs, (f,),, cs, (f2) can be derived analogously.
Second, define constraints for approximated labeling functions fi, f2 to ensure they exist in the proper hypothesis spaces H1, H2 based Eq
LH, (f1) LHa = 7es; (f2) + (1- (/2),
where ér, (f2) is estimated using a pseudo-label- based cross-entropy loss 18
čr, (fz) Er log fo (1(()()()))))
Then, we define the discrepancy term based on CMD:
Lais Cs,, h2
Ex,rr//md fz: ((:)) emd(h, f1: Benas
cmd(h, fz; g (xx))))
Finally, by introducing a trade-off parameter 3 to balance the classification loss and the discrepancy, the overall objective function can be written as follows:
min LH, + LLHz -3Lais
minkE LH, Lu,, + Lee BLdis
Tn complex datasets, regularization terms are introduced to prevent f2 from overfitting noisy pseudo-labels A.1.3

---

Published in Transactions on Machine Learning Research (06/2023)
Algorithm 1 MJE
Input: source data S and unlabeled target data T
Parameter: constrained classifiers f1, f2; shared feature extractor 9; upper bound classifier h; hyperpa- rameters 3; learning rate a
Output: updated parameters g, for iteration 1.2.... do
Step A: optimize 9, S1, f2 to satisfy the hypothesis constraint Compute LH, LHz
Update 9, J2:
(g. + (g, fi, f2) ad(g, fi, f2) where (g.. f2) a(Ls, + Lmz))
f1-f))
Step B: maximize the discrepancy w.r.t. fi,f2 within the hypothesis space constraint
Compute LH, LH2, Lais
Update f1.
(f1, + (fi, f2) + f2) where (2) Ki +1 - -Ihdis)
Step C: minimize the entire larget error upper bound w.r.t. y. for fixed S1, .2
Compute Lees Ldis Update h:
h) - h) + (l)g,h),, where (gg,)) = +lLaatBLau) 8(g.h)
end for
The procedures described in A during one training cycle, we first train g, f1 2 to satisfy the hypothesis space constraint; second, we train fi. f2 to maximize the discrepancy term within the hypothesis space constraint; finally, we train g. h to minimize the entire upper bound. Step C executed four times in a training cycle, according to Saito al. (20176). See A.9 for further details on this objective.
4 Evaluation
In this section, evaluate our proposed method using several different datasets (Digit Netzer al. 2011
LeCun et al. 1998: Hull 1994), VisDA (Peng et al. 2017). Office-Home (Venkateswara et al.) 2017 and Office-31 (Saenko et al. 2010)). J. We choose MCD (Saito et al. 2017b) as the bascline for major comparison with ours because the two methods share the same concept. tightening the upper bound established by Ben-David et al. (2010). In addition, the two methods utilize the same network architecture and training procedure, enabling their results to be comparable. We conduct an ablation study (A.11) to demonstrate the
contributions of each part of our proposal. We manually create an imbalanced label situation and demonstrate the robustness of our proposed method 10. The details of the experimental settings, hyperparameter
selection, and training objectives provided in A.1 A.7 and A.9 respectively.
4.1 Experiment on Digit Dataset
In this experiment, our proposed method was assessed in four types of adaptation scenarios by adopting commonly used digit datasets: MNIST (LeCun et al. 1998). street view house numbers (SVHN) (Netzer et al,, 2011), and USPS (Hull 1994), such that the results can be easily compared with other popular methods. All the experiments are performed in an unsupervised manner without any data augmentation. Tab. | lists the accuracies of the different mothods. Our proposed method improves the performance in nearly all settings, except for a single result, compared with GPDA (Kim al.) 2019). However, their solution requires sampling, which increases the data size and is equivalent to adding Gaussian noise the last layer of the classifier, which is considered type of augmentation. The first setting of our proposal is THS + L1, where 7 serves as a direct comparison with MCD because the two methods share the same architecture, except for the upper bound, which demonstrates rates the importance of the joint error. The second and third settings of our proposal show the influence of the CMD and hypothesis space constraint, respectively.
We 7 0.1 MNIST SVHN
8

---

Published in Transactions on Machine Learning Research (06/2023)
Table 1: Accuracy of models adapted on digits datasets (* denotes the entire training set)
SVHN to MNIST
MNIST to SVHN
MNIST to USPS
MNIST to USPS*
USPS
METHOD
to
MNIST
Source Only
67.1
21.3
76.7
79.7
63.4
MDD
Long et al.
2015
71.1
81.1
DANN
Ganin al.
2016
71.1
25.1
77.3
85.1
73.2
DRCN
Ghilary el
2016
82.0+1.11
40.1
91.8.10.1
73.7+:..1
ADDA Tzeng et al.
2017
76.0
89.4+0.2
90.1
MCD Saito et al.]
2017b
96.240.4
11.211111
94.2+10.7
96.5::0.3
94.1
GPDATKimm et al
[2099]
98.2+0011
96.5+0.2
98.1+0.1
96.4+0.1
ours (THS + L1 17 -=
96.8
30.4+1.
94.5
96.810.3
95.240.2
ours (THS + CMD, 11=11
97.5+0022
31.5+
95.3++0.3
97.2+0.22
95.6+0.2
ours
- - 0.9)
98.6
5..3+1.3
96.8:10.2
97.9
96.9
1.8
1.40
0.50
88
206
M--5
20.00
5.5
S-M
0m0
8.0
MCD
MCD THS IL,
0.
0.00
THIS L
0.00
CMS
0.1
170
0.00
0.50
(a) domain affects (fixed
(b) M-S
(c) S +
(d) M -
1)
Figure 4: (a) Sensitivity w.r.t. 7: (b) Comparisons for marginal discropancy; (c)-(d) Comparisons for convergence rate; All plots are drawn based on the average of three runs.
Fig. 4a shows the result when we fix 7 and we can observe that the hyperparameter y is slightly sensitive to the domain gap. In summary, setting = yields the best performance in most situations because fr can be considerably close after aligning the distributions, particularly in these easily adapted scenarios. However, in MNIST + SVHN, setting 7 = 0.1 provides the optimum, which means that fs, fr are far away owing large domain gap in which no fcature extractor is capable of introducing an identical conditional distribution in the feature space. In addition, setting and find that the accuracy remains high and varies subtly; therefore, omit this figure. Furthermore, Fig. 4b empirically proves that simply minimizing the discrepancy between the marginal distributions does not necessarily lead to reliable adaptation. In addition, Figs. 4cland 4d show the effectiveness of the CMD, which accelerates convergence and provides a slightly better result.
lest the performance different values for 7 = 7 A
4.2 Experiment on VisDA Dataset
The VisDA dataset (Peng al. 2017) is designed for 12-class adaptation task from synthetic to real object images. The source domain contains 152,397 synthetic images generated by rendering 3D CAD models, whereas the target domain is collected from MSCOCO (Lin et al. 2014 and consists of 55,388 real images. Tab. 2 lists the accuracies of the different methods and the proposed method performed well under all settings. Images in this dalaset are more complex than digits; however, our method provides reliable performance. Another key observation is that some competing methods (e.g., DANN and MCD), which can be categorized as distribution matching based on adversarial lcarning, perform worse than MDD (which simply matches statistics) in classes such as plane and horse, whereas our method performs better across all classes. This clearly demonstrates the importance of joint error.

---

Published in Transactions on Machine Learning Research (06/2023)
Table 2: Accuracy ResNet- 101 model fine-tuned on VisDA dataset within 10 epochs
METHOD
plane
beyel
bus
car
horse
knife
meyel
person
plant
sktbbed
train
truck
avg
Source Only
55.1
53.3
61.9
59.1
80.6
17.9
79.7
31.2
$1.0
26.5
73.5
8.5
52.4
MDD
MDD
Long dt al.
2015
87.1
63.0
76.5
42.0
90.3
42.9
85.9
53.1
49.7
36.3
85.8
20.7
61.1
DANN
Gani et a
2016
81.9
77.7
82.8
41.3
81.2
29.5
65.1
28.6
51.9
54.6
82.8
7.8
57.4
MCD Salto ef al 20176
87.0
60.9
83.7
61.0
88.9
79.6
84.7
76.9
88.6
40.3
83.0
25.8
71.9
GPDA Kim et a 2019
83.0
74.3
80.4
66.0
87.6
75.3
$3.8
73.1
90.1
57.3
80.2
37.9
73.3
MCC Jm ct al
2020
88.1
80.3
80.5
71.5
90.1
93.2
85.0
71.6
80.4
73.8
85.0
36.9
78.8
outs (THS 11.7 =1=
86.3
82.7
83.7
68.7
87.9
72.7
85.4
61.5
87.3
55.5
75.2
34.1
73.4
ours (THS +
88.4
83.3
74.8
78.0
88.1
43.2
88.2
68.9
87.6
65.5
92.6
58.5
76.4
OuES (TH5 CMD,
93.8
79.5
79.3
55.9
93.9
93.8
86.5
80.3
91.6
87.7
85.4
51.6
81.6
1.90
0.50
0.70
MCD THS+L
0.70
0.80
F0.00
0.70
0.76
MCD THSS L,
VidDA
THS+
A.1
8.50
0.00
opends
(a) fixed 17
(b) various values for
(c) faster convergence
(d) higher discrepancy
Figure 5: (a)-(b) Sensitivity w.rtt. 7:77: (c) Comparisons for convergence rate; (d) Comparisons for marginal
discrepancy; All plots are drawn based on the average of three runs.
Fig. 5a shows the result when we fix 1 and the performance decreases while relaxing the constraint, which is perplexing to us. We expect an improvement here because it difficult to believe that fs. fr eventually exist similar space, determining from the relatively low prediction accuracy. Fig. 1lg-5b/e shows the adaptation performance of different values for ? = 77 전 setting, where the prediction accuracy decreases when r is lower than 0.9. One possible reason is that f2 and h might nearly agree the target domain, such that the prediction of h cannot provide more accurate information on the target domain without introducing noisy pseudo-labels. Figs. 5c and 5d demonstrate the effectiveness of the cross-margin discrepancy and the importance of the joint error, respectively.
4.3 Experiment on Office-Home Dataset
Office-Home (Venkateswara et al. 2017) a complex dataset containing 15,500 images from four significantly different domains: Art (Ar), Clipart (CI), Product (Pr), and World (Rw). Tab. 3 lists the accuracies of the different methods. The adaptation accuracy of the source only method is low, indicating that large domain gap is likely to exist. In such cases, we believe that simply minimizing the discrepancy between the source and target domains can fail because the joint error may increase when aligning the distributions. Our proposal generally provides better performance when adaptation is difficult (1st, 8th, and 11th columns). These results suggest that our proposal, which incorporates the joint error into the target error upper bound, can boost performance, particularly when there is a large domain gap. SPL. (Wang Breckon 2020) and SRDC (Tang al., 2020) share similar strategy, which can be described cluster-based iterative pseudo-labeling. Despite their outstanding performances, these methods lack theoretical guarantee of the generalization error bound. In addition, they assume that samples in the target domain well clustered within the feature space and could be labeled by the Euclidean distance from clusters obtained using K-means. The first. assumption is not always true because high-dimensional features can lie in a low-dinensional manifold (e.g., the Swiss Roll dataset), where no cluster exists. Regarding the second assumption, the labeling scheme can be vulnerable because K-mcans sensitive to initialization, and the Euclidean distance can suffer from the curse of dimensionality such that there is little difference the distances between different pairs of points.
10

---

Published in Transactions on Machine Learning Research (06/2023)
Table 3: Accuracy of ResNet-50 model fine-tuned on the Office Home dataset. We repeated each experiment five times and reported the average of the accuracy.
METHOD
AR-+Ct
An-+Pa
AR-RR Rw
CL-AAR
Cl.+PP
Cl.+RRw
PR-+AR
PR-CCE
PR-sRw
Ro-+AA
Rou--Ch
PPc--PP
AVG
SOURCE ONLY
34.9
50.00
58.0
37.4
41.0
26.2
36.5
31.2
60.4
55.0
41.2
395.0
16.1
39.3
76.8
MCD
61.9
70.7
74.3
09.0
68.4
68.8
08.2
61.6
76.1
63.6
55.8
79.3
66..5
CDAN
LoiN b9
G1.7
70.6
76.0
07.5
70.0
71.0
D7.4
66.9
77.8
70.8
56.7
81.6
66.8
ADA
63.4
70.9
65.99
68.8
77.6
62.8
72.8
78.5
74.2
79.5
74.6
82.7
67.6
SPL
WASt & BIESKKON
2014
64.3
77.8
61.9
65.1
75.0
81.1
06.0
61.1
82.8
69.9
55.3
866.1
71.0
AATIA YANe BY
61.0
71.3
77.5
60.5
70.8
71.2
18.1
61.8
76.9
71.0
07.4
81.8
67.0
72.7
T8.7
68.1
74.5
79.9
72.6
07.8
GARS (7 P8 C.MAY
0.9)
00.3
77.8
81.0
0650
20.4
74.5
0007
09.5
81.8
74.2
092.7
651.3
72.0
5
S
(a) conditional discrepancy (ours) (b) conditional discrepancy (MCD)
(c) marginal discrepancy (ours)
Figure 6: Feature visualization with t-SNE in scenario Real to Art; (a)-(b) shows the conditional discrepancy by using different colors for each class; (c) shows the marginal discrepancy by using the same color for all the classes.
In addition, we plot the learned features using t-SNE (van der Maaten & Hinton 2008 in Fig. which shows Real to Art scenario. In Fig. 6a] we illustrate the source (dot) and target (cross) features using different colors that represent their classes. In the proposed method, most of the target features are clustered to their corresponding sources and do not show a large variance from the class center compared with MCD (Fig. 6b). We also visualize the features of the source (red dots) and target (blue crosses) domains. Fig, 6c shows the marginal discrepancy.
4.4 Experiment on Office-31 Dataset
Office-31 (Saenko et al. 2010 is a widely used dataset for verifying the effectiveness of a DA algorithm that contains three diverse domains, Amazon (A), Webcam (W), and DSLR (D), with 4,652 images unbalanced classes. Tab. f lists the results for Office-31. In tasks D-* and W-*A, determining by the relatively low adaptation accuracy across all methods, large domain gap between the source and target domains is likely to exist. Our method functions well in such cases, demonstrating that it penalizes the mismatch between the source and target domains. However, this advantage is not remarkable, particularly for the task A-W, where our proposed method shows relatively high variance and poor performance. One possible reason for this is that our method depends on learning reliable classifiers for the source domain lo satisfy the constraints of the hypothesis space. However, the Amazon domain contains considerable noise, and the entire dataset lacks diversity (c.g., some domains only have several hundred samples with many duplicates), rendering it difficult for an error-bound based method that generally requires sufficient sample size to learn reliable classifier.
11

---

Published in Transactions on Machine Learning Research (06/2023)
Table Accuracy ResNet-50 model fine-tuned on the Office-31 dataset, We repeated each experiment five times and recorded the average and the standard deviation of the accuracy.
METHOD
A-WW
W
W-D
A-+
D-+
W-+A
AvG
SOURCE ONLY
68.410.2
96.710.1
99.310.1
68.910.2
62.510.3
60.710.3
76,1
DANN
GANIN ET AL.
2016
82.040.4
96.940.2
99.110.1
79.740.4
68.210.4
67.410.5
82.2
ADDA TZENG RT AL..
2017
86.2+0.5
96.2+0.3
98.4+0.3
77.8+0.3
69.5+0.4
68.9+0.5
82.9
MCD SAITO ET
20LYB
88.610.2
98.520.1
100.01:0
92.210.2
69.510.1
69.710.3
86.5
LONG ET AL.
2018
94.140011
98.620011
100.04.0
92.910.2
71.020.3
69.310.3
87.7
SYMNETS ZHANG ET AL...
2019A
90.8+0.1
98.8+0.3
100.0+00.0
93.9+0.5
74.6+0.6
72.5+0.5
88.4
SPL WANG & BRECKON
2020
92.7+0.00
98.1=0.0
99.8+0.0
93.710.0
76.420.0
76.910.0
89.6
MCUNJN ET AL.
22020
95.5++.22
98.6+0.1
100.0+0.0
94.4+0.3
72.9+0.2
74.9+0.3
89.4
SRDO LANG
2020
94.6+1.0
99.2+0.5
100.0+0.0
92.6+0.6
78.1+13
76.3+0.2
90.1
SCAL WANG ET AL.
2022
93.510.2
98.5#0.1
100.0
93.440.3
72.4+0.1
74.040.3
88.6
OURS (THS CMD,7
0.9)
91.9+0.5
99.0#0.2
100.0
93.740.5
76.1
77.810.2
89.8
5 Conclusion
In this research, we propose a novel upper bound that considers joint errors. Subsequently, we pursue a tighter bound with reasonable constraints on the hypothesis space. Additionally, we adopt a novel cross-domain discrepancy for dissimilarity measurement, which alleviates instability during adversarial learning. Extensive empirical evidence shows that an invariant representation is not sufficient to guarantce good generalization performance in the target domain because the joint error affects, particularly when the domain gap is large. We believe that our results contribute significantly in understanding UDA and stimulate future work on the design of stronger adaptation algorithms.
6 Acknowledgements
This research is partially supported by JST Moonshot R&D Grant Number JPMJPS2011, CREST Grant Number JPMJCR2015 and Basic Research Grant (Super AI) of Institute for AI and Beyond of the University of Tokyo.
References
Peter L. Bartlett and Shahar Mendelson. Rademacher and gaussian complexities: Risk bounds and structural results. J. Mach. Learn. Res., 3:463-482, 2002.
Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Vaughan. A theory of learning from different domains. Machine Learning, 79:151-177 175, 2010.
Konstantinos Bousmalis, Nathan Silberman, David Dohan, Dumitru Erhan, and Dilip Krishnan. Unsupervised pixel-level domain adaptation with generative adversarial networks. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 95-104, 2017.
Jun Deng, Wei Dong, Richard Socher, Li-Jia Li, Kuntai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-255, 2009.
Kamran Ghasedi Dizaji, Amirhossein Herandi, Cheng Deng, Weidong Cai, and Heng Huang. Deep clustering via joint convolutional autoencoder embed- ding and relative entropy minimization. In International Conference on Computer Vision, 2017.
Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In Proceedings of the 32nd International Conference on Machine Learning, volume 37, pp. 1180-1189, JMLR.org, 2015.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural networks. Journal of Machine Tearning Research, 17(1):2096-200 22000, 2016.
12

---

Published in Transactions on Machine Learning Research (06/2023)
Muhammad Ghifary, W. Bastiaan Kleijn, Mengjie Zhang, David Balduzzi, and Wen Li. Deep reconstruction- classification networks for unsupervised domain adaptation. In European Conference on Compuler Vision, volume 9908, pp. 597 613. Springer, 2016.
Ian Goodfellow, Jean Pouget-Abadie, Mchdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems, pp. 2672 2680. Curran Associates, Inc., 2014.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. IEEE Conference Computer Vision and Pattern Recognition, pp. 770 778, 2015.
Judy Hoffman, Eric Tzeng, Tacsung Park, Jun-Yan Zhu, Phillip Isola, Kate Sacnko, Alexci Efros, and Trevor Darrell. CyCADA: Cycle-consistent adversarial domain adaptation. In Proceedinggs of the 35th International Conference on Machine Learning, volume 80, pp. 1989-1998. PMLR, 2018.
Jonathan Hull. Database for handwritten text recognition research. IEEE Transactions on Pattern Analysis and Machine Intelligence, 16:550 554, 1994.
Ying Jin, Ximei Wang, Mingsheng Long, and Jianmin Wang. Minimum class confusion for versatile domain adaptation. In European Conference Computer Vision, 2020.
Minyoung Kim, Pritish Sahu, Behnam Gholami, and Vladimir Pavlovic. Unsupervised visual domain adaptation: A deep max-margin gaussian process approach. In JEEE Conference on Computer Vision and Pottern Recognition, 2019.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR, abs/ 1412.991, 2014.
V. Koltchinskii and D. Panchenko. Empirical margin distributions and bounding the generalization error of combined classifiers. Ann, Statist., 30(1):1 50, 2002.
Vladimir Koltchinskii and Dmitriy Panchenko. Rademacher processes and bounding the risk of function learning. In High Dimensional Probabilily II, pp. 447. Birkhäuser Boston, 2000.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Sradient-based learning applied to document recognition. In Proceedings of the IEEE, volume 86, pp. 2278 2324, 1998.
Yanghao Li, Naiyan Wang, Jianping Shi, Xiaodi Hou, and Jiaying Liu. Adaptive batch normalization for practical domain adaptation. Pullern Recognition, 80:109 117, 2018.
Tsung-Yi Yi Lin, Michael Maire, Serge J. Belongie, Lubomir D. Bourdev, Ross B. Girshick, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. In European Conference on Computer Vision, 2014.
Ming-Yi Yu Liu and Oncel Tuzel. Coupled genorative adversarial networks. In Advances in Neurol Information Processing Systems, pp. 469 477. Curran Associates, Inc., 2016.
Mingsheng Long, Jianmin Wang, Guiguang Ding, Jia- Jia-Guang Sun, and Philip S. Yu. Transfer feature learning with joint distribution adaptation. IEEE International Conference on Computer Vision, pp. 2200-2207, 2013.
Mingsheng Long, Yue Cao, Jianmin Wang, and Michael I. Jordan. Learning transferable features with deep adaptation net works. In Proceedings of the 32nd International Conference on International Conference on Machine Learning, volume 37, pp. 97-105. JMLR.org, 2015.
Mingsheng Long, Han Zhu, Jianmin Wang, and Michael I. Jordan. Deep transfer learning with joint adaptation networks. In Proceedings of the 34th International Conference on Machine Learning, volume 70, pp. 2208 2217. oMgg. 2017.
13

---

Published in Transactions on Machine Learning Research (06/2023)
Mingsheng Long, ZHANGJIE CAO, Jianmin Wang, and Michael I Jordan. Conditional adversarial domain adaptation. In Advances in Neural Information Processing Systems, pp. 1640-1650. Curran Associates, Inc., 2018.
Yishay Mansour, Mchryar Mohri, and Afshin Rostamizadch. Domain adaptation: Learning bounds and algorithms. In COLT, 2009.
Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets, 2014.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. ArXiv,
2018.
Zak Murez, Soheil Kolouri, David J. Kriegman, Ravi Ramamoorthi, and Kyungnam Kim. Image to image translation for domain adaptation. IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 4500-4509, 2017.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupercised Feature Learning, 2011.
S J Pan, I W Tsang, J T Kwok, and Q Yang. Domain adaptation via transfer component analysis. IEEE Transactions on Neural Networks, 22(2):199-211 2011.
Sinno Jialin Pan, Ivor Wai-Hung Teang, James T. Kwok, and Qiang Yang. Domain adaptation via transfer component analysis. IEEE Transactions on Neural Networks, 22:199-21 2209, 2009.
Xingchao Peng, Ben Usman, Neela Kaushik, Judy Hoffman, Dequan Wang, and Kate Saenko. Visda: The visual domain adaptation challenge. ArXiv, abs/1710.06922, 2017.
Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell. Adapting visual category models to new domains. In European Conference on Computer Vision, pp. 213-226 226. Springer-Verlie Verlag, 2010.
Kuniaki Saito, Yoshitaka Ushiku, and Tatsuya Harada. Asymmetric tri-training for unsupervised domain adaptation. In Proceedings of the 34th International Conference on Machine Learning, volume 70, pp. PMLR, 2017a.
Kuniaki Saito, Kohei Watanabe, Yoshitaka Ushiku, and Tatsuya Harada. Maximum classifier discrepancy for unsupervised domain adaptation. IEFE/CVE Conference on Computer Vision and Pattern Recognition, pp. 3723 3732, 2017b.
Swami Sankaranarayanan, Yogesh Balaji, Carlos D. Castillo, and Rama Chellappa. Generate to adapt: Aligning domains using generative adversarial networks. CoRR, abs/ Noss1704.01105, 2017.
Ozan Sener, Hyun Oh Song, Ashutosh Saxena, and Silvio Savarese. Learning transferrable representations for unsupervised domain adaptation. In Advonces in Neural Information Processing Systems, pp. 2110 2118. Curran Associates, Inc., 2016.
Ashish Shrivastava, Tomas Pfister, Oncel Tuzel, Josh Susskind, Wenda Wang, and Russell Webb. Learning from simulated and unsupervised images through adversarial training. IEEE Conference on Compuler Vision and Pottern Recognition, pp. 2242-225, 2016.
Hui Tang, Ke Chen Chen, and Kui Jia. Unsupervised domain adaptation via structurally regularized deep clustering. In IEEE Conference Computer Vision and Pattern Recognation, 2020.
Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. IEEE Conference on Computer Vision and Pattern Recognition, pp. 2962-2971, 2017.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. Journal of Machine Learning Research, 9:2579 22257926005 2605, 2008.
14

---

Published in Transactions on Machine Learning Research (06/2023)
Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hash- ing network for unsupervised domain adaptation. IEEE Conference on Compuler Vision and Paltern Recognition, pp. 5385 5394, 2017.
Hui Wang, Jian Tian, Songyuan Li, Hanbin Zhao, Qi Tian, Fei Wu, and Xi Li. Structure-condttioned adversarial learning for unsupervised domain adaptation. In Neurocompuling, volume 497, pp. 216 229, 2022.
Qian Wang and Toby P. Breckon. Unsupervised domain adaptation via structured prediction based selective pseado-labeling. In AAAI Conference on Artificial Intelligence, 2020.
Yifan Wu, Fara Winston, Divyansh Kaushik, and Zachary Lipton. Domain adaptation with asymmetrically- relaxed distribution alignment. In Proceedings of Machine Learning Research, volume 97, pp. 6872 6881. PMLR, 2019.
Jianfei Yang, Han Zou, Yuxun Zhou, Zhaoyang Zeng, and Lihua Xie. Mind the discriminability: Asymmetric adversarial domain adaptation. In European Conference on Computer Vision, 2020.
Weichen Zhang, Wanli Ouyang, Wen Li, and Dong Xu. Collaborative and adversarial network for unsupervised domain adaptation. IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3801-3809, 2018.
Yabin Zhang, Hui Tang, Kui Jia, and Mingkui Tan. Domain-symmetrt networks for adversarial domain adaptation. In IEEE Conference on Computer Vision and Pattern Recognition, 2019a.
Yuchen Zhang, Tiaule Liu, Mingsheng Long, and Michael Jordan. Bridging theory and algorithu for domain adaptation. In Proceedings of the 36th International Conference on Machine Learning, volume 97, pp. 7404-7413. PMLR, 2019b.
Han Zhao, Remi Tachet Des Combes, Kun Zhang, and Geoffrey Gordon. On learning invariant representations for domain adaptation. In Proceedings of Machine Learning Research, volume 97, pp. 7523-7532. PMLR, 2019.
A Appendix
A.1
Experimental Setting
A.1.1 Digits Dataset
SVHN ++ MNIST We first examine the adaptation from SVHN (Fig. 7a to MNIST (Fig. 7b
We used
standard training and testing sets for both the source and target domains. The feature extractor contains three 5 x convolutional layers with stride two 3 x 3 max pooling placed after the first two convolutional layers and a single fully-connected layer. For the classifiers, used 2-layer fully-connected networks.
MNIST + USPS As for the adaptation between MNIST and USPS (Fig, 7ch, we followed the training protocol established in Long et al. (2013) by sampling 2000 images from MNIST and 1800 from the USPS. For the test samples, the standard version was used for both source and target domains. The feature generator contains two
convolutional layers with stride two 2 x 2 max pooling placod after cach convolutional layer and single fully-connected layer. For the classifiers, we used 2-layer fully-connected networks.
Throughout the experiment, employed the CNN architecture used in Saito et al. (2017)), where batch normalization was applied to cach layer, and a 0.5 rate of dropout was used between fully-connected layers. In addition, spectral normalization (Miyato et al. 2018) was deployed for the classifiers in all subsequent experiments to stabilize adversarial learning. The major reason for utilizing this technique is that we observe a performance drop when the network is overtrained (this generally occurs after several hundreds of epochs). However, we are unsure whether this caused by an overfitting of the source error or a general neural network training problem related to early stopping (this phenomenon can also be observed in other
15

---

Published in Transactions on Machine Learning Research (06/2023)
algorithms). One common solution is to use gradient reversal layer Ganin et al. 2016) to balance the weight between the source error and discrepancy between the two domains. However, this introduces additional typerparameters for tuning. Additionally, in the proposed CMD method, as illustrated in Fig. g we maximize log fi y1) + +ogg(1 - fz(z, y1)) with respect to the classifiers but minimize only log(1 1 f2 v1)) (a
part of the objective) with respect. to the feature extractor to avoid unnecessary oscillation during verssriial learning. This implics that the objectives we attempt to maximize and minimize are not is not equivalent. In such cases, it is difficult to employ gradient reversal layer, which intended to train the classifiers and feature extractors simultaneously in single objective. With the help of spectral normalization, ensure that the classifiers are approximately Lipschitz such that performance drop at an early stage can be avoided because the gradient with respect to the classifiers will be relatively small. Adam (Kingia & Ba 2014 was used for optimization, with a minibatch size of 128 and a learning rate of 10-1
19.31
(a) SVHN
(b) MNIST
(c) USPS
Figure 7: Random samples from each dataset.
log fx yn) log (2(x
log (2x log fi (xx.)2)
g
log fikkilhhooo
discrepancy
Y1
class label
Y/
Figure 8: Illustrated details for the cross-margin discrepancy
A.1.2 VisDA Dataset
The proposed method was further evaluated for object classification. The VisDA dataset (Pnn et al. 2017). which is designed for a 12-class adaptation task from a synthetic object to real object images, was used. The source domain contained 152,397 synthetic images generated by rendering 3D CAD models. Data from the target domain were collected from MSCOCO (Lin et al., 2014), which consisted of 55,388 real images. Because 3D models are generated without background or color diversity, the synthetic domain is slightly different from the real domain, rendering it considerably more difficult problem than digit adaptation. In addition, this experiment was performed in an unsupervised fashion, and no data augmentation technique, excluding horizontal flipping, was allowed. By following the prolocol established by /Salo el al. (2017b), We evaluated our method by fine-tuning a ResNet-101 (He et al.] 2015)) model pretrained on ImageNet (Deng ct al. 2009). The model, except for the last layer combined with single-layer bottleneck, was used as a feature extractor, and randomly initialized 2-layer connecconeette network was used as a classifier, where batch normalization was applied to each layer and a 0.5 dropout rate was conducted. Stochastic gradient.
16

---

Published in Transactions on Machine Learning Research (06/2023)
descent (SDG) with Nesterov moment was used for optimization with minibatch size of 32 and an initial learning rate of 10-3, which decayed exponentially. The network architecture used in Saito et al. (2017)) was originally 3-layer fully-connected classifier. However, as mentioned in the main paper, we have demonstrated that a relatively weaker classífier improves the stability of adversarial learning. Therefore, we leverage the Tramework proposed in Zhang et al. (2019b) by considering the top layer of the classifier, that is, bottleneck, as part of the feature extractor (it does not change the entire complexity of the model). Another crucial point is that the network parameters with respect to ResNet-101 and the bottleneck were updated separately at ratio of 1:10, which is the same as in Zhang et al. (2019b). A similar approach was used in Saato et al. (2017b) to decelerate the update of ResNet-101. The common reason is that excessively updating the feature extractor is not recommended because ResNet-101 is extremely powerful that it can easily overfit the source error. We applied horizontal flipping of the input images during training as the only data augmentation. For the hyperparameter, we tested y = {0.1, 0.5, 0.9, 1) and = {0, 0.5, 0.8, 0.93. For direct comparison, we reported the accuracy after 10 epochs.
(a) synthetic images
(b) real images
Figure 9: (a) Samples from source domain. (b) Samples from target domain.
A.1.3 Oofe Dataset
officeHHome (Venkateswara et al. 2017 was a complex dataset (Fig. 10 containing 15,500 images from four significantly different domains: Art (paintings, sketches, and/or artistic depictions), Clipart (clip art images), Product (images without background), and Real-world (regular images captured with a camera). In this experiment, following the protocol of Zhang et al. (2019b), we evaluated our method by fine-tuning a ResNet-50 (He et al., 2015) model pretrained on ImageNel (Deng et al. 2009). The model, except. for the last layer combined with single-layer bottleneck, was used as a feature extractor, and randomly initialized 2-layer fully-conneeted network with a width of 1024 was used as a classifier, where batch normalization was applied to each layer and 0.5 dropout rale was conducted. For optimization, we used SGD with a Nesterov momentum term fixed 0.9, where the batch size was 32, and the learning rate was adjusted according to Ganin et al. (2016). We applied horizontal flipping and resized the cropping of the input images during training data augmentation, similar to that in Zhang et al. (2019b).
We introduce a slightly different loss function for čr, (f2): For unlabeled target data {af E TT) the network predictions given by h, f2 are represented by {q: E [Q) and (pr E P)m where the confidence of class k is defined by
4r, = h)g(())) k), PLA = f=(((()))))
In addition to the hypothesis space constraint, we add class balance regularization term for f2 to encourage entropy maximization of the label distribution in the target domain, which improves the performance, particularly when the class number large Saito et al. 2017b
1 KL(p,lla) + min P
=i= PLRk log
Σ
-==
Pi.k
For fixed Q, by setting the approximate gradient to zero, we obtain the following closed- form solution to the aforementioned problem Dizaji al. 2017
Di.k/ Di-l Rikk
Pi.
1k=1 /i.k/ Eizı Pi.k
17

---

Published in Transactions on Machine Learning Research (06/2023)
Finally, we train 9. to push P toward Pi by minimizing the following KL divergence:
K pl log f2(g(1)), k))
Er, (J2) =
(8)
Art
Real World Product Clipart
Spont
Desk-Lamp
Figure 10: Sample images from the Office-Home dataset. (Venkateswara et al. 2017
A.1.4 Office- 31 Dataset
Offico-31 (Sacnko et al., 2010) (Fig. 11 is widely used dataset for verifying the effectiveness of DA algorithm. It contains three diverse domains: Amazon from the Amazon website, Webcam by web camera, and DSLR by digital SLR camera with 4,652 images in 31 unbalanced classes. Several noisy samples are shown in Fig 12 to backup our assumption for the failure case. In this experiment, following the protocol of Zhang et al.
2019b), we evaluated our method by fine-tuning a ResNet-50 (He et al. 2015) model pretrained on Image. Net. (Deng et al. 2009). The model used here nearly identical to that used the Office-Home experiment, except for a different width of 2,048 for the classifiers. For optimization, we used SGD with Nesterov momentum term fixed 0.9, where the batch size 32, and the learning rate was adjusted according to Ganin et al. (2016). We applied horizontal flipping and resized the cropping of the input images during training as data augmentation, similar to that in Zhang et al. (2019b).
AMAZON
am
DSLR
WEBCAM
bike
monitor
bottle
pen
speaker
phone
scissors
mug
mouse
ruler
projector printer
Figure 11: Sample images from the Office-31 dataset.
A.2 Comparisons with Other Methods
This section present that, under certain conditions, the proposed joint error-based upper bound can be reduced to other popular upper bounds, demonstrating the generality of our proposal.
18

---

Published in Transactions on Machine Learning Research (06/2023)
Sootch
Decho Soled
Panasonic.
ORF
ruler
trash tape_ dip_edise pen
bottle
scissors
binder
phone
punchers
computer
Figure 12: Noisy images from the Office-31 dataset.
A.2.1
Margin Disparity Discrepancy
Zhang et al. (019b) proposed novel margin-aware generalization bound based scoring functions and defined new divergence MDD. The training objective used in MDD can be alternatively expressed as (here, e(h, f) denotes the margin disparity) follows:
min (h), max (er, (f.h) cs,(f..)) NEHF.9 .9
JEH"
From 3 3 if we set fi = f2 and free the constraint of to any H², proposal degrades exactly to MDD. As discussed previonsly, the assumption of an identical true labeling function fs = fr is dangerous because we are not aware of the location of fr and perfect alignment between the conditional distributions of the source and larget domains is not likely to occur in practice. In addition, an unconstrained hypot space for f not helpful in building a tight bound.
A.2.2 Maximum Classifier Discrepancy
Saito et al. (017b) proposed two spek-sp classifiers f1, fe that are used to separate the decision boundary on the source domain, such that the feature extractor is encouraged to produce features near the support of the source samples. The objective used in MCD can be alternatively expressed as follows:
ming [cs, (fi) + max, (f1)
s.t. H. = {f] arg min mei 3 es, (f)}
From Eq 3 if set y 1 1 and h f1, proposal reduces to MCD. As proved in 3.1 the proposed upper bound is optimized when h = ʃs. However, this no longer holds after the upper bound is relaxed by obtaining the supremum; that is, setting h = fi does not necessarily minimize the objective. In addition, as discussed previously, assuming H₃ = Hac = H, lacks generality because fr can be far from fs and does not necessarily classify all source samples, which means that the assumption MCD is not likely to be applicable cases in which large domain gap exists.
A.3 Rademacher Complexity
Let H be a set of real-valued functions defined over set X. Given a sample S E X", the empirical Rademacher complexity of is defined follows:
(H)
F sup kEH
Σ
ach(a))) (a), Fm)
The expectation is calculated over a (a1, ...y An), where os independent uniform random variable that accepts values in +1}.
Rademacher complexity measures the ability of a class of functions to fit noise. It has the additional advantage of being data-dependent and can be measured from finite samples, which can lead to lighter bounds than those based on other measures of complexity, such as the VC-dimension. Following the established theory proposod by Mansour et al. (2009), we denote the empirical average of hypothesis h and its expectation over samples drawn according to the distribution considered by R(h). The following is a version of the Rademacher complexity bound (Koltchinskii & Panchenko, 2000 Bartlett & Mendelson, 2002
(0, by R(h))
19

---

Published in Transactions on Machine Learning Research (06/2023)
Let H be class of functions mapping X x Y + 10 and S = ((21, v1). (Ims Bm)) be finite sample drawn i.i.d. according to the distribution Q. Then, for any > 0 with probability of at least 1 over a sample S of size m, the following inequality holds for all h E H:
log 2 a 2m
R(h) - A(h) + R((H)) + +3V
From our proposed upper bound, we have: (r(h) - <s(h) + (ss fr, where Cg.r is further bounded by: sup fa (er(fi, + Esfi, f2) + eT fi) Eh f2)}. We relax this bound by applying es(fi, f2) - cs(fi, h) + Ccg(f2, h) and name it d(S, T:h) s MM. (er(f. f2) + cs(f,, h) er(fi, We assume that the loss function is bounded by M > Oze(h, > M for all h, h' E H. Let Q be a distribution over X and let Q denote the corresponding empirical distribution for sample S = (f1...., Tm)) Then, we can scale the loss to [0, 1] by dividing by M and denote the new class by H2, which represents the class of functions for - ((f fz())) fi E H1, fa E H2}. From the aforementioned theorem, for any o > with a probability of at least a, the following inequality holds for all fi E H1, f2 E H2:
(Q(Si,J2) M
solfinh) M
log 2
ua MM M + +3V
2m
The empirical Rademacher complexity has the property that M(at) = af((H) for any hypothesis class and positive number a simplifics the aforementioned bound. Let S be a distribution over X and be the corresponding empirical distribution for sample S, and let T be a distribution over X and 7 be the corresponding empirical distribution for sample T', Then, for any a 0, with a probability of least 1-o o over samples S of size m drawn according to S and samples T of size n drawn according to T,, we can write the following:
d(S,T;h) S
sup / (eg(fi, + =g((i, ( +( (f)
+ Re(( + Rz(En, ) + Ry (cn,
log
log 2
+ 3M(
2m + 2\
2n
A.4 Compatibility
Despite the assumption that H includes fr was made in previous research like Mansour et al. (2009). we agree that assuming the true labeling functions lie in a specific hypothesis space lacks generality. However, if the algorithm is run within finite samples, it is possible that a function inside a specific hypothesis space with enough complexity can perfectly mimic the behavior of the true labeling function for those samples. Therefore, even if the hypothesis space use does not contain true labeling functions, it does not harm the actual learning process. The proof is as follows:
(h) = er(h, fr)
= hhtp::hhh-..c.kkii-i----a-aahha-
es(h,
=
+cs(f
= +c((h (c
where 0 = ff)
(cs fr) fr(.
Let f = arg migmmin 's f f) and fi = arg min 22s Ur(r,,)) such that f3. fi E H can perfectly mimic the behavior of fs, fr on S and T. Then 0 can be completely ignored, and the obtained bound becomes compatible with the original upper bound by replacing fs, fr with f3, f7-
(fs:
20

---

Figure 13: Estimated means of ground truth and upper bound by three runs in scenario Pr-+I adapted Published in Transactions on Machine Learning Research (06/2023)
A.5
Validity
In Sec 3.2
assume that it is possible to create two subspaces H1 H2 S Hy such that:
Cs. fr. h max A
Cs.T(fi,f2.))
This assumption difficult prove thcoretically; thus, we show the validity of the incquality using experimental results. In this section, select adaplation scenario in which the domain gap is large (Product to Clipart scenario of the Office-Home dataset). We used the full source and target labels to estimate (s, fr, h) as the ground truth. The upper bound is computed with two subspaces H1, H2 given by the target-driven hypothesis space proposal (Sec 3.2.1 that utilizes pseudo-labels, as follows:
H1 {/1 arg wigminn m
H2 = {fzl men + (1 mfrr(
Fig 13 shows that our proposal remains a valid upper bound in practice even the domain gap is extremely large that the subspace H2 is unlikely to include fr.
0.0
ground truth upper bound
-0.5
-1.0
Cs,7,
-1.5
22.0
2.5
-3.00
5000
10000
15000
20000
Iteration
with THS CMD proposal.
A.6 Proof
A.6.1 Proof for the Upper Bound
Given the aforementioned notation and the triangle incquality, have the following:
(h) = (h, fr) + | (s(h):+(t((t_]]]-_((( (es(h, es(h, fr)) -0(( es(h, (s), ST(
I er fr) th cs(,
A.6.2 Proof for the Optimum
We demonstrale that our upper bound is minimized when h = Js and is equivalent to er ST).
First, by setting = fs, we obtain
(s(h) + er(ss JT) es(ss, fT) er 1s) es(h, er(ss:
21

---

Published in Transactions on Machine Learning Research (06/2023)
Then, according to the triangle inequality and nonnegativity of the distance metric, we have
[es(h)) + (s(ss [r)] + (rs,, fr) + er eshh,,
2 csh + er(fs. + crh, es fr)
= =rrffs fr) + chh,
2 IT(
A.6.3 Proof for the Relation to Joint error
We demonstrate that when our upper bound is minimized, it is equivalent the upper bound of the optimal joint error À because of the following:
crfs. cr(s,, cc((ss.fs)
er((ss +es(fs)
2 min(er(h) +es()) kelf
A.7 Hyperparameter Selection
In all experiments, the hyperparameters 7. were based on the validation performance on a separate dataset consisting of few hundred labeled target samples. As shown in the main paper, the performance of our proposed method is sensitive to changes in the hyperparamcters. However, it also suggosts that significantly boost the performance, which enabled us to select
appropriate choice of hyperparameters the hyperparameters via validation dataset with 200 samples. Fig. shows two examples of the validation performance of different adaptation tasks with respect to 7,
1.0
1.0
0.9
0.9
Accuracy
0.8
Accuracy
0.7
0.7
0.6
0.6
S-MM
VisDA
0.5
0.5
0.0
0.2
1.0
0.0
0.2
1.0
7
7
(s) THS L1 where =1
(b) THS CMD where
Figure 14: Validation performance when varying the hyperparameters on separate datasel
A.8 Relations between the Margin Theory and CMD
The margin between the data points and the classification surface plays significant role in achieving strong generalization performance. To quantify into a differentiable measurement as surrogate for the 0-1 loss, introduce the margin theory developed by Kootchhiskki & Panchenko (2002), where the margin loss is interpreted as follows:
E (( [max(0, (eax)) max s(y(x)) s(y(x))),
#'
22

---

Published in Transactions on Machine Learning Research (06/2023)
where the output of the score function s(ylæ) for multiclass classification indicates the confidence of the prediction for class
We aim to utilize this concept to further improve the reliability of our proposed method by leveraging this margin loss to define a novel measurement of the discrepancy between the two hypotheses f1, f2 (e.g., a multilayer perceptron with an output layer of softmax function) over distribution D, namely, the CMD:
cD(fi, f2) Deedd f2; f2)
Before further discussion, we first introduce two distributions D D/2 induced by labeling functions fi, fa, respectively, where Dy {( I, (2))]a: ~ D} and Dj {(x, f (x)) ~ D) (function returns the predictive labels). Then, consider the case in which the two hypotheses (fi and f2) disagree, that is, = In (I) # (x)) and the primitive loss is defined as follows:
amd(f1,f;:1 = log (x)lz)= - log (2(1(z)) + log f2(12(z) - log fi (g2(2) = log fi (11 log (v2l + log f2(v2(x) - log f2(v1 (x)
Then, the CMD en(si,f2) can be expressed as follows:
(( [max log Jo (v/ - (o + B(a,))D (max log f2(1/(x)) - log [2(1(=)) v'#y
This is the sum of the margin loss for f1 D3 and the margin loss for f2 Dj if we use the logarithm of softmax as the score function.
Following the method introduced by Goodfellow et al. 2014) for mitigating the burden exploding or vanishing gradients, we optimize the dual form of the aforementioned primitive loss in practice:
and(Ji,/2;a)) = log fi (v) + log(1 - fi (v2lz)) + log f2(v2(x) + log(1 - f2(21 (x))
A.9 Details of the Training Objective
This section briefly describes the training objectives of the proposed method. We consider the THS objective as example, and each term in the objective is defined as follows:
minneh g (h) eH. Cs (f1, f2, h)]
s.t. Hi H2 (fi, fal arg min /. .el.egg..ssss.. Lss,, (fi) + yes, (f2) + 1))T,, (f2))}.
where g represents a feature extractor and h. f1, f2 are classifiers with a softmax output (i.c., Max y) represents the confidence of sample classified as label y). First, we can use the cross-entropy loss to achieve the following hypothesis space constraint:
(fs, = -Er,y~s log(fi(vl(()))
cs, (fi)
YES, (1 n)ēT, (f2) = ys log(fa(vlo(z))) (1 (1--)) E) (og(((()))))((((
We then use the proposed CMD to reorganize CS, (2. f2. h). Before further derivation, we introduce three proxy distributions Sn=f2: Sy andS induced by f1, where S = {a, y/ S, S, (x) = (x) S {x, y/æ S.y (x) * (z), Syy, {z, y/a S. S, (z) # 11,(x) 8) (the labeling function returns labels of the most confident. prediction by f). Analogously, define the others
23

---

Published in Transactions on Machine Learning Research (06/2023)
Table 5: Influence 3 on the adaptation -MNIST under our proposal (THS + CMD, 0).
SVHN to MNIST
3 = 0.001
3 = 0.01
3 = 0.1
Acc
98.4
98.610.11
0.3
S According to the definition of CMD (Sec 3.3
Tg (2.
Rz. 8 f1=12 (log (y o(s)), fa(ulo(=))) log lagmax(( (v())((=)) 1 f2(vlo(>)))
+ P2.y011 (lox (((la(()) lox(1 (2(xla(=))) /,,1" [ext (u(l(((=)) fox/2(xila(e))
= [lo (vllate)) fa(ula(=))) Log (x( f2(vio(==))1
F...yTTT Dox (ull((z) lex() 12 |ox[[ f1(vla(=))) f2x/2())
+ B, Bn.yTTi
k(vlo(x))) log (olo(=))) ((yla(=)))
+ f (Dxx (((vl((=)) lex(t - ((vla(=))))) Ee.y~TTA, [lxx(1 f1 (x(la(==)) lox h(vl()))()))
Fu.ym (logma(h((vlatt))) (a(ale(=))) lag h(ylo(z)), fa(vla(=)))
Fa.y 1a Box (a(xla(e))
h(xla(()))) E2.g0 [ox(11 (a(xia(=)) hoxlat)))))
where part of this objective can be regarded as CGAN objective aiming align the conditional distributions of three pairs of hypothesis induced distributions (S Tz= S/ e TTyi Saan <> TAA))) under minor assumption that f1, f2 more confident about S,T, respectively.
As explained in A.1.1. to avoid unnecessary oscillations during adversarial learning, only optimize part of the objective (the part considered as fake by the classifier) w.r.t. g, h such that it can be consistent with the general CGAN objective. This implics the objective (f,,,, f2, h)) that we attempt to maximize is not equivalent to (Čs,,7, (f1, h2, that attempt to minimize.
T U1. 12.h)
Ra 11. logmax((1 (vis(()))) f2(vlo(x))) -
Pa.y0//1E logtt - f2(ulo(=))) - logit (1(1a(()))
Fu.y-Th-fp log (1l_)_)))))))))____)) f2(voe((3))
Es.y-TTyV. Lox(1 b=(ll--_pp0iidcc)))=2___-__.___2 lox(1 (1(va((=))
Fu.y~Tnh log max(1 (olo(())) -
Px.unTT lng(1 k(y h(v))(())))))))) Pa.p fo0lot)))
Rr.00 log 10g s(=)) (2(ul(ce))
R lool1 - F2.g~ Ins(1 f2(1l((=)))
(10)
Because we apply different measurements to the source error and discrepancy, we introduce a scaling factor B to ensure that neither of them can dominate back propagation. 0.01 used in all the experiments because we observed that the source error generally approximately le - 2 and the discrepancy generally approximately 1e0. We also conducted simple experiment to check the influence of B and the results are listed in Tab
A.10 Imbalance Label Distribution
In this section, describe additional experiment in which the label distribution of the source is significantly different from that of the target domain. We choose offico-Home datasct and manually remove the samples of an entire class ('clock') from the larget (Clipart) domain to create an imbalance scenario. The adaptation accuracy is listed T In theory, our proposal does not suffer from an imbalance in the label distribution. Unlike other mothods that can only match marginal distributions, the proposed method designod to align conditional distributions. In practice, the experimental results demonstrate the validity of our theory because remarkable performance drop exists in our proposed method compared to other methods. Despite this
24

---

Published in Transactions on Machine Learning Research (06/2023)
Table 6: Accuracy ResNet-50 model fine-tuned on the Office-Home dataset. represents the imbalance
label distribution setting, where we manually remove the samples of an entire class from the target domain.
METHOD
AR-=CL
Rw-+CL PR-CL
SOURCE ONLY
34.9
31.2
41.2
DANN GANIN ET AL.
2016
15.6
43.7
51.8
MCD AITO ET AL.
2017B
51.9
51.6
55.8
CDAN LONG ET AL.
2018
50.7
50.9
56.7
SYMNETS ZHANG ET AL.
2019A
47.7
48.8
52.6
SPL & BRECKON
2020
54.5
53.1
55.3
AADA YANG ET AL.
YANG
2020
54.0
51.8
57.4
SRDC TANG ET AT.
2020
52.3
53.8
57.1
SCAL WANG ET AL.
2022
55.3
51.6
57.8
OURS (THS+ CMD,
0.9)
60.3
59.2
62.7
DANN'
43.1
40.5
49.2
MCD
50.5
49.7
54.3
OURS (THS CMD 0.9)
60.0
58.5
62.2
Table 7: Simple ablation study to demonstrate the contribution from each part of the proposal (V in the column "optimal joint error" means to include the joint error in the upper bound, in contrast to the training objective proposed by MCD; / in the column "CMD" means to use the proposed discrepancy measurement for 6, whereas indicates that hypothesis space" means the hypothesis space for f2 is created based which is corresponding to our THS proposal in the main paper, where v o 1). We repeat the adaptation from SVHN to MNIST five times and record the average and the standard deviation of the accuracy.
I norm alternatively; / in the column "pseudo-labels induced
pseudo-labels of target samples,
OPTIMAL
CROSS
ESEUDO-LABELS
SVHN TO MNIST
METHÒD
JOINT
MARGIN
INDUCED
ERROR
DISCREPANCY
HYPOTTHESI SPACE
BASELINE
X
96.220.4
V
96.610.22
X
V
97.110..3
V
97.6++0.2
OURS
X
96.810.2
V
97.5++0.2
98.240.22
V
98.6+0011
extreme imbalance in the label distribution, our proposed method still outperformed other methods trained the full target domain.
A.11 Ablation Study
In this section, we conduct a simple ablation study to show the way each part of the proposal, that is, the objective, including the joint error, the CMD, and the psondo-label-nddeeed hypothesis space, contributes to the performance gain. For comparison, we chose MCD(Saito et al., 2017b) the baseline because its assumption regarding the hypothesis space is similar to that of our proposal. In addition, the two methods shared the same network architecture in their implementation, rendering the experimental results directly comparable. In Tab. 7 the fifth row shows the effectiveness of the proposed upper bound, where we use Lı as the discrepancy measurement without involving any pseudo-targel information, and improves the performance compared to MCD in the first row. The sixth row shows the results when it is replaced with the CMD. The last row shows the results of leveraging information from the psoudo-labeled target samples create a more reliable hypothesis space for fz. These results show that every part our proposal improves the performance. To verify the effectiveness of the proposed method further, conducted another ablation
25

---

Published in Transactions on Machine Learning Research (06/2023)
Table 8: Simple ablation study to demonstrate the contribution from each part of the proposal (V in the column "optimal joint error" means to include the joint error in the upper bound, in contrast to the training objective proposed by MCD; in the column "CMD" means to use the proposed discrepancy measurement for c, whereas indicates that use L1 norm alternatively; v in the column "pscudo-labels induced hypothesis space" means the hypothesis space for f2 is created based which is corresponding to our THS proposal in the main paper, where 17 * 1). We repeat the adaptation on VisDA dataset three times and record the average and the standard deviation of the accuracy.
pseudo-labels of target samples,
OPTIMAL
CROSS MARGIN DISCREPANCY
PSEUDO-LABELS
METHOD
JOINT
INDUCED
VIsDA
ERROR
HYPOTHESIS SPACE
BASELINE
X
71.9+0.44
73.1+0.3
X
V
76.220.3
V
76.8=0.2
OURS
X
73.4=0.3
V
V
76.4=0.3
X
V
79.210.2
81.6110.2
study on VisDA dataset. The results in Tab. 8 show a similar conclusion that each part of our proposal is beneficial for adaptation, despite a more complicated scenario.
26


Reviews:
[{"review_id": "QSTGaNyOhD", "reviewer": "Reviewer_rDS2", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The version has better organization and clearer explanations such as equations. No significant changes in experiments. \n\nThough this version addresses some of my concerns, I still feel confused about the training algorithm, e.g., Fig. 4. Is it possible to replace it with an algorithm listed in the paper (or along with the figure)? Also, I do not see a demo code for reproducing results, so I am suspicious that there may be some tricks in the implementation to achieve such performance. I strongly encourage the authors to release the code somewhere.  ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposed an unsupervised domain adaptation method through minimized joint error. This is a resubmission.\n", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See my comments in Weakness.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Kjc8QCqgds", "reviewer": "Reviewer_w1Z7", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- trying to minimize the Joint Error makes sense, and the proposed contribution is a reasonnable contribution in this direction\n- Experimental results are good\n\nWeaknesses\n- there are some unclear parts in the derivation of the method and the notations. See my comment below.\n- The claim that this paper is one of the first to try to 'directly minimize the joint error' (page 3) seems a little exagerated, and could be undertoned. Notably, I strongly encourage the authors to clearly present the differences with the work from Saito and colleagues [1]. Though this paper [1] does present a similar idea from a less theoretical perspective (though [1, Section 4] is enlighting), it seems to the reviewer that the overall approach is similar, and the differences with the proprosed CMD discrepancy should be more clearly stated. Howvere approaches, such as [2] or [3], should also be examined under the same lense.  \n\n[1] Asymmetric Tri-training for Unsupervised Domain Adaptation\nKuniaki Saito, Yoshitaka Ushiku, Tatsuya Harada Proceedings of the 34th International Conference on Machine Learning, PMLR 70:2988-2997, 2017. \n\n[2] CyCADA: Cycle-Consistent Adversarial Domain Adaptation\nJudy Hoffman, Eric Tzeng, Taesung Park, Jun-Yan Zhu, Phillip Isola, Kate Saenko, Alexei Efros, Trevor Darrell Proceedings of the 35th International Conference on Machine Learning, PMLR 80:1989-1998, 2018. \n\n[3] Bridging Theory and Algorithm for Domain Adaptation. Zhang, Yuchen & Liu, Tianle & Long, Mingsheng & Jordan, Michael. (2019).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a novel unsupervised domain adaptation (UDA) method which takes into account the joint error (how much a classifier can be effective on both source and target at the same time). While in most traditional approaches to UDA ignore this term due to intractability, the paper proposes to minimize an upper bound of this joint error, which relies on pseudo-labels obtained on target samples. Authors propose a novel cross-margin discrepancy, and show results and empirical evidences of effectivity on classical UDA datasets (digits, VisDA, Office-Home, etc.).", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Overall, while I acknowledge that the authors made some good efforts to provide a clear and illustrated exposition of their ideas, I must admit I had troubles understanding the core concepts as there are some notation issues. I listed some below, but at this stage and from the current state of the paper, I need to make too many assumptions about the meaning of the theoretical concepts to really assert the quality of the paper, and the novleties wrt. state-of-the-art. \n\n\nMinor comments:\n - on page 4, you write $H_f \\leq H$. What does it mean ?\n - Section 3.2, you write $g \\in H_g$. I am a little bit confused here, in wich sense is the feature extractor belonging to an hypothesis space ? Maybe the meaning of $H$ in your paper, though classical in the machine learning literature, should be more clearly specified ? \n - section 3.2.1, typo: 'the all the samples'\n - I had troubles understanding Figure 2 and the corresponding description in Section 3.2.2. I encourage the authors to work on this subsection.\n - Section 3.3. there is something unclear, if $f_1$ and $f_2$ are hypothesis function, with outputs are probability distributions over classes, how can we write $1-f_1$ or $\\text{log}(f_1)$ ?  ", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "DkTOKWBmMj", "reviewer": "Reviewer_n4XW", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n========\n- The proposed approach is derived from a bound that is more appropriate than previous ones in that the joint error term is taken into account\n- Thorough empirical investigation on the relevant datasets / benchmarks for domain adaptation\n- In some cases, the proposed method outperforms others\n\nWeaknesses\n===========\n- It’s not clear in practice how big a deal it is to ignore the joint error. While I agree that the scenario illustrated in Figure 1 is possible in principle, it’s not clear how often this actually occurs. It would thus be great to directly investigate whether the phenomenon illustrated in Figure 1 occurs in the baseline approaches used in the experiments, and whether (or how often) the ‘failure’ of those baselines (e.g. worse performance compared to the proposed method) is indeed associated with this phenomenon occurring.\n- The paper has a number of clarity issues (see below)\n- Some important aspects of experimental design are missing (from the main paper at least) that need to be discussed. \n- The proposed method is in several cases outperformed by other baselines\n- The proposed procedure seems complex in terms of stages of training (which involves several design choices to be made, e.g. which submodules are frozen/trainable in each phase) and has some key hyperparameters which isn’t clear how should be tuned\n- Some issues with soundness of claims (see below)\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper tackles the problem of domain adaptation, where we assume access to a labeled source dataset as well as unlabeled data from the target dataset that we want to ultimately perform well on. Revisiting the bound (for the target error) proposed in early seminal work in the field by Ben-David et al, the authors remark that most recent work on domain adaptation ignores the ‘joint error’ term and focuses on learning embeddings where the discrepancy between the marginal distributions of the source and target is minimized. The issue that can arise from this is that the two distributions are indistinguishable from each other in terms of features, but with different classes mixed together in embedding space. Based on this observation, they propose an upper bound that takes the ‘joint error’ into account. Their upper bound is given in terms of the optimal labeling functions for the source and target, which in practice we don’t have access to, so they relax it to replace those by a supremum, introducing a min-max flavor into their optimization problem, reminiscent of other adversarial approaches. Then, they tighten the bound by constraining the space of hypotheses from which the supremum can be taken over: one hypothesis space is constrained to be the set of hypotheses that work on the source domain (this is easier to characterize as we have access to source labels) while the other is constrained to be the set of hypotheses at the intersection of the sets that work well on the source and target domains (for the latter, pseudolabels are used, obtained by the source model). They provide a practical algorithm for the resulting min-max optimization that has 3 phases, optimizing different sets of parameters with different sub-parts of the objective. They perform a thorough experimentation on different datasets showing that their approach sometimes outperforms some baselines.\n", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- [Related work] The authors say that the problem of ignoring the joint error ‘remains unsolved’, but several works mentioned in the related work section seem to address this, so it’s not clear which aspect is actually still unsolved. For instance, the authors effectively dismiss some previous work because they say that it involves hyperparameters. But don’t most methods have that property (including the proposed one)?  \n\n- [Related work] On a related note, why don’t the authors compare experimentally against the similarly-motivated methods mentioned in the related work: (Sener et al. 2016), (Zhang et al, 2018) and (Wu et al, 2019), for example?\n\n- [Related work] On another related note, it would be great to elaborate more on the differences between the proposed method and this most related categories of works. The authors say they are the first to ‘directly’ minimize the joint error for UDA. Do other approaches minimize it indirectly then? If so, what are the benefits of the direct approach? Also, the authors say in the experiments that their approach is very similar to MCD. Please explain what exactly is the difference.\n\n- [Clarity / Motivation] It wasn’t clear to me why the hypothesis space H2 is defined as the intersection the hypothesis space of source models and that of target models (according to pseudolabeling). Why not define it as just the latter, rather than the intersection of the two? On a related note, Figure 2 shows the ‘improper constraint’ in subfigure a (by the way, I’m not sure what ‘improper constraint’ vs ‘alleviated constraint’ means. Which constraint exactly is this referring to? Why is one improper?) where the improper constraint is setting the hypothesis space H2 to be that of the source models. But a more intuitive choice would be setting it to be that of the *target* models. It would be great to clarify why this is the reference point used there. \n\n- [Clarity] A very prominent clarity issue is that the ‘joint error’ is at the forefront of the narrative of the paper (incl. the abstract, intro, etc) but it’s not actually defined (formally or intuitively) until very late (it’s defined the first time as far as I can tell only in page 3, *after* having presented the proposed upper bound even). It would really help readability to explain what exactly is meant by joint error.\n\n- [Clarity] Confusing notation in several places. For example, the error \\epsilon is defined as taking 2 arguments (corresponding to prediction and ground truth). The source error \\epsilon_S should then be defined as \\epsilon(h, f_S) rather than as \\epsilon_S(h, f_S) – currently, \\epsilon_S appears on the RHS of the equation that defines it. \n\n- [Clarity] Furthermore, the definition of \\epsilon_S says that it takes just 1 argument (the hypothesis h), because the second argument is fixed to be the optimal labeling function for the source. But then, several terms in Equation 1 use \\epsilon_S (and correspondingly \\epsilon_T for the target) with two arguments which is confusing.\n\n- [Clarity] Also, in Equation 1, it is not clear what \\epsilon_S(f_S, f_T) is versus \\epsilon_T(f_S, f_T) – did the authors instead mean \\epsilon(f_S, f_T) versus \\epsilon(f_T, f_S)? Notice the different order of arguments there. I.e. these quantities are the error on the target induced by using the optimal hypothesis on the source, and the other way around (error on the source if using the hypothesis that is optimal for the target). Is this what was intended? If so, please correct. If not, can you please explain?\n\n- [Clarity] In later equations, new symbols like \\epsilon_{S_g} start appearing too, without being defined (as far as I can tell; might be missing it). Same for C_{S_g, T_g}. Same for f_{1or2}. I understand the meaning here for all of these, but it’s best to be precise, and introducing all these symbols that the reader must search (or infer) the definitions for causes fatigue.\n\n- [Clarity] In Section 3.2.2 I find the text confusing. The authors talk about points being ‘outside’ of f2, for example. But f2 is a function. What does it mean to be outside a function?\n\n- [Clarity] ‘All experiments are performed in an unsupervised fashion’ – this sentence confused me. I thought labels from the source domain were used e.g. in Step A of training?\n\n- [Clarity] Add the missing ‘details’ regarding the pseudolabeling process. Are these computed online or offline? In stage 1 of training (Figure 4i), pseudolabels are used for the target loss that optimizes f2 (and g) – are the pseudolabels obtained from f1? These are learned jointly in this phase. At the start of training when f1 is random, is it possible that the produced pseudolabels are of very poor quality, which throws off the training of f2?\n\n- [Experiment design] How are hyperparameters tuned?  \\gamma and \\eta seem to be particularly important (but also standard hyperparameters like learning rate, when to stop training, etc). This is important to discuss in UDA since we assume no labeled validation set from the target domain (by definition of the problem setting). So, it’s important to disclose how model selection happens\n\n- [Experiment design] Related to the above, in some cases (e.g. in Table 2), the authors report several variants of their method (with different hyperparameters) and it seems that for different classes (columns in Table 2), different versions perform significantly better/worse than others. Ultimately when proposing a method though one needs to select a single model to deploy (we can’t decide based on the test set results which one to choose in each case…). So some of the results are showing that *there exists* a variant (i.e. setting of hyperparameters) that (sometimes) can perform well, without really offering a procedure for arriving at that variant.\n\n- [Experiment design] Why do different tables compare against different subsets of baselines?\n\n- [Soundness] The authors claim that their method sometimes outperforming others demonstrates that their proposed approach manages to penalize ‘the undesired matching between the source and target domain’. I don’t think this necessarily follows by simply looking at the numbers in the tables. Can’t there be several other reasons for the better or poorer performance of different models? It seems that the authors should design a separate experiment to investigate this in particular if they want to make this claim.\n\n- [Soundness] The authors claim that one of their contributions is showing that their proposed novel measurement of dissimilarity can ‘alleviate the instability during adversarial learning’. But this isn’t formally defined (instability in terms of what?) and I don’t see where this is shown. There are some plots showing faster convergence using the proposed measurement compared to others, which might relate to but isn’t the same as mitigating ‘instability’. \n\n- [Soundness] ‘Fig. 5b empirically proves simply minimizing the discrepancy between marginal distributions does not necessarily lead to a reliable adaptation’. This figure shows the marginal discrepancy over epochs, but I’m not sure how this relates to the quality of adaptation, and what evidence is used exactly to support this claim.\n", "broader_impact_concerns": "I have no broader impact concerns about this work in particular.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper is a re-submission of an earlier TMLR paper.  One of the three reviewers was also a reviewer on the previous submission, and two of the reviewers are new.  The previous reviewer is happy with the current version and recommended acceptance.  One of the two reviewers was willing to recommend acceptance after the latest revision, and the final reviewer still had concerns about the clarity of the paper, though they did not see the final revision (their recommendation came in before the final revision).

I took a look at the revision.  For the most part, it seems that this revision addresses the concerns of the negative reviewer.  However, while reading the paper I do agree with the general concern amongst the new reviewers that there are some clarity issues here, and they continue to persist.  I think at this point it's mainly a polishing issue, as there are some lingering grammatical issues.  For instance, in the first paragraph of the introduction, the first sentence is not entirely grammatically correct, nor is the fourth.  There are many such places in the text like this.  I think this paper needs a full polish to fix these issues, and to ensure that the notation and clarity is OK.  With a bit more polish this paper should be good to go.
Decision: Accept (poster)