Supervised Domain Adaptation Based on Marginal and Conditional Distributions Alignment
Authors: Ori Katz
Ronen Talmon
Uri Shaham
Venue: Accepted by TMLR ()

Abstract: Supervised domain adaptation (SDA) is an area of machine learning, where the goal is to achieve good generalization performance on data from a target domain, given a small corpus of labeled training data from the target domain and a large corpus of labeled data from a related source domain. 
In this work, based on a generalization of a well-known theoretical result of \citet{ben2010theory}, we propose an SDA approach, in which the adaptation is performed by aligning the marginal and conditional components of the input-label joint distributions.  
In addition to being theoretically grounded, we demonstrate that the proposed approach has two advantages over existing SDA approaches. First, it applies to a broad collection of learning tasks, such as regression, classification, multi-label classification, and few-shot learning. Second, it takes into account the geometric structure of the input and label spaces. Experimentally, despite its generality, our approach demonstrates on-par or superior results compared with recent state-of-the-art task-specific methods.

Published in Transactions on Machine Learning Research (07/2024)
Supervised Domain Adaptation Based on Marginal and Conditional Distributions Alignment
Ori Katz
orikats orikats@gmail.c com
Viterbi Faculty of Electrical and Computer Engineering Technion Israel Institute of Technology
Ronen Talmon
ronen Conennec..echtiion.coo www.wtto.ccciicccc.c.o.
Viterbi Facully of Electrical and Computer Engineering Technion - Israel Institute of Technology
Uri Shaham
wri.haham
Department of Computer Science
Bor-Iln University
Reviewed on OpenReview: http::://ppeeeeveee net/ forum? <d- ffBj12yh55
Abstract
Supervised domain adaptation (SDA) is an area of machine learning, where the goal is to achieve good generalization performance on data from a target domain, given small corpus of labeled training data from the target domain and a large corpus of labeled data from a related source domain. this work, based on a generalization of a well-known thcoretical result of Ben-David et al. (2010), we propose an SDA approach, which the adaptation performed by aligning the marginal and conditional components of the input-label joint distributions. In addition to being theoretically grounded, we demonstrate that the proposed approach has two advantages over existing SDA approaches. First, it applies to broad collection of learning lasks, such as regression, classification, multi-label classification, and few-shot learning. Second, it takes into account the geometric structure of the input and label spaces. Experimentallyy despite its generality, our approach demonstrates on-par or superior results compared with recent state-of-the-arl lask-specific methods. Our code available here.
1 Introduction
The Empirical Risk Minimization (ERM) principle, which is the theoretical basis of supervisod machine learning, is based the assumption that the training data is sampled from the data distribution that the model will encounter after deployment. Unfortunately, in practice, this assumption does not hold in many applications and tasks, and often significant discrepancies between the train and test distributions are prevalent. Mitigating these discrepancies is systematically addressed by domain adaptation (DA). In the literature, many contexts of DA are considered. Here, we focus on supervised domain adaptation (SDA) in the following sctting- Given large corpus of labeled data from a source domain and small corpus of data from
labeled data from larget domain, the goal is to facilitate learning with good generalization the target domain. This setting is highly suitable for scenarios such emerging domains (e.g. early stages of pandemic), rare events (e.g. natural disasters), costly data acquisition (e.g. remote sensing or certain types of environmental monitoring), and privacy concerns (e.g. in healthcare or finance sectors, where data privacy regulations may limit even unlabeled data use). These challenges necessitate effective SDA approaches.
In their seminal work, Ben-David et al. (2010) presented an upper bound on the generalization error on the target domain for binary classification and deterministic labeling functions. Here, we generalize this result by considering arbitrary supervised learning tasks, and (ii) by defining the domain as input-label

---

Published in Transactions on Machine Learning Research (07/)024)
joint distribution. The derived upper bound consists of two terms. The first term, often referred to as the covariate shift, conveys the discrepancy between the marginal distributions on the input space. The second term represents the discrepancy between the conditional distributions on the label space, and therefore, we term it as the cross-domain conditional agreement (CDCA) crror. The covariate shift has been widely investigated in the literature of unsuperoised DA (UDA) Pan et al. (2010); Long et al. (2018); SSn et al. (2017): Zhang et al. (2019b), and many of the existing algorithms for UDA attempt to minimize it. These methods aim to mitigate this shift by mapping the input space into a representation space of invariant features across domains. However, ensuring similarity in the conditional distribution of the label space given the mapped input space remains challenging Gong et al.] (2016). To face this gap, recent research in DA has also explored aligning conditional distributions. [Gong et al. (2016) proposed an unsupervised method that focuses on aligning conditional transferable components, considering a causal system in which labels are the cause for inputs. Under the assumption that the conditional distribution of the transferable components is invariant after a location-scale transformation, Gong et al.] (2016) devisod a UDA method for classification lasks. A similar approach, addressing the alignment of the conditional distributions, was proposed by RRchard et al. (2010), they derived a bound based on the
et al. 2021 J. Inspired by the bound proposed in conditional distribution of the input space given the considered hypothesis function and devised a method for finding a hypothesis function minimizing the derived bound. Considering the conditional distributions has been utilized in multi-source domain adaplation as well. For example, Heinze-Demi & Meinshausen (2021) addressed the challenges raised by domain shifts in image classification and introduced conditional variance penalties to enhance domain-shift robustness. Although this example is not directly related to conditional alignment, it underscores the broader relevance of considering conditional distributions into account in the DA landscape.
In contrast to UDA, existing methods for SDA are not driven by the theoretical bound of Ben-David et al. (2010), and both the covariate shift and the CDCA term have been overlooked. More specifically, the main paradigm of SDA methods, which are designed specifically for classification tasks, is to employ heuristics that aim to map samples from the same class to a single point, ignoring the distribution of the samples in the input space.
While our generalization of the upper bound has its own merit, we show that it introduces new SDA approach, in which the input space and the label space are decoupled. Specifically, based on the covariate shift and the CDCA error terms, we formulate the problem of SDA as multi-term optimization problem and propose to solve it using a Siamese neural network. In our experiments, we demonstrate the broad applicability of the proposed method to classification, regression, multi-label classification, and few-shot learning tasks, where we show on-par or superior results compared to recent slate-of-the-arl methods that are specific for each task.
The main contributions of this paper are follows. First, we present
approach to SDA that is theoretically grounded and widely applicable. Second, we extend the upper bound of the generalization error in the target domain to broader and more practical scenarios. Third, driven by the extended upper bound, we introduce a new loss term, and subsequently, a geometry-aware SDA method that is based on a Siamese neural network.
2 Related Work
The task of SDA lies in the interface of UDA, multi-task learning (MTT.) Zhang et al. (2019ap: Dakota et al. (2021), and multi-source domain adaptation (MSDA) [Sun et al. (2015), and is highly related to continual learning Tang al. (2021) and catastrophic forgetting (Toompson et al. (2019); ;; Li et al. (2022).
While UDA focuses on transferring knowledge solely between the input spaces (marginal distributions of inputs), in MTL the ultimate goal is to improve performance on multiple related tasks by sharing informa- tion essential to these lasks. Therefore, MTL focuses on the distributions of the label space, often assuming the same input space distribution. SDA leverages both the input space and the label space (i.e., the joint distributions of inputs and labels), allowing exploitation of the relatodness between source and target do- mains, even when the joint distributions slightly differ. Another highly related field is MSDA Sun et al. 2015). While MSDA can be applied to SDA problems by treating each source domain as separate task,
2

---

Published in Transactions on Machine Learning Research (07/2024)
SDA typically focuses on scenarios with single source domain and limited amount of labeled data in the target domain. This focus on few-shot learning from the target domain typically differentiates SDA from MSDA, often necessitating more complex machinery to handle data from multiple sources.
SDA is also related to continual learning Tang et al. (2021) and catastrophic forgetting Thompson et al. (2019); Li et al. (2022). In continual learning, the goal is to train a model on sequence of tasks while preserving the knowledge acquired from previous tasks. This is similar to SDA in the sense that both approaches aim to handle evolving data distributions. However, contimual learning typically deals with single domain and focuses on mitigating forgetting during training, whereas SDA addresses adaptation between two different domains.
Since our SDA approach shares principles with UDA approaches, we begin this section with a survey of UDA methods. Then, we focus on SDA techniques.
2.1 Unsupervised Domain Adaptation
The large majority of meth for UDA can be broadly divided into two paradigms- - instance-based hods and beature-based methods. The instance-based paradign is based on samples re-weighting. The core idea behind this paradigm is to approximate the target error using a weighted sum of source errors. One notable shortcoming of instance-based methods is that they assume large overlap between the supports of the source and target densities, and hence, are limited when the covariate shift. is relatively large Teshima et al.) (2020). In these cases, the second paradigm of casture-based methods becomes preferable. The common trait shared by methods in the feature-based paradigm is the application of a mapping function to the input features. Early methods in UDA attempt to minimize the covariate shift by finding a mapping from the target distribution to the source distribution Gooalan et al. (2011); Gong et al. (2012); Fernando et al. (2013). A different, yet related approach, which is often used for domain generaliza litataaiziiiin,, is to find a latent space in which the representation is domain-invanti invariant. [Long et al.) samples from the source domain are labeled, the labels are also taken into account. In the last decade, due to the emergence of deep neural networks, utilizing Siamese neural networks (NN) for domain adaptation has become the leading technique to incorporate the source labels, e.g. Tzeng et al. (2014); Sun el al. (011 7
(2013); Pan et al.) (2010). When the
Ganin & Lempitsky
(2015); Tzeng et al. [2017 2015); Long et al. (2018 2015
2017p. In some methods,
domain invariance is obtained by incorporating probability mcasures such as maximum mean discrepancy
(MMD) Tzeng et al. (2014), multi-kernel MMD Long el al. (2015), joint MMD Long el al. (2017), or even
second-order statistics [Sun et al. (2017). Recently, UDA has seen significant progress by incorporating (2017); 1; Long et al. (2018);
methods utilizing adversarial approaches, e.g., Ganin al. (2016); Tzeng al.
Zhang et al. (2019b). These methods leverage generative adversarial networks, where a generator aiming
to extract domain-agnostic features is pitted against discriminator aiming to distinguish between features
extracted from the source domain and features extracted from the target domain. This adversarial training
is carried out by a minimax optimization algorithm and encourages the generator to learn a shared feature
space where the domains become indistinguishable. This idea was first introduced in Ganin et al. generalized in TTenng et al. (2017), and paved the way for many techniques. For example, [ong et al.
(0016)
(2018)
proposed conditioning the generator on class labels, promoting better adaptation for classification tasks.
Particularly relevant to our work is Zhang et al. (201b)), who extended the theoretical result of BaviD et al. (010)) to multi-class classification and rotrodde the Margin Disparity Discrepancy (MDD), a novel measurement with rigorous generalization bounds, mcasuring the discrepancy between the source and target domains in terms of their classification margins. The results of el al. (2019b) established a theoretical foundation for adversarial UDA methods, bridging the gap between Ben-Davia et al. (2010)'s 's theory and a practical UDA algorithm.
2.2 Supervised Domain Adaptation
While minimizing the covariate shift is a compromise when no labels from the target domain are at hand, one could argue that same approaches for UDA can be employed in SDA in order to match the conditional distributions as well. However, when facing a small amount of data points from the target domain, this is unfortunately not the case. Therefore, existing methods for SDA employ other strategies. These strategies
3

---

Published in Transactions on Machine Learning Research (07/2024)
can be broadly divided according to the downstream task, or more specifically, according to the marginal distributions of labels. In recent years, substantial body of literature in SDA addressed classification tasks, while regression tasks have gained less attention. For regression, the main strategy is re-weighting, e.g.: Yao & Dorctto (2010); Pardoe & Stone (2010). Recently, in accordance with many UDA mothods, imployyng NNs combined with an adversaiial mechanism was proposed for the weights selection fd Mathelin ct al. p2021p In Teshima et al. (2020), the authors proposed a new SDA approach for regression tasks that does not fall into the previously mentioned paradigms, and is based on data augmentation, where the target dataset is augmented using the learned statistics from the source dataset. In classification, the main strategy can be viewed as feature-based approach. The main paradigm for SDA classification is contrastive- based, e.g., Motiian et al. (2017b The objective of these methods is to map samples from the same class to a single point while separating the points representing different classes. Note that in contrast. to UDA, these methods do not explicitly aim to minimize the covariate shift or the CDCA term. Moreover, their objective often contracts the within-class goometry and ignores the betwoen-class goometry. The second paradigm for SDA classification is geometry-based, e-g., Xu et al. (2019); Morsing al. (2021), ; in which locally geometry-aware loss is employed. For example, in Xu et al (2019) this is distance which is employed via stochastic neighborhood embeddingg while in Morsing et al. 2021 this is a modification of the Rayleigh quotient. Note that although being geometry-aware these methods do not aim to preserve between-class and within-slass geomet.ries.
3 Learning Setup and Problem Formulation
We adopt the formulation of Ben-David el al. (2010) with some modifications. Tet X' be an input space and y be the label space. We will use the terms label space and output space interchangeably to emphasize that our work is not restricted to discrete or categorical label spaces.
Definition 3.1 (Domain). A joint distribution P defined on the input-output product space X' x x is called domain. We assume that a domain P has a density function, denoted by P(x,y)
Definition 3.2 (Hypothesis and Error). A hypothesis is a function h: X' + y. The ertor of a hypothesis
h with respect to a domain P is (h) & ((a))~p E(h(z),)) y), where É y x y - R is a loss function that is typically determined by the specific downstream task.
Consider a source domain ps and a target domain pf, and consider N samples {(xg 2 } drawn from the source domain ps and N samples {(=F' w/ > drawn from the target domain pT, where vx),, lid p for {S, T3. Following the common practice, we assume that NT & SN Our goal is to find hypothesis h" such that: h² é argmin, Epr cpr(h). (h).
4 Bounding the Error of the Target Domain
Den-David et al. (2010) presented an upper bound of the target error of a hypothesis. Their bound is derived for doterministic labeling functions and binary output spaces. We generalize their result to (i) the definition of a domain as a joint distribution and (ii) arbitrary supervised learning tasks.
Let Q" denote the marginal distribution over the input space in domain E {S, T} equipped with the density function Q" = Jyey (a y) for any X. Let d(2", 2T) denote the total variation distance between the marginals, given by: d(QS, )T) supBes (Q$( - QTT(T))( where we define, with slight abuse of notation, Q" (B) = Jeen (z))) and B is the o-algebra of all measurable subsets in the support of Q$ and Q"¹ We define the pross-domale conditionals agreement (CDCA) error of domain E (S,T) by:
X"
E
E
(1)
3' d'-p2
where p denotes the conditional distribution of the output space given x E X in domain e E (S,T).
We further assume that there exists an upper bound, denoted by M, for the expectation of the loss M A max (sup,, 6ps (h), supa 6pr (h)). Note that, for example, if is the 0 loss, then M 1. We remark that
4

---

Published in Transactions on Machine Learning Research (07/2024)
for the Cross-entropy (CE) loss this assumption does not hold, however, since CE is used as surrogate for the intractable 0 - 1 loss, it does not affects the generality of this assumption.
Theorem 4.1. For any hypothesis function h X + y and loss function satisfying: 1e(z,y) e(z, 2)1 [e(y, 2)1, we have:
Cps (h) S eps + M ( - minn(x" ,x'}.
(2)
The proof is in Appendix A Note that the requirement on the loss function in Theorem is satisfied by frequently-used loss functions, such as the 0 - 1, LI, and L2 losses. CE does not generally satisfy this assumption. However, in certain cases involving densitics that result from CE minimization, it does [hhoe & Johnson (981).
The derived upper bound in equation 2 consists of three terms. The first term, Eps (h), is the error of the hypothesis with respect to the source domain. The second term, d(Q 25), is originated from the covariate shift between the marginal distributions of the input space. The last term, min(x" xx"), conveys the discrepancy between the conditional distributions of the two domains.
5 Proposed Approach
5.1 From the Theory to an Optimization Algorithm
At first glance, the bound in Theorem 4.1 may be viewed only as a theoretical guarantee for the adaptation capability between two domains. However, a closer look from an algorithmic viewpoint suggests a practical utility. Specifically, consider a (measurable) mapping function & from the input space X to some embedding space E. Let pm, E {S, be the joint distribution defined on the embedded-output space that is induced by the marginal pushforward distribution, i.e., the joint distribution of (ø(x), y), where (x, ~ P
Considering the embedded space & as an analog of the input space X' introduces another degree of freedom that depends on the mapping function é. Specifically, we can now use the upper bound in Theorem find hypothesis and a mapping function & that minimizes the following objective:
(Ø lit) argmin eps (h) + (()22))) + min (x ,x*) (0.h)
where Qr and X" denote the marginal pushforward distribution and the corresponding CDCA error for U E {5,T), respectively, and E + y is a hypothesis function defined on the embedded space.
Putting the optimization in equation 3 into practical use in the context of the considered SDA problem involves two elements. The first element is the approximation of the terms based on the available data at hand. A straightforward approximation of the error (p=(h) is given by: (p.(h) & 1 N, ((((((()))))))) uf). The approximation of the covariate shift, denoted by d(Os, ÖT), can be obtained by employing any UDA technique to the embedded training data via ø. For example, Long et al. (2015) approximated this term using the MMD distance between (d(=;))) and (d(=)))
The approximation of the term min XJ) is more challenging. By observing the definition of the CDCA error from Equation 4 we see that the approximation of x requires input-label samples (z. from the source domain, which are available, but also input-label samples (r,, where is sampled from the source domain Q but y' from the marginal distribution induced by the target domain pr. An analogous challenge is raised in the approximation of रT. We propose to address these challenges using kernel regression in Section 5.2
The second practical element is the incorporation of the available labeled data from the larget domain. So far, our focus was on the minimization of equation 3] which based on the upper bound of the hypothesis error in the target domain in the r.h.s. of equation 2 Importantly, in our SDA setting, a small number of input-output samples from the target domain are available. Although they are not sufficient for accurate meaningful learning, the available target samples facilitate approximation of the error: 6 (h) a
5

---

Published in Transactions on Machine Learning Research (07/2024)
Ni E ((h(6(x)))) vf). For any 0 a I 1, we can express the bound from eq. (2) as follows:
(h) = (1 - a) (h) + ocpr (h) $ (1 a))pp(h) (h) + a a(cps(h) (h) + Md(Q", ( + min(x , x²)).
which gives rise to the following optimization problem:
(0))")= = argmin(1 - =)) (h) + aléps (h) + align(ps, p))), (6.h)
(4)
where: align(Ps. ) d(@ ) + min (T)) In practice, the selection of the hyper-parameter a reflects the prior belief for the capability to estimate h from the available target samples. We remark that the proposed objective assumes the form of typical objective functions for SDA consisting of a source term, a target term, and an alignment term.
5.2 Implementation
We adopl a popular approach employed in DA and utilize a Siamese network architect.re Motiaa et al.] (2017b); Xu et al. (2019). We consider the last layer as the hypothesis function h, and the rest of the network is viewed as the mapping function & of the input data to an embedded space. During training, this network is simultancously fed with pairs of mini-batches from the source and target domains.
Let (Xx, )s), (XT, YT) denote mini-batches of n samples from the source and target training sets, re- spectively. We denote their mappings by o( o(x") E Read for
{S,T), where d is hyperparameter denoting the dimensionality of the output layer of ø. In this setting, the implementation of éps (h) and Epr (h) translates to the following loss term: L(h(o(X))) Y") + L(M(((x5). YS),, where L(X, Y) is an aggregation operator of the considered loss function P: L(X,Y) & E_,((X, Yi, and X; and Y, denotes the ith sample in X and Y. respectively. The implementation of the empirical covariate shift, d( )),, is given by LUDA(Ø(X X ()X)), where LUdA denotes the loss term of the chosen UDA technique.
In our experiments, we used the UDA method proposed in Sun et al. (017)), which attempts to minimize the covariate shift by aligning the second-order statistics of the source and target distributions. This method does not require hyperparameters tuning, is casy to apply, and according to a recent survey achieves state- of-the art performance on standard benchmarks (see Table in Preciado-Grijalee & Muthireddy (2021)). ). We note that we also tested other UDA techniques, e.g., MMD Tzeng et al. (2014) and the adversarial method from Tzeng et al. 2207)) which led to similar results.
The CDCA term in equation 4 is approximated by using kernel regression (KR) Watson 19644), aiming to generate labels from a certain domaín given the mapped features and labels of its cross- More specifically, let e denote the cross-domain of E {5,T),
use KR in order to generate ú that admits the distribution of Pe where x Q". Then we use these generated samples in order to generate pairs of (o(x). 9) to approximate x" The exact procedure is described in Algorithm [ We remark that the motivation for using kernel regression, rather than standard parametric estimator like a neural network, is to avoid overfitting due to the small number of available training samples from the target domain. This choice raises computational complexity and limitations, which we further discuss in Appendix D
Let p ". E {), denote the output of the application of Algorithm 1 ((X"), ((XF), and Y Then, the computation of the CDCA term is given by: m(Y L(Y YT)}.
To conclude, the proposed method includes a Siamese NN, as presented in Figure I During training,
the network is fed with pairs of labeled mini-batches: (XS, YS) and (XT, YT). The training objective,
analogous to equation is to minimize the following loss term:
I(h(o(xT)), ),,Y), YT) L(h(o(x))),, 55 + Laliga ((x)) ),YY' YS ((xx)) YT).
(5)
where the alignment term Latign; analogous to align (Ps,p0 ,TT) in equation
is given by:
Latign ()(x)), Y ,0(x)) LUDA( ((x)))(())))) min SS),L(Y? (TT)..
6

---

Published in Transactions on Machine Learning Research (07 (07/202) 2024)
bxx* S
X
L(h(o(X))), Y
Lalign (6(X Y ,6(x²), Y
Φ ((X²)
X
L(h(o(())) Y
Figure 1: schematic illustration of our proposed approach.
In practice, the terms in the proposed loss can be weighted according to hyperparameters tuning.
Algorithm Cross-domaie samples generation via KR.
Input: 1.
Two mapped features of bani-b one from each domain 6(X") Rdx" U (S,T). 2. The
labels for the mini-batch of the cross-domain Y
Output: A mini-batch sample Y where each y E Po admits the distribution of po, where x ~ Q"
Compute [. [(x'))) o(ff)ll,, for i,j I, n, where X2 denotes the ith row of X (corresponding to the ith sample in the batch).
Compute: K [i,j exp (-
where max; miny [r,] Da -(
3: Approximate y using dernel-based weighted average: Yo = (diag (Kr K Y where is a column vector of all ones.
6 Experimental Study
6.1 with commonly-used benchmarks for classification. We compare approach to state-of-tho-art (SOTA) methods for classification and show that our approach obtains par superior results. In Section 6.2 we focus scenarios that highlight the benefits stemmingg from laking the geometry of the input and output spaces into account. We present new datasets and learning tasks and demonstrate suporior results compared to the SOTA bascline methods used in Section 6.1 We complete this section with an evaluation of regression tasks Section 6.3 There, we show superior results compared to SOTA methods for regression, remonstrating the universality of our method and its broad applicability. Tn summary, throughout this section, we demonstrate ou-par superior results on multiple tasks compared to the SOTA methods tailored specifically for the respective task (i.e., classification or regression).
Our experimental study begins in Section
6.1 Classification on benchmark tasks
We consider three uommonly-used benchmark tasks for DA. The "Digits" task consists of two domains (datasets): the MNIST dataset LeCun et al. (2010), denoted by M, and the USPS dataset LeCun et al. (1989), denoted by U. The "Office" task (Saenko et al. (2010) consists of domains: Amazon, Webcam, and DSLR, denoted by A,W, and D, respectively. The "VisDA-C" task Peng et al. (2018) consists of two domains: synthetic domain of 3D rendered objects, denoted by S, and real domain of images-in-the- wild, denoted by R. An illustration of the datascts is presented in Figure 2 We compare our method to three state-of-the-art SDA methods. The first method, termed CCSA Motiian el al. (2017b), represents contrastive-baase approaches. The second method, d-SNE et al. 22019 represents geometric approaches. The third method, termed NEM Wang et al. (2019) combination of these two approaches. All of these methods share the same objective of mapping samples from the same class to single point while separating the points representing different classes. In addition to these methods, compare approach with its

---

Published in Transactions on Machine Learning Research (07 (204))
M
u
A
D
W
S
R
4
4
3
3
Figure 2: Illustraivve samples from the "Digits" (M and U), "Office" (A, W and D) and the "VisDA-C" (S and R) tasks.
ablated versions. In the first ablation, denoted by " "Ab-UDA", we discard the UDA loss term LUDA, and in the second ablation, denoted by "Ab-CDCA", we discard the CDCA term. For reference, we further report the results of three naïve baselines that include training using the source or target domain only (denoted by S-Only or T-Only, respectively) and simultaneously training on both domains (denoted by S+T).
In the first experiment, we consider the "Digits" tasks. We follow the same experimental protocol from Motiian et al. (2017b); Xu et al. (2019) (see details in Appendix C. We conduct several experiments, and in each experiment, we consider a different number of samples per class from the target domain. We repeat each experiment 10 times. In this experiment, 200 samples per class from the source domain and X samples per class from the target domain are randomly selected as the training set, where X E f1. 7... We note that in most real-world problems, the data from the source domain is considered to be unlimited. Still, for the sake of fair comparison, chose to adopt the common evaluation protocol employed by the considered baselines Motiian et al. (2017b); Xu et al. (2019). In the sequel, we conduct experiments where consider the entire source dataset for evaluation and observe similar results. The average results and their standard deviations are presented in Table [ We note that we reproduced the baselines' results, see Appendix B for details. Bold indicates the best results, and underline indicates the second-best. proposed approach achieves on-par results with the compared baselines. In addition, we see the contribution of each term in the proposed loss in equation 5 Specifically, we see that the contribution of the UDA and the CDCA terms is especially pronounced when considering fewer samples per class.
the table, we see that our
In the second experiment, we consider the "Office" tasks. We consider six DA tasks: A - A W.D D.D. D, + W, W + A and W + D. We follow the same experimental protocol from Motiian et al. (2017b); Xu et al. (2019). For the source domain, we randomly pick 20 samples per class from Amazon and samples per class from Webcam and DSLR. For the larget domain, we randomly pick 3 samples per class. For more details, see Appendix C. We repeat this procedure times, the average results and their standard deviations are presented in Table Same in Table D we see that our approach attains on-par results with the current SOTA methods.
In the third experiment, we consider a DA task from the synthetic domain (S) of the 3D rendered objects in "VisDA-C" to the real domain (R). We follow the experimental protocol from Xu et al. (2019). For more details, see Appendix C We conduct several experiments, and in each experiment, we consider a different number of samples per class from the target domain. We repeat each experiment 10 times, the average results and their standard deviations are presented in Table Here as well we see that our approach achieves on-par results with the current. SOTA methods.
6.2 Leveraging the intrinsic geometry of a dataset
In this section, we highlight the scenarios in which our method is expected to excel and validate it empirically, In the first experiment, we present a variation of the SDA task. We randomly pick some classos and discard their associated training samples from the target domain. For the remaining target classes, we take one sample per class. We term the target classes we discard "cold" target classes and this entire task zero-
8

---

Published in Transactions on Machine Learning Research (07/2024)
Table 1: Classification accuracy for the "Digits" tasks. In the upper part, we consider adaptation from M. In the lower part, we consider the opposite direction. In each column, we a different number of samples per class from the target domain.
5
# Best
S-Only
59.35 2.55
63.74 + 1.76
59.56 + 1.59
60.02 2.57
T-Only
46.26 + 0.98
65.24 2.13
76.85 1.79
82.73 + 1.26
S+T
77.40 = 1.31
85.99 + 1.44
87.35 1.36
91.68 - 0.78
CCSA
82.40 1.56
88.00 = 1.24
90.05 = 1.02
90.82 + 1.21
U
NEM
75.75 = 1.82
83.76 + 1.43
88.27 = 1.29
89.70 + 1.14
d-SNE
79.42 1.88
85.35 + 1.16
89.09 = 0.93
90.93 0.91
Ours
81.67 - 1.73
86.31 1 1.07
90.20 + 1.14
92.08 + 0.50
Ab-UDA
74.24 1.67
84.43 + 1.77
87.78 1.51
91.09 0.71
Ab-CDCA
78.43 = 1.80
85.60 + 1.38
89.38 = 0.91
90.84 1.04
S-Only
78.07 + 1.04
78.04 + 1.25
78.25 + 1.13
76.72 + 1.47
T-Only
61.87 1.58
76.46 + 1.63
81.18 + 1.41
84.41 1.28
S+T
82.41 1.50
$7.94 1.11
89.23 = 1.38
90.14 0.77
CCSA
85.11 1.23
87.51 + 1.15
89.60 0.94
90.35
M-U
NEM
84.38 = 1.22
87.16 + 1.32
89.88 = 1.01
90.54
d-SNE
84.39 1.42
89.04 + 1.14
89.42 1 1.05
91.34
Ours
85.85 + 0.87
89.46 0.90
90.80 1 0.67
91.52 0.88
Ab-UDA
83.09 = 1.48
87.67 + 1.34
89.46 = 0.69
91.05 + 0.83
Ab-DDCA
84.51 1.18
87.35 + 1.03
90.62 0.90
91.46 0.80
Table 2: Classification accuracy for the "Office" tasks. In each column, we consider a different task.
A- D
A - W
D - A
D - W
W + A
W -
# Best
S-Only
57.86 + 1.75
54.11 1.62
50.10 + 1.16
90.05 + 1.36
48.39 + 1.24
92.77 = 1.66
0
T-Only
82.03 1.00
81.38 1.52
59.14 + 1.46
79.74 1.45
59.26 + 1.53
84.38 1.73
0
S+T
82.81 1.35
1.17
66.44 + 0.85
93.02 1 0.61
64.51 0.91
97.27 1 1.00
1
CCSA
70.41
71.72
59.33 + 1.37
04.37=0.00
57.26 1.07
94.53 1.21
1
NEM
85.02 1..77 1.02
63.44
94.32 1.20
64.32
06.96+0.3
1
d-SNE
83.50 4: 2.25
84.58 + 1.52
65.52 0.99
94.06 + 0.77
64.59 1.33
96.35 - 1.01
0
Ours
87.56 1.65
83.50 = 0.82
65.56 + 0.94
93.08 1.05
66.28 + 1.40
97.42 + 0.75
3
shot SDA. We use the datascts from the "Digits" tasks as in Section 6.1 and follow the same experimental protocol. In this experiment, 200 samples per class from the source domain and one sample per class from the target domain are randomly selected as the training set. In addition, in each experiment, we consider a different number of "cold" target classes which then ablated from the training set. We remark that in the remainder of this section, the entire source datasets are utilized for the evaluation. For brevity, consider two representative baselines from Section 6.1 CCSA Motiian et al. (2017b), a contrastive-baset approach, and d-SNE Xu et al. (2019), a geometry-based approach. The results are presented in Table g First, comparing d-SNE to CCSA, we see a clear benefit incorporating geometric considerations into the model's design. Next, comparing our approach with d-SNE, we see that incorporating the marginals distribution alignment into the loss term attains even more significant improvements, espocially when considcring large number of "cold" classes. This improvement is evident both in the average accuracy and in the standard deviations.
In the second experiment, we test a multi-label classification task in a zero-shot setting that is based on the "Digits" datasets. We consider an input space that consists of a concalenation of digit slots, where each slot can either contain an instance of a digit or be blank. The associated output space consists binary indicator vectors, where the ith entry indicates whether the ith digit appeared in the concatenated image. The concatenated images in the source domain consist of MNIST digits and the concatenated images in the farget domain consist of USPS digits. In addition, the source domain admits zero-shot setting, where the
9

---

Published in Transactions on Machine Learning Research (07/2024)
Classification accuracy for the "VisDA-C" task. In each row, we test a different number of samples per class from the target domain.
Table 3:
S-Only
T-Only
S+T
CCSA
NEM
dSNE
Ours
10 40.55 1.14 53.98 = 1.47 60.64 1 1.36 48.43
1.46
61.88 1 1.33
60.54 1 1.05 62.90 + 1.13
15 38.89
1.24 60.14
1.35
62.76 1 0.77
47.43
0.99
64.66 + 0.94
63.71
1 1.20
64.39 1.07
20 35.12 + 1.12 62.62 = 0.71 66.26
1 0.73 49.55 0.86 65.80 + 0.80 65.41 1 0.98 66.55 + 1.22
Table 4: Classification accuracy for zero-shot DA task. In each column, we consider a different proportion (number) of "cold" classes.
70% (7)
50% (5)
20% (2)
CCSA
72.67 = 5.68
75.29 + 2.58
77.19 5.02
d-SNE
73.41 = 4.33
78.09 + 4.23
82.99 2.83
Ours
77.24 1: 2.25
81.36 + 1.26
84.00 + 2.35
training set contains only images that are associated with 1-hot output vectors, i.., they consist of a single digit. For the training set from the target domain, we follow a similar approach. We randomly pick 32 labels, and for each label we randomly generate 3 training datapoints. An illustration of these domains is presented in Figure 3(a).
We repeat this experiment 10 times and report in Table 5 three evaluation metrics for this multi-label classification: F1-Score, AUC, and the Hamming distance between the predicted indicator vectors and the true indicator vectors. We see that our approach obtained superior results compared to CCSA and d-SNE. We note that for fair comparison we adapted both CCSA and d-SNE to multi-label classification. See Appendix C for details.
In the third experiment, we consider a colored version of the "Digits" tasks. Specifically, we replace the original single-digit grayscale images with colored versions of multi-digit images. Fach image is a (1+ 1% N%1)) 1)- digit number, where No denotes the number of colors in the color palette. For example, for No = 3, we get 2-digit images. The digits in the image are randomly sampled from the corresponding domain (MNIST or USPS). The color of each image is determined by the [% most significant digits in the image, and the label set according to the presented number the image. The resulting dataset consists of 10.N, classes, divided into No clusters dominated by the color of the images. In Figure 3(b) we present an illustrative t-SNE embedding Van der Maaten & Hinton (2008) of a colored version of the MNIST dataset for No = Observing Figure 3 b),
that the input space geometry is indeed dominated by the colors of the images. We repeat the same experiment in Section 6.1 but for DA task from colored MNIST to colored USPS. We expect our approach to capture the geometry of the output space, in contrast to the competing methods that map input data of the same class to a single point. This may be conveyed in our approach by mapping input data with the same color but different digits to ncarby locations in the embedded space, thereby facilitating fewer errors in predicting the true color. Therefore, in addition to the accuracy measure, we report the proportion of partial errors, where the color of the predicted classes different than the color of the true class. The results are presented in the table in Table 5
In the upper part of the table, we show the obtained accuracy for N. 10, 20,30. We see that the geometric-aware methods (ours and d-SNE) obtain superior results compared to the contrastive-based ap- proach (CCSA). At the lower part of Table we present the proportion of out-of-color (OOC) errors. We see that the geometric-aware methods (ours and d-SNE) manage to yield significantly lower 00C errors.
6.3 Regression
We demonstrate the universality of our method and apply it to regression tasks. In contrast to classification tasks, the best of our knowledge, there no definitive benchmark for SDA in regression tasks. Here, we consider two datasets that, were used in two recent. SDA methods for regression.
10

---

Published in Transactions on Machine Learning Research (07 (07/2024) /024)
0
3
6250
G 5
5
6
2 0
3
0
3
6
25
1
/ 3
(b)
Figure 3: (a) Illustration of the images from the multi-label classification task. The images on the left (right) side are associated with the source (target) domain. (b) An illustrative t-SNE embeddings of the raw datapoints from a colored MNIST dataset for N₂ = 3. Each image a 2-digit number, where the color is encoded the tens digit.
Table 5: F1-Score, AUC (higher better), and Hamming distance (lower is better) for the multi-label DA task.
F1-Score (t)
AUC (t)
Hamming (L)
CCSA
0.22 0.25
0.55 0.16
0.46 + 0.10
d-SNE
0.69 0.15
0.73 0.10
0.31 + 0.08
Ours
0.74 + 0.15
0.80 + 0.16
0.26 + 0.15
The first experiment is taken from de Mathelin et al. 2021 and is based on the CityCam vehicle counting dataset Zhang et al. (2017). This dalaset consists of images acquired by traflic cameras, where each image is annotated with a label indicating the number of vehicles in certain range of interest. We follow the same experimental protocol in de Mathelin al. (2021) where the authors consider images from 4 cameras: two located on highway and two located at intersection. The images from one of the intersections are considered as the target. domain, and the rest are considered the source domain. For more details, see Appendix C We consider 20, 50, 100 and 200 samples from the target domain and compute the obtained mean-absoluto-orror (MAE). In Table 7 we present our results along with the results from de Mathelin et al. h202 We see that our approach outperforms classical approaches such as TrAdaB Pardoe & Stone (2010), KLIEP Sugiyama al. (2077), and KMM Huang al. (2006), while obtaining competitive results compared to the recent WANN de Mathelin et al. 2021); especially when fewer samples from the target domain are available.
The second experiment is taken from Teshima et al. 20200. and it is based on gasoline consumption dataset (William 22008,, p.284, Example 9.5). This dataset consists of tabular data describing gasoline usage in 18 of the OECD countries over 19 years. We follow the same experimental protocol as in Teshima et al. (0020)
times. Fach time, one country considered as the target domain, and the rest are considered as the source domain. The evaluation metric considered in Teshima al. 20200
where the experiment repeated
is normalized version of the mean square error (NMSE). For more details, Appendix C In Table
present results along with the results from Teshima et al. (2020). For convenience, in addition to
the evaluated metric from Teshima et al. (2020), on the right side of Table S we present the performance ranking obtained by each method for each larget country. We see that our approach performs betler for most of the domains as well as on average.
7 Conclusions
In this paper, we proposed new approach for SDA. We presented an upper bound of the generalization error in the target domain, which extends the setting considered by Ben-David et. al. (2010). Based
11

---

Published in Transactions on Machine Learning Research (07/2024)
Table 6: Adaptation results of the colored MNIST to the colored USPS. each column, we consider a different number of colors Ne. In the upper part, we show the accuracy obtained by cach method, and in the lower part, we present the proportion of out-of-color errors.
10
20
CCSA
77.49 + 0.84
74.90 2.53
68.86 1.76
Acc (t)
d-SNE
75.55 + 1.72
75.22 + 1.25
71.71 + 2.81
Ours
78.54 + 1.74
80.45 - 0.69
80.71 1.14
CCSA d-SNE
0.67 + 0.52
7.95 + 1.73
25.07 + 4.68
OOC (4)
0.73 + 0.17
5.33 + 1.41
16.14 + 3.70
Ours
0.23 + 0.17
1.32 + 0.45
5.67 + 0.92
Table 7: MAE for the CityCam vehicle counting dataset. In each row, we consider different mumber of samples from the target domain.
TrAdaB.
KLIEP
KMM
WANN
Ours
20 3.30 + 0.18 3.05 + 0.48 2.79 + 0.16
2.79 + 0.22
2.67 + 0.30
50 2.96 + 0.13 2.60 + 0.15 2.52 + 0.10
2.48 + 0.08
2.41 + 0.19
100
2.52 0.16
0.1 2.32
0.07
2.26 + 0.04
2.21 + 0.32
200 2.28 + 0.11 2.07 = 0.07 2.06 + 0.09
1.98 0.07
2.05 = 0.23
Table 8: Results for the gasoline dataset. In each row, different country is considered the target domain. On the left side, we present the Normalized MSF as presented in Teshima et al. (2020) On the right side, we present the ranking of the methods (lower is better).
Target Country
Target Comnty
T-Only
S-Only
S+T
CMTra
Ours
T-Only
S-Only
S+T
CMTra
Ours
AUT
5.88
11.00
9.67 =0.37
9.84 20.62
5.39
#1.86
2.89 + 1.47
AUT
3
5
2
BEL
10.70 +7.50
8.19 20.68
9.48 #0.91
7.94 +2.19
3.36 + 1.48
BEL
5
2
CAN
5.16 +1.3
157.74 48.83
156.65
10.69
3.84 10.98
15.29 + 1.03
CAN
2
4
1
3
DNK
3.26 10.61
30.79 20.93
28.12 1167
3.23 10.63
1.97
DNK
2
1
FRA
2.79
4.67 =0.41
3.05 #0.11
1.92 +0.66
1.20 2.067
FRA
3
2
DEU
16.99 18.01
229.65 19.13
210.59 11.99
11.23
30.71
1.51
DEU
GRC
3.80 =2.21
5.30 +0.90
5.75 20.68
3.55 11.79
1.85
GRC
3
2
IRL
3.05
00.4
135.57 +5..44
12.34 +0058
4.35 +1.25
82.22 +
IRL
3
2
4
ITA
13.00 1:1.55
35.29 1.83
39.27 +2.52
14.05 +1.81
4.11
ITA
2
4
5
3
JPN
44.67 10.35
8.10 +1.05
8:38 +1.07
12.32 4-4.55
0.48 0.61
JPN
3
5
NLD
3.75 +0.80
+0.05
3.87 +0.79
NOR
2.70 +0.51
1.86 +0.29
20.11
2.82 +0.73
80.66 0.59
NOR
ESP
5.18 +1.05
5.17 +1.14
4,29 +0.72
6.09 +1..33
0.54 1 0.48
ESP
SWE
6.44 +2.66
2.48 +0.23
2.02 +0.21
5.47 +2.63
1.42 + 1.05
SWE
5
2
CHE
3.51 +0.16
43.59 1.77
7,48 +0.49
2.90 +0.37
1.93 + 1.23
CHE
3
4
2
TUR
1.65 10.47
1.22 +0.18
0.91 +0.09
1.06 10.15
6.30 9.98
TUR
2
5
5.95
+1.86
15.92
10.05
USA
+1.96 1.98
21.53 +3.30
12.28 +2.52
1.60 +0.42
18.41 3.220
USA
#Best
Mean
3.33
2.41
this bound, formulated the task of SDA optimization problem using Siamese neural network. Experimental results demonstrate the effectiveness and broad applicability of the proposed approach for various classification and regression tasks.
optimization problem. We proposed solve this
We hope this work will spark interest employing the principles of (Ben-David et al. 2010 to SDA tasks. future work, we will extend the proposed approach for semi-supervised domain adaptation. This a natural extension of the proposed approach that follows the decoupling between the input and labels distributions in the derived bound.
12

---

Published in Transactions on Machine Learning Research (07/2024)
Acknowledgments and Disclosure of Funding
We thank the editor and the reviewers for their important comments and suggestions. The work of OK and RT was supported by the European Union's Horizon 2020 research and innovation programme under grant agreement No. 802735-RRG-DIFFOP RT acknowledges the support of the Schmidt Career Advancement Chair in Al.
13

---

Published in Transactions on Machine Learning Research (07 2024)
References
Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan. A theory of learning from different domains. Machine learning, 79(1):151-175, 2010.
Daniel Dakota, Zeeshan Ali Sayyed, and Sandra Kübler. Bidircctional domain adaptation using weighted multi-task learning. In Proceedings of the 17th International Conference on Pursing Technologies and the IWPT 2021 Shared Task on Parsing into Enhanced Universal Dependencies (IWPT 2021), pp. 93 105, 2021.
Antoine de Mathelin, Guillaume Richard, François Deheeger, Mathilde Mougeot, and Nicolas Vayatis. Ad- versarial weighting for domain adaptation in regression. In 2021 IEEE 33rd International Conference on Tools with Artificial Intelligence (ICTAI), pp. 49 56. IEEE, 2021.
Basura Fernando, Amaury Habrard, Marc Sebban, and Tinne Tuytelaars. Unsupervised visual domain adaptation using subspace alignment. In Proceedings of the IEEE international conference computer vision, pp. 2960-2967, 2013.
Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In Interna- tional conference on machine learning, pp. 1180-1189. PMLR, 2015.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Nomain-adversariin training of neural networks. The journal of machine learning research, 17(1):2096 2030, 2016.
Boqing Gong, Yuan Shi, Fci Sha, and Kristen Grauman. Geodesic flow kernel for unsupervised domain adaptation. In 2012 IEEE conference on computer vision and pattern recognition, pp. 2066-2073 IEEE, 2012.
Mingming Gong, Kun Zhang, Tongliang Liu, Dacheng Tao, Clark Glymour, and Bernhard Schölkopf. Domain adaptation with conditional transforable components. In International conference on machine learning, pp. 2839 2848. PMLR, 2016.
Raghuraman Gopalan, Ruonan Li, and Rama Chellappa. Domain adaptation for object recognition: An unsupervised approach. In 2011 international conference on computer vision, pp. 999-1006. IEEE, 2011.
Lukas Hedegaard, Omar Ali Sheikh-Omar, and Alexandros Iosifidis. Supervised domain adaptation: A graph embedding perspective and a rectified experimental protocol. IEEE Transactions on Image Processing, 30:8619-8631, 2021.
Christina Demn and Nicolai Meinshausen. Conditional variance penalties and domain shift robustness. Machine Learning, 110(2):303 348, 2021.
Jiayuan Huang, Arthur Gretton, Karsten Borgwardt, Bernhard Schölkopf, and Alex Smola. Correcting sample selection bias by unlabeled data. Advances in neural information processing systems, 19, 2006.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6900 2014.
Shinya Kobayashi, Yukinari Ota, Yayoi Harada, Ayataka Ebita, Masami Moriya, Hirokatsu Onoda, Kazu- toshi Onogi, Hirotaka Kamahori, Chiaki Kobayashi, Hirokazu Endo, el al. The jra-55 reanalysis: General specifications and basic characteristics. Journal of the Meteorological Society of Jopan. Ser. 1I, (3(1):5-44) 48, 2015.
Yann LeCun, Bernhard Boser, John Denker, Donnie Henderson, Richard Howard, Wayne Hubbard, and Lawrence Jackel. Handwrilten digit recognition with a back-propagation network. Advances in newral information processing systems, 2, 1989.
Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. att labs, 2010.
14

---

Published in Transactions on Machine Learning Research (07/2024)
Dingcheng Li, Zheng Chen, Eunah Cho, Jie Hao, Xiaohu Liu, Fan Xing, Chenlei Guo, and Yang Liu. Over- coming catastrophic forgetting during domain adaptation of seq2seq language generation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 5441-5454, 2022.
Mingsheng Long, Guiguang Ding, Jianmin Wang, Jiaguang Sun, Yuchen Guo, and Philip S Yu. Transfer sparse coding for robust image representation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 407-414, 2013.
Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan. Learning transferable features with deep adaptation networks. In International conference on machine learning, pp. 97-105. PMLR, 2015.
Mingsheng Long, Han Zhu, Jianmin Wang, and Michael I Jordan. Deep transfer learning with joint adapta- tion networks. In International conference on machine learning, pp. 2208-2217. PMLR, 2017.
Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael Jordan. Conditional adversarial domain adaptation. Advances in neural information processing systems, 31, 2018
Lukas Hedegaard Morsing, Omar Ali Sheikh-Omar, and Alexandros Iosifidis. Supervised domain adaptation using graph embedding. In 2020 25th International Conference on Pottern Recognition (ICPR), pp. 7841 7847. IEEE, 2021.
Saeid Motiaan, Quimn Jones, Seyed Iranmanesh, and Gianfranco Doretto. Few-shot adversarial domain adaptation. Advances in neural information processing systems, 30, 2017a.
Sacid Motiian, Marco Piccirilli, Donald A Adjeroh, and Gianfranco Doretto. Unified deep supervised domain adaptation and generalization. In Proceedings of the IEFE international conference. on computer vision, pp. 5715 5725, 2017b.
Sinno Jialin Pan, Ivor W Tsang, James T Kwok, and Qiang Yang. Domain adaptation via transfer component analysis. IEEE transactions neural networks, 22(2):199 210, 2010.
David Pardoe and Peter Stone. Boosting for regression transfer. In ICML, 2010.
Xingehao Peng, Ben Usman, Neela Kaushik, Dequan Wang, Judy Hoffman, and Kate Saenko. Visda: A synthetic-to-real benchmark for visual domain adaptation. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognation Workshops, pp. 2021-2026, 2018.
Alan Preciado- Preciado-Grijalvva and Venkata Santosh Sai Ramireddy Muthireddy. Evaluation of deep neural network domain adaptation techniques for image recognition. arXiv preprint orXiv:2109. arX1::2100.33420, 2021.
Guillaume Richard, Antoine de Mathelin, Georges Hébrail, Mathilde Mougeot, and Nicolas Vayatis. Unsu- pervised multi-source domain adaptation for regression. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2020, Ghent, Belgium, Seplember 14-18, 2020, Pro- ceedings, Part I, pp. 395 411. Springer, 2021.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211-253) 2015.
Kale Saenko, Brian Kulis, Mario Fritz,, and Trevor Darrell. Adapting visual calegory models to new domains. In European conference on computer vision, pp. 213-226. Springer, 2010.
John Shore and Rodney Johnson. Properties of cross-entropy minimization. IEFF Transactions on Infor- mation Theory, 2744)4472-482, 1981.
Masashi Sugiyama, Shinichi Nakajima, Hisashi Kashima, Paul Buenau, and Motoaki Kawanabe. Direct importance estimation with model selection and its application to covariate shift adaptation. Advances in neural information processing systems, 20, 2007.
15

---

Published in Transactions on Machine Learning Research (07/2024)
Baochen Sun, Jiashi Feng, and Kate Saenko. Correlation alignment for unsupervised domain adaptation. In Domain Adaplation in Computer Vision Applications, pp. 153 173-171. Springer, 2017.
Shiliang Sun, Honglei Shi, and Yuanbin Wu. A survey of multi-source domain adaptation. Information Fusion, 24:84-92 92, 2015.
Shixiang Tang, Peng Su, Dapeng Chen, and Wanli Ouyang. Gradient regularized contrastive learning for continual domain adaptation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 2665-2673, 2021.
Takeshi Teshima, Issei Sato, and Masashi Sugivama. Few-shot domain adaptation by causal mechanism transfer. In International Conference on Machine Learning, pp. 9458 9469. PMLR, 2020.
Brian Thompson, Jeremy Gwinmup, Huda Khayrallah, Kevin Duh, and Philipp Kochn. Overcoming catas- trophic lorgetting during domain adaplation of neural machine translation. In 2019 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pp. 2062 2068. Association for Computational Linguistios, 2019.
Eric Tzeng, Judy Hoffman, Ning Zhang, Kate Sacnko, and Trevor Darrell. Deep domain confusion: Maxi- mizing for domain invariance. Xiv preprinl ar arXiv:1412.34 2014.
Eric Tzeng, Judy Hoffmman, Trevor Darrell, and Kate Saenko. Simultaneous deep transfer across domains and tasks. In Proceedings of the IEEE international conference on computer vision, pp. 4068-4076, 2015.
Eric Tzeng, Judy Hoflman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7167-7176, 2017.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
Zengmao Wang, Bo Du, and Yuhong Guo. Domain adaptation with neural embedding matching. IEEE transactions on neural networks and learning systems, (1())23877-2377, 2019.
Geoffrey S Watson. Smooth regression analysis. Sankhyā: The Indian Journal of Statistics, Series A, pp. 359 372, 1964.
Yandong Wen, Kaipeng Zhang, Zhifeng Li, and Yu Qiao. A discriminative feature learning approach for deep face recognition. In European conference on computer vision, pp. 499-515. Springer, 2016.
H. Greene William. Econometric Analysis. 6. Business Publishing: Donna Battista, 2008.
Xiang Xu, Xiong Zhou, Ragav Venkatesan, Gurumurthy Swaminathan, and Orchid Majumder. d-sne: Do- main adaptation using stochastic neighborhood embedding. Proceedings of the IEEE/CVE Conference on Computer Vision and Pottern Recognition, pp. 2497-2506, 2506, 2019.
Yi Yao and Gianfranco Doretto. Boosting for transfer learning with multiple sources. In 2010 IEEE computer society conference on computer vision and pattern recognition, pp. 1855-1862. IEFE, 2010.
Jing Zhang, Wanqing Li, and Philip Ogunbona. Unsupervised domain adaptation: A multi-task learning- based method. Knowledge- Based Systems, 186:104975, 2019a.
Shanghang Zhang, Guanhang Wu, Joao P Costeira, and Jose MF Moura. Understanding traffic density from large-scale web camera data. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5898 5907, 2017.
Yuchen Zhang, Tianle Liu, Mingsheng Long, and Michael Jordan. Bridging theory and algorithm for domain adaptation. In International Conference on Machine Learning, pp. 7404-740 7413. PMLR, 2019b.
16

---

Published in Transactions on Machine Learning Research (07 (07/2024)
Appendix
A Proof for Theorem 4.1
Proof. Recall the definition for the error of a hypothesis: ep(h) - E. e(h(z). v), which we can recast
the error as (phh) E E E(h(z), y). In addition, let (-) denote the density function of the marginal xEQ
distribution Q" for v E (S, T) Then, we have:
epr ch) tpr (h) eps (h) (1)
E ((h(z),))
E ((h(r,,)
(6)
ZEQ DEPT
xEQ DEPT
S <pss(h) +
E xEQ*
E ((h(z),
Eps + cpr (h)
E
E
((h()),))
(7)
yEP?
ZEQ
DEPT
= ch) +
E
E ((h(z).y)
E SEQSW'EP?
E
((h(z).
+
#EO$
yEP!
E
E ((h(z).))
- E
E
E(h(x),))
(8)
xEQT VEPT
SEQ SEPT
= Eps(h) (h) +
E
E [(h(z).)) e(h(z))
ZEQS V'-P2
9~PS
E
E {(M(z),1) - E E E(h(z),))
(9)
xEQ VEPT
=E VEPT
S (p) +
E
E
[((h(z),)) v) ((b(z),v)))+
y-PS
y VNp!
Q (x)
E. {(h(z).v) - -Q"(x) E
e(h(z).v) dz
(10)
VNPT
WNPT
8EX
Eps (h)
E
E [e(h(z), v) ((h(z).8))+
y~PS 1/ WNPT
Q (x: Q2"(z) E [(h(z), v)dx yuPF
(11)
xEX
(lu.v') + M / 1Q0 (2)))))((a)))
S Eps (h) +
E
E
dr
(12)
yup8
xEX
8 Eps (h) + + Md 2*).
(13)
where the transition from equation 11 to equation 12 is due to the assumption: [((x, -(r,,))((xx 2) S 18 2)
and the upper bound of the expected loss function M. Repeating the same derivation, but replacing 6ps (h) by Epr (h) in the r.h.s of equation results in similar bound: (h)(h) I Epe (h) + x" A Q³).
implying that:
Epr (h) S eps (h) + Md(eS, ΦT) - mmnnfx S,xT.
17

---

Published in Transactions on Machine Learning Research (07 (204))
B Baselines Reproducibility on Benchmarks Dataset
In our experimental study, we faced difficulties reproducing the results on the benchmark datasets of "Digits" and "Office" of the two most recognized methods for SDA: CCSA Motiian et al. 2017bb and dSNE Xu et al. 2019h. These difficulties stem from the following reasons:
When considering few-shot learning scenarios the randomly picked training datapoints from the target domain can highly affect the results. Therefore, common practice is to repeat each experiment multiple times. This is indeed the practice that appears in the source code published by the authors of CCSA D However, in their code, the model parameters are not initialized between each experiment. As a consequence, the model accumulates knowledge from previous experiments. Note that this is a known issuc 2 which the authors have not yet addressed. Similarly, the authors of dSNE published their code However, we, among others, could not reproduce their result:
Known implementation issues:
Backbone inconsistencies: In recent years, many DA methods are based on Siamese neural networks. The choice of the backbone, its architecture, and whether is pre-trained or not, highly affect the results (for example, see the bottom line in Table in Hedegaard al. (2021)). Therefore, in order to fairly compare DA techniques, it is crucial to perform the evaluation with the same backbone. We note that in the literature, there are works that do not follow this guideline. For example, in dSNE, for the "Digits" datasets, instead of using the same backbone as in Motiaan et al.) (2017b a a close look at the code reveals that LeNet++ was used Wen et al. (2016).
Datasets: At first glance, the protocol for preparing the benchmark datasets is forward,, described in Section 6 and in more detail in Appendix C
However, the following aspects are often
overlooked:
Augmentations. Datasets angmentations can highly improve the model's ability to generalize, especially when ovaluating the ability of a model that was trained on catalog images (Amazon) to classify images from the wild (DLSR or Webcam). Similar to the backbone, a fair comparison between methods should rely on the same augmentations protocol.
Parrtiiioning Although the train-test partitioning is clearly defined, it is not clear how to construct the validation set. For example, in the extreme case of 1 sample per target class, choosing the validation set is not straightforward. Inspecting the code of recently published papers shows that the prevalent practice to use the test set as validation. The absence of a validation set introduces a risk for model selection, especially when systematic hyperparameter tuning is conducted. This concern was recently raised by Hedegaard et al. (2021). ) where the authors proposed rectified experimental protocol and released Python package for benchmark datasets
1 This protocol is yet to be considered a slandard, however, in our experiments, we adopted their protocol and we wish to encourage others to do so as well. We note that this protocol only mitigates the difficulties that stem from the inability to properly define a validation set. Although the target datapoints in the validation set are not explicitly introduced to the model during training, they do affect the model's hyperparameters. Therefore, order to perform a fair comparison between different models is it important to ensure that the same computational budget used in the hyperparameter tuning of each method.
Hyperparameters inconsistency: Similarly to the choice of the backbone, the selection of the hy perparameeeers can highly affect the model's performance. Therefore, in order to conduct a fair comparison between different models, is important to ensure that the same hyperparameters are tuned in each experiment. Otherwise, especially in the absence of validation sct, the experiment
l hattpp///ghaub.cm//aantttt/////
hhtpes//gghaabbmmm//ammtaaaaaa/ /CCSA/ ACSAA/SsSSS///////
5 http:///hhhbbcco////////////////
6 htttps: http://wg...nng//oppe//m/mmttttt
18

---

Published in Transactions on Machine Learning Research (07/2024)
results might reffect the quality of the hyperparameters tuning procedure rather than the quality of the evaluated model. Many works, however, do not follow this practice. For example, in Hedegaard et a.. (2021), the set of hyperparameters employed by the model was significantly larger than the oncs considered in Motiian et al. (2017b Xu et al. (2019). Morcover, the hyperparameters tuning protocol conducted by |Hedegaard et al. (2021) requires large computational resources.
Therefore, when considering the results obtained by a specific method in a specific experiment, we see a wide range of values reported different papers. The results for CCSA Motiian et al. (2017b) and dSNE Xu et al. (2019), as reported by different papers for the MNIST USPS experiment are presented Table We see that although the same method was evaluated the same benchmark, different. papers report significantly different results. A similar inconsistency was observed in the experiments related to the "Office" datasct, can be seen in Table 10 Note that the implementation of CCSA in Morsing et al. (2021) and in Hedegaard et al. (2021) deviates from the original one proposed in Motiaan et al. (2017b) and includes the explicit error term on the target domain.
In order cope with the inability to rely on prior results, we reproduced the baseline results using the same pipeline employed for all the methods.
Table 9: Experimental results for the MNIST to USPS experiment as reported by different papers (Table) in this paper). The results in the upper part are related to CCSA and the results in the lower part are related to dSNE.
Reported by
3
5
7
Motiian (2017b)
(Authors)
85.0
90.1
92.4
92.9
Reproduced by
85.11 + 1.23
87.51 + 1.15
89.60 + 0.94
90.35 + 0.71
CCSA
Morsing et al.
(2021
75.6 + 2.1
85.0 = 1.4
87.8 = 0.7
89.1 = 0.7
Hedegaard el a
89.1 +: 1.1
91.2 + 0.9
93.8 + 0.4
94.3 + 0.4
Xu et al. (2019)
Authors)
92.9
93.55
95.13
96.13
Reproduced by us
84.39 + 1.42
89.04 = 1.14
89.42 1.05
91.34 = 0.88
dSNE
Morsing et al. 2002
69.0 + 1.7
80.4 + 1.7
86.1 1.9
87.7 + 0.9
Hedegaard al.
2021
88.3 1.7
91.4 + 1.2 93.1 + 0.5
93.6 0.6
Table 10: Experimental results for the "Office" experiment as reported by different papers (Table 2 in this paper). The results in the upper part are related to CCSA and the results in the lower part are related dSNE.
Reported by
A
A W
D A
D + W
W A
W D
Motilan et al.
2017b
(Authors)
89.0
1.0
71.8
+
72.1 + 1.0
97.6 +
Reproduced
70.41 1..:1... 1.52
71.72 + 1.27
59.33 1.37
94.37 0.86
1.07
94.53 1.21
CCSA
Morsing et al.
2021
848421
875
66.5
0.7
6..01.6
98.6
Tedegaamm et al
202
2..4
84.5 2.1
65.5
97.5 0.9
60.8+:1.5
98.4 + 1.0
Xu ct al. 2019
Authors)
91.4
90.1 1.007
71.1 0.18
97.1 1.007
71.7
17 0.42
97.5 07.50..22
Reproduced by us
83.50 1.225
84.58 1 1.52
63.52
94.06 1 0.77
64.59 1: 1.33
06.35
1.01
dSNE
Moxsing et al.
202
86.5 - 2.5
88.7 + 1.9
65.9
97.6
63.9
99.0 0.5
Hedegaari et, al
202
84.7 + 1.3
82.3 + 2.4
65.1 +
98.2 + 0.4
59.9+11
99.7 - 0.4
C Technical Details and Additional Results
this section, we provide more details and additional results to the experimental study Section Our 0
code is publicly available here
C.1 Benchmark datasets
In these experiments, we follow the experimental setting from Motiian et al. (017b): Xu al. 2019 The
data partitioning is carried out. according to the rectified experimental protocol suggested in Hedegaard
19

---

Published in Transactions on Machine Learning Research (07 (024))
et al. (2021) using their released Python packages The hyperparameterss set includes the mini-batch size, optimizer selection (Adam Kingma & Ba (2014) SGD), learning rate, weight-decay, and the number of training epochs. Hyperparameters tuning was carried out manually based on the validation set for each experiment, and then, the same set of hyperparamcters was used for all the methods. The selected hyper- parameters for each experiment appear in our published code. The training loss these experiments the binary cross-entropy (BCE) loss. The CDCA term is applied by representing the labels as 1-hot vectors RC,, where C denotes the cardinality of the labels space.
C.1.1 "Digits" experiments
In this setting, each experiment is repeated 10 times. In each experiment, 200 samples per class from the source domain and X samples per class from the target domain are randomly selected the training set, where X E 1, The images from the MNIST datasct are resized to fit the size of the images from the USPS dataset. No data augmentation was applied. The network architecture consists of two convolutional layers with a kernel size of 3 x 3 and 32 filters followed by max-pooling layers and 2 fully connected layers of sizes 120 and 84. We note that this architecture deviates from the architecture reported in Motiian et al.) (2017b), but is taken from their published source code (see lines 94-98). We further emphasize that we used the same architecture for all the evaluated methods.
C.1.2 "Office" and "VisDA-C" C" experiments
In this setting, each experiment is repeated 5 times. each experiment, the samples for each class are sampled according to the quantities described Section G Tn line with the experimental protocol from Motiian et al. (2017b); Xu et al. (2019), for the "Office" tasks we perform the default data augmentations used in the Python package TLlb These augmentations are based 224 x 224. The network architecture consists of the convolutional layers of a VGG-16 model Kobayashi el al.) (2015) pre-trained on ImageNet Russakovsky et al. (2015). followed by 2 fully connected layers of sizes 1024 and 128.
random resized crop of size
C.2 Zero-shot, multi-label, and colored "Digits" experiment
For the zcro-shot experiment, we used the same architecture as in the "Digits" experiment. For the multi-label and colored experiment, we made subtle modifications to support the size of the modified image. Specifically, for the multi-label experiment, we modified the fully-connected layer located after the max-pooling layer from 1152 120 10368 x 120. For the colored experiment, we changed the first convolutional layer to support 3 (RGB) input channels. The training loss in the zero-shot and the colored digits experiments is the BCE loss, and in the multi-label experiment it is the sum of the BCE loss applied to each slot. The CDCA term is implemented by representing the labels as vectors in R,C, where C denotes the cardinality of the label space. In the zero-shot and the colored digits experiments, these vectors are 1-hot, whereas in the multi-label experiment, they may contain any binary representation.
C.3 CityCam experiment
We followed the experiment protocol from de Mathelin et al. (2021) The CityCam dataset is taken from Zhang et al. (2017) and publicly available Four cameras from the dataset were selected: "253", "511", "572", and "495". Cameras "253", "511", and "572" are considered the source domain, while camera "495" is considered the target domain. Each experiment
repeated 10 times. In each experiment, X datapoints from the target domain were randomly selected for training, while the rest were kept for testing, All the datapoints from the source domain are used for training. The images acquired by cach camera were pre-processed using & pre-trained 5e model. The pre-processing code can be found here 2 and the
Inttps//ppypi.oor/ppoeeee proppeettttiisssppsss
"hhttss//ggthuub.com//uue
haw.bboooommmmmm//aaaaatttte Twaaal///aaaaerrraaa Transfft-Learningg-Lirree
hnttps//wwww.dtycam-mmmi /dataset
20

---

Published in Transactions on Machine Learning Research (07 (07/2024)
processed dataset can be found here We used the same network architecture in de Mathelin et al. p202T, which consists of multi-layer perceptron with two hidden layers of size 100 and 10. The training loss is the MSE loss.
C.4 Gasoline experiment
We followed the experiment protocol from Teshima et al. 2020)) The Gasoline dataset is publicly available here [4] its parsed version is available here - The dataset consists of gasoline usage in 18 of the OECD countries over 19 years. Each record in the dataset describes the gasoline consumption in a certain year and country and contains four variables: per-capita income, gasoline price, stock of cars per capita, and gasoline consumption per car. The first three are considered as the predictor variable and the latter is considered the label. We follow the same procedure as in Teshima al. (2020), where the time-series structure (i.e., temporal order) is ignored, and instead the data is considered country, we perform the following procedure. We randomly pick 6 datapoints from their associated records to create the target training dataset, while the rest of its associated datapoints are used for testing. The source training dataset consists of all the datapoints from the rest of the countries. The training loss is the MSE loss. We use the same evaluation metric as in Teshima et al. (2020), which the MSE distance normalized by the closest label in the larget set (including the test. datapoints). We repeat this procedure 10 times with different train-test splits of target domain data.
i.i.d. samples for each country. For each
Implementation, Computational Complexity and Limitations
Incorporating gcometrical considerations typically requires the ability process large mini-batches (sce for example Xu et al. (2019)). ). Indeed, traditional methods for UDA, such DDC Tzeng al. (2014) DAN Long et al. (2015), and CORAL (Snn al. (017)) require large
However, this limitation be mitigated by considering adversarial approaches for UDA, such as ADDA Tzeng et al. (2017). In addition to the covariate shift, which is addressed using UDA methods, our approach relics on the approximation of the CDCA term. Since we approximate this term using non-parametric kernel regression, its resolution is also limited by the size of the mini- batch. This challenge will be addressed in future work by implementing the CDCA term paramctrically. More specifically, following the recent advances in UDA and the emergence of adversarial methods, we plan use an adversarial mechanism to sample from the cross-domain, rather than generate samples using the nonpparametric kernel regression.
wnttpes//bes..wlee.com htw.://icccc.oeoee/////o/ocooonn http://wwwooo.ooooeeeeoooeeoeeee 10211
21


Reviews:
[{"review_id": "DMZjbe7PiA", "reviewer": "Reviewer_8Dhw", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths:\n\nThis manuscript is quite easy to read, and the claims are largely explicit.  \n\nThe algorithm is compared on a variety of tasks (not just classification), especially useful since regression tends to act differently than classification.\n\nI appreciated the honest discussion in the appendix about reproducibility and model selection are nice.\n\n## Weaknesses/areas to improve:\n\n### Backgrounds/Methods:\n\nIn Section 4, we the authors claim that they generalize the theory of Ben-David et al. (2010), and this is also noted in the introduction as a contribution.  However, it is unclear exactly which part is novel.  For example, [1] considered a generic loss function as an extension of the Ben-David based on the prior extension of Li et al. (2018).  The contribution in this section needs to be clarified.\n\n[1] Richard, Guillaume, et al. \"Unsupervised multi-source domain adaptation for regression.\"  ECML PKDD 2020. \n\nAdditionally, I do not see authors discuss multi-task learning at all.  There is so much overlap of concept between multi-task learning and supervised domain adaptation that it really should be discussed.\n\nEquation 4 is not well motivated.  In particular, (2) is a lower bound on the target error, whereas adding in an estimate of the target error will loosen the lower bound. I get the motivation for using a noisy estimate of the target error, but this should be more carefully considered and explained.  Additionally, the equation now is basically: target error <= noisy realization of target error + bound on target error.  It seems like the different loss terms should be weighted.  For example, if there is a fair amount of target data, then it would seem prudent to use primarily the target error estimate rather than the bound.  Please add a discussion of this equation and whether weighting the samples is appropriate.\n\n### Empirical evaluations:\n\nThe optimization approach requires second order statistics to estimate the loss terms.  It is surprising to me that this can be reasonably estimated by such a small number of target samples.  I would appreciate a section that elaborates on the errors and feasibility of that optimization with limited target samples.\n\nOne of the things that concerns me is the very precise settings to compare the algorithms.  For example, in the digits dataset the source data and target data are both highly limited, which is a scenario repeated throughout the empirical evaluation.  It leaves me with a question on whether the results hold conclusions hold broadly.  In particular, most UDA papers have a pretty large source domain, and most real-world problems I consider have a fair amount of source data.  The authors need to clearly motivate limiting the source data, and discuss this extensively.  Additionally, the authors are considering highly data-limited target regimes.  As someone who has applied SDA in practice, many of our real world problems we have hundreds to thousands of labeled target data, so I am unable to evaluate from these results whether I should consider their algorithm  I would the authors to more clearly elaborate on what happens in different data regimes, and clearly describe the scenarios where their method is beneficial.\n\nAddition, the authors need to include more experimental details in the main paper.  I could not evaluate many of the claims without first reading through the appendix.  Key details need to be in the flow of the text.\n\nThe reproducibility of results is an interesting argument made in the appendix. In particular, the “Hyperparameter inconsistency” point argues that the same hyperparameters need to be tuned for each experiment.  I disagree with this assertion in general but understand the reasonableness in local scenarios.  I would like the authors to elaborate on this claim.  Secondly, as they point out, the lack of a validation set is a problem for tuning.  This highlights that the authors need to describe the validation and performance estimation approach very clearly.  While the authors note that they adopt the strategy of Hedegaard et al. (2021), the key details need to be repeated here so that the manuscript is more self-contained.  E.g., what data was used as validation, test, etc?  How well can the algorithms do with generic default parameters?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This manuscript proposes a new method for supervised domain adaptation.  Specifically, the authors extend theory that uses an extension of theory from Ben-David on unsupervised domain adaptation to work with supervised domain adaptation, and then design an algorithm to minimize the theoretical bound.  This algorithm is run on a number of empirical tasks and compared to several existing algorithms, showing on-par or improved performance", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "## Necessary:\n\nInclude details of validation procedure and discussion of its reasonableness.\n\nModify draft so that key necessary details are contained within the main paper.\n\nIncrease discussion of related works (e.g., multi-task, related theory) to more clearly delineate contributions\n\nDiscuss reasonableness of second-order statistic estimation.\n\nProvide results/discussion on different levels of data, including when source data is more available.\n\n## Minor:\n\nReference format:  Several references should be (name, year) not name (year).", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "IMOXbHlJJ2", "reviewer": "Reviewer_jTBJ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The conditional distributions are aligned with kernel regression that considers the geometric structure of the input and label spaces.\n\n- The model is verified on both classification and regression tasks.\n\nWeaknesses:\n\n- The proposed Theorem 4.1. seems identical to that of Ben-David et al. (2010). \n\n- Despite a theoretical approach, I do not observe any attempt to address the potential problem in this specific SDA setting, where limited labeled target data can increase generalization error.\n\n- Unlike SSDA, SDA does not allow unlabeled target data during training. However, if my understanding is correct, you still need unlabeled target data to verify your model. This problem setting seems impractical in dealing with real-world tasks where unlabeled data is always available.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes an SDA approach by aligning the marginal and conditional distributions based on Ben-David et al. (2010). Specifically, the conditional distributions are aligned with kernel regression that takes into account the geometric structure of the input and label spaces. The model is evaluated on both classification and regression tasks.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Overall, while I acknowledge that the authors made some good efforts to provide a clear exposition of their ideas, I found the core concept very similar to Ben-David et al. (2010) but less rigorous and theoretically motivated. I listed several concerns as follows.\n\n\n- I think it would be better to clarify the difference from the SSDA task. \n\n- How does Theorem 4.1. differ from Theorem 1 in Ben-David et al. (2010) which proposed:\n\n$\\epsilon_T(h) \\leq \\epsilon_S(h) + d_1(S,T) + \\min(E_{x\\sim S}|f_S(x)-f_T(x)|, E_{x\\sim T}|f_S(x)-f_T(x)|)$ \n\n- How did you get $M$ out of the integral in Eq.12?\n\n\n- The derivation from Eq.3 to E.4 is not theoretically motivated, where the classification loss of labeled target data is simply added to the objective.\n\n\n- The theoretical analysis is incomplete, e.g., the generalization error of Eq.2 is missing.\n\n\n- Where is the definition of $\\ell$ in Eq.5?\n\n\n- In UDA, we are forced to align marginal distributions $Q^S, Q^T$ since there are no target labels, which may increase joint error [1]. In this SDA setting, we can directly align $P^S_y, P^T_y$ to resolve the covariate shift. Therefore, I do not see the motivation to break this conditional alignment into marginal alignment ($Q^S=Q^T$) and CDCA ($P^S_x=P^T_x$).\n\n- In that sense, I would like to see a baseline sorely based on conditional distribution alignment such as [2] to justify your motivation. \n\n\n- How does your kernel regression method compare to simply training classifiers $h_S,h_T$ for $S, T$ and approximating $\\hat{Y}^S=h_T(S),\\hat{Y}^T=h_S(T)$?\n\n\n- I doubt NEM or d-SNE published in 2019 is still SOTA.\n\n\n- In Fig.3(b), the feature space seems not well-clustered. Consequently, I wonder if $\\|\\phi(X_i)-\\phi(X_j)\\|$ is still discriminative since the difference would be marginal for high dimensional vectors.\n\n\n[1]On Learning Invariant Representations for Domain Adaptation, ICML 2019\n\n[2]Conditional Adversarial Domain Adaptation, NIPS 2018", "broader_impact_concerns": "none", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "WC8e0Sa8uH", "reviewer": "Reviewer_DCB5", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The paper shows an upper bound for the error of supervised domain adaption under mild assumptions.\n\n2. The paper shows that such an upper bound can be utilized as a training objective for a Siamese network to get good performance in certain settings for supervised domain adaption.\n\nWeaknesses:\n1. The tightness of the upper bound is unknown. Indeed, there could be some steps during the derivations (e.g., eq 10 and 12) contributing to large bounding factors. Therefore, it is not clear that such an upper bound could be a good objective to optimize either in theory or in practice.\n\n2. The final optimization approach is quite similar to contrastive learning, but its connections and differences are not well discussed.\n\n3. The proposed method does not show very consistent results across settings. For example, why does it perform better for the M to U adaptation in Table 1 but not vice versa? Also, the network structures used for the experiments are a bit small, considering the recent literature. Recent work on these datasets typically uses ResNet-50 or similar architectures.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper studies the problem of supervised domain adaptation. It shows an upper bound for the error. The paper then utilizes the upper bound as a training objective and proposes training a Siamese network for the supervised domain adaptation problem. The paper shows that the proposed learning method can achieve certain improvements, especially for zero- or few-shot tasks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please address the weaknesses part.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper propose to address the problem of supervised domain adaptation (SDA) by minimizing an upper bound inspired by the work on Ben David et al. in unsupervised domain adaptation. The authors describe how to implement the proposed approach (using a kernel regression on the labels) and present experiments on classification, multi-label and regression DA to highlight the generality of the method. 

The reviewers found the paper interesting and well written but had several major comments and questions. The authors clarified most of the questions and updated the paper accordingly. The answers were appreciated by all reviewers but a few remaining concerns were raised during the final discussion. First the method is quite incremental wrt the bound of Ben David, and the assumptions on L prevent the used of the most commonly used classification loss  and the bound does not seem to be very tight with missing comparison to existing results. Also the positioning of the SDA method wrt other existing work especially Unsupervised DA and Multi-task learning is still lacking despite some localized changes and more references in the paper. The last concern was the limited performance gain in the numerical experiments compared to competitors. 

Despite those concerns that limit the potential impact of the paper, it is well written and the claims are reasonable and justified. The approach is also of interest to the community so the paper does respect the requirement for publication in TMLR. But some remaining work in positioning  is necessary before acceptance :
+ The positioning of the approach wrt existing method especially multi-task learning and multi-source DA should be done with more details that adding a few references. The SDA problem as described can be solved with MTL and MSDA so a description of what is done in the literature and why the proposed approach is different is necessary. For instance the proposed approach looks similar to [A] and [B] that minimize the other conditional distribution in the embedding.
+ As noted by a reviewer it is important to position and discuss more in detail the interest of the proposed SDA approach wrt the similar Margin Disparity Discrepancy from [C].  

[A] Gong, Mingming, et al. "Domain adaptation with conditional transferable components." International conference on machine learning. PMLR, 2016.

[B] Heinze-Deml, Christina, and Nicolai Meinshausen. "Conditional variance penalties and domain shift robustness." Machine Learning 110.2 (2021): 303-348.

[C] Zhang, Yuchen, et al. "Bridging theory and algorithm for domain adaptation." International conference on machine learning. PMLR, 2019.
Decision: Accept (poster)