Secure Domain Adaptation with Multiple Sources
Authors: Serban Stan
Mohammad Rostami
Venue: Accepted by TMLR ()

Abstract: Multi-source unsupervised domain adaptation (MUDA) is a   framework to address the challenge of annotated data scarcity in a target domain via transferring knowledge from multiple annotated source domains.  When the source domains are distributed, data privacy and security can become significant concerns and protocols may limit data sharing, yet existing MUDA methods overlook these constraints. We develop an algorithm to address MUDA when source domain data cannot be shared with the target or across the source domains.  Our method is based on aligning the distributions of source and target domains indirectly via  estimating the source feature embeddings and predicting over a confidence based combination of domain specific model predictions. We provide  theoretical analysis to support our approach and conduct   empirical experiments to demonstrate that our algorithm is effective.

Published in Transactions on Machine Learning Research (october/022)
Secure Domain Adaptation with Multiple Sources
Scrban Stan
sston edu
Department of Computer Science University of Southern Cakifrrna
Mohammad Rostami
rostamim Celaamimuusc..mm Prscc edu
Department of Computer Science University of Southern California
Reviewed on OpenReview: htips: // openreview. net/
met/ forum? id- UDttiStaaPpSrotte.d
Abstract
Multi-source unsupervised domain adaptation (MUDA) a framework to address the chal- lenge of annotated data scarcity a target domain via transferring knowledge from multiple annotated source domains. When the source domains are distributed, data privacy and se- curity can become significant concerns and protocols may limit data sharing, yet existing MUDA methods overlook these constraints. We develop algorithm to address MUDA when source domain data cannot be shared with the target or across the source domains. Our method is based on aligning the distributions of source and target domains indirectly via est est imating the source feature embeddings and predicting over a confidence based combi- nation of domain specific model predictions. We provide theoretical analysis to support our approach and conduct empirical experiments demonstrate that our algorithm is effective.
1 Introduction
Advances in deep learning have led to significant increase in the performance of machine learning (ML) algorithms in many applications (Russakovsky et al. (2015)). However, deep learning relies on access to large quantities of labeled data. for end-to-end blind training. Even if neural model is trained using annolated data, in settings where domain--hift (Torralba & Efros (2011)) exists between the training domain and deployment domain, neural models have been shown to suffer from sub-optimal performance. In such sconarios, the naive solution is to retrain the models on each domain from scratch. This opcration however necessitates annotating large databases persistently, which has proven to a lime-consuming and expensive process Rostami et al. (2018). Unsupervised Domaïn Adaptation (UDA) (Long et al. (2016)) framework developed to address the challenge of domain-shift and permit model generalization between source domain which has access to labeled data and related target domain in which only unannotated data is accessible.
Another recent line of research considers Invariant Risk Minimization (TRM) (Arjovsky et al. 2019 ), where model generalization on multiple domains is desirable (Ahuja et al. (2020)) ). The key difference compared to the UDA framework that in UDA minimizing target error is preferred to maintaining joint high performance on both source and target. Thus, UDA methods operate in a more relaxed environment compared to IRM, and allow for better target generalization (Krueger et al. (2020): Kamath al. (2021)
An effective technique to address UDA is to map data points from source and target domain into shared latent embedding space at which distributions are aligned Rostami (0221)) The latent embedding space is often modeled as the output-space of deep encoder, trained to produce shared representation for both domains. This outcome can be achieved using adversarial learning (Hoffman et al. (2018): Dou el al. (2019h: Tzeng et al. (2017); Boussaali et al. (2017)), where the distributions are matched indirectly through competing generator and discriminator networks to learn a domain-agnostic embodding. Alternatively, a probability metric can be selected and minimized to align the distributions directly in the embedding (Chen
et al. (2019); Sun et al. (2017); Lee et. al. (2019): Rostami & Galstyan 2020 Stan & Rostami (2021)

---

Published in Transactions on Machine Learning Research 2october/202)
Most existing UDA algorithms consider a single source domain for knowledge transfer. Recently, single-
source unsupervised domain adaptation (SUDA) has been extended to multi-source unsupervised domain adaptation (MUDA), where several distinct sources of knowledge are available (Peng et al. (2019a); Zhao et al.
(2020g; Rostami et al. (2019): Lin et al. 2020)) Guo et al. (2020): Tasar et al. (2020); : Venkat et al. (2020a)) ).
The goal in MUDA is to benefit. from the collect informat ion encoded in several distincl annolated source
domains to improve model generalization on an unannotated target domain. Compared to SUDA, MUDA
algorithms require leveraging data distribution discrepancies between pairs of source domains, as well as between the sources and the target. Thus, assumption in most MUDA algorithms is that the annotated source datasets are centrally accessible. Such premise however ignores privacy/securicg regulations or bandwidth limitations that may constrain the possibility of joint data access between source domains.
In practice, it is natural to assume source datasets are distributed amongst independent entities, and sharing data between them may constitute a privacy violation. For example, improving mobile keyboard predictions is performed by securely training models on independent computing nodes without centrally collecting user data (Yang et al. (2018)). Similarly, in medical image processing applications. data is oflen distributed amongst different medical institutions. Due to privacy regulations (Yan et al. (2021)) sharing data can be prohibited, and hence central access to data for all the source domains simultancously becomes infeasible. MUDA algorithms can offer privacy between the sources and target by operating in a source-free regime (Ahmed et al. (2021)) ), i.e., during the adaptation process source samples are considered to be unavailable, and only the source trained model or source data statistics are assumed to be accessible, However, approaches operating under this premise require retraining if new source domains become available, or if a set of source domains becomes inaccessible. This downside leads to increased time and computational resource cost.
We relax the need for centralized processing of source data in MUDA while maintaining cross-domain privacy, Our approach is robust to accessibility changes for different source domains, allowing for relearning the target decision function without end-to-end retraining. We relax the need for direct access to source domain samples for adaptation by approximating the distribution of source embeddings. We perform sourco-free adaptation with respect lo each source domain, and propose a confidence based pooling mechanism for larget inference. In the present work, we: (i) Address the challenge of data privacy for MUDA by maintaining full privacy between pairs of source domains, and between the sources and the target. (ii) Propose efficient distributed optimization process for MUDA to process each dataset locally while encoding high-level learned knowledge in a shared latent embedding space. (ii) Provide theoretical justification for our method by proving that our algorithm minimizes an upper-bound of the target error. We conduct extensive empirical experimental results on five standard MUDA benchmark datasets to demonstrate the effectiveness of our approach.
2 Related work
Single-Source UDA: Single source UDA aims to improve model generalization for an unlabeled target domain using only single source domain with annotated data. SUDA has been studied extensively. A primary workflow employed in recent UDA works consists of training a deep neural network jointly on the labeled source domain and the unlabeled target domain to achieve distribution aligmment between both domains in a latent embedding space. This goal has been achieved by employing generative adversarial networks (Goodfellow et al. (2014)) to encourage domain alignment (Hoffman et al. (2018): ; Dhouib et al. Luc et al.] (2016); Tzeng et al. (2017); Sankaranarayanan et al. (2018)) as well as directly minimizing
g2 y an appropriate distributional distance between the source and target embeddings (Long et al. (2015) 2017b); Morcrio et al.] (2018)). ). SUDA algorithms do not leverage inter-domain statistics the presence of several source domains, and thus extending single-source UDA algorithms to a multi-source setting is nontrivial.
Multi-Source UDA: The MUDA setting is recent extension of SUDA, where multiple streams of data are concomitantly leveraged for improved target domain generalization. Xu et al.] (2018) minimize discrepancy
between source and larget domains by optimizing an adversarial loss. Peng et al. (2019 adapt on multiple domains by aligning inter-domain statistics of the source domains in an embedding space. Guo et al. (2018) Iearn to combine domain specific predictions via meta-lcarning. |Venkat et al. (2020a) use pscudo-labels to improve domain alignment. The increased amount of source data in MUDA not necessarily an advantage over SUDA, as negative transfer between domains needs to be controlled during adaptation. Li et al. (0018)
2

---

Published in Transactions on Machine Learning Research (october/202)
exploit domain similarity to avoid negative transfer by leveraging model statistics in shared embedding space. Zhu et al. (019) achieve domain alignment by adapting deep networks at various levels of abstraction. Zhao et al. (2020) align target features against source trained features via optimal transport, then combine source domains proportionally to Wasscrstein distance. Wen et al. (0200) use a discriminator to exclude data samples with negative generalization impact. Such approaches admit joint access to source domains during the training process, making them infeasible in scttings where data privacy and security are of concern.
Privacy in Domain Adaptation: The importance of inter-domain privacy has been recognized and explored for single-source UDA, specifically in source-free adaptation. Note this framework is relevant in many important practical settings even for SUDA, where privacy regulations limit the possibility of sharing data (Peng et al. (2019b); Li et al., (2020); Liang et al. (2020a) (021)). Several UDA approaches consider maintaining an approximatinon of the source domain for adaptation. Kurmi et al. (2021)) benefit
from GANs to generate source-domain like samples during the adaptation phase. Yeh et al. (2021) align
distributions via minimizing the KL-divergenee in addition to a variational autoencoder reconstruction loss.
Similar to our approach, Yang et al. (2022) model the source distribution and use clustering of target samples to assign the correct class is done by minimizing an VAE reconstruction error. [Tian et al. (2022)
approximate the latent source space during adaptation and use adversarial learning in their approach. Ding ct al. (2022) also estimate the source distribution and choose specific anchors to guide the distribution learning process. Adaptation is done by optimizing class conditional maximum mean discrepancy wetween samples from the learnt approximation distributions and the target samples. These above works consider only a single source. In multi-source adaptation, the strategy for combining information across source domains is key in achieving competitive performance. Peng et al. (2019b) perform collaborative adaptation under privacy restrictions between souree domains under the framework of federated learning. [Ahmed ct al. (2021 approach privacy-preserving MUDA via information maximization and pseudo-labeling, Dong et al. (2021 choose high confidence target samples as class anchors and pseudo-labels labels are then assigned according to the closest anchors. Unlike our approach, Ahmed et al. (2021); Dong et al. (2021) require simultaneous access to all source trained models during adaptation. This makes these method unsuitable for scenarios such as asynchronoss federated learning (Peng et al.] (2019b)), ) where communication between individual domains may be broken, or processing different source domains may be done at irregular time intervals (McMahan et al. (2016)). Our main adaptation tool is represented by optimal transport based optimization, followed by a confidence based pooling mechanism, making the adaplation process considerably more weigt.
We address a more constrained yet practical setting, where privacy should be preserved both between pairs of source domains and with respect to the target. Our approach allows for efficient distributed optimization, not requiring end-to-end retraining if different source domains become inaccessible due to privacy obligations McMahan et al.] (2016); Peng et al. (2019b), or more source domaīns become available after initial training has finished, allowing for accumulative learning from several domains. Our method builds on extending the idea of probability metric minimization, explored in UDA (Chen et al. (2019: Lee et al. (2019); ; Stan & Rostami (2022) Rostami (2021a) 2022.) to MUDA. The latent source and target features are represented via the output space of neural encoder. Domain alignment implics a shared embedding space for these representations. To achieve this, a suitable distributional distance metric is chosen between these two sets of embeddings and minimized. In this work, we used the Sliced Wasserstein Distance (SWD) (Rabin et al. (2011): ;; Bonncel et al. (2015)) for this purpose. SWD is metric for approximating the optimal transport metric (Redko et al. (2019)) J. It is a suitable choice for UDA because: (i) it possesses non-vanishing gradients for two distributions with non-overlapping supports. As result, it is suitable objective function for deep learning gradient- based optimization techniques. (ii) It can be computed efficiently based on a closed-form solution using only empirical samples, drawn from the two probability distributions.
3 Problem formulation
Let Sy, S, be the data distributions of n annotated source domains and T be the data distribution of an unannotated target domain. Assume the source and target domains share the same feature space RXxHXC where W, H. C describe an image by width, height and number of channels respectively. We con- sider all domains having common label-space Y, but not necessarily sharing the same label distribution. For each source domain k we observe the labeled samples {( ...1)...., (2% yk.ne) } where of Sk.
3

---

Figure 1: Block-diagram of our proposed approach: (a) source specific model training is done independently for each source domain (b) the distribution of latent embeddings of each source domain is estimated via Published in Transactions on Machine Learning Research (tttobrr/2022)
Training and Adaptation
Test-time Inference
Source Domain
CNN
FC
L
Cls.
CNN
FC1
Cis.
w_1
(b)
The final predictions are a convex combination over predictions from each adapted model
Adaptation
Prototypical
initialization with
source weights
GMM
distribution
Target Domain
(d)
CNN n
I
FCn
I
Cls.
w
(c)
Adaptation
CNN
FC
via L +L
W2 ent
a mixture of Gaussians, (c) for each source trained model, adaptation is performed by minimizing the distributional discrepancy between the learnt GMM distribution and the target encodings (d) the final target domain prodictions are obtained via a learnt convex combinations of logits for each adapted model
We only observe unlabeled samples {æf...., x from the larget domain T. The goal is to train a model fo : RWxxxC + RR!* capable of inferring target labels, where [Y] is the number inference classes. The first step in our approach is independently train decision models for each source domain via empirical risk minimization (ERM) by minimizing the cross-entropy loss Lee 0₂ = arg ming "f Since under our considered setting the target and source domains share common input and label space, these models can be directly used the target to derive a naive solution. However, given the distributional discrepancy between source domains and target, c.g., real world images versus clip art, generalization perfor- mance will be poor. The goal of our MUDA approach is to benefit from the unannotated target datasel and the source-trained models in order to improve upon model generalization while avoiding negative transfer.
1 E Lee (fo Bk.)).
To this end, we decompose the model fo into a feature extractor gu(-): R + Rdz and a classifier subnetwork hp(()= Rdz + RI with learnable parameters and v, such that ʃ(-) = (h o g.). We assume input data points are images of size W И C and the latent embedding shape is of size dz. In a SUDA setting, we can improve generalization of each sourco-specific model the target domain by aligning the distributions of the source and the target domain in the latent embedding space. Specifically, we minimize distributional discrepancy metric D),,)) across both domains, e.g., SWD loss, to update the learnable parameters: u arg min, (Sa((S) (u))). By aligning the distributions the source trained classifier h2 will generalize well on the target domain T. In the MUDA setting, the goal improving upon SUDA by benefiting from the collective knowledge of the source domains make predictions on the target. This can be done via weighted average of predictions made by each of the domain-spccific models, i.c., models with learnable parameters 0g2 (ug, vx). For sample of in the target domain, the model prediction will be Σn -h= wxfop (X)), where wy denotes a set of learnable weights associated with the source domains.
We note the above general approach requires simultancous access to source and target data during adaptation. We relax this constraint and consider the more challenging setting of source-free adaptation, where we lose access to the source domains once source training finishes. To account for applications with sensitive data, o.g. medical domains, we also forbid interaction between source models during adaptation. Hence, the source distributions S₃ and their representations in the embodding space, i..., g(Sx), will become inaccessible. To circumvent this challenge, rely on intermediate distributional estimates of the source latent embeddings.
4

---

employ Gaussian Published in Transactions on Machine Learning Research (October/200) 2202)
4 Proposed algorithm
Our proposed approach for MUDA with private data is visualized in Figure We basc our algorithm two levels of hierarchies. First, adapt each source-trained model while preserving privacy (left and middle subfigures). We then combine predictions of the source-specific models on the target domain according to their reliability (right subfigure). To tackle the challenge of data privacy, we approximate the distributions of the source domains in the embedding space as a multi-modal distribution and usee these distributional estimates for domain alignment (Figure 1 left). We can benefit. from these estimates because once source training is completed, the input embodding distribution should be mapped into [ distribution enable the classifier subnetwork to separale the classes. Note, each separated distributional mode encodes left). To approximate these internal distributions
one of the classes (see Figure [ Mixture Models (GMM), with learnable mean and covariance paramcters μkg Σg. Since we have access to labeled source data points, we can learn P4 and Σς in supervised fashion. Let 1c(x) denotes the indicator function for x then the maximum likelihood estimates for the GMM parameters would be:
Mkk Eee ( (af Ee -== 1e (o(H))))
Eis Lo(Na.)) (4%%.) a) (k, (2hh,) mR,e)
Ee.e
(1)
ΣE (MN.s)
Learning aa and Σg for each domain enables us sample class conditionally from the GMM distributional estimates and approximate the distribution g(Sx) in the absence of the source dataset.
We adapt the source-trained model by aligning the target distribution T and the GMM distribution in the embedding space. To preserve privacy, for each source domain k we generate intermediate pseudo- domains A& with pseudo-samples " J by drawing random samples from the estimated GMM distribution. The pseudo-domain is used as approximation of the corresponding source embeddings. To align the two distribution, a suitable distance metric D):,) needs to be used. We rely on the SWD due to its mentioned appealing properties. The SWD acts as an estimate for the Wasserstein Distance (WD) between two distributions (Rabin et al. (2011)), ) by aggregating the tractable 1-dimensional WD over projections onto the unit hypersphere. In the context of our algorithm, the SWD discrepancy measure becomes:
1 Σ (o(x)))))) di) - -2k.,, 22)1
D(g(T),Au) -
(2)
where di projection direction, and i; are indices corresponding to the sorted projections. While the source and target domains share the same label space y they do not necessarily share the same distribution of labels. Since the prior probabilities on classes are not known in the larget domain, minimizing the SWD at the batch level may lead to incorrectly clustering samples from different classes together, depending on the discrepancy between the label distributions. To address this challenge, we take advantage of the conditional entropy loss (Grandvalet & Bengio (2004)) as regularization term based on information ion maximization. The conditional entropy acts as a soft clustering objective that ensures aligning target samples to the wrong class via SWD will be penalized. We follow the approximation presented in Eq. 6 in Grandvalet Bengio (2004):
Lent Lene (fo(())T))
Σ Coe (fo(z)) n
(3)
To guarantee this added loss term influences the latent representations produced by the feature extractor, the classifier frozen during model adaptation. Our final combined adaptation loss is descrbied as:
D(g(T), A) (fo)((((TT)
(4)
5

---

Published in Transactions on Machine Learning Research (october/202)
for regularizer ?. Once the source-specific adaptation is completed across all domains, the final model predictions on the target domain are obtained by combining probabilistic predictions returned by each of the n domain-specific models. The mixing weights are chosen as convex vector w = (W1 ww), i.e, > 0 and E, 1, with final predictions taking the form Σ w.fe. The choice of is critical, as assigning large weights to a model which does not generalize well will harm inference power. We utilize the source model prediction confidence on the target domain as proxy for generalization performance. We have provided empirical evidence for this choice in Section We thus set a confidence threshold and assign wa:
W6 1 (max for (x)) > A),
Wkk W
Σ
Wk
where f()) denotes the model ouput just prior to the final SoftMax layer which correspond to probability,
Note the only cross-domain information transfer in our framework is communicating the latent. means and covariance matrices of the estimated GMMs plus the domain-specifie model weights which pro- vide a warm start for adaptation. Data samples are never shared between any two domains during pretraining and adaptation. As result, our ap- proach preserves data privacy for scenarios at which the source datasets are distributed across several entities. Additionally, the adaptation process for each source domain is performed independently. As a result, our approach can be used incorporate new source domains as they become available over time without requiring end-to-end retraining from scratch. We will only require to update the nor- malized mixing weights via Equation 5] which takes negligible runtime compared to model training. Our proposed privacy preserving approach, named Se- cure MUDA (SMUDA), is presented in Algorithm
Secure Multi-source Unsupervised Do- main Adaptation (SMUDA)
Algorithm
procedure SMUDA(S ......... T,, for k + to n do
pkg 2g. 0g = Train((s))
Generate A& basod on Mk: Ex
Compute we (Eq. 5)
0 = Adopt (p. Ak. T,L,7)
return wh Wn, 01
On
procedure TRAIN(S,)
Learn 0g. (uk, the) by min. Lcr(So (8x), Learn parameters uki Ei (Eq. I
return Pk: Σx. 0%
procedure ADAPT(0s, A, I,
Initialize network with weights O2
04
arg ming ((g As) + (ooe)(
(Eq. return 0%
5 Theoretical analysis
We provide analysis to demonstrate that algorithm minimizes upper bound for the target domain error. We adopt the framework developed by Redko & Sebban (2017) for single source UDA using Wusser- stein distance to provide a theoretical justification for the algorithm we proposed. Our analysis is performed in the latent embedding space. Let H represent the hypothesis space of all classifier subnetworks. Let h& (-) denote the model learnt by each domain--pecific model. We also set eD(.), where E {8) ........ S.. T}, to be the true expected error returned by some model h(-) H. in the hypothesis space on the domain D. Additionally, let jAs, मे. Σ"E ((((()))) Ar, 1 E #2,,, and üuT = / Eih (((xt)) denote the emn pirical distributions that are built using the samples for the source domain, the intermediate pscudo-domain, and the target domain in the latent space. Then the following theorem holds for the MUDA setting:
Theorem 1. Consider Algorithm for MUDA under the explained conditions, then the following holds
V
I N + M + ec, (M2)
(2 (Th)) Σ wx(es, (ha) + ((,T, + D(pr, (As,) + (2 (
(6)
where Ck is the combined error loss with respect to domain k and ht is the optimal model with respect to this loss when shared model is trained jointly on annoloted datasels from all domains simultaneously.
Proof: the complete proof is included in the Appendix.

---

Published in Transactions on Machine Learning Research (october/202)
We note the target domain error is upper bounded by the convex combination of the domain-specific adap- lation errors. Algorithm 1 minimizes the right-hand side of Equation G as follows: for each source domain, the source expected error is minimized by training the models using ERM. The second term is minimized by closing the distributional gap between the intermodiate pseudo-domain and the target domain in the latent space. The third term corresponds to how well the GMM distribution approximates the latent source samples. Our algorithm does not directly minimize this term, however if the model forms a multi-modal distribution in the source embedding space, necessary for good performance, this term will be small. The second to last term is dependent on the number of available samples in the adaptation problem, and becomes negligible when sufficient samples are accessible. The final term measures the difficulty of the optimization, and is dependent only on the structure of the data, For related domains, this term will also be negligible.
6 Experimental validation
Datasets We validate on five datasets: Office-31, Office-Homas
Inage-Clef and DomainNet.
Office-31 (Saenko et al. 2010)) is a datasel with 31 classes consisting of 4110 images from an olfice en- vironment pertaining to three domains: Amazon, Webcam and DSLR. Domains differ in image quality, background, number of samples and class distributions.
Caltech (Gong et al. (2012)) contains 2533 from 10 classes of office related images from four domains: Amazon, Webcam, DSLR, Caltech. Office- Home /eenaateswara et al. (2017)) contains 65 classes and 30475 from four different domains: Art (stylized images), Clipart (clip art sketches), Product (images with no background) and Real- World (realistic images), making it more challenging that Office datasets. Image- Clef (Long et al. (2017a)) contains 1800 images under 12 generic categories sourced from three domains: Caltech, Imagenet and Pascal. DomainNet (Peng ct al. (2019a)) is a larger, more recent dataset containing 586,575 images from 345 general classes, with different class distributions for each of its domains: Quickdraw, Clipart, Painting, Infograph, Sketch, Real.
Preprocessing & Network structure: we follow the literature for fair comparison. For each domain we re-scale images to a standard size of (224, 224,3). We ResNet50 (He et al. (2016)) network backbone for the feature extractor, followed by four fully connected layers. The network classification head consists of a linear layer, and source-training is performed using cross-entropy loss. The ResNet layers of the feature extractor are frozen during adaptation. We report classification accuracy, averaged across five runs. As hardware we used a and NVIDIA Titan Xp GPU. Our code is provided as part of the supplementary material and available online at https:/
To test the effectiveness of our privacy preserving approach for MUDA, we compare our method against state-of-tth the art SUDA and MUDA approaches. Benchmarks for single best (SB), source combined (SC) and
multi source (MS) performance are reported based on DAN (Long et al. (2015)) D-CORAT, Sun & Saenko
(2016) RevGrad (Ganin & Lempitsky (015)). ). We include most existing MUDA algorithms: DCTN (Xu
et al. () , FADA (eng et al. (0019b)) MFSAN (Zhu et al. (2019)) MDDA (Zhao et al. (2020)) ) SimpAl (Venkat et al. (2020b)), ), JAN (Long et al. (2017b)), MEDA Waan et al. (2018)) MCD (Saito et al. (2018)), M3SDA (Pong et al. (2019ab) ) MDAN (Zhao et al. (2018) MDMN (Li et al. (2018)), DARN (Wen et al.
(2020), DECISION (Ahmed et al. (2021)). Note that we maintain full domain privacy throughout training and adaptation. Hence, most of the above works should be considered as upperbounds in performance, they address more relaxed problem, by allowing joint and persistent access to source data. While Ahmed et al. (2021) also performs sourco-free adaptation, they benefit from jointly accessing the source trained models during adaptation, while our method only assumes joint access when pooling predictions. Despite this additional constraint, results prove our algorithm is competitive and at times outperforming the aforementioned methods. We next present quantitative and qualitative analysis of our work.
6.1 Performance Results
Table I presents our main results. For Office-31, we observe state-of-the-art performance (SOTA) on the -> + + A tasks and near SOTA performance on the remaining task. Note that the domains DSLR and Webcam share similar distributions, as exemplified through the Source-Only results, and for these domains obtaining good adaptation performance involves minimizing negative transfer, which our method
7

---

Published in Transactions on Machine Learning Research (october/202)
successfully achieves. In the case of Image-clef, we obtain SOTA performance all tasks, even though the methods we compare against are not source-free. On the Office-caltech dataset, we obtain SOTA performance on the + A task, with close to SOTA performance on the three other tasks. The domains of the Office-home dataset have larger domain gaps with more classes compared to the three previous datasets. Our approach obtains near SOTA performance on the + and + R lasks and competitive performance on the remaining tasks. Finally, the DomainNett dataset contains much larger number of classes and variation in class distributions compared to the other datasets, making the most challenging considered task in experiments. Even 8o, are able to obtain SOTA performance three of the six tasks with competitive results on the other three. We reiterate most other MUDA algorithms serve upper-bounds to our work, as they either access source data directly, simultaneously use models from all sources for adaptation, or both. Results across all tasks demonstrate that not only are we able to compare favorably against these methods while preserving data privacy, but we also set new SOTA on several tasks.
Method
D
WW
Avg.
Method
+P
+C
-1
Avg.
Method
WW
+D
+C
Avg.
Source Only
96.7
62.5
86.2
Source Only
74.8
91.5
83.9
83.4
Source Ouly
99.0
98.3
87.8
86.1
92.8
8R
DAN
99.7
98.0
65.3
67.7
DAN
75.0
93.3
86.2
84.8
DAN
99.3
98.2
89.7
94.8
95.5
D-CORAL
99.7
98.0
65.3
87.7
D-CORAL
76.9
93.6
88.5
86.3
JAN
99.4
91.2
91.8
D-CORAL
RevGrad
DAN
99.6
99.3
97.8
98.0
67.6
07.1
66 2
88.3
87.5
88.1
RevGrad
DAN
75.0
77.6
96.2
93.3
87.0
92.2
86.1
87.7
DCTN
DAN
99.5
99.4
99.0
99.1
89.2
90.2
91.6
91.6
91.8
94.8
MEDA
99.3
99.2
91.4
92.9
95.7
RevGrad
88.5
D-CORAL
77.1
93.6
91.7
87.5
MCD
99.5
99.1
91.5
92.1
95.6
MESAN
SimpAl
MDDA
DCTN
99.2
99.5
99.3
97.1
98.2
98.5
56.2
72.7
64.2
89.0
84.2
90.2
87.2
RevGrad
MFSAN
DCTN
75.0
79.1
77.9
95.7
95.4
93.7
90.3
93.6
91.8
89.4
87.8
87.0
FADA
SImpAA1
MSSDA
99.4
88.1
99.3
99.2
99.8
87.1
91.5
92.2
88.7
84.2
94.1
95.3
96.1
96.7
87.1
DECISION
99.6
98.4
75.4
91.1
MS
SImpAl
77.5
93.3
91.0
87.3
DECISION
99.6
100
95.9
95.9
98.0
SMUDA
99.8
98.5
75.4
91.2
SMUDA +
79.4
96.9
93.9
90.1
SMUDA
99.3
97.6
93.9
95.9
96.6
(a) Office-31
(b) Image-clef
(c) Office-caltech
Method
-A
+
-P
Avg.
Method
+
Q
+
C
P
+
S
+
Avg.
Source Only
65.3
49.6
79.7
75.4
67.5
Source Only
11.8
39.6
33.9
8.2
23.1
41.6
26.4
DAN
68.2
56.5
80.3
75.9
70.2
DAN
16.2
39.1
33.3
11.4
29.7
42.1
28.6
D-CORAL
67.0
53.6
80.3
76.3
69.3
35.3
32.5
25.7
43.1
26.7
RevCrad
67.9
55.9
80.4
75.8
70.0
ADDA
14.9
39.5
29.1
14.5
30.7
41.9
28
DAN
65.5
59.4
79.0
82.5
72.4
MCD
42.6
19.6
33.8
50.5
32.2
D.CORAL
68.1
58.6
79.5
82.7
72.2
Source Only
38.1
13.0
33.7
51.9
32.9
RevGrad
68.4
59.1
79.5
82.7
72.4
DAN
15.3
45.4
36.2
12.8
34.0
48.6
32.1
MFSAN
72.1
62.0
80.3
81.8
74.1
SC
JAN
12.1
40.9
35.4
11.1
32.3
45.8
29.6
M'SDA
64.1
62.8
76.2
78.6
70.4
ADDA
14.7
47.5
36.7
11.4
33.5
49.1
32.2
MS
SlmpAl
70.8
56.3
80.2
81.5
72.2
MCD
7.6
54.3
45.7
22.1
43.5
58.4
38.5
MDAN
68.1
67.0
81.0
82.8
74.8
DCTN
7.2
48.6
48.8
23.5
47.3
533.5
38.2
MDMN
68.7
67.6
81.4
83.3
75.3
M
6.3
58.6
52.3
26.0
49.5
62.7
42.6
DARN
70.0
68.4
82.8
83.9
76,2
FADA"
7.9
45.3
38.9
16.3
26.8
46.7
30.3
DECISION
74.5
59.4
84.4
83.6
75.5
MS
DECISION
18.9
61.5
54.6
21.6
51
67.5
45.9
SMUDA
69.1
61.5
83.5
83.4
74.4
SMUDA
14.6
62.4
53.6
24.4
49.9
68.3
45.5
(d) Office-home
(c) DomainNet
Table 1: Results on five benchmark datasets. Single best (SB) represents the best performance with respect to any source, source combined (SC) represents performance obtained by pooling the source data together from different domains, and multi source (MS) represents mothods performing multi source adaptation. indicates source-free adaptation, guaranteeing privacy between sources and the target. + indicates privacy between source models. Results in bold correspond to the highest accuracy amongst the nource-free approaches.
6.2 Ablative Experiments and Empirical Analysis
We perform ablative experiments by investigating the effect of each loss term in Eq. on performance, and present results in Table 2 We observe combining the two terms yields improved performance for all datasets besides Office-caltech, where the difference negligible. On the other hand, minimizing the SWD is more impactful on the Image-clef and Office-home datascts. The conditional entropy contributes more for Offich- and some Office-31 tasks. Our insight conditional entropy is more impactful when the source trained models have higher initial performance the target (e.g.- + D, W on Office-91), while the
8

---

Published in Transactions on Machine Learning Research (october/202)
Method
++
-W
-+A
Avg.
Method
-W
-+D
-CC
A
Avg.
SWD only
92.2
94.1
73.1
86.4
SWD only
98.1
97.8
92.1
95.5
95.9
Cent only
99.8
98.1
66.2
88.1
Lent only
99.4
97.7
94
96
96.8
SMUDA
99.8
98.5
75.4
91.2
SMUDA
99.3
97.6
93.9
95.9
96.6
(a) Office-31
(b) Office-caltech
Method
-PP
+CC
-+I
Avg.
Method
-
+
-P
->
R
Avg.
SWD only
79.3
96.5
94.2
90
SWD only
66.6
59.1
80.9
82.2
72.2
Lest only
78.5
96
91.7
88.7
Los only
64.5
49.4
77.8
72.2
66
SMUDA
79.4
96.9
93.9
90.1
SMUDA
69.1
61.5
83.5
83.4
74.4
(c)
Image-clef
(d) Offico-home
Table 2: Results when only the SWD objective, the entropy objective both (SMUDA) are used.
SWD term is more beneficial when there is a larger discrepancy between the source domains and the target domain (o.g., + on Office-31). Experiments conclude using both terms further improves performance.
Preserving privacy when performing adaptation limits information access belween domains. is then ex- pected that privacy preserving methods are at disadvantage compared to methods that do not impose any privacy restrictions. We show that in the case of our method, this observed disadvantage is negligible. To study the effect of preserving privacy UDA performance,
perform experiments where source data is shared eit between sources with the target domain. We consider three primary scenarios for sharing source data: (i) SWD loss is computed using the source domain latent features (SW); (ii) Source domains' data is Combined into a single source (SC); Supervised Source loss is computed for joint UDA (SS). We re- port results for natural combinations these approaches Table 3 We observe SMUDA performs similarly SW. Joint adaptation (SS), sourco-combined performance (SC), or a combination of the two offer improved performance on all datasets. The improvements from sacrifieing privacy are however negligible compared to SMUDA. We conclude under the considered test cases. This leads our adaptation approach to achieve comparable performance to settings where privacy is not enforced, with the added benefit of not sharing data between domains.
source domain approximation using GMMs captures sufficient source information
Method
D
W
-
Avg.
Method
-W
+D
C
A
Avg.
SW
99.5
98.5
75.5
91.2
SW
99.4
96.9
93.9
95.9
96.5
SC
98.1
96.8
76
90.3
SC
99.7
96.8
94.1
96
96.6
SS
99.8
98.7
76.3
91.6
SS
99.6
97.2
94.1
95.9
96.7
SW+SS
99.8
98.6
75.7
91.3
SW+SS
99.7
97.4
94.1
96
96,8
SC+SS
98.4
96.9
76.1
90.5
SC+SS
99.7
96.8
94.2
96.2
96.7
SC+SW+S5
99.0
97.7
76.1
90.9
SC+SW+SS
99.6
97.2
93.3
95.9
SMUDA
99.8
98.5
75.4
91.2
SMUDA
99.3
97.6
93.9
95.9
96.6
(a) Office-31
(b)
Offico-caltedl
Method
+
+
+1
Avg.
Method
-A
+C
Avg.
SW
79.5
95.2
91.3
SW
68.9
60.8
83.3
83.4
74.1
SC
79.8
96.6
94.2
90.2
SC
69.6
62.9
85.3
84.7
75.6
SS
79.4
95.6
91.6
88.9
SS
68.9
61.1
84
83.6
74.4
SW+SS
79.4
95.6
91.8
88.9
SW+SS
68.5
61
83.8
83.8
74.3
SC+SS
79.9
96.6
93.1
SC+SS
69.4
85.5
75.7
SC+SW+SS
79.5
95.2
91.3
88.6
SC+SW+SS
68.8
62.7
85.1
84.5
75.3
SMUDA
79.4
96.9
93.9
90.1
SMUDA
69.2
61.1
83.2
83.5
74.3
(c) Image-clef
(d) Office-home
Table Results comparing SMUDA to non-private variants.
To compare our method against ensemble models of existing single-source source-free UDA (SFUDA) meth- ods, we performed experiments on the Office-31 dataset. We compare against five recent SFUDA approaches in Table 4 and the ensemble of these methods in Table We observe that although in the SFUDA setting,

---

Published in Transactions on Machine Learning Research (otoobrr/022)
despite being competitive, method trails some of the methods, it outperforms the methods in MUDA. We conclude that our method alleviates the effect of negative transfer successfully and indeed can boost performance of a weaker single-source performance. We also note that we likely can improve the SFUDA performance for our method if we benefit from better probability metrics or model regularization.
Method
A++
W-D
A
D-W
D
W
W
Avg.
USEDA
Kundu et a) 2020
64.5
96
71
93.3
62.8
63.6
75.2
SHOT
Jang et nl, 20206
94.0
99.9
90.1
98.4
74.7
74.3
88.6
AFN
Xu et al. 2019
90.7
99.8
90.1
98.6
73.0
70.2
$7.1
Zhang al. 2019
100
98.4
72.2
GVB GD
98.7
SMUDA (ours)
99.8
87.9
98.5
72.1
75.4
87.7
Table 4: Single source results
Method
DD
-+ W
-+.A
Avg.
USFDA
Kundu et al.
2020
96.0
93.1
65.5
84.9
SHOT-ens
Ahmed
2002
97.8
91.9
75.0
89.3
AFN Xu et al.
2019
98.4
96.4
71.3
88.7
MDD Zhang
2019
95.4
99.3
74.1
89.6
CVB-CD Oui et
2020
97.2
95.6
74.9
89.2
SMUDA-Umitrr
94.2
74.8
75.4
86.4
SMUDA (ours)
98.5
75.4
Table 5: Uniformly combined predictions
We study the effect of hyper-parameters on SMUDA performance. We first empirically validate our approach for computing the mixing parameters Wk- We consider four scenarios for combining model predictions: (i) Eq. (ii) setting weights proportional to SWD between the intermediate and the target domains (a cross- domain measure of distributional similarity), (iil) using uniform average, and (iv) assigning all mixing weight to the model with best target performance. Average performance for tasks from four of the datascts are reported in Table 0 We observe our choice leads to maximum performance. Single best performance is able to slightly outperform on one dataset, however suffers on tasks where significant pairwise domain gap exists. This is expected, as using several domains is beneficial when they complement each other in terms of available information. Assigning weights proportional to D(y(T), As) may seem intuitive, given that similarity between pseudo-datasets and target latent features indicates better classifier generalization. However, this method performs better only compared uniform averaging. We conclude model reliability is superior criterion of combining predictions. Uniform averaging leads to decreased generalization the target domain as it treats all domains equally. As result, models with the least. generalization ability on the target domain harm collective performance.
Dataset Office-31
High confidence
W2
Uniform
Single Best
91.2
88.6
85.1
91.2
Image-clef
90.1
89.6
89.8
89.6
Office-caltech
96.6
96.6
96.6
97
Office-home
74.4
74.2
72.8
Total avg.
88.1
87.2
86.4
87.6
Table 6: Analytic experiments to study four strategies for combining the individual model predictions. Mixing based on model reliability proves superior to other popular approaches.
We additionally study the effect of the SWD projection hyper- parammeer SWD utilizes L random projections, as detailed in Equation 2 While large L leads to tighter approximation of the optimal transport metric, it also incurs a computational resource penalty. We investigate whether there is a range of L values offering sufficient adaptation performance, and analyze the impact of this parameter using the Office-31 dataset. Figure 2 we reported performance results for L E (1,10, 50, 100, 200, 350, 500). The SWD approximation becomes tighter with an increased number of projections, which we see translating on all three tasks. We also note that above certain threshold, i.e. I 200, the gains in performance from increasing L are minimal.
10

---

Published in Transactions on Machine Learning Research (October/202) 2022)
Office ->W
Cffice D>
Office ->A
97.75
99.7
Target Accurac) (C
99.50
Target Accurac)
75
97.25
99.5
74
97.00
99.4
96.75
.99.3
72
96.50
99.2
96.25
70
10
50
100
200
350
500
1
50
100
200
350 500
10
100 200 350 500
Num. Projections
Num. Projections
Num. Projections
Figure 2: Performance for different numbers of latent projections used in the SWD on Office-31.
1.0
1.0
combinesipefformance
combirediaefforancee
combined performance
W2 + Ent Target
0.8
0.8
0.8
0.6
0.6
0.6
adaptation
Starts
adaptation Starts
44a00aa S3a sate
0.4
0.0
0.0
0.0
10 15
25
5
10
20
25
10
20
25
Iterations (thousands)
Iterations (thousands)
Iterations (thousands)
Figure 3: Effect of the adaptation process on the Office-home dataset: from left to right, we consider Art, Clipart and Product as the source domains, and Real World as the target domain.
In Figure we explore the bchavior of our adaptation strategy with respect to a Office-home task. For cach of the three source domains, we observe an increase in larget accuracy once adaptation starts, which is line with our previous results. Note this increase in target accuracy also correlates with the minimization of the SWD and entropy losses. We additionally note that the combined multi-source performance using all three source domains outperforms the three SUDA performances. The biggest difference is observed for the Clipart trained model, which exhibits the highest discrepancy from the target domain Real World.
Art Real World
100
Clipart
Product
Real World
Real World
Target accurecy
Target accuracy
Target accuracy
40
20
20
0
(0. (2 4] 64 (6,88 8] (,81) Source classifier confidence
10..2] (.2.4] (.4.6) [66 (.8,1)
(0. 2] (.2.4] (4 (6,,8] 8] (881) Source classifier confidence
Source classifier confidence
Figure Prediction accuracy on Office-home target domain tasks under different levels of source model confidence, and our choice of A. Target predictions above this threshold attain high accuracy.
The confidence threshold À controls the assignment of mixing weights wp. For each source domain, the number of target samples with confidence greater than is recorded, and these normalized values produce Wk. In order determine whether a certain value of À leads to satisfactory choice of mixing weights, it is important to determine whether the high confidence samples are indeed correctly predicted. Figure 4 provides the prediction accuracy on target domain samples the Office-home dataset for different confidence ranges. We consider 5 different confidence probability ranges: 20, 20 40, 40 - 60, 60 - 80, 80 100). observe low-confidence predictions offer poor accuracy for the target domain. For example, in cases when the confidence is less than 0.2, prediction accuracy below 40%. Conversely, for larget samples with a predicted confidence greater than .6, we observe accuracy of more than 90% on all the three tasks of Office-home. This experiments supports our intuition that the amount of high confidence target samples can be used as a proxy for the domain mixing weights wg. We also note the amount of high confidence samples is calculated using the source only models, as adaptation artificially increase confidence across the whole dataset..
100100. We
11

---

Published in Transactions on Machine Learning Research (ottober/202)
Office-31 Accuracy
Image-clef Accuracy
Accuracy
Accuracy
90
90
88
88
86
86
0.0
0.2
0.4
0.6
0.8
0.0
0.2
0.4
0.6
0.8
Confidence Threshold
Confidence Threshold
Figure 5: Results on the Office-31 and Image-clef datasets for different values of the confidence parameter X. The dotted line corresponds to .5 used for reporting results in Table
We further investigate performance regards to the parameter. While Figure 5 we observe target accuracy increase correlated to higher levels of classifier confidence, the amount of high confidence samples proportional to dataset size is equally important for appropriate choice of confidence threshold. Setting the À parameter too high may lead to mixing weights that do not capture model behavior on the whole target distribution, just on a small subset of samples, leading to degraded performance. Conversely, a low value of will lead to results that are equivalent to uniformly combining predictions. Figure portrays both these behaviors on the Office-31 and Image-clef datasets. We observe our choice of .5 is able to obtain best performance on the Office-31 datasct, and close best performance on the Image-clef datasct. We also note the choice of À is relatively robust, as values in the interval [.2. .7] offer similar performance.
gaussians
source-only adapted
Figure 6: UMAP latent space visualization for Qffice-caltech with Amazon as the target. Sources in order: Callech, DSLR, and Webcam. Adaptalion shifts target embeddings towards the GMM distribution.
Our approach attempts to minimize the distributional distance between target embeddings and GMM esti- mations of source embeddings. We provide insight into this process in Figure [] where we reduce the data representation dimension to two using UMAP (Meinnes et al. (0018)). We display GMM samples, target la- tent embeddings before adaptation, and target latent emboddings post-adaptation. For each source domain, the adaptation process reduces the distance between target domain embeddings (yellow points) and the GMM samples (red points). This empirically validates the theoretical justification for our algorithm. Given classifiers trained on the source domains are able to generalize the GMM samples result of protraining, conclude that source-specific domain alignment translates to improved collective performance.
7 Conclusion
We develop privacy-preserving MUDA algorithm based the assumption that input distribution is mapped into multi-modal distribution in embedding space. We maintain cross-domain privacy by minimizing the SWD loss between an intermediate GMM distribution and the target domain distribution in the latent embedding. We then combine the source-specific models according to their reliability. We provide theoretical analysis to justify our algorithm. Our experiments demonstrate that our algorithm performs favorably against SOTA MUDA algorithms using five UDA benchmarks while preserving privacy. Future direction includes considering setting where the target domain shares different classes with each of sources.
12

---

Published in Transactions on Machine Learning Research (ottober/202)
References
Sk Miraj Ahmed, Dripta S. Raychaudhuri, Sujoy Paul, Samet Oymak, and Amit K. Roy-Chowdhury. Un- supervised multi-source domain adaptation without access to source data, 2021.
Kartik Ahuja, Karthikeyan Shanmugam, Kush Varshney, and Amit Dhurandhar. Invariant risk minimization games. In International Conference on Machine Learning, pp. 145-155. PMLR, 2020.
Martin Arjovsky, Léon Botton, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization, 2019.
URL httpp://aarii...rrrab////// 028893
Nicolas Bonneel, Julien Rabin, Gabriel Peyré, and Hanspeter Pfister. Sliced and Radon Wasserstein barycen- ters of measures. Journal of Mathematical Imaging and Vision, 51(1):22-45, 2015.
Konstantinos Bousmalis, Nathan Silberman, David Dohan, Dumitru Erhan, and Dilip Krishnan. Unsuper- vised pixel-level domain adaptation with generative adversarial networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 3722-3723 3731, 2017.
Chaoqi Chen, Weiping Xie, Wenbing Huang, Yu Rong, Xinghao Ding, Yue Huang, Tingyang Xu, and Junzhou Huang. Progressive feature alignment for unsupervised domain adaptation. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 627 636, 2019.
Shuhao Cui, Shuhui Wang, Junbao Zhuo, Chi Su, Qingming Huang, and Qi Tian. Gradually vanishing bridge for adversarial domain adaptation. In Proceedings of the IEEE/CVF conference computer vision and pollern recognition, pp. 12455 12464, 2020.
Sofien Dhouib, levgen Redko, and Carole Lartizien. Margin-aware adversarial domain adaptation with optimal transport. In seiethh International Conference on Machine Learning, 2020.
Ning Ding, Yixing Xu, Yehui Tang, Chao Xu, Yunhe Wang, and Dacheng Tao. Source-free domain adaptation via distribution estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pallern Recognition, pp. 7212-720 2022.
Jiahua Dong, Zhen Fang, Anjin Liu, Gan Sun, and Tongliang Liu. Confident anchor-induced multi-source free domain adaptation. Advances in Neural Information Processing Systems, 34:2848 14:2848-2860, 2021.
Qi Dou, Cheng Ouyang, Cheng Chen, Hao Chen, Ben Glocker, Xiahai Zhuang, and Pheng- Ann Heng. Pnp-adanet: Plug-and-play adversarial domain adaptation network unpaired cross-modality cardiac segmentation. IEFE Access, 7:99065-99076, 2019.
Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In Proceedings of International Conference on Machine Learning, pp. 1180 1189, 2015.
B. Gong, Y. Shi, F. Sha, and K. Grauman. Geodesic flow kernel for unsupervised domain adaptation. In Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on, pp. 2066-2073. IEEE, 2012.
Ian Goodfellow, Jean Pouget-Abadic, Mchdi Mirza, Bing Xu, David Wardo-Farley, Sherjil Ozair, Aaron Courville, and Yoshua. Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems, pp. 2680, 2014.
Y. Grandvalet and Y. Bengio. Semi-supervisel learning by entropy minimization. In Advances in Neural Information Processing Systems, volume 17, pp. 529 536, 2004.
Han Guo, Ramakanth Pasunuru, and Mohit. Bansal. Multi-source domain adaptation for text classification via distancenet-banddis.. In Proceedings of the AAAT Conference on Artificial Intelligence, volume 34, pp. 7830 7838, 2020.
Jiang Guo, Darsh Shah, and Regina Barzilay. Multi-source domain adaptation with mixture of experts. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 4694-4703, 2018.
13

---

Published in Transactions on Machine Learning Research (october/202)
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference. on Computer Vision and Paltern. Recognilion, pp. 770- 778, 2016.
Judy Hoffman, Eric Tzeng, Taesung Park, Jun- Yan Zhu, Phillip Isola, Kate Saenko, Alexei Efros, and Trevor Darrell. Cycada: Cycle-consistent adversarial domain adaptation. In International conference on machine learning, pp. 1989 1998. PMLR, 2018.
Pritish Kamath, Akilesh Tangella, Danica Sutherland, and Nathan Srebro. Docs invariant risk minimiza- tion capture invariance? In Arindam Banerjee and Kenji Fukumizu (eds.), Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, volume 130 of Proceedings of Machine Learning Research, pp. 4069 4077. PMLR, 13 15 Apr 2021. URL https: pttpa://proceedingg mlr presa/
#130/kamath2in
David Krueger, Ethan Caballero, Joern-Henrik Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang, Remi Le Priol, and Aaron Courville. Out-of-distribution generalization via risk extrapolation (rex), 2020. URL
00688
Jogendra Nath Kundu, Naveen Venkat, Rahul M V, and R. Venkatesh Babu. Universal sourco-free domain adaptation. June 2020.
Vinod K Kurmi, Venkatesh K Subramanian, and Vinay P Namboodiri. Domain impression: A source data free domain adaptation method, In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 615-625, 2021.
Chen- Yu Lee, Tanmay Batra, Mohammad Haris Baig, and Daniel Ulbricht. Sliced wasserstein discrepancy for unsupervised domain adaptation. In Proceedings of the IEEE Conference Computer Vision and 2019.
Pallern Recognilion, pp.
Rui Li, Qianfen Jiao, Wenming Cao, Hau-San Wong, and Si Wu. Model adaptation: Unsupervised domain adaptation without source data. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 9641-9650, 2020.
Yitong Li, michael Murias, geraldine Dawson, and David E Carlson. Extracting relation- ships by multi-domain matching. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Newral Information Processing Systems, vol- ume 31. Curran Associates, Inc., 2018. URL httpps://proceedings htarippp.ccppapt//2/////11 pdf.
Jian Liang, Dapeng Hu, and Jiashi Feng. Do really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. In International Conference on Machine Learning, pp. 6028 6039. PMLR, 2020a.
Jian Liang, Dapeng Hu, and Jiashi Feng. Do we really need to access the source data? Source hypothesis transfer for unsupervised domain adaptation. In Hal Daumé III and Aarti Singh (ods.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learn- ing Research, pp. 6028 6039. PMLR, 13 18 Jul 2020b. URL https://proocee liang20a. html
mlr prtr.peess/vi1/
Jian Liang, Dapong Hu, Yunbo Wang, Ran Hc, and Jiashi Feng. Source data-absent unsupervised domain adaptation through hypothesis transfer and labeling transfer. IEEE Transactions on Pallern Analysis and Machine Intelligence, 2021.
Chuang Lin, Sicheng Zhao, Lei Meng, and Tat-Seng Chua. Multi-source domain adaptation for visual sentiment classification. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 2661 2668, 2020.
Mingshong Long, Yue Cao, Jianmin Wang, and Michael Jordan. Learning transferable features with deep adaptation nerworks. In Proceedings of International Conference on Machine Learning, pp. 97-105, 2015.
14

---

Published in Transactions on Machine Learning Research (october/202)
Mingsheng Long, Han Zhu, Jianmin Wang, and Michael I Jordan. Unsupervised domain adaptation with residual transfer networks. In Advances in Newral Information Processing Systems, pp. 136-144, 2016.
Mingsheng Long, Han Zhu, Jianmin Wang, and Michael I Jordan. Deep transfer learning with joint adapta- tion networks. Tn Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 2208-2217. JMLR. org, 2017a.
Mingsheng Long, Han Zhu, Jianmin Wang, and Michael I. Jordan. Deep transfer learning with joint adapta- tion networks. In Doina Procup and Yee Whye Tch (ods.), Proceedings of the 34th International Conference on Machine Learning. volume 70 of Proceedings of Machine Learnång Research, pp. 2208-200222222 2217. PMLR, 06-11 Aug 201 URL http:://proceedings..tt wwapangssmlr...rr... wtt.:prseeos/voo///aa.a httppppaaaaaag.....pppppe///oo/a
Pauline Luc, Camille Couprie, Soumith Chintala, and Jakob Verbeek. Semantic segmentation using adver- sarial networks. In NIPS Workshop on Adversarial Training, 2016.
Leland Mclnnes, John Healy, Nathaniel Saul, and Lukas Großberger. UMAP: Uniform manifold approxima- tion and projection. Journal of Open Source Software, 3(29):861, 2018.
H. Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agüera y Arcas. Communication-effiiene learning of deep networks from decentralized data. 2016. doi: 10.48550/ (0.48550/ARXIV ARXIV.
1602.05629. URL btttps//arxxv.org. wwwp/ag/g///......////////////ea 05629
Pietro Morerio, Jacopo Cavazza, and Vittorio Murino. Minimal-entropy correlation alignment for unsuper- vised deep domain adaptation. In ICLR, 2018.
Xingchao Peng, Qinxum Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 1406-1415, 2019a.
Xingchao Peng, Zijun Huang, Yizhe Zhu, and Kate Saenko. Federated adversarial domain adaptation, 2019b.
J. Rabin, G. Peyré, J. Delon, and M. Bernot. Wasserstein barycenter and its application to texture mixing. In International Conference on Scale Space and Variational Methods in Computer Vision, pp. 435-446. Springer, 2011.
A. Redko, Land Habrard and M. Sebban. Thcoretical analysis of domain adaptation with optimal transport. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 737-753. Springer, 2017.
levgen Redko, Nicolas Courty, Rémi Flamary, and Devis Tuia. Optimal transport for multi-source do- main adaptation under target shift. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 849-858. PMLR, 2019.
Mohammad Rostami. Lifelong domain adaptation via consolidated internal distribution. Advances in Newral Information Processing Systems, 34:11172-11183, 2021a.
Mohammad Rostami. Thansfer Learning Through Embedding Spaces. CRC Press, 2021b.
Mohammad Rostami. Increasing model generalizability for unsupervised domain adaptation. In Proceedings of the Conference on Lifelong Learning Agents, 2022.
Mohammad Rostami and Aram Galstyan. Sequential unsupervised domain adaptation through prototypical distributions. arXio preprint orXiw:2007.00197 2020.
Mohammad Rostami, David Huber, and TsaigChing Lu. A crowdsourcing triage algorithm for geopolitical event forccasting. In Proceedings of the 12th ACM Conference on Recommender Systems, pp. 377-381, 2018.
Mohammad Rostami, Soheil Kolouri, Eric Eaton, and Kyungnam Kim. Sar image classification using few- shot cross-domain transfer learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pallern Recognition Workshops, pp. 0 0-0, 2019.
15

---

Published in Transactions on Machine Learning Research (october/202)
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211 2000:222111221111 2015.
K. Saenko, B. Kulis, M. Fritz, and T. Darrell. Adapting visual category models to new domains. In European Conference Computer Vision, pp. 213 226. Springer, 2010.
Kuniaki Saito, Kohei Watanabe, Yoshitaka Ushiku, and Tatsuya Harada. Maximum classifier discrepancy for unsupervised domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3723 3032, 2018.
S. Sankaranarayauan, Y. Balaji, C. D Castillo, and R. Chellappa. Generate to adapt: Aligning domains using generative adversarial networks. In CVPR, 2018.
Serban Stan and Mohammad Rostami. Unsupervised model adaptation for cont inutl semantic segmentation. In AAAI, 2021.
Serban Stan and Mohammad Rostami. Privacy preserving domain adaptation for semantic segmentation of medical images. In BMVC, 2022.
Baochen Sun and Kate Saenko. Deep coral: Correlation alignment for deep domain adaptation. In European conference computer vision, pp. 443 450. Springer, 2016.
Baochen Sun, Jiashi Feng, and Kate Saenko. Correlation alignment for unsupervised domain adaptation. In Domain Adaptation in Computer Vision Applications, pp. 153 171. Springer, 2017.
Onur Tasar, Yuliya Tarabalka, Alain Giros, Pierre Alliez, and Sébastien Clerc. Standardgan: Multi-source do- main adaptation for semantic segmentation of very high resolution satellite images by data standardization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 192-193, 2020.
Jiayi Tian, Jing Zhang, Wen Li, and Dong Xu. Vdm-da: Virtual domain modeling for source data-fres free domain adaptation. IEEE Transactions on Circuits and Systems for Video Technology, 324))3374--3760, 2022. doi: 20L.1//TTTT..221..1111111
A. Torralba and A. A. Efros. Unbiased look at dataset bias. In Computer Vision and pattern Recognition (CVPR), 2011 IEEE Conference on, pp. 1521 1528. IEEE, 2011.
Eric Tzeng, Judy Hoffman, Kate Sacnko, and Trevor Darrell. Adversarial discriminative domain adaptation. In Proceedings of the IEEE Conference on Compuler Vision and Pallern Recognition, pp- 7167-7176, 2017.
Naveen Venkat, Jogendra Nath Kundu, Durgesh Kumar Singh, Ambareesh Revanur, and R Venkatesh Babu. Your classifier can secretly suffice multi-source domain adaptation. In NewrIPS, 2020a.
Naveen Venkat, Jogendra Nath Kundu, Durgesh Kumar Singh, Ambareesh Revanur, and Venkatesh Babu
R. Your classifier can secretly suffice multi-source domain adaptation. In Newr 2020b. URI proceedings. httpdd/......ss.aaar...aaaaaasse Abst:aatttaaaaaattttttttt...e
Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5018-5027, 2017.
Jindong Wang, Wenjie Feng, Yiqiang Chen, Han Yu, Meiyu Huang, and Philip S Yu. Visual domain adaptation with manifold embedded distribution alignment. In Proceedings of the 26th ACM international conference on Multimedia, pp. 402 410, 2018
Junfeng Wen, Russell Greiner, and Dale Schuurmans. Domain aggregation networks for multi-source domain adaptation. In Hal Daumé III and Aarti Singh (ods.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 10224. PMLR, 13-18 Jul 2020. URL http://proceedings mlr wttpp///ve//eeeee.e../p htmml
16

---

Published in Transactions on Machine Learning Research (october/202)
Ruijia Xu, Ziliang Chen, Wangmeng Zuo, Junjic Yan, and Liang Lin. Deep cocktail network: Multi-source unsupervised domain adaptation with category shift. In Proceedings of the IEEE Conference Compuler Vision and Pattern Recognition, pp. 3964 3973, 2018.
Ruijia Xu, Guanbin Ti, Jihan Yang, and Liang Lin. Larger norm transferable: An adaptive feature norm approach for unsupervisod domain adaptation. In Proceedings of the IEEE/CVE International Conference on Compuler Vision, pp. 1426 1435, 2019.
Zengiiang Yan, Jeffry Wicaksana, Zhiwei Wang, Xin Yang, and Kwang- Swang-Ting Cheng. Variation-aware feder- ated learning with multi-source decentralized medical image data. IEEE Journal of Biomedical and Health Informatics, 25(7):2615 (0:):261-22222 2628, 2021. doi: 10.
2020.3040015.
Baoyao Yang, Hao- Wei Yeh, Tatsuya Harada, and Pong C. Yuen. Model-induced generalization error bound for information-thoortiicc representation learning in sourco-data-fres unsupervised domain adap- (alion. IEEE Transactions on Image Processing, 31:419 432, 2022. doi: (010099TPP.202..31111
Timothy Yang, Galen Andrew, Hubert Eichner, Haicheng Sun, Wei Li, Nicholas Kong, Daniel Ramage, and Françoise Beaufays. Applied federated learning: Improving google keyboard query suggestions. arXiv 029002 2018.
preprint
Hao- Wei Yeh, Baoyao Yang, Pong C Yuen, and Tatsuya Harada. Sofa: Source-data-fres feature alignment for unsupervised domain adaptation. In Proceedings of the IEEE/CVE Winter Conference on Applications of Computer Vision, pp. 474-483, 2021.
Yuchen Zhang, Tianle Liu, Mingsheng Long, and Michael Jordan. Bridging theory and algorithm for domain adaptation. In International Conference on Machine Learning, pp. 7404-7413, 2019.
Han Zhao, Shanghang Zhang, Guanhang Wu, José MF Moura, Joao P Costeira, and Geoffrey Gordon. Adversarial multiple source domain adaptation. Advances in neural informalion processing systems, 31: 8559 8570, 2018.
Sicheng Zhao, Guangzhi Wang, Shanghang Zhang, Yang Gu, Yaxian Li, Zhichao Song, Pengfei Xu, Runbo Hu, Hua Chai, and Kurt Keutzer. Multi-source distilling domain adaptation. In Proceedings of the AAAI Conference Arlificial Intelligence, volume 34, pp. 12975 12983, 2020.
Yongchun Zhu, Fuzhen Zhuang, and Deqing Wang. Aligning domain-specific distribution and classifier for cross-domain classification from multiple sources. In Proceedings of the AAAT Conference on Artificial Intelligence, volume 33, pp. 5989-5996, 2019.
17

---

Published in Transactions on Machine Learning Research (October/2D) 2022)
A Appendix
A.1 Proof of Theorem 5.1
We offer a proof for Theorem 5.1 from the main paper. Consider the following results.
Theorem 2. Theorem from Redko & Sebban 2017
Let h be the hypothesis learnt by model, and h² the hypothesis that minimizes es + ET. Under the assumplions described in our framework, consider the exislence of N source samples and M largel samples, with empirical source and target distributions üs and fiT in Rd. Then, for any d' > d and Ç < v2, there exists a constant number No depending on such that for any & > and min(N, M) 2 No max(5" (d +2) ,2),1) 1) with probability al least E,, the following holds:
er(h) Ses(h) ((rr,As)+
/ (21ogt /9)
V
M M
(7)
+ N
+ ec(1))
The above theorem provides upper bound the target error with respect to the source error, the distance between source and target domains, a term that is minimized based on the mumber of samples, and constant = =s(hh) eT (h*) describing the performance of an optimal hypothesis on the present set of samples.
We adapt the result Theorem to provide an upper bound in our multi-source setting. Consider the following two results.
Lemma 1. Under the definitions of Theorem
W(As & W(As.Ap) + W(Pp.
(8)
where pu the GMM distribulion learnal for source domain S.
Proof. As W is distance metric, the proof is immediate application of the triangle inequality.
Lemma 2. Let h be the hypothesis describing the multi-source model, and let h2 be the hypothesis learnt for source domain k. If (h) is the error function for hypothesis h domain T, then
eT(h) < Σ wger (hx)
(9)
Proof. Let p(X) En= wnfa(X (X) with Σ wy 1, wy 0 be the probabilistic estimate returned by our model for some input X, and let y be the label associated with this input. The proof for the Lemma proceeds follows
18

---

Published in Transactions on Machine Learning Research (October/2D) 2022)
er(h) = Ecx 1,)
E(XWI~ - log (x)
-(XNNNT log( Σ wx (x)x(())lu
< E(x,)~T
wh( log (x)) Jensen's Ineq.
Wk (f
wker(ha (hx)
We now exlend Theorem 2 as follows
Theorem 3. Solti-Souure unsupervised enror bound (Theorem 5.1 from the main paper)
Under the assumptions of our framework and using the definitions from Theorem
er(h) (h) S Σ (yses (hx) + W (pp. + W WsPpe
km1
(10)
(2log (I I N: + M ec2 (bf)
where Pk is the sample GMM distribution learnt for source domain k, Ne is the sample size of domain k, Ck the combined error loss with respect to domain k, and hi the optimal model with respect this loss. Proof.
a wher (hx) From Lemma
eT (h) S
k=1
Wkk (es, (Iaa) + W(āT, Pss)+
(2 log( {
(()) No (M)
ec,, (h1)) by Theorem 2
wx (a(es, (Ix)) + W(ĀT. )p.) ) W(pp,
(2log( (/)() No + +V M Ec, (ht)) by Lemma 1
A.2 Experimental parameters
We use the Adam optimizer with source learning rate of le for each source domain for all datasets. Target learning rates are chosen between le - 5 and le 7 for adaptation. The number of training iterations
19

---

Published in Transactions on Machine Learning Research (tttbber/2022
and adaptation iterations differs per dataset: Office-31 (12k, 48k), Domain-net (80k, 160k), Image-clef (4k, 3k) Office-home (40k, 10k), Office-CalTech (4k, 6k). The training batch size is either 16 or 32, with little difference observed between the two. The adaptation batch size is usually chosen around 10x the number of classes for each dataset, to ensure a good class representation when minimizing the SWD distance. The network size the same across all datasels, with the SWD minimization space being 256 dimensional. The above mentioned parameters are also provided the config.py py file the codebase.
A.3 Additional Results
We extend the runtime results from Section 6.3 of the main paper to the Domain-Net dataset. As seen in Figure g the result in Figure / share similar trend. After the start of the adaptation process target accuracy improves for each source trained model. Additionally, pooling information from each of the five source domains leads to improved overall predictive quality of the model.
1.0
adapt. storts
edept Alatis.
1.0
wwwooi. Alotl
Accuracy Normalzeed Loss
Accurany Norraalzzed Loss
Accuracy Normallzed Loos
0.8
0.8
0.8
0.6
0.6
0.6
0.4
0.4
0.2
0.0
0.0
0.0
8
12
12
12
Eeeations [tens of Uhousands)
Ieeraions (tens of thousands)
lterations (tens of thousands]
Adapt, stauts.
adapt. starts
W2-Luss
CE-Loss
0.8
Accuracy Normalzed
Accuracy Nomalized alized
Target accuracy
0.6
0.6
combined
combined
0.4
0.4
0.2
0.2
0.0
0.0
9 10 Iterations thousands)
2
10
Iterations thousanes)
Figure 7: Effect of the adaptation process on the Domain-Nee Net dataset, where sketch is the target. Sources are order Clipart, Infograph, Painting, Quickdraw, Real.
In Table 7 we report results for different choices of the y parameter the Office-31 datasct. The main results in Table [ are generated for the choice of .22. We test wider range of values to identify how robust this parameter needs to be. We observe large values of y put too much emphasis on the entropy loss, harming the optimal transport based distribution matching. Small values of y share a similar problem, as we lose the soft-clustering benefit provided by the entropy loss. Overall we notice different tasks have different high performing y ranges For example, on the + D task, any choice of the parameter greater than 0.02 lcads to best performance, while on the A task y less than 0.04 offer a competitive range of values. While setting task specific 7 values may lead to improved performance, we show that the more robust setting where choose the same 7 per dataset still works reasonably well in practice.
We give justification for the beneficial effect of using all the available souree domains for inference. In Table we present results obtained from successively adding source domains to our ensemble, for the Office-home dataset. We also include single best performance as a baseline, representing the highest target accuracy obtained across source domains. The rows of the table correspond the four MUDA problems considered for Office-home, while the columns correspond the number of source domains considered in ensembling. For example, the ACP R task with 2 sources considers the problem AC - R. Similar to Figures 3 and
20

---

Published in Transactions on Machine Learning Research (october/202)
Method
+
-W
+A
Avg.
y=1
99.8
97.6
64.8
87.4
y= .4
99.8
98.7
72.8
90.4
7
99.8
98.4
74.2
90.8
7=
99.8
98.5
75.4
91.2
7= 01
98.8
97.8
76.3
91
7= le-3
89.3
92.4
74.4
85.4
7= 1e-4
87.4
90.7
74.2
84.1
Table Performance analysis for different values of y
we observe that ensembling is superior to single best performance on all tasks. Additionally, our mixing strategy proves to be robust with respect to negative transfer, as adding new domains is always beneficial the reported performance.
Method
Single best
First src, domain
First 2 src. domains
All 3 src. doomais
ACP R
81.1
81.2
82.3
83.7
ACR
83.1
77.1
77.9
83.3
APR
60.1
PCR +
67.2
67.3
67.6
68.2
Table 8: Performance analysis when source domains are introduced sequentially,
We investigate the representation quality of the GMM distribution surrogate for the source distribu- tion. Note that having GMMs that good approximation of the source latent features is crucial for approach. In Figure 8 we present visualized data representations for the estimated GMMs and the source domain distributions for the Image-clef dataset. We note that for both source domains, their latent space distributions after pretraining are multi-modal distributions with 12 modes, each corresponding to one class. This observation confirms that we can approximate the source domain distribution with GMM with modes. We also note that for both source domains the estimated GMM distribution offers close approxi- mation of the original source distribution. This experiment empirically validates that the third term in Eq. 4 these intermediate cross-domain distributions.
is small in practice and
src. embeddings gaussians
Figure 8: Source and GMM embeddings for the Image-clef dataset with Pascal and Caltech sources. For both datasets, the GMM samples closely approximate the source embeddings.
We additionally analyze the latent source approximation used for adaptation, and compare it to the scenario when fine tuning the model on the source domain can be done after adaptation starts. We present these findings in Figure g When privacy is not a concern and we have access to all source data, there is no need to approximate the source distribution during adaptation, as we can just directly use the source samples. For empirical exploration, results on how the latent distribution changes when source access is permitted during adaptation (corresponding the SS + SW case in Table 3) is presented in Figure 9 We notice that training progresses, the means slightly shift. Visually however, the change compared to using the latent
21

---

Published in Transactions on Machine Learning Research (october/202)
space only after source training is negligible. This supports the idea that the GMMs learned at the end of source training will provide sulficient approximation of the source distribution.
Figure 9: Latent distributions for two domains the Office-31 dataset when considering the D + and W + tasks. Fach color gradient represents a latent feature distribution snapshot of the source domains. Darker colors correspond later training iterations, lighter colors earlier iterations.
22


Reviews:
[{"review_id": "lsWPdoNPWC", "reviewer": "Reviewer_rxyr", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. (Motivation) The problem addressed by the paper is indeed important in the field of MUDA and the privacy-protecting setting of the study is of practical use. The authors have also made this point clear and convincing enough through the examples of mobile keyboard predictions and medical image processing applications.\n2. (Privacy) The major contribution lies in how the proposed method avoids using inter-domain statistics, such as the data, distribution, or representations in the embedding space, as many of the existing MUDA methods would do. The proposed method achieved model adaptation using only the means/covariances as the only cross-domain information. \n3. (Scalability) The proposed method is also robust to changes in the accessibility of the source domains, such as increasing or decreasing the available source domains. End-to-end retraining is avoided as the model-ensemble structure made each source domain work in a plug-in mode.\n4. (Theoretical) Theoretical analysis is provided to show the upper bound of the target domain error.\n5. (Empirical) The experiment results show the effectiveness of the proposed method in various datasets.\n\nWeaknesses:\n\n1. (Presentation) There is too little information about the key technologies used in this paper disclosed in the abstract. Some high-level summarization should be mentioned to improve the readability, such as the usage of GMM, model ensembling, etc. Also, Figure 1 looks a bit scrambled with too many elements of different styles, and thus needs revision. \n2. (Presentation) All the texts in the figures (legends, labels, axis ticks, annotations) are suggested to be of similar font size to that of the texts in the paper. The current font size is not consistent (see legends in Figures 4, 6, 7, etc). Also, the figures are suggested to use vector diagrams, which can avoid low-resolution problems (Figure 7). \n3. (Privacy) The choice of the model ensemble is a double-edged sword. It is used to protect privacy initially. However, I am afraid in such a scenario, since the model for each source domain is separate, wouldn’t the privacy attack against the model for a particular source domain be easier to perform compared to those MUDA methods using only one model for all the source domain?\n4. (Time and Memory Consumption) It is self-explanatory that the inference time, as well as the memory consumption of the proposed method, is proportionate to the number of the source domains. Take the membership inference attack (MIA) [1, 2] as an example, which is a very commonly used privacy attack. Many MIA is based on the confidence or the entropy of the output, which are both used in the proposed method. Thus, I wonder if the proposed method would suffer from MIA more compared to the non-model-ensemble-based methods.\n5. (Hyper-parameter Sensitivity) The paper criticized the adversarial-learning-based UDA methods for their high dependence on optimal hyper-parameters. In this paper, it seems the newly introduced hyper-parameters $\\lambda$ exert great influence on the performance as well (Figure 5) and we do not know the influence of the $\\gamma$ for now. \n6. It is claimed that the proposed method is friendly to source domain change, while there is no empirical evidence in the paper. I would like to see how the performance will be if more source domains are added to the well-trained model of SMUDA.\n7. (Related Works) It seems the proposed method has some similarities to invariant risk minimization (IRM) [3-5], and I suggest the authors look at them as well as add some discussion on the similarity/difference.\n\n> [1] Carlini N, Chien S, Nasr M, et al. Membership inference attacks from first principles[C]. 2022 IEEE Symposium on Security and Privacy (SP). IEEE, 2022: 1897-1914.\n>\n> [2] Shokri R, Stronati M, Song C, et al. Membership inference attacks against machine learning models[C]. 2017 IEEE symposium on security and privacy (SP). IEEE, 2017: 3-18.\n>\n> [3] Arjovsky M, Bottou L, Gulrajani I, et al. Invariant risk minimization[J]. arXiv preprint arXiv:1907.02893, 2019.\n>\n> [4] Ahuja K, Shanmugam K, Varshney K, et al. Invariant risk minimization games[C]. International Conference on Machine Learning. PMLR, 2020: 145-155.\n>\n> [5] Krueger D, Caballero E, Jacobsen J H, et al. Out-of-distribution generalization via risk extrapolation (rex)[C]. International Conference on Machine Learning. PMLR, 2021: 5815-5826.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a multi-source unsupervised domain adaptation (MUDA) method targeting the use case, where (both source and target) datasets are distributed among independent entities and can not be shared across some central accessibility due to privacy protection. To adapt to each source domain while preserving privacy, the proposed method chooses to use a model ensemble tactic, which approximates the distributions of the source domains in the embedding space and uses these distributional estimations for domain alignments through Gaussian Mixture Models (GMM). More specifically, for each source domain, a model is independently trained along with the calculation of the pair of mean and covariance parameters in the latent space. The models are then adapted to pseudo-domains through the GMM distribution in the embedding space and conditional cross-entropy. The final output of the input from the target domain will be the weighted sum of the models from all the source domains based on the confidence scores of the prediction. Both theoretical and empirical results show the effectiveness of the proposed method. Regarding privacy, this method achieves using only the latent means and covariance of the estimated GMMs as the cross-domain information and data samples are never shared between any two domains during either the training or adaptation phase.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "In general, the requested changes all correspond to the weaknesses listed above.\n\n1. (Weakness 1 & 2) Please refer to weaknesses 1 and 2 to improve the readability of the figure or the introduction of the paper.\n2. (Weakness 3) It is recommended to add some discussion and analysis on the performance of different methods against possible privacy attacks (and experiments if applicable).\n3. (Weakness 5) It is recommended to add some ablation studies to study the sensitivity of the proposed method to the choice of hyper-parameter $\\gamma$. If possible, please also add some discussion of the sensitivity of the proposed method to hyper-parameter choices, as was mentioned in the literature review in the first place.\n4. (Weakness 6) Please add experiments to support your argument on source domain change.\n5. (Weakness 7) Please add some discussions on invariant risk minimization.\n", "broader_impact_concerns": "No concerns on the broader impact from the reviewer.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "90J3n7Icw0", "reviewer": "Reviewer_7zKM", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1.This paper is over-all well written and the references seem sufficient.\n\n2.It is sort of variable to enable the independent adaptation from each source domain.\n\n3.Necessary experiments are conducted to support the proposed idea.\n\nWeaknesses:\n\n1.The biggest concern is that the novelty is weak, compared to a related work DECISION （Ahmed et al. 2021), which also explores MUDA without the access of source domain data. The only difference from DECISION is that this work claims the benefit of independent adaptation from each source domain. This results in the next issue.\n\n2.The motivation of performing independent adaptation is still unclear or not well-grounded. The authors claim that this setting is constrained yet practical, which seems quite subjective. DECISION assumes the access of all pretrained models where the privacy of all source data is still maintained. More importantly, DECISION allows the end2end finetuning on the target data (including the retraining of the feature extractors of all source domains and the combination weights), leading to better performance than the proposed model in this paper (as illustrated in Table c, d, e). While the authors state that this work allows for accumulative learning from several domains when new source domain comes, there is no experimental justification on this point. \n\n3.GMM is still a weak surrogate distribution for more practical and complicated datasets. It is expected that GMM is unable to encode the sufficient patterns within the input data. What if we change GMM to other choices? Such as GAN?\n\n4.As stated above, the combination weights (Eq. 6) are hand-crafted, leading to suboptimal performance compared to DECISION where the feature extractor and the weights are learned simultaneously and end to end. \n\nWriting issues: \n\n1.\tThe abstract and introduction require rewriting, as main important features such as GMM and the difference compared to DECISION are not included.\n\n2.\tEq. (4) is unclear. More details needed.\n\n3.\tEq. (6) is confusing. The function f should output a label other than the logit. And what about multi-class classification? Not just binary classification. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies the task of multi-source unsupervised domain adaptation (MUDA) where the data of each source domain are assumed to be unavailable to keep privacy during adaptation. For this purpose, the authors propose to learn the latent representation of each source domain via GMM, and then apply the surrogate distribution for the alignment between the source and target domain. The adaptation for each source domain is carried out independently and the predictions of target samples are obtained via a hand-crafted convex combination of logits of all adapted models. Experiments are conducted on several benchmarks with certain ablations studies performed as well. ", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "See the weakness part above.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "fi6jAKKTD7", "reviewer": "Reviewer_Q4QX", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall the paper addresses an important problem in MUDA and proposes an effective algorithm to solve the problem. \nThe paper extensively ablates different hyperparameters used in the algorithm and presents a nice comparison that justifies the different design choices. \nThe online nature of the algorithm, in that it can incorporate new source domains without having to retrain the entire pipeline gives a computational advantage to the method.\n\nHowever, I would like to get an idea of how worse the algorithm performs in comparison to the scenario when privacy is not enforced. In particular, how different is the accuracy of the model on the target domain both theoretically and empirically, if you could use all sources and the target simultaneously? \n\nAnother concern is around the assumption of feature space learned using individual sources to be a gaussian mixture model. Although the last paragraph above the conclusion, provides an explanation of why this works, I am more interested in understanding how practical is this assumption. Does this hold for most datasets or is it being used here to make the computation easy.\n\nRelated to the previous concerns, the algorithm does not share data across the domains but only the estimated means and variances. \nHow different are the means and variances of the domains estimated with and without privacy concerns? \nDoes the privacy-preserving MUDA suffer significantly in comparison to vanilla MUDA algorithms? If so, is it possible to narrow this gap if we computed the means and variances of all source domains (just as is done in lines 3 and 4 of Alg 1) but used them together to do the adaptation step? Particularly, instead of getting \\theta_k^A on line 6 for every k, we only got one \\theta_A = Adapt(all \\theta_k, all A_k, T, L, \\gamma). \n\nLastly, is there a direct way of measuring privacy leaks? Is it possible that sharing means and variances also lead to leaks in privacy?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes an algorithm for multi-source unsupervised domain adaptation (MUDA) for the scenarios when source and target domain data is not available at the same time due to privacy and security concerns. \n\nThe algorithm starts by learning a model for each source domain separately and assuming the representation space of the encoder to be a gaussian mixture model estimates their means and variances, separately for each source domain (regardless of other sources and the target).\n\nTo address the concerns of privacy the domain adaptation part is done based on the learned means and variances of the individual source domain and the target domain. \n\nAfter the adaptation part of each source and the given target is complete the final classifier is obtained by using a weighted combination of the individual classifier. \n\nA confidence-based technique is used to combine the classifiers for prediction of a point at test time. Using ablation studies the paper shows confidence-based scheme to be the best among different possible combination schemes. \n\nExtensive experiments and ablation tests have been performed in the paper to clarify the effectiveness of the proposed algorithm and the different design choices used.\nThe paper also shows that the proposed algorithm is theoretically motivated and minimizes the upper bound on the target domain error.\n\n", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Answers to questions raised in the Weakness/Concern section.\n\nIt would be great to see some empirical results or discussion on the differences between the performances of the algorithms with and without privacy to estimate how much performance is lost by enforcing privacy constraints. I would also be interested in seeing a discussion on the difference in the upper bound on the target domain error with and without privacy constraints.\n\nAdditional discussion on why the GMM choice is suitable and practical would also be beneficial to the audience trying to incorporate privacy in their methodology.\n\nLastly, some discussion about how privacy leaks are measured should be added. It would be beneficial to add previous works on this topic too if there are any that the authors are aware of.\n\nMinor: \n1. Change Eq. 3 to add L and refer to it on line 14 of Alg. 1\n2. Typo in spelling above eq 5.\n", "broader_impact_concerns": "-", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "CQ1yC6VWya", "reviewer": "Reviewer_TfHu", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n\n1. This paper addresses the multi-source UDA problem, where source data is not required when training models in the target domain. This is a practical approach for privacy-sensitive issues such as medical data.\n2. The model adaptation between the source and target is performed independently for each source domain so that if the source domain increases or decreases, only the changed source domain needs to be adapted.\n3. Theoretical analysis has been performed and shows that the proposed algorithm minimizes the upper bound of errors in the target domain.\n4. The effectiveness of the proposed method is verified using five standard datasets in UDA.\n5. The proposed algorithm is simple and easy to implement.\n6. The paper is well written and easy to understand.\n\nWeakness\n1. The proposed method adapts single-source Source-free UDA (SFUDA) to each source domain and computes an ensemble of them. Therefore, the novelty of the proposed method is not significant in the general framework.\n2. Source-free UDA by estimating the probability density distribution of the source domain and aligning that distribution with the target distribution has already been proposed  [c, d, e, f, g in the next section], so the method is not novel at this point.\n3. The target model copies the parameters of the source model in the early stages of learning, which is somewhat problematic from a privacy perspective since the source's post-training parameters are accessible.\n4. The number of models increases with the number of source domains, making model size inefficient. In addition, when data from the target domain is input, the inference must be performed in proportion to the number of source models, which increases inference time in proportion to the # of source domains.\n5. In the experimental results in Table 1, the method DECISION performs better than the proposed method, especially for more complex data sets such as Office-home and DomainNet.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, the authors propose a new multi-source unsupervised domain adaptation (MUDA) method that does not require source data when training the model in the target domain, thus avoiding issues such as data privacy. The proposed method performs domain adaptation in each source and target domain. The final output is obtained by computing a weighted sum of the resulting adapted models for each source domain based on their confidence levels. During the adaptation of each source and target domain, the distribution of the source domain data is modeled as a mixture of Gaussians to avoid direct access to the source domain data, and domain adaptation is performed to align the estimated source distribution with the target distribution. The authors also show that the proposed method minimizes the upper bound on errors in the target domain through theoretical analysis. Furthermore, extensive experiments verify the effectiveness of the proposed method.\n", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "1. Many single-source SFUDA methods have been proposed. This paper needs a comparison with the ensemble models of existing single-source SFUDA methods that adapt the target model to each source domain.\n2. This paper should show how much performance improvement the proposed SFUDA (a part of the proposed model w/o ensemble) between each source and target has over other single-source SFUDA methods.\n3. Since other multi-source SFUDA methods exist besides DECISION [a], this paper needs at least a qualitative comparison with them (e.g., [b]).\n4. Methods for domain adaptation by estimating the probability density distribution of the source domain and aligning that distribution with the target distribution have already been proposed [c, d, e, f, g]. This paper should compare the proposed one at least qualitatively with them.\n5. In training the target model, the initial parameter values are copied from the parameters of the source model. The reviewer would like to know if there are any privacy issues.\n6. This paper should describe the values of the hyperparameters when the results of TABLE 1 are obtained. The readers would like to know how much optimization is done for each dataset.\n\n\n- [a] Ahmed, Sk Miraj, et al. \"Unsupervised multi-source domain adaptation without access to source data.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.\n- [b] Dong, Jiahua, et al. \"Confident anchor-induced multi-source free domain adaptation.\" Advances in Neural Information Processing Systems 34 (2021): 2848-2860.\n- [c] (align with GAN-generated source-like data) Kurmi, Vinod K., Venkatesh K. Subramanian, and Vinay P. Namboodiri. \"Domain impression: A source data free domain adaptation method.\" Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2021.\n- [d] (GMM-based) Yeh, Hao-Wei, et al. \"Sofa: Source-data-free feature alignment for unsupervised domain adaptation.\" Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2021.\n- [e] (GMM-based) Yang, Baoyao, et al. \"Model-Induced Generalization Error Bound for Information-Theoretic Representation Learning in Source-Data-Free Unsupervised Domain Adaptation.\" IEEE Transactions on Image Processing 31 (2021): 419-432.\n- [f] (GMM-based, means and variances are computed by weights of source classifier) Tian, Jiayi, et al. \"Vdm-da: Virtual domain modeling for source data-free domain adaptation.\" IEEE Transactions on Circuits and Systems for Video Technology (2021).\n- [g] (GMM-based, means and variances are computed by target features from source model) Ding, Ning, et al. \"Source-Free Domain Adaptation via Distribution Estimation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n", "broader_impact_concerns": "There are no concerns on the ethical implications of the work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: While all reviewers see some merits in this work, the authors' response and revision did not fully address the reviewers' comments. Therefore, I demand necessary changes and clarification on the following two action items before accepting this submission.

1. Justification on why the accuracy would not be degraded when the privacy constraint is enforced. There are known trade-offs in utility and privacy for many machine learning tasks and models, so the reported results need proper justification on why the proposed method is an exception (Please see Reviewer Q4QX's comments for details)

2. Clarification of differences to existing works (Please see Reviewer TfHu's comments for details): 

The paper states, "We address the challenge of data privacy for MUDA by maintaining full privacy between pairs of source domains, and between source and target domains." as one of the main contributions. However, it is difficult to say maintaining 'full' privacy because the parameters of the source model can be accessed during model initialization. SFUDA methods that do not access source model parameters have already been proposed (e.g., [d, e]). In this respect, the paper may be overstating its contribution.

The second contribution is that "We propose an efficient distributed optimization process for MUDA to process each dataset locally while We propose an efficient distributed optimization process for MUDA to process each dataset locally while encoding high-level learned knowledge in a shared latent embedding space." However, the novelty of the proposed method is questionable because a method for encoding high-level learned knowledge in a shared latent embedding space using GMMs has already been proposed for SFUDA [c, d, e, f, g], similar to the proposed method.
Decision: Accept (poster)