Confident Anchor-Induced Multi-Source Free Domain Adaptation
Authors: Jiahua Dong
Zhen Fang
Anjin Liu
Gan Sun
Tongliang Liu
Venue: NeurIPS 2021 Poster (2021)
Keywords: Transfer Learning
Unsupervised Learning
Multi-Source-Free Domain Adaptation

Abstract: Unsupervised domain adaptation has attracted appealing academic attentions by transferring knowledge from labeled source domain to unlabeled target domain. However, most existing methods assume the source data are drawn from a single domain, which cannot be successfully applied to explore complementarily transferable knowledge from multiple source domains with large distribution discrepancies. Moreover, they require access to source data during training, which are inefficient and unpractical due to privacy preservation and memory storage. To address these challenges, we develop a novel Confident-Anchor-induced multi-source-free Domain Adaptation (CAiDA) model, which is a pioneer exploration of knowledge adaptation from multiple source domains to the unlabeled target domain without any source data, but with only pre-trained source models. Specifically, a source-specific transferable perception module is proposed to automatically quantify the contributions of the complementary knowledge transferred from multi-source domains to the target domain. To generate pseudo labels for the target domain without access to the source data, we develop a confident-anchor-induced pseudo label generator by constructing a confident anchor group and assigning each unconfident target sample with a semantic-nearest confident anchor. Furthermore, a class-relationship-aware consistency loss is proposed to preserve consistent inter-class relationships by aligning soft confusion matrices across domains. Theoretical analysis answers why multi-source domains are better than a single source domain, and establishes a novel learning bound to show the effectiveness of exploiting multi-source domains. Experiments on several representative datasets illustrate the superiority of our proposed CAiDA model. The code is available at https://github.com/Learning-group123/CAiDA.

Confiden--Ancho--nndueed Fre Domain Adaptation
Jiahua Dongl.2,, Zhen Fang", Anjin Liu', Gan Sun't, Tongliang Liu
State Key Laboratory of Roboties, Shenyang Institute of Automation, Chinese Academy of Sciences.
University of Chinese Academy of Sciences.
e DeSI Lab, AAII, University of Technology Sydney. "TML Lab, University of Sydney. {dongjiahual fzjlyt, sungan1412/@gmaill com, anjin. liubuts. edu. au, tongliang. liudsydney.y.oo edu.au
Abstract
Unsupervised domain adaptation has attracted appealing academic attentions by transferring knowledge from labeled source domain to unlabeled target domain. However, existing methods assume the source data drawn from single domain, which cannot successfully applied to explore complementarily trans- ferable knowledge from multiple source domains with large distribution discrep- ancies. Moreover, they require access to source data during training, which are inefficient and unpractical due privacy preservation and memory storage. To multi-
address these challenges, we develop novel source-free Domain Adaptation (CAiDA) model, which is pioneer exploration of knowledge adaptation from multiple source domains to the unlabeled target domain without any source data, but with only pre-trained source models. Specif- ically, source-specific transfcrable perception module is proposed to automati- cally quantify the contributions of the complementary knowledge transferred from multi-source domains to the target domain. To generate pseudo labels for the target domain without access the source data, we develop a confident-anchor induced pseudo label generator by constructing confident anchor group and signing each unconfident target sample with a semantic-nearest confident anchor. Furthermore, dlass-relationship-aveee consistent inter-class relationships by aligning soft confusion matrices across do- mains. Theoretical analysis answers why multi-source domains are better than single source domain, and establishes a novel learning bound to show the effec- tiveness of exploiting multi-source domains. Experiments several representa- tive datasets illustrate the superiority our proposed CAiDA model. The code is available srttps://gittee www..//aaningggooonooooo httpaammmmnnnggruppp//////AA
consistency loss is proposed to preserve
Introduction
captures transferable knowledge from labeled data in source domain to classify unlabeled data in target domain. UDA has achieved remarkable successes in many applications, e.g., object detection [23], medical diagnose [S, [2], sentiment anal-
Unsupervised Domain Adaptation (UDA)
PM 5y [X
ysis [62]], etc. Generally, most of existing methods employ adversarial learning [17]] to encourage the
learned souree and target features to be indistinguishable from each other [TO, 15], or minimize the
distribution discrepancy across domains by matching the statistical moments of distributions [II]].
"Equal contributions
t Corresponding author
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

However, the above-mentioned methods have strong assumption that the source data are merely drawn from a single domain. Unfortunately, the source data are often collected under different deployed environments (i.e., multiple source domains with large distribution discrepancies) in real- world applications, which makes them difficult to explore complementarily transferable knowledge from the multi-source domains for target prediction. To achieve this, Multi-Source Domain Adap- tation (MSDA) [22 62) bi]] is proposed to match the features across domains and then quantify the contributions of source domains [2. 911 60]. Additionally, [29. [88 aim to weight the source contribu- tions by normalizing the distance similarities between source and target domains.
Unfortunately, recent MSDA methods [29 34, IT A require massive labeled source data when adapting source domains to the target domain. This could make them inefficient and unpractical in real-world applications with sensitive information (e.g., medical diagnosis [T2]] and recommendation
system [241), due to privacy preservation issues, storage and security concerns 15551,,, To this end, a new challenging and practical problem named Multi-Source--ie Free Domain Adaptation (MSFDA) is researched, which explores transferable knowledge from multiple source domains to target domain with only pre-trained source models and without access to any source data. The trivial solutions for tackling MSFDA via using existing single-source- free domain adaptation methods [255 G0 6% 57]]
are to adapt each source model individually and simply take average prediction of source models. However, they cannot explore the contributions of the complementary information transferred from different source domains, due to the lack of source data. Therefore, tackling the MSFDA problem is a challenging but researeese task.
To address the MSFDA problem, we develop a novel Confident-AAchor-inddeee multi-source- freee Domain Adaptation (CAiDA) model, which is a pioneer exploration to capture transferable infor- mation from multiple source models to promote target prediction without access to source data. Specifically, a source-specific transferable perception module is designed to calibrate the contribu- tions of the transferability from multiple source domains. We develop confidentaanchorrinduceel pseudo label generator to mine pseudo labels for the unlabeled target data, by incorporating with the quantified source transferability contributions. We construct confident anchor group to assign each target sample with a semantic-neare confident anchor, and perform feature augmentation be- tween them to generate confident target pseudo label. A clasx-relationship-avw proposed to ensure the semantic consistency of underlying inter-class relationships across domains via the alignment of soft confusion matrices. Furthermore, based some mild assumptions, the- oretical analysis guarantees that multiple source models could help generate more reliable pseudo labels. Our theoretical analysis also provides novel learning bound for MSFDA, which reveals that multiple source models help achieve a tighter generalization error bound for the target domain. We verify the effectiveness of our proposed model via comparison experiments on benchmark datasets. The main contributions of our work are summarized as follows:
consistency loss is
multi-source-ffe Domain Adaptation (CAiDA) model to explore transferable knowledge from multiple source domains to assist target prediction with pre-trained source models and without access to source data. To our best knowledge, this paper is a pioneer exploration of multi-source- domain adaptation in the field of transfer learning.
We develop novel
We propose a novel MSFDA theory, which shows that multiple source models could improve the possibility of obtaining more reliable pseudo labels under some mild assumptions. Our theoretical analysis also provides novel generalization bound for MSFDA to show the effect of multiple source models. This novel bound implies a positive answer to the solvability of MSFDA problem.
A source-specific transferable perception module and a class-relationshipp- are designed quantify the contributions of the transferability of multiple source domains and ensure the semantic consistency of underlying inter-class relationships across domains, respectively.
consistency loss
Based on the confident pseudo labeling strategy in theoretical analysis, a aunfident-anchor-innuee pseudo label generator is proposed to generate pseudo labels for the target domain by establishing a confident anchor group and assigning each target sample with semantic-nearest confident anchor.
2 Related Work
Unsupervised Domain Adaptation aims to borrow transferable knowledge from source domain to promote the prediction of the unlabeled target domain. After Hoffman et al. [22]] introduce ad-

---

versarial learning [| into domain adaptation, enormous adversarial baaed methods [8, I1l D5 59
are proposed to perform feature-level or pixel-level distribution alignment. Besides, some mo- ment matching-based methods [04, 35, [I]] focus on matching the distribution statistical moments
at different orders to minimize the distribution discrepancy across domains. Furthermore, some re-
searches design the adversarial dropout [28], batch normalization [52]] and auxiliary reconstruction tasks [[]][]] to narrow the domain discrepancy. Unfortunately, the methods mentioned above assume massive labeled source data are available. This is unpractical due to privacy and security concerns.
Source-Free Domain Adaptation (SFDA) [E] is studied to tackle the above challenge. A common strategy in SFDA methods is to mine the confident pseudo labels for target domain. To this end, [[3] uses self-supervised pseudo labeling strategy, and [25]] designs a confidence-based sample filtering method. [+2] alleviates the negative transfer brought by noisy pseudo labels through confidence re- weighting and regularization. In addition to the pseudo labeling strategy, the model adaptation strategy has also been studied. For example, [50, 5 employ an adversarial learning strategy to perform model adaptation with only pre-trained source models. However, they cannot be applied to tackle the MSFDA problem, due to the distribution discrepancies across different source domains.
Multi-Souree Domain Adaptation is extended from vanilla domain adaptation [02 IS [I]] by ex- ploring transferable knowledge from multiple sources. To capture the relationship between different source domains and a given target domain, Guo et al. [ design a mixture-of-experr model for unsupervised domain adaptation from multiple sources. [56]] focuses on determining which source domain is the best for target prediction via dynamic curriculum learning. Some discrepancy- based methods aim to narrow the distribution discrepancy across domains by minimizing different mea- sures such as the Rényi-divergence [21]] and maximum mean discrepancy [199]. Moreover, some adversarial-bart
methods focus on optimizing the H-divergence [+] 600], generative adversarial loss [T] and Wasserstein distance [[2]] to make features from multiple sources indistinguishable
for a shared discriminator. [62] B perform knowledge adaptation at the pixel-level by replaying multiple source domains. Due to lack of source data, above strategies in MSDA may be invalid and
unsuitable to address the challenging MSFDA problem. To this end, Ahmed et al. [M]] utilize nearest distance measure to mine target pseudo labels, and weight the predictions from multiple source mod- els for the MSFDA task. It may result in that the generation process has high probability to obtain noisy labels [A] G1, 54] when the strategy is not matched with the target data, while our model could generate confident pseudo labels from two different perspectives, ie., geometry and probability.
3 Problem Setting
Let X' and y = [K] := {1....,] K} denote the feature space and label space. A domain is a joint distribution Pxy on X' x Y). There are n source domains ( (Pxy } For any source domain Pxy,, a corresponding neural-network-bor domain p' with unlabeled target data T = fx' { ~ Pk, ii.dd, the aim of multi-souuce--fre domain adaptation (MSFDA) is to classify the unlabeled target data by utilizing T and (h" hh)
predictor (model) h' X + R is given. Given target
Let l denote non-negative loss function defined over RK x RR Given a hypothesis space E(x.)) ((h(x), (gy) and Li(h)
H C (h X RK,, we denote (h) B(x.))-P l(h(x), (z)) as the risks with respect to the i-th source domain and a given target domain, where &b y + R maps any label y to corresponding one-hot vector.
The source predictor h¹ is vector-valued function, iee, hx() = [x),,, ..., h (x)] and consists of two basic components: feature extractor f X - IRd and classifier ct Rd R.K where d is the dimension of extracted features. Therefore, h can be rewritten as c o fo. After using softmax as
the activation function in the output layer, we have hj 2 O and Σk= hi = 1. To ensure that each predictor h' is a relatively accurate predictor for the domain Pxy, we assume that the predictor h' is e-accurate under E, loss, i.c., E(x.)) Pky hx(x)) - 4 (y)lle < e, for any i E [n]]
4 Theoretical Analysis
Without any relations between source domains and target domain, MSFDA cannot be effectively addressed from the theoretical view. To bridge source and target domains, Mansour et al. [00] and
3

---

Miraj Ahmed et al. [M]] assume that Pxx = Ei= XxX and Prix Pyix = pixx for any i E [n], where X, 2 and Σim >; This assumption is very strong and may be unrealistic in many real-world applications. Motivated by the meta learning [[]] and domain generalization [C), in this paper, we propose some novel and mild assumptions to address MSFDA problem.
Assumption (Meta Assumption.) P'y- :xy: Pxy are drawn (i.i.d.) from a meta distribu- tion P, which is defined over a joint distribution space XXY.
Assumption 2 (Regular Domain.) Let the joint distribution space Pxy is endowed with total variation distance d.v(:..). The larget domain Pxy is regular domain, i.e. for any ( Xxy > O, where No. = =(P: :P: (P, Pxx) <0)) o). f'x
is easy to check, the meta distribution P is discrete or continuous with continuous density function, then the target domain x drawn by P is a regular domain with probability 1. In addition, to weaken the assumption Py Pv x for any i, E [n]]. our key strategy is consider the anchor point assumption that is studied in label-noise learning [[6]].
We say a given point x is a r-anchor point, if there exists a predictor h with the largest score (x), such that li{(x) - (x(x)) 2 т, for any c E [K] and e + k. h² the r-anchor predictor for x. Assumption (r-Anchor Point Assumption.) Given T-anchor point x, suppose that all anchor predictors for x are h" ...., ...,,% h²,, then the true label of the T-anchor point is k. if tij (xx is the largest score among scores he ...., h.., hūg for any c [K].
When the source predictors are accurate enough, Assumption E implies that the source and target conditional distributions are similar in the high confidence region. Hence, it is much weaker than the traditional assumption py = Piix = P for i, E [n]. T is regarded as a threshold to distinguish which data has highly confident prediction. In general, r is close to 1.
Given any hi, it is easy to check that there exists matrix function A' (' [xk](x)] such that (x) = Ex a Px rxx (1)x) with ΣK a 1. We say the diagonal elements {a/)/=> as the transfer factor for h' and py Generally, A' Ax(x) [x)] is not unique, thus the transfer factor may be not unique. The following theorem indicates that Assumption B holds if we give proper assumptions for transfer factor.
)/5
Theorem Suppose that the Bayesian label is true label [TA]. If there exist transfer factors and constant B < K such that [aX dn B, then Assumption B holds with 7 > 1 B/K.
Theorem D provides a theoretical support for Assumption B and indicates that when the transfer fac- tors are positive, Assumption B always holds with proper т. To further study the highly confident pseudo labeling strategy, the following theorem provides a lower bound to estimate the number of highly confident pseudo labels, i.e., the number of r-anchor points.
Theorem 2 Assume Assumptions 0 and D hold and the conditional distribution Pr can be pre- sented as a labeling function, i.e., p (ulx) = Or 1. Given 1i > 0, if 2 n and (1-1))1--)) >
E ++ 20 + 2 (2mm 2)/mm, then with probability at least least 17m larget data are --nchhr points, where is the upper bound of the accuracies of source predictors, and is introduced in Assumption 2
- (1 (NN Px,v ))a > at
Theorem D indicates that multi-source predictors improve the probability to obtain more r-anchor points. To further understand the effect of multi-source domains, we build a novel learning bound for the MSFDA task. Let A, be the set consisting of all r-anchor points. Denote the empirical risk 2g (h) by TAPTI Ex (()(x), (3)), where y the label of the anchor point x.
Theorem 3 Given Assumption E and some assumptions used in Theorem D and suppose that the loss E has upper bound M > 0 and hypothesis space H has finite Natarajan dimension, for nj > if m 2 n and (1-n) (1 7) > e e 20 2llog((2m (2m/6//1 /2mm, then for any h E and b E (0, 1),
there exists a constant Cbbb, K) such that with the probability at least 1 - 28 2( 9 (NF pt ))"
log(2/8)
20 +
(.(h)-E,00 S
MC (CC(,KK)
77-6m116 M I 2a -
where is the upper bound of accuracies of source predictors, and a is introduced in Assumption P
4

---

Conrident Anchor
Detance Messure
Clbssllone
Target
Extractor F!
-100
Luss
Pseudd Labol
Edis
Fnn (xe))
a(:))
Source Specific Transferable Perception
- fel)
ConfidertAAccho Incluced Pseudo Label Generator
(x))
h²(xx)
Class- Rless-Relaliooahiii--- Ritis--rreee
Extractor 1
Clessifer e*
Corsistency
Overview of our model, mainly including source-specific transferable perception strat- egy to quantify the contributions of the transferability of source domains, a confiden-ancche induced pseudo label generator generate pseudo labels for target domain, and class-relationship- aware consistency loss to ensure the semantic consistency of underlying inter-class relationships.
Figure 1:
Theorem & shows that multiple source domains improve the probability to ensure a tighter gener- alization bound, i.e., Eq. (I) holds. Note that the Natarajan dimension used in Theorem B is bit outdated. However, the Natarajan dimension can be replaced and Theorem Gl can be updated without any technical barriers, if there exists better generalization theory for supervised learning.
Summary of Theoretical Analysis: The reasons to develop MSFDA are to study the solvability of MSFDA and understand how multi-source predictors benefit the target domain's classification. By Theorems @ and B, we realize that multi-source predictors improve the probability to obtain more highly confident pseudo labels, resulting in tighter generalization bound. The generalization bound in Theorem Gl gives a positive answer to the solvability of MSFDA. Additionally, Theorems MI and also imply two interesting and important results: Theorem W provides the first theoretical support to the confident pscudo labeling strategy, and Theorem Q provides the first lower bound of the number of highly confident pseudo labels. As know, the theoretical results in Theorems II and A are novel.
5
The Proposed CAiDA Model
The graphical illustration of our proposed model is depicted in Figure MI It mainly consists of three significant components: source-specifie transferable perception, munfident-anchho--nnueee pseudo label generator and class-relationshii-aaaal consistency loss, which are elaborated as follows.
5.1 Source-Specific Transferable Perception
Generally, in multi-source domain adaptation (MSDA), different source domains have different con- tributions to improve the performance on target domain 61]. To this end, many previous MSDA methods match the features across different domains, and then quantify the contributions of source domains by taking the average of the trained source predictors [IT], 650]. or weight the trained source predictors normalizing the distance similarities [29,5]] between source and target domains. How- ever, due to the lack of source data, these methods cannot employ source data to match features and cannot be successfully applied to multi-source free domain adaptation (MSFDA) tasks.
Therefore, source-specific transferable perception module is developed to automatically quantify the contributions of the transferability of source domains, shown in Figure MI Specifically, the one-hot encoding vector u' E R of the i-th source domain can considered as the unique do- main characterization, which is then employed as network input to quantify the contribution of the transferability. We then concatenate all source domains' one-hot characterizations together to obtain U [u', ] E _=x", U is then forwarded into a Multi-Layer Perceptron (MLP) network Ω to automatically quantify the contribution (2(U) = 22(u', uu,..., . ), such that Σ 14 = 1, where μı denotes the quantified contribution of the i-th source domain to the target prediction.
the transferability μ E IR" of n source domains:
When the source data are unavailable, it is difficult to narrow distribution discrepancy across do- mains, due to lack of any supervised information. Inspired by [GK], we freeze the network parameters of classifiers [e=)=_ and solely perform distribution adaptation across domains feature extrac- tors [ via information maximization [26], since (c 1 "_ contain class distribution information of source domains. However, Liang et al. [< cannot be effectively applied to multi-source- -frec do-
5

---

main adaptation scenario, where different source domains have different transferable contributions on target prediction. Therefore, Let is proposed to minimize conditional entropy of target outputs by incorporating the source-specific transferable perception ja:
1
Lent
hx(x') log(hx(x')))
where h2 (x') ) is the k-th coordinate value of hxxx), here h(x²) - (xe)) denotes the com- bination of source predictions. The larger value of μ4; indicates the larger contribution of transferabil- ity of the i-th source domain for target adaptation. Unfortunately. our proposed CAiDA model may suffer from the trivial solution of Eq. (2) by predicting all target data as a single class to minimize Eq. (2). To tackle this issue, Laiv is designed to consider class prediction diversity by maximizing the entropy of empirical label distribution [66] predicted by different source domains:
K
Ldiv
Pie Pkg
Pk:
(3)
1=1
where Pk
1 m
f== hx(x²) denotes the mean prediction probability over target data.
5.2 Confident-Annhho--nnueee Pseudo Label Generator
Although the minimization of both Lent and Laiv promotes the class diversity and knowledge adap- tation between multiple unseen source data and target data, it cannot eircumvent erroneous label assignment due to the noisy target prediction brought by domain discrepancy. To alleviate this issue, based on Assumption B, a confident-anchor-nndiee confident pseudo labels for target data, as shown in Figure II Specifically, based on the r-anchor assumption, we select target data x as confident anchor when the maximum category prediction probability hx- (xx) is larger than the rest of category prediction probabilities h4 (x') (k & k*) by a threshold Tp. For each confident anchor, we integrate the features extracted from multiple source extractors together to obtain a probability-based confident anchor group Cp:
pseudo label generator is developed to mine
Cp = ([(('')) (x²) - hx(xx)) 2 Tp. Ak k 1,2, .. m).
where f(x)) = Ei= (x²) denotes the feature of the j-th target data. Tр is defined as the median value of the probability difference between the largest and the second largest probabilities over all target data. Furthermore, inspired by the proposed r-anchor assumption, to circumvent the noisy anchor in Cp. we construct distance-based confident anchor group Ca. The feature centroid of the k-th class induced by the i-th source domain for the whole target data is defined as E En=1 ( (xd) m (x))). {()) are then weighted with source-specific transferability to obtain the feature centroid &u of the k-th class over all source domains via & = = M&E The distance between the feature of x and the k-th feature centroid & is denoted as d(f(xx) Ek). Ex). where d(.;) :) is distance measure function. When the minimum distance d(f(x') ), Ex- ) is shorter than the rest of distances d(f (x) (Is # k*) by threshold Ta. we select the target data x into Cy:
C. = (f(x') d((())))))))))))))))))(())))dd ). Ex) d(f(x'), Ex-) 2 Ta, Vk + k" j 1,2, m..
(5)
where we set as the median value of the distance difference between the shortest and the second shortest distances over all target data. Therefore, the final confident anchor group is obtained by performing an intersection operation between C, and C,, i.e., C = C, Ca.
To generate confident pseudo labels for unconfident target data, select semantic-neares, confi- dent anchor from C for each target data via continual similarity searching. To be specific, given an unconfident target data x', we search a serial of unconfident guiding data consecutively using the distance measure function -)- in the feature space, until the confident anchor from C' is detected. During each searching process, the previous searched guiding data are not considered in the follow- ing iteration. Denote as the set containing previously searched guiding data, x as the searched confident anchor closest to x². The guiding sample x)() in the (-th search could be obtained by:
xi(1) arg min d' d(f(x)(t-11) {(x))), subject to x0) + (t--1) x((1) $ I,
(6)
x ET

---

Algorithm The Searching Process of
Algorithm 2 The Optimization of Our Model.
Semantic-Nearest Confident Anchor x
1: Input: {hr {h'j I E epoches, B batches;
Input: x2, F, C.
2: Initialize: K1, R2. the parameters of St;
2: Initialize: (t-1 x², I R;
3: For e = 1, 1...
do
3: While (t-1) & do
4: Obtain pseudo labels via Eq. (2);
4: Obtain x0 via Eq. (6);
5: For 1 = 1, B do
5: Update via adding (1) into 1;
6: Select mini-batch of target data;
6: Update (t-1)) via (t-1)) t x1));
7: Update (fr and 5 via Eq.(2);
7: End
8: End
8: Obtain x via xic + x
9: End
9: Return xfc.
10: Return: (h' J==1 and QS
where (t-1)) is the guiding data in the previous search. The searching process of x for the j-th target data summarized in Algorithm MI Moreover, fuse the target data and their corresponding confident anchor in the feature space for feature augmentation, and the synthetic feature is denoted [o, 1] is the random weight to determine the influence of confident anchor fayn (x'). Therefore, the confident-anchor-nnddeee
as fayn (x²) (1 f)(('') + wf (x'", where
pseudo label j of the j-th target data x² and the classification loss Lets of whole target data are formulated
e' arg min ((f Cele kE[K]
ga - (og(hx(xx)))
5.3 Class- Roass-Relationship-Aeee Consistency Loss
The inherent relationships between different classes have semantic consistency across domains, re- gardless of distribution discrepancy. In light of this, aligning class relationships could promote more shared transferable knowledge from source domains towards target adaptation. To achieve this, as depicted in Figure M. a class-re rewatrrosssiii--aaaaaeeee data from the same class to be compactly clustered together while preserving the intrinsic inter-class relationships via soft confusion matrix alignment. To be specific, the soft label distribution st of the k-th class predicted via the i-the source predictor formulated as s 1 - ("
consistency loss Lere is designed to encourage target
The collection of soft label distributions { k= represents kind of soft confusion matrix associ- ated with a particular domain, encoding inter-class relationships learned by the i-th source predictor (e.g., computers have more similar semantic relationship with desks than horses). Without access to the source data, Lere aims to align soft confusion matrices from different source predictors:
n
Lere
(KL($ + KL(
(8)
2n
=1=11k1
where KT.(pll)) E, log Pr. denotes the Kullback-Leibler (KI.) divergence. The complexity of Lere is not problematic in practice, due to the limited number source domains.
In summary, the overall optimization objective to optimize {f (f' and Ω is formulated as:
L = Lent Laiv + K1 Lets + K2 KaLere
where K1, are the balanced weights. The optimization of our model is presented in Algorithm 0
6 Experiments
6.1 Datasets and Baseline Methods
Datasets: Office-31 [2] consists of three representative domains with 31 shared object categories
the office environment, i.e., Amazon (A), Webcam (W) and DSLR (D), Office-Caltech [08] an extension dataset of Office-31 [22]] by adding additional subset called Caltech-256 (C) on it and extracting 10 common object classes among them. Office-Home [Н6] is composed of four different domains including Product (Pr), Clipart (CI), Art (Ar), and Realworld (Re). Each of these subsets
7

---

Table 1: Comparisons between model and other competing methods Office-31 P2 dataset
(the left block) and Office-Caltech [TX] dataset (the right block).
Methods
Source Dasa
A. 1-D--
A.
D. WA
Avg.
A. D.C
A.
3D
C,D, W
-A
A.
W
Avg.
Source only [A]]
97.1
92.0
51.6
80.2
93.3
94.2
90.6
87.3
91.3
MDAN
[
99.2
95.4
55.2
83.3
99.4
98.7
93.5
91.6
95.8
DCTN
99.6
54.9
83.8
99.3
99.4
94.1
91.3
96.0
MPSDA 1:
99.4
96.2
55.4
83.7
99.5
99.2
94.5
92.2
96.4
MDDA [SM]]
99.2
97.1
56.2
84.2
99.3
99.6
95.3
92.3
96.6
L.C. MSDA
[AX]]
99.6
97.2
$6.9
84.6
99.4
99.7
93.7
95.1
97.0
Souree model only
X
95.4
97.3
60.2
84.4
98.0
99.5
96.3
92.1
96.5
BATT
98.5
89.5
97.5
97.5
95.7
97.2
PıDA AN
93.8
96.7
73.2
87.9
97.6
97.
97.3
94.6
96.7
SHOT
94.9
97.8
75.0
89.3
95.7
95.8
97.0
MA [+
96.1
97.3
75.2
89.5
99.8
97.2
95.7
95.6
97.1
DECISION
98.4
75.4
91.1
100.0
95.9
95,9
98.0
Ours-w/oEnt
97.3
74.2
90.3
94.3
96.3
97.2
Ours-wloDiv
97.2
98.6
73.7
89.8
98.6
99.3
94.1
95.7
96.9
Ours w/oCls
96.7
73.0
89.4
97.3
93.6
95.2
96.1
Ours-w/dd wurs-w/oCre
98.6
99.5
75.4
91.2
99.6
100.0
95.3
96.5
97.9
Ours
98.9
99.8
75.8
91.6
99.8
100.0
96.8
97.1
98.4
(a) Source model only
(b) Ours
(c) Source model only
(d) Ours
Figure 2: t-SNE 2SS visualizations on Office-31
22 dataset when performing D, W A (a)(b) and
A. WD+ (c)(d) domain adaptation tasks.
consists of 65 shared object categories. Digits-Five | contains five digit recognition subsets including MNIST-M (MM), USPS (UP), MNIST (MT), SVHN (SV) and Synthetic Digits (SY).
Baseline Methods: To validate the effectiveness of our model, we conduct comparison experiments with wide array of baseline methods. Specifically, MDAN M DCTN [a>I] MDDD [НI MDDA [62] and LIC-MSDA with access to source data. BAIT [5Z]], PrDA [25]], SHOT [XK] and MA
[ are traditional representative multi-source domain adaptation methods
[+T] focuses on unsupervised
single-source domain adaptation without access to source data. We compare against the multi-source extensions of [25. KUGGs 5%5
$3.57 57] by taking an average of target soft predictions from all adapted source
models. DECISION [M]] combines the source adaptation models with suitable weights automatically for multi-source- free domain adaptation. Furthermore, Source only denotes the performance of evaluation target data when combining the rest of source data for training, and Source model only represents the average performance over the predictions of all source models.
6.2 Experiments on Office-31 and Office-Caltech Datasets
Performance Comparisons: The comparisons between our CAiDA model and other state-of-the art methods on Office-31 [22] and Office-Caltech [[S] datasets are presented in Table We have the following conclusions from the results Table II: 1) When compared with the multi-source domain adaptation methods [1]]] 37 53, 601 (62 that employ source data for training, our proposed model without access to source data significantly outperforms them by a large margin 1.4%~~8.3% in terms of mean accuracy. It verifies the superiority of our model to tackle multi-source-fire domain adaptation. 2) Our model performs better than the multi-source extensions of single-source- domain adaptation methods [2 50L (3, 57], which validates the effectiveness of our pseudo label generation process. 3) The performance of our model is better than [II] for all evaluation tasks, since the annfideet-anchor--oe induced pseudo label generator and class relationship-awwene. loss promote the adaptation performance. Figure shows that our model significantly narrows distribution discrepancy across domains Office-31 [ when compared with Source model only.
consistency
Ablation Studies: This subsection introduces the effectiveness of each component our model via ablation studies on Office-31 [[2]] and Office-Caltech [[]] datasets, as shown in Table II. Ours-

---

0.4
Ofice-
Accuracy (8)
85.
Aocuracy (52)
03
Waights
18 1
80
023
W
6.D.W.AA
D.
03
0.5
0.5
to 1o
2
6
(a) Office-31
(b) Office-Caltech
Weights of domain A
(d) Convergence
Figure 3: Qualitative analysis about parameters { (a)(b), weights of domain A (c) and con-
vergence Office-Caltech (d), where in (c) denotes the rest of domains except for the target.
Table 2: Comparisons between our model and other competing methods on Office-Home
[6
Methods
Source Data
Arr
Ar, CI. -
Cr, Rc Ar
Avg.
Source only [1]
67.8
71.3
51,8
53.4
61.1
MDAN
77.5
77.6
62.2
70.6
DCTT
78.7
78.3
63.8
66.4
71.8
MPSDA
79.4
79.1
63.5
67.2
72.3
MDDA 0X
79.6
79.5
62.3
66.7
71.0
LxC-MSDA
[330
80.1
79.2
64.1
67.4
72.7
Souree model only
76.3
18.8
50.1
50.9
64.0
BAIT [+
77.2
79.4
59.6
71.1
71.8
PrDA [A4
76.8
79.1
57.5
69.3
70.7
SHOT
[4C
82.9
82.8
59.3
72.2
74.3
[00
73.5
DECISION [
X
83.6
84.4
59.4
74.5
75.5
Ours-w/oEin
82.6
83.0
58.7
74.2
74.6
Ours wioDiv
Ours wloCls
81.4
82.7
57.9
73.1
73.8
Ours
84.2
84.7
60.5
75.2
76.2
w/oEnt, Ours-w/oDiv, Ours-w/oCls and Ours-w/oCre are the abbreviations of training our proposed model without Leut, Ldiv. Lets and fere: respectively. When any of component of our model is abandoned, the performance degrades 0.4% in terms of average accuracy, which illustrates the rationality and effectiveness of all components to cooperate together. All modules play indis- pensable role in improving performance, even though our model has no access to source data.
in range of {0.1, 0.3, 0.5, 0.7, 0.7.0.9) 0.9) and K2 in range of {10-1 1, 10-3 1002 100100111 on Office-31 [22]] and Office-Caltede [[]]] datasets, shown in Figure B (a)(b). It validates that our proposed model achieves stable performance over wide range of syper-parameter selection. Morcover, the best performance of our proposed model on target domain is obtained when N₁ 0.7 and K2 10-2
Parameter Investigation: This subsection investigates the effects of hyper-parameters
Contribution Weights and Convergence Analysis: FigureB(c))d) (c) present the contribution weights of domain A and convergence curves of our proposed model on Office-31 [22] and Office-Caltech
[ datasets. The source-specific transferable perception module could quantify the transferabil- ity contributions of complementary information from multiple source domains to target prediction. Furthermore, the accuracy on Office-Caltech [[8] dataset converges stable value after a few epoches, which demonstrates the convergence effectiveness of our proposed CAiDA model.
6.3 Experiments on Office-Home Dataset
Performance Comparisons: As presented in Table 2L we conduct comparison experiments on Office-Home [[6] dataset to illustrate the effectiveness of our model. We have the following ob- servations from Table D: 1) Without access source data, model significantly outperforms the representative multi-source domain adaptation methods надд 60, 52 by 3.5%~~566 in terms of average accuracy. 2) The confident pseudo label generator encourages perform better than [[]] which verifies the superiority our model for multi-source- adaptation. The performance of our model improves large margin of 1.9%~5.5% in terms of mean accuracy, compared with [25 G0, KU, B3, 57]] validates the efficiency of source-specific transfer- able perception strategy and class-rlationshippaaaan consistency narrow distribution discrepancy.
proposed model to domain
Ablation Studies: introduced Table R we conduct ablation studies of our model on Office- Home [0]] dataset illustrate the rationality of all designed modules. When compared with Ours,

---

Table 3: Comparisons between model and other competing methods Digits-Five 1 dataset,
where R denotes the rest of four domains except for the single target domain.
Methods
Souree Data
R - MM
R + MT
R
UP
R + SV
R + SY
Avg.
Source only [M]
63.4
90.5
88.7
63.5
82.4
77.7
MDAN
98.0
87.4
83.3
DCTN SS
70.5
96.2
92.8
77.6
86.8
84.8
MYSDA H
72.8
98.4
96.1
81.3
89.6
87.7
MDDA 0M
78.6
98.8
93.9
79.3
89.7
88.1
LiC-MSDA [800
99.0
98.3
91.8
Source model unly
25.2
90.0
93.3
42.8
77.8
65.8
BAIT [Y
87.6
96.2
96.7
60.6
90.5
86.3
PrDA
95.4
83.9
SHOT
90.4
98.9
97.7
58.3
83.9
85.8
MA [50]]
90.8
98.4
98.0
59.1
84.5
86.2
DECISION [
93.0
99.2
97.8
82.6
97.5
94.0
Ours-w/oEnt
92.1
97.3
96.0
80.7
96.3
92.5
Ours-w/Div
91.7
97.0
96.8
82,2
96.5
92.8
Ours-w/oCls
91.3
96.6
96.4
80.5
95.8
92.1
Ours-wloCre
92.8
98.2
98.1
93.9
Ours
93.7
99.1
98.6
83.3
98.1
94.6
the performances of Ours-w/oline Ours-w/oDiv, Ours-w/oCls and Ours-w/oCre degrades 1.6%, 1.9%, 2.4% and 0.6% in terms of average accuracy, respectively. It validates that all designed mod- ules could cooperate well to address MSFDA task. Moreover, confident-anchor-induee pseudo label generator could reduce the distribution discrepancy via mining confident pseudo labels.
6.4 Experiments on Digits-Five Dataset
This subsection presents the ablation studies and comparison experiments between model and other competing methods on Digits-Five [РI dataset, as introduced in Table E. Some conclusions drawn from the presented results in Table 1) Our proposed model performs the best in terms of average accuracy when compared with multi-source domain adaptation methods (1,,4,45,55
and single-source-free adaptation methods (25] The significant performance improvement demonstrates the effectiveness of our model, even without access to source data. 2) Our model outperforms [25] 5T. [R. 5 by 8.3%~~10.7% mean accuracy, since automatically quantifies the contributions of different source domains to target adaptation. When we compare our model with [1I, MT, 47. 55, 60. 62], the confident-anchi cenet-anccoo-nnuueeee pseudo label generator and glass-relationship. aware consistency loss facilitate the performance improvement by mining confident pseudo labels and aligning soft confusion matrices across domains. 3) The performance degradation ablation studies (i.e., Ours-w/oEnt, Ours-w/oDiv, Ours-w/oCls and Ours-w/oCre validates that all proposed components are designed effectively and reasonably to explore transferable knowledge.
Conclusion and Future Work
This paper proposes novel manfiddet--nchor-inniie multi-sone source-free domain adaptation (CAiDA) model to capture transferable knowledge from multiple source domains without access source data. To be specific, a source-speciiie transferable perception module is developed auto- matically weight the contributions of the transferability of source domains. Meanwhile, we design saaieeettannco--innuueeee pseudo label generator to mine confident pseudo labels for the target do- consistency
main by establishing a confident anchor group, and develop class-relationshipaawwe loss to capture consistent inter-class relationships across domains. Theoretical analysis provides some new perspectives to the highly confident pseudo labeling strategy, and gives theoretical sup- port for MSFDA task under some proper mild assumptions. Extensive experiments illustrate superiority of our proposed model. In the future, we will extend MSFDA to the multi-label [48 99] or open-set [ 67.53 53
scenarios and use MSFDA techniques to study pandemic [38 59]
Acknowledgments
This work was partially supported by National Nature Science Foundation of China under Grant 62003336: National Postdoctoral Innovative Talents Support Program under Grant BX20200353; Nature Foundation of Liaoning Province of China under Grant 2020-KF-11-4 1--00; and Australian search Council Projects under Grant DP-180100442 19010011, and IC-190100031.
10

---

References
[1] Sk Miraj Ahmed, Dripta Raychaudhuri, Sujoy Paul, Samet Oymak, and Amit K. Roy-Chowdhury. Unsupervised multi-source domain adaptation without access to source data. In CVPR, pages 10103- 10112, 2021.
[2] Zechen Bai, Zhigang Wang, Jian Wang. Di Hu, and Errui Ding. Unsupervised multi-source domain adaptation for person re-identificationn. In CVPR, pages 12914-12923, 2021.
[3] Yogesh Balaji. Swami Sankaranarayanan,
Rama Chellappa. Metareg: Towards domain generalization using meta regularization. In NeurlPS, volume 31, pages 1006-10166 2018.
[4] Antonin Berthon, Bo Han, Gang Niu, Tongliang Liu, and Masashi Sugiyama. Confidence scores make instance-dependent label-noise learning possible. In ICML, volume 139, pages 825-836, 2021.
[5] Konstantinos Bousmalis, George Trigeorgis, Nathan Silberman, Dilip Krishnan, and Dumitru Erhan. Do- main separation networks. In NewrlPS, pages 343-351, 2016.
[6] John Bridle, Anthony Heading. and David MacKay. Unsupervised classifiers, mutual information and phantom targets. In NeurlPS, volume 4, pages (096-1101) 1992.
[7] Jiacheng Cheng. Tongliang Liu, Kotagiri Ramamohanarao, and Dacheng Tao. Learning with bounded instance and label-dependent label noisc. In /CML, volume 119, pages 1789-1799, 2020.
[8] Haoang Chi, Feng Liu, Wenjing Yang. Long Lan, Tongliang Liu, Bo Han, William K. Cheung, and James T. Kwok. TOHAN: A one-step approach towards few-shot hypothesis adaptation. 2021.
[9] Jiahua Dong, Yang Cong, Gan Sun, and Dongdong Hou. scopie lesions segmentation. In ICCV, pages 10711 100220, 2019.
weakly-supervised endo-
[10] Jiahua Dong, Yang Cong, Gan Sun, Yuyang Liu, and Xiaowei Xu. Csel: Critical semantic-consistent lcarning for unsupervised domain adaptation. In ECCV, volume 12353, pages 745-762, 2020.
[11] Jiahua Dong, Yang Cong, Gan Sun, Yunsheng Yang, Xiaowei Xu, and Zhengming Ding- Weakly- supervised cross-domain adaptation for endoscopic lesions segmentation. IEEE Transactions on Circuits and Syxtems for Video Technology, 31: 31:2020-2033, 2020.
[12] Jiahua Dong, Yang Cong, Gan Sun, Bineng Zhong, and Xiaowei Xu. What he transferred: Unsuper- vised domain adaptation for endoscopic lesions segmentation. In CVPR, pages 4022-4031, 2020.
[13] Zhen Fang, Jie Lu, Anjin Liu, Feng Liu, and Guangquan Zhang. Learning bounds for open-set learning.
In ICML, volume 139, pages 3122-3132, 2021.
[14] Zhen Fang, Jie Lu, Feng Liu, Junyu Xuan, and Guangquan Zhang. Open set domain adaptation: Theoreti- cal bound and algorithm. IEEE Transactions on Neural Networks and Learning Systems, abs/1907.0883 1907.08375, 2019.
[15] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. adverraadv training of neural networks. J. Mach. Learn. Res., 17:20962030, 2016.
[16] Muhammad Ghifary, W. Bastiaan Kleijn, Mengjie Zhang. David Balduzzi, and Wen Li. Deep reconsruactionclaasfffilll networks for unsupervised domain adaptation. In ECCV, volume 9908, pages 597-613, 2016.
[17] Ian Goodfellow, Jean Pouget-Abadic, Mchdi Mirza, Bing Xu, David Warde-Farlcy, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NeurlPS, volume 27, 2014.
[18] Kristen Grauman. Geodesic flow kernel for unsupervised domain adaptation. In CVPR, page 20662073, 2012.
[19] Jiang Guo, Darsh Shah, and Regina Barzilay. Multi-source domain adaptation with mixture of experts. In EMNLP, pages 4694-4703, 2018
[20] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pages 770-778, 2016.
[21] Judy Hoffmann, Mehryar Mohri, and Ningshan Zhang. Algorithms and theory for multiple-source adapta- tion. In NewrlPS, volume 31, pages 8256 8266, 2018.
[22] Judy Hoffmann Eric Tzeng. Taesung Park, Yun Zhu, Phillip Isola, Kate Saenko. Alexei Efros, and Trevor Darrell. CyCADA: Cycle-consistent adversarial domain adaptation. In ICML, volume 80, pages 1989-1998, 2018
[23] Han-Kal Kai Hsu, Wei-Chih Hung, Hung- Yu Tseng, Chun-Han Yao, Yi-Hsuan Tsai, Mancesh Singh, and Ming- Hsuan Yang. Progressive domain adaptation for object detection. CVPR Workshops, pages 738 746, 2019.
[24] Shatha Jaradat. Deep cross cross-domain fashion recommendation. In RECSYS, page 407410, 2017.
11

---

[25] Youngeun Kim, Donghyeon Cho, Priyadarshini Panda, and Sungeun Hong. Progressive domain adapta- tion from source pre-trained model. arXiv preprini arXiv:2007 201554.
[26] Andreas Krause, Pietro Perona, and Ryan Gomes. Discriminative clustering by regularized information maximization. In NeurIPS, volume 23, pages 775-783, 2010.
[27] Y. Lecun, I.. Bottou, Bengio, and P. Haffner. Gradient-hand learning applicd to document recognition. Proceedings of the IEEE, 86:2278-2324, 1998.
[28] Scungmin Lce, Dongwan Kim, Namil Kim, and Scong- -uyng Jeong. Drop to adapt: Learning discrimina- tive features for unsupervised domain adaptation. In ICCV, pages 91-100, 2019.
[29] Keqiuyin Li, Jie Lu, Hua Zao, and Guangquan Zhang. Multi-source contribution learning for domain adaptation. IEEE Transactions on Neural Networks and Learning Systems, pages 1-15, 2021.
[30] Rui Li, Qianfen Jiao, Wenming Cao. Hau-San Wong. and Si Wu. Model adaptation: Unsupervised domain adaptation without source data. In CVPR. pages
2020.
[31] Xuefeng Li, Tongliang Liu, Bo Han, Gang Niu, and Masashi Sugiyama, Provably end-to-end label-noise learning without anchor points. In ICML, volume 139, pages 6403-6413, 2021.
[32] Yitong Li, michael Murias, geraldine Dawson, and David E Carlson. Extracting relationships by multi- domain matching. In NeurIPS, volume 31, pages 6799 1119996810,-1 6810, 2018.
[33] Jian Liang. Dapeng Hu, and Jiashi Feng. Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. In ICML, pages 6028-6039, 2020.
[34] Chuang Lin, Sicheng Zhao, Lei Meng, and Tat-Seng Chua. Multi-source domain adaptation for visual sentiment classification. AAAI, 34:2661-2668 2020.
[35] Feng Liu, Wenkai Xu, Jie Lu, Guangquan Zhang, Arthur Gretton, and Danica J. Sutherland. Learning deep kernels for non-parametric two-sample tests. In ICML., volume 119, pages 6316-6326, 2020.
[36] Tongliang Liu and Dacheng Tao. Classification with noisy labels by importance reweighting. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38:447-461, 2016.
[37] Yadan Luo, Zijian Wang, Zi Huang, and Mahsa Baktashmotlagh. Progressive graph learning for open-set domain adaptation. In ICML, volume 119, pages 6468-6478 2020.
[38] Qianqian Ma. Yang- Yu Liu, and Alex Olshevsky. Optimal lockdown for pandemic control. arXiv preprint arXiv:2010. 12923, 2020.
[39] Qianqian Ma. Yang-Yi Yu Liu, and Alex Olshevsky. Optimal vaceine allocation for pandemic stabilization. arXiv preprint arXiv: 2109.04612, 2021.
[40] Yishay Mansour, Mehryar Mohri, and Afshin Rostamizadeh. Multiple source adaptation and the rényi divergence. In UAI, pages 367-374, 2009.
[41] Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In ICCV, pages 1406-1415, 2019.
[42] Zhen Qiu, Yifan Zhang. Hongbin Lin, Shuaicheng Niu, Yanxia Liu, Qing Du, and Mingkui Tan. Source- free domain adaptation via avatar prototype generation and adaptation. In IJCAI, pages 1921-2927, 2021.
[43] Aravind Rajeswaran, Chelsea Finn, Sham Kakade, and Sergey Levine. Meta-learning with implicit gradients. In NeurlPS, volume 32, pages 113-124, 2019.
[44] Paolo Russo, Tatiana Tommasi, and Barbara Caputo. Towards multi-source adaptive semantic segmenta- tion. In Image Analysis and Processing, pages 292-301, 2019.
[45] Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 9:2579-2605, 2008.
[46] Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hash- 2017.
ing network for unsupervised domain adaptation. In CVPR, pages
[47] Hang Wang, Minghao Xu, Bingbing Ni, and Wenjun Zhang. Learning to combine: Knowledge aggrega- tion for multi- source domain adaptation. In ECCV, pages 727 744,744,277 2020.
[48] Lichen Wang, Zhengming Ding, and Yun Fu. Generie multi-label annotation via adaptive graph and marginalized augmentation. ACM Transactions on Knowledge Discovery Data, 16:1-20, 2021.
[49] Lichen Wang, Yunyu Liu, Can Qin, Gan Sun, and Yun Fu. Dual relation semi-supervised multi-label learning. In AAAI. volume 34, pages 0227-6234, 2020.
[50] Lixu Wang. Shichao Xu, Xiao Wang, and Qi Zhu. Eavesdrop the composition proportion of training labels in federated learning. arXiv preprint 191v:190000004, 2019.
[51] Lixu Wang. Shichao Xu, Xiao Wang, and Qi Zhu. Towards class imbalance in federated learning. arXiv preprint arXiv:2008.06211 2020.
12

---

[52] Ximei Wang, Ying Jin, Mingsheng Long, Jianmin Wang, and Michael I Jordan. Transferable normaliza- tion: Towards improving transferability of deep neural networks. In NeurlPS, volume 32, pages 1951- 1961. 2019.
[53] Zijian Wang, Yadan Luo, Ruihong Qiu, Zi Huang, and Mahsa Baktashmotlagh. Learning to diversify for single domain generalization. In ICCV, 2021.
[54] Songhua Wu, Xiaobo Xia, Tongliang Liu, Bo Han, Mingming Gong. Nannan Wang, Haifeng Liu, and Gang Niu. Class2simi: A noise reduction perspective on learning with noisy labels. In ICML, volume 139, pages 11285-11295 2021.
[55] Ruijia Xu, Ziliang Chen, Wangmeng Zuo, Junjic Yan, and Liang Lin. Deep cocktail network: Multi- source unsupervised domain adaptation with category shift. In CVPR, pages 3964-3973, 2018.
[56] Luyu Yang, Yogesh Balaji, Ser-Nam Lim, and Abhinav Shrivastava. Curriculum manager for source selection in multi-source domain adaptation. ECCV, pages 608-624, 2020.
[57] Shiqi Yang, Yaxing Wang, Joost de Weijer, Luis Herranz, and Shangling Jui. Unsupervised domain adaptation without source data casting bait. arXiv preprint arXiv:2010. 12427, 2020.
[58] Jun Zhang, Weien Zhou, Xianqi Chen, Wen Yao, and Lu Cao. Multisource selective transfer framework in multiobjective optimization problems. IEEE Transactions Evolutionary Computation, 24:424-438, 2020.
[59] Yiyang Zhang, Feng Liu, Zhen Fang, Bo Yuan, Guangquan Zhang, and Jie Lu. Clarinet: A one-step approach towards budget-friendle unsupervised domain adaptation. In I/CAI. pages 2526-2532. 2020.
[60] Han Zhao, Shanghang Zhang, Guanhang Wu, José F. Moura, Joao P Costeira.
Geoffrey J Gordon.
Adversarial multiple source domain adaptation. In NeurIPS, volume 31, pages 8568 8579, 2018
[61] Sicheng Zhao, Bo Li, Xiangyu Yue, Yang Gu, Pengfei Xu, Runbo Hu, Hua Chai, and Kurt Keutzer. Multi-souree domain adaptation for semantic segmentation. NeurIPS, volume 32, pages 7285-7298, 2019.
[62] Sicheng Zhao, Guangzhi Wang. Shanghang Zhang, Yang Gu, Yaxian Li, Zhichao Song, Pengfei Xu, Runbo Hu. Hua Chai, and Kurt Keutzer. Multi-source distilling domain adaptation. AAAI, 34:12975- 12983, 2020.
[63] Li Zhong, Zhen Fang, Feng Liu, Jie Lu, Bo Yuan, and Guangquan Zhang. How does the combined risk affect the performance of unsupervised domain adaptation approaches? In AAAI, pages 11079-11087, 2021.
13


Reviews:
[{"review_id": "uT5x1mPGer", "reviewer": "Reviewer_tYwH", "summary": "In this paper, an algorithm for multi-source domain adaptation is proposed. The algorithm is developed in a source-free regime to address the concern for privacy. A loss function is designed to integrate the source-specific end-to-end classifiers using solely the target domain samples, which preserves privacy. The loss function consists of four terms to benefit from enforcing source-specific transferability, anchor-induced pseudo-labels, and class relationship-aware consistency. A PAC-learning style theoretical analysis is provided to demonstrate that using multiple sources may improve the generalizability on the target domain. Finally, experiments on five standard UDA datasets are provided to demonstrate that the method is effective.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. The proofs for both theorems seem correct. However, I feel the analysis does not offer much benefit to the paper. When it comes to the assumption, note:\n- the loss function usually is not bounded in deep learning, e.g., using cross-entropy.-  PAC-learning analysis is not very useful for deep neural networks because they are known to have poor VC-dimension or Natarajan dimension. Indeed, this is a primary reason that deep learning cannot be well understood within the standard PAC-learning analysis based on notions such as VC-dimension.- It is not clear to me that the used terms in the loss function satisfy the requirements for Theorem 2.\nHence the provided analysis does not offer much support when deep neural networks are used. Additionally, analyses based on PAC-learning have existential nature broadly speaking and beyond deep learning, you need to have an idea of whether the conditions are met in practice. In this sense, I don't think the theoretical analysis is very valuable. I highly am skeptical to use mathematical analysis when it does not provide much benefit, just for the sake of making the work more rigorous. My assessment would be totally different of course, if it is demonstrated that the analysis is relevant to the experiments. \n\n2. The proposed method outperforms the multi-source UDA methods, despite being a source-free method. This is impressive but goes against my intuition. Because other multi-source UDA methods seem to upper-bounds for the proposed method because they are using more information, i.e., the source samples. I was wondering what is the reason for this observation? Is there any way to show that if the source data points for accessible, the proposed method can have even better performances? I think such an experiment can serve both as an ablative experiment and also broaden the novelty of your work even in simple multi-source UDA setting.\n\n3. The small improvement of the proposed method over the single-source UDA algorithms questions necessity of the proposed approach, i.e., improvement over DECISION is about 0.5% on average. If this is the case, one might ask why not just use the existing source-free methods by averaging? I understand we see a marginal improvement, but this also comes with a significantly more rigorous procedure. Additionally, not all source-free methods are used for comparison. It may be possible that if we use other methods, e.g., Universal Source-Free Domain Adaptation, we get better results. So, at least all the existing source-free methods should be used to demonstrate the proposed method offers an advantage in terms of performance. I would like to include the best source-free single domain UDA algorithm in the results. Please check the literature for this purpose.\n\n4. The primary concern that I have for this work is selecting the values for the hyper-parameters k1 and k2. Figure 3 (a) and (b) are not accurately interpretable in my eyes but I think they show performance can vary a few percent when these hyper-parameters are not tuned properly. Hence, proper value selection for these hyper-parameters seems to be essential to observe the reported performances. Now, the question is how do you tune these parameters? From the descriptions, it seems that the authors have used performance on the target domain to select the optimal values. This is however mathematically wrong because it is as if test error has been used for hyper-parameters tuning. Using the test error for hyper-parameter tuning questions whether the proposed method can stably outperform the baselines given the small differences between the performance, as low as 0.3% in some cases. Please clarify how have you tuned the hyper-parameters in your results and how should the user tune them for other new datasets. If the test error has been used, then the reported performances are not acceptable. \n\n5. I would like to know how does Algorithm 1 scale when we have many source domains? Is it possible to offer complexity analysis? I think theoretical analysis on direct aspects of the algorithm will be more useful.\n\n\nMinor comments\n\n- The title is confusing: in the absence of punctuation, multi-source free is confusing. Maybe it is better to change it to multisource-free or use different wordings.\n\n- Line 38-40 are a little confusing. It is implied having a massive labeled source of data is the concern for the current MSDA methods, whereas privacy is the issue you address. Irrespective of implementing UDA in a source-free regime, we will need a labeled source domain data. \n\n- Figure 2 is difficult to decode, e.g., what are the colors for each domain?. I understand you have a page limit, but it can be clarified in the Appendix.\n\n----------------------------------\nFollowing internal discussions with the rest of reviewers, I finalized and maintained my updated rating.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I think the authors have addressed these aspects adequately. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "ejBWhz6MyUa", "reviewer": "Reviewer_BbBx", "summary": "This paper addresses the multi-source free domain adaptation problem where multiple pretrained source models and unlabeled target data are given. Theoretical analysis aims at proving that multiple source domains improve the probability to ensure a generalization bound. The method is developed based on mutual information maximization, pseudo labeling and consistency losses. Experiments show that the proposed method achieves slightly better results. ", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper addresses a new problem but with limited technical novelty. The proposed method combines a number of well-known techniques. The two loss terms in Eq.(2) and (3) in Section 5.1 form components of the standard mutual information criterion used for semi-supervised learning but without providing the similar insights.  The linear combination of multiple source models is also a very common integration strategy.  The pseudo-label generator in Section 5.2 follows the most common confident-based pseudo label determination strategy but with a bit tedious process. It lacks any more insightful principle. \n\nMoreover, the techniques adopted are mainly semi-supervised learning techniques and there is not really any domain adaptation factor. Does this mean source free domain adaptation can only be solved from the semi-supervised learning perspective?\n\n\nThe related work section is insufficient. It only very briefly mentions the source-free domain adaptation methods. Are there any major technical differences between this proposed method and the methods in the literature? It fails to mention and compare their method with the existing work [1] for the same multi-source free domain adaptation problem in the related work section even though this work [1] also uses information maximization, prototype-base pseudo labeling and meta-learning to learn the weighting coefficients for multiple source domains.\n\nThe experiments are not very convincing with slight performance gains. Are all the comparison methods used the same feature extraction networks?  The feature extraction network can lead to performance differences.\n\nOther questions: \n1. For the Source-Specific Transferability Perception, in line 180, why are the weight vectors taken as prototypes if the classifiers compute inner-product instead of Euclidean distance?\n\n\n[1] Sk Miraj Ahmed, et al. Unsupervised multi-source domain adaptation without access to source data. CVPR 2021. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "see main review", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "x"}, "extra_text": {}}, {"review_id": "dQdIWsxWx3e", "reviewer": "Reviewer_R2JJ", "summary": "This work presents CAiDA for Multi Source-Free Domain Adaptation in the absence of source data. The core idea is to define confident anchors to obtain pseudo labels for target domains. In the presence of multiple source domains, the contribution of each source domain for the target sample is measured using source-specific transferability perception modules. Further a class-relationship aware consistency loss is proposed to maintain semantic consistency across domains. The proposed method consistently performs better than prior arts on several datasets.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, the novelty is limited as the approach ties multiple existing ideas together. However, this work can be improved by focusing more on the distinguishing elements.\n\n1. Novelty: \n\nOne of the major drawbacks is that the novelty is limited in the proposed framework, especially when the improvement over prior work DECISION [1] is only marginal (e.g. 0.4% overall improvement on Office-Caltech dataset in Table 1). Specifically, there are two major components which are employed in previous works:\n\n    - The conditional entropy minimization ($\\mathcal{L}_{ent}$) and diversity promoting ($\\mathcal{L}_{div}$) losses are quite common in the Domain Adaptation literature. For instance, DECISION [1] employs both these losses.\n\n    - While the proposed pseudo-labeling approach is based on the nearest confident anchor (high confidence target sample), DECISION [1] labels target samples based on the nearest cluster centroid. These two methods are notionally similar - both assign target pseudo labels based on the nearest high confidence target samples.\n\nThe major distinguishing factor of this work is the Class-Relationship Aware Consistency loss and the synthetic feature augmentation (L234), which should be highlighted upon in my opinion. Also note that certain works such as [P1] optimize class confusion matrices that naturally results in feature alignment during adaptation.\n\n[P1] Jin et al, “Minimum Class Confusion for Versatile Domain Adaptation”, ECCV 2020\n\n\n2. Readability / Missing Details: \n\n    - For most parts, the manuscript is easy to follow, however there are some inconsistencies in the notation. For eg. the LHS in $\\mathcal{L}_t(\\boldsymbol(h))$ (L113) should have superscript $i$. In L184, $\\mu$ should be bold (vector).\n\n    - Could the authors clarify what is meant by “probability over all target data” in L200? Does it refer to the target data in a mini-batch, or the entire target dataset (L110)? If it is the entire target data (of size $m$) then each backpropagation through loss $\\mathcal{L}_{div}$ would require 1 epoch.\n\n    - L183 (“$\\boldsymbol{U}$ is then forwarded into a three-layer convolution network $\\boldsymbol{\\Omega}$”): Could the authors clarify the rationale behind the use of a convolution network? Here, the ordering of the sources in the matrix $\\boldsymbol{U}$ would play a major role in determining the performance of the network, due to the nature of the convolution operation. Furthermore, I couldn’t find details about this network.\n\n    - The assumptions in Section 4 are better understood by referring to the detailed proofs in the supplementary material. I suggest making Section 4 more elaborate (for e.g. the discussion in Introduction in L30-L77 is repetitive which can be trimmed down to save space).\n\n\n3. Approach / Evaluation:\n\n- It seems that the Consistency Loss $\\mathcal{L}_{crc}$ (Eq. 8) is computationally intensive, with $O(n^2 * K^2)$ operations. However, from the ablative analysis in Tables 1-3, the consistency loss results in the least gains which suggests that the loss is inefficient. Could the authors comment on this aspect? For instance, what is the increase in training time by incorporating this loss?\n\n- An ablative study on the use of $\\mathcal{C}_p$ and $\\mathcal{C}_d$ is missing. Note that, using $\\mathcal{C}_d$ is similar to using the cluster centroid for pseudo-labeling in DECISION [1]. Therefore, it is important to understand how the combination of $\\mathcal{C}_p$ and $\\mathcal{C}_d$ helps.\n\nOn a side note, it would be a good idea to present standard deviation along with the mean statistics since the average performance of the methods are close.\n\n---------\nPost Rebuttal comments: The authors have addressed my major concerns. After going through all the reviews and the author responses, I would like to vote for an acceptance.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Adequately addressed in the Supplementary Material.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "7jUiOp2f6s8", "reviewer": "Reviewer_4wZc", "summary": "This work considers unsupervised domain adaptation issue. A novel framework called CAiDA is proposed which uses the pre-trained source models rather than source data which are commonly used in traditional methods. To demonstrate the effectiveness and superiority of the proposed method, the authors did theoretical analyses as well as numerical experiments. The numerical experimental results show the proposed model can do better than state of art methods.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\nThis paper is well organized, the motivation is clear and logical, and it is easy to follow the major idea.\n\nThe results are valid. Extensive experimental results are provided to show the effectiveness of the proposed method. Besides, some theoretical analyses are also provided to provide insights behind the proposed model.\n\nCons:\nThe writing of this paper can be further polished, especially the theoretical part. The theorems should be stated in a clearer and more formal way. The theorem and assumptions should use the italic form, but the proofs should not. Besides, there are too many steps in the proof of the two theorems, these should be simplified.\n\nThere are also some typos in the paper, these should be revised in the final version.\n\nThe authors only discussed the limitations about the assumptions in theoretical analysis. The limitations of this model and why it does not work very well on some specific datasets should also be included.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "See Cons above.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2h"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The theoretical contribution for multi-source domain adaptation without source data is nice. Though some components in the proposed method are borrowed from existing techniques, the overall design of the proposed method is based on the insights of the theoretical analysis. As [1] is the first work (if I am not wrong), which only provides theoretical analysis, the authors should add a section to discuss the difference in terms of the theoretical findings between the proposed method and [1] in a revision. 

Overall, I recommend acceptance for this work.
Decision: Accept (Poster)