Confident-Anchor-Induced Multi-Source-Free Domain Adaptation
Year: 2021

Abstract:
Unsupervised domain adaptation has attracted appealing academic attentions by transferring knowledge from labeled source domain to unlabeled target domain. However, most existing methods assume the source data are drawn from a single domain, which cannot be successfully applied to explore complementarily transferable knowledge from multiple source domains with large distribution discrepancies. Moreover, they require access to source data during training, which are inefficient and unpractical due to privacy preservation and memory storage. To address these challenges, we develop a novel Confident-Anchor-induced multisource-free Domain Adaptation (CAiDA) model, which is a pioneer exploration of knowledge adaptation from multiple source domains to the unlabeled target domain without any source data, but with only pre-trained source models. Specifically, a source-specific transferable perception module is proposed to automatically quantify the contributions of the complementary knowledge transferred from multi-source domains to the target domain. To generate pseudo labels for the target domain without access to the source data, we develop a confident-anchorinduced pseudo label generator by constructing a confident anchor group and assigning each unconfident target sample with a semantic-nearest confident anchor. Furthermore, a class-relationship-aware consistency loss is proposed to preserve consistent inter-class relationships by aligning soft confusion matrices across domains. Theoretical analysis answers why multi-source domains are better than a single source domain, and establishes a novel learning bound to show the effectiveness of exploiting multi-source domains. Experiments on several representative datasets illustrate the superiority of our proposed CAiDA model. The code is available at https://github.com/Learning-group123/CAiDA.

heading: 1 Introduction
text: Unsupervised Domain Adaptation (UDA) [22,59,63] captures transferable knowledge from labeled data in a source domain to classify unlabeled data in a target domain. UDA has achieved remarkable successes in many applications, e.g., object detection [23], medical diagnose [9,12], sentiment analysis [34], etc. Generally, most of existing methods employ adversarial learning [17] to encourage the learned source and target features to be indistinguishable from each other [10, 15], or minimize the distribution discrepancy across domains by matching the statistical moments of distributions [41].
∗Equal contributions †Corresponding author
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
However, the above-mentioned methods have a strong assumption that the source data are merely drawn from a single domain. Unfortunately, the source data are often collected under different deployed environments (i.e., multiple source domains with large distribution discrepancies) in realworld applications, which makes them difficult to explore complementarily transferable knowledge from the multi-source domains for target prediction. To achieve this, Multi-Source Domain Adaptation (MSDA) [32, 34, 61] is proposed to match the features across domains and then quantify the contributions of source domains [2,41,60]. Additionally, [29,58] aim to weight the source contributions by normalizing the distance similarities between source and target domains.
Unfortunately, recent MSDA methods [29, 34, 41, 44] require massive labeled source data when adapting source domains to the target domain. This could make them inefficient and unpractical in real-world applications with sensitive information (e.g., medical diagnosis [12] and recommendation system [24]), due to privacy preservation issues, storage and security concerns [50, 51]. To this end, a new challenging and practical problem named Multi-Source-Free Domain Adaptation (MSFDA) is researched, which explores transferable knowledge from multiple source domains to target domain with only pre-trained source models and without access to any source data. The trivial solutions for tackling MSFDA via using existing single-source-free domain adaptation methods [25, 30, 33, 57] are to adapt each source model individually and simply take an average prediction of source models. However, they cannot explore the contributions of the complementary information transferred from different source domains, due to the lack of source data. Therefore, tackling the MSFDA problem is a challenging but rarely-researched task.
To address the MSFDA problem, we develop a novel Confident-Anchor-induced multi-source-free Domain Adaptation (CAiDA) model, which is a pioneer exploration to capture transferable information from multiple source models to promote target prediction without access to source data. Specifically, a source-specific transferable perception module is designed to calibrate the contributions of the transferability from multiple source domains. We develop a confident-anchor-induced pseudo label generator to mine pseudo labels for the unlabeled target data, by incorporating with the quantified source transferability contributions. We construct a confident anchor group to assign each target sample with a semantic-nearest confident anchor, and perform feature augmentation between them to generate confident target pseudo label. A class-relationship-aware consistency loss is proposed to ensure the semantic consistency of underlying inter-class relationships across domains via the alignment of soft confusion matrices. Furthermore, based on some mild assumptions, theoretical analysis guarantees that multiple source models could help generate more reliable pseudo labels. Our theoretical analysis also provides a novel learning bound for MSFDA, which reveals that multiple source models help achieve a tighter generalization error bound for the target domain. We verify the effectiveness of our proposed model via comparison experiments on benchmark datasets. The main contributions of our work are summarized as follows:
• We develop a novel Confident-Anchor-induced multi-source-free Domain Adaptation (CAiDA) model to explore transferable knowledge from multiple source domains to assist target prediction with pre-trained source models and without access to source data. To our best knowledge, this paper is a pioneer exploration of multi-source-free domain adaptation in the field of transfer learning.
•We propose a novel MSFDA theory, which shows that multiple source models could improve the possibility of obtaining more reliable pseudo labels under some mild assumptions. Our theoretical analysis also provides a novel generalization bound for MSFDA to show the effect of multiple source models. This novel bound implies a positive answer to the solvability of MSFDA problem.
• A source-specific transferable perception module and a class-relationship-aware consistency loss are designed to quantify the contributions of the transferability of multiple source domains and ensure the semantic consistency of underlying inter-class relationships across domains, respectively.
• Based on the confident pseudo labeling strategy in theoretical analysis, a confident-anchor-induced pseudo label generator is proposed to generate pseudo labels for the target domain by establishing a confident anchor group and assigning each target sample with a semantic-nearest confident anchor.

heading: 2 Related Work
text: Unsupervised Domain Adaptation aims to borrow transferable knowledge from source domain to promote the prediction of the unlabeled target domain. After Hoffman et al. [22] introduce ad-
versarial learning [17] into domain adaptation, enormous adversarial-based methods [8, 11, 15, 59] are proposed to perform feature-level or pixel-level distribution alignment. Besides, some moment matching-based methods [14, 35, 41] focus on matching the distribution statistical moments at different orders to minimize the distribution discrepancy across domains. Furthermore, some researches design the adversarial dropout [28], batch normalization [52] and auxiliary reconstruction tasks [5,16] to narrow the domain discrepancy. Unfortunately, the methods mentioned above assume massive labeled source data are available. This is unpractical due to privacy and security concerns.
Source-Free Domain Adaptation (SFDA) [33] is studied to tackle the above challenge. A common strategy in SFDA methods is to mine the confident pseudo labels for target domain. To this end, [33] uses a self-supervised pseudo labeling strategy, and [25] designs a confidence-based sample filtering method. [42] alleviates the negative transfer brought by noisy pseudo labels through confidence reweighting and regularization. In addition to the pseudo labeling strategy, the model adaptation strategy has also been studied. For example, [30, 57] employ an adversarial learning strategy to perform model adaptation with only pre-trained source models. However, they cannot be applied to tackle the MSFDA problem, due to the distribution discrepancies across different source domains.
Multi-Source Domain Adaptation is extended from vanilla domain adaptation [12, 15, 41] by exploring transferable knowledge from multiple sources. To capture the relationship between different source domains and a given target domain, Guo et al. [19] design a mixture-of-experts model for unsupervised domain adaptation from multiple sources. [56] focuses on determining which source domain is the best for target prediction via dynamic curriculum learning. Some discrepancy-based methods aim to narrow the distribution discrepancy across domains by minimizing different measures such as the Rényi-divergence [21] and maximum mean discrepancy [19]. Moreover, some adversarial-based methods focus on optimizing the H-divergence [41, 60], generative adversarial loss [55,61] and Wasserstein distance [32] to make features from multiple sources indistinguishable for a shared discriminator. [34, 44] perform knowledge adaptation at the pixel-level by replaying multiple source domains. Due to lack of source data, above strategies in MSDA may be invalid and unsuitable to address the challenging MSFDA problem. To this end, Ahmed et al. [1] utilize nearest distance measure to mine target pseudo labels, and weight the predictions from multiple source models for the MSFDA task. It may result in that the generation process has high probability to obtain noisy labels [4, 31, 54] when the strategy is not matched with the target data, while our model could generate confident pseudo labels from two different perspectives, i.e., geometry and probability.

heading: 3 Problem Setting
text: Let X and Y = [K] := {1, ...,K} denote the feature space and label space. A domain is a joint distribution PXY on X × Y . There are n source domains {P iXY }ni=1. For any source domain P iXY , a corresponding neural-network-based predictor (model) hi : X → RK is given. Given a target domain P tXY with unlabeled target data T = {xj}mj=1 ∼ P tX , i.i.d., the aim of multi-source-free domain adaptation (MSFDA) is to classify the unlabeled target data by utilizing T and {hi}ni=1.
Let ℓ denote a non-negative loss function defined over RK × RK . Given a hypothesis space H ⊂ {h : X → RK}, we denote Lis(h) = E(x,y)∼P iXY ℓ(h(x),Φ(y)) and Lt(h) = E(x,y)∼P tXY ℓ(h(x),Φ(y)) as the risks with respect to the i-th source domain and a given target domain, where Φ : Y → RK maps any label y to a corresponding one-hot vector. The source predictor hi is a vector-valued function, i.e., hi(x) = [hi1(x), ..., h i K(x)]
⊤, and consists of two basic components: feature extractor f i : X → Rd and classifier ci : Rd → RK , where d is the dimension of extracted features. Therefore, hi can be rewritten as ci ◦ f i. After using softmax as the activation function in the output layer, we have hik ≥ 0 and ∑K k=1 h i k = 1. To ensure that each predictor hi is a relatively accurate predictor for the domain P iXY , we assume that the predictor h i is ϵ-accurate under ℓ1 loss, i.e., E(x,y)∼P iXY ∥h i(x)−Φ(y)∥ℓ1 < ϵ, for any i ∈ [n].

heading: 4 Theoretical Analysis
text: Without any relations between source domains and target domain, MSFDA cannot be effectively addressed from the theoretical view. To bridge source and target domains, Mansour et al. [40] and
Miraj Ahmed et al. [1] assume that PXY = ∑n i=1 λiP i XY and P t Y |X = P i Y |X = P j Y |X , for any
i, j ∈ [n], where λi ≥ 0 and ∑n
i=1 λi = 1. This assumption is very strong and may be unrealistic in many real-world applications. Motivated by the meta learning [43] and domain generalization [3], in this paper, we propose some novel and mild assumptions to address MSFDA problem. Assumption 1 (Meta Assumption.) P tXY , P 1XY , ..., PnXY are drawn (i.i.d.) from a meta distribution P , which is defined over a joint distribution space PXY . Assumption 2 (Regular Domain.) Let the joint distribution space PXY is endowed with total variation distance dTV(·, ·). The target domain P tXY is a regular domain, i.e., for any σ > 0, P(N σP tXY ) > 0, where N σ P tXY = {P : dTV(P, P tXY ) < σ}. It is easy to check, if the meta distribution P is discrete or continuous with continuous density function, then the target domain P tXY drawn byP is a regular domain with probability 1. In addition, to weaken the assumption P tY |X = P i Y |X = P j Y |X , for any i, j ∈ [n], our key strategy is to consider the anchor point assumption that is studied in label-noise learning [36].
We say a given point x is a τ -anchor point, if there exists a predictor hi with the largest score hik(x), such that hik(x)− hic(x) ≥ τ , for any c ∈ [K] and c ̸= k. hi is the τ -anchor predictor for x. Assumption 3 (τ -Anchor Point Assumption.) Given a τ -anchor point x, suppose that all τ - anchor predictors for x are hi1 , ...,hil , then the true label of the τ -anchor point x is k, if hjk(x) is the largest score among scores hi1c , ..., h il c , for any c ∈ [K].
When the source predictors are accurate enough, Assumption 3 implies that the source and target conditional distributions are similar in the high confidence region. Hence, it is much weaker than the traditional assumption P tY |X = P i Y |X = P j Y |X , for i, j ∈ [n]. τ is regarded as a threshold to distinguish which data has highly confident prediction. In general, τ is close to 1.
Given any hi, it is easy to check that there exists a matrix function Ai(x) = [aikl(x)] such that hik(x) = ∑K l=1 a i kl(x)P t Y |X(l|x) with ∑K k=1 a i kl = 1. We say the diagonal elements {aill}Kl=1 as the transfer factor for hi and P tY |X . Generally, A i(x) = [aikl(x)] is not unique, thus the transfer factor may be not unique. The following theorem indicates that Assumption 3 holds if we give proper assumptions for transfer factor. Theorem 1 Suppose that the Bayesian label is true label [7]. If there exist transfer factors and a constant B < K such that maxi∈[n],l∈[K] aill ≥ B, then Assumption 3 holds with τ > 1−B/K. Theorem 1 provides a theoretical support for Assumption 3 and indicates that when the transfer factors are positive, Assumption 3 always holds with a proper τ . To further study the highly confident pseudo labeling strategy, the following theorem provides a lower bound to estimate the number of highly confident pseudo labels, i.e., the number of τ -anchor points. Theorem 2 Assume Assumptions 1 and 2 hold and the conditional distribution P tY |X can be presented as a labeling function, i.e., P tY |X(y|x) = 0 or 1. Given η > 0, if m ≥ n and (1−η)(1−τ) > ϵ + 2σ + 2 √ log(2m/δ)/2m, then with probability at least 1 − δ − (1 −P(N σPXtYt ))
n > 0, at least ηm target data are τ -anchor points, where ϵ is the upper bound of the accuracies of source predictors, and σ is introduced in Assumption 2.
Theorem 2 indicates that multi-source predictors improve the probability to obtain more τ -anchor points. To further understand the effect of multi-source domains, we build a novel learning bound for the MSFDA task. Let Aτ be the set consisting of all τ -anchor points. Denote the empirical risk L̂τs (h) by 1|Aτ∩T | ∑ x∈Aτ∩T ℓ(h(x),Φ(y)), where y is the label of the anchor point x.
Theorem 3 Given Assumption 3 and some assumptions used in Theorem 2, and suppose that the loss ℓ has upper bound M > 0 and hypothesis space H has finite Natarajan dimension, for η > 0, if m ≥ n and (1 − η)(1 − τ) > ϵ + 2σ + 2 √ log(2m/δ)/2m, then for any h ∈ H and b ∈ (0, 1), there exists a constant C(b,K) such that with the probability at least 1− 2δ− 2(1−P(N σP tXY )) n:
∣∣Lt(h)− L̂τs (h)∣∣ ≤MC(b,K) √ log(2/δ)
η1−bm1−b +M
2σ + ϵ
1− τ − 2σ − ϵ , (1)
where ϵ is the upper bound of accuracies of source predictors, and σ is introduced in Assumption 2.
Theorem 3 shows that multiple source domains improve the probability to ensure a tighter generalization bound, i.e., Eq. (1) holds. Note that the Natarajan dimension used in Theorem 3 is a bit outdated. However, the Natarajan dimension can be replaced and Theorem 3 can be updated without any technical barriers, if there exists better generalization theory for supervised learning.
Summary of Theoretical Analysis: The reasons to develop MSFDA are to study the solvability of MSFDA and understand how multi-source predictors benefit the target domain’s classification. By Theorems 2 and 3, we realize that multi-source predictors improve the probability to obtain more highly confident pseudo labels, resulting in a tighter generalization bound. The generalization bound in Theorem 3 gives a positive answer to the solvability of MSFDA. Additionally, Theorems 1 and 2 also imply two interesting and important results: Theorem 1 provides the first theoretical support to the confident pseudo labeling strategy, and Theorem 2 provides the first lower bound of the number of highly confident pseudo labels. As we know, the theoretical results in Theorems 1 and 2 are novel.

heading: 5 The Proposed CAiDA Model
text: The graphical illustration of our proposed model is depicted in Figure 1. It mainly consists of three significant components: source-specific transferable perception, confident-anchor-induced pseudo label generator and class-relationship-aware consistency loss, which are elaborated as follows.

heading: 5.1 Source-Specific Transferable Perception
text: Generally, in multi-source domain adaptation (MSDA), different source domains have different contributions to improve the performance on target domain [21,61]. To this end, many previous MSDA methods match the features across different domains, and then quantify the contributions of source domains by taking the average of the trained source predictors [41,60], or weight the trained source predictors by normalizing the distance similarities [29,58] between source and target domains. However, due to the lack of source data, these methods cannot employ source data to match features and cannot be successfully applied to multi-source-free domain adaptation (MSFDA) tasks.
Therefore, a source-specific transferable perception module is developed to automatically quantify the contributions of the transferability of source domains, as shown in Figure 1. Specifically, the one-hot encoding vector ui ∈ Rn of the i-th source domain can be considered as the unique domain characterization, which is then employed as network input to quantify the contribution of the transferability. We then concatenate all source domains’ one-hot characterizations together to obtain U = [u1, · · · ,un]⊤ ∈ Rn×n. U is then forwarded into a Multi-Layer Perceptron (MLP) network Ω to automatically quantify the contribution of the transferability µ ∈ Rn of n source domains: µ = Ω(U) = Ω([u1,u2, · · · ,un]⊤), such that ∑n i=1 µi = 1, where µi denotes the quantified contribution of the i-th source domain to the target prediction.
When the source data are unavailable, it is difficult to narrow distribution discrepancy across domains, due to lack of any supervised information. Inspired by [33], we freeze the network parameters of classifiers {ci}ni=1 and solely perform distribution adaptation across domains on feature extractors {f i}ni=1 via information maximization [26], since {ci}ni=1 contain class distribution information of source domains. However, Liang et al. [33] cannot be effectively applied to multi-source-free do-
main adaptation scenario, where different source domains have different transferable contributions on target prediction. Therefore, Lent is proposed to minimize conditional entropy of target outputs by incorporating the source-specific transferable perception µ:
Lent = − 1
m m∑ j=1 [ K∑ k=1 hk(x j) log(hk(x j))], (2)
where hk(xj) is the k-th coordinate value of h(xj), here h(xj) = ∑n i=1 µih i(xj) denotes the combination of source predictions. The larger value of µi indicates the larger contribution of transferability of the i-th source domain for target adaptation. Unfortunately, our proposed CAiDA model may suffer from the trivial solution of Eq. (2) by predicting all target data as a single class to minimize Eq. (2). To tackle this issue, Ldiv is designed to consider class prediction diversity by maximizing the entropy of empirical label distribution [6] predicted by different source domains:
Ldiv = ∑K
k=1 p̂k log p̂k, (3)
where p̂k = 1m ∑m j=1 hk(x j) denotes the mean prediction probability over target data.

heading: 5.2 Confident-Anchor-Induced Pseudo Label Generator
text: Although the minimization of both Lent and Ldiv promotes the class diversity and knowledge adaptation between multiple unseen source data and target data, it cannot circumvent erroneous label assignment due to the noisy target prediction brought by domain discrepancy. To alleviate this issue, based on Assumption 3, a confident-anchor-induced pseudo label generator is developed to mine confident pseudo labels for target data, as shown in Figure 1. Specifically, based on the τ -anchor assumption, we select target data xj as confident anchor when the maximum category prediction probability hk∗(xj) is larger than the rest of category prediction probabilities hk(xj) (k ̸= k∗) by a threshold τp. For each confident anchor, we integrate the features extracted from multiple source extractors together to obtain a probability-based confident anchor group Cp:
Cp = {f(xj)|hk∗(xj)− hk(xj) ≥ τp,∀k ̸= k∗, j = 1, 2, · · ·m}, (4) where f(xj) = ∑n
i=1 µif i(xj) denotes the feature of the j-th target data. τp is defined as the
median value of the probability difference between the largest and the second largest probabilities over all target data. Furthermore, inspired by the proposed τ -anchor assumption, to circumvent the noisy anchor in Cp, we construct a distance-based confident anchor group Cd. The feature centroid of the k-th class induced by the i-th source domain for the whole target data is defined as ξik =∑m
j=1 h i k(x
j)f i(xj)/ ∑m
j=1 h i k(x j). {ξik}ni=1 are then weighted with source-specific transferability to obtain the feature centroid ξk of the k-th class over all source domains via ξk = ∑n i=1 µiξ i k. The distance between the feature of xj and the k-th feature centroid ξk is denoted as d(f(xj), ξk), where d(·, ·) is a distance measure function. When the minimum distance d(f(xj), ξk∗) is shorter than the rest of distances d(f(xj), ξk)(k ̸= k∗) by a threshold τd, we select the target data xj into Cd:
Cd = {f(xj)|d(f(xj), ξk)− d(f(xj), ξk∗) ≥ τd, ∀k ̸= k∗, j = 1, 2, · · ·m}, (5)
where we set τd as the median value of the distance difference between the shortest and the second shortest distances over all target data. Therefore, the final confident anchor group C is obtained by performing an intersection operation between Cp and Cd, i.e., C = Cp ∩ Cd. To generate confident pseudo labels for unconfident target data, we select a semantic-nearest confident anchor from C for each target data via continual similarity searching. To be specific, given an unconfident target data xj , we search a serial of unconfident guiding data consecutively using the distance measure function d(·, ·) in the feature space, until the confident anchor from C is detected. During each searching process, the previous searched guiding data are not considered in the following iteration. Denote I as the set containing previously searched guiding data, xjc as the searched confident anchor closest to xj . The guiding sample xj(t) in the t-th search could be obtained by:
xj(t) = arg min xj∈T d(f(xj(t−1)), f(xj)), subject to xj(t) ̸= xj(t−1),xj(t) /∈ I, (6)
Algorithm 1 The Searching Process of Semantic-Nearest Confident Anchor xjc.
1: Input: xj , C, t = 1; 2: Initialize: xj(t−1) = xj , I = ∅; 3: While xj(t−1) /∈ C do 4: Obtain xj(t) via Eq. (6); 5: Update I via adding xj(t) into I; 6: Update xj(t−1) via xj(t−1) ← xj(t); 7: End 8: Obtain xjc via xjc ← xj(t−1) 9: Return xjc.
Algorithm 2 The Optimization of Our Model. 1: Input: {hi}ni=1, T , E epoches, B batches; 2: Initialize: κ1, κ2, the parameters of Ω; 3: For e = 1, · · · , E do 4: Obtain pseudo labels via Eq. (7); 5: For b = 1, · · · , B do 6: Select a mini-batch of target data; 7: Update {f i}ni=1 and Ω via Eq.(9); 8: End 9: End 10: Return: {hi}ni=1 and Ω.
where xj(t−1) is the guiding data in the previous search. The searching process of xjc for the j-th target data is summarized in Algorithm 1. Moreover, we fuse the target data and their corresponding confident anchor in the feature space for feature augmentation, and the synthetic feature is denoted as fsyn(xj) = (1 − ω)f(xj) + ωf(xjc), where ω ∈ [0, 1] is the random weight to determine the influence of confident anchor on fsyn(xj). Therefore, the confident-anchor-induced pseudo label ŷj of the j-th target data xj and the classification loss Lcls of whole target data are formulated as:
ŷj = arg min k∈[K] d(fsyn(x j), ξk); Lcls =
1
m m∑ j=1 K∑ k=1 [−1ŷj=k log(hk(xj))]. (7)

heading: 5.3 Class-Relationship-Aware Consistency Loss
text: The inherent relationships between different classes have semantic consistency across domains, regardless of distribution discrepancy. In light of this, aligning class relationships could promote more shared transferable knowledge from source domains towards target adaptation. To achieve this, as depicted in Figure 1, a class-relationship-aware consistency loss Lcrc is designed to encourage target data from the same class to be compactly clustered together while preserving the intrinsic inter-class relationships via soft confusion matrix alignment. To be specific, the soft label distribution sik of the k-th class predicted via the i-the source predictor is formulated as sik = 1 m ∑m j=1[1ŷj=kµih
i(xj)]. The collection of soft label distributions {sik}Kk=1 represents a kind of soft confusion matrix associated with a particular domain, encoding inter-class relationships learned by the i-th source predictor (e.g., computers have more similar semantic relationship with desks than horses). Without access to the source data, Lcrc aims to align soft confusion matrices from different source predictors:
Lcrc = 1
2n2 n∑ i=1 n∑ i′=1 K∑ k=1 ( KL(sik||si ′ k ) + KL(s i′ k ||sik) ) , (8)
where KL(p||q) = ∑
r pr log pr qr denotes the Kullback-Leibler (KL) divergence. The complexity of Lcrc is not problematic in practice, due to the limited number of source domains. In summary, the overall optimization objective to optimize {f i}ni=1 and Ω is formulated as:
L = Lent + Ldiv + κ1Lcls + κ2Lcrc, (9)
where κ1, κ2 are the balanced weights. The optimization of our model is presented in Algorithm 2.

heading: 6 Experiments

heading: 6.1 Datasets and Baseline Methods
text: Datasets: Office-31 [22] consists of three representative domains with 31 shared object categories in the office environment, i.e., Amazon (A), Webcam (W) and DSLR (D). Office-Caltech [18] is an extension dataset of Office-31 [22] by adding an additional subset called Caltech-256 (C) on it and extracting 10 common object classes among them. Office-Home [46] is composed of four different domains including Product (Pr), Clipart (Cl), Art (Ar), and Realworld (Re). Each of these subsets
consists of 65 shared object categories. Digits-Five [41] contains five digit recognition subsets including MNIST-M (MM), USPS (UP), MNIST (MT), SVHN (SV) and Synthetic Digits (SY).
Baseline Methods: To validate the effectiveness of our model, we conduct comparison experiments with a wide array of baseline methods. Specifically, MDAN [60], DCTN [55], M3SDA [41], MDDA [62] and LtC-MSDA [47] are traditional representative multi-source domain adaptation methods with access to source data. BAIT [57], PrDA [25], SHOT [33] and MA [30] focuses on unsupervised single-source domain adaptation without access to source data. We compare against the multi-source extensions of [25, 30, 33, 57] by taking an average of target soft predictions from all adapted source models. DECISION [1] combines the source adaptation models with suitable weights automatically for multi-source-free domain adaptation. Furthermore, Source only denotes the performance of evaluation on target data when combining the rest of source data for training, and Source model only represents the average performance over the predictions of all source models.

heading: 6.2 Experiments on Office-31 and Office-Caltech Datasets
text: Performance Comparisons: The comparisons between our CAiDA model and other state-of-theart methods on Office-31 [22] and Office-Caltech [18] datasets are presented in Table 1. We have the following conclusions from the results in Table 1: 1) When compared with the multi-source domain adaptation methods [41, 47, 55, 60, 62] that employ source data for training, our proposed model without access to source data significantly outperforms them by a large margin of 1.4%∼8.3% in terms of mean accuracy. It verifies the superiority of our model to tackle multi-source-free domain adaptation. 2) Our model performs better than the multi-source extensions of single-source-free domain adaptation methods [25, 30, 33, 57], which validates the effectiveness of our pseudo label generation process. 3) The performance of our model is better than [1] for all evaluation tasks, since the confident-anchor-induced pseudo label generator and class-relationship-aware consistency loss promote the adaptation performance. Figure 2 shows that our model significantly narrows distribution discrepancy across domains on Office-31 [22] when compared with Source model only.
Ablation Studies: This subsection introduces the effectiveness of each component in our model via ablation studies on Office-31 [22] and Office-Caltech [18] datasets, as shown in Table 1. Ours-
w/oEnt, Ours-w/oDiv, Ours-w/oCls and Ours-w/oCrc are the abbreviations of training our proposed model without Lent, Ldiv, Lcls and Lcrc, respectively. When any one of component of our model is abandoned, the performance degrades 0.4%∼2.3% in terms of average accuracy, which illustrates the rationality and effectiveness of all components to cooperate together. All modules play an indispensable role in improving performance, even though our model has no access to source data.
Parameter Investigation: This subsection investigates the effects of hyper-parameters κ1 in a range of {0.1, 0.3, 0.5, 0.7, 0.9} and κ2 in a range of {10−4, 10−3, 10−2, 10−1, 1} on Office-31 [22] and Office-Caltech [18] datasets, as shown in Figure 3 (a)(b). It validates that our proposed model achieves stable performance over a wide range of hyper-parameters selection. Moreover, the best performance of our proposed model on target domain is obtained when κ1 = 0.7 and κ2 = 10−2.
Contribution Weights and Convergence Analysis: Figure 3 (c)(d) present the contribution weights of domain A and convergence curves of our proposed model on Office-31 [22] and Office-Caltech [18] datasets. The source-specific transferable perception module could quantify the transferability contributions of complementary information from multiple source domains to target prediction. Furthermore, the accuracy on Office-Caltech [18] dataset converges to a stable value after a few epoches, which demonstrates the convergence effectiveness of our proposed CAiDA model.

heading: 6.3 Experiments on Office-Home Dataset
text: Performance Comparisons: As presented in Table 2, we conduct comparison experiments on Office-Home [46] dataset to illustrate the effectiveness of our model. We have the following observations from Table 2: 1) Without access to source data, our model significantly outperforms the representative multi-source domain adaptation methods [41, 47, 55, 60, 62] by 3.5%∼5.6% in terms of average accuracy. 2) The confident pseudo label generator encourages our proposed model to perform better than [1], which verifies the superiority of our model for multi-source-free domain adaptation. 3) The performance of our model improves a large margin of 1.9%∼5.5% in terms of mean accuracy, compared with [25, 30, 33, 57]. It validates the efficiency of source-specific transferable perception strategy and class-relationship-aware consistency to narrow distribution discrepancy.
Ablation Studies: As introduced in Table 2, we conduct ablation studies of our model on OfficeHome [46] dataset to illustrate the rationality of all designed modules. When compared with Ours,
the performances of Ours-w/oEnt, Ours-w/oDiv, Ours-w/oCls and Ours-w/oCrc degrades 1.6%, 1.9%, 2.4% and 0.6% in terms of average accuracy, respectively. It validates that all designed modules could cooperate well to address MSFDA task. Moreover, a confident-anchor-induced pseudo label generator could reduce the distribution discrepancy via mining confident pseudo labels.

heading: 6.4 Experiments on Digits-Five Dataset
text: This subsection presents the ablation studies and comparison experiments between our model and other competing methods on Digits-Five [41] dataset, as introduced in Table 3. Some conclusions are drawn from the presented results in Table 3: 1) Our proposed model performs the best in terms of average accuracy when compared with multi-source domain adaptation methods [1,41,47,55,60,62] and single-source-free adaptation methods [25,30,33,57]. The significant performance improvement demonstrates the effectiveness of our model, even without access to source data. 2) Our model outperforms [25, 30, 33, 57] by 8.3%∼10.7% mean accuracy, since it automatically quantifies the contributions of different source domains to target adaptation. When we compare our model with [1, 41, 47, 55, 60, 62], the confident-anchor-induced pseudo label generator and class-relationshipaware consistency loss facilitate the performance improvement by mining confident pseudo labels and aligning soft confusion matrices across domains. 3) The performance degradation in ablation studies (i.e., Ours-w/oEnt, Ours-w/oDiv, Ours-w/oCls and Ours-w/oCrc) validates that all proposed components are designed effectively and reasonably to explore transferable knowledge.

heading: 7 Conclusion and Future Work
text: This paper proposes a novel confident-anchor-induced multi-source-free domain adaptation (CAiDA) model to capture transferable knowledge from multiple source domains without access to source data. To be specific, a source-specific transferable perception module is developed to automatically weight the contributions of the transferability of source domains. Meanwhile, we design a confident-anchor-induced pseudo label generator to mine confident pseudo labels for the target domain by establishing a confident anchor group, and develop a class-relationship-aware consistency loss to capture consistent inter-class relationships across domains. Theoretical analysis provides some new perspectives to the highly confident pseudo labeling strategy, and gives theoretical support for MSFDA task under some proper mild assumptions. Extensive experiments illustrate the superiority of our proposed model. In the future, we will extend MSFDA to the multi-label [48, 49] or open-set [13, 37, 53] scenarios and use MSFDA techniques to study pandemic [38, 39].

heading: Acknowledgments
text: This work was partially supported by National Nature Science Foundation of China under Grant 62003336; National Postdoctoral Innovative Talents Support Program under Grant BX20200353; Nature Foundation of Liaoning Province of China under Grant 2020-KF-11-01; and Australian Research Council Projects under Grant DP-180103424, DE-190101473, and IC-190100031.

Review summaries:
In this paper, an algorithm for multi-source domain adaptation is proposed. The algorithm is developed in a source-free regime to address the concern for privacy. A loss function is designed to integrate the source-specific end-to-end classifiers using solely the target domain samples, which preserves privacy. The loss function consists of four terms to benefit from enforcing source-specific transferability, anchor-induced pseudo-labels, and class relationship-aware consistency. A PAC-learning style theoretical analysis is provided to demonstrate that using multiple sources may improve the generalizability on the target domain. Finally, experiments on five standard UDA datasets are provided to demonstrate that the method is effective.

This work presents CAiDA for Multi Source-Free Domain Adaptation in the absence of source data. The core idea is to define confident anchors to obtain pseudo labels for target domains. In the presence of multiple source domains, the contribution of each source domain for the target sample is measured using source-specific transferability perception modules. Further a class-relationship aware consistency loss is proposed to maintain semantic consistency across domains. The proposed method consistently performs better than prior arts on several datasets.

This paper addresses the multi-source free domain adaptation problem where multiple pretrained source models and unlabeled target data are given. Theoretical analysis aims at proving that multiple source domains improve the probability to ensure a generalization bound. The method is developed based on mutual information maximization, pseudo labeling and consistency losses. Experiments show that the proposed method achieves slightly better results. 

This work considers unsupervised domain adaptation issue. A novel framework called CAiDA is proposed which uses the pre-trained source models rather than source data which are commonly used in traditional methods. To demonstrate the effectiveness and superiority of the proposed method, the authors did theoretical analyses as well as numerical experiments. The numerical experimental results show the proposed model can do better than state of art methods.