Confident Anchor-Induced Multi-Source Free Domain Adaptation

1 Introduction . Unsupervised Domain Adaptation ( UDA ) [ 22,59,63 ] captures transferable knowledge from labeled data in a source domain to classify unlabeled data in a target domain . UDA has achieved remarkable successes in many applications , e.g. , object detection [ 23 ] , medical diagnose [ 9,12 ] , sentiment analysis [ 34 ] , etc . Generally , most of existing methods employ adversarial learning [ 17 ] to encourage the learned source and target features to be indistinguishable from each other [ 10 , 15 ] , or minimize the distribution discrepancy across domains by matching the statistical moments of distributions [ 41 ] . ∗Equal contributions †Corresponding author 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . However , the above-mentioned methods have a strong assumption that the source data are merely drawn from a single domain . Unfortunately , the source data are often collected under different deployed environments ( i.e. , multiple source domains with large distribution discrepancies ) in realworld applications , which makes them difficult to explore complementarily transferable knowledge from the multi-source domains for target prediction . To achieve this , Multi-Source Domain Adaptation ( MSDA ) [ 32 , 34 , 61 ] is proposed to match the features across domains and then quantify the contributions of source domains [ 2,41,60 ] . Additionally , [ 29,58 ] aim to weight the source contributions by normalizing the distance similarities between source and target domains . Unfortunately , recent MSDA methods [ 29 , 34 , 41 , 44 ] require massive labeled source data when adapting source domains to the target domain . This could make them inefficient and unpractical in real-world applications with sensitive information ( e.g. , medical diagnosis [ 12 ] and recommendation system [ 24 ] ) , due to privacy preservation issues , storage and security concerns [ 50 , 51 ] . To this end , a new challenging and practical problem named Multi-Source-Free Domain Adaptation ( MSFDA ) is researched , which explores transferable knowledge from multiple source domains to target domain with only pre-trained source models and without access to any source data . The trivial solutions for tackling MSFDA via using existing single-source-free domain adaptation methods [ 25 , 30 , 33 , 57 ] are to adapt each source model individually and simply take an average prediction of source models . However , they can not explore the contributions of the complementary information transferred from different source domains , due to the lack of source data . Therefore , tackling the MSFDA problem is a challenging but rarely-researched task . To address the MSFDA problem , we develop a novel Confident-Anchor-induced multi-source-free Domain Adaptation ( CAiDA ) model , which is a pioneer exploration to capture transferable information from multiple source models to promote target prediction without access to source data . Specifically , a source-specific transferable perception module is designed to calibrate the contributions of the transferability from multiple source domains . We develop a confident-anchor-induced pseudo label generator to mine pseudo labels for the unlabeled target data , by incorporating with the quantified source transferability contributions . We construct a confident anchor group to assign each target sample with a semantic-nearest confident anchor , and perform feature augmentation between them to generate confident target pseudo label . A class-relationship-aware consistency loss is proposed to ensure the semantic consistency of underlying inter-class relationships across domains via the alignment of soft confusion matrices . Furthermore , based on some mild assumptions , theoretical analysis guarantees that multiple source models could help generate more reliable pseudo labels . Our theoretical analysis also provides a novel learning bound for MSFDA , which reveals that multiple source models help achieve a tighter generalization error bound for the target domain . We verify the effectiveness of our proposed model via comparison experiments on benchmark datasets . The main contributions of our work are summarized as follows : • We develop a novel Confident-Anchor-induced multi-source-free Domain Adaptation ( CAiDA ) model to explore transferable knowledge from multiple source domains to assist target prediction with pre-trained source models and without access to source data . To our best knowledge , this paper is a pioneer exploration of multi-source-free domain adaptation in the field of transfer learning . •We propose a novel MSFDA theory , which shows that multiple source models could improve the possibility of obtaining more reliable pseudo labels under some mild assumptions . Our theoretical analysis also provides a novel generalization bound for MSFDA to show the effect of multiple source models . This novel bound implies a positive answer to the solvability of MSFDA problem . • A source-specific transferable perception module and a class-relationship-aware consistency loss are designed to quantify the contributions of the transferability of multiple source domains and ensure the semantic consistency of underlying inter-class relationships across domains , respectively . • Based on the confident pseudo labeling strategy in theoretical analysis , a confident-anchor-induced pseudo label generator is proposed to generate pseudo labels for the target domain by establishing a confident anchor group and assigning each target sample with a semantic-nearest confident anchor . 2 Related Work . Unsupervised Domain Adaptation aims to borrow transferable knowledge from source domain to promote the prediction of the unlabeled target domain . After Hoffman et al . [ 22 ] introduce ad- versarial learning [ 17 ] into domain adaptation , enormous adversarial-based methods [ 8 , 11 , 15 , 59 ] are proposed to perform feature-level or pixel-level distribution alignment . Besides , some moment matching-based methods [ 14 , 35 , 41 ] focus on matching the distribution statistical moments at different orders to minimize the distribution discrepancy across domains . Furthermore , some researches design the adversarial dropout [ 28 ] , batch normalization [ 52 ] and auxiliary reconstruction tasks [ 5,16 ] to narrow the domain discrepancy . Unfortunately , the methods mentioned above assume massive labeled source data are available . This is unpractical due to privacy and security concerns . Source-Free Domain Adaptation ( SFDA ) [ 33 ] is studied to tackle the above challenge . A common strategy in SFDA methods is to mine the confident pseudo labels for target domain . To this end , [ 33 ] uses a self-supervised pseudo labeling strategy , and [ 25 ] designs a confidence-based sample filtering method . [ 42 ] alleviates the negative transfer brought by noisy pseudo labels through confidence reweighting and regularization . In addition to the pseudo labeling strategy , the model adaptation strategy has also been studied . For example , [ 30 , 57 ] employ an adversarial learning strategy to perform model adaptation with only pre-trained source models . However , they can not be applied to tackle the MSFDA problem , due to the distribution discrepancies across different source domains . Multi-Source Domain Adaptation is extended from vanilla domain adaptation [ 12 , 15 , 41 ] by exploring transferable knowledge from multiple sources . To capture the relationship between different source domains and a given target domain , Guo et al . [ 19 ] design a mixture-of-experts model for unsupervised domain adaptation from multiple sources . [ 56 ] focuses on determining which source domain is the best for target prediction via dynamic curriculum learning . Some discrepancy-based methods aim to narrow the distribution discrepancy across domains by minimizing different measures such as the Rényi-divergence [ 21 ] and maximum mean discrepancy [ 19 ] . Moreover , some adversarial-based methods focus on optimizing the H-divergence [ 41 , 60 ] , generative adversarial loss [ 55,61 ] and Wasserstein distance [ 32 ] to make features from multiple sources indistinguishable for a shared discriminator . [ 34 , 44 ] perform knowledge adaptation at the pixel-level by replaying multiple source domains . Due to lack of source data , above strategies in MSDA may be invalid and unsuitable to address the challenging MSFDA problem . To this end , Ahmed et al . [ 1 ] utilize nearest distance measure to mine target pseudo labels , and weight the predictions from multiple source models for the MSFDA task . It may result in that the generation process has high probability to obtain noisy labels [ 4 , 31 , 54 ] when the strategy is not matched with the target data , while our model could generate confident pseudo labels from two different perspectives , i.e. , geometry and probability . 3 Problem Setting . Let X and Y = [ K ] : = { 1 , ... , K } denote the feature space and label space . A domain is a joint distribution PXY on X × Y . There are n source domains { P iXY } ni=1 . For any source domain P iXY , a corresponding neural-network-based predictor ( model ) hi : X → RK is given . Given a target domain P tXY with unlabeled target data T = { xj } mj=1 ∼ P tX , i.i.d. , the aim of multi-source-free domain adaptation ( MSFDA ) is to classify the unlabeled target data by utilizing T and { hi } ni=1 . Let ℓ denote a non-negative loss function defined over RK × RK . Given a hypothesis space H ⊂ { h : X → RK } , we denote Lis ( h ) = E ( x , y ) ∼P iXY ℓ ( h ( x ) , Φ ( y ) ) and Lt ( h ) = E ( x , y ) ∼P tXY ℓ ( h ( x ) , Φ ( y ) ) as the risks with respect to the i-th source domain and a given target domain , where Φ : Y → RK maps any label y to a corresponding one-hot vector . The source predictor hi is a vector-valued function , i.e. , hi ( x ) = [ hi1 ( x ) , ... , h i K ( x ) ] ⊤ , and consists of two basic components : feature extractor f i : X → Rd and classifier ci : Rd → RK , where d is the dimension of extracted features . Therefore , hi can be rewritten as ci ◦ f i . After using softmax as the activation function in the output layer , we have hik ≥ 0 and ∑K k=1 h i k = 1 . To ensure that each predictor hi is a relatively accurate predictor for the domain P iXY , we assume that the predictor h i is ϵ-accurate under ℓ1 loss , i.e. , E ( x , y ) ∼P iXY ∥h i ( x ) −Φ ( y ) ∥ℓ1 < ϵ , for any i ∈ [ n ] . 4 Theoretical Analysis . Without any relations between source domains and target domain , MSFDA can not be effectively addressed from the theoretical view . To bridge source and target domains , Mansour et al . [ 40 ] and Miraj Ahmed et al . [ 1 ] assume that PXY = ∑n i=1 λiP i XY and P t Y |X = P i Y |X = P j Y |X , for any i , j ∈ [ n ] , where λi ≥ 0 and ∑n i=1 λi = 1 . This assumption is very strong and may be unrealistic in many real-world applications . Motivated by the meta learning [ 43 ] and domain generalization [ 3 ] , in this paper , we propose some novel and mild assumptions to address MSFDA problem . Assumption 1 ( Meta Assumption . ) P tXY , P 1XY , ... , PnXY are drawn ( i.i.d . ) from a meta distribution P , which is defined over a joint distribution space PXY . Assumption 2 ( Regular Domain . ) Let the joint distribution space PXY is endowed with total variation distance dTV ( · , · ) . The target domain P tXY is a regular domain , i.e. , for any σ > 0 , P ( N σP tXY ) > 0 , where N σ P tXY = { P : dTV ( P , P tXY ) < σ } . It is easy to check , if the meta distribution P is discrete or continuous with continuous density function , then the target domain P tXY drawn byP is a regular domain with probability 1 . In addition , to weaken the assumption P tY |X = P i Y |X = P j Y |X , for any i , j ∈ [ n ] , our key strategy is to consider the anchor point assumption that is studied in label-noise learning [ 36 ] . We say a given point x is a τ -anchor point , if there exists a predictor hi with the largest score hik ( x ) , such that hik ( x ) − hic ( x ) ≥ τ , for any c ∈ [ K ] and c ̸= k. hi is the τ -anchor predictor for x . Assumption 3 ( τ -Anchor Point Assumption . ) Given a τ -anchor point x , suppose that all τ - anchor predictors for x are hi1 , ... , hil , then the true label of the τ -anchor point x is k , if hjk ( x ) is the largest score among scores hi1c , ... , h il c , for any c ∈ [ K ] . When the source predictors are accurate enough , Assumption 3 implies that the source and target conditional distributions are similar in the high confidence region . Hence , it is much weaker than the traditional assumption P tY |X = P i Y |X = P j Y |X , for i , j ∈ [ n ] . τ is regarded as a threshold to distinguish which data has highly confident prediction . In general , τ is close to 1 . Given any hi , it is easy to check that there exists a matrix function Ai ( x ) = [ aikl ( x ) ] such that hik ( x ) = ∑K l=1 a i kl ( x ) P t Y |X ( l|x ) with ∑K k=1 a i kl = 1 . We say the diagonal elements { aill } Kl=1 as the transfer factor for hi and P tY |X . Generally , A i ( x ) = [ aikl ( x ) ] is not unique , thus the transfer factor may be not unique . The following theorem indicates that Assumption 3 holds if we give proper assumptions for transfer factor . Theorem 1 Suppose that the Bayesian label is true label [ 7 ] . If there exist transfer factors and a constant B < K such that maxi∈ [ n ] , l∈ [ K ] aill ≥ B , then Assumption 3 holds with τ > 1−B/K . Theorem 1 provides a theoretical support for Assumption 3 and indicates that when the transfer factors are positive , Assumption 3 always holds with a proper τ . To further study the highly confident pseudo labeling strategy , the following theorem provides a lower bound to estimate the number of highly confident pseudo labels , i.e. , the number of τ -anchor points . Theorem 2 Assume Assumptions 1 and 2 hold and the conditional distribution P tY |X can be presented as a labeling function , i.e. , P tY |X ( y|x ) = 0 or 1 . Given η > 0 , if m ≥ n and ( 1−η ) ( 1−τ ) > ϵ + 2σ + 2 √ log ( 2m/δ ) /2m , then with probability at least 1 − δ − ( 1 −P ( N σPXtYt ) ) n > 0 , at least ηm target data are τ -anchor points , where ϵ is the upper bound of the accuracies of source predictors , and σ is introduced in Assumption 2 . Theorem 2 indicates that multi-source predictors improve the probability to obtain more τ -anchor points . To further understand the effect of multi-source domains , we build a novel learning bound for the MSFDA task . Let Aτ be the set consisting of all τ -anchor points . Denote the empirical risk L̂τs ( h ) by 1|Aτ∩T | ∑ x∈Aτ∩T ℓ ( h ( x ) , Φ ( y ) ) , where y is the label of the anchor point x. Theorem 3 Given Assumption 3 and some assumptions used in Theorem 2 , and suppose that the loss ℓ has upper bound M > 0 and hypothesis space H has finite Natarajan dimension , for η > 0 , if m ≥ n and ( 1 − η ) ( 1 − τ ) > ϵ + 2σ + 2 √ log ( 2m/δ ) /2m , then for any h ∈ H and b ∈ ( 0 , 1 ) , there exists a constant C ( b , K ) such that with the probability at least 1− 2δ− 2 ( 1−P ( N σP tXY ) ) n : ∣∣Lt ( h ) − L̂τs ( h ) ∣∣ ≤MC ( b , K ) √ log ( 2/δ ) η1−bm1−b +M 2σ + ϵ 1− τ − 2σ − ϵ , ( 1 ) where ϵ is the upper bound of accuracies of source predictors , and σ is introduced in Assumption 2 . Theorem 3 shows that multiple source domains improve the probability to ensure a tighter generalization bound , i.e. , Eq . ( 1 ) holds . Note that the Natarajan dimension used in Theorem 3 is a bit outdated . However , the Natarajan dimension can be replaced and Theorem 3 can be updated without any technical barriers , if there exists better generalization theory for supervised learning . Summary of Theoretical Analysis : The reasons to develop MSFDA are to study the solvability of MSFDA and understand how multi-source predictors benefit the target domain ’ s classification . By Theorems 2 and 3 , we realize that multi-source predictors improve the probability to obtain more highly confident pseudo labels , resulting in a tighter generalization bound . The generalization bound in Theorem 3 gives a positive answer to the solvability of MSFDA . Additionally , Theorems 1 and 2 also imply two interesting and important results : Theorem 1 provides the first theoretical support to the confident pseudo labeling strategy , and Theorem 2 provides the first lower bound of the number of highly confident pseudo labels . As we know , the theoretical results in Theorems 1 and 2 are novel .

Summary: In this paper, an algorithm for multi-source domain adaptation is proposed. The algorithm is developed in a source-free regime to address the concern for privacy. A loss function is designed to integrate the source-specific end-to-end classifiers using solely the target domain samples, which preserves privacy. The loss function consists of four terms to benefit from enforcing source-specific transferability, anchor-induced pseudo-labels, and class relationship-aware consistency. A PAC-learning style theoretical analysis is provided to demonstrate that using multiple sources may improve the generalizability on the target domain. Finally, experiments on five standard UDA datasets are provided to demonstrate that the method is effective.