Confident Anchor-Induced Multi-Source Free Domain Adaptation

1 Introduction . Unsupervised Domain Adaptation ( UDA ) [ 22,59,63 ] captures transferable knowledge from labeled data in a source domain to classify unlabeled data in a target domain . UDA has achieved remarkable successes in many applications , e.g. , object detection [ 23 ] , medical diagnose [ 9,12 ] , sentiment analysis [ 34 ] , etc . Generally , most of existing methods employ adversarial learning [ 17 ] to encourage the learned source and target features to be indistinguishable from each other [ 10 , 15 ] , or minimize the distribution discrepancy across domains by matching the statistical moments of distributions [ 41 ] . ∗Equal contributions †Corresponding author 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . However , the above-mentioned methods have a strong assumption that the source data are merely drawn from a single domain . Unfortunately , the source data are often collected under different deployed environments ( i.e. , multiple source domains with large distribution discrepancies ) in realworld applications , which makes them difficult to explore complementarily transferable knowledge from the multi-source domains for target prediction . To achieve this , Multi-Source Domain Adaptation ( MSDA ) [ 32 , 34 , 61 ] is proposed to match the features across domains and then quantify the contributions of source domains [ 2,41,60 ] . Additionally , [ 29,58 ] aim to weight the source contributions by normalizing the distance similarities between source and target domains . Unfortunately , recent MSDA methods [ 29 , 34 , 41 , 44 ] require massive labeled source data when adapting source domains to the target domain . This could make them inefficient and unpractical in real-world applications with sensitive information ( e.g. , medical diagnosis [ 12 ] and recommendation system [ 24 ] ) , due to privacy preservation issues , storage and security concerns [ 50 , 51 ] . To this end , a new challenging and practical problem named Multi-Source-Free Domain Adaptation ( MSFDA ) is researched , which explores transferable knowledge from multiple source domains to target domain with only pre-trained source models and without access to any source data . The trivial solutions for tackling MSFDA via using existing single-source-free domain adaptation methods [ 25 , 30 , 33 , 57 ] are to adapt each source model individually and simply take an average prediction of source models . However , they can not explore the contributions of the complementary information transferred from different source domains , due to the lack of source data . Therefore , tackling the MSFDA problem is a challenging but rarely-researched task . To address the MSFDA problem , we develop a novel Confident-Anchor-induced multi-source-free Domain Adaptation ( CAiDA ) model , which is a pioneer exploration to capture transferable information from multiple source models to promote target prediction without access to source data . Specifically , a source-specific transferable perception module is designed to calibrate the contributions of the transferability from multiple source domains . We develop a confident-anchor-induced pseudo label generator to mine pseudo labels for the unlabeled target data , by incorporating with the quantified source transferability contributions . We construct a confident anchor group to assign each target sample with a semantic-nearest confident anchor , and perform feature augmentation between them to generate confident target pseudo label . A class-relationship-aware consistency loss is proposed to ensure the semantic consistency of underlying inter-class relationships across domains via the alignment of soft confusion matrices . Furthermore , based on some mild assumptions , theoretical analysis guarantees that multiple source models could help generate more reliable pseudo labels . Our theoretical analysis also provides a novel learning bound for MSFDA , which reveals that multiple source models help achieve a tighter generalization error bound for the target domain . We verify the effectiveness of our proposed model via comparison experiments on benchmark datasets . The main contributions of our work are summarized as follows : • We develop a novel Confident-Anchor-induced multi-source-free Domain Adaptation ( CAiDA ) model to explore transferable knowledge from multiple source domains to assist target prediction with pre-trained source models and without access to source data . To our best knowledge , this paper is a pioneer exploration of multi-source-free domain adaptation in the field of transfer learning . •We propose a novel MSFDA theory , which shows that multiple source models could improve the possibility of obtaining more reliable pseudo labels under some mild assumptions . Our theoretical analysis also provides a novel generalization bound for MSFDA to show the effect of multiple source models . This novel bound implies a positive answer to the solvability of MSFDA problem . • A source-specific transferable perception module and a class-relationship-aware consistency loss are designed to quantify the contributions of the transferability of multiple source domains and ensure the semantic consistency of underlying inter-class relationships across domains , respectively . • Based on the confident pseudo labeling strategy in theoretical analysis , a confident-anchor-induced pseudo label generator is proposed to generate pseudo labels for the target domain by establishing a confident anchor group and assigning each target sample with a semantic-nearest confident anchor . 2 Related Work . Unsupervised Domain Adaptation aims to borrow transferable knowledge from source domain to promote the prediction of the unlabeled target domain . After Hoffman et al . [ 22 ] introduce ad- versarial learning [ 17 ] into domain adaptation , enormous adversarial-based methods [ 8 , 11 , 15 , 59 ] are proposed to perform feature-level or pixel-level distribution alignment . Besides , some moment matching-based methods [ 14 , 35 , 41 ] focus on matching the distribution statistical moments at different orders to minimize the distribution discrepancy across domains . Furthermore , some researches design the adversarial dropout [ 28 ] , batch normalization [ 52 ] and auxiliary reconstruction tasks [ 5,16 ] to narrow the domain discrepancy . Unfortunately , the methods mentioned above assume massive labeled source data are available . This is unpractical due to privacy and security concerns . Source-Free Domain Adaptation ( SFDA ) [ 33 ] is studied to tackle the above challenge . A common strategy in SFDA methods is to mine the confident pseudo labels for target domain . To this end , [ 33 ] uses a self-supervised pseudo labeling strategy , and [ 25 ] designs a confidence-based sample filtering method . [ 42 ] alleviates the negative transfer brought by noisy pseudo labels through confidence reweighting and regularization . In addition to the pseudo labeling strategy , the model adaptation strategy has also been studied . For example , [ 30 , 57 ] employ an adversarial learning strategy to perform model adaptation with only pre-trained source models . However , they can not be applied to tackle the MSFDA problem , due to the distribution discrepancies across different source domains . Multi-Source Domain Adaptation is extended from vanilla domain adaptation [ 12 , 15 , 41 ] by exploring transferable knowledge from multiple sources . To capture the relationship between different source domains and a given target domain , Guo et al . [ 19 ] design a mixture-of-experts model for unsupervised domain adaptation from multiple sources . [ 56 ] focuses on determining which source domain is the best for target prediction via dynamic curriculum learning . Some discrepancy-based methods aim to narrow the distribution discrepancy across domains by minimizing different measures such as the Rényi-divergence [ 21 ] and maximum mean discrepancy [ 19 ] . Moreover , some adversarial-based methods focus on optimizing the H-divergence [ 41 , 60 ] , generative adversarial loss [ 55,61 ] and Wasserstein distance [ 32 ] to make features from multiple sources indistinguishable for a shared discriminator . [ 34 , 44 ] perform knowledge adaptation at the pixel-level by replaying multiple source domains . Due to lack of source data , above strategies in MSDA may be invalid and unsuitable to address the challenging MSFDA problem . To this end , Ahmed et al . [ 1 ] utilize nearest distance measure to mine target pseudo labels , and weight the predictions from multiple source models for the MSFDA task . It may result in that the generation process has high probability to obtain noisy labels [ 4 , 31 , 54 ] when the strategy is not matched with the target data , while our model could generate confident pseudo labels from two different perspectives , i.e. , geometry and probability . 3 Problem Setting . Let X and Y = [ K ] : = { 1 , ... , K } denote the feature space and label space . A domain is a joint distribution PXY on X × Y . There are n source domains { P iXY } ni=1 . For any source domain P iXY , a corresponding neural-network-based predictor ( model ) hi : X → RK is given . Given a target domain P tXY with unlabeled target data T = { xj } mj=1 ∼ P tX , i.i.d. , the aim of multi-source-free domain adaptation ( MSFDA ) is to classify the unlabeled target data by utilizing T and { hi } ni=1 . Let ℓ denote a non-negative loss function defined over RK × RK . Given a hypothesis space H ⊂ { h : X → RK } , we denote Lis ( h ) = E ( x , y ) ∼P iXY ℓ ( h ( x ) , Φ ( y ) ) and Lt ( h ) = E ( x , y ) ∼P tXY ℓ ( h ( x ) , Φ ( y ) ) as the risks with respect to the i-th source domain and a given target domain , where Φ : Y → RK maps any label y to a corresponding one-hot vector . The source predictor hi is a vector-valued function , i.e. , hi ( x ) = [ hi1 ( x ) , ... , h i K ( x ) ] ⊤ , and consists of two basic components : feature extractor f i : X → Rd and classifier ci : Rd → RK , where d is the dimension of extracted features . Therefore , hi can be rewritten as ci ◦ f i . After using softmax as the activation function in the output layer , we have hik ≥ 0 and ∑K k=1 h i k = 1 . To ensure that each predictor hi is a relatively accurate predictor for the domain P iXY , we assume that the predictor h i is ϵ-accurate under ℓ1 loss , i.e. , E ( x , y ) ∼P iXY ∥h i ( x ) −Φ ( y ) ∥ℓ1 < ϵ , for any i ∈ [ n ] . 4 Theoretical Analysis . Without any relations between source domains and target domain , MSFDA can not be effectively addressed from the theoretical view . To bridge source and target domains , Mansour et al . [ 40 ] and Miraj Ahmed et al . [ 1 ] assume that PXY = ∑n i=1 λiP i XY and P t Y |X = P i Y |X = P j Y |X , for any i , j ∈ [ n ] , where λi ≥ 0 and ∑n i=1 λi = 1 . This assumption is very strong and may be unrealistic in many real-world applications . Motivated by the meta learning [ 43 ] and domain generalization [ 3 ] , in this paper , we propose some novel and mild assumptions to address MSFDA problem . Assumption 1 ( Meta Assumption . ) P tXY , P 1XY , ... , PnXY are drawn ( i.i.d . ) from a meta distribution P , which is defined over a joint distribution space PXY . Assumption 2 ( Regular Domain . ) Let the joint distribution space PXY is endowed with total variation distance dTV ( · , · ) . The target domain P tXY is a regular domain , i.e. , for any σ > 0 , P ( N σP tXY ) > 0 , where N σ P tXY = { P : dTV ( P , P tXY ) < σ } . It is easy to check , if the meta distribution P is discrete or continuous with continuous density function , then the target domain P tXY drawn byP is a regular domain with probability 1 . In addition , to weaken the assumption P tY |X = P i Y |X = P j Y |X , for any i , j ∈ [ n ] , our key strategy is to consider the anchor point assumption that is studied in label-noise learning [ 36 ] . We say a given point x is a τ -anchor point , if there exists a predictor hi with the largest score hik ( x ) , such that hik ( x ) − hic ( x ) ≥ τ , for any c ∈ [ K ] and c ̸= k. hi is the τ -anchor predictor for x . Assumption 3 ( τ -Anchor Point Assumption . ) Given a τ -anchor point x , suppose that all τ - anchor predictors for x are hi1 , ... , hil , then the true label of the τ -anchor point x is k , if hjk ( x ) is the largest score among scores hi1c , ... , h il c , for any c ∈ [ K ] . When the source predictors are accurate enough , Assumption 3 implies that the source and target conditional distributions are similar in the high confidence region . Hence , it is much weaker than the traditional assumption P tY |X = P i Y |X = P j Y |X , for i , j ∈ [ n ] . τ is regarded as a threshold to distinguish which data has highly confident prediction . In general , τ is close to 1 . Given any hi , it is easy to check that there exists a matrix function Ai ( x ) = [ aikl ( x ) ] such that hik ( x ) = ∑K l=1 a i kl ( x ) P t Y |X ( l|x ) with ∑K k=1 a i kl = 1 . We say the diagonal elements { aill } Kl=1 as the transfer factor for hi and P tY |X . Generally , A i ( x ) = [ aikl ( x ) ] is not unique , thus the transfer factor may be not unique . The following theorem indicates that Assumption 3 holds if we give proper assumptions for transfer factor . Theorem 1 Suppose that the Bayesian label is true label [ 7 ] . If there exist transfer factors and a constant B < K such that maxi∈ [ n ] , l∈ [ K ] aill ≥ B , then Assumption 3 holds with τ > 1−B/K . Theorem 1 provides a theoretical support for Assumption 3 and indicates that when the transfer factors are positive , Assumption 3 always holds with a proper τ . To further study the highly confident pseudo labeling strategy , the following theorem provides a lower bound to estimate the number of highly confident pseudo labels , i.e. , the number of τ -anchor points . Theorem 2 Assume Assumptions 1 and 2 hold and the conditional distribution P tY |X can be presented as a labeling function , i.e. , P tY |X ( y|x ) = 0 or 1 . Given η > 0 , if m ≥ n and ( 1−η ) ( 1−τ ) > ϵ + 2σ + 2 √ log ( 2m/δ ) /2m , then with probability at least 1 − δ − ( 1 −P ( N σPXtYt ) ) n > 0 , at least ηm target data are τ -anchor points , where ϵ is the upper bound of the accuracies of source predictors , and σ is introduced in Assumption 2 . Theorem 2 indicates that multi-source predictors improve the probability to obtain more τ -anchor points . To further understand the effect of multi-source domains , we build a novel learning bound for the MSFDA task . Let Aτ be the set consisting of all τ -anchor points . Denote the empirical risk L̂τs ( h ) by 1|Aτ∩T | ∑ x∈Aτ∩T ℓ ( h ( x ) , Φ ( y ) ) , where y is the label of the anchor point x. Theorem 3 Given Assumption 3 and some assumptions used in Theorem 2 , and suppose that the loss ℓ has upper bound M > 0 and hypothesis space H has finite Natarajan dimension , for η > 0 , if m ≥ n and ( 1 − η ) ( 1 − τ ) > ϵ + 2σ + 2 √ log ( 2m/δ ) /2m , then for any h ∈ H and b ∈ ( 0 , 1 ) , there exists a constant C ( b , K ) such that with the probability at least 1− 2δ− 2 ( 1−P ( N σP tXY ) ) n : ∣∣Lt ( h ) − L̂τs ( h ) ∣∣ ≤MC ( b , K ) √ log ( 2/δ ) η1−bm1−b +M 2σ + ϵ 1− τ − 2σ − ϵ , ( 1 ) where ϵ is the upper bound of accuracies of source predictors , and σ is introduced in Assumption 2 . Theorem 3 shows that multiple source domains improve the probability to ensure a tighter generalization bound , i.e. , Eq . ( 1 ) holds . Note that the Natarajan dimension used in Theorem 3 is a bit outdated . However , the Natarajan dimension can be replaced and Theorem 3 can be updated without any technical barriers , if there exists better generalization theory for supervised learning . Summary of Theoretical Analysis : The reasons to develop MSFDA are to study the solvability of MSFDA and understand how multi-source predictors benefit the target domain ’ s classification . By Theorems 2 and 3 , we realize that multi-source predictors improve the probability to obtain more highly confident pseudo labels , resulting in a tighter generalization bound . The generalization bound in Theorem 3 gives a positive answer to the solvability of MSFDA . Additionally , Theorems 1 and 2 also imply two interesting and important results : Theorem 1 provides the first theoretical support to the confident pseudo labeling strategy , and Theorem 2 provides the first lower bound of the number of highly confident pseudo labels . As we know , the theoretical results in Theorems 1 and 2 are novel .

Summary: This paper addresses the multi-source free domain adaptation problem where multiple pretrained source models and unlabeled target data are given. Theoretical analysis aims at proving that multiple source domains improve the probability to ensure a generalization bound. The method is developed based on mutual information maximization, pseudo labeling and consistency losses. Experiments show that the proposed method achieves slightly better results.