Source-free Domain Adaptation via Distributional Alignment by Matching Batch Normalization Statistics

1 INTRODUCTION . In typical statistical machine learning algorithms , test data are assumed to stem from the same distribution as training data ( Hastie et al. , 2009 ) . However , this assumption is often violated in practical situations , and the trained model results in unexpectedly poor performance ( QuioneroCandela et al. , 2009 ) . This situation is called domain shift , and many researchers have intensely worked on domain adaptation ( Csurka , 2017 ; Wilson & Cook , 2020 ) to overcome it . A common approach for domain adaptation is to jointly minimize a distributional discrepancy between domains in a feature space as well as the prediction error of the model ( Wilson & Cook , 2020 ) , as shown in Fig . 1 ( a ) . Deep neural networks ( DNNs ) are particularly popular for this joint training , and recent methods using DNNs have demonstrated excellent performance under domain shift ( Wilson & Cook , 2020 ) . Many domain adaptation algorithms assume that they can access labeled source data as well as target data during adaptation . This assumption is essentially required to evaluate the distributional discrepancy between domains as well as the accuracy of the model ’ s prediction . However , it can be unreasonable in some cases , for example , due to data privacy issues or too large-scale source datasets to be handled at the environment where the adaptation is conducted . To tackle this problem , a few recent studies ( Kundu et al. , 2020 ; Li et al. , 2020 ; Liang et al. , 2020 ) have proposed source-free domain adaptation methods in which they do not need to access the source data . In source-free domain adaptation , the model trained with source data is given instead of source data themselves , and it is fine-tuned through adaptation with unlabeled target data so that the fine-tuned model works well in the target domain . Since it seems quite hard to evaluate the distributional discrepancy between unobservable source data and given target data , previous studies mainly focused on how to minimize the prediction error of the model with unlabeled target data , for example , by using pseudo-labeling ( Liang et al. , 2020 ) or a conditional generative model ( Li et al. , 2020 ) . However , due to lack of the distributional alignment , those methods heavily depend on noisy target labels obtained through the adaptation , which can result in unstable performance . In this paper , we propose a novel method for source-free domain adaptation . Figure 1 ( b ) shows our setup in comparison with that of typical domain adaptation methods shown in Fig . 1 ( a ) . In our method , we explicitly minimize the distributional discrepancy between domains by utilizing batch normalization ( BN ) statistics stored in the pretrained model . Since we fix the pretrained classifier during adaptation , the BN statistics stored in the classifier can be regarded as representing the distribution of source features extracted by the pretrained encoder . Based on this idea , to minimize the discrepancy , we train the target-specific encoder so that the BN statistics of the target features extracted by the encoder match with those stored in the classifier . We also adopt information maximization as in Liang et al . ( 2020 ) to further boost the classification performance of the classifier in the target domain . Our method is apparently simple but effective ; indeed , we will validate its advantage through extensive experiments on several benchmark datasets . 2 RELATED WORK . In this section , we introduce existing works on domain adaptation that are related to ours and also present a formulation of batch normalization . 2.1 DOMAIN ADAPTATION . Given source and target data , the goal of domain adaptation is to obtain a good prediction model that performs well in the target domain ( Csurka , 2017 ; Wilson & Cook , 2020 ) . Importantly , the data distributions are significantly different between the domains , which means that we can not simply train the model with source data to maximize the performance of the model for target data . Therefore , in addition to minimizing the prediction error using labeled source data , many domain adaptation algorithms try to align the data distributions between domains by adversarial training ( Ganin et al. , 2016 ; Tzeng et al. , 2017 ; Deng et al. , 2019 ; Xu et al. , 2019 ) or explicitly minimizing a distributionaldiscrepancy measure ( Long et al. , 2015 ; Bousmalis et al. , 2016 ; Long et al. , 2017 ) . This approach has empirically shown excellent performance and is also closely connected to theoretical analysis ( Ben-David et al. , 2010 ) . However , since this distribution alignment requires access to source data , these methods can not be directly applied to the source-free domain adaptation setting . In source-free domain adaptation , we can only access target data but not source data , and the model pretrained with the source data is given instead of the source data . This challenging problem has been tackled in recent studies . Li et al . ( 2020 ) proposed joint training of the target model and the conditional GAN ( Generative Adversarial Network ) ( Mirza & Osindero , 2014 ) that is to generate annotated target data . Liang et al . ( 2020 ) explicitly divided the pretrained model into two modules , called a feature encoder and a classifier , and trained the target-specific feature encoder while fixing the classifier . To make the classifier work well with the target features , this training jointly conducts both information maximization and self-supervised pseudo-labeling with the fixed classifier . Kundu et al . ( 2020 ) adopted a similar architecture but it has three modules : a backbone model , a feature extractor , and a classifier . In the adaptation phase , only the feature extractor is tuned for the target domain by minimizing the entropy of the classifier ’ s output . Since the methods shown above do not try to align data distributions between domains , they can not essentially avoid confirmation bias of the model and also can not benefit from well-exploited theories in the studies on typical domain adaptation problems ( Ben-David et al. , 2010 ) . 2.2 BATCH NORMALIZATION . Batch normalization ( BN ) ( Ioffe & Szegedy , 2015 ) has been widely used in modern architectures of deep neural networks to make their training faster as well as being stable . It normalizes each input feature within a mini-batch in a channel-wise manner so that the output has zero-mean and unit-variance . Let B and { zi } Bi=1 denote the mini-batch size and the input features to the batch normalization , respectively . Here , we assume that the input features consist of C channels as zi = [ z ( 1 ) i , ... , z ( C ) i ] and each channel contains nc features . BN first computes the means { µc } Cc=1 and variances { σ2c } Cc=1 of the features for each channel within the mini-batch : µc = 1 ncB B∑ i nc∑ j z ( c ) i [ j ] , σ 2 c = 1 ncB B∑ i nc∑ j ( z ( c ) i [ j ] − µc ) 2 , ( 1 ) where z ( c ) i [ j ] is the j-th feature in z ( c ) i . Then , it normalizes the input features by using the computed BN statistics : z̃ ( c ) i = z ( c ) i − µc√ σ2c + , ( 2 ) where is a small positive constant for numerical stability . In the inference phase , BN can not always compute those statistics , because the input data do not necessarily compose a mini-batch . Instead , BN stores the exponentially weighted averages of the BN statistics in the training phase and uses them in the inference phase to compute z̃ in Eq . ( 2 ) ( Ioffe & Szegedy , 2015 ) . Since BN renormalizes features to have zero-mean and unit-variance , several methods ( Li et al. , 2018 ; Chang et al. , 2019 ; Wang et al. , 2019 ) adopted domain-specific BN to explicitly align both the distribution of source features and that of target features into a common distribution . Since the domain-specific BN methods are jointly trained during adaptation , we can not use these methods in the source-free setting . 3 PROPOSED METHOD . Figure 2 shows an overview of our method . We assume that the model pretrained with source data is given , and it conducts BN at least once somewhere inside the model . Before conducting domain adaptation , we divide the model in two sub-models : a feature encoder and a classifier , so that BN comes at the very beginning of the classifier . Then , for domain adaptation , we fine-tune the encoder with unlabeled target data with the classifier fixed . After adaptation , we use the fine-tuned encoder and the fixed classifier to predict the class of test data in the target domain . To make the fixed classifier work well in the target domain after domain adaptation , we aim to obtain a fine-tuned encoder that satisfies the following two properties : • The distribution of target features extracted by the fine-tuned encoder is well aligned to that of source features extracted by the pretrained encoder . • The features extracted by the fine-tuned encoder are sufficiently discriminative for the fixed classifier to accurately predict the class of input target data . To this end , we jointly minimize both the BN-statistics matching loss and information maximization loss to fine-tune the encoder . In the former loss , we approximate the distribution of unobservable source features by using the BN statistics stored in the first BN layer of the classifier , and the loss explicitly evaluates the discrepancy between source and target feature distributions based on those statistics . Therefore , minimizing this loss leads to satisfying the first property shown above . On the other hand , the latter loss is to make the predictions by the fixed classifier certain for every target sample as well as diverse within all target data , and minimizing this loss leads to fulfilling the second property . Below , we describe the details of these losses . 3.1 DISTRIBUTION ALIGNMENT BY MATCHING BATCH NORMALIZATION STATISTICS . Since the whole model is pretrained with source data and we fix the classifier while finetuning the encoder , the BN statistics stored in the first BN in the classifier can be seen as the statistics of the source features extracted by the pretrained encoder . We approximate the source-feature distribution by using these statistics . Specifically , we simply use a Gaussian distribution for each channel denoted by N ( µ̂c , σ̂2c ) where µ̂c and σ̂2c are the mean and variance of the Gaussian distribution which are the stored BN statistics corresponding to the c-th channel . To match the feature distributions between domains , we define the BN-statistics matching loss , which evaluates the averaged Kullback-Leibler ( KL ) divergence from the target-feature distribution to the approximated source-feature distribution : LBNM ( { xi } Bi=1 , θ ) = 1 C C∑ c=1 KL ( N ( µ̂c , σ̂2c ) ||N ( µc , σ2c ) ) = 1 2C C∑ c=1 ( log σ2c σ̂2c + σ̂2c + ( µ̂c − µc ) 2 σ2c − 1 ) , ( 3 ) where { xi } Bi=1 is a mini-batch from the target data , θ is a set of trainable parameters of the encoder , and µc and σc are the BN statistics of the c-th channel computed from the target mini-batch . Note that , since µc and σc are calculated from the features extracted by the encoder , they depend on θ . Here , we also approximate the target-feature distribution with another Gaussian distribution so that the KL divergence can be efficiently computed in a parametric manner . By minimizing this loss , we can explicitly reduce the discrepancy between the distribution of unobservable source features and that of target features . In Eq . ( 3 ) , we chose the KL divergence to measure the distributional discrepancy between domains . There are two reasons for this choice . First , the KL divergence between two Gaussian distributions is easy to compute with the BN statistics as shown in Eq . ( 3 ) . Moreover , since these statistics are naturally computed in the BN layer , calculating this divergence only requires tiny calculation costs . Secondly , it would be a theoretically-inspired design from the perspective of risk minimization in the target domain . When we consider a binary classification task , the expected risk of any hypothesis h in the target domain can be upper-bounded under some mild assumptions as the following inequality ( Ben-David et al. , 2010 ) : rT ( h ) ≤ rS ( h ) + d1 ( pS , pT ) + β , ( 4 ) where rS ( h ) and rT ( h ) denote the expected risk of h under the source-data distribution pS and target-data distribution pT , respectively , d1 ( p , q ) represents the total variation distance between p and q , and β is a constant value that is expected to be sufficiently small . This inequality roughly gives a theoretical justification to recent domain adaptation algorithms , that is , joint minimization of both the distributional discrepancy between domains ( corresponding to the second term of the bound in Eq . ( 4 ) ) and the prediction error of the model ( corresponding to the first term of the bound in Eq . ( 4 ) ) . Here , the total variation distance can be related to the KL divergence by Pinsker ’ s inequality ( Csiszar & Körner , 2011 ) : d1 ( p , q ) ≤ √ 1 2 KL ( p||q ) . ( 5 ) Consequently , we can guarantee that minimizing the KL divergence between domains minimizes the bound of the target risk .

Summary: In the work, the authors focus on tackling the problem of source free domain adaptation. The proposed method mainly has two parts, in which the second is nearly the same as the SHOT-IM as in Liang et al., 2020 [1], while the first part aims at coping with this problem from a new perspective to align the distribution of target features extracted by the fine-tuned encoder to that of source features extracted by the pre-trained encoder. To achieve this, they utilize batch normalization statistics stored in the pre-trained model to approximate the distribution of unobserved source data.