Source-free Domain Adaptation via Distributional Alignment by Matching Batch Normalization Statistics
Authors: Masato Ishii
Masashi Sugiyama
Venue:  (2021)
Keywords: domain adaptation
transfer learning

Abstract: In this paper, we propose a novel domain adaptation method for the source-free setting. In this setting, we cannot access source data during adaptation, while unlabeled target data and a model pretrained with source data are given. Due to lack of source data, we cannot directly match the data distributions between domains unlike typical domain adaptation algorithms. To cope with this problem, we propose utilizing batch normalization statistics stored in the pretrained model to approximate the distribution of unobserved source data. Specifically, we fix the classifier part of the model during adaptation and only fine-tune the remaining feature encoder part so that batch normalization statistics of the features extracted by the encoder match those stored in the fixed classifier. Additionally, we also maximize the mutual information between the features and the classifier's outputs to further boost the classification performance. Experimental results with several benchmark datasets show that our method achieves competitive performance with state-of-the-art domain adaptation methods even though it does not require access to source data. 

Under review as a conference paper at ICLR 2021
SOURCE-FREE DOMAIN ADAPTATION VIA DISTRIBUTIONAL ALIGNMENT BY MATCHING BATCH NORMALIZATION STATISTICS
Anonymous authors Paper under double-blind review
ABSTRACT
this paper, we propose a novel domain adaptation method for the source-free setting. In this setting, we cannot access source data during adaptation, while unlabeled target data and model pretrained with source data are given. Due to lack of source data, cannot directly match the data distributions between domains unlike typical domain adaptation algorithms. To cope with this problem. we propose utilizing batch normalization statistics stored in the pretrained model to approximate the distribution of unobserved source data. Specifically, we fix the classifier part of the model during adaptation and only fine-tune the remaining feature encoder part so that batch normalization statistics of the features extracted by the encoder match those stored in the fixed classifier. Additionally, maximize the mutual information between the features and the classifier's 's outputs further boost the classification performance. Experimental results with several benchmark datasets show that our method achieves competitive performance with ffate-of-the-ard domain adaptation methods even though does not require access to source data.
also
INTRODUCTION
In typical statistical machine learning algorithms, test data are assumed to stem from the same distribution as training data (Hastie et al. 2009). However, this assumption is often violated in practical situations, and the trained model results in unexpectedly poor performance (Quionero- Candela et al. 2009). This situation is called domain shift, and many researchers have intensely worked on domain adaptation (Csurka) 2017: Wilson Cook 2020) to overcome it. A common approach for domain adaptation is to jointly minimize distributional discrepancy between domains a feature space as well as the prediction error of the model (Wilson & Cook (020), as shown in Fig. [[a)] Deep neural networks (DNNs) particularly popular for this joint training, and recent methods using DNNs have demonstrated excellent performance under domain shift (Wilson) & Cook 2020).
Many domain adaptation algorithms assume that they can access labeled source data as well as target data during adaptation. This assumption essentially required evaluate the distributional discrepancy between domains as well as the accuracy of the model's 's prediction. However, it can be unreasonable in some cases, for example, due data privacy issues or too large-scale source datasets to be handled at the environment where the adaptation is conducted. To tackle this problem, a few recent studies (Kundu al.) 2020 Li al.] 2020: Liang et al., 2020) have proposed source-free domain adaptation methods in which they do not need to access the source data.
In source-free domain adaptation, the model trained with source data is given instead of source data themselves, and it is fine-tuned through adaptation with unlabeled target data so that the fine-tuned model works well in the target domain. Since seems quite hard to evaluate the distributional dis- crepancy between unobservable source data and given target data, previous studies mainly focused on how to minimize the prediction error the model with unlabeled target data, for example, by using pseudo-labeling (Liang et al. 2020) ever, due to lack of the distributional alignment, those methods heavily depend on noisy target labels obtained through the adaptation, which can result in unstable performance.
conditional generative model (Li et al. 2020). How-

---

Under review as a conference paper at ICLR 2021
Source data
Target data
Target data
Weight sharing or regularinatin
Weiggtt intiiaizatiinn
Feature Encoder
Feature Eneader
Festure Encodkr
Fenture Eacoder
Stored BN statistics
Classifier
Minimize distributional distrepancy
Pretrsined
Minimize BN-statitiis discrepancy
Minimiic
Maximie:
classification CITO
Mutual information
(a) General setup commonly adopted in recent typical domain adaptation methods. This visualization in- spired by (Wilson & Cook (020).
(b) Our setup for source- free domain adaptation.
Figure Comparison between typical domain adaptation methods and our method. A rectangle with solid lines represents trainable component, while that with dotted lines represent fixed component during adaptation.
In this paper, we propose a novel method for source-free domain adaptation. Figure T(b)
shows
setup in comparison with that of typical domain adaptation methods shown in Fig. ((a) In method, we explicitly minimize the distributional discrepancy between domains by utilizing batch normalization (BN) statistics stored in the pretrained model. Since we fix the pretrained classifier during adaptation, the BN statistics stored in the classifier can be regarded representing the dis- tribution of source features extracted by the pretrained encoder. Based on this idea, to minimize the discrepancy, we train the target-specific encoder so that the BN statistics of the target features extracted by the encoder match with those stored in the classifier. We also adopt information max- imization as in Liang et al. (2020) to further boost the classification performance of the classifier in the target domain. Our method is apparently simple but effective; indeed, we will validate its advantage through extensive experiments on several benchmark datasets.
2 RELATED WORK
In this section, we introduce existing works on domain adaptation that are related to ours and also present a formulation of batch normalization.
2.1 DOMAIN ADAPTATION
Given source and target data, the goal of domain adaptation is to obtain good prediction model that performs well in the target domain (Csurka 2017; Wilson & Cook) (020). Importantly, the data dis- tributions significantly different between the domains, which means that we cannot simply train the model with source data to maximize the performance of the model for target data. Therefore, in addition to minimizing the prediction error using labeled source data, many domain adaptation algorithms try to align the data distributions between domains by adversarial training (Ganin et al., 2016 Tzeng al. 2017 Deng al. 2019 Xu al. 2019 or explicitly minimizing a distributional- discrepancy measure (Long al.) 2015 Bousmalis et al. 2016 Long al.) 2017). This approach has empirically shown excellent performance and also closely connected to theoretical analysis (Ben-David al. 2010). However, since this distribution alignment requires access to source data,
these methods cannot directly applied to the source-free domain adaptation setting.
In source-free domain adaptation, can only access target data but not source data, and the model pretrained with the source data is given instead of the source data. This challenging problem has been tackled in recent studies. [i et al. (2020) proposed joint training of the target model and the conditional GAN (Generative Adversaria Network) (Mirza Osindero, 2014) that is to generate annotated target data. Liang al. 2020) explicitly divided the pretrained model into modules,
2

---

Under review as a conference paper at ICLR 2021
called feature encoder and classifier, and trained the target-specific feature encoder while fixing the classifier. To make the classifier work well with the target features, this training jointly conducts both information maximization and self-supervised pseudo-
with the fixed classifier. Kundu et al. (2020) adopted a similar architecture but has three modules: a backbone model, a feature extractor, and classifier. In the adaptation phase, only the feature extractor tuned for the target domain by minimizing the entropy of the classifier's output. Since the methods shown above do try to align data distributions between domains, they cannot essentially avoid confirmation bias of the model and also cannot benefit from well-exploited theories in the studies on typical domain adaptation problems [Ben-Daves al. 2010).
2.2 BATCH NORMALIZATION
Batch normalization (BN) (Toffe & Szegedy 2015) has been widely used in modern architectures of deep neural networks to make their training faster as well as being stable. It normalizes each input feature within a mini-batch in channel-wis manner so that the output has zero-mean and vari-nari Let B and { denote the mini-batch size and the input features to the batch normalization, respectively. Here, assume that the input features consist of C' channels as (e.") ..... [ and each channel contains ne features. BN first computes the means {Mela=1 and variances fTel== of the features for each channel within the mini-batch:
B
The
Tce
Mc BBB
[b. of BBB
-D
(c) [i] - -2e)²
(1)
j
where 2 []] is the j-th feature in zi BN statistics:
(c)
Then, it normalizes the input features by using the computed
He
(2)
00 +
where e is small positive constant for numerical stability. In the inference phase, BN cannot always compute those statistics, because the input data do not necessarily compose mini-batch. Instead, BN stores the exponentially weighted averages of the BN statistics in the training phase and uses them in the inference phase to compute
Eq. (Toffe & Szegedy (2015).
Since BN renormalizes features to have zero-men mean and unit-variance several methods (Li et al. 2018 Chang et al.) 2019) Wang al., (219) adopted domain-specifie BN to explicitly align both the distribution of source features and that of target features into common distribution. Since the domain-specific BN methods are jointly trained during adaptation, we cannot use these methods in the source-free free setting.
3 PROPOSED METHOD
Figure 2 shows an overview of our method. We assume that the model pretrained with source data given, and conducts BN at least once somewhere inside the model. Before conducting domain adaptation, we divide the model in two sub-models: feature encoder and classifier, so that BN comes at the very beginning of the classifier. Then, for domain adaptation, we fine-tune the encoder with unlabeled target data with the classifier fixed. After adaptation, use the fine-tuned encoder and the fixed classifier to predict the class of test data in the target domain.
To make the fixed classifier work well in the target domain after domain adaptation, we aim to obtain a fine-tuned encoder that satisfies the following two properties:
The distribution of target features extracted by the fine-tuned encoder is well aligned that of source features extracted by the pretrained encoder.
The features extracted by the fine-tuned encoder are sufficiently discriminative for the fixed classifier to accurately predict the class of input target data.
To this end, we jointly minimize both the BN-statistics matching loss and information maximization loss to fine-tune the encoder. In the former loss, we approximate the distribution of unobservable
3

---

Under review as a conference paper at ICLR 2021
Pretraining phase
Adaptation phase
Source data
Target data
Conv
Conv
Encoder
Initialize
Encoder
Calculate BN statisties
FC
FC
(c.
(dc.
Classifier
BN
Copy & fix
Classifier
BN
BN statistics
Softmax
Softmax
matching loss: LBNS
Classification loss
Information maximization loss: LIM
Figure 2: An overview of the proposed method.
source features by using the BN statistics stored in the first BN layer of the classifier, and the loss explicitly evaluates the discrepancy between source and target feature distributions based on those statistics. Therefore, minimizing this loss leads to satisfying the first property shown above. On the other hand, the latter loss is to make the predictions by the fixed classifier certain for every target sample as well as diverse within all target data, and minimizing this loss leads to fulfilling the second property. Below, we describe the details of these losses.
3.1 DISTRIBUTION ALIGNMENT BY MATCHING BATCH NORMALIZATION STATISTICS
Since the whole model is pretrained with source data and we fix the classifier while finetuning the encoder, the BN statistics stored in the first BN the classifier can be seen as the statistics of the source features extracted by the pretrained encoder. We approximate the source-feature distribution by using these statistics. Specifically, we simply use a Gaussian distribution for each channel de- noted by Nier a²) where ü. and of the mean and variance of the Gaussian distribution which are the stored BN statistics corresponding to the e-th channel.
To match the feature distributions between domains, which evaluates the averaged Kullback-Leiblee Leibler (KL) divergence from the target-feature distribution the approximated source-feature distribution:
define the BN-statistics matching loss,
LENM((=;)) 1
Σ KL (N WHe) o²))
a + (fcc - Ho)²
02
log
+
-
(3)
20
0.
o²
where {x: is mini-batch from the target data, 0 is a set of trainable parameters of the encoder, and He and a. are the BN statisties of the c-th channel computed from the target mini-batch. Note that, since Me and de calculated from the features extracted by the encoder, they depend 0. Here, we also approximate the target-feature distribution with another Gaussian distribution so that the KL divergence can be efficiently computed in parametric manner. By minimizing this loss, can explicitly reduce the discrepancy between the distribution of unobservable source features and that of target features.
In Eq. we chose the KL divergence to measure the distributional discrepancy between domains. There two reasons for this choice. First, the KI. divergence between Gaussian distributions
4

---

Under review as a conference paper at ICLR 2021
easy compute with the BN statistics as shown in Eq. (3) Moreover, since these statisties naturally computed in the BN layer, calculating this divergence only requires tiny calculation costs. Secondly, it would be theoretically- -inspired design from the perspective of risk minimization in the target domain. When we consider a binary classification task, the expected risk of any hypothesis in the target domain can be hoper-bounded under some mild assumptions the following inequality (Ben-David et al. 2010):
(T(h) S rg)(h) d1 (PS.PP)) 3,
(4)
where (h) and (h) denote the expected risk of under the source-data distribution ps and target-data distribution pr, respectively, d1 (p, q) represents the total variation distance between and 9, and 3 is a constant value that expected be sufficiently small. This inequality roughly gives theoretical justification to recent domain adaptation algorithms, that is, joint minimization of both the distributional discrepancy between domains (corresponding to the second term of the bound in Eq. (4)) and the prediction error of the model (corresponding to the first term of the bound in Eq. 44). Here, the total variation distance can be related to the KL divergence by Pinsker's 's inequality (Csiszar & Körner 2011):
di (p.g) S kL(()))))
(5)
Consequently, we can guarantee that minimizing the KL divergence between domains minimizes the bound of the target risk.
3.2 DOMAIN ADAPTATION WITHOUT ACCESS TO THE SOURCE DATA
Only aligning the marginal feature-distributions between domains does not guarantee that the fixed classifier works well in the target domain, because the features extracted by the encoder are not necessarily discriminative. If the features are sufficiently discriminative for the classifier, we can expect that the output of the classifier is almost always a one-hot vector but is diverse within the target data. Therefore, following the approach presented in [iang al. (2020), also adopt the information maximization loss to make the classifier work accurately.
B [IM(fx)) - - [B
B + B D H
(6)
where H(p(v)) = Ey p(y') is the entropy function, and fo(x)) denotes the output of the classifier. The first term in the right-hand side of Eq. (6) is the negative entropy of the averaged output of the classifier, and minimizing it leads to large diversity of the output within the mini- batch. The second term is the averaged entropy of the classifier's output, and minimizing it makes the outputs close to one-hot vectors. Therefore, the features extracted by the target encoder are induced to be discriminative by minimizing the information maximization loss.
Finally, our source-free free domain adaptation method is formulated as joint minimization of both the BN-statistics matching loss in Eq. (3) and the information maximization loss in Eq. (6):
min E (=,) ~D. [L
ALNNM((::)
where D. the target dataset from which the mini-batch is sampled, and a hyper-paramedt controls the balance between the two terms. Note that this optimization can be conducted without the source data, which means that we do not need to access to the source data during adaptation.
EXPERIMENTS
We conducted experiments with several datasets that are commonly used in existing works on do- main adaptation. Specifically, we used digit recognition datasets (MNIST (LeCun al.) 1998), USPS (LeCun et al. 1990). and SVHN (Netzer et al. 2011)) and an object recognition dataset 3Office-31 dataset (Saenko et al. 2010 In the experiment, source training data. Following the setup in Liang et al. (200), we used standard cross-entropy loss with label smoothing for this pretraining. Then, apply source-free domain adaptation
first pretrained the model with the
5

---

Under review as a conference paper at ICLR 2021
Table 1: Experimental results with Office-31 dataset. The bold number represents the highest test accuracy among the source-free free domain adaptation methods, and the underline represents the second
highest one,
Method
A-=D
A-W
Dー>
D
W-A
W-+
SHOT
94.0
90.1
74.7
98.4
74.3
99.9
Liang et al. 2020)
Model adaptation
92.7++0.4
93.7++0.2
75.3+0.5
98.5+001
77.8
99.8++0.2
[Li et al.
2020)
Our method
89.0++.22
91.7+11.0
7.2
98.9+0.1
76.6+0077
100.0+0.00
ADDA
77.
86,2
0..
96.210.3
68.9
98.4
Tzeng et al. 2017
Grad CAT
90.8+1.8
94.4
72.2+0.6
98.0++0.1
70.2+0.1
100.0+0.0
Deng al. 2019)
d-SNE
94.7:10.4
96.610.11
75.5+0.4
99.1
74.2+102
100.0+0.00
Xu et al.
2019)
method to fine-tune the pretrained model with the target training data. We used Adam optimizer for both pretraining and adaptation. The number of iterations in the optimization was set 30,000, and the batch size was set to 64. The hyper-paramette parameter À in Eq. is set to 10 in all experiments except for those in section 4.3 The performance of the domain adaptation is evaluated by test accuracy of the fine-tuned model on the target test data. We report the averaged accuracy as well as the standard deviation over five runs with random initialization of the model at the pretraining phase.
We compared the performance of our method with those of the state-of-the-an art methods for source- free domain adaptation (Li et al. 2020; Liang et al. 2020), which are most related to our work. We did not include the work by Kundu al. (2020) in this comparison, because it is designed for more difficult setting, called universal domain adaptation. For reference, we also show the performance of the recent methods for typical domain adaptation (Tzeng al. 2017 Deng et al., 2201 Xu al. 2019), though they require access to the source data during adaptation.
4.1 OBJECT RECOGNITION DATASET
The Office-31 dataset comprises three domains: Amazon (A), DSLR (D), and Webcam (W). We examined all possible combinations for the adaptation, which results in six scenarios. Following the setup Ganin al.) (2016); [iang et al. (2020), we used ResNet-50 pretrained with the ImageNet backbone model. We removed the original FC layer from the pretrained ResNet-50 and added a bottleneck FC layer (256 units) and a classification FC layer (31 units). A BN layer is put before and after the bottleneck layer, and we used the last one to calculate BN- ot-statistics matching loss. Note that the backbone part is fixed in our experiments.
classification dataset
Table I shows the test accuracy of the adapted models the target data. The results shown above the double line are those of the source-free domain adaptation methods, while the remaining ones those of the other typical domain adaptation methods. Our method achieved the best accuracy three out of six scenarios, and, surprisingly, its performance reached or exceeded the performance of the state-of-the-art typical domain adaptation methods in those cases. Moreover, our method also shows competitive performance in the other scenarios except for A - D. Since SHOT (Liang et al.) 2020) also adopts the information maximization loss, these results indicate that our BN-ssttisticss matching loss substantially improves the performance of the adaptation by successfully reducing the distributional discrepancy between domains. The model adaptation (Li et al. 2020) also works well through the all scenarios. However, considering that it requires training of conditional GAN while adaptation, our method is quite appealing due to simplicity of its training procedure as well as its high performance.
4.2 DIGIT RECOGNITION DATASETS
We examined USPS + MNIST and SVHN + MNIST scenarios. Following the previous studies Long al. 2018 Liang et al. 2020), we used the classical LeNet-5 network for the former scenario,

---

Under review as a conference paper at ICLR 2021
Experimental results with digit recognition datasets. The bold number represents the high- est test accuracy among the source-free domain adaptation methods, and the underline represents the second highest one.
Table 2:
Method
USPS MMSSTMNIST
MNIST
SHOT (Liang et al. 2020)
98.4100.6
98.0:0.22
98.91.0.0
Model adaptation (Li ct al. 2020
99.3+0.1
97.3+t.22
99.4=0.11
Our method
99.1
97.7+0..2
99.1
ADDA (Tzeng et 2017)
90.1
89,4
76.01188
rRevGrad+ CAT (Deng et al. 2019)
96.0+:099
94.0+L0.7
98.8+00.0
d-SNE Xu et al. 2019
98.5+-0.4
99.0+F0.1
96.5++0.2
99.0
97.5
98.8
Test accuracy
95.0
92.5
98.4
90.0
Test
98.2
87.5
MNIST USPS
98.0
85.0
USPS MNIST
SVHN >> MNIST
97.8
62.5
10-2
10-1
10"
101
10a
10a
hyper-parameter
The number of the target data
Figure 3: Sensitivity of the performance to the Figure 4: Sensitivity of the performance to the hyper-parameter X.
dataset size.
while a variant of LeNet, called DTN, is used for the latter one. For both models, we used the last BN layer in the model to calculate the BN statisties matching loss in method.
Table shows the experimental results with the digit recognition datasets. Although our method did not achieve the best performance among the source-free methods, it stably achieved the second highest accuracy in all scenarios. Similarly in the results with Office-31 dataset, our method exceeds the performance of the typical domain adaptation methods two scenarios, namely USPS- and SVHN MNIST.
4.3 PERFORMANCE SENSITIVITY TO THE HYPEER-PARAMETE AND DATASET SIZE
We investigated the performance sensitivity of our method to the hyper-parameter setting and that the size of the target dataset. The experimental settings same with those in the previous experiment unless otherwise noted.
Our method introduces single hyper-parametee which is in Eq. 7 We first investigated the performance sensitivity to the value of X. Since we can only access the unlabeled target data during adaptation, is essentially hard appropriately tune the hyper parameter. Therefore, high stability of the performance under suboptimal setting of the hyper-paramett is required in the source-free domain adaptation. In the experiment, we varied the value of from 0.01 to 100 and used it in our method to conduct the adaptation with the digit recognition datasets. Figure 3 shows how the test accuracy of the adapted model changes according to the value of In all adaptation scenarios, the performance of our method is quite stable against the change of the value of X. It keeps almost same within the wide range the value X, specifically 0.2 < X < 50.
We also investigated the performance of our method in case of small-scale target data. This in- vestigation crucial, because, considering the motivation domain adaptation, we cannot always expect sufficiently large amount of the target data. Since MNIST is the largest target dataset used in our experiments, we conducted this investigation with SVHN-MNN -MNIST adaptation. To make the small-scale target data, randomly selected subset of the original target training data while

---

Under review as a conference paper at ICLR 2021
100.0
600 target samples
99.5
6,000 target samples
60,000 target samples
accuracy
99.0
98.5
98.0
97.5
97.0
5000
10000
15000
20000
25000
30000
The number iterations
Figure 5: Test accuracy curves during adaptation in our method.
keeping the class prior same with that the original dataset. Figure f shows how the test accuracy after the adaptation changes according to the size of the target dataset. As decreasing the number of the target data, the performance of our method becomes deteriorated to some extent. However, even when there are only 600 samples in the target dataset. our method still achieved 98.0%, which comparable performance with those of the typical domain adaptation methods using full target dataset well source dataset.
Figure 5 shows how the test accuracy by the model changes during adaptation. The accuracy is stably and monotonically improved even when the number of the target data is small. It means that our method can effectively avoid overfitting to the small-scale target dataset.
CONCLUSION
We proposed novel domain adaptation method for source-free setting. match the distribu- tions between unobservable source data and given target data, we utilize the BN statistics stored the pretrained model to explicitly estimate and minimize distributional discrepancy between domains. This approach is quite efficient in terms of the computational cost and can be justified from the perspective risk minimization in the target domain. Experimental results with several benchmark datasets have shown that method performs well even though it does not require the access to the source data. Moreover, its performance was empirically quite stable against suboptimal hyper-parameter setting or limited size of the target dataset. In conclusion, argue that method quite promising to tackle many real- world problems that are hard to solve with existing domain adaptation methods.
REFERENCES
Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wort- man Vaughan. A theory learning from different domains. Machine learning, 195--2): 2010.
Konstantinos Bousmalis, George Trigeorgis, Nathan Silberman, Dilip Krishnan, and Dumitru Erhan. Domain separation networks. In Advances neural information processing systems, pp. 343-351, 2016.
Woong-Gi Chang, Tackgeun You, Seonguk Seo, Suha Kwak, and Bohyung Han. Domain-specific batch normalization for unsupervised domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7354-7362, 2019.
Imre Csiszar and János Kömer. Information theory: coding theorems for discrete memoryless sys- fems. Cambridge University Press, 2011.
8

---

Under review as a conference paper at ICLR 2021
Gabriela Csurka (ed.). Domain Adaptation in Computer Vision Applications. Advances in Computer Vision and Pattern Recognition. Springer, 2017.
Zhijie Deng, Yucen Luo, and Jun Zhu. Cluster alignment with a teacher for unsupervised domain adaptation. Proceedings of the IEEE International Conference on Computer Vision, pp. 9944- 9953, 2019.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand. and Victor Lempitsky. aammainadveerariall training of neural net- works. The Journal of Machine Learning Research, 17(1):2096 2030, 2016.
Trevor Hastie, Robert Tibshirani, and Jerome Friedman. The Elements of Statistical Learning Data Mining, Inference, and Prediction. Springer, second edition, 2009.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning, pp. 448-456, 2015.
Jogendra Nath Kundu, Naveen Venkat, R Venkatesh Babu, et al. Universal source-free domain adap- tation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4544-4553 2020.
Yann LeCun, Bernhard Boser, John Denker, Donnie Henderson, Richard E Howard, Wayne E Hubbard, and Lawrence D Jackel. Handwritten digit recognition with a back-propagation net- work. Advances in neural information processing systems, pp- 396 396-404, 1990.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):227)) 2222222-22222 1998.
Rui Li, Qianfen Jiao, Wenming Cao, Hau-San Wong, and Wu. Model adaptation: Unsuper- vised domain adaptation without source data. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 9641-9650, 2020.
Yanghao Li, Naiyan Wang, Jianping Shi, Xiaodi Hou, and Jiaying Liu. Adaptive batch normalization for practical domain adaptation. Pattern Recognition, 80: 10:1009117 117, 2018.
Jian Liang, Dapeng Hu, and Jiashi Feng. Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. In International Conference on Machine Learning, 2020.
Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan. Learning transferable features with deep adaptation networks. In International conference on machine learning, pp. PMLR,, 2015.
Mingsheng Long, Han Zhu, Jianmin Wang, and Michael Jordan. Deep transfer learning with joint adaptation networks. In International conference machine learning, pp. 2208-2217... PMLR, 2017.
Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael I Jordan. Conditional adversarial domain adaptation. In Advances Neural Information Processing Systems, pp. 1640-1650, 2018.
Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets.
arXiv preprint
arXiv: 1...1.11....... 2014.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning. 2011.
Joaquin Quionero-Candela, Masashi Sugiyama, Anton Schwaighofer, and Neil D Lawrence. Dataset shift in machine learning. The MIT Press, 2009.
Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell. Adapting visual category models to new domains. In European Conference Computer Vision, pp. 213-226, 2010.

---

Under review as a conference paper at ICLR 2021
Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 2962-2971. 2017.
Ximei Wang, Ying Jin, Mingsheng Long, Jianmin Wang, and Michael Jordan. Transferable nor- malization: Towards improving transferability of deep neural networks. In Advances in Neural Information Processing Systems, pp. 1953-1963, 2019.
Garrett Wilson and Diane J Cook. A survey of unsupervised deep domain adaptation. ACM Trans- actions Intelligent Systems and Technology (TIST), 11(5):1-4,, 2020.
Xiang Xu, Xiong Zhou, Ragav Venkatesan, Gurumurthy Swaminathan, and Orchid Majumder. d- sne: Domain adaptation using stochastic neighborhood embedding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2497-2506, 2019.
10


Reviews:
[{"review_id": "xPdeUw59khO", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Summary\nThis paper proposes a domain adaptation technique when source data is not available. The exponentially weighted average of BN statistics from source training along with the trained model is utilized to align source and target distributions. Source model is divided into feature encoder and classifier components based on the presence of the last BN layer. BN statistics matching loss minimizes the distribution discrepancy between source and target, whereas information maximization loss enforces the classifier to be sufficiently discriminative. Experiments on several benchmark datasets showed competitive performance with state-of-the-art domain adaptation methods.\n\n### Strong Points\n 1. Sound modelling with thorough experiments on benchmark datasets along with small-scale datasets\n 2. Application of joint optimization ensures discrimitiveness between classes at the same time ensuring domain matching.\n\n### Weak Points\n 1. As classifier part of the model is fixed I am curious how the model handles label shift (i.e. when labels distribution differs between source and target)\n 2. How does the technique performs when source model has multiple BN layers and till the first BN layer is considered as encoder will be interesting. Does choosing the last BN always perform better than first BN?", "extra_scores": {}, "extra_text": {}}, {"review_id": "NHQeiafkCgy", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "After author response: I've read the response and other reviews and keep my score (weak accept). R3 says that \"The second part of information maximization loss is nearly the same as that of SHOT-IM\", but I don't see it as a problem. The first part seems interesting and novel, and to get strong results they need to combine it with an existing technique, which seems typical. I agree with the authors and disagree with R3: that their \"method outperforms it (SHOT) in 7 out of 9 scenarios.\" So the positive is that it's a nice idea, simple method, and performs quite well.\n\nHowever, I agree with R3 that there should be more extensive ablations to understand the effect of each part, and the sensitivity analysis of lambda should be done on more datasets. Additionally, I'd like to see more detailed comparisons to related work like https://arxiv.org/abs/2006.10963 that uses batchnorm for domain adaptation. The results aren't stellar, so without a good conceptual explanation, or empirical investigation, I don't see this as a must accept.\n\n#########################################################################\n\nSummary:\n\nThis paper tackles the problem of unsupervised domain adaptation, where there may be privacy constraints on the source, so we have access to a source model but not the source data. Prior work (Liang et al) essentially adapts the model by entropy minimization / self-training on the unlabeled data. This paper proposes in addition aligning the batchnorm statistics between the source and target data (inspired by domain adversarial training). They see improvements in some digits and office adaptation tasks.\n\n#########################################################################\n\nReasons for score:\n\nAligning batch-norm statistics is a simple and nice idea, and this direction seems promising. The method even performs comparably to regular unsupervised domain adaptation methods (no privacy), that use domain adversarial training and are notoriously difficult to tune so this approach could simplify tuning in addition to preserving privacy. They show small improvements on 7/9 tasks compared to Liang et al 2020. It’s an interesting piece of work!\n\n#########################################################################\n\nPros:\n\n- I like the idea of aligning batch norm statistics when we do not have access to the source data and cannot do domain adversarial training.\n\n- Results seem quite promising, especially compared to the closest related work (Liang et al). Some other methods do better, but they seem substantially more complicated.\n\n#########################################################################\n\nCons:\n\n- It’s not easy for me to tell if there are other differences relative to Liang et al 2020 besides the addition of the L_{BNM} loss. Are the regularization, augmentation, number of epochs, etc trained the same? I noticed that the numbers in table 1 and 2 are taken directly from Liang et al. Given that the numbers aren’t clearly better (7/9 cases, and by small amounts), I’d like a more clear ablation to see that the improvement is indeed coming from L_{BNM}, and not e.g. because of a different data augmentation strategy. \n\n- Relatedly to the previous point, is Liang et al the same as your method in the case when lambda = 0? However, it seems from Figure 3, that the accuracy when you set lambda -> 0 is very low (around < 93%). But Liang et al get 98.9% on SVHN -> MNIST. Can you explain this discrepancy?\n\n- I’d compare to this simpler method: where you naively normalize the representations in the target so that the statistics match the source. That is, if mu_c, sigma_c is the target statistic for channel c, and \\hat{mu}_c, \\hat{sigma}_c is the source statistic for channel c, then just normalize the feature, so if x_c is the value of channel c for an example, then normalize it to get [(x_c - mu_c) / sigma_c] * \\hat{sigma}_c + \\hat{mu}_c. Effectively, this is explicitly / directly aligning the source and target batchnorm statistics. After doing this explicit alignment, you can then just train on the loss L_{IM} to further refine the model. This would shed insight into whether you need to jointly optimize L_{IM} and L_{BNM}, or can just do the explicit alignment of batchnorm statistics and then optimize L_{IM}.\n\n#########################################################################\n\nQuestions and things to improve:\n\n- This didn’t affect my score, but the claims about theoretical connections need to be toned down substantially. It’s fine for a method to be a heuristic that is loosely inspired by theory, but the paper just needs to be upfront about this.\n\nFor example, page 1 says “, the performance of those methods is not guaranteed by theories exploited in typical domain adaptation studies”, however a number of recent works have explained why self-training based methods do work, e.g. Kumar et al 2020, Chen et al 2020. Page 2 says domain adversarial training “is also validated in theory”. This is too strong, domain adversarial training is only inspired by theory and not validated by it. The theory in Ben-David et al and follow up works assume that there exists a classifier that does well on both the source and the target - and even if this is true in the input space, it may not be true in the learned representation space obtained from domain adversarial training. See Zhao et al 2019. In the conclusion, the paper says the method is “theoretically validated”, but the paper does not give any theoretical explanation for why the method works. This can be changed to “theoretically inspired” if you like. Similarly the use of “theoretically justified” should be toned down in section 3.1 (e.g. top of page 5)\n\n#########################################################################\n\nA. Kumar, T. Ma, P. Liang. Understanding Self-Training for Gradual Domain Adaptation. ICML 2020.\n\nY. Chen, C. Wei, A. Kumar, T. Ma. Self-training Avoids Using Spurious Features Under Domain Shift. NeurIPS 2020.\n\nH. Zhao, R. T. des Combes, K. Zhang, and G. J. Gordon. On learning invariant representation for domain adaptation. ICML 2019.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "6yWC4oa4iVt", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In the work, the authors focus on tackling the problem of source free domain adaptation. The proposed method mainly has two parts, in which the second is nearly the same as the SHOT-IM as in Liang et al., 2020 [1], while the first part aims at coping with this problem from a new perspective to align the distribution of target features extracted by the fine-tuned encoder to that of source features extracted by the pre-trained encoder. To achieve this, they utilize batch normalization statistics stored in the pre-trained model to approximate the distribution of unobserved source data. \n\nPros: \n1.\tTackling the problem of source free domain adaptation from a new perspective of aligning the distribution of target features extracted by the fine-tuned encoder to that of source features extracted by the pre-trained encoder, specifically, the BN statistic. They also provide a roughly promising theoretical analysis. To me, the first part is new, elegant, and interesting. \n2.\tThis paper is well-written and crystal-clear, making it enjoyable to read.\n\nBefore reading the second part of information maximization loss proposed by Liang et al. 2020 [1], which also tackles the source free domain adaptation, I really want to accept this paper. However, after reading the remaining part, the drawbacks of this paper are too obvious to be ignored. My major concerns can be concluded as follows:\n \n1.\tThe second part of information maximization loss is nearly the same as that of SHOT-IM as in Liang et al., 2020 [1], resulting in a limited novelty of the whole paper. To me, the second part should at least provide some insights or some different perspectives to make the technique novelty enough for a top-tier conference.\n2.\tMeanwhile, the experiments section also makes me a little disappointed. Neither state-of-the-art results nor comprehensive ablation studies are seen. First, in most domain adaptation tasks, the performance of the proposed method is obviously lower than SHOT Liang et al., 2020 [1], and Model adaptation (Li et al., 2020) [2]. For a new paper that meets the bottom line of a top-tier conference, at least little improvement should be seen in most tasks.\n3.\tFurther, since the proposed method consists of two parts, some basic ablation studies should be conducted to verify the effectiveness of both parts. Without taking apart the whole method, we will never know how much improvement each part contributes.\n4.\t Another concern is that the performance sensitivity experiment is only conducted on a simple task SVHN→MNIST, which is too weak to draw a conclusion that the proposed method keeps almost the same within the wide range of the value of λ, specifically 0.1 ≤ λ ≤ 50.\n\n\n[1] Jian Liang, Dapeng Hu, and Jiashi Feng. Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. In International Conference on Machine Learning, 2020.\n\n[2] Rui Li, Qianfen Jiao, Wenming Cao, Hau-San Wong, and Si Wu. Model adaptation: Unsupervised domain adaptation without source data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9641–9650, 2020.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This submission develops a novel technique for domain adaptation for the setup where only a trained model (but no data) from the source task is available. The authors propose to fine-tune the feature encoder using batch norm statistics of the features extracted. Additionally their criterion also promotes increasing the the mututal information between features and target classification. The developed method is experimentally evaluated on several benchmarks.

Pros:
- The problem considered is of practical relevance and general interest in ICLR community
- The proposed methodology is well motivated and shows good performance

Cons:
- There is no thorough formal analysis of when the method would work and not work; not even on an intuitive level (state conditions under which the proposed method should be expected to work better/worse than other state of the art optimization criteria for the same setup
- Alternatively to a sound theoretical analysis, the authors should provide a more extensive set of ablation experiments (this was mentioned by several reviewers)

In the current format, it remains unclear, how the research community would benefit from the study presented.
Decision: Reject