Title: Revisiting Batch Normalization For Practical Domain Adaptation
Authors: Yanghao Li
Naiyan Wang
Jianping Shi
Jiaying Liu
Xiaodi Hou
Venue: ICLR 2017

Abstract: Deep neural networks (DNN) have shown unprecedented success in various computer vision applications such as image classification and object detection. However, it is still a common annoyance during the training phase, that one has to prepare at least thousands of labeled images to fine-tune a network to a specific domain. Recent study shows that a DNN has strong dependency towards the training dataset, and the learned features cannot be easily transferred to a different but relevant task without fine-tuning. In this paper, we propose a simple yet powerful remedy, called Adaptive Batch Normalization (AdaBN) to increase the generalization ability of a DNN. By modulating the statistics from the source domain to the target domain in all Batch Normalization layers across the network, our approach achieves deep adaptation effect for domain adaptation tasks. In contrary to other deep learning domain adaptation methods, our method does not require additional components, and is parameter-free. It archives state-of-the-art performance despite its surprising simplicity. Furthermore, we demonstrate that our method is complementary with other existing methods. Combining AdaBN with existing domain adaptation treatments may further improve model performance.

TL;DR: We propose a simple yet effective approach for domain adaptation on batch normalized neural networks.

Review discussion:
1. p. 5, 3.3, What is the purpose of the second paragraph? I'm not sure I understand the link between this one and the previous one.
2. p. 6, Table 1, What is the evaluation protocol for the Office dataset? DAN results are for the transductive setting; RevGrad results differ from the ones in the ICML paper (also transductive setting)
3. The authors are using Inception BN as a base model while the other competing methods are built on top of AlexNet. Could that happen that those methods would also benefit from having a stronger base model. Is it possible to conduct a fair comparison?
Thanks a lot for your questions. Our answers are shown as follows.
1. In section 3.3, we make further discussions about two questions of our AdaBN after explaining the algorithm in section 3.1 and section 3.2. This section is not to explain more implementation details of the algorithm, but to discuss about some complementary thoughts of our method.
a) We explain why the simple translation and scaling operations in our AdaBN could approximate complex non-linear domain transfer function. This discussion demonstrates that our method has potentially strong transfer power despite of the simplicity.
b) We discuss about why we choose to transform neural response independently instead of decorrelating and then re-correlating the covariance matrix.
2. In our knowledge, there are two unsupervised protocols for the Office dataset.
a) Use all source examples with labels and all target examples without labels in DAN (Long et al., 2015), CORAL (Sun et al., 2016) and RevGrad (Ganin & Lempitsky, 2015)
b) Subsample source examples (8 samples per category for webcam/dslr and 20 for amazon) in Saenko et al., 2010.
We use the first protocol in Table 1. In DAN (Long et al., 2015), CORAL (Sun et al., 2016) and RevGrad (Ganin & Lempitsky, 2015), the reproduced results are inconsistent with the same method (such as RevGrad) although these papers use the same protocol.
In our paper, the results of AlexNet, DDC and DAN are taken from the Table 1 in DAD (Long et al., 2015). The results of Deep CORAL are taken directly from its original paper. The results of RevGrad were taken from CORAL (Sun et al., 2016) before. We have updated the results of RevGrad from its original paper now.
3. Since our method has to use deep models with BN layers, AlexNet is not suitable and we choose to use the Inception-BN model. For a fair comparison, we tried our best to transfer other methods to the Inception-BN model. We have implemented SA, GFK, LSSA and CORAL to reproduce results based on Inception-BN.
However, it is not easy to transform from AlexNet to Inception-BN for other methods. There are two main reasons.
First, some methods (e.g. DDC, Deep CORAL) do not have publicly available implementations.
Second, AlexNet has 3 fully-connected layers, which is different with Inception-BN.
However, some strategies in these methods are based on the structure of AlexNet.
For example, DAN uses MK-MMD among the 3 fully connected layers. We tried to implement some of them based on Inception-BN but did not get promising results even worse than baseline method. This may be due to some missing implementation technologies or the unsuitableness of these methods for different network structures.
Thus, we do not report these results for fair comparisons.
Overall I think this is an interesting paper which shows empirical performance improvement over baselines.
However, my main concern with the paper is regarding its technical depth, as the gist of the paper can be summarized as the following: instead of keeping the batch norm mean and bias estimation over the whole model, estimate them on a per-domain basis. I am not sure if this is novel, as this is a natural extension of the original batch normalization paper.
Overall I think this paper is more fit as a short workshop presentation rather than a full conference paper.
Detailed comments:
Section 3.1: I respectfully disagree that the core idea of BN is to align the distribution of training data. It does this as a side effect, but the major purpose of BN is to properly control the scale of the gradient so we can train very deep models without the problem of vanishing gradients. It is plausible that intermediate features from different datasets naturally show as different groups in a t-SNE embedding. This is not the particular feature of batch normalization: visualizing a set of intermediate features with AlexNet and one gets the same results.
So the premise in section 3.1 is not accurate.
Section 3.3: I have the same concern as the other reviewer. It seems to be quite detatched from the general idea of AdaBN. Equation 2 presents an obvious argument that the combined BN-fully_connected layer forms a linear transform, which is true in the original BN case and in this case as well. I do not think it adds much theoretical depth to the paper. (In general the novelty of this paper seems low)
Experiments:
- section 4.3.1 is not an accurate measure of the "effectiveness" of the proposed method, but a verification of a simple fact: previously, we normalize the source domain features into a Gaussian distribution. the proposed method is explicitly normalizing the target domain features into the same Gaussian distribution as well.
So, it is obvious that the KL divergence between these two distributions are closer - in fact, one is *explicitly* making them close.
However, this does not directly correlate to the effectiveness of the final classification performance.
- section 4.3.2: the sensitivity analysis is a very interesting read, as it suggests that only a very few number of images are needed to account for the domain shift in the AdaBN parameter estimation. This seems to suggest that a single "whitening" operation is already good enough to offset the domain bias (in both cases shown, a single batch is sufficient to recover about 80% of the performance gain, although I cannot get data for even smaller number of examples from the figure). It would thus be useful to have a comparison between these approaches, and also a detailed analysis of the effect from each layer of the model - the current analysis seems a bit thin.
Thanks for your insightful comments and suggestions of our work.
1.
About section 3.1
Thanks for your suggestion and we have updated our writing.
However, we think aligning the distribution of training data is not just a side effect. It is the key way to achieve the purpose of BN which is to avoid the problem of vanishing gradients and help optimization. In original BN paper, the authors’ motivation is to address the problem of “internal covariate shift”, which means “the change in the distributions of layers’ inputs”.
Thus, BN is proposed to “reduce internal covariate shift” and make “the distribution of nonlinearity inputs more stable”.
(2) We also directly visualize the intermediate features with Inception-BN network instead of our BN features in Sec. 3.1. The figure can be seen at this link (https://s30.postimg.org/fdamc2l1t/a2d_feature_tsne.png). Red circles are features of samples from training domain (Amazon) while blue ones are testing features (DSLR). It blends much more than that in Figure 2. This demonstrates the statistics of BN layer indeed contain the traits of the data domain. The features of individual image cannot be separated directly in terms of different domains.
2.
About section 3.3 and section 4.3.1
Thanks for your suggestion. We have revised section 3.3 to make it clearer and merged it to the section 3.2.
According to your and Reviewer2’s suggestion, we remove the previous section 4.3.1.
3.
About section 4.3.2
Thanks for your suggestion. We have updated additional experimental results in the revised paper.
(1) We have experiments with smaller number of samples and found that the performance will drop more (e.g. 0.652 with 16 samples, 0.661 with 32 samples.) We have updated the results in the section “Sensitivity to target domain size” (section 4.3.1 now).
(2) In the section “Adaptation Effect for Different BN Layers” (section 4.3.2 now), we add the detailed analysis of adaptation effect for different BN layers of our AdaBN method.
Update: I thank the authors for their comments. I still think that the method needs more experimental evaluation: for now, it's restricted to the settings in which one can use pre-trained ImageNet model, but it's also important to show the effectiveness in scenarios where one needs to train everything from scratch. I would love to see a fair comparison of the state-of-the-art methods on toy datasets (e.g. see (Bousmalis et al., 2016), (Ganin & Lempitsky, 2015)). In my opinion, it's a crucial point that doesn't allow me to increase the rating.
This paper proposes a simple trick turning batch normalization into a domain adaptation technique. The authors show that per-batch means and variances normally computed as a part of the BN procedure are sufficient to discriminate the domain. This observation leads to an idea that adaptation for the target domain can be performed by replacing population statistics computed on the source dataset by the corresponding statistics from the target dataset.
Overall, I think the paper is more suitable for a workshop track rather than for the main conference track. My main concerns are the following:
1.
Although the main idea is very simple, it feels like the paper is composed in such a way to make the main contribution less obvious (e.g. the idea could have been described in the abstract but the authors avoided doing so).
2. (This one is from the pre-review questions) The authors are using much stronger base CNN which may account for the bulk of the reported improvement. In order to prove the effectiveness of the trick, the authors would need to conduct a fair comparison against competing methods. It would be highly desirable to conduct this comparison also in the case of a model trained from scratch (as opposed to reusing ImageNet-trained networks).
Thanks for your insightful comments and suggestions of our work.
1.
About writing.
We have updated the writing of abstract to make our idea clearer.
2.
About experiments.
(1) We agree with the reviewer that same base network should be used for a fair comparison.
Thus, we tried our best to transfer other methods to the Inception-BN model. As shown in our answers to the pre-review questions, we have implemented SA, GFK, LSSA and CORAL to reproduce results based on Inception-BN. For other methods, we had not reproduced promising results. This may be due to some missing implementation details (some methods do not have public implementations) or the uniqueness of AlexNet (AlexNet has intrinsic different structure with Inception).
Thus, we do not report these controversial results.
(2) We respectfully disagree with the reviewer’s suggestion of training a model from scratch. Currently, most computer vision tasks (including classification, detection and segmentation) use an pre-trained model, and it has significant impact on the performance. For our domain adaptation task (especially for Office dataset), the number of images is too small (e.g. 500 images for DSLR domain) compared to that of ImageNet dataset (1M images).  Training a DNN like AlexNet or Inception-BN with such amount images would suffer from severe over-fitting and significantly drop the performance.
Thus, we think this comparison with a model trained from scratch is meaningless. It cannot manifest the difference of different methods.
This paper proposes a simple domain adaptation technique in which batch normalization is performed separately in each domain.
Pros:
The method is very simple and easy to understand and apply.
The experiments demonstrate that the method compares favorably with existing methods on standard domain adaptation tasks.
The analysis in section 4.3.2 shows that a very small number of target domain samples are needed for adaptation of the network.
Cons:
There is little novelty -- the method is arguably too simple to be called a “method.” Rather, it’s the most straightforward/intuitive approach when using a network with batch normalization for domain adaptation.  The alternative -- using the BN statistics from the source domain for target domain examples -- is less natural, to me. (I guess this alternative is what’s done in the Inception BN results in Table 1-2?)
The analysis in section 4.3.1 is superfluous except as a sanity check -- KL divergence between the distributions should be 0 when each distribution is shifted/scaled to N(0,1) by BN.
Section 3.3: it’s not clear to me what point is being made here.
Overall, there’s not much novelty here, but it’s hard to argue that simplicity is a bad thing when the method is clearly competitive with or outperforming prior work on the standard benchmarks (in a domain adaptation tradition that started with “Frustratingly Easy Domain Adaptation”).  If accepted, Sections 4.3.1 and 3.3 should be removed or rewritten for clarity for a final version.
Thanks for your insightful comments and suggestions of our work.
1. As mentioned in the original BN paper: “the means and variances are fixed during inference.” ( Ioffe & Szegedy, 2015).
In practice, many methods “freeze the batch normalization parameters to those estimated during ImageNet pre-training” (“Speed/accuracy trade-offs for modern convolutional object detectors”) for detection and segmentation tasks.
Thus, we think modulating BN statistics in the inference is far from been explored, especially for the domain adaptation task.
2.
About section 3.3 and section 4.3.1
Thanks for your suggestions. We have updated our paper.
(1) We have revised section 3.3 to make it clearer and merge it to the section 3.2.
(2) We have removed the original section 4.3.1.
At the same time, we also added a new analysis section “Adaptation Effect for Different BN Layers”.
We do not think simplicity is our drawback of our method.
On the contrary, we believe this is a great advantage of our method. Being technically simple does not mean no novelty, and it should never be a reason to reject a paper.
In fact, Batch Normalization is a very simple method, while dropout is even simpler. These techniques have had huge impacts to the field despite the simplicity. As the Reviewer2 indicates, there are also prior simple but important methods in domain adaptation. (e.g. “Frustratingly Easy Domain Adaptation” and “Return of Frustratingly Easy Domain Adaptation”) Further, to our best knowledge, there are no prior works to exploit Batch Normalization for domain adaptation.
The main contributions in our paper is that we propose a simple yet effective AdaBN method for domain adaptation by modulating the statistics in all BN layers, which outperforms the states-of-the-art methods.
Furthermore, we demonstrate that our method is complementary with other existing methods.
Thus, we think it is valuable for the researchers in the field.
According to the reviewers’ helpful suggestions, we have revised and updated our paper. The main modifications are as follows:
(1) We have updated the writing of our abstract and revised section 3.3 to make it clearer and merge it to the section 3.2.
(2) We have removed the original section 4.3.1, and revised section “sensitivity to target domain size” by adding the experimental results of using smaller number of images.
(3) We have added a new analysis section “Adaptation Effect for Different BN Layers” in section 4.3.2.
This work also inspired our another recent work "Demystifying Neural Style Transfer"(https://arxiv.org/abs/1701.01036), which explains why Gram matrix represents the artistic style in neural style transfer. In that work, we also exploit the idea in our AdaBN method: matching BN statistics to align distributions in different domains, for neural style transfer and achieves comparable results with more complicated methods such as MMD. We believe this demonstrates the effectiveness of our method in domain adaptation field.
We find another related paper [a] to support the effectiveness of our AdaBN. In that paper, the authors also proposed a simple but effective method to learn multiple styles at the same time for neural style transfer. In our previous comment, we have shown that neural style transfer can be intrinsically considered as a domain adaptation problem, which shares the same core process: distribution alignment. In [a], authors share a similar idea with our AdaBN method: modulating BN layers for distribution alignment. Differently, [a] learns different slope and bias terms in BN layers for different 'domains' (if we consider a target style image is an individual domain).
While our AdaBN uses the mean and variance statistics in BN layers for different domains.
We believe our AdaBN and [a] both exploit the functionality of BN layers for distribution alignment, which is then applied in domain adaptation and neural style transfer, respectively.
At the same time, our paper "Demystifying Neural Style Transfer" also connects the neural style transfer and domain adaptation. We think these three papers together provide a more holistic and systematical perspective of these two fields and the connection with BN layers.
[a] A Learned Representation For Artistic Style (https://openreview.net/forum?id=BJO-BuT1g)

Meta-review:
role: PC
data: comment: The paper performs domain adaptation using a very simple trick inspired by BatchNorm. The paper received below margin scores. The reviewers both, liked the simplicity of the approach, and at the same time felt that the contribution was too thin. Given the high bar of ICLR, this paper falls short.

Decision: Reject