Collaborative Normalization for Unsupervised Domain Adaptation

Abstract: Batch Normalization (BN) as an important component assists Deep Neural Networks achieving promising performance for extensive learning tasks by scaling distribution of feature representations within mini-batches. However, the application of BN suffers from performance degradation under the scenario of Unsupervised Domain Adaptation (UDA), since the estimated statistics fail to concurrently describe two different domains. In this paper, we develop a novel normalization technique, named Collaborative Normalization (CoN), for eliminating domain discrepancy and accelerating the model training of neural networks for UDA. Unlike typical strategies only exploiting domain-specific statistics during normalization, our CoN excavates cross-domain knowledge and simultaneously scales features from various domains by mimicking the merits of collaborative representation. Our CoN can be easily plugged into popular neural network backbones for cross-domain learning. One the one hand, \textbf{theoretical analysis} guarantees that models with CoN promote discriminability of feature representations and accelerate convergence rate; on the other hand, \textbf{empirical study} verifies that replacing BN with CoN in popular network backbones effectively improves classification accuracy with \textbf{4\%} in most learning tasks across three cross-domain visual benchmarks.

Reviews and author responses:

**Summarize what the paper claims to contribute.**
 
The paper proposes a normalization technique called collaborative normalization, which can substitute batch normalization in any neural network and achieve better performance on domain adaptation tasks than the previous works. The proposed method introduces an additional domain attention, collaborative translation, which calibrates the output of domain-specific batchNorm to exploit common domain information by exploiting cross-domain projection error.
The author gives an geometrical interpretation of the proposed method and it is the first trial to analyze a normalization in domain adaptation in a theoretical manner.
The performance of the proposed method is evaluated on the several domain adaptation benchmarks and it achieves the best performance compared to its counterpart techniques.
 
**List strong and weak points of the paper.**
 
*strong points*
 
The proposed method introduces collaborative normalization a closed-form domain attention to calibrate the latent features in the batch normalization. It boosts performance without additional learning parameters.
 
For my best understading, it is the very first paper trying theoretical analysis on normalization for domain adaptation, while the other works [R1,R2,R3,R4, AdaBN, TransNorm] only show empirical performance.
 
*weak points*
 
The proposed method adds an incremental contribution compared to TransNorm (Wang et al. (2019a)). The versatility of the proposed method is limited to only two domain cases, although domain-specific BN is applicable to multi-source domain adaptation.
 
As my understanding, the proof is not directly related with domain discrepancy. and the proof don’t compare the proposed method with the other methods, BatchNorm, TransNorm.
 
**Clearly state your recommendation (accept or reject) with one or two key reasons for this choice**
 
I give “Ok but not good enough - rejection (4)” to this paper. The paper is well-organized and it firstly tries to give the proof on normalization for domain adaptation. But, the proposed method is incremental to the previous work [TransNorm]. I think the proof is not in a complete form. I write the questions on the proof in below section.
 
**Ask questions you would like answered by the authors to help you clarify your understanding of the paper and provide the additional evidence you need to be confident in your assessment.**
 
I have questions on collaborative normalization because I can’t fully understand the description in <section 3.2 collaborative normalization>. Thank you in advance.
 
First, in the end of the section 3.2, you mentioned that “effectively eliminates the cross-domain discrepancy”. I have a confusion because domain descrepancy usually is usually formated as a kind of distance function between source and target distribution. (1) Could you explain how the sample-specific error term leads to the entire domain discrepancy?
 
Second, could you give an inquality that CoN achieves better approximated error (or domain discrepancy) than TransNorm and AdaBN in terms of approximated error $\tilde{\epsilon_i}$ or related terms?
 
Third, a distribution on latent features can not be fully represented by mean and standard deviation of batch normalization. If the underlying distribution on latent feature is Gaussian distribution, then it can be represented as mean and covariance. In this reason, I think that the given proof is a kind of very good draft for future proof.
 
**Provide additional feedback with the aim to improve the paper.** 
 
Several related works for domain adaptation with batch normalization are missing. Although I know that there are too many relevant works for domain adaptation, this work misses too many works on normalization-inspired works for domain adaptation. The works [R1, R2] can be an alternative for AdaBN, TransNorm.
 
There are works on combining several normalizations [R3, R4] to achieve better performance on several tasks even for domain adaptation [R3]. What do you think about those works? and how they are related to your work?
 
[R1] Domain-Specific Batch Normalization for Unsupervised Domain Adaptation, CVPR 19
 
[R2] Unsupervised Domain Adaptation using Feature-Whitening and Consensus Loss, CVPR 19
 
[R3] Batch-Instance Normalization for Adaptively Style-Invariant Neural Networks, NeurIPS 18
 
[R4] Switchable Whitening for Deep Representation Learning, ICCV 19


This paper proposes a novel normalization method termed Collaborative Normalization (CoN) to eliminate domain discrepancy for unsupervised domain adaptation. In practice, CoN firstly exploits domain-specific statistics to normalize features from source and target domain. Then it calculates the collaborative representations across domains and gradually aligns samples from one domain to the other by using these representations. Extensive experimental results demonstrate the effectiveness of the proposed normalization method in various unsupervised domain adaptation benchmarks.


Pros: 
+ Neat motivation; 
+ Novel methods with clear contributions;
+ The writing of this paper is excellent;
+ Extensive experiments and good performance.

Cons:

I would like to hear more about the implementation details. 
- Are all of the models fine-tuned from ImageNet? If yes, is it stable to replace CoN from BN in the training process?


- Do you replace all of the BN layers in ResNet50? Is it reasonable to apply global pooling for low-level features that haven’t captured enough semantic information?


- The authors aim to estimate the location of the source (target) features in the target (source) domain. In practice, the Collaborative Translation is realized across the mini-batch in each iteration. Is the proposed method sensitive to the mini-batch size and the distribution of the categories in a mini-batch, especially compared with TN? How about the performance of the proposed method and TN by using a smaller batch size?

--Paper summary--

The authors propose extended batch normalization, called collaborative normalization, to tackle unsupervised domain adaptation. It normalizes features using domain-specific statistics and shifts the normalized features to gradually align their distributions between domains through the model. Experimental results with several datasets validate the advantage of the proposed method.


--Review summary--

This paper is well-written and well-organized. The proposed module should be widely applicable to various DNN architectures, and it works well in the experiments with several datasets. However, some points remain unclear or are not convincing, which makes my score a bit conservative. To appropriately determine my score, I would appreciate if the authors clarify these points in their response.


--Details--

Strength

- The proposed module can be widely applied to modern DNN architectures.
- This paper is well-written and well-organized. 


Weakness and concerns

- Some points in the design of the proposed method remain unclear or are not convincing.
	- The definition of \eta in Eq. (4) is intuitively unreasonable and needs more justification. When the distributions are perfectly aligned between domains, \eta should diverge, because the denominator becomes zero.
	- It should be also justified to use d in the numerator. Why do not we need any other coefficient to appropriately scale the shift value?
	- It seems reasonable to define A^{st/ts} using m^{t/s} as shown in the end of "With respect to the second challenge." However, instead of them, the authors use cosine similarities between Gs to compute A^{st/ts}, and it is not justified theoretically or empirically.

- As far as I understand, the mini-batch from target data is assumed to contain all class candidates, but this assumption can be easily violated if the number of class candidates is large or the size of mini-batch is small. Does the performance of the proposed method heavily depend on the number of class candidates and the size of mini-batch?

- Just after Eq. (7), the authors state "since y^s has the highest cosine similarity with y_i^t," but why is it?

- The existing studies on subspace-alignment based domain adaptation (e.g., [R1]) should be referred.  
[R1] "Unsupervised visual domain adaptation using subspace alignment," ICCV 2013.

- Figure 2 should be further improved. For example, \hat{y}^S should be \tilde{y}^S, and \tilde{y}^S should be \overline{y}^S.


Meta-review: Although all reviewers acknowledge that the paper has some merit, the work lacks in novelty. The idea of using normalisation techniques to reduce the domain discrepancy in UDA is well established in the DA and DG community. The theoretical analysis is an interesting first step in providing some insights on this class of approaches, but it still does not support an understanding on why some of these methods work better than others. Given all these considerations, the work is not ready for acceptance at ICLR.

Decision: withdrawn-rejected-submissions