Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: After author response: I've read the response and other reviews and keep my score (weak accept). R3 says that "The second part of information maximization loss is nearly the same as that of SHOT-IM", but I don't see it as a problem. The first part seems interesting and novel, and to get strong results they need to combine it with an existing technique, which seems typical. I agree with the authors and disagree with R3: that their "method outperforms it (SHOT) in 7 out of 9 scenarios." So the positive is that it's a nice idea, simple method, and performs quite well.

However, I agree with R3 that there should be more extensive ablations to understand the effect of each part, and the sensitivity analysis of lambda should be done on more datasets. Additionally, I'd like to see more detailed comparisons to related work like https://arxiv.org/abs/2006.10963 that uses batchnorm for domain adaptation. The results aren't stellar, so without a good conceptual explanation, or empirical investigation, I don't see this as a must accept.

#########################################################################

Summary:

This paper tackles the problem of unsupervised domain adaptation, where there may be privacy constraints on the source, so we have access to a source model but not the source data. Prior work (Liang et al) essentially adapts the model by entropy minimization / self-training on the unlabeled data. This paper proposes in addition aligning the batchnorm statistics between the source and target data (inspired by domain adversarial training). They see improvements in some digits and office adaptation tasks.

#########################################################################

Reasons for score:

Aligning batch-norm statistics is a simple and nice idea, and this direction seems promising. The method even performs comparably to regular unsupervised domain adaptation methods (no privacy), that use domain adversarial training and are notoriously difficult to tune so this approach could simplify tuning in addition to preserving privacy. They show small improvements on 7/9 tasks compared to Liang et al 2020. It’s an interesting piece of work!

#########################################################################

Pros:

- I like the idea of aligning batch norm statistics when we do not have access to the source data and cannot do domain adversarial training.

- Results seem quite promising, especially compared to the closest related work (Liang et al). Some other methods do better, but they seem substantially more complicated.

#########################################################################

Cons:

- It’s not easy for me to tell if there are other differences relative to Liang et al 2020 besides the addition of the L_{BNM} loss. Are the regularization, augmentation, number of epochs, etc trained the same? I noticed that the numbers in table 1 and 2 are taken directly from Liang et al. Given that the numbers aren’t clearly better (7/9 cases, and by small amounts), I’d like a more clear ablation to see that the improvement is indeed coming from L_{BNM}, and not e.g. because of a different data augmentation strategy. 

- Relatedly to the previous point, is Liang et al the same as your method in the case when lambda = 0? However, it seems from Figure 3, that the accuracy when you set lambda -> 0 is very low (around < 93%). But Liang et al get 98.9% on SVHN -> MNIST. Can you explain this discrepancy?

- I’d compare to this simpler method: where you naively normalize the representations in the target so that the statistics match the source. That is, if mu_c, sigma_c is the target statistic for channel c, and \hat{mu}_c, \hat{sigma}_c is the source statistic for channel c, then just normalize the feature, so if x_c is the value of channel c for an example, then normalize it to get [(x_c - mu_c) / sigma_c] * \hat{sigma}_c + \hat{mu}_c. Effectively, this is explicitly / directly aligning the source and target batchnorm statistics. After doing this explicit alignment, you can then just train on the loss L_{IM} to further refine the model. This would shed insight into whether you need to jointly optimize L_{IM} and L_{BNM}, or can just do the explicit alignment of batchnorm statistics and then optimize L_{IM}.

#########################################################################

Questions and things to improve:

- This didn’t affect my score, but the claims about theoretical connections need to be toned down substantially. It’s fine for a method to be a heuristic that is loosely inspired by theory, but the paper just needs to be upfront about this.

For example, page 1 says “, the performance of those methods is not guaranteed by theories exploited in typical domain adaptation studies”, however a number of recent works have explained why self-training based methods do work, e.g. Kumar et al 2020, Chen et al 2020. Page 2 says domain adversarial training “is also validated in theory”. This is too strong, domain adversarial training is only inspired by theory and not validated by it. The theory in Ben-David et al and follow up works assume that there exists a classifier that does well on both the source and the target - and even if this is true in the input space, it may not be true in the learned representation space obtained from domain adversarial training. See Zhao et al 2019. In the conclusion, the paper says the method is “theoretically validated”, but the paper does not give any theoretical explanation for why the method works. This can be changed to “theoretically inspired” if you like. Similarly the use of “theoretically justified” should be toned down in section 3.1 (e.g. top of page 5)

#########################################################################

A. Kumar, T. Ma, P. Liang. Understanding Self-Training for Gradual Domain Adaptation. ICML 2020.

Y. Chen, C. Wei, A. Kumar, T. Ma. Self-training Avoids Using Spurious Features Under Domain Shift. NeurIPS 2020.

H. Zhao, R. T. des Combes, K. Zhang, and G. J. Gordon. On learning invariant representation for domain adaptation. ICML 2019.

review_title: Cool idea, some questions about comparison with Liang et al 2020

final_score: rating: 4: Ok but not good enough - rejection
review_content: In the work, the authors focus on tackling the problem of source free domain adaptation. The proposed method mainly has two parts, in which the second is nearly the same as the SHOT-IM as in Liang et al., 2020 [1], while the first part aims at coping with this problem from a new perspective to align the distribution of target features extracted by the fine-tuned encoder to that of source features extracted by the pre-trained encoder. To achieve this, they utilize batch normalization statistics stored in the pre-trained model to approximate the distribution of unobserved source data. 

Pros: 
1.	Tackling the problem of source free domain adaptation from a new perspective of aligning the distribution of target features extracted by the fine-tuned encoder to that of source features extracted by the pre-trained encoder, specifically, the BN statistic. They also provide a roughly promising theoretical analysis. To me, the first part is new, elegant, and interesting. 
2.	This paper is well-written and crystal-clear, making it enjoyable to read.

Before reading the second part of information maximization loss proposed by Liang et al. 2020 [1], which also tackles the source free domain adaptation, I really want to accept this paper. However, after reading the remaining part, the drawbacks of this paper are too obvious to be ignored. My major concerns can be concluded as follows:
 
1.	The second part of information maximization loss is nearly the same as that of SHOT-IM as in Liang et al., 2020 [1], resulting in a limited novelty of the whole paper. To me, the second part should at least provide some insights or some different perspectives to make the technique novelty enough for a top-tier conference.
2.	Meanwhile, the experiments section also makes me a little disappointed. Neither state-of-the-art results nor comprehensive ablation studies are seen. First, in most domain adaptation tasks, the performance of the proposed method is obviously lower than SHOT Liang et al., 2020 [1], and Model adaptation (Li et al., 2020) [2]. For a new paper that meets the bottom line of a top-tier conference, at least little improvement should be seen in most tasks.
3.	Further, since the proposed method consists of two parts, some basic ablation studies should be conducted to verify the effectiveness of both parts. Without taking apart the whole method, we will never know how much improvement each part contributes.
4.	 Another concern is that the performance sensitivity experiment is only conducted on a simple task SVHN→MNIST, which is too weak to draw a conclusion that the proposed method keeps almost the same within the wide range of the value of λ, specifically 0.1 ≤ λ ≤ 50.


[1] Jian Liang, Dapeng Hu, and Jiashi Feng. Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. In International Conference on Machine Learning, 2020.

[2] Rui Li, Qianfen Jiao, Wenming Cao, Hau-San Wong, and Si Wu. Model adaptation: Unsupervised domain adaptation without source data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9641–9650, 2020.

review_title: I appreciated the first new part,  but I felt disappointed of the second part previously proposed by others, the experiment results and the inadequate ablation studies.

final_score: rating: 6: Marginally above acceptance threshold
review_content: ### Summary
This paper proposes a domain adaptation technique when source data is not available. The exponentially weighted average of BN statistics from source training along with the trained model is utilized to align source and target distributions. Source model is divided into feature encoder and classifier components based on the presence of the last BN layer. BN statistics matching loss minimizes the distribution discrepancy between source and target, whereas information maximization loss enforces the classifier to be sufficiently discriminative. Experiments on several benchmark datasets showed competitive performance with state-of-the-art domain adaptation methods.

### Strong Points
 1. Sound modelling with thorough experiments on benchmark datasets along with small-scale datasets
 2. Application of joint optimization ensures discrimitiveness between classes at the same time ensuring domain matching.

### Weak Points
 1. As classifier part of the model is fixed I am curious how the model handles label shift (i.e. when labels distribution differs between source and target)
 2. How does the technique performs when source model has multiple BN layers and till the first BN layer is considered as encoder will be interesting. Does choosing the last BN always perform better than first BN?
review_title: Domain adaptation with meta information when source data is not available

Meta-review: This submission develops a novel technique for domain adaptation for the setup where only a trained model (but no data) from the source task is available. The authors propose to fine-tune the feature encoder using batch norm statistics of the features extracted. Additionally their criterion also promotes increasing the the mututal information between features and target classification. The developed method is experimentally evaluated on several benchmarks.

Pros:
- The problem considered is of practical relevance and general interest in ICLR community
- The proposed methodology is well motivated and shows good performance

Cons:
- There is no thorough formal analysis of when the method would work and not work; not even on an intuitive level (state conditions under which the proposed method should be expected to work better/worse than other state of the art optimization criteria for the same setup
- Alternatively to a sound theoretical analysis, the authors should provide a more extensive set of ablation experiments (this was mentioned by several reviewers)

In the current format, it remains unclear, how the research community would benefit from the study presented.
Decision: Reject