Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper studies the impact of embedding complexity on domain-invariant representations. By incorporating embedding complexity into the previous upper bound explicitly, the authors demonstrate the limitations of previous theories and algorithms. Based on their theoretical findings, the authors propose to control the embedding complexity with implicit regularization. Specifically, aligning source and target feature distributions in multiple layers controls both embedding complexity and domain discrepancy. The proposed algorithm can achieve similar performance as DANN with manual selection of embedding depth.

By noting that the hypothesis space can be decomposed in to the feature extractor and the classifier, the authors propose to address the domain discrepancy separately. D_H\DeltaH is termed latent divergence, which the algorithm attempts to minimize. D_G\DeltaG is treated as embedding complexity, which is the intrinsic property of the feature extractor. Thus, domain-invariant representations should seek a proper tradeoff between those two terms. 

The paper is well-written and the contributions are stated clearly. The exploration on the layer division is really insightful.

However, I have several concerns: 
1.	The proposed upper bound is insightful, but it has several limitations. Compared to the version applied to the feature space in equation (3), the proposed upper bound is looser. The embedding complexity terms includes two encoders, which are deep neural networks in practice, thus it can be excessively large. As the authors point out, in equation (3), the embedding complexity is not addressed explicitly, but it is implicit in the adaptability \lambda in a more reasonable way. Previous works [1], [2], [3] have already taken them into consideration. Proposition 5 is a direct application of proposition 1 in [1].
2.	On the claim of implicit regularization. By applying domain adversarial training to multiple layers, the authors claim that the encoder in higher layers is implicitly restricted. However, they do not validate this regularization effect. Is the embedding complexity controlled? Theoretical analysis or experimental results would be helpful.
3.	The proposed MDM method seems to be incremental. [4] has probed into the effect of multi-layer adaptation strategy. Besides, applying domain adversarial training to many layers leads to more computational cost and may slow down training significantly. 


[1]Fredrik D Johansson, Rajesh Ranganath, and David Sontag. Support and invertibility in domain- invariant representations. arXiv preprint arXiv:1903.03448, 2019.
[2]Han Zhao, Remi Tachet des Combes, Kun Zhang, and Geoffrey J Gordon. On learning invariant representation for domain adaptation. arXiv preprint arXiv:1901.09453, 2019.
[3] Hong Liu, Mingsheng Long, Jianmin Wang, and Michael Jordan. Transferable adversarial training: A general approach to adapting deep classifiers. In International Conference on Machine Learning, pp. 4013–4022, 2019.
[4] Mingsheng Long, Yue Cao, Jianmin Wang, and Michael I. Jordan. Learning transferable features with deep adaptation networks. In Proceedings of the 32nd International Conference on International Conference on Machine Learning, volume 37, pp. 97–105, 2015.

review_title: Official Blind Review #4

final_score: rating: 1: Reject
review_content: This paper studies the problem of domain adaptation via learning invariant representations. The main argument here is that when the total depth of layers in a neural network is fixed, tradeoffs exist between feature alignment and prediction power. Furthermore, the authors argue that richer feature extractor can sometimes significantly overfit the source domain, leading to a large risk on the target domain. 

Overall the paper is well-written and easy to follow. My major concern is that the paper, including the motivation and illustrative example, are too similar to previous work [1-2]. More detailed discussions are needed to highlight the difference of this work compared with [1-2]. The main contribution lies in Theorem 4. However, the upper bound is both loose and misleading. Compared with the original generalization upper bound [3], the one proposed in this paper contains a constant $\lambda$ that contains FOUR optimal error terms. Note that the original one in [3] only contains two such terms. In fact even a bound containing 2 such terms could potentially be very loose, since it's perfectly fine that a hypothesis can have large risk on the source domain while still attaining a small risk on the target domain. The bound is misleading in the sense that this $\lambda$ term cannot be computed or approximated, hence only the first two terms in (6) could be minimized in practice. However, this again can potentially lead to large target risk when the label distributions of source and target domains differ. 

The experiments on using different number of layers of the network as feature extractors are quite interesting. The main message here is that general tradeoff exists with richer encoding function class. However, similar phenomenons have already been observed [4, Section 6.4], and it's not clear to me what's new here. 

[1].    On Learning Invariant Representations for Domain Adaptation, ICML 2019.
[2].    Domain Adaptation with Asymmetrically-Relaxed Distribution Alignment, ICML 2019.
[3].    Analysis of representations for domain adaptation, NIPS 2007.
[4].    A DIRT-T APPROACH TO UNSUPERVISED DOMAIN ADAPTATION, ICLR 2018.
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: This paper proposes a new theory for domain adaptation considering the complexity of representation extractors. This paper gives a new bound for target error in domain adaptation, which contains the classic distribution distance related to the hypothesis space of high-level classifiers and a new distribution distance defined on the embedding space. This paper also proposes Multilayer Divergence Minimization algorithm based on the theory and evaluates it on real-world dataset. 
Positive points: 
(a) This paper proposes an interesting insight that the complexity of embeddings is also important in domain adaptation.
(b) This paper defines a new distribution divergence and build an interesting theory based on it.
(c) The proposed algorithm could automatically reach the best result of trying DANN on each layer.
Negative points:
(a) There is no proof that this new bound is better than classic domain adaptation theory (Ben-David et al., 2010). Although this bound involves new insight, the novelty is limited if it is looser than existing upper bound. Furthermore, there are no creative tools in the mathematical proof part, which is a direct extension of the classic theory. 
(b) There is no analysis about the generalization when estimating this upper bound from finite samples. It could be easily seen that the sample complexity of embedding complexity is at least of the same order than classic \mathcal{H}\Delta\mathcal{H}-divergence (Ben-David et al., 2010).
(c) The analysis on the monotonicity of the divergences across the layers is very limited. It will be better if there is a discussion about when the monotonicity is strict.
(d) What is the role of embedding complexity in the algorithm? It seems that only high-level classifier divergence is minimized.
(e) Why minimizing the sum of divergences computed on all layers can control the proposed upper bound? It seems that if the embedding complexity of each layer is a constant, minimize divergence of a single layer can further minimize the minimum. Furthermore, there are previous method that minimizes divergences on all layers [A]. Please give a discussion on this method.
(f)The empirical evaluation is relatively weak. There is no experiment based on convolutional networks, which are widely used on the Digit and Office-31 datasets.

Although the insight is interesting, the novelty of this paper is not enough for being accepted by ICLR. So I vote for rejecting this submission. 

[A] Zhang, Weichen, et al. "Collaborative and adversarial network for unsupervised domain adaptation." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018.

review_title: Official Blind Review #3

Meta-review: This paper studies the impact of embedding complexity on domain-invariant representations by incorporating embedding complexity into the previous upper bound explicitly.

The idea of embedding complexity is interesting, the exploration has some useful insight, and the paper is well-written. However, Reviewers and AC generally agree that the current version can be significantly improved in several ways:
- The proposed upper bound has several limitations such as looser than existing ones.
- The embedding complexity is only addressed implicitly, which shares similar idea with previous works.
- The claim of implicit regularization has not been explored in-depth.
- The proposed MDM method seems to be incremental and related closely with the embedding complexity.
- There is no analysis about the generalization when estimating this upper bound from finite samples.

There are important details requiring further elaboration. So I recommend rejection.
Decision: Reject