Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This submission provides a new theoretical framework for domain adaptation. In order to tackle the adaptability term in the classical domain adaptation theory, this submission proposes a new upper bound that enlarge the hypothesis space in the adaptability term. A weighted version of this theory is also given. Authors further support their conclusion by empirical results.

Pros:
1. This submission studies an important problem in domain adaptation.
2. This submission proposes new theoretical insight about compression and adaptability.
3. The conclusions of this paper can be partially proved by the empirical results.

Cons:
1.	As the author says in their future work, the source constraint is too strong that need to control the feature unchanged across all source domain. For this condition is not build on samples but on the support of source domain. It seems that authors use $L_0$ to constrain \phi’ to have same value with \phi on source dataset, which may be only a small part with zero measure of source support set. 
2.	There is no generalization error analysis for these upper bounds. This submission provides weighted version of the main theory in the section 5. It seems that weighted version of upper bound could be further minimized by find a good weight. But add weight will add variance in the complexity term [A].
3.	This submission adds \beta term to change the adaptability term of $\tilde{\mathcal{H}}$ into the adaptability term of $\mathcal{H}_0$. The reason why \beta can be estimated from finite sample is not clarified, which is the premise of being trainable and should be mainly discussed in this paper. We can see that to estimate \beta is a domain adaptation problem under the fact that the labeled functions are same. \beta is a term that can’t be computed from small finite samples if there is no more assumption: It is not easy to approximate $\tilde{f}_s$ uniformly, otherwise the estimation of \beta will suffer from distribution shift. This submission claimed that the term can be trainable by giving Proposition 2, a proof of the consistency. However, this proposition is built on the assumption that there exists a series of \phi minimizing the distribution distance to zero. But this is impossible for finite sample estimation, when there will always be generalization error of estimating the distribution distance. Furthermore, it is usually impossible to make the two embedded distributions completely the same in empirical. In addition, if there is a series of \phi, how to control other terms in the upper bound? Every \phi will induce new $\tilde{\mathcal{H}}$ and $\mathcal{H}_0$ which will change all other terms. In summary, the main theory in this submission changes the unknown adaptability to a new term that is very hard to estimate. And there is no sufficient empirical or theoretical evidence in this paper that could support the fact that \beta is small. The contribution is therefore limited.
4.	The theory also fails to give upper bounds on compression term and adaptability term, or some explicit upper bounds for certain hypothesis spaces as examples. Readers could not have a clear image of how large will these terms be. Furthermore, if the support sets of source and target domain coincide a lot, the adaptability of $\mathcal{H}^s$ will not be too smaller than the previous one.
5.	The organization of the submission makes it hard to read:
a)	The symbol of this submission is chaotic. For example, $\tilde{\mathcal{H}}$, $\tilde{\mathcal{H}}^s$ ,$\tilde{\mathcal{H}}_h$ are defined based on $\phi$, $\pi(h)$ is defined based on $\tilde{\mathcal{H}}$, but all these facts are not revealed in their symbols. 
b)	For clarity, all loss functions defined in Section 4 should be stated in a independent line. The Section 5 should be moved to the front of Experiment part.
c)	I recommend the authors to restate all new defined symbols as a list on the top of appendix. It really troubles me during checking the proof.

I think this submission discusses about an important problem and provides new insight, but it is not a thorough theoretical work because of above reasons. So, I vote for rejecting this paper.

[A] Cortes, Corinna, Yishay Mansour, and Mehryar Mohri. "Learning bounds for importance weighting." Advances in neural information processing systems. 2010.

review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: Summary
-------
This paper presents a revisit of existing theoretical frameworks in unsupervised domain adaptation in the context of learning invariant representation. They propose a novel bound that involves trainable terms taking into account some compression information and a novel interpretation of adaptability. The authors mention also contribution showing that weighting representations can be a way to improve the analysis. 

Evaluation
-----
The ideas are novel and the result brings novel and interesting light on the difficult problem of unsupervised domain adaptation. 
However, the practical interest in terms of applicability of the proposed framework is not fully demonstrated, the properties of the proposed analysis have to be studied more in details and some parts better justified. The experimental evaluation brings some interesting behavior but is somewhat limited. The weighting aspect of the contribution is not supported by any experiment.

Other comments
------------

-I am a but puzzled by the use of the term "compression". This is maybe subjective, but in the context of learning representation, I would have interpreted it as a way to sparsify the representation, and thus compression could then be measured with respect to a given norm (L2?) or another criterion (Kolmogoroff, ...). 

In the paper, the notion of compression is related to a reduction of the hypothesis space after application of a transformation \phi, so I am wondering if using "hypothesis space reduction" would not be more appropriate.
In this case, however, there are maybe links with structural risk minimization that could be investigated here.
A side remark: there is no particular restriction on the space of transformations, we wonder if it would be useful to indicate if all the possible transformations are included as subspaces of a given latent space. Since, to be very general, one can imagine the existence of an unbounded number of transformations that correspond to an increase of the input dimension. For transformations leading to different representations of different dimensions, the way the deduced hypothesis can be compared should also be indicated (for defining properly the inclusion H(\phi_1)\subset H(\phi_2).

On the other hand, the authors seem to need the use of norms over transformations as illustrated in the definition of H_0^\eta in the experimental section. So I suggest that the analysis could be revisited by directly incorporating (representation) norms in the theoretical framework and in particular for defining more properly H_0.

-One weakness of the theoretical framework is for me the lack of definition of H_0 in Section 3. We just know that it is included between two classes of hypothesis of interest, but there is no clear characterisation of H_0 which makes the analysis fuzzy: we have a bound that involves an object without any clear definition and it is for me difficult to really interpret the bound. Trying to define H_0 with some restrictions related to the norm of the transformations, as evoked before, could be a way to address this point (and actually the way the experiments are done tend to confirm this point).

-Another weak point is the lack of qualitative analyse of the bound in Inequality 3 (the same applies for Inequality 5). I would have appreciated if the authors could provide an analysis similar to the one of (Mansour et al., COLT 2009) - it is cited in the paper - when they compared their result to the one of (Ben-David et al., 2007). For example, what happens when source is equal to the target, when is the bound significantly loose, significantly tight, different from other existing results, ...

In particular, if we compare the bound with the one of Ben-David et al. (we can also consider the one of Mansour et al.), there is two additional term, one is weighted by a factor 2, another one involved a supremum and one can think that this bound is rather loose and does not provide any insightful information and said differently it could not give a strong framework for practical considerations.
I may understand that when the bound is tight we could deduce that the compression term is low, but finding cases leading to a tight interesting bound does not seem obvious.

-The experimental evaluation presents some expected behavior in the context of the bound, but I miss a real study trying to make use of the proposed framework to do adaptation in practice with comparisons to other strategies.
Additionally, having additional studies with other models and tasks will probably reinforce the analysis.

-At the beginning of Section 3.2, the authors mention that they restrict their analysis to the square loss, however I think the analysis is true for larger class of losses with more general properties. In the experimental evaluation, the cross entropy is used, so I think that the experimental evaluation should also be consistent with the theoretical analysis by considering the square loss.


-Paragraph below Definition 5 is unclear: the notion of L2 norm has not been introduced in this context, so the message of the authors is a bit unclear.

-I do not find the notation \gamma(\phi,H) appropriate, I woud rather suggest to use \gamma(H\cdot \phi)

-The biblioggrgaphy can be improved by adding the right conferences/journals where the papers have been published in addition to the ArXiv reference.

review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: This paper introduces the compression risk in domain-invariant representations. Learning domain-invariant representations leads to larger compression risks and potentially worse adaptability. To this end, the authors presents gamma(H) to measure the compression risk. Learning weighted representations to control source error, domain discrepancy, and compression simultaneously leads to a better tradeoff between invariance and compression, which is verified by experimental results.

The paper presents an in-depth analysis of compression and invariance, which provides some insight. However, I have several concerns:
* In Section 4, the authors propose a regularization to ensure h belongs to H_0. How is the regularization chosen? How does it perform on other datasets? Experimental results only on digit datasets are not convincing.
* In Section 5, the authors introduce weighted representations to alleviate the curse of invariance. However, they do not provide experiments to validate their improvement. 
* The organization of this manuscript is poor and difficult to follow. Starting from Section 3, the authors use several definitions to introduce their main theorem. However, these definitions are somewhat misleading. I cannot get the point until the end of Section 3. Besides, the notations are confusing, so I have to go back to the previous sections in case of misunderstanding.




review_title: Official Blind Review #2

Meta-review: This paper provides a new theoretical framework for domain adaptation by exploring the compression and adaptability.

Reviewers and AC generally agree that this paper discusses about an important problem and provides new insight, but it is not a thorough theoretical work. The reviewers identified several key limitations of the theory such as unrealistic condition and approximation. Some important points still require more work to make the framework practical for algorithm design and computation. The presentation could also be improved.

Hence I recommend rejection.
Decision: Reject