Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The paper provides a rigorous theory for domain generalization (DG) and multi-source domain adaptation (MSDA). It presents two bounds on the target domain risk of some classifier. The first one suggests that obtaining a general domain invariant (DI) representation can be achieved via minimizing a weighted sum of source domains' risks in the DG setting and that the DI representation should also align the target domain's feature marginal distribution with that of the target for MSDA. The second one requires, in addition, an alignment of all the source domains' feature marginal distribution, resulting in compressed DI representations. The characteristics of both types of DI representations are thoroughly theoretically examined with the advantages and drawbacks of each one being discussed: while the first one increases the span of the class-conditionals of the source mixture, the second guarantees a tighter generalization gap for learning on the mixture of sources. The last theoretical aspect addressed is the effect of a class imbalance between the considered mixture of source domains and the target domain, which generalizes previous work on the drawbacks of learning a DI representation.  Finally, empirical evaluation is carried out to support the theoretical findings.
main_review: The paper is well written, easy to follow, and presents important theoretical findings that help to advance the understanding of both domain generalization and multiple source domain adaptation.

## Strengths

* The theoretical results clearly help to understand both the problem of domain generalization and multiple-source domain adaptation.  It rigorously clarifies the properties of each method of obtaining a DI representation in the case of learning with multiple sources. The bounds in terms of the Hellinger distance are novel to the best of my knowledge, and the loss function is not required to verify the triangle inequality, which corresponds better to the classification setting. 
* Theorem 8 is surprising when looking at Theorems 1 and 3: in these latter two theorems, the non-estimable quantity in the MSDA setting is the maximum expected disagreement between a source labeling function and that of the target. It does not depend on the feature map $g$, so one would expect that it is safe to learn a DI representation (indeed, the result analogous negative result from [3] is intuitively motivated by the fact that the ideal joint error term may increase due to mapping the data via $g$). Despite this, Theorem 8 proves the limit of DI representation learning whenever the class marginals change between the sources mixture and the target.

## Weaknesses

* Although [1] figures in the references (reference [50]), it is not cited in the paper, whereas it is a method that should be compared to at least theoretically, as it tackles multiple source domain adaptation after establishing a theoretical bound on the target domain risk. It should also be mentioned in Section 4 "Related Work".
* The bounds suggest an alignment in the sense of the Hellinger distance. However, in practice optimization, the hard constraint $\mathbb P_g^T = \mathbb P_g^\pi$. Using another distance $D$ int the optimization problem after line 136 would be better justifier if a bound $d_{1/2}(\mathbb P_g^T, \mathbb P_g^\pi) \leq D(\mathbb P_g^T, \mathbb P_g^\pi)$ was given for example.

## Comments
* The $\gamma_{i,c}$ term (line 191) was not defined.



## Suggestions
* A summary table defining each type of DI representation with its associated constraints and optimization problem would further facilitate understanding the paper.
*  Indicating that the second term of the RHS of Equation (3) is just a constant to ensure the non-negativity of the divergence would facilitate reading.
* It would be better to have a number for the optimization problem just after line 137.


## Questions
* Can the infinite capacity assumption of Theorems 5 and 6 be weakened? In particular, can it lead to a weak form of divergence that can be null even with different distributions, but that is useful for classification (as it is the case with the $\mathcal H \Delta \mathcal H-$divergence [2])? 
* Line 189: "Intuitively, for each domain $\mathbb D^{S,i}$, the minimal loss $\min_{\hat h \in \mathcal H}\mathcal L(\hat h, h^{S,i}, \mathbb P_g^i)$ should be inversely proportional to the divergence between the class-conditional $\mathbb P^{S,i,c}_g$ ". Is it possible to develop more on why this is intuitive (and true)?
limitations_and_societal_impact: Limitations of the work were not addressed. Societal impact also was not as the paper is theoretical.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes a theoretical foundation for learning domain invariant representation.
Their theory applies to both multi-source domain adaptation (MSDA) (a generalization of the
classical domain adaptation) and domain generalization (DG), the setup where the target
domain is unknown during training. It proposes two views of the domain invariant representation learning: a *compressed*
representation where the source domains are regularized to map onto the same embedding, and a *general* where the source
domains are not regularized to map onto the same embedding. The paper studies the trade-offs between these two views
both theoretically and empirically.
main_review: The paper proposes an interesting perspective to domain invariant representation learning.
In general, the contribution of this work is more theoretical in that it bridges methods already
present in the literature. This paper also presents the trade-off between the two approaches explored.
While I enjoyed reading the theoretical perspective, the empirical exploration is unconvincing.

Below, I list some of my concerns for the authors to address.

### Questions/comment
* You claim that the task of domain invariant representation learning becomes significantly challenging for MSDA. It is not clear to me why that is true.
* In Figure 2a. The trade-off is not clear. It would be better to have a confidence interval over, e.g. 10 runs to assess that the variation is not due to noise.
* Similarly for Figure 2b and Figure 3a. there does not seem to be a benefit to using general DI, except in the first 10k iterations.
* Please specify if you are in the MSDA or DG setting in your experiments.

========
## Post-rebuttal: 
I thank the authors for taking the time to answer my questions and the other reviewers'. After reading the reviews and the rebuttal, I am confident to raise my score from 5 -> 6.
limitations_and_societal_impact: The authors did not discuss some limitations of their work, nor the societal impact.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: This paper studies multi-source domain adaptation (MSDA) and domain generalization (DG). The authors first propose two types of invariant representations in MSDA and DG, and then analyze their benefits and drawbacks theoretically. Furthermore, the authors verify the results with experiments.


main_review: MSDA and DG are both important problems in transfer learning. The paper aims at understanding the role of invariant representations in MSDA and DG theoretically. The authors state their results and implications clearly. 

However, I have some questions about the proposed theory:
First, although the upper bounds involve multiple source domains and feature learning, it is of the same spirit as [3]: the target accuracy can be bounded with source accuracy, distribution distance and label shift. (e.g. Theorem 1 of [3]). The extension of [3] to multi-source setting has also occurred in previous works. The lower bounds are also simple extensions of [49] (with some triangle inequalities).

Second, the two kinds of invariant representations seem not complementary. The “compressed” version is another form of the “general” version with stricter constraint. Theorem 3 is a direct corollary of Theorem 1. Therefore, a representation satisfying the conditions of Theorem 3 (the compressed DI) must satisfy those of Theorem 1 (the general DI). 

Another question is the analysis in Section 2.3.3. Since we only have access to finite samples from the source domains, we can only constrain the empirical version of the feature distribution. Thus, Theorem 7 should take into account the gap between the empirical and population versions of distribution distance. Also, if we use dataset S to minimize the loss of \hat h on S, we should use uniform convergence or similar results to deal with the generalization gap. 

Post-rebuttal:
I appreciate the endeavor that the authors made to address the concerns. The authors further clarified the difference between the proposed upper bound and previous works. Nonetheless, the following concerns are not fully addressed, and I decide to keep my score.

1. I admit that the algorithms of the two kinds of DI representations are different. However, since Theorem 3 is a direct corollary of Theorem 1 by applying Lemma 4 in the appendix, the “compressed” version is another form of the “general” version with stricter constraint, and it is quite well known to tune the constraint carefully in domain adaptation algorithms (such as carefully tune the $\lambda$ in line 293 of the main paper).

2. As to the choice of distance, I do admit the Hellinger distance can be applied to the case where KL divergence is infinity. However both of them belong to the F-divergence and satisfy the data processing inequality (line 229 of the appendix). It is quite straightforward to extend Zhao et al. 2019 to get the multi-source lower bound, since it's also based on the data processing inequality. 
limitations_and_societal_impact: The paper does not describe the limitations and potential social impact.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes new theorems for transfer learning, particularly on multi-source domain adaptation and domain generalization. For domain-invariant representations, authors propose two variants: general DIR and compressed DIR. Experiments have shown its effectiveness.
main_review: Domain-invariant representation: is domain-invariant representation enough for learning transferrable representations? Reference [49] pointed out that DIR is not enough. Then, can you show the theorems in this paper can generally work in MSDA and DG settings by only considering DIR?

Experiments are only based on MNIST and PACS datasets, which are not enough. Plus, there needs to be comparisons with existing methods, particularly with DANN to show that the theorem actually works.

Finally, the theorems are based on certain assumptions. Can you show the bridge between the assumptions with real datasets? When does these theorem work and when doesn't?
limitations_and_societal_impact: No negative societal impact
review_title: null

Meta-review: This paper studied new theories on multi-source domain adaptation (MSDA) and domain generalization (DG), where two types of invariant representations in MSDA and DG were proposed and analyzed. Reviewers believed that the problems are interesting and the current study is worthwhile to the transfer learning community. However, they still hold some important suggestions for improvement: i) The compressed DI is a special case of the general DI, which does not introduce new surprising insight; ii) More importantly, the multi-source lower bound can be directly extended from Zhao et al. 2019, both falling into the F-divergence; iii) The detailed connection between the theories and existing algorithms is only conceptual, but there is no term-to-term correspondence, hence the gap between them may still be large; iv) The empirical evaluation is relatively shallow, and many recent methods such as DRO and CORAL are not compared with.

AC does hold a high preference of further research into the underlying theoretical aspects of domain adaptation and generalization. After reading the paper and comparing it with Zhao et al., I found that this paper is in its current form not essentially different from existing work. Note that I'm very familiar with Zhao et al., and their conclusions are well-known in our literature. In addition, the paper does not introduce novel algorithms but is limited to explain existing algorithms. As a consequence, the gap between theory and algorithms still remains, and our desire for a truly effective DG algorithm is still far away (as pointed out by the DomainBed work) -- even with this paper published. So authors are highly encouraged to improve their work towards this direction.
Decision: Accept (Poster)