Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper studies the problem of domain generalization via invariant feature learning studied from the lens of information theory. The motivation is to use rate distortion theory to obtain a domain-invariant representation. The authors propose an algorithm titled Twins that uses the aforementioned rate distortion theory coupled with a multivariate Gaussian assumption to provide an additional penalty based on cross-correlation, i.e., minimize the distances between pairs of points belonging to different domains but the same label. The authors demonstrate, that under suitable assumptions, this approach will recover the invariant feature representation. The authors construct several hand-crafted problems where indeed this assumption (and the proposed algorithm) work well, and also provide some experiments on MNIST-type data.
main_review: Strengths:
- The paper is well-written and presents the central contributions clearly.
- The authors list the assumptions in the data-generating process and the algorithm (Section 3) quite clearly.
- The approach is developed following a consistent and a (mostly) rigorous line of argumentation, with some theoretical guarantees on convergence under suitable assumptions.

Weaknesses:
- In my opinion, while this work presents some good arguments, its central contributions are not that distinctive from prior work. Minimizing some notion of cross-domain covariance has been studied extensively in both the machine learning and computer vision communities, to varying levels of success (see, e.g., https://openaccess.thecvf.com/content_cvpr_2018/papers/Li_Domain_Generalization_With_CVPR_2018_paper.pdf, https://www.sciencedirect.com/science/article/abs/pii/S003132031930425X for practical examples, and the papers from Samory Kpotufe's group, e.g., https://arxiv.org/abs/1803.01833 that study this in the context of domain adaptation). The central idea is not really inspiring, albeit the connections to rate distortion theory are interesting. Furthermore, it has been shown recently that for real-world problems of domain generalization, it is very hard to beat naive ERM [https://arxiv.org/abs/2007.01434] which makes me wonder whether the assumptions in this paper (and more generally, in a lot of invariant feature research) really hold in practice.
- This brings me to my second point: The contributions within this paper (both theoretical and the experimental results) do not really further any insight into the domain generalization problem IMHO. The assumptions the authors make are far too simplistic for real-world settings (e.g., zero-main Gaussian feature distributions, which are patently not true for image classification that regularly exhibits long-tailed label distributions), and hence do not provide a complete picture, and can also be at odds with prior work. The experimental results are on handcrafted problems and toy datasets, which again, are too simplistic (and in the case of handcrafted problems, not generalizable to real-world problems). The problem of domain generalization suffers specifically from this issue, where a lot of the proposed work is valid only under certain restrictive assumptions, and usually fail when deployed in practice. It would be nice to see if these assumptions were validated by experiments on data that is not carefully curated and drawn from real-world problems.
summary_of_the_review: In summary, I recommend a weak reject. I am quite familiar with the related work in this area and I believe that the authors should put in more effort to identify where their approach fails in the real-world and subsequently see how they can relax assumptions made in the paper by analyzing these failure cases. Domain generalization research has seen a rapid increase in methodological work that propose various lenses to look at the problem, however, it is hard to disambiguate and see the value of these approaches when run in the real-world.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a method for domain generalization. They formulate the problem as a rate-distortion problem and introduce the information bottleneck penalty. Specifically, they realize the objective as a contrastive loss that encourges embeddings across domains to be similar if they share the same label. In the experiments, the method improves the corresponding baselines in several toy datasets.
main_review: - It is good that the paper provides the theoretical gurantee of the algorithm for linear binary classiciation.

- After several approximations, the actual contrastive loss formulation in Eq.(14) is adopted. This formulation is closely related to some existing works [A,B] that also employ the constrastive loss for domain generalization. Since they share the similar idea, proper discussion and comparison with them will be necessary.

- The batch construction method and the batch size will directly affect the loss Eq.(14) used for the training. For example, the amount of hard examples included in the contrastive pairs will depend on these choices. It would be good to include the experiemtnal analysis as well as discussion on these factors.

- It is good that the proposed method consistently improves the corresponding baseline methods in two experiments, linear unit tests and mnist-type datasets. However, I have concern that evaluation on more realistic datasets is missing. As the method assumes Gaussian bottleneck which could be a over-simplification for real world problems, I think evaluation on other datasets, such as VLCS, PACS, and OfficeHome, in domainbed would be helpful to show the generalization performance of the method.

- How does the hyperparemter mu affects the accruacy? Analysis of the effect of hyperparameter is missing.

[A] Feature Stylization and Domain-aware Contrastive Learning for Domain Generalization, ACM MM 2021.
[B] SelfReg: Self-supervised Contrastive Regularization for Domain Generalization, ICCV 2021.
summary_of_the_review: The paper proposes an interesting approach based on the information bottleneck. However, I think the discussion and comparison with related work on more realistic datasets is necessary.

---  
After authors' response  
I appreciate the authors' feedback. They provided additional results on more realistic datasets (PACS and OfficeHome) and results for different $\mu$. After reading the feedback as well as other reviews, I would like to keep my score. As also mentioned by other reviewers, the theoretical guarantee of the algorithm is based on quite strong assumptions, which do not likely hold under practical conditions, and I am a bit skeptical about its significance.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Authors address a problem of domain generalization using  information bottleneck method. Authors assume that learning domain invariant features is a key to solve the domain generalization problem and then prove that  information bottleneck penalty guarantees that domain-invariant features can be learned. 
main_review: Weakness: 

1) Authors strongly suggest that learning domain invariant features is the only way to solve the domain generalization problem. They prove that  information bottleneck penalty guarantees that domain-invariant features can be learned which in turn guarantees that there is a solution to the domain generalization problem. This may not be completely correct as authors missed out on some of the literature. 

2) Weak literature survey. Please cite the papers [1-6] that may be missing from the current paper. 

3) Weak experiments section. 

Strength: 

1) Clear assumption and claims (even through they are strong in some ways, clarity in writing helps understand what authors are trying to convey). 

2) Strong justification on why their method could work. 

3) Experiments with synthetic data for better understanding. 

Main Review:

1) “Domain generalization deals with the difference in the distribution between the training and testing datasets, i.e., the domain shift problem, by extracting domain-invariant features”: This is not correct. Extracting domain-invariant features is just one way to address the domain generalization problem. 

2) “The literature on domain generalization aims to learn the features that are invariant across multiple training domains, and to use those features for out-of-distribution generalization.”: Again not correct. Please look at section 2 of Li et al 2019 [1] to get an idea of some non domain invariant learning methods. You could also take a look at [2-5]. These references also should be cited in the current paper.  

3) Assumption 1 and invariant feature map: In “Domain Generalization by Marginal Transfer Learning”, there is no assumption like assumption 1 in the paper. Does this assumption 1 help in anyway to tighten the learning theoretic bound in terms of number of source domains and number of training examples available in the source domains?

4) Assumption 4: “Strictly separable invariant features” This looks like a very strong assumption. What happens if this assumption is not satisfied all the time for all the source domains? 

5) The experimental section is not complete as it does not contain most datasets that recent papers use: For example, PACS or VLCS datasets are missing [6]. Can authors get results on one of these datasets at least? 
 


[1] Li, Yiying, Yongxin Yang, Wei Zhou, and Timothy Hospedales. "Feature-critic networks for heterogeneous domain generalization." In International Conference on Machine Learning, pp. 3915-3924. PMLR, 2019.

[2] Deshmukh, Aniket Anand, Ankit Bansal, and Akash Rastogi. "Domain2vec: Deep domain generalization." arXiv preprint arXiv:1807.02919 (2018).

[3] Niu, Li, Wen Li, and Dong Xu. "Multi-view domain generalization for visual recognition." In Proceedings of the IEEE international conference on computer vision, pp. 4193-4201. 2015.

[4] Blanchard, Gilles, Aniket Anand Deshmukh, Ürün Dogan, Gyemin Lee, and Clayton Scott. "Domain Generalization by Marginal Transfer Learning." arXiv preprint arXiv:1711.07910 (2017) J. Mach. Learn. Res. (JMLR) 22 (2021): 2-1.

[5] Duan, Lixin, Ivor W. Tsang, and Dong Xu. "Domain transfer multiple kernel learning." IEEE Transactions on Pattern Analysis and Machine Intelligence 34, no. 3 (2012): 465-479.

[6] Li, Da, Yongxin Yang, Yi-Zhe Song, and Timothy M. Hospedales. "Deeper, broader and artier domain generalization." In Proceedings of the IEEE international conference on computer vision, pp. 5542-5550. 2017.

summary_of_the_review: Authors should address the weaknesses of the paper. Improved literature survey and comparing/contrasting with suggested papers will help improve the score. Stronger experiments section can further improve the score.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: In this work, the authors study the problem of domain generalization. Recent works such as invariant risk minimization [Arjovsky et al.] have invigorated a lot of interest in the problem of domain generalization. The paper brings together ideas from rate distortion and information bottleneck principle (recently used in Ahuja et al. for domain generalization) and the work of model based domain generalization [Robey et al.]. 
In Ahuja et al., the authors combine invariance and information bottleneck to address domain generalization. The representation learned in Ahuja et al. is constrained by the label and information bottleneck constraints. In this work, the authors add some more structure to the generation of the observations X from the latent and say that there is a common instance X that is transformed by the domain transformation model to get the image in the current domain, i.e. X^e = G(X,e). The authors try to recover the common instance X that is shared across domains. 


main_review: **Strengths:** The problem addressed in the paper is very important. It is a good idea to combine the ideas introduced in model based domain generalization [Robey et al.] and the ideas of information bottleneck and invariance [Ahuja et al.].  The experiments show some promise. The paper is easy to read as well. 


**Weaknesses:** I highlight concerns with the paper below in a pointwise fashion

1. **Access to X to prove theorems makes theorems vacuous** The authors theory assumes access to domain invariant label. If we already have access to X to prove the theorems, the theorems are vaccuous because you can just use X to learn the predictor in any domain and that should work fine. I will describe a data generation process to further my point. 

   $X^e \leftarrow G(X,e)$

   $Y^e \leftarrow f(X) + N $
  
  In the above model the whole difficulty arises because we have access to $X^e$ and not $X$. If we have access to $X$ and have $f$ in the space of functions that the learner searches, then with enough data $f$ would be learned given $X$ as input. Therefore, the theorem in the current version do not make much sense.

2. **Proxy for X is not reliable** The authors for the experiments do not assume $X$ is available and instead rely on a proxy. The proxy is that for an instance $X^e$  the authors use an alternate $X^{e'}$ from a different environment but with the same label. If the authors are borrowing the idea from Robey et al. already why do they not use the $G$ function from [Robey et al.] and use the domain invariant $X$. Take the $G$ model from Robey et al. for each $X^{e}$ find the alternate $X^{e'}$ using their model. Table 1 in [Robey et al.] very clearly illustrates what I am saying. Using the learned model $G$ seems a much more appropriate proxy and not just any point from another domain with the same label.

3. **Regarding experiments** I would encourage the authors to modify their method to incorporate $G$ model in it. However, if the authors insist that their approach is already working and robust, then some more experiments and justification (ablation studies for the same are needed).  Take datasets such as Terra Incognita, Camelyon etc. used in DomainBed and WILDS repository and show that the idea continues to work in more diverse settings. 


summary_of_the_review: Overall, the paper proposes an interesting approach to combine information bottleneck and model based domain generalization ideas recently used in [Ahuja et al.] and [Robey et al.]. However, there are severe shortcomings in the paper, the theory seems vacuous under assumption of access to $X$ (authors should please correct this). The proxy for $X$ is not reliable and instead $G$ model from Robey et al. should be used to get good results in large scale settings.
review_title: null

Meta-review: Casting domain generalization as a rate-distortion problem and developing an information-theoretic approach to solving it looks like an interesting idea. While the proposed method is technically sound, the assumption made in the proposed method is too strong to hold in real-world applications. Though in the rebuttal the authors provided additional experiments on two benchmark datasets, reviewers' concerns about the strong assumption made in the proposed algorithm still remain. To address this issue, I think besides conducting more extensive experiments, the authors also need to analyze when the assumption does not hold in practice, why the proposed algorithm could still perform well compared with other domain generalization methods.

In summary, this is a borderline paper below the acceptance bar of ICLR.
Decision: Reject