Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This work focuses on solving domain generalization. Given labeled data of multiple source domains, the goal is to predict the labels of target domain data. In domain generalization, the target domain data are not provided at training time. The main challenge of domain generalization is to learn a domain-invariant representation. To solve the problem, most existing approaches focus on the marginal distribution alignment p(z|d) without considering the conditional distribution p(y|z,d). However, it is needed to align both the marginal and the conditional distributions to classify the target data. To address the challenge, they propose a theoretically grounded method for learning a domain-invariant representation based on domain density transformation functions. They enforce the representation to be invariant under the transformations among the source domains to the representation become more robust under other domain transformations. It aligns both the marginal and the conditional distributions. The transformation functions are based on GANs in practice.
main_review: Strengths
- Most existing approaches only focus on the marginal distribution alignment. They point out the limitation of the previous approaches with an eye-catching illustration.
- They propose a novel approach to align both the marginal and the conditional distributions with a theoretical analysis.
- They empirically show the effectiveness of the proposed method by performing experiments on real-world datasets. The proposed method outperforms the relevant baselines.

Weaknesses
- They need to more clarify the differences between the meta learning and domain generalization, and to compare briefly the existing approaches for meta learning and their approach.
- They need to train a complex network (StarGAN) to align the distributions. The classification performance will be greatly affected by training the network.
- They need to explain the reasons for the performance differences with the key ideas of the baselines.

limitations_and_societal_impact: - They clearly present the limitations of the proposed method.
- The proposed method is heavily dependent on the transformation network performance. Thus, it requires great effort to set up and train the transformation network.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes a novel method, DRIT-GAN, to learn domain invariant representation for domain generalization.
main_review: Originality: The task of domain generalization (DG) is classic. The proposed method leverages the well-known technique, StarGAN. The novelty is limited. 

Quality: Theoretically, I am afraid that the paper is not technically sound. Remarks 1 & 2 are trivial and immediate from the definition. The author claims several prior works used those conditions without noting them. However, I am not clear about the value of remarks 1 & 2 in developing the proposed method. Here are questions to the author:
1. You use remark 1 to justify your assumption that y and d are independent. So how does the assumption of y and d being independent guide your algorithm design? Will your method be broken if y and d are not independent?
2. By remark 2, the author tries to say that enforcing representation z to be independent of domain d is not enough since it only aligns the marginal. However, most prior methods about DG not only enforce 'z' and 'd' being independent. They also use z to predict y. So at least implicitly, they are aligning the conditional distribution. Does the author aware of it?

Then the author put a new assumption, existing invertible transformations between domains, and proposes a method that works under this assumption.
1. How do you validate your assumption? How likely it satisfies in actual practice?
2. I notice the author assumes that the transformations only depend on data x, domain d, but are independent of label y. Can you justify why it is independent of y?

The theorem says the proposed method is valid (correct), but it does not say it is optimal. Theorem gives a sufficient condition but not a necessary one. So I do curious about the optimality of the method. Let's consider the motivating example that the author presented in Figure 1. Does your method work in this case? Can StarGAN learn the correct transformation between two domains? I feel the StarGAN also has the risk of being trapped into the suboptimal as the author described in the introduction and ending with 0% accuracy.

Empirically, the author does a good job of showing performance improvement. Clearly, your method shows an advantage over prior works on three benchmarks. A deeper analysis would be good if the author can shed more light on why the proposed method improves the performance.

Clarity: The paper is well-written. As a reader, I can easily follow it.
limitations_and_societal_impact: As mentioned, the paper has limited depth in theory. The author could try to think about theoretical analysis that bounding your model's error. For example, what if your StarGAN cannot perfectly learn the transformation? How its error of learning transformation translates to the error of your final prediction model?
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: The paper presents a method for domain generalization in the presence of multiple source domains.  Given a pre-trained domain translation model f, the proposed method regularizes ERM by minimizing the l2 distance between  g(x) and its translation in domain d'; g(f_d'(x)) , where g is the feature extractor. 
main_review: Strengths of the paper:
-------------------------------
The idea of using domain translation as an intervention to learn invariant features is interesting. I think this idea could be expanded to include arbitrary transformations for arbitrary domains. 

Weaknesses of the paper:
-----------------------------------
- The authors designate a complete page on using invertible function for domain translation but finally opt for stargan. I don't believe stargan learns invertible functions. 
- shouldn't  the goal is to learn invariant functions for p(y|g(x))? I.e for any domain transformation f, p(y|g(x) and p(y|g(f(x))) should be the same. In other words, why is the regularization done in feature space? A better discussion on this is needed. 
- since we do have access to labels in the source domain, what would happen if we just minimize “dist(g(x_{d1}|y=c), g(x_{d2}|y=c))” where d1 and d2 represent domain 1 and 2. Probably this is a big regularization on the model since this way the model would learn common information in x_d1 and x_d2 which relate to class c and this may not generalize well to other domains but an experiment like this could act as an ablation study and highlight the merits of the proposed approach. 
- the paper lacks comparison with IRM and IRM based methods.



limitations_and_societal_impact: The authors briefly talk about the limitation of their work as having to train an additional domain translation method (STARGAN). 
Another potential limitation of the model could be its ability to deal with label distribution shift. An experiment is needed to verify if that is indeed the case.
review_title: null

Meta-review: This paper proposes an approach to align both the marginal and the conditional distributions with a theoretical analysis. Authors have done a good job to address many concerns from reviewers. After the discussion period, all three reviewers are inclined to accept the paper (note that one increased the score from 4 to 6 without editing the original review.) 
Decision: Accept (Poster)