Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The paper addresses the problem of out-of-distribution prediction by proposing a causal model which captures the underlying data generation process. The causal model consists of a latent variable which generates both the input images and the associated output labels. The novelty of the paper stems from the fact that the authors split the latent variable into two parts. The first part (s) captures the semantics of the generation process and the second part (v) captures the spurious domain specific variations. Since maximizing the likelihood of the data under this model is intractable, the authors propose an approximate and efficient training and inference mechanism. The authors show theoretically that their model will learn semantic features which are responsible for the final prediction. The claims are backed by a collection of experiments on simple datasets.  

main_review: The paper seems to be technically sound and the claims are backed up by the experiments. The generative model proposed in the paper is fairly intuitive and well motivated. The ideas are reasonably novel, though incremental. I do have a couple of objections with the way the paper currently stands: 

1. The paper is very poorly written and it is extremely difficult to follow the arguments and the motivations behind the arguments. The authors really need to revise the presentation of the paper to make it easier to understand before it is accepted. 

2. It is not clear what is the full benefit of splitting the latent variable z, which is generating the x and y. While the authors claim to show some results of the ablation study in the appendix by combining the two variables, the error bars of the two methods have a significant overlap (especially on the non MNIST datasets). 

3. There are no insights provided around what semantic features is the model learning on the ImageCLEF-DA and PACS datasets. 

4. The experimental section backing up the claims of out-of-domain generalization is somewhat weak. The images in the ImageCLEF-DA datasets are not very different visually. In the PACS datasets the images are fairly different however pooling across three domains to create the training set is bound to make the task easier. It would be interesting to see the performance of the model on one domain when trained on images only from a single domain. 

Originality
The ideas in the paper, although somewhat incremental, are fairly original. 

Quality
The research presented in this paper is of good quality however the presentation of the research is extremely poor. 

Clarity
The paper is rather poorly written and extremely hard to follow. Furthermore it is certainly not a self contained paper with a lot of important details left to the appendix.  

Significance
I believe more thorough empirical evidence is required to infer the true significance of this approach. 


limitations_and_societal_impact: The paper does not have any concerns around societal impact and they seem to have called out most of the limitations of their model.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes a Causal Semantic Generative model (CSG) for out-of-distribution (OOD) generalization. It assumes a semantic latent variable $s$ and a variation latent variable $v$, following the causal graph that $s,v$ are the causes of the input $x$ and $s$ is the only cause of the label $y$. The prior $p(s,v)$ is domain-specific, but the causal mechanisms $p(x|s,v)$ and $p(y|s)$ are invariant across domains. To learn the model, the authors adopt variational inference by introducing $q(s,v|x)$ and training with ELBO. The authors consider single-domain OOD prediction and domain adaptation. For single-domain, they advocate to use an independent prior p(s)p(v) to eliminate spurious correlation. For domain adaptation, a new prior is learned on the target domain.

The authors provide theoretical analysis, showing that under additive noise assumption, the ground-truth CSG is identifiable, and subsequently the OOD generalization error and domain adaptation error are bounded. Experiments are conducted on Shifted MNIST, ImageCLEF-DA and PACS datasets. The proposed model achieves superior performance compared to CNBB in single domain and DANN, DAN, CDAN in domain adaptation.
main_review: Causal learning and OOD generalization are very important problems. This paper proposes a novel model that's well motivated, theoretically justified, and empirically achieves good results. I'd like to recommend acceptance.

The paper is well-written and I enjoyed reading it. I just found two typos. Line 203 "use $q(s, v|x)p(y|x)$ as $q(s, v, y|x)$", should be $p(y|s)$ instead of $p(y|x)$. Line 321 "it is scales larger", remove "is".
limitations_and_societal_impact: Limitations and potential negative societal impact are not discussed in the paper.

As the proposed CSG is a generative model, if you can visualize the effects of $s$ and $v$ on $x$ and $y$, it will help understand what the model has learned.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The authors propose a specific graphical model, Causal Semantic Generative model, which is based on the principles of casual reasoning. The graphical model attempts to explicitly distinguish between meaningful semantic factors of variation and spurious correlations. The design of the model is targeted at out-of-domain generalization and domain adaptation performance. The authors describe an approach for training such a model using variational inference, and demonstrate theoretically and experimentally that their approach gives superior OOD generalization and domain adaptation performance compared to relevant baselines.
main_review: Originality: While latent variable models aiming to separate out some form of content and style have been proposed before, the specific model proposed by the authors along with in-depth intuitive and formal justification represents a novel contribution.

Quality: The authors make theoretical claims about the properties of their model, and experimental claims about the value of their formulation. As far as I can tell their theoretical claims are well supported.

However, I feel their specific experiment claims become lost in jargon. Specifically, the authors make a number of strong claims based on intuition (e.g. lines 68, 72-73, 79-80, 83, 100-101, all of section 3.2). These are central claims to the justification of the authors’ model, and are made repeatedly and strongly, but I don’t feel that the experimental evidence is commensurate with the strength of the authors’ claims. For example, there are a number of works the authors do no cite that outperform their method on ImageCLEF-DA or PACS (e.g. Wang, Qian, and Toby Breckon. "Unsupervised domain adaptation via structured prediction based selective pseudo-labeling." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 34. No. 04. 2020, Huang, Zeyi, et al. "Self-challenging improves cross-domain generalization." Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16. Springer International Publishing, 2020.). Similarly, despite making claims about the shortcomings of many other works (lines 66-67, 74-75, 81-85), the authors do not compare to these models. 

If the authors' claims are solely supported by empirical evaluation, then they should be evaluated against actual state of the art methods. Moreover, if the authors are going to make such strong claims about the shortcomings of many other models they need to demonstrate that their approach is superior. Instead of pure accuracy results, these claims could alternatively be supported by analysis showing that the representations learned by their method do indeed separate out true signal from spurious signal better than competing methods. But they do not provide any such evidence as it stands.

Clarity: The organization of the paper is strong, although as a reader I felt there was a large amount of unnecessary language and appeal to intuition which hides what the authors actually did and how they evaluated their hypotheses. Section 3.1 has a lot of this in my opinion, with the e.g. very formal Principle 2 and section 3.2.

Significance: Learning models that are robust to domain shift is one of the major open challenges in machine learning right now in my opinion, and methods that attempt to improve this robustness are very much worthwhile.


Overall this paper presents an interesting model along with theoretical analysis and a small amount of experimental validation. The model itself and theoretical analysis may merit publication on its own, but I am concerned with the amount and strength of claims made by the authors in proportion to the evidence for their claims. As a reader I felt like formality and jargon were used as a tool to convince me of the correctness of claims, instead of evidence. It would not take a huge amount of experimental evidence or experimentation at a large scale to push me to accept, and in fact it may be as simple as toning down the unfounded rhetoric, but as written I do not feel the authors have supported their claims with good-faith experimentation.
limitations_and_societal_impact: The authors do not discuss the limitations of their work.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper focuses on addressing the domain generalization problem through the lens of causality. Instead of learning environment-invariant representation/inference p(z|x) like [31,97], the paper proposes to learn the other direction p(x|z). The paper proposes a novel graph, where it assumes the latent semantic-object representation (s) and domain-specific representation (v) cause the image. The generative mechanism works (s,v) -> x, while the prior p(s, v) can be prone to environment change at test time.   Therefore, the model proposes a CSG-ELBO to maximize the likelihood of E_{p(x, y)}[log p(x,y)], and further derive a CSG-ind to disentangle v from s. The paper conducts three experiments on different datasets for both OOD generalization and domain adaptation to show the advantage of their proposed model.
main_review: Pros:
1. I think the causal graph proposed to estimate (x -> y) is novel. Instead of directly maximizing p(y|x), the paper proposes to model the causal direction (s,v->x) and uses the variational method to approximate the original objective function. 
2. The paper establishes a theory for the identification of the semantic factor (cause of prediction) and subsequent merits for OOD generalization and domain adaptation.

Cons:
1. The paper is somewhat elusive. Maybe because I'm not an expert in Causality/Domain-Adaptation area, it took me very long to understand section 3. It would be good to make this paragraph more understandable. Due to the clarity issue, I'm not convinced of the design of the proposed CSG. 
2. The used datasets are somewhat toy and not quite representative in the field. Is there any chance to apply your method on ColoredMNIST or Office to see its performance?

Questions:
1. In line 166, why do you suggest p(x|z) is nearly deterministic and invertible? Are you suggesting that once we know the object position z, we can deterministically generate an image x? While in line 173, you suggest that multiple z can lead to the same x, therefore, identifying z from a single x is difficult, which seems to contradict the original assumption. 
limitations_and_societal_impact: Yes
review_title: null

Meta-review: This paper attempts to robustify models learned on a particular dataset to OOD examples which could plausibly be addressed if the model learns based on the semantics of training observations, rather than spurious factors. The proposed solution is—inspired by recent work in causality—to introduce a pair of latent variables modelling the semantic relationship between inputs and labels, and the natural spurious variations/noise in the input, respectively. Neural variational inference is used to jointly learn this generative model. Experiments show it works on some relatively toy data, which is arguably complex enough to prove the concept.

I personally like the idea, and would like to have seen a richer discussion here. I have read all the reviews and rebuttals carefully, it is seems like the main issues had to do with clarity, the strength of some of the claims in the paper, and the appropriateness of comparisons and datasets used. I feel in the rebuttals the authors have responded to the latter two points to my satisfaction, and if the concerned reviewers are not satisfied with those rebuttals, they have not made this known to me. The authors have committed to reviewing the strength of some the assertions about the approach in their rebuttal, and I count on them to make good on this in the preparation of their camera ready. Regarding clarity, I did find section 4 a bit hard to read due to the density of mathematical expressions. My suggestion would be to summarise the findings of section 5 in the main paper, move the rest of section 5 to the appendix, and expand section 4 to give a little bit of visual breathing room and more detail where needed. I don't think this sort of change warrants asking for further reviews, and believe this paper will be of interest to NeurIPS attendees. I recommend acceptance.
Decision: Accept (Poster)