Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: **Summary**
The paper focuses on the causal perspective of domain-generalization and domain adaptation setup for images. I.e. classifying an image under some distribution shift at test time. Similar to previous work [1-4], it assumes that some latent semantic-object representation (s) and semantic-domain representation (v) cause the image, and that these causal (generative) mechanisms (s,v ->x ) are stable, while their prior p(s,v) is prone to change at test time. It develops a new variational approach to estimate the generative distributions, and test the approach on two datasets for domain-generalization and domain-adaptation.

Overall, the paper suggests a novel approach and theory to an important problem. Its major weaknesses are in its clarity and the experimental part. 

**Strong points**
Novelty: The paper provides a novel approach for estimating the likelihood of p(class|image), by developing a new variational approach for modelling the causal direction (s,v->x).
Correctness: Although I didn’t verify the details of the proofs, the approach seems technically correct. Note that I was not convinced that s->y (see weakness)

**Weak points**
Experiments and Reproducibility:
The experiments show some signal, but are not through enough:
• shifted-MNIST: it is not clear why shift=0 is much better than shift~$N(0,\sigma^2)$, since both cases incorporate a domain shift
• It would be useful to show the performance the model and baselines on test samples from the observational (in) distribution.
• Missing details about evaluation split for shifted-MNIST: Did the experiments used a validation set for hyper-param search with shifted-MNIST and ImageCLEF? Was it based on in-distribution data or OOD data?
• It would be useful to provide an ablation study, since the approach has a lot of "moving parts".
• It would be useful to have an experiment on an additional dataset, maybe more controlled than ImageCLEF, but less artificial than shifted-MNIST.
• What were the ranges used for hyper-param search? What was the search protocol?

Clarity:
• The parts describing the method are hard to follow, it will be useful to improve their clarity.
• It will be beneficial to explicitly state which are the learned parametrized distributions, and how inference is applied with them.
• What makes the VAE inference mappings (x->s,v) stable to domain shift? E.g. [1] showed that correlated latent properties in VAEs are not robust to such domain shifts.
• What makes v distinctive of s? Is it because y only depends on s?
• Does the approach uses any information on the labels of the domain?

Correctness: I was not convinced about the causal relation s->y. I.e. that the semantic concept cause the label, independently of the image. I do agree that there is a semantic concept (e.g. s) that cause the image. But then, as explained by [Arjovsky 2019] the labelling process is caused by the image. I.e. s->image->y, and not as argued by the paper. The way I see it, is like a communication channel: y_tx -> s -> image -> y_rx. Could the authors elaborate how the model will change if replacing s->y by y_tx->s ?


**Other comments:**
• I suggest discussing [2,3,4], which learned similar stable mechanisms in images.
• I am not sure about the statement that this work is the "first to identify the semantic factor and leverage causal invariance for OOD prediction" e.g. see [3,4]
• The title may be confusing. OOD usually refers to anomaly-detection, while this paper relates to domain-generalization and domain-adaptation.
• It will be useful to clarify that the approach doesn't use any external-semantic-knowledge.
• Section 3.2 - I suggest to add a first sentence to introduce what this section is about.
• About remark in page 6: (1) what is a deterministic s-v relation? (2) chairs can also appear in a workspace, and it may help to disentangle the desks from workspaces.
	
[1] Suter et al. 2018, Robustly Disentangled Causal Mechanisms: Validating Deep Representations for Interventional Robustness
[2] Besserve et al. 2020, Counterfactuals uncover the modular structure of deep generative models
[3] Heinze-Deml et al. 2017, Conditional Variance Penalties and Domain Shift Robustness
[4] Atzmon et al. 2020, A causal view of compositional zero-shot recognition





**EDIT: Post rebuttal**

I thank the authors for their reply. Although the authors answered most of my questions, I decided to keep the score as is, because I share similar concerns with R2 about the presentation, and because experiments are still lacking. 

Additionally, I am concerned with one of the author's replies saying *All methods achieve accuracy 1 ... on the training distribution*, because usually there is a trade-off between accuracy on the observational distribution versus the shifted distribution (discussed by Rothenhäusler, 2018 [Anchor regression]): Achieving perfect accuracy on the observational distribution, usually means relying on the spurious correlations. And under domain-shift scenarios, this would hinder the performance on the shifted-distribution.




review_title: Strengths: Novel approach and theory. Weaknesses: Clarity and experiments

final_score: rating: 7: Good paper, accept
review_content: The Causal Semantic Generative Model (CSG) presents an approach for learning both semantic and diverse latent causal variables in supervised settings using variational Bayes.  Contrary to many similar approaches which assume the label as the causal variable, this assumes a hidden causal variable which produces both the labels and observed features. Furthermore, this approach separates this latent causal variable into two components one for semantics, which impacts label generation and one for the diversity/variation which in combination with the semantic variable generates the observed features.

The authors present variations for learning this model which account for correlation/independence between semantic and diversity latent variables (CSG and CSG-ind) and also extend to settings where some data does contain labels (CSG-DA). The authors also show under which data generating assumptions the modeling holds and show how changes to the prior distribution of semantic and diversity variable distributions impact Out-of-Distribution Generalization error and Domain Adaptation Error.

The work presents empirical results demonstrating the effectiveness of this approach in both OOD settings (no test adaptation) and domain adaptation settings (test adaptation). For the presented experiments the authors show superior results in OOD generalization and competitive results in the domain adaptation setting. While the experiments present a compelling proof of concept, the tasks Shifted MNIST and ImageCLEF-DA are not the most representative challenges in their respective domains.  Would be interested to see performance in ColoredMNIST task for causal identification and OOD generalization as the generative structure is well understood as well as performance capabilities. The same could be said for the domain adaptation task, with Office-Home, VisDA-17, DomainNet or a variety of more challenging and representative tasks giving more empirical credibly to the experiments performed.  
review_title: By learning separate latent causal variables for both diversity and semantics, this work presents a tractable method to learn supervised models invariant to distributional shift.

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes a Causal Semantic Generative model (CSG) to model semantic and variation factors separately and also provides a variational approach to learn the model.  It is proved that under some (perhaps strong) assumptions, CSG identifies the semantic factor on the training domain. Authors further show the boundedness of OOD generalization error based on the above result. Two  experiments are presented to validate the proposed method.

While the paper seems to have many interesting ideas and theoretic results, it is poorly presented and not well prepared, making it hard to evaluate the technical correctness. I also notice that the paper changes the original latex formatting by reducing much vertical space for almost all section titles, theorems, and equations. Also, the top margin of page 8 is heavily reduced. There is approximately 2/3 - 1 page more content than that of the original format with required page limit. Given this consideration, I also lower my score. 

In summary, I recommend a rejection for the present version. I highly suggest the authors revise the paper to make it more readable, self-contained, and concise, and resubmit the paper to another top conference/journal.

Please find my questions/suggestions below.

1. The variables $s$, $v$, $x$, $y$ are introduced in the introduction part but then not mentioned when they are actually used in Section 3 to describe CSG. I suggest introducing the variables in Section 3.

2. Authors use many italic words, and more crucially, lots of mixed uses of italic and normal fonts, particularly in Section 3. It really affects the reading flow. For introducing the CSG part, I recommend firstly introduce the model and then use another few paragraphs to present the examples for illustration.

3. For the invariance principle 3.2, $p(s,v)$ is the only source of domain shift. Does it allow $p(v)$ and $p(s|v)$ (or $p(s)$ and $p(v|s)$) change while keeping $p(s,v)$ unchanged?

3. (a) Sometimes you use 'Theorem' and also 'Thm.' in the main text, please be consistent. (b) The places for citation also affects the reading flow. For example, right after Assumption 5.2,  'It is a common (Janzing et al., 2009; Shalit et al., 2017; Khemakhem et al., 2019; Lee et al., 2019) sufficient condition for the fundamental (Peters et al., 2014, Prop. 7) requirement of causal minimality
(Peters et al., 2014; 2017) for identifiability.' (c)  Some notations are not introduced, e.g.,  $q^{\indep}(x)$ in CSG-ind. 

4. Why is it necessary to introduce CSG-ind? And in what case shall we consider CSG-ind? I did not find it well explained.

5.  Also for Assumption 5.2, 'It is a common sufficient condition for the fundamental requirement of causal minimality for identifiability.' However, causal minimality is not equivalent to that $f$ being bijective. 

6. The paper should provide more details in comparing with IRM, and in fact I didn't get much information about the difference with IRM from Section 3.2.  Can the authors give more details about 'For noisy or degenerate mechanisms, ambiguity
occurs during inference (Fig. 2), and the inferred result notably relies on the prior.'? Also, IRM is not compared in the experiments.

7. In my opinion, the biggest contribution of this paper is to propose CSG, within which several theoretic results (identifiability, OOD error bound, etc.) are established under some necessary but maybe strong conditions. However, there is much less content on verifying that CSG does bring many benefits. One way is to conduct extensive experiments to verify so, but I find the experiments are not sufficient and also several methods (like IRM) are not included for comparison.

** after reading rebuttal**

Thanks for clarifications and an improved version. I decide to increase my evaluation to 5.

However, I think that the paper needs to take more content to illustrate the practical benefits of the proposed CSG frameworks, for the following reasons: 
- the framework is proposed based on empirical observations like 'intervening an image by e.g. breaking a camera sensor unit when taking the image, does not change how the photographer labels it', which is not mathematically rigorous; 
- the principles and assumptions are rather strong (though I understand one generally has to make assumptions in causality), and in practice it is not clear when such assumptions hold and how many applications satisfy these assumptions; 
- the interesting derivations and theorems are also based on the CSG framework, which means, if the framework is incorrect, then these results may fail; 
- the experiment settings are rather limited in the current version. I hope the authors to add further content in their next version, regardless of whether the paper gets accepted or rejected.

Lastly, I still feel it a bit tricky to change the original formatting in the previous submission.
review_title: Interesting causality based model and several theoretic results. Insufficient experiments and poor presentation

Meta-review: The paper formalizes domain adaptation by taking the causal (generative) direction of dependencies p(image | class, domain).  They evaluate an ELBO surrogate loss by fitting a reverse q function that is new for this setup, and add a term to the loss that induces independence between class and domain. The paper also  proves identifiability conditions. The approach is then evaluated on two semi-synthetic and small datasets, showing some improvement. 

Reviewers were concerned about presentation and the experimental validation. The authors addresses some of the concerns in their rebuttal, but several reviewers found that the experimental evidence was still lacking, and that the authors should evaluate their approach in more standard and realistic benchmark datasets.  As a result, the paper cannot be accepted in its current form



 


Decision: Reject