Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a general framework for so-called "out-of-distribution generalization" that can handle non-linear associations and causal links between variables. It provides proofs for identifiability and generalization. It proposes a general causal model that is sensible for many prediction problems. It proposes a conditionally non-factorized prior. It provides a tractable method that can be used in practice.
main_review: **Strengths**

This paper has significant technical and empirical novelty. Moreover, it is very well written and communicates the central contributions clearly.

**Weaknesses**

Overall, I only have 2 issues with this paper. **First**, a very minor concern regarding readability. Since your solution builds off of the VAE literature in part, I think that it would be more readable to use $X$ for observed random variables and $Z$ for latent random variables, rather than $O$ and $X$ respectively. While $O$ makes sense since it is the first letter in observed, I think it is burdensome to ask computer science readers to disassociate it from its usage in "Big O notation".

**Second**, and more significantly. I'm not convinced that "Out-of-Distribution Generalization" is a sensible term, and I'm also concerned that it is misleading. I think you clearly present one graphical model that defines one distribution. All environments are a part of that distribution, and I would argue that unseen environments are out-of-sample rather than out-of-distribution.

For me, out-of-distribution would correspond to a fundamental change in the relationship between variables. For example, let's say $Y$ is a binary variable indicating whether or not a sequence of characters $O$ signifies addition. Under one distribution the plus sign (+) (represented by some $X_p$) would have a strong causal association with $Y$. This would be an invariant causal association across whether the characters were printed, hand written, displayed on a screen, blue, purple, etc. If we never observed an orange "+" during training, under the assumption of the same distribution (graphical model) it makes sense that we would want to generalize to this *out-of-sample* environment. But what if we see samples from a different graphical model, say one in which $Y$ now signifies the logical or operation and under this distribution all plus signs are instead associated with that $Y$? If our original model saw samples from this distribution it would confidently and incorrectly classify such examples as signifying addition, thus failing to generalize to the new distribution. This is just one example of why I think it is incorrect to characterize what this method does as out-of-distribution generalization. I am open to further discussion about this point, but I think it would be more precise to use the term out-of-sample in place of out-of-distribution.
summary_of_the_review: This paper has significant technical and empirical novelty. Therefore I recommend acceptance. However, I think we need to have a discussion about whether Out-of-Distribution Generalization is a sensible term.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes invariant Causal Representation Learning (iCaRL) for OOD generalization in the nonlinear setting. The work extends iVAE to a somewhat more general setting and shows the direct cause of the target can be discovered. iCARL is then developed based on the direct cause. . Extensive experiments verify the effectiveness of the proposed method.
main_review: Section 4.1 presents the theoretical foundation, with a complete and clear proof. This is a valuable contribution.

Section 4.2 is less than rigorous: a) “one observation is that for any two latent variables Xi and Xj , only when both are causes of Y do we have that the dependency between them increases after conditioning on Y .” Note that Xi and Xj are actually correlated due to the confounder E, how to justify this claim? Or do you mean conditional on E? b) ”when there exist at least
two causal latent variables, it is trivial to test all pairs of latent variables with independence testing (Gretton et al., 2007) and conditional independence testing (Zhang et al., 2012) to discover all of them by comparing p-values from these two tests.” I am not sure whether the p-values can be compared. Also, p-value itself is not a statistical measure. Please justify the methods in this section. 

Section 4.3 gives the algorithm wrt. both training datasets and new test data. Here is my main concern: I wonder whether this method is for DG and OOD generalization, or indeed a DA method. Note that “ Phase 3:  … When in a new environment, we first infer Pa(Y) from O by solving Eq. (12) and then leverage the learned w for prediction.” This means, the method requires knowledge of test domain data, although in a not very complicated way. To me, if test domain data are needed in adjusting the algorithm, then it should be a DA method. Note that this is different from simple normalization, as this operation can be included in the trained model. Further, what if we have a test dataset as a mixture of several domains, then is this method affected? Please clarify.

Minor concerns and questions:
1.	The causal graph is explained well wrt practical considerations. However, it is not clear how practical the assumptions in Theorem 1 and 2 are. For instance, is it true to have O=f(x) injective in VLCS datasets? Why is it needed to assume n <= d?
2.	“It also makes sense in Assumption 1c that the generative mechanism p(O| X) is invariant across all the environments. Otherwise, it is impossible to infer X from O in any unseen environment” Is it possible to assume only $P(O|X_pa)$ remains invariant? 
3.	Footnote 7 is missing. Also maybe move some footnotes to the main text in later version (e.g., camera ready version if accepted)? Footnote kind of affecs reading flow.
4.	In practical algorithms, how to select the dimension of X? 
5.	(very minor concern) maybe to include one or two more real datasets like PACS?

summary_of_the_review: In summary, the paper has adequate theoretical contributions in terms of identifiability. The algorithm part seems less justified. Also, the method appears to be a DA method, not for OOD generalization, thus authors should compare their algorithm to recent DA methods. I look forward to authors' response.

== after reading response ==
Based on the current response, I decide to increase my evaluation.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes Invariant Causal Representation Learning (iCaRL), which learns causal representation for OOD generalization and can be viewd as a generalization of IRM to the settings with both nonlinear representations and nonlinear classidiers. The problem is important but part of the algorithm seems not technically correct.
main_review: strengths
- a general underlying causal model that covers many real-world scenarios is peoposed
- assumptions of the general causal model are clearly presented and explained
- theoretical guarantee on the identifiability of the latent varibles is provided
- the results of synthetic data experiments are carefully analyzed and can well corroborate the theoretical claims


weakness
- the algorithm seems a bit cumbersome
- there appears to be a technical flaw in Phase 2 of the algorithm
- some popular benchmark data sets are missing
summary_of_the_review: This paper introduces an interesting idea about a general causal model, and presents related assumptions and theoretical results clearly. There seems to be some technical flaws in the proposed algorithm and the experimental results could be improved by adding some popular benchmark  real-world data set.


detailed comments
- the general casual structure (Fig. 1(c)) lasks an edge from E to Y compared with Fig. 1(b). Will this difference affect its expressiveness such that some data-generating processes of Fig. 1(b) can not be characterized by Fig. 1(c)
- when determining Pa(Y) in phase 2, the algorithm compares the p-values of independent testing IndTest(X_i, X_j) and conditional independence testing IndTest(X_i, X_j | Y). Since these two methods derive test statistic from differernt rationale, I was wondering whether the p-values of these two test can be directly compared to conclude the evidence of independence.
- I was wondering whether phase 2 is able to distinguish Y's parents between the its ancestors. it seems an ancestor of a direct cause of Y will also be included in Pa(Y) after phase 2.
- this paper only provedes the results on VLCS data set while existing works usually evaluate the prerformance on at least two real-world data set. It would be much better to include one more real-world data set (e.g. PACS)
review_title: null

Meta-review: The paper sets up a complex algorithm for out-of-distribution generalization. The algorithm requires first, a generalization of identification results for variational autoencoders, the followed by second, a causal discovery subroutine, and third, learning an invariant predictor using the discovered causes. The procedure reads sound, and the results on common benchmarks look good, though I do not know how practical the approach would be in general.
Decision: Accept (Poster)