Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper proposes a method for learning invariant (nonlinear) data representations and classifiers, using data from multiple domains. A key step in the method is to discover the direct causes of the outcome of interest from a set of latent variables that are recovered from observed variables via identifiable VAE. The problem being tackled is significant, and the general idea is interesting and sensible. The empirical results also look encouraging. However, there appears to be a major flaw in the theoretical setup. In order to apply identifiable VAE, the method needs to assume that any two latent variables are independent conditional on the outcome variable and the domain index (Assumption 1). But what about latent variables that are direct causes of the outcome variable? If both X_1 and X_2 are direct causes of Y, generically they will not be independent conditional on Y and E, will they? In the motivating example, only one latent variable is a direct cause of the outcome, so this issue does not arise, but the ambition, as I understand it, is to handle any number of direct causes. I don't see how Assumption 1 can be justified as a plausible assumption in the general case where more than one latent variable is a direct cause of the outcome.

Moreover, Theorem 4 is less than fully rigorous and is misleading. For example, the proof of Theorem 4 invokes a method for inferring causal directions in Zhang et al. (2017), but as far as I know, that method does not yet have a rigorous theoretical justification. As it is formulated, Theorem 4 sounds like a theoretical identifiability result, and as such is not rigorously established by the proof given in the paper.
review_title: Interesting approach to an important problem, but with a substantial flaw

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper proposes an invariant causal representation learning paradigm in the nonlinear setting. Based on a conditional factorial assumption, they proved identifiability up to a linear transform. The ICRL objective, in this case, is able to discover all the direct causes of the outcome, and thus enables OOD generalization.

The novelty of the paper seems to be in the generalization of the IRM framework to the nonlinear case which is interesting to me. 
The authors combined iVAE and IRM to solve this problem. Overall the paper is clearly written and easy to follow, but some conceptual issues remain.

Here are my issues with the paper:
- How to verify assumption 1 in a real case? Although the authors argued this assumption is not very restrictive and similar to the assumption in the iVAE paper, I think there is a difference between these two papers. In the iVAE paper, they assumed the latent variables to be conditionally factorial, while here the authors assume the potential causals (unobserved variables) are independent. 

- After discovering direct causes, they still need the IRM phase to learn an invariant predictor. IRM itself can identify spurious causes and learn an invariant predictor, so what is the gain of learning the first two phases? What if some spurious causations are wrongly detected by the second phase, will it affect the predictor?

-The synthetic data experiment is not convincing at all. ICRL outperforms ERM and IRM in a very extreme case, where all the algorithms perform terribly, I don't think I can conclude ICRL is a better algorithm among others from this test case. If the authors can visually show the invariant representation of ICRL is more robust, that would be a good illustration. 

- In the colored MNIST experiment, I assume the setting is the same as the IRM paper. While they said their IRM can reach 66.9+-2.5 acc, which is 7 percent higher than this paper and even higher than ICRL. So I wonder what causes this gap.
review_title: Interesting topic but the assumption and the experiments are not convincing.

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper proposes Invariant Causal Representation Learning, which seeks to learn representations for downstream tasks that are based on only causally invariant latent variables so the representation is robust to shifts in the test environment.  

A model is assumed where an environment variable is a cause of all the latent variables and a target. The iVAE algorithm is used to learn the latent variable model. Then, a series of conditional independence tests and bivariate causal discovery methods are used to distinguish which latent variables correspond to causes (effects) of the target. Finally representations are learned from the observed variables to the causal latent variables of the target and then from these variables to the target.

The approach is evaluated using synthetic data and semi-synthetic data based on MNIST.

The clarity and organization of the paper could be improved. The algorithm should be moved from the appendix to the main text and the procedure should be described more holistically to give the reader an outline before diving into the details of each component section. The experiments section is also very unclear.

The main issue that remains unclear to me is how the environment variable E is being used explicitly. It doesn’t seem clear to me that you would generally have access to E, but it is required in the rules necessary to determine the direct causes of Y. What are the E variables being used in each of the experiments?

The novelty seems somewhat limited. It seems the theoretical results can be divided into (a) results about identifiability of the latent variable model and (b) the method for identifying the direct causes. Some of (a) follows directly from Khemakhem et al. (2020) - it is difficult to determine whether there is sufficient novelty here. (b) follows directly from well known constraint-based and bivariate causal discovery approaches. 

The experiments are unconvincing. The proposed method outperforms existing approaches in a high noise synthetic data example and a kind of adversarial example where grayscale MNIST is colored in a way that is strongly correlated with the class label (the experimental setup and evaluation metric is very confusing here). It would be more convincing to the proposed method evaluated in a more realistic setting.

Further, there are no (synthetic) experiments which confirm that the proposed method does in fact learn the causally relevant latent variables and its robustness in doing so. Since identifying the correct causal latent variables requires multiple conditional independence tests and bivariate causal discovery methods (on latent variables which may be estimated incorrectly), there is an obvious concern about how robust this procedure is in practice. It would be more convincing to see (e.g.) precision and recall with regard to selecting the correct causal latent variables when the ground truth is known.

In summary, the paper introducing some interesting ideas, but the clarity could be improved, the novelty may be somewhat limited and the experimental results could be improved.
review_title: Proposes some interesting ideas, but the novelty is limited and experimental results are unconvincing 

Meta-review: The paper aims to provide a framework for learning non-linear feature mappings such that are invariant to environments. The critical concern raised by the reviewers is their assumption: that causal features of the label are conditionally independent given the label. But in any DAG, conditioning on a common child (here, the label) renders the parents dependent. Their assumption thus is not going to hold other than on a measure zero set of parameters.
Decision: Reject