Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: **Summary**
The paper considers the task of coarsely-disentangling a data-generating process into independent and shared modules. In more detail, it is assumed that each observation in a given dataset was generated by exactly one out of a set of independent generative processes referred to as independent causal mechanisms (ICM) in combination with a global mechanism that is shared across modules. The paper proposed the use of a mixture prior to model separate ICMs within a single architecture/generative model, chosen to be a GAN (Wasserstein-GAN with gradient penalty). Learning is performed by combining the GAN loss with self-supervision by using a separate encoder and adding loss terms resemblant of cycle-consistency-losses. The paper claims to prove a notion of identifiability of the coarse-grained modules in the sense of separation into disjoint manifolds and conducts some experiments on (variations of) MNIST and Fashion-MNIST, comparing their method against vanilla and disentangled VAE variants on some downstream tasks.

**Pros**
- the paper tackles a very interesting and highly-relevant topic (disentanglement with non-factorising latent space and connections to causality)
- the coarse-grained view and the separation into independent and shared mechanisms seems like a useful abstraction and (to the best of my knowledge) has not been investigated before in this form
- the authors support their proposed method with some theoretical analysis
- the latent traversal in Figure 2 and some of the quantitative results look promising 

**Cons**
- even though I am very familiar with the topic of the submission, I found the paper very hard to follow
- many of the statements relating to causality and ICMs are inaccurate, misleading, or wrong (see detailed comments below)
- the notion of identifiability used in the paper is very unconventional and not consistent with prior work
- the assumed generative model is never fully specified (in particular, the mapping from latents to observations and the role of the mechanisms is not described in the text or given as formula) and the mixture prior is inconsistent with the graphical model given in Fig.1
- some of the cited works are misrepresented or wrongly protrayed (see detailed comments)


**Evaluation**
I think the paper tackles a very important problem and presents some interesting ideas. However, the paper contains too many errors, unclear parts, and inconsistencies in the current form which makes it very difficult to gain a clear picture of the proposed method and to assess its correctness. Significant improvements in terms of clarity and exposition are needed before publication and I therefore recommend rejection at the current stage.

**Detailed Comments and Questions**
- The ICM principle states that "the data generating process is composed of independent and autonomous modules that do not inform of influence each other" (Peters et al., 2017). I believe you should cite this. Note that this is different from the notion of isolated, as different ICMs may feed into each other. In fact this is how ICMs are conventionally understood in causality: each conditional distribution of a variable given its causal parents $P(X_i|PA_{X_i})$ is interpreted as an ICM, and together they form the data generating process for the joint $P(X_1, ..., X_n)=\prod_{i=1}^n P(X_i|PA_{X_i})$. ICMs are thus traditionally understood as independent modules that can be (re-)combined.
- I do not understand why the term "ICM-conditioned mechanism" is used: what is an "independent causal mechanism-conditioned mechanism"? 
- Related Work: you mention three aspects but only two are given.
- Functional Causal Models: there are several key differences between FCMs and Bayesian networks (BN), so I think the presentation here is misguided. Firstly, BNs have no causal connotation to them but are only a way to represent a particular factorisation of a probability distribution. Causal graphical models or causal BNs on the other hand are endowed with a notion of intervention and are thus closer to FCMs. Perhaps this is what you intended to say? Moreover, it is unusual to refer to $P(X|PA_X)$ as a posterior---this is precisely what an ICM refers to and other common names include (causal) conditional or Markov kernel. 
- the entire discussion of causality and FCMs misses the notion of intervention and counterfactual which is crucial for defining causal concepts
- The description of FCMs is incorrect: The deterministic relationships (structural equations) in SCMs are a set of assignments of the form $X_i:=f_i(PA_{X_i}, U_i)$ where $U_i$ are the unobserved variables and $f_i$ are indeed deterministic. 
- I have never heard or read about this and would like to ask about the source of this characterisation: "If each function in FCM represents an autonomous mechanism, such FCM is called a structural model. Moreover, if each mechanism only determines the value of one and only one variable, such a structural model is called a structural causal model (SCM)." ? This seems strange to me.
- (Cai et al., 2019; Monti et al., 2020) are cited incorrectly: actually these aim to discover the graph which is different from learning an SCM as the former supports interventional reasoning, while the latter supports counterfactuals. 
- Figure 1: the rectangles M_i are not described in the caption.
- ICA: the description of recent advances in ICA is incorrect: it refers to "requiring a relatively large number of independent components" when this should read "... large number of environments" or more generally values of the auxiliary variable which renders sources conditionally independent
- the specification of the mixture prior at the end of page 3 seems incorrect as it does not integrate to 1. Perhaps you meant to include mixture weights? However, this is also inconsistent with Fig.1: (c) would suggest a prior which factorises as $p(z)=p(z_C)p(z_S)\prod_{i=0}^N p(z_{M_i}|z_C)$, while (d) suggests $p(z)=p(z_S)\prod_{i=0}^N p(z_{M_i})$ both of which are different from the text. As I infer it from reading between the lines, $z_C$ is a categorical variable that switches between different mechanisms, and $z_M$, $z_S$ are both inputs to the generative process. Why the need for the many $z_M$'s? Would a simple mixture distribution on $z_M$ with weights and parameters depending on $z_C$ not do the job? If not, can you explain why?
- How is the isolation constraint implemented, i.e., how does it translate mathematically? Can you please specify the full generative process $p(z, x)$?
- the last paragraph in 3.2 is very hard to follow
- 3.3 suddenly mentions the encoder, but no encoder has been introduced up to this point
- can you justify the loss function in more detail?
- Wasserstain --> Wasserstein
- 4. The stated version of identifiability does not seem to make sense to me. It is almost always possible to learn the ground truth model, the question of identifiability is whether we are *guaranteed* to learn the right model. The cited work of Khemakem et al (2020) provides a nice and intuitive definition of identifiability in terms of equivalence classes in parameter space that seems more principled than the notion used here. 
- Thm.3 This is the first time M_k is mentioned 
- I am not sure what the point of the paragraph at the end of page 5 is? This does not really provide any intuition on the Theorem or its proof. Moreover, you assume $h:Z\rightarrow Z$ and then have $h(z)\not\in Z$ which seems to contradict each other. Please clarify.
- the paper refers to "co-variant shift" throughout; to the best of my knowledge "covariate shift" is the accepted terminology. is the different notation intentional, and if so can you clarify the difference?
- some experiments are not described in sufficient detail (e.g., what is being predicted in 5.2? why are there no error bars in Tables 1 or 3)
- the paper would benefit considerably from some professional proofreading and grammar checking (though this is not a key factor in my evaluation)



POST-REBUTTAL UPDATE:
I thank the authors for the detailed response. Based on the proposed changes I will slightly increase my score, but I still believe the paper needs additional work before meriting publication.
review_title: Interesting paper, but clarity and exposition need to be improved

final_score: rating: 2: Strong rejection
review_content: I hardly agree with the assumption made in the graphical model c) and d) that each cluster of data has a different generating mechanism from others. First, it is named the ICM-conditional mechanism in the paper, which is very confusing since it contradicts the definition in the literature of ICM [1]. The ICM, in the literature, originally describes the independent autonomous for generating a sequence of variables in the causal graph, e.g., p(v_1,...,v_k) = \Pi_k p(v_k | Pa(k)), based on the assumption that the exogenous variables are independent with each other. Besides, the difference in terms of variation for digits 1 and 2, in the example made for supporting this assumption, should be encoded in the latent variables z, rather than the generating process. Since the latent variable z is defined to describe the high-level abstractions or concepts, including but not limited to the thickness, width, length in the example of MNIST. Therefore, the graphical model should be y -> z -> x, which is aligned with existing literature in nonlinear ICA.  Besides, even if the assumption is correct, the posterior distribution of p(z_Mk | x_Mk) should be varied across k, which hence cannot be learned by the same encoder E. 

Although the experimental results yield good results, it can not be interpreted in this way. The high-level spirit of this paper is based on an invalid assumption, which makes the promising results in experimental parts non-important. 


[1] Schölkopf, Bernhard. "Causality for machine learning." arXiv preprint arXiv:1911.10500 (2019).
review_title: This work assumes the disentanglement of the generation process, with each process corresponding to each cluster of data. The whole methodology is standing on this assumption, which is unreasonable for me.

final_score: rating: 5: Marginally below acceptance threshold
review_content: Authors propose a new method to learn independent causal mechanisms from data, which is achieved by designing a mixture prior consisting of shared mechanisms and Independent Causal Mechanisms (ICM) conditioned mechanisms. Specific orthogonal structure or separability is imposed on the ICM-conditioned mechanisms. A single generative model is used with such a mixture prior to learn mechanisms from data. Authors also prove the subspace of the mechanisms are separated and thus identifiable. Experiments on MNIST dataset demonstrate that the method is able to learn independent mechanisms and it improves the robustness against intervention, co-variant shift and noise. 

Detailed Comments:
1. I am not quite clear about the meaning of ICM-conditioned mechanisms. Could authors elaborate it a little bit what “conditioned” means.

2. The Independent Causal Mechanisms first proposed by (Scholkopf, 2019) is referring to the independence of each sub-modules in a whole generating process, e.g. for a cause-effect pair (x, y), ICM usually means that the process of generating the cause which is p(x) is independent of the process that maps the cause to the effect which is p(y|x). However, in this paper, authors are considering a very different setting where they argue that for different data groups, the generative mechanisms are different. I am not very sure they are the same thing. Could authors explain the connection between the two?

3. Authors use single generative model which is claimed to be one of the advantage of their method. However, in Fig 1(d) they use M_N, …, M_0 to indicate that for different data groups, the generating mechanisms are different. Could authors explain this discrepancy?

4. It appears to me that the method can only work in the scenario where we know beforehand the number of different generating mechanisms in the data. However, for real applications it is difficult to have such prior information. Can authors also comment about the applicability of the method?

5. It appears to me that the proposed method is incremental to existing disentangled representation learning methods except for the mixture prior.

review_title: a new method for disentangling independent causal mechanisms

Meta-review: This paper proposes to learn representations in an unsupervised manner using a generative model in which observations are generated by combining independent causal mechanisms (ICMs), in combination with a global mechanism. The authors introduce an unconventional mixture prior for the shared and independent components of the representation and train an encoder, discriminator and generator using a Wasserstein GAN with additional terms that enforce consistency in the data and latent space. Experiments consider variations of MNIST and Fashion-MNIST and perform comparisons against a standard VAE, a β-VAE, and the Ada-GVAE. 

Reviewers are broadly in agreement that this submission is not ready for publication in its current form. R4 in particular has left very detailed comments regarding clarity. The authors were able to in part address these comments, and R4 raised their score in response. That said, from a read of the manuscript in its latest form, the metareviewer (who is very familiar with literature on disentangled representations) is inclined to agree with the reviewers that this is work that has value, but is very difficult to follow in its current form. The metareviewer would like to suggest that the authors regroup, think carefully about how to improve clarity (in addition to addressing concrete points raised in reviews) and resubmit to a different venue. 
Decision: Reject