Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper presents a novel and interesting method to formulate learning of correlated disentangled causal factors as a part of encode-decoder framework using a Gan-style learning approach. The proposed idea is interesting but I think the wording of the paper as well as the theoretical and experimental results could be improved. 

Please find my detailed comments:
1- In equation (6), I suggest it to be modified to L(E, G, F)=L_{gen}(E,F,G)+\lambda L_{sup}(E). In other words, it would make sense to remove min_{E, G, F}.

2- Proposition 1: I think a necessary condition for a to be not equal to zero is that if the ground truth latent factors are correlated. Therefore, I suggest this assumption to be added to the assumptions of the proposition. Regarding the proof, I found the following typos. First, in the third line of the proof, I think x_{j} should be \xi_{j} instead. In the last two lines of the proof, L(E^*, G)\geq a+b^* should be replaced with L(E^*, G)\geq a+\lambda b^*.

3- Regarding theorem 1, I found the sentence "Then DEAR learns the disentangled encoder E*" not accurate. It is because assuming E, G and f does not guarantee that this framework is able to actually learn E*. Therefore, I suggest to change this statement to be more like the following statement: The encoder for the optimal solution of the proposed loss fn will be disentangled encoder E*. I found the following typos in the proof of this theorem. First, in the third line, inside the parenthesis, y should be replaced with y_i. In the 8th line, the derivative should be with respect to E_i(x) and not \sigma(E_i(x)). In the 11th line, p(x,z) should be replaced with p_G(x, z).

4- In the section 4.3 Algorithm, I found the following typos. First, in the second line, "implicit generated conditional" should be "implicit generative conditional". In the 5th line, "theorem" should be replaced with "Lemma". In Lemma 1, in the presented gradients, L_{gen} should be L.

5- Regarding the Experiments section, I did not find the qualitative experiments to clearly present the superiority of this method to disentangle correlated causal factors over the previous methods with a similar objective.  

review_title: This paper presents a novel and interesting method to formulate learning of correlated disentangled causal factors as a part of encode-decoder framework using a Gan-style learning approach. The proposed idea is interesting but I think the wording of the paper as well as the theoretical and experimental results could be improved. 

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper propose to learning causal disentangled representation which conforms human's cognition. The learned representation can be applied in causal controllable generation and benefit downstream tasks. The uses of this method is interesting.
The paper has some advantages:
It only requires part of sample is supervised, the previous methods need fully supervied information.
The previous method unreasonably force the latent factors to be independent. However, this method incorporate a SCM as prior distribution of latent representations. Therefore, the learned factors can be causally correlated which is consistent to practice.
The experiments are extensively conducted. The results demonstrate the effectiveness of this method.

I have some questions about this method:
In section 4.3, the gradient in Algorithm 1 is not consistent with the gradient in Lemma 1. Which one is correct?
In section 5.2.2, the test set is grouped into 4 groups according to the two binary labels, the target one and the spurious one. It is not clear what is the meaning of target one and spurious one.
The paper require the information about SCM as prior, which is a strong requirement. I think it is of great significance to investigate how to learn causal disentanlged representation with weaker prior knowledge in the future. I understant it is very difficult, but it is worth researching.

review_title: Disentangled Generative Causal Representation Learning

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper presents a latent variable model where the variables in the latent space are causally disentangled, i.e. the disentanglement is ensured according to a structural causal model (SCM). The resulting model is made up of two parts. The first one, a generative unsupervised part, is essentially a VAE and is defined with the VAE ELBO loss. The second part is supervised and accounts for the causal disentanglement of the factors that are assumed to underlie the distribution; the authors claim the fewer supervised samples are required to estimate the second part of the loss alone. The two parts are then combined with a hyperparameter.

The authors motivate their approach by comparing it to models using an independent prior for the latent factors and to ones using structural causal models in the latent space. For the former case, they claim that in reality the latent factors are frequently dependent, which their model is able to capture. For the latter, they argue that the competing models use SCMs for conditional factors rather than unconditional as in the proposed approach.

Subsequently, disentangled representation is defined as one where a 1-to-1 function can be established between the data and the underlying factors corresponding to the nodes of the SCM (Definition 1). The authors then propose to use the general nonlinear SCM model in the latent space which concludes the actual definition of their model (part 2 of the loss mentioned before). Finally, they note that the model ensures disentanglement as defined before and describe a GAN-like algorithm for minimizing the compound loss (part 1 + part 2). The paper is concluded with a series of experiments which include a quantitative comparison of accuracy, sample efficiency and distributional robustness against a number of VAE-based disentanglement methods.


*****Strengths:*****

The incorporation of structural causal models to generative modelling, especially in the context of disentanglement and the resulting explainability, is an important topic and this paper is certainly relevant in this area.

The experiments show a quantitative improvement over a number of competing disentanglement methods.


*****Weaknesses:*****

Generative modelling with SCMs, being an important area of research, was addressed before. The review of related work provided here is rather limited. Moreover, the authors decided to contrast their approach mostly to disentanglement strategies with an independence prior. I think the paper would be much more convincing if more weight were put on direct comparison with other methods using SCMs for generative modelling such as causalGAN and cognate methods (which is absent from the experiment section at the moment). As a matter of fact even the limited high-level comparison to such models provided in section 2 left me confused (what is the “unidirectional nature” of the other models compared to DARE? What is meant by “direct access to attributes” in the other methods – labels? Factors?).


*****Questions / feedback:*****

See weaknesses.

What is the role of the hyperparameter lambda (equations 2, 6)? Can we alter the sample complexity (wrt labels y) by varying it, or make the model less dependent on the assumed generalized SCM?

Why is there a difference in notation between Z and \xi? Is the DAG defined by Z not the same as the SCM defined by \xi?


*****Typos / minor comments:*****

Section 4.2: justification on -> justification of

Section 4.1: interventional distribution is never defined

I found Figure 1 unreadable.


*****Post Rebuttal*****

I would like to thank the authors for the rebuttal. While I think more discussion / comparison to approaches without an explicit independence prior would further improve the paper, the authors have clarified many of my doubts. I have therefore decided to raise my final score.
review_title: An interesting paper on generative latent space modelling with SCMs, more discussion on relation to similar models needed.

final_score: rating: 5: Marginally below acceptance threshold
review_content: **Summary of the paper**
This submission addresses the topic of causal representation learning/causal disentanglement, i.e., learning explanatory latent factors which are not mutually independent, but instead connected via an underlying causal model. The authors propose the DEAR framework to address this task. DEAR consists of an encoder-decoder architecture (similar to VAEs) which is combined with a separate discriminator. The key learning procedure is to match two joint distributions over data x and latents z: (i) the joint induced by pushing the (empirical?) data distribution q(x) through the encoder q(z|x); and (ii) the joint induced by pushing the (model?) prior p(z) through the generator p(x|z). The task of distribution matching is done using a separate discriminator trained to distinguish q(x,z) from p(x,z) while encoder and decoder are trained to fool the discriminator. 
Another key ingredient is the form of the model prior p(z) which does not factorise, but instead takes a particular form inspired by structural causal models (SCMs).
In order to learn the dependent latent variables, two additional strong assumptions are made: (i) full supervision in the form of observations of all ground truth causal factors y for (a subset of) the training data; (ii) ground truth knowledge of the causal ordering or full causal graph. The latter (ii) is used to construct the prior p(z) to allow for the same DAG structure as the true factors, and the former (i) is used to augment the joint distribution matching objective with a supervised regulariser that penalises deviation of the encoded factors from the groundtruth factors.
The authors claim to prove non-identifiability for models assuming a factorised prior, as well as identifiability for their model, but I have some reservations and concerns about the level of detail and technical correctness of the proofs (more details below).
Experiments are performed on three datasets with causally related latent variables: Pendulum, CelebA-Smile and CelebA-Attractive. The proposed DEAR approach is compared against a baseline (S-$\beta$-VAE) in terms of latent traversals, controllable generation, and two downstream tasks (sample efficiency and distributional robustness) and outperforms the baseline on these.

**Summary of evaluation**
While the paper addresses a very important problem and contributes some interesting ideas and results, the submission has too many technical flaws and misses too many important details to merit publication in the current form. 

**Pros**
- the setting of learning causally related latent factors considered in the paper is a very interesting and highly relevant
- the approach of matching the two joint distributions induced by encoder and decoder in an adversarial fashion is well-motivated, interesting, and (as far as I can tell) novel
- the authors (attempt to) provide some theoretical insights along with the proposed method which is appreciated
- the datasets seem suitable for the task, and some of the results (e.g., Fig.3) are quite interesting and promising

**Cons**
- the main part of the paper (Sections 3 & 4) is quite hard to parse due to inconsistencies, non-standard use of notation (CE, q(x) for data distribution, ...), and important missing details (e.g., the assumed data generating process in terms of $\xi$ is never specified)
- the model relies on known ground truth causal ordering AND supervision in the form of annotated ground truth factors which is a very strong assumption and too unrealistic to be useful for causal representation learning in practice: learning the causal variables and their structure is precisely the task in causal disentanglement and it seems that most information is assumed given. At the same time, the importance of these assumptions is not stressed enough in the abstract and introduction and formulations like "using a suitable GAN loss" are misleading since they suggest an unsupervised approach.
- only linear relationships between the causal variables $V=f^{-1}(Z)$ with additive noise are allowed (before a component-wise nonlinearity f is applied) and it is not clear why this restriction is necessary or why the component wise nonlinearity f cannot simply by absorbed into the decoder
- the theoretical results claimed in the paper are not backed up by rigorous proofs (I checked proofs for Proposition 1 and Theorem 1, but not for the derivations of the gradients/Lemmas): there are several mathematical errors, missing assumptions, and unclear steps brushed under the carpet
- there is no information about the choice of two key quantities, the hyperparameter $\lambda$ determining the weight of the supervised regulariser, and the fraction/amount of labelled examples $N_S$ (nor any ablation on how performance depends on these) which harms transparency and reproducibility

**Comments regarding the Proof of Proposition 1**
- 2nd sentence: E should be E*.
- The assumption of a non-factorising distribution over $\xi$ is not stated as part of the proposition. This should be stated as a clear assumption in which case the first paragraph of the proof becomes obsolete. 
- 2nd paragraph, 2nd sentence refers to distributions of z and E(x), but I believe E*(x) is meant. Otherwise the claim that "the intersection of their distribution families" (could also be clarified) is incorrect or at least not justified.
- 3rd paragraph refers to $p_E(x,z)$ which is not defined. I assume $q_E(x,z)$ is meant? 
- A key problem I see with the definition of b as the minimum of $L_{sup}$ subject to the constraint $L_gen=0$. It is not clear why this minimum should exist, i.e., why it should be possible to satisfy the constraint if the prior $p(z)$ factorises, but the true generative process is based on dependent latents. Certainly, there are not sufficient details provided to assess the correctness of the statement.
- there are missing $\lambda$'s in $L^*$, should be $L^*$ $\geq a+\lambda b^* > \lambda b^*$

**Comments regarding the proof of Theorem 1**
- as stated before, cross entropy is between two distributions does not involve a sigmoid function
- what is the distribution $p(y_i|x)$? this was not defined in the paper
- the partial derivative is incorrect, should be $y/\sigma(E_i)-(1-y_i)/(1-\sigma(E_i))$ inside the bracket
- the second partial derivative misses a factor 2
- the main part of the proof is only informally sketched in natural language: this is not sufficiently rigorous for a prove that claims to show identifiability; more detail is needed to be able to assess the correctness of the claim. It is also not clear whether infinite data is assumed.

**More detailed comments and questions**
- the use of $q_x(x)$ for the data distribution is unconventional: usually this is denoted $p_{\text{data}}(x)$ or simply $p(x)$; also it would be good to distinguish between the empirical distribution provided by the training data and the true distribution which is unknown at different parts in the paper
- the related work section is very short and key references are not described in any detail: it would make the paper more accessible if key building blocks of the proposed method (e.g., Locatello et al., Yu et al.) were explained in more detail here
- I do not follow the logic  of the first sentence at the top of page 3, please clarify or rephrase
- the assumed true data generating process to be captured is never specified: in particular, what is the generative process for the ground truth factors $\xi$ and how are these factors decoded to give rise to the observations?
- I do not understand the need to consider separate $\xi$ and $y$? what does the addition of $y$ add? also what distribution is the expectation in the definition of $y$ with respect to: $\xi_i=\mathbb{E}[y_i|x]$ ?
- the definition of cross entropy stated in 3.2 does not coincide with the universally accepted form of cross entropy (which does not have the sigmoid term)
- what is $\phi$ in $L_{sup}(\phi)$ in 3.2? it is not defined at this point, and why does it not appear as argument to the other $L_{sup}$'s?
- Definition 1 seems strange to me as it does not allow for permutations of the ground truth factors. I would have expected that for all i there is a j s.t. $E_i(x)=g_j(\xi_j)$. Can you elaborate?
- last paragraph in Section 3: The claim made here seems incorrect: (E',G') as constructed in Proposition 1 minimises neither (1) nor (2), but $L_{sup}(E)$ subject to $L_{gen}=0$; see also my point on the feasibility of this constraint in the context of the proof of proposition 1
-Sec.4.1: if f is a component wise tranformation, it is not clear what it's addition adds, and why you do not consider a causal model over $V=f^{-1}(z)$ instead. the particular choice of a linear SCM is not motivated further which makes it hard to understand. Also, it would help to specify $p(\epsilon)$ earlier as otherwise one wonders what the point of $h$ is.
- you refer to the "direction" and scale of causal effects, but the direction is given by assumption in the form of causal order. do you mean the "sign" of the linear effect?
- the procedure described for computing interventions is not motivated from the latent SCM, and in fact it appears to be closer in nature to a counterfactual (rung 3) rather than an interventional (rung 2) quantity since the noise variables are kept fixed when performing the intervention as far as I understand. For an intervention, the exogenous variables would have to be re-sampled.
- I do not understand the reasoning in the last paragraph of Sec. 4.1.: the claim that k=m fails due to capacity issues is not explained and does not make sense to me. I do not understand why one would consider different sets of latent, since independent factors can also be incorporated within the latent SCM, these would simply have zeros in the corresponding rows and columns in the adjacency matrix.
- Theorem 1 assumes that "the true binary adjacency matrix" can be learned. This is too vague: what does "can be learnt" mean exactly? Also, why is the adjacency matrix binary all of the sudden?
- 4.3 states that unlike prior work, the prior distribution has learnable parameters, but there are a number of works considering learning the prior. you should either rephrase or reference such works. 
- Algorithm 1: $\psi$ is not defined at this point. I assume it refers to the discriminator parameters?
- 5.1 you state that you can only manipulate the causal factors whilr leaving their effects unchanged, but this is incorrect. Effects will change unless you also intervene on those and fix them (which I believe is what you mean, given the experiments).
- There is a reference to Figure 3-9 which confused me.
- 5.2 I find the choice of defining "attractiveness as a slim young woman with makeup and thick hair" very questionable from an ethical perspective. I think that we should try our best in academia to overcome such outdated notions, and given that this was a design choice you made, I think it would be nice to consider a more neutral/less controversial example. (This is just a personal note and has no bearing on my evaluation!)

**Post rebuttal comments**
I thank the authors for the detailed response. I think that some points have been clarified and corrected, and I have increased my score slightly to reflect this. I still think the paper needs another iteration before publication though.
review_title: Very interesting topic, but too many technical flaws, inaccuracies, and missing details in the current form

Meta-review: This paper presents a method to formulate learning of causally disentangled representation as a part of the encode-decoder framework. Although the reviewers agree that the paper presents some interesting ideas, they feel the paper is not ready for publication yet.  In particular, I encourage the authors to take the feedback of reviewer R2 into account, which is quite detailed and provides substantive ways of improving the work. After all, I recommend rejection.


Decision: Reject