Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes a novel principle for representation learning using an autoencoder, named interventional consistency. The tractable objective functions, namely Invariance Score and Equivariance Score, are formulated, and Explicit Causal Latent Block is proposed to model the causal structure of the latent representation.
main_review: Strengths
- The proposed concept, the interventional consistency, seems highly novel. It extends the notion of consistency in a meaningful way.

Weaknesses
- The largest weakness of the paper is the lack of clarity. The paper is very difficult to follow, and I believe the difficulty is attributed to the unpolished exposition. In general, Section 2 assumes the readers have a strong background in causal representation learning to an unnecessary extent. 
  - The ICM principle is introduced without stating its full name (independent causal mechanisms, I guess?). 
  - Since ICM works as a fundamental underlying principle for the work, it deserves a more detailed illustration and justification.
  - It would be helpful for readers if it is stated that PA states "parents". 
  - What is the definition of the intervention distribution? What are the examples? What properties of them are assumed?
  - Providing a concrete, real-world example of interventional consistency would be helpful.
  - While the paper says $m$ and $k$ are assumed to be equivalent, Figure 1 depicts the scenario where $m$ and $k$ are different. Additionally, I couldn't come up with a meaningful real-world case where $m\neq k$.
- The other parts of the paper also have clarity issues.
  - Texts in Figure 2, Figure 3, Figure 5, Figure 8 are too small and difficult to read.
  - The figure numbering is missing for Figure 5.
  - There is a broken reference link on page 6.
summary_of_the_review: I vote to reject the paper, due to the lack of clarity. To improve the clarity of the paper, significant rewriting seems to be needed.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors argue that the interventional consistency is a desired property for the representation of auto-encoders and proposed a new metric on it. The authors further develop a new training procedure for auto-encoder that has extra regularization terms to enforce the interventional consistency.
main_review: Frankly, I am not quite convinced why the interventional consistency is desired even after reading the draft. Some assumptions seem too strong to me and I feel the main objective of the paper is not conveyed well.

Some typos and issues listed here:

1) Unexplained notations: 

      a) section 2, composition R = E\circle D. here neither E nor D is defined. 

      b)  Figure 4 comes before the figure 3

      c) the third paragraph of section 3, related work: there is a citation error with question mark


2) Lack of clarity:

a) The major algorithms 1) and 2) are not described in the main paper.  We have to reach out to the appendix for the algorithm description. 

b) The optimization objective L_{INV} and L_{EQV} are not specified in the main paper either.

c) The response map is mainly introduced by citing a related work. However, given its importance to this paper it is better explained and self-contained.

d) There is not much explanation on the latent traversals.


3) Assumptions:

There is an assumption that the statistical dependencies in the prior are preserved by the response map.  This assumption looks to me very strong. Not sure if such an assumption makes sense for real data.

summary_of_the_review: I feel the draft is not in a status for publication yet. The authors may need to polish it to have a clear story line. Also it needs to be self-contained.


########Update after the authors' rebuttal###########

I would like to thank the authors' explanations on the notations as well as the assumption concerns. Overall I feel this draft has potential to be a good one. As stated in the authors feedback, in its current status more work is needed to make it easy to read. (For example, the algorithm descriptions are still hidden in the appendix.) As such I am keeping my ratings unchanged.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes to use interventional consistency to regularise representation learning in VAEs.
The idea is well-motivated by the ICM principle and theoretically justified.
The paper suggests to use interventional consistency for both training and evaluation of the representation learnt by VAEs.
Results show that the proposed idea can give more modular and interpretable representation.
main_review: Strengths
- The idea is well-motivated and seems to be theoretically justified.
- The empirical results are promising and support the main claim of the paper.
- The limitation of the proposed method is also discussed (in the paragraph right before Section 2.1).

Weaknesses
- Presentation of the paper can be improved
  - ICM is not defined when it's firstly used in page 2.
  - There is a broken citation in the 3rd line of the 3rd paragraph of Section 3
  - The first two figures in Section 5 is not numbered.
    - The order of all methods in the two figures is weird/inconsistent. You should keep a consistent order for all baseline + variants.
    - I also suggest better a better way to label them: You can colour each baseline model with a colour and fill the variants with (no filling, dots, dashed lines).

Questions
- For Figure 2, how do I know the traversals are supposed to give low errors? i cannot tell if the traversal is along any disentangled dimension from the images.
summary_of_the_review: The paper proposes a well-motivated and (seemingly) theoretically justified way to regularise representation learning in VAEs.
The results also support the main claim of the paper.
Therefore I believe this paper is beneficial to the community.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This work proposes the notion of "interventional consistency" as a beneficial property learned representations should have and introduces a regularization term to enforce it in autoencoders. Moreover, the authors introduce an "explicit latent causal block" that allows learning a structural causal model (SCM) over the latent factors. The experimental section argues that both the regularization and the "causal layer" improve interventional consistency. 
main_review: - The paper is motivated by the goal of "causal representation learning" and explicitly mentions it does not address the important question of identifiability. The most important problem is that there is no justification as to why the authors' notion of interventional consistency has anything to do with what researchers in the past have meant by a "causal representation" (e.g. disentangled, modular, robust, ...etc.). 
- I found the paper hard to read even though the ideas proposed are rather simple. The paper is framed within the larger context of causality, but I am not sure this is the best way to present the contribution, which might explain my confusion. 
- The motivation for the "explicit causal latent block" is unclear. Why does it make sense to have the output of the encoder be the noise of the SCM over $Z$?
- Some very important quantity like $\mathcal{L}\_\text{INV}$ and $\mathcal{L}\_\text{EQV}$ (which are the actual regularizer proposed) are defined only in the appendix. Important contributions should be presented in the main text. Also, the explanations for these central quantities lack clarity.
- It is not clear why learning a SCM in the latent space of MNIST makes sense. For instance, I can't imagine the shape of a digit causing its thickness. 
- Figure 6 shows three latent traversals for a baseline and two variants of their approach and claim that their approach yield more modular traversal. However, it is not clear what is meant here by modularity and I do not see how the latent traversals of their approach are better.
- The experiments show that indeed the regularizer improves "interventional consistency". But, as I said earlier, it is not clear how this relates to "causal representation learning" or why it is something beneficial regarding disentanglement, robustness, modularity etc. 

Minor:

- The notation is sometimes imprecise: for example, in the equation between (2) and (3), it is not clear what is meant by $Z'\_{j \not= i}$. Is it all components of $Z'$ that are not $i$? If so the decomposition is wrong. Is it $Z'\_{j<i}$ for some ordering of the nodes? If so what is the ordering? This point matters because (4) goes on specifying what is meant for the "ICM principle to be preserved by the response map" based on this above equation.
- The notation is sometimes hard to parse: for instance, Section 2.1 has a few double "hat" like this: $\bar{\tilde{Z}}$. 
-  Table 1 and 2 have no error bars, making it hard to asses whether the gaps between different methods are significant or not. 

summary_of_the_review: Given the points raised above, I cannot recommend acceptance. My main concerns are with (i) the lack of motivation for both the "interventional consistency" and the "explicit causal latent block", as well as (ii) the (sometimes significant) lack of clarity. The paper did not make a convincing case that "interventional consistency" is a valuable tool for representation learning.
review_title: null

Meta-review: The paper introduces the notion of interventional consistency of a representation learned using autoencoders, which is claimed to be a desirable property for disentanglement. The reviewers agree that the contributions are novel and relevant, but they also found the paper hard to follow due to a lack of clarity and motivation. Further, they considered the underlying assumptions very strong and possibly hard to find practical instances where they may hold (e.g., the assumption that statistical dependencies in the prior are preserved by the response map). The reviewers also noted that some real-world examples showing the interventional consistency would be helpful. 

After all, the paper contains interesting ideas and we would like to encourage the authors to pursue this line of work. Still, the paper in its current form is not ready for publication. We encourage the authors to address the reviewers' comments explicitly in a future version of the manuscript.
Decision: Reject