Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: **Property controllable variational autoencoder via invertible mutual dependence**

The paper proposes an approach for learning disentangled latent representation in the VAEs where parts of the latent space should correspond to observed properties of interest of the data so that these could be controlled at generation. They replace the total correlation regularization term of Chen2018 with more appropriate group- and property-wise KL terms promoting independence between the *relevant* (w) and the *remaining* (z) part of the latent space and inner independence between dimensions of the w part of the space (dropping the term promoting independence in the z part.) More importantly, they link the data property y with the latent space w by a learned mapping.

The is another contribution to the rather extensive literature on disentanglement of latent representation of VAEs. It builds on previous papers in the area, mainly Chen2018 and Klys2018. The change in the ELBO as compared to Chen2018 (TC vs group/property-wise) is well explained and motivated though seems rather trivial.
The introduction of the invertible mapping between y and w is in my view more interesting and important. Though somewhat contrived at first reading (some improvements in the text could help, see comments), it seems to correspond well to what the model shall achieve. 
This  as is also demonstrated in the experimental section focusing mainly on the dSprites and QM9. The paper would benefit from a more extensive experimental evaluation (another standard dataset such as CelebA) though I understand this may be difficult to do given the deadlines and computational resources and therefore do not consider as critical.
The paper is generally well written though there are places which are difficult to parse and comprehend and suggest last minute drafting. These shall be made clear for the final version (see comments below).
The problem of correlated properties is addressed by a rather trivial extension of the basic model and requires the user to provide to the model groups of correlated properties. I feel this deserves a lot more attention and search for better solutions is a possible direction for future work.

I recommend to accept the paper as I find it interesting for the community, addressing a lively area of research via a new approach which, though not dramatically innovative, seems to yield the desired results. I do have a few comments as to the clarity of the statements in the paper which I hope will be addressed by the authors during the rebuttal period and in the final version.

*Comments / questions:*
* I find calling the mapping between y and w *invertible* as misleading. The mapping $w \to y$ is defined as stochastic via the learned generative distribution $p(y | w)$. Each w thus corresponds to multiple possible values of $y$.

* eq (2): please state clearly (in equations) the independence and conditional independence assumptions leading to this result

* eq (4) holds only in expectation over $p(x)$, right? ($E_{p(x)}$ in front of the DKL in the left hand side.)

* p4 before eq (5): "Roughly disentangling all pairs of latent variables without emphasis could lead to poor convergence and incur exponential number of pairs among properties and latent variables for such enforcement." What do you mean? Please explain / elaborate / rephrase.

* p4, "... no strict assumptions of parameters for $p(y_k)$ and $q(w_k |y_k)$". $y$ does not exist in the inference model so what does $q(w_k |y_k)$ refer to?

* p4 "The most straightforward way to do this is to model both the mutual mapping between $y_k$ and its relevant latent variable $w_k$." Both mappings $y_k \to w_k$ and its inverse $w_k \to y_k$? Or what do you mean?

* p5, para5 "Thus, we utilize the Naı̈ve Monte Carlo approximation based on a mini-batch
of samples to **underestimate** $q(z), q(w)$ and $q(w_k)$, as described by Chen et al. (2018)" Underestimate? What do you mean?

* p6: you say you use the normalized mutual information between $w_k$ and $y_k$. Please explain how you define this and how you calculate in practice.

* p7, fig3 "... when traversing three latent variables in subset z ...". There are only 3 latent variables z? If more, how did you decide which shall be traversed?

* p8, tab3: How do you predict the property $y_k$ from a molecule? You first infer w and z and then use the mean of p(y | w) as prediction? Or you sample y from p(y | w)? Please elaborate.

* p7, tab2: Do I read correctly that MSE of cLogP is bigger from the PCVAE (corr) then from the standard PCVAE? What does this say about the advantages of PCVAE (corr)?

* experiments over QM9: are all your evaluation metrics calculated only over valid molecules? Usually, generative models for molecules are evaluated using the validity, unicity and novelty scores. It would be instructive to complement your results with these.


*Minor comments / questions*
* Please provide references to models displayed in Fig1(a)-(c).

* You use the term *correletad* a lot. I'm guessing here you mean any sort of dependence, not just linear?

* p2: "Directly enforcing such mutual independence inherently between all pairs of latent variables incurs exponential number of pairs among properties and latent variables for such enforcement." Exponential number of pairs? What do you mean?

* p2, para3: "...  as these have been shown to be relatively resilient with respect to the complex variants involved (Bengio et al., 2013)." Complex variants? What do you mean?

* p3, para2:  $y = {y_k \in R}_{k=1}^K$. Is this the same as $y \in R^K$?


*Typos or phrasing improvements needed:*

* p1, para1: "**Knowing such properties is crucial** for many applications that depend on being able to interpret the data and control the data generation **to yield the desired properties**." ???

* p2. para1: "Also, many cases require to generate data with properties of which the values **are?** unseen during training process."

* p3 just before eq (2): $\log P_{θ,γ} (x, y, w, z)$ (capital P?)

* p5, para3: "As stated in the third challenge in Section 1, there are usually several groups of properties involved in **formatting**? data x ..."

* p6, para5: "... each encoded latent variable $w_k$ and the property $y_k$ **that is?** assigned to it ..."

* p7, caption Tab2: "...  (PCVAE (cor) denotes the **extensive**? model for correlated propertie"

review_title: Another tweak on disentanglement in VAEs with continous properties for supervision

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper presents the new Property-controllable VAE (PCVAE) to inductively bias the latent representation to capture explicit data properties. Towards this, the paper proposes group-wise and property-wise disentanglement terms. The group-wise disentanglement term separates two subsets of the latent representation. In contrast, the property-wise disentanglement term promotes the disentanglement of an individual component of one of the subsets in the latent space. Furthermore, the model enforces individually disentangled latent subset to account for the given property of the dataset via invertible constraint. The presented work is evaluated on two datasets from two domains. Below I present some pros, cons, suggestions, and clarifying questions for the authors. 

- The paper is clear to understand. The presented experimental studies are clear and demonstrate the efficacy of the model.  For instance, the precision with which the proposed model controls the property generation is impressive. Qualitatively this is demonstrated by Fig 4. Are these demonstrated on the train or the test set? 

- For both group-wise and property-wise disentanglement terms, the TC term is considered. Can the authors clarify why they claim them to be novel? Can authors present training statistics to give insight into the model's optimization in relation to these up-weighted TC terms? 

- dSprites is considered to be a relatively simpler dataset for disentanglement. If authors wanted a dataset with ground truth generative factors in the dataset, they could consider 3DShapes (Burgess & Kim, 2018), which would have more generative factors and is more challenging than dSprites. Also, 3DShapes is a widely considered dataset beside dSprites for disentanglement related study.  

Burgess, C, & Kim, H. (2018). 3D Shapes Dataset. https://github.com/deepmind/3dshapes-dataset/. 

- The literature review in the paper seems broad. I would encourage authors to discuss some recent approaches (e.g., Gyawali et al., 2019) that disentangle latent space into two subsets with one being related to certain property of the dataset. 

Gyawali, P. K., Horacek, B. M., Sapp, J. L., & Wang, L. (2019). Sequential factorized autoencoder for localizing the origin of ventricular activation from 12-lead electrocardiograms. IEEE Transactions on Biomedical Engineering, 67(5), 1505-1516.

Minor comments:
- Please proofread your manuscript carefully to avoid grammatical errors. e.g., "latent presentation".   
review_title: VAE for capturing explicit data properties. 

final_score: rating: 6: Marginally above acceptance threshold
review_content: **Update**

I appreciate the effort by the authors to clarify some of the issues, most of which are addressed in the rebuttal, so I will raise my score to 6.
I still feel like the $I(w, y)$ part needs to be dealt with a bit carefully, especially there is a invertible mapping between the two on the generative side. The simple graphical model seems like $w \leftarrow x \rightarrow y$, where left is encoder and right is data generation procedure. 

**Summary**
The paper proposes property controllable VAE, with the aim to learn certain latent variables correlated to the property and are disentangled. This is done by variational inference + total-variation based disentanglement terms, with the aim to "control the properties".

**Strengths**
Empirical improvements over similar VAE approaches, such as Semi-VAE and CSVAE, showing the ability to learn and control the properties.

**Weakness**
 - The paper has several important details that are missing, and seems not self-contained enough to reproduce if the reader is not familiar with the disentanglement literature (see questions below).
 - I don't quite understand the motivation behind the method. If you want to control the value y (which is assumed to exist for all x), then why not replace w with y entirely (i.e. invertible network is identity) and operate directly on y? On inference side, predict $q(y | x)$, on generation side use empirical distribution (or some simple KDE) on y to get $p(y)$. 
- Granted, it is possible that $y_i$s can be dependent, whereas you want $w_i$s to be independent; but it seems that Section 3.2.4 adds the assumption that $y_i$ are independent because you can group dependent variables?

**Questions**
- How are the total correlation objectives evaluated? For example, the explicit log-likelihood values for either $\log q(w)$ or $\log q(w_i)$ cannot be efficiently computed since that is the aggregate posterior? Is this implemented as an adversarial objective?
- What is the difference between PCVAE and PCVAE_tc (write down the equations)? Performance-wise they seem relatively similar.
- How does the invertible network work when y is discrete? If I made any error in w, then it maps to some y value outside of the sample space?
- How do you evaluate $I(w, y)$, and for which distribution is this defined? This confusion is because in reality there are two "worlds" concerning both variables (generative and inference directions) and because you have a invertible function between w and y (so it seems that optimizing MI is useless since you don't lose information anyways?).
- Why do we use avgMI as a metric? Do we expect the optimal MI value to be 1? If that is the case, then how do we get an avgMI of 1.004 for Semi-VAE in the first place?

review_title: Nice experiments, but not self-contained enough to readers who are not familar with disentanglement literature.

final_score: rating: 6: Marginally above acceptance threshold
review_content: To encourage disentanglement in the latent space of a variational autoencoder (VAE), the authors propose to learn two sets of latent z and w: the dimensions of w are independent of each other and each dimension w_i maps to a known ground truth generating factor y_i. Latent z captures all the other factors. The well studied Total Correlation regularisation is used to enforce the independence of z and w, and the same is used to enforce the independence of the dimensions of w. Each dimension is learned to predict a corresponding ground truth factor. The key difference from the previous approach is the use of invertible and Lipschitz smooth mapping to learn monotonic mappings from w to y.

Pros:
+ The proposed regularization is shown to be useful for controlled manipulation for dsprites and QM9 dataset and improves upon comparable baselines.
+ Can be applied when the ground truth factors are continuous-valued.
+ Ability to handle the case where the ground truth factors are correlated. 

Cons + Questions: 
- Ablation studies on the use of spectral normalization will be helpful. 
- The choice of Gaussian distribution for modeling the relationship between latent and ground truth factors could be elaborated upon.
- A more complex dataset at least in the case of natural images with large number of ground truth factors with correlated latent structure will be useful. In such cases how does the choice of dimensions of z and w impact the learning and their independence structure?


Typos:

Sec 2: earning -> learning

Sec 3.2.1 likely-hood -> likelihood

Fig 1(d) Y is missing from the model.

\phi is missing from RHS of eq 4.

review_title: Well motivated approach and good results on limited datasets

Meta-review: This paper proposes a new approach to learning deep generative models with induced structure in the latent representation. All four reviewers gave the same score of 6 to this paper, showing a consensus that the paper is above the bar for acceptance. The authors did a commendable job of detailed replies to reviewer comments, which as R1, R3, and R4 all note has improved the clarity and quality of the paper, addressing their concerns.
Decision: Accept (Poster)