Property Controllable Variational Autoencoder via Invertible Mutual Dependence
Authors: Xiaojie Guo
Yuanqi Du
Liang Zhao
Venue: ICLR 2021 Poster (2021)
Keywords: deep generative models
interpretable latent representation
disentangled representation learning

Abstract: Deep generative models have made important progress towards modeling complex, high dimensional data via learning latent representations. Their usefulness is nevertheless often limited by a lack of control over the generative process or a poor understanding of the latent representation. To overcome these issues, attention is now focused on discovering latent variables correlated to the data properties and ways to manipulate these properties. This paper presents the new Property controllable VAE (PCVAE), where a new Bayesian model is proposed to inductively bias the latent representation using explicit data properties via novel group-wise and property-wise disentanglement. Each data property corresponds seamlessly to a latent variable, by innovatively enforcing invertible mutual dependence between them. This allows us to move along the learned latent dimensions to control specific properties of the generated data with great precision. Quantitative and qualitative evaluations confirm that the PCVAE outperforms the existing models by up to 28% in capturing and 65% in manipulating the desired properties.

Published as a conference paper at ICLR 2021
PROPERTY CONTROLLABLE VARIATIONAL AUTOEN- CODER VIA INVERTIBLE MUTUAL DEPENDENCE
Xiaojie Guo
Yuanqi Du
Liang Zhao' Department of CS Emory University Atlanta, GA 30322, USA
Department of IST George Mason University Fairfax, VA 22030, USA xguo70 gmu edu
Department of CS George Mason University Fairfax, VA 22030, USA ydu gm edu
liang. . zhao@emoryyyrrooo edu
ABSTRACT
Deep generative models have made important progress towards modeling com- plex, high dimensional data. Their usefulness is nevertheless often limited by a lack of control over the generative process or a poor understanding of the latent representation. To overcome these issues, attention is now focused on discovering latent variables correlated to the data properties and manipulating these proper- ties. This paper presents the Property-controllable VAE (PCVAE), where new Bayesian model is proposed to inductively bias the latent representation using explicit data properties via novel group-wise and property -wise disentanglement terms. Each data property corresponds seamlessly to a latent variable, by enfore- ing invertible mutual dependence between them. This allows learned latent dimensions to control specific properties of the generated data with great precision. Quantitative and qualitative evaluations confirm that the PCVAE outperforms the existing models by to 28% in capturing and 65% in manip- ulating the desired properties. The code for the proposed PCVAE is available at/attps: //tttsu com/ Xquo7/PCVAB
move along the
INTRODUCTION
Important progress has been made towards learning the underlying low-dimensional representation and generative process of complex high dimensional data such as images (Pu et al., (016)). natural languages (Bowman et al. 2016), chemical molecules (Kadurin al. 2017: Guo et al. 2019) and geo-spatial data (Zhao) 2020) via deep generative models. In recent years, surge of research has developed new ways to further enhance the disentanglement and independence of the latent dimen- sions, creating models with better robustness, improved interpretability, and greater generalizability
with inductive bias (see Figures MTa) and [bb) (Kingma et al. [2014: Kulkarni et al., 2015 Creswell
et al. 2017) without any bias (Higgins al. 2017;] Chen et al. 2018 Kumar al.) 2018). Al- though it is generally assumed that the complex data is generated from the latent representations, their latent dimensions are typically not associated with physical meaning and hence cannot reflect real data generation mechanisms such as the relationships between structural and functional charac- teristies. critical problem that remains unsolved is how best identify and enforce the correspon- dence between the learned latent dimensions and key aspects of the data, such as the bio-physice properties of molecule. Knowing such properties crucial for many applications that depend on being able to interpret and control the data generation process with the desired properties.
In an effort to achieve this, several researchers (Klys et al., 2018 Locatello al. 2019b) have sug- gested methods that enforce subset of latent dimensions correspond to targeted categorical prop- erties, as shown in Figure Tc). Though the initial results have been encouraging. critical challenges remain unsolved such as: (1) Difficulty in handling continuous-valuee properties. The control imposed on data generation limits existing techniques to categorical (typically binary) propertics, to enable tractable model inference and sufficient coverage of the data. However, continuous-valued properties (e.g., the scale and light level of images) are also common in real world data, while their model inference usually can be easily intractable. Also, many cases require to generate data
Cooresponning author: liang.zhan @ emory.edu

---

Published as a conference paper at ICLR 2021
(a) CondVAE
(b) CondVAE-inle
(c) CSVAE
(d) POVAE (ours)
Figure 1: While most existing models (e.g.. Sub-figures (a) (Kingma et al. 2014 Kulkarni et al. 2015 and (b) (Creswell et al.] (017)) do not explicitly learn the correspondence between latent dimensions and data properties, some recent work (Sub-figures (c) (Klys al, 2018) and (d)) has started to explore this. The generative model (right) and its model inference (left) are shown each sub-figure. Dotted arrows represent the enforcement of independence and double arrows represent the invertible dependence between two variables. refers to data, and w refer to two subsets of latent variables, and refers to the properties.
with properties of which the values are unseen during training process. This cannot achieved by conventional techniques such conditional models without making strong assumption the model distributions. (2) Difficulty in efficiently enhancing mutual independence among latent the properties. This problem requires ensure that each
variables relevant and irrelevant property is only correlated to its corresponding latent variable(s) and independent of all the oth- ers. Directly enforcing such mutual independence inherently between all pairs of latent variables incurs quadratic number of optimization efforts. Hence an efficient way is imperative. (3) Difficulty in capturing and controlling correlated properties. It is feasible that several independent latent variables can capture multiple independent properties. But when the properties are correlated, they cannot "one-on-one" mapped corresponding independent latent variables anymore. However, correlated properties commonly found in formatting real world data.
To solve the above challenges, we propose a new model, Property-comtrollable VAE (PCVAE), where new Bayesian model is proposed to inductively bias the latent representation using explicit data properties via novel group-wise and property-wist disentanglement terms. Each data property is seamlessly linked the corresponding latent variable by innovatively enforcing invertible mutual dependence between them, as shown Figure Md). Hence, when generating data, the corresponding latent variables manipulated to simultaneously control multiple desired properties without influencing the others. We have also further extended our model to handle inter-correlated properties. Our key contributions are summarized follows:
A new Bayesian model that inductively biases the latent representation using explicit real data properties is proposed. A variational inference strategy and inference model have been customized ensure effecttive Bayesian inference.
Group-wise and property-wise disentanglement terms are proposed to enhance the mutual inde- pendence among property, relevant and irrelevant latent variables.
The invertible mutual dependence between property- latent variable pair is achieved by enforcing an invertibility constraint over residual-basee decoder.
The quantitative and qualitative evaluation performed for this study revealed our PCVAE outper- forms existing methods by up to 28% capturing and 65% manipulating the desired properties.
RELATED WORKS
Disentanglement Representation Learning. An important relevant area of research disentan- gled representation learning (Alemi et al., 2017] Chen et al. 2018 Higgins et al.) 2017 Kim & Mnih 2018), which structures the latent space by minimizing the mutual information between all pairs of latent variables. The goal here is to learn representations that separate out the underlying explanatory factors that are responsible for variations the data, as these have been shown be rel- atively resilient with respect to the complex variants involved (Bengio et al.) 2013 Ma et al. 2019 Guo et al. 2020), and thus can be used to enhance generalizability as well as improve robustness against adversarial attack. As noted by Locatello et al. (2019a), it is impossible for disentangled representation learning to capture the desired properties without supervision and inductive biases.
Learning latent representations via supervision. This ensures that the latent variables capture the desired properties though supervision, generally by directly defining properties as latent variables in the model (Locatello et al. 2019b). Unfortunately, apart from providing explicit variable for the labelled property, this yields no other easily interpretable structures, such as discovering latent variables that are correlated to the properties, the model proposed in the current study does. This
2

---

Published as a conference paper at ICLR 2021
is also an issue with other methods of structuring latent space that have been explored, such as batching data according to labels (Kulkarni et al. |2155] 20a5g et al. 2020) or using a discriminator network in non-generative model (Lample et al. 2017). Some researchers addressed this problem by introducing the architecture bias through a two-way factored autnencoder and realize the super- vision based a pair-wise contrastive loss (Gyywalli et al. 2019). Other researchers addressed this problem by linking latent variables with observed labels through adversarial learning (Creswell et al. 2017 Edwards & Storkey, 2015; Ganin al., 2016 Mathieu et al. 2016). The most relevant work for our purpose is CSVAE (Klys et 2018)) where subset of latent variables are correlated with binary properties via an adversarial learning. All the above works can not handle multiple continuous-valued properties due to their strict assumptions on the distribution of properties.
Data manipulation and generation, Here, trained machine learning models are utilized to manipu- late and generate data in controllable way with the desired properties, which is especially useful for applications in the image domain. Several works have specifically considered transferring attributes in images, which is the same goal as that in the CASVE. These earlier works (Zhou et al. 2017: Xiao ct al. 2017 2018) all transfer attributes from a source image onto a target image. These models can only perform categorical attribute transformation between images (e.g., "splice the beard style of image A onto image B"), but only through interpolation between existing images. Once trained, our proposed model can generate objects with any value of certain property (either observed or unobserved during training) that can be encoded in the subset of latent variables.
3 PROPERTY CONTROLLABLE VAE
3.1 PROBLEM FORMULATION
Suppose are given dataset D where each data instance is (xx, y) with E R and y [yk B] to represent K properties of interest of x. For example, if x is molecule, then we may have properties of interest, such as cLogP and cLogS. We assume that the data (z. are generated by some random process from continuous latent random variables (z, w). Each variable w controls one of the properties of interest in y, while the variables in control all the other aspects of a.
Our goal is to learn such a generative model involving (z, y) and (2,w). where the subset of vari- ables (i.e., 2) are disentangled from subset w, and the variables inside w are disentangled from each other. Once this model has been learned, then we can expect different elements of variables in w to control different properties of interest, which is highly desirable goal for many data generation downstream tasks. For example, we may want to decrease the value of a specific property (e.g.. pro- tein energy) by changing the value of the corresponding element in It is also possible directly set a desired property value (e.g., the mass of a molecule) and then generate the corresponding x with this target value (i.e., molecule with the target mass value).
3.2 OVERALL OBJECTIVE
In this section, we first introduce the Bayesian variational inference of PCVAE. Then we introduce the griup-wise and property- wise disentanglement terms as part of the overall objective. Following this, an invertibility constraint is introduced to enforce mutual dependence between each property- latent variable pair. At last, PCVAE extended to capture and control multiple correlated properties.
3.2.1 BAYESIAN VARIATIONAL INFERENCE OF PCVAE
The goal in Section 3.1 requires us to not only model the dependence between and (w, 2) for latent representation learning and data generation, but also model the dependence between y and w for property manipulation. We propose to achieve this by maximizing a form of variational lower bound on the joint log likelihood p(2, y) of our model. Given an approximate posterior ( w ar, y), we can use the Jensen's 's equality to obtain the variational lower bound of p(æ, y) as:
log v) = log [r( w, 2)/((( x/ w|a, [r,y)]] v)]
(I) The joint likelihood log p(x, y, w. 2) be decomposed as log p(z, ø12, w) + logp(z, w). We have two assumptions: (1) w only encodes the information from y. namely, x and y are condi- tionally independent given w (i.e., x 1 y/ w)): (2) is independent from and y, namely = 1 w
Le p(x, w, =)/9(=, w w)]
3

---

Published as a conference paper at ICLR 2021
and 1 y. which is equal to y 1 2/w (see derivation in Appendix A.3 First, based the two assumptions, we can get a 1 yl(2, w) (see derivation in Appendix A.4. Thus, we have log p(z, 2|2, w) = log p(1)2, w) log plyl2, w). Then, based on the assumption y 1 2/w, we can have (ug w) = log p(ylw) Then we get log p(x, 212, w) = logp(<=|2, w) + log pogp(yle (v) To explicitly represent the dependence between and (z, w) well as the dependence between y and w, we can parameterize the joint log- log-likelihood as log Po., (o,, w, z) with and as:
Pe. (2, y, w, =) = logpo(>1=,, w) logp(z, w) + log logp-(ylw).
Given the condition that a parameterized (2 w/ 2 y) go w|x) = (o(2(1)9 (w|x) the information on is included in æ), by taking Eq. into the above variational lower bound term in Eq. [ we obtain the negative part an upper bound on log po,, (x, y) (as shown in the right sub- sub-figure of Figure [d)):
( (since
4 = Bo Logg
w)] Begtcchsllooppp((ll))) + Dxz (1,(2, w w[=]]l((,,,) w))
(3)
This gives us the proposed Bayesian variational inference of PCVAE. The detailed derivation of Eq. [ can be found in Appendix [A.]] As there K properties of interest in y which signed and disentangled by the latent variables in w, the second term in Eq. 3 can be detailed as Σx (nola Pyy (yul [.
3.2.2 GROUP- WISE AND PROPERTY-WISS WISE DISENTANGLEMENT
Considering that the above derivation is conditional on two requirements: (1) z is independent from and y and (2) the variables in independent from each other, while in practice minimizing the above objective L1 will not imply that our model will satisfy these conditions. We therefore propose further penalize the novel Group-wise- and Property-wise Disentanglement terms.
We first decompose the KL (Kullback-Leibber) divergence term in Eq. 3]as:
Exxs w| x [))([ w)]] Dx (2, w. ll ((=, w))(x))
+ Duu(a(=, w) II FI., ((x))(w)))) + >, Daz(a(A) I())))) + Σ, PKc(a(w,) Il p(w,))
(4)
total correlation term
The second term in the right of the above equation is referred to as the total correlation (TC), as de- fined by Chen et al. (2018), which is one of many generalizations of mutual information to more than two random variables. The detailed derivation of this decomposition can be found in Appendix A The penalty on this TC term forces the model to find statistically independent factors in the data distribution. A heavier penalty on this term induces a more disentangled representation among all variables in both z and w. but as stated in our problem formulation, we only require that (1) vari- ables in are disentangled to capture different properties, and (2) although is disentangled from w, the latent variables inside do not need be disentangled from each other. Roughly enforcing the disentanglement between all pairs of latent variables in w and z, as done by the existing TÖ term, can incur at least quadratic number of redundant optimization efforts and could lead to poor convergence. Thus, we further decompose and analyze the TC term as:
Dxı(a(z, w) II II,, ((x))(ws)) DKIla(Z, w) II ((=)g(w)) Drr(a(w) I. ((::)
group-ntir disentaglemment
property wise disentungloment
+ Dxx(a(=)
I. a(=:)).
(5)
The first term in the right part of above decomposition enforces the independence between the two subsets of latent variables z and w, which we term group-wise disentanglement. The second term enforces the independences of variables inside w, ensuring that each latent variable can only capture the information of the single property assigned to it. We term this property-wist disentanglement Imposing a heavy penalty these two terms can satisfy the two requirements mentioned above. We can now obtain the second part for the objective of PCVAE by introducing the coefficient p as:
Lg =
+ pD pDxx(((w))) I. ((w)))
3.2.3 INVERTIBLE CONSTRAINT FOR PROPERTY CONTROL.
As stated in the problem formulation, an important goal for our new model is to generate data point x that retains the original property value of a given property ys with great precision. More importantly, there should be no strict assumptions of parameters for p())) and qf ((wx) The most
lyx).
4

---

Published as a conference paper at ICLR 2021
straightforward way to do this is to model both the mutual dependence between yr and its relevant latent variable wa, namely, ((wglBk)) and Wy), which, however, could incur double errors in this two-way mapping. To address it, we innovatively propose instead an invertible function that mathematically ensures the exact recovery of we given yı based on the following deduction.
In the above objective, we only explicitly model the conditional distribution of | wwx). hence, to achieve the precisely control of property via 2 and w, which is necessary to generate x with a certain property yı m, we need to maximize the probability that yk m as follows:
x ~ po(xlz,,)
2, W - arg max; --<=),,-( (yk = m ml,,w),
which is equal to:
po w), p(=), Wjitk ~ p(w)); Wkk - arg max P-, (yk ml/w)
(8)
where Wy can be determined as follows (N in the followings denotes Gaussian distribution):
Wk arg max -p(w)) Py (y( ml/wk)
W. arg max p( log Log.N(v mlfa(wnin))
(9)
= We arg max max p(wo)) (m fa fu((w:n))) = Wk fx (m)
Therefore, by learning an invertible function fx (w(wx:)) 7) from Wi. to the expectation of to model Pry (v% /wg), we can easily achieve the desired precise control of the property. The above derivation are based on the assumption that y is a continuous-valie when y is discrete property, detailed in Appendix [A.2] To learn invertible function (wxx?)) we propose to leverage an invertible neural network. Inspired by the invertible ResNet (Behrmann jet al. 2019), we decompose the function (wg; 7) as fa 7) = f((wwi)) 7) + wg- As proved by Behrmann et al. (209)) the sufficient condition that fx(wa;7) is invertible is that Lip(fa) < 1, where Lip( fi) is the Lipschitz-constant of f& (wg:7).
It can also be extended into the situation
Thus, the overall objective of the proposed PCVAE finally formalized as: minCi + aC2 with 0,0.7
subject to Lip(Fa) for all E K, where a is the coefficient parameter.
Remark (Monotonic relationship of property-latent variable pair). Given the condition that (wwii) is invertible and continuous (Lip(fs) is less than 1), fi fi((wi)) is thus a monotonic function. This is very important to increase (or decrease) the value of property ys by increasing (or decreasing) Wh. especially when the desired value of property is not available.
3.2.4 GENERALIZATION OF HANDLING CORRELATED PROPERTIES
As stated in the third challenge in Section there are usually several groups of properties involved in describing the data and each group has several properties. These different groups are independent, but the properties within the same group are correlated. Thus, we can further generalize the above objective framework to handle the correlated properties inside the same group.
The notation for y4 is extended to yı (yJ,k E R) signifying that there are M₁ correlated prop- erties inside the k-th property group. The properties inside the same property set yk are correlated, while the different property sets (e.g. Jp and yı are independent. Similarly, the notation for w; is extended group of latent variables to control the corresponding property set. For the properties inside the same group, we assume all depend on the same group of latent variables Wp. as
pluj, (wx) (M ((.)),
(10)
where S& (wxi [i]] denotes the j-th element of the output of (wws)). Thus, the second term in Eq. can be generalized as Σ ΣS Mr
[ga(w|=)[lopp p. (ww,)
3.3 NEURAL NETWORK ARCHITECTURE OF PCVAE
As shown in Figure dd), there is encoder (left-hand side of Figure I(d)) that models the dis- tribution q(2, w (x) and two decoders (right-hand side of Figure (d)) that model the distribution w) and p(= (2, w). To implement the encoder and decoders in the first objective (i.e., Li) use Multi-penn caltiipprreetionss (MLPs) Convolution Neural Networks (CNNs) or Graph Neural Networks (GNNs) to represent the distributions over relevant random variables.
5

---

Published as a conference paper at ICLR 2021
To implement the second part L2, it is necessary to calculate the group- wise and property- disentanglement terms. Noting that the calculation of the density ((). q(w) and (w;)) in group- wise and property- wise disentanglement terms depends on the entire data space. As such, it is inaccessible to compute it exactly during training. Thus, as the same operation conducted by Chen et al.] (2018), we utilize the Naive Monte Carlo approximation based underestimate q(z), ((w) and ((wx). The detailed operation is described in Appendix B.3
mini-batch of samples to
To implement the invertible constraint and model the distribution of p-, ( we utilize MLPs to model the function (+. (.). Since the function f&(.) modeled by MLPs a composition of contractive nonlincarities (c.g.. ReLU, ELU, tanh) and linear mappings, based on the definition of Lipschitz- constant we have Lip(fix fi)) 1 if I| Wi l12< for ( E L, where Wi refer to the weights of the 1-th layer in fa. || ll2 denotes the spectral norm, and L refers to the number of layer in the MLPs. To realize the above constraint on weights of neural networks, we propose to use the spectral normalization for each layer of MLPs, as introduced by |Behrmann et al. (019)).
3.4 PRECISELY PROPERTY CONTROLLABLE GENERATION
Our proposed model can be applied to an important downstream task, namely precisely property controllable generation. Given the value of required property yk. the goal of property controllable generation is to generate a data z which holds the same value as this desired property. To achieve this, three steps are conducted: (1) infer the value of Wtg based on the well-trained neural network (+) and the given property Mı via fixed-poine iteration by following wik i+1 := y/s - fe(ww) 7). where wi is the updated latent variable at the i-th iteration step and W2g yks (2) randomly sample the
values of and the remaining variables in w from their prior distributions (i.e., Gaussian distribution)
to obtain all the latent variables; and (3) generate a data x using the decoder based on the latent variables that are inferred from the previous two steps.
EXPERIMENT
This section reports the results of the qualitative and quantitative evaluation carried out to test the performance of the proposed model on two datasets in two domains, namely images and molecules. All experiments were conducted on a 64-bit machine with an NVIDIA GPU (GTX 1080 Ti, 11016 MHz, 11 GB GDDR5). The architectures and hyper-parameetrs can be found in Appendix B The code for the proposed PCVAE is available at/https: http::::///gthbbbb com/ X
4.1 EXPERIMENT SETUP
4.1.1 DATASETS
The dSprites dataset (Matthey et al. 2017) consists of 2D shapes procedurally generated from ground truth independent semantic factors. The factors that explored as properties of data in this experiment are scale, and the x and y positions (mentioned as x pos and pos) of a sprite. All possible combinations of these semantic factors are used for generating total of 730k images, where 580k/ / 146k is the training/testing set split. The 3Dshapes dataset (Burgess & Kim 2018) consists of 3D shapes procedurally generated from ground truth independent semantic factors. The factors that are explored as properties of data in this experiment are wall hue, floor hue and scale. All possible combinations of these semantic factors are used for generating total of 480k images, where 390k/ 90k is the training/testing set split. The QM9 dataset (Ramakrishnan et al., 2014) consists of 134k stable small organic molecules, where 120k/20X 20k is the training/testing set split.
4.1.2 COMPARISON METHOOS
In order to validate the superiority of our proposed model in capturing and manipulating the prop- erty during generation, we compare the performance of PCVAE to those achieved by three com- parison models that are most relevant to our problem: (1) VAmi-VAE (Locatello et al.) 2019b) is semi-supervised model that enforces the value of each latent variable to be equal to the value of each property. Here we utilize all the labels for supervision for fairness; (2) CSVAE (Klys) et al., 2018) is VAE-based model that utilizes mutual information minimization to learn latent dimensions associated with only binary properties. Here we adjust this model to handle contin- uous property by assuming Gaussian distribution; and (3) PCVAE te is baseline model that

---

Published as a conference paper at ICLR 2021
holds the same inference model and property controlling strategy as PCVAE, of which the pro- posed group-wise and property-wise disentanglement terms are replaced with the TC term, namely, ( w wIll [[; ((2))9(w)))) wy), as proposed in 3-TCVAE (Chen et al. 2018). It is used as an ablation study to validate the effectiveness of the proposed group and property-wis disentan- glement. (4) PCVAE is a baseline model that holds the same architecture and disentanglement terms PCVAE except for the spectral normalization. It is used an ablation study validate the effectiveness of spectral normalization.
4.2 EVALUATION FOR DISENTANGLED LATENT VARIABLES
In this section, we explore (1) whether each variable wy successfully captures the information of the property that is assigned to it through supervision, and (2) whether the subset 2 of latent variables is independent from the properties.
Quantitative evaluation. We calculate the normalized mutual information between each encoded latent variable Wg and the property yı: that is assigned it, as well as the average mutual informa- tion between latent and each 1/x- Figure " shows the mutual information heat map by each model on (Spritessresults on QM9 datasets can be found in Appendix g The element in the row of Zoog denotes to the average of all the mutual information between and each property. In addition, we utilize the metric MvgM/ proposed by Locatello al. (2019b) to show an overall quantitative com- parison between different methods, as shown in Table 1 The proposed PCVAE achieves the least avgMI of 0.257, which demonstrate its strength in enforcing the relationship between wy and yis- These results also validate the effectiveness and necessity of the proposed group-wise and property- wise disentanglement term, as PCVAE outperforms the baseline model PCVAE.tt on avgMI by around 28%. Though CSVAE shows an good performance in disentangling and w, its latent vari- ables in w turn in a poor performance for capturing the properties. Similar conclusions can also be drawn from the results on the Dshapps and QM9 dataset. For example, PCVAE outperforms the comparison models by about 16% in capturing two independent properties cLogP and Molweights.
Semi- VAE
CSVAB
PCVAB.C Ic
PCVAE
Zung
0.001
0.006
0.0096
Zgso
ZaRo
0.026
0.060
Zurg
06
0.0036
0.0037
14:
0.0033
0.0063
06
1/2
0.0000
0.0005
M4
0.0015
-22
Mg
0.0029
0.73
0.0000
0.62
WW's
0.00099
0000077
0.0032
Scale
x_pos
y_pos
Scale
X_pos
y_pos
Scale
x_pos
y_pos
Scale
X_pos
y_pos
Figure 2: Hcat-maps of the mutual information between latent variables and three properties by each model for the dSprites dataset. Data value in each cell denotes the normalized mutual information.
Table 2: MSE between the expected and actual prop- erty of the generated molecules (PCVAE (cor) denotes the extended model for correlated properties).
Table 1:
The avgMI achieved by each model on the dSprites and QM9 datasets.
Methods
dSpirites
3Dshapes
QM9
Model
cLogP
Molweights
LLogS
Semi- VAE
0.439
0.115
1.413
Semi- VAE
1.40
122.34
N/A
CSVAE
0.868
1.348
1.411
CSVAE
4.69
180.92
N/A
PCVAE tc
0.285
0.018
1.245
PCVAE.IC
5.02
131.15
N/A
PCVAE_
0.266
0.031
1.162
PCVAE.ns
1.81
176.94
N/A
PCVAE
0.257
0.016
1.125
PCVAE
1.29
87.62
N/A
PCVAE (cor)
1.33
53,49
1.15
Qualitative evaluation. We also qualitatively evaluate the dependence of each latent variable and its relevant property by visualizing the variation of the properties when traversing the priors of each latent variable. As shown Figure 3 as the values of W1. W2 and w3 change between (-0.5, 0.5), the continuous variations of the assigned properties of scale, x-position and y-position of the gen- crated images are clearly visible (as highlighted in red rectangle). The variables in 2 = {2) 22, 23} have almost no influence on these properties, which validates the effectiveness of the group-wise disentanglement term. More evaluation results the other two datasets can be found in Appendix
MT =|| T(e, v) F(A)) I|,,, where & is the number of properties. T(w, v) is mutual information matrix. The details can be found in Appendix B.4
7

---

Published as a conference paper at ICLR 2021
2
22
43
W1
W2
W3
Figure 3: The generated images by PCVAE when traversing three latent variables in subset (upper
3 rows) and three latent variables in subset w (bottom 3 rows) for the dSprites dataset.
4.3 EVALUATION FOR PROPERTY CONTROLLABLE GENERATION
In this section, validate the performance of the property controllable generation. Specifically, given a predefined value of property yk, the aim is to explore whether the proposed model could generate a data point r with a property that is the same as Vk-
Qualitative evaluation. For dSprites and 3Dshapes dataset, since we have no ground-truth method with which calculate the property yi the generated images, we directly visualize the images that are generated given different values of property Y4- In Figure + each column contains four images generated given the same value of y/s (here y/s refers to the x --oosttoon property) but given the different values for the other two properties. The objects of the generated images in the same column clearly share the same x-position. Similar results can be observed in Figure 3 the wall hue flooe hue of four images in the same column are the same given the same desired property. More
visualizations for the other properties can be found in Appendix C
0.05
0.10
0.15
0.20
0.25
0.30
0.35
0.40
0.45
0.50
0.55
0.60
0.65
0.70
0.75
0.80
0.85
0.90
0.95
1.00
Figure 4: The generated images on dSprites dataset when traversing the desired value of property X- X-position. Each column of images generated given the same value of the desired property.
0.05 0.10
0.15 0.20 0.25 0.30 0.35 0.40
0.45
0.20 0.25 0.30 0.35 0.40 0.45 0.50 0.55 0.60
(b)Control the property of "floor hue" Figure 5: The generated images on 3Dshapes dataset when traversing the desired value of property (a) wall hue and (b)floor hue. Each column of images generated given the value of the desired property and random values of other properties.
(a)Control the property of "wall hue"
Quantitative evaluation. For the QM9 dataset, since the properties can not be visualized directly from the molecule, we quantitatively measure the property controllable performance in terms the MSE (mean squared error) between the actual property y of generated molecule and the desired property Sk, shown in Table 2 The proposed PCVAE outperforms the other comparison mod- els in successfully controlling cLogP and Molweights molecule generation with a smaller MSE than that of the comparison methods by around 65.15 % and 40.5% average, respectively. This
8

---

Published as a conference paper at ICLR 2021
demonstrate the superiority of PCVAE in precisely controlling the continuous- properties due to the effective invertible property prediction network. In addition, the obvious advantage of PC- VAE over PCVAE te demonstrate the effectiveness and necessity of the proposed group wouup and property-wise disentanglement term in precisely property controllable generation.
4.4 EVALUATION FOR HANDLING CORRELATED PROPERTIES
In this section, we access the ability of the proposed model to capture and control the correlated properties. The performance is tested on the QM9 molecule set for two tasks: property prediction and property controllable generation. Three properties are selected: Molweights, cLogP and cLogS. Molweights is independent from cLogP and cLogS, while cLogP and cLogS are inner-correlated.
First, we evaluate whether the subset of la- tent variables w can power the ability of prop- erty prediction, which is a very important task for new compound design in drug discovery. Given an input molecule, the trained encoder (inference model) is used to get the relevant la- tent variable Wg. and then utilize the invertible function of p(yk|wa) to predict the property y/.- Table B compares the performance of prop- erty prediction task on PCVAE and comparison models for uncorrelated properties, as well as the extended model (denoted as PCVAF(cor)) for correlated properties. Though PCVAF(cor) deals with a more difficult case than PCVAE, where an additional property clogS that is correlated with cLog? is included, PCVAE(cor) still successfully captures the information of the added property cLogS with ignorable influence the prediction of cLogP and Molweight. Specifically, as shown in Table S regarding the prediction of independent properties, PCVAE outperforms semi-VAE and CSVAE with a smaller MSE of 33.33 in terms of Molweights. It can be also observed that the prediction results of PCVAE_nsp is better than PCVAE, which shows that enforcing both directions' dependence (i.e., p( w|y well as p(yl ww) via spectral normalization) can introduce more errors than only modelling the dependence p(ylw).
Table 3: Evaluation of the property prediction task in term of MSE between the predicted and real properties.
Latent Variable
Molweights
cLog?
cLogS
semi- VAF
975.18
1.44
NIA
CSVAE
63.78
1.21
N/A
PCVAF
34.37
0.86
N/A
PCVAE nsp
31.50
0.64
N/A
PCVAE
33.33
1.21
N/A
POVAE (cor)
33.04
0.96
0.53
Next, further explore the performance of the PCVAE(cor) control the generation of the corre- lated properties. As shown in Table PCVAE(cor) achieves the smallest MSE on all the properties. demonstrate that adding the supervision of its correlated property cLogS does influence the control of the property cLogP. This also demonstrates the effectiveness of the invertible function for handling multi-input and nulti--utpuut data. To test the necessities of the proposed PCVAE (cor), we also evaluate the performance of PCVAE and comparison models in dealing with correlated properties, of which the results could be found in Appendix C.4
5 CONCLUSION
In this paper, we have proposed the PCVAE and its extended model, which learns a latent space that separates information correlated with the properties into a predefined subset of latent variables. To accomplish this, we first propose a novel Bayesian variational inference of PCVAE to jointly learn the distribution of data and properties followed by novel group-wind wise and property- -wise disentan- glement term to deal with the complex dependency of subsets of latent variables. Then, we propose to enforce an invertible mutual dependence to allow the precise property controllable generation. At last, demonstrate through quantitative and qualitative evaluations from three aspects that our proposed model achieves better performance than existing and baseline models. In future work, we plan to extend PCVAE to semi-supervised setting, where some of the property labels are missing.
ACKNOWLEDGMENTS
This work supported by the National Science Foundation (NSF) Grant No. 1755850, No. 1841520. No. 2007716, No. 2007976, No. 1942594, No. 1907805, a Jeffress Memorial Trust Award, Amazon Research Award, NVIDIA GPU Grant, and Design Knowledge Company (subcontract number: 10827.002. 12..00......)).......

---

Published as a conference paper at ICLR 2021
REFERENCES
Alexander A Alemi, Ian Fischer, Joshua V Dillon, and Kevin Murphy. Deep variational information bottleneck. In International Conference on Machine Learning, 2017.
Jens Behrmann, Will Grathwohl, Ricky TQ Chen, David Duvenaud, and Jörn-Henrik Jacobsen. Invertible residual networks. International Conference on Machine Learning, pp. 573-582, 2019.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions pattern analysis and machine intelligence, 3588):1798-1822 2013.
Samuel Bowman, Luke Vilnis, Oriol Vinyals, Andrew Dai, Rafal Jozefowiez, and Samy Bengio. Generating sentences from a continuous space. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, pp. 10-21, 2016.
Chris Burgess and Hyunjik Kim. 3d shapes dataset. https///stthbbooom//emmmm/////ss 2018.
Russel E Caflisch et al. Monte carlo and quasi-monte carlo methods. Acta numerica, 1998:1-49, 1998.
Ricky TQ Chen, Xuechen Li, Roger B Grosse, and David K Duvenaud. Isolating sources of disen- tanglement in variational autoencoders. In Advances in Neural Information Processing Systems, pp. 2610-2620, 2018.
Antonia Creswell, Anil Bharath, and Biswa Sengupta. Conditional autoencoders with adversarial information factorization. space, 19:24, 2017.
Harrison Edwards and Amos Storkey. Censoring representations with an adversary. arXiv preprint arXiv: 2015.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversaria training of neural net- works. The Journal of Machine Learning Research, 17(1):2096-203) 2016.
Xiaojie Guo, Liang Zhao, Cameron Nowzari, Setareh Rafatirad, Houman Homayoun, and Sai Manoj Pudukotai Dinakarrao. Deep multi-atrributed graph translation with node-edge co- evolution. In 2019 IEEE International Conference on Data Mining (ICDM), pp. 250-259. IEEE, 2019.
Xiaojie Guo, Liang Zhao, Zhao Qin, Lingfei Wu, Amarda Shehu, and Yanfang Ye. Interpretable deep graph generation with node-edge co-disentanglement. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 1697-1707, 2020.
Prashnna Kumar Gyawali, B Milan Horacek, John L Sapp, and Linwei Wang. Sequential factorized autoencoder for localizing the origin of ventricular activation from 12-lead electrocardiograms. IEEE Transactions on Biomedical Engineering. 67(5):1505-155 2019.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. In International Conference on Learning Representationss 2017.
Artur Kadurin, Sergey Nikolenko, Kuzma Khrabrov, Alex Aliper, and Alex Zhavoronkov. drugan: an advanced generative adversarial autoencoder model for de novo generation of new molecules with desired molecular properties in silico. Molecular pharmaceutics, (4(9):309831108 2017.
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. In International Conference on
Machine Learning, pp. 2649-2666666 2658. 2018.
10

---

Published as a conference paper at ICLR 2021
Durk P Kingma, Shakir Mohamed, Danilo Jimenez Rezende, and Max Welling. Semi Semi-supervised learning with deep generative models. In Advances in neural information processing systems, pp. 3581-3589, 2014.
Jack Klys, Jake Snell, and Richard Zemel. Learning latent subspaces variational autoencoders. In Advances in Neural Information Processing Systems, pp.
2018.
Tejas D Kulkarni, William F Whitney, Pushmeet Kohli, and Josh Tenenbaum. Deep convolutional inverse graphics network. In Advances in neural information processing systems, pp. 2539-2547, 2015.
Abhishek Kumar, Prasanna Sattigeri, and Avinash Balakrishnan. Variational inference of disentan- gled latent concepts from unlabeled observations. International Conference on Learning Repre- sentations, 2018.
Matt J Kusner, Brooks Paige, and José Miguel Hernáneez-Lobtto. Grammar variational autoen- coder. In Proceedings of the 34th International Conference on Machine Learning Volume 70, pp- 1945-1954. JMLR. org, 2017.
Guillaume Lample, Neil Zeghidour, Nicolas Usumier, Antoine Bordes, Ludovic Denoyer, and Mare' Aurelio Ranzato. Fader networks: Manipulating images by sliding attributes. In Advances in neural information processing systems, pp. 5967-5976, 2017.
Yujia Li, Oriol Vinyals, Chris Dyer, Razvan Pascanu, and Peter W. Battaglia. Learning deep gen- crative models of graphs. CoRR, abs/1803.03324, 2018. URL
org/ abs/
1803. I803.03324 03324
Qi Liu, Miltiadis Allamanis, Mare Brockschmidt, and Alexander Gaunt. Constrained graph varia- tional autoencoders for molccule design. In Bengio, H. Wallach, Larochelle, K. Grauman, N. Resa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems 31, pp. 7795-7804. 2018a.
Qi Liu, Miltiadis Allamanis, Mare Brockschmidt, and Alexander L. Gaunt. Constrained graph vari- ational autoencoders for molecule design. The Thirty-second Conference on Neural Information Processing Systems, 2018b.
Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. In international conference on machine learning, pp. 41 4114-4124, 2019a.
Francesco Locatello, Michael Tschannen, Stefan Bauer, Gunnar Rätsch, Bernhard Schölkopf, and Olivier Bachem. Disentangling factors of variations using few labels. In International Conference on Learning Representations, 2019b.
Jianxin Ma, Peng Cui, Kun Kuang, Xin Wang, and Wenwu Zhu. Disentangled graph convolutional networks. In International Conference on Machine Learning, pp. 4212-4221, 2019.
Michael F Mathieu, Junbo Jake Zhao, Junbo Zhao, Aditya Ramesh, Pablo Sprechmann, and Yann LeCun. Disentangling factors of variation in deep representation using adversarial training. In Advances in neural information processing systems, pp. 5040-5048, 2016.
Loic Matthey, Irina Higgins, Demis Hassabis, and Alexander Lerchner. dsprites: Disentanglement testing sprites dataset. URL https://github. comdeepminn//ssriissslanne/l/// (Accessseee////eeeeeeeeeceeeeeee 2018- 05-08), 2017.
Yunchen Pu, Zhe Gan, Ricardo Henao, Xin Yuan, Chunyuan Li, Andrew Stevens, and Lawrence Carin. Variational autoencoder for deep learning of images, labels and captions. In Advances in neural information processing systems, pp. 2352-2360, 2016.
Raghunathan Ramakrishnan, Pavlo Dral, Matthias Rupp, and Anatole Von Lilienfeld. Quantum chemistry structures and properties of 134 kilo molecules. Scientific data, 1(1):1-7, 2014.

---

Published as a conference paper at ICLR 2021
Taihong Xiao, Jiapeng Hong, and Jinwen Ma. Dna-gan: Learning disentangled representations from multi-attribute images. Workshop of International conference on machine learning, 2017.
Taihong Xiao, Jiapeng Hong, and Jinwen Ma. Elegant: Exchanging latent encodings with gan for transferring multiple face attributes. In Proceedings of the European conference computer vision (ECCV), pp. 168-184, 2018.
Liming Zhang, Liang Zhao, Shan Qin, and Dieter Pfoser. Tg-gan: Deep generative models for continuously-tims temporal graph generation. arXiv preprint arXiv:2005 F0X3:20005.08333, 2020.
Liang Zhao. Event prediction in big data cra: A systematic survey. arXiv preprint arXiv:2007.09815, 2020.
Shuchang Zhou, Taihong Xiao, Yi Yang, Dieqiao Feng, Qinyao He, and Weiran He. Gene- gan: Learning object transfiguration and attribute subspace from unpaired data. arXiv preprint arXiv:1705.04492, 2017.
12

---

Published as a conference paper at ICLR 2021
A ADDITIONAL DERIVATIONS ABOUT METHODOLOGY
A.1 DETAILED DERIVATION OF BAYESIAN VARIATIONAL. INFERENCE OF PCVAE
Given an approximate posterior q(2, w w/. y). we can use the Jensen's equality to obtain the varia- tional lower bound of p(x, y) as:
p(z, y) 2. [log p(2,
y, w, 2)/((2///22/0 w/.,
We have two assumptions: (I) w only encode the information from y, namely, æ and are condi- tionally independent given (i.e., 1. y/w); (2) is independent from and y. namely 1 w and 2 1 y. which is equal to y 1 z/w (see derivation in Appendix A..3). First, based on the two assumptions,
can get I yl(2, w) (see derivation in Appendix A.4). Thus, we have log p(a,, y| 2, w) = log (x]22, w: + log p(yl=, Then, based on the assumption 1 z/wr, we can have log ((y=,, w) log p(ylw). Then get log p(x, 212, w) log p(122, w) log p(ylw) To explicitly represent the dependence between x and (z, w) as well as the dependence between y and w, we can parameterize the joint log-likelihood log Pa, ,, y, w, 2) with and as:
log Po,, (x, y, w. 2) = log (2 w) log 2(2, w) + log (ull
(11)
Given the condition that parameterized (2, w1, v) ( w|xx = the information on y is included in x), by taking into the above variational lower bound term, we upper bound
(since
obtain the negative part
Po, (z, as:
L
y a wl [lo (=(22 (1]g, w) log r(2 w) - log p. (vlw)) log (2, v)]
Re log Pote w)]
ol Dog
+ [ [lgg v) - log p( w)]
(12)
= - w)] - [ [lop + Be 2/2, wx))
p(z,w)
= - -e - + (1s(1 wlx)llp(2, w))
Based on the above derivation
D (1 (x we could further decompose as:
9 wz)
D wkllqa(,,,l=)llloc.... = Eg(
p(2,
Bog w/[x - log p(z, w)]
((2 w|x)
((2,w)
log I1 (2)) P(2;)
I. (us)) II.(w;))
=
log
log
((2,00) w)
Ih (<)))(w)))
Considering that q(2, w) (2, w/ w|x), we can get:
Ppte)Dkk w/ x)) 11)), w))]
= [log ((2, w|x)
q(2, w) IIi {(z))(w;)))
II, ((:()) [T,((z))
[l:(; Il.(ww,) 11
+ log
+ log
+ loo
((2Ww) w)
w))
gle, w)
E, q(2, ()x)w)p() p [log
Ihis ((())(())))
[T. ((1) ] + E Eg(w I.((w:))
+ EggesDoog [T,((2))
(13)
I[,
=DKIlo(Z,,un w) q(2, w)(()) + DKu(a(2, II I {(4)g(w)))
+ Σ, DuLl((#)) ll x()))) Σ, Dus((())) p(w,))
(14)
A.2 EXTENSION OF THE INVERTIBLE FUNCTION TO DISCRETE-VALURE PROPERTIES
Here we consider the situation when property y is discrete- valued and we denote yA (0, 1] E R one-hot vector here to represent its category and C is the number of categories. In the overall
13

---

Published as a conference paper at ICLR 2021
objective, only explicitly model the conditional distribution of py (yx/ (gg) hence, to achieve the precisely control of property via 2 and w, which is necessary to generate x with a certain property yr that belong to the mth category, need to maximize the probability that yk = M follows:
x ~ po (2|2z, w), 2, w + arg -() Py (Vk = 2 w),
(15)
where M is also an one-hot vector with M[m] = 1. Then the above equation is equal to:
x (, where Wi can be determined as follows based cross entropy objective (Cat in the followings denotes categorical distribution):
p(=), Woith ~ p(w;); Wi - arg max max Py M(w)) (16)
Wy + arg m&x -pw)) Py (y M)
Wik + arg max -ppmm) log Cat(uu= = MI (lxx(w)i)))
p( wik arg max p( log (up((u((wi
Σj explux(wus
Wk uK (M)
(17)
A.3. DERIVATION PROCESS EXPLANATION 1
In this section, we proof that if = 1 w and y. we can have 1 2/we. First, based on the Bayesian theory, could have ply, z|w) p(2ly, wp(y)))())u)) plylz, w))(w()) namely,
pl w (y(lw))) = p(u12, w))((=/w))
(18)
Then, considering that v 1 w and 1 y. we can have pl ((2|w) = p(z) and also p(= w) p(z).
Then the right and left sides of Equation 18 can be replaced as: p(=) :))p(/w) p u))z),
and then we have p(ylw) (w) = p(vls,w), w) Given p(ylw) = p(ulz, w). both sides of the equations are multiplied by p(2/w). and we have p(zlw)p(ylw)) plyl2, w)p(2/w) p(y, z/w). Thus,
y 1 =|w.
A.4 DERIVATION PROCESS EXPLANATION 2
In this section, we proof that if x 1 ylw, y 1 2, and = w, we can have x 1 vl, 2). First, based the Bayesian theory, could have p(r,ylu,2) p(ulæ, 2, w (xll), p(aly, 2, w) w))((() 22, w) namely,
plyl,, 2, w)p(>lz w) u,))ll
(19)
Then, considering that z and w (namely 1 2/w. as proved in Section A.3) as well as y I. æ0| w. can have ply,, z, w) = pvy us) and also p(ylz, w) p(ylw). Then the right and left sides of Equation 19 can be replaced as p(ylw) (ylw)p((zz, w) = p(212, y, w)p(ylw) w), and then we have p(x12, w) p(=ly, w). Thus, we get vl(
B ARCHITECTURE AND HYPER- PHIER-PARAMETERS
B. ARCHITECTURE
FOR DSPRITS AND 3DSHAPES DATASET
Based on the description of the implementation of the proposed objective, there are three compo- nents, namely, encoder_ to model Po(z, 2, w|x), decoder_ to model P. (1 2, w) and decoder 2 to model P., (wly). When evaluate the dSprites data, the number of latent dimensions in is 3 and the number of latent dimensions in is also 3. The detailed architectures are shown in Table The hyper-paramter used for training is detailed in Table 6
B.2 ARCHITECTURE AND SUPPER-PRAAMTERS FOR MOLECULE QM9 DATASET
The architecture used for evaluation on the QM9 dataset are totally borrowed from the work pro- posed by Liu al. (2018b). A molecule is represented graph G(X, A), where each atom is
14

---

Published as a conference paper at ICLR 2021
Table 4: Encoders and decoders architectures of PCVAE for dSprites dataset (Each layers expressed in the format as < kernel size >< layer type > Num channel >< Activation Addiattionfunction function >< stride FC refers the fully connected layers).
Encoder
Decoder
Decoder 2
Input: X R
Input[ x, w] E R
Input:w;;
x Conv.32 ReL.Ustride 2
FC.256 ReLU
FC.50 ReLU
1 x 1 Conv.32 ReI.U.stride 2
FC.256 ReLU
Spectral Norm Layer
x Conv.32 strride 2
FC.512 ReLU
FC11
1 x 1 Conv.32 ReL.U.ttride 2
x 1 Conv Transpose..33 ReLU.stride 2
N/A
FC.256 RcLU
X 4 Conv Transposesss ReLU.stride 2
N/A
FC.256 ReLU
X Conv Transsssppsse33 ReI.U.stride
N/A
FC.12
x 1 Conv ConvTrannpoose.66 ReL.U.trride 2
N/A
node and X refers to the features for all nodes; each bond is an edge, where A denotes to the adjacent matrix of the graph. Considering it not the concentration of this paper, we briefly introduce the model and provide it architecture parameters in T We recommend the reader to the work (Liu al. 2018b) for more details.
Table Encoders and decoders architectures of PCVAE for QM9 dataset (Each layers is expressed in the format as < kernel size >< layer type >< N um channel >< Activation function >< stride size >. FC refers to the fully connected layers).
Encoder_1 1
Decoder_1 1
Decoder 2
Input: G(X, A), ER ER
Input[ z, w] E E
hayype E R 101
Input:en, E IR
FC.10N ReLU
FC.100 ReLU
FC.20 ReI.U
GGNN. TOO ReLU
GGNN. 100 ReLU
Spectral _Nrm Layer
GGNN. 100 ReLU
GGNN. ReLU
FC.1
FC.100
FC.9 (node) FC.3 (edge)
N/A
Molecule Encoder and Decoder. A encoder_/ / is constructed model (2, w|x) based on a gated graph neural network (GGNN). As a result, by sampling from the modelled distribution, (z, w) are obtained variables containing the graph representation vectors. The molecule decoder / models the distribution pa(a| 2z, w) to generate the molecule graph G. The molecule decoder_2 models the predict the properties y. The process proceeds in an auto-regressive style. In each step a focus node is chosen to be visited, and then the edges are generated related to this focus node. The nodes are ordered by using the breadth-fire first traversal. The molecule decoder mainly contains three steps, namely node initialization, node update and edge selection and labelling.
distribution po(y)w)
Node Initialization. We first define N as an upper bound on the number of nodes in the final gen- erated graph. An initial state
h! (t=0) is assigned with each node v, in set of initially unconnected nodes. Specifically, (t=0) is the concatenation as [(2, T;] where тÚ is an one-hot vector indi- cating the atom type. 7; is derived from (z, w) by sampling from the softmax output of learned mapping т; f(2)). From these node-level states, we can calculate global representations H(t), which the average representation of nodes in the connected component at generation step t. In addition to N working nodes, special "stop node" is also initialized to a learned representation hend for managing algorithm termination detailed as below.
Edge Selection and Labeling At each step t, focus node v; is picked from the queue of nodes. Then edges egj is selected from node v; node Uy with label Exg- Specifically, for each non-
focus node Uy, we construct feature vector 7/. [h]] hr di.j H()), H())], where d; is the
graph distance (i.e. path) between two nodes Us. 2/.- We use these representations to produce a dis-
(t) The parameters calculated softmax outputs from neural networks. New edges are sampled
(t) p ((E,,l
tribution over candidate edges as p(ei.j- ExyIn Ti of the distribution one by one from the above learned distributions. Any nodes that are connected to the graph for the first time during this edge selection added to the node queue.
15

---

Published as a conference paper at ICLR 2021
Node Update. Whenever we obtain new graph ((++1) step I, the previous node states h1 is discarded and new node representations hi (t+1) for each node is calculated by taking their (possibly changed) neighborhood into account. To this end, a standard gated graph neural network (GGNN)
utilized through S steps, which defined as a recurrent operation over messages r:
Termination. In the edge generation process of each node, the edges to node vi is kept added until an edge to the stop node is selected. Then we move the focus from the node v,, and regard v as a "closed" node. The next focus node is then selected from the focus queue. In this way, a single connected component is grown in a breadth-fire manner. The node and edge generations continue until the queue of nodes is empty.
Table 6: hyper-paramter used for training on dSprites and QM9 datasets
Dataset
Learning
Batch_size
Num_iteration
с (spectral.norm)
dSprites
5c-4
64
0.97
QM9
5c-4
32
100
B.3 ESTIMATION OF GROUP-WISE AND PROPERTY-WER WISE DISENTANGLEMENT TERMS
To evaluate the density y gf ((w) and g(w;) in the second loss L2, a naïve Monte Carlo approxi- mation (Caflisch et al. 1998) is utilized for the estimation. We describe the operation by taking 9(=) as an example. A naïve Monte Carlo approximation based on a minibatch of samples from p(n) (n is the data sample index) is likely to underestimate 9(=). As stated by Chen et al. (2018). this can be intuitively seen by viewing ((=) as a mixture distribution where the data index n indicates the mixture component. With randomly sampled component, (=2[n) is close to 0, whereas q( ((=Im) would be large if n is the component that z came from. So it is much better to sample this compo- nent and weight the probability appropriately. Thus, can use weighted version for estimating the function log 2(2) during training. When provided with a mini-bate batch of samples {NI...., R& }, we can use the estimator as:
Σ d(=(n))m,)), J=1
E Bg(ss[log = M
[log
(20)
MN
where 2(n,) is sample from g(2(n), and N is the count of the whole samples in dataset. M is the count of samples in a mini-batch.
B.4 DETAILED DESCRIPTION OF AVGMI
To evaluate the performance of the disentangled representation learning of the inference model, adopt the metric avgMI proposed by Locatello et al.) (2019b). The goal of avgMI to evaluate the whether each latent variable wa only capture the information of the relevant property Vi. and has nothing correlation with the other properties. We utilize the MI matrix (i.e. the matrix of pairwise mutual information between and y) represent the overall disentanglement performance. The optimal MI matrix should be like an identity matrix with diagonal entries all 1 and other entries all where the mutual information between each wy and y/ is 1 and the MI between w. and other property is O. Then avgMI score is calculated as the distance between the real MI matrix and the optimal MI matrix. The smaller the avgMI is, the better the performance are.
Each entry. namely, mutual information MI(w;, y; in the mutual information matrix [(z, y) is cal- p(w,,s)) ).. Therefore, to empirically estimate
culated as: MI( w., y₂) = Σ Σg, ([((;,,)) 41) log p( p(w;). p(v;). and p( w. y ). we need to have w and in the experiments. And as we know, we have the observations æ and y, and is generated from by the encoder.
16

---

Published as a conference paper at ICLR 2021
C ADDITIONAL EXPERIMENT RESULTS
C.I EVALUATION ON THE QUALITY OF GENERATION ON QM9
We evaluate the quality of the generated molecules on the QM9 dataset by three metrics: Novelty measures the fraction of generated molecules that are not in the training dataset; Uniqueness mea- sures the fraction of generated molecules after and before removing duplicates; Validity measures the fraction of generated molecules that are chemically valid. The results of the evaluation are shown in Table 7 As shown in Table 7 the proposed PCVAE still achieve 100% valdity and 99.5%, which is desirable in the problem of controlling generation. We could also found that the proposed PC- VAE can have an influence on the uniqueness of the generated data, which may be explained by the supervision of the model. However, considering our focus is data generation given the desired property, is not a critical issue, whereas the validity and novelty are still very high.
Table 7: Quantitative evaluation of the generated molecules.
Method
Validity
Novelty
Uniqueness
Grammar VAE Kusner el al. 2017)
30.00%
95.44%
9.30%
GAE (L al. 2018
61.00%
85.00%
40.90%
CGVAE Liuetal 2018a
100.00%
96.33%
98.03%
PC
100.00%
99.10%
63.50%
PCVAE
100.00%
99.50%
33.40%
C.2 EVALUATION ON DISENTANGLED LATENT VARIABLES
Evaluation results on dSprites. We provide the qualitative evaluation on the comparison experi- ments when traversing the values of latent variables in Fig. 6 As shown here, the latent variables w learned by the baseline model PCVAE to could successfully capture the three properties, which validate the effectiveness of the proposed overall inference model. The latent variables w2 and wy learned by CSVAE can capture the x_pos and y-pos properties, while 10 fails in capture the scale property. Vemi can capture the three properties but the quality of the generated images is very bad and biased.
Evaluation results on 3Dshapes. We provide the qualitative evaluation on the 3Dshapes when traversing the values of each latent variables w in Fig. 6 As shown here, the latent variables w learned by the proposed model PCVAE could successfully capture the three properties, object scale, wall hue and the floor hue in the images, which validate the effectiveness of the proposed overall inference model.
Evaluation results on QM9 dataset. We calculate the mutual information between each encoded latent variable Wa and the property yk that assigned to it, as well as the average mutual information between latent and each yk. as shown in Figure for the results molecule QM9 dataset. For this difficult task in the molecule domain which contains the implicit properties, the proposed PCVAE still shows significant advantage in capturing the Mol_weight and cLog? properties. We also qual- itative evaluate the relationship of each latent variable and its relevant properties. We visualize the variation of the properties on QM9 datasets, when traversing the priors of each latent variable, as shown in Figure The variable wi and Wy could successfully capture the properties Molweight and cLog?.
C.3 EVALUATION FOR PRECISE PROPERTY CONTROL
Evaluation on dSprites dataset. For dsprits dataset, since we have no ground-truth method to calculatcthe properties yi of the image, we directly visualize the images that are generated given different properties y) three comparison models. As shown in Figure 10 each column shows four images generated given the same value of yk. but given the different values of other properties. It could be easily observed that the objects of the generated images in the same column have the same value of yk- We provide the visualization of y pos in Fig 10
17

---

Published as a conference paper at ICLR 2021
Semi-VAL VAE E
22
23
W
W2
W3
CSVAE
Z1
2
33
W
W2
W
PCVAE tc
2
72
Z3
W 1
22
W3
Figure 6: The generated images by comparison models when traversing on three latent variables in subset of latent (upper3 rows) and three latent variables in subset of latent w (bottom 3 rows) for the Dsprits dataset
18

---

Published as a conference paper at ICLR 2021
Scmi- VAE
scale
wall hue
floor hue
CSVAB
scale
wall hue
floor hue
PCVAE
scale
wall hue
floor huc
PCVAE
scale
wall hue
floor hue
Figure 7: The generated images by PCVAE and comparison methods when traversing on three latent variables in subset of latent (bottom 3 rows) for the 3Dshapes dataset
Semi-VA VAE
CSVAE
PCVAE_tC tc
PCVAE
Z evg
00006
00005
0,000
Zav
7ays
020
0.001
0.009
W4
8.0006
00006
W4
W1
0.001
W2
W2
W.
W2
0.057
0.17
cLogP
Mol_weight
cLogP
Mol. weight
cLoogP
Mol_ weight
cLog?
Mol weight
Figure 8: Comparison on Mutual Information between different latent variables on different proper- ties molecule QM9 dataset.
C.4 EVALUATION ON THE NECESSARIES OF PCVAF(COR) FOR DEALING WITH CORRELATED PROPERTY
To validate the necessity of PCVAE (cor) for dealing with the correlated properties, we evaluated the performance of PCVAE and comparison models in dealing with correlated properties. As shown in Table for generation task, the proposed PCVAE (cor) achieved much smaller MSE than those achieved by CSVAE and PCVAE by averagely around 72.9%. 52.5% and 58.0% on the control of
latent vartable aı vs property curve
latent variable au us property curve curve
Istent variable Mu vs property curve
Istent variable we us property curve
aropetty
prunatty 8.5
2.6
Aangery 34
3.5
V/
0.0
Figure 9: The properties of generated molecules when traversing on the corresponding latent vari- ables in sub-space w by ControlVAE. (left) and ControlVAE (right).
19

---

Published as a conference paper at ICLR 2021
PCVAE tc
0.65
0.50
0.95
PCVAE
0:15
C:25
0.45
The generated images when traversing on the given desired property of Y-position. Each column of images share the same value of the desired property.
Figure 10:
Molweigt, cLog? and cLogS, respectively. This validates that traditional disentangled-based VAE models cannot handle the controllable generation for correlated properties. For prediction task, the proposed PCVAE (cor) also achieved much smaller MSE than those achieved by CSVAE and PCVAE by averagely around 53. 12%, 13.2% and 22.1% on the prediction of Molweight, cLogP and cLogS respectively. The bad performance on PCVAE and CSVAE in dealing with correlated properties caused by the conflicts between the independence enforcement latent variables w and the dependence relationship enforcement on and the correlated properties y, which largely deteriorate the optimization of the whole model.
Table 8: Comparison of models in dealing with the correlated properties
Prediction task
Generation task
Model CSVAE
Molweight
cLog?
cLogS
Molweight
cLog?
cLogS
88.03
1.21
0.63
183.41
3.84
2.56
PCVAE
52.43
1.03
0.73
168.52
1.76
2.95
PCVAE (cor)
33.04
0.53
53.49
1.33
1.15
20


Reviews:
[{"review_id": "t0NYsqVm0tL", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Update**\n\nI appreciate the effort by the authors to clarify some of the issues, most of which are addressed in the rebuttal, so I will raise my score to 6.\nI still feel like the $I(w, y)$ part needs to be dealt with a bit carefully, especially there is a invertible mapping between the two on the generative side. The simple graphical model seems like $w \\leftarrow x \\rightarrow y$, where left is encoder and right is data generation procedure. \n\n**Summary**\nThe paper proposes property controllable VAE, with the aim to learn certain latent variables correlated to the property and are disentangled. This is done by variational inference + total-variation based disentanglement terms, with the aim to \"control the properties\".\n\n**Strengths**\nEmpirical improvements over similar VAE approaches, such as Semi-VAE and CSVAE, showing the ability to learn and control the properties.\n\n**Weakness**\n - The paper has several important details that are missing, and seems not self-contained enough to reproduce if the reader is not familiar with the disentanglement literature (see questions below).\n - I don't quite understand the motivation behind the method. If you want to control the value y (which is assumed to exist for all x), then why not replace w with y entirely (i.e. invertible network is identity) and operate directly on y? On inference side, predict $q(y | x)$, on generation side use empirical distribution (or some simple KDE) on y to get $p(y)$. \n- Granted, it is possible that $y_i$s can be dependent, whereas you want $w_i$s to be independent; but it seems that Section 3.2.4 adds the assumption that $y_i$ are independent because you can group dependent variables?\n\n**Questions**\n- How are the total correlation objectives evaluated? For example, the explicit log-likelihood values for either $\\log q(w)$ or $\\log q(w_i)$ cannot be efficiently computed since that is the aggregate posterior? Is this implemented as an adversarial objective?\n- What is the difference between PCVAE and PCVAE_tc (write down the equations)? Performance-wise they seem relatively similar.\n- How does the invertible network work when y is discrete? If I made any error in w, then it maps to some y value outside of the sample space?\n- How do you evaluate $I(w, y)$, and for which distribution is this defined? This confusion is because in reality there are two \"worlds\" concerning both variables (generative and inference directions) and because you have a invertible function between w and y (so it seems that optimizing MI is useless since you don't lose information anyways?).\n- Why do we use avgMI as a metric? Do we expect the optimal MI value to be 1? If that is the case, then how do we get an avgMI of 1.004 for Semi-VAE in the first place?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "eFx1dJ6FyhO", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents the new Property-controllable VAE (PCVAE) to inductively bias the latent representation to capture explicit data properties. Towards this, the paper proposes group-wise and property-wise disentanglement terms. The group-wise disentanglement term separates two subsets of the latent representation. In contrast, the property-wise disentanglement term promotes the disentanglement of an individual component of one of the subsets in the latent space. Furthermore, the model enforces individually disentangled latent subset to account for the given property of the dataset via invertible constraint. The presented work is evaluated on two datasets from two domains. Below I present some pros, cons, suggestions, and clarifying questions for the authors. \n\n- The paper is clear to understand. The presented experimental studies are clear and demonstrate the efficacy of the model.  For instance, the precision with which the proposed model controls the property generation is impressive. Qualitatively this is demonstrated by Fig 4. Are these demonstrated on the train or the test set? \n\n- For both group-wise and property-wise disentanglement terms, the TC term is considered. Can the authors clarify why they claim them to be novel? Can authors present training statistics to give insight into the model's optimization in relation to these up-weighted TC terms? \n\n- dSprites is considered to be a relatively simpler dataset for disentanglement. If authors wanted a dataset with ground truth generative factors in the dataset, they could consider 3DShapes (Burgess & Kim, 2018), which would have more generative factors and is more challenging than dSprites. Also, 3DShapes is a widely considered dataset beside dSprites for disentanglement related study.  \n\nBurgess, C, & Kim, H. (2018). 3D Shapes Dataset. https://github.com/deepmind/3dshapes-dataset/. \n\n- The literature review in the paper seems broad. I would encourage authors to discuss some recent approaches (e.g., Gyawali et al., 2019) that disentangle latent space into two subsets with one being related to certain property of the dataset. \n\nGyawali, P. K., Horacek, B. M., Sapp, J. L., & Wang, L. (2019). Sequential factorized autoencoder for localizing the origin of ventricular activation from 12-lead electrocardiograms. IEEE Transactions on Biomedical Engineering, 67(5), 1505-1516.\n\nMinor comments:\n- Please proofread your manuscript carefully to avoid grammatical errors. e.g., \"latent presentation\".   ", "extra_scores": {}, "extra_text": {}}, {"review_id": "_zR4qt-vk5", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Property controllable variational autoencoder via invertible mutual dependence**\n\nThe paper proposes an approach for learning disentangled latent representation in the VAEs where parts of the latent space should correspond to observed properties of interest of the data so that these could be controlled at generation. They replace the total correlation regularization term of Chen2018 with more appropriate group- and property-wise KL terms promoting independence between the *relevant* (w) and the *remaining* (z) part of the latent space and inner independence between dimensions of the w part of the space (dropping the term promoting independence in the z part.) More importantly, they link the data property y with the latent space w by a learned mapping.\n\nThe is another contribution to the rather extensive literature on disentanglement of latent representation of VAEs. It builds on previous papers in the area, mainly Chen2018 and Klys2018. The change in the ELBO as compared to Chen2018 (TC vs group/property-wise) is well explained and motivated though seems rather trivial.\nThe introduction of the invertible mapping between y and w is in my view more interesting and important. Though somewhat contrived at first reading (some improvements in the text could help, see comments), it seems to correspond well to what the model shall achieve. \nThis  as is also demonstrated in the experimental section focusing mainly on the dSprites and QM9. The paper would benefit from a more extensive experimental evaluation (another standard dataset such as CelebA) though I understand this may be difficult to do given the deadlines and computational resources and therefore do not consider as critical.\nThe paper is generally well written though there are places which are difficult to parse and comprehend and suggest last minute drafting. These shall be made clear for the final version (see comments below).\nThe problem of correlated properties is addressed by a rather trivial extension of the basic model and requires the user to provide to the model groups of correlated properties. I feel this deserves a lot more attention and search for better solutions is a possible direction for future work.\n\nI recommend to accept the paper as I find it interesting for the community, addressing a lively area of research via a new approach which, though not dramatically innovative, seems to yield the desired results. I do have a few comments as to the clarity of the statements in the paper which I hope will be addressed by the authors during the rebuttal period and in the final version.\n\n*Comments / questions:*\n* I find calling the mapping between y and w *invertible* as misleading. The mapping $w \\to y$ is defined as stochastic via the learned generative distribution $p(y | w)$. Each w thus corresponds to multiple possible values of $y$.\n\n* eq (2): please state clearly (in equations) the independence and conditional independence assumptions leading to this result\n\n* eq (4) holds only in expectation over $p(x)$, right? ($E_{p(x)}$ in front of the DKL in the left hand side.)\n\n* p4 before eq (5): \"Roughly disentangling all pairs of latent variables without emphasis could lead to poor convergence and incur exponential number of pairs among properties and latent variables for such enforcement.\" What do you mean? Please explain / elaborate / rephrase.\n\n* p4, \"... no strict assumptions of parameters for $p(y_k)$ and $q(w_k |y_k)$\". $y$ does not exist in the inference model so what does $q(w_k |y_k)$ refer to?\n\n* p4 \"The most straightforward way to do this is to model both the mutual mapping between $y_k$ and its relevant latent variable $w_k$.\" Both mappings $y_k \\to w_k$ and its inverse $w_k \\to y_k$? Or what do you mean?\n\n* p5, para5 \"Thus, we utilize the Naı̈ve Monte Carlo approximation based on a mini-batch\nof samples to **underestimate** $q(z), q(w)$ and $q(w_k)$, as described by Chen et al. (2018)\" Underestimate? What do you mean?\n\n* p6: you say you use the normalized mutual information between $w_k$ and $y_k$. Please explain how you define this and how you calculate in practice.\n\n* p7, fig3 \"... when traversing three latent variables in subset z ...\". There are only 3 latent variables z? If more, how did you decide which shall be traversed?\n\n* p8, tab3: How do you predict the property $y_k$ from a molecule? You first infer w and z and then use the mean of p(y | w) as prediction? Or you sample y from p(y | w)? Please elaborate.\n\n* p7, tab2: Do I read correctly that MSE of cLogP is bigger from the PCVAE (corr) then from the standard PCVAE? What does this say about the advantages of PCVAE (corr)?\n\n* experiments over QM9: are all your evaluation metrics calculated only over valid molecules? Usually, generative models for molecules are evaluated using the validity, unicity and novelty scores. It would be instructive to complement your results with these.\n\n\n*Minor comments / questions*\n* Please provide references to models displayed in Fig1(a)-(c).\n\n* You use the term *correletad* a lot. I'm guessing here you mean any sort of dependence, not just linear?\n\n* p2: \"Directly enforcing such mutual independence inherently between all pairs of latent variables incurs exponential number of pairs among properties and latent variables for such enforcement.\" Exponential number of pairs? What do you mean?\n\n* p2, para3: \"...  as these have been shown to be relatively resilient with respect to the complex variants involved (Bengio et al., 2013).\" Complex variants? What do you mean?\n\n* p3, para2:  $y = {y_k \\in R}_{k=1}^K$. Is this the same as $y \\in R^K$?\n\n\n*Typos or phrasing improvements needed:*\n\n* p1, para1: \"**Knowing such properties is crucial** for many applications that depend on being able to interpret the data and control the data generation **to yield the desired properties**.\" ???\n\n* p2. para1: \"Also, many cases require to generate data with properties of which the values **are?** unseen during training process.\"\n\n* p3 just before eq (2): $\\log P_{θ,γ} (x, y, w, z)$ (capital P?)\n\n* p5, para3: \"As stated in the third challenge in Section 1, there are usually several groups of properties involved in **formatting**? data x ...\"\n\n* p6, para5: \"... each encoded latent variable $w_k$ and the property $y_k$ **that is?** assigned to it ...\"\n\n* p7, caption Tab2: \"...  (PCVAE (cor) denotes the **extensive**? model for correlated propertie\"\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "7H0vVzihIwl", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "To encourage disentanglement in the latent space of a variational autoencoder (VAE), the authors propose to learn two sets of latent z and w: the dimensions of w are independent of each other and each dimension w_i maps to a known ground truth generating factor y_i. Latent z captures all the other factors. The well studied Total Correlation regularisation is used to enforce the independence of z and w, and the same is used to enforce the independence of the dimensions of w. Each dimension is learned to predict a corresponding ground truth factor. The key difference from the previous approach is the use of invertible and Lipschitz smooth mapping to learn monotonic mappings from w to y.\n\nPros:\n+ The proposed regularization is shown to be useful for controlled manipulation for dsprites and QM9 dataset and improves upon comparable baselines.\n+ Can be applied when the ground truth factors are continuous-valued.\n+ Ability to handle the case where the ground truth factors are correlated. \n\nCons + Questions: \n- Ablation studies on the use of spectral normalization will be helpful. \n- The choice of Gaussian distribution for modeling the relationship between latent and ground truth factors could be elaborated upon.\n- A more complex dataset at least in the case of natural images with large number of ground truth factors with correlated latent structure will be useful. In such cases how does the choice of dimensions of z and w impact the learning and their independence structure?\n\n\nTypos:\n\nSec 2: earning -> learning\n\nSec 3.2.1 likely-hood -> likelihood\n\nFig 1(d) Y is missing from the model.\n\n\\phi is missing from RHS of eq 4.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a new approach to learning deep generative models with induced structure in the latent representation. All four reviewers gave the same score of 6 to this paper, showing a consensus that the paper is above the bar for acceptance. The authors did a commendable job of detailed replies to reviewer comments, which as R1, R3, and R4 all note has improved the clarity and quality of the paper, addressing their concerns.
Decision: Accept (Poster)