Multi-objective Deep Data Generation with Correlated Property Control
Authors: Shiyu Wang
Xiaojie Guo
Xuanyang Lin
Bo Pan
Yuanqi Du
Yinkai Wang
Yanfang Ye
Ashley Ann Petersen
Austin Leitgeb
Saleh AlKhalifa
Kevin Minbiole
William Wuest
Amarda Shehu
Liang Zhao
Venue: NeurIPS 2022 Accept (2022)
Keywords: deep generative models
controllable generation
correlated properties
disentangled learning
variational autoencoders

Abstract: Developing deep generative models has been an emerging field due to the ability to model and generate complex data for various purposes, such as image synthesis and molecular design. However, the advance of deep generative models is limited by the challenges to generate objects that possess multiple desired properties because: 1) the existence of complex correlation among real-world properties is common but hard to identify; 2) controlling individual property enforces an implicit partially control of its correlated properties, which is difficult to model; 3) controlling multiple properties under variour manners simultaneously is hard and underexplored. We address these challenges by proposing a novel deep generative framework that recovers semantics and correlation of properties through disentangled latent vectors. The correlation is handled via an explainable mask pooling layer, and properties are precisely retained by the generated objects via the mutual dependence between latent vectors and properties. Our generative model preserves properties of interest while handles correlation and conflicts of properties under a multi-objective optimization framework. The experiments demonstrate our model's superior performance in generating objects with desired properties.

Multi-objective Deep Data Generation with Correlated Property Control
Shiyu Wang', Xiaojie Guo', Xuanyang Lin', Bo Pan', Yuanqi Du Yinkai Wang', Yanfang Ye,, Ashley Ann Petersen', Austin Leitgeb", Saleh AlKhalifa', Kevin Minbiole", William Wuest', Amarda Shehu", Liang Zh
Emoryy University, Sshiyu.wang mike. bo.pan, william. wuest, liang.zhao| @emory.edu
IBB Thomas.J. Watson Research Center, xguo7 @ gmu.edu
CComell University, yd392 cornell.edu
*Tufts University, yinkai. wang tufts.edu
University of Notre Dame, yye7 @ nd.edu
Villanova University, (apeter24, austin,leitgeb, kevin.minbiole @ villanova.edu
Reecursiv LLC, salehesam @ gmail.com GGorrge Mason University, ashehu @ gmu.edu
Abstract
Developing deep generative models has been an emerging field due to the ability to model and generate complex data for various purposes, such image synthesis and molecular design. However, the advancement of deep generative models is limited by challenges to generate objects that possess multiple desired properties: 1) the existence of complex correlation among real-world properties is common but hard to identify;
controlling individual property enforces an implicit partially control of its correlated properties, which is difficult to model; 3) controlling multi- ple properties under various manners simultaneously hard and under-explored. We address these challenges by proposing novel deep generative framework, CorrVAE, that recovers semantics and the correlation of properties through disen- tangled latent vectors. The correlation is handled via explainable mask pooling layer, and properties are precisely retained by generated objects via the mutual dependence between latent vectors and properties. Our generative model preserves properties of interest while handling correlation and conflicts of properties under a multi-objective optimization framework. The experiments demonstrate our model's superior performance in generating data with desired properties. The code CorVVAB is available at httpps: https:://itthi //gpth/bb.ttttt. com/ shi yu- wuy-uuuwga..... CorrVAE 7pu-wang/CoorVAE
Introduction
Developing powerful deep generative models has been an emerging field due its capability to model and generate high- -ighnsimenio complex data for various purposes, such as image synthesis
protein design [4 16), co-authorship network analysis (6] Extensive efforts have been spent learning underly- ing low-dimensional representation and the generation process of high-dimensional data through
30), molecular design 24 47
and natural language generation 22 32
deep generative models such as variational autoencoders (VAE) (27 35 generative adversarial networks (GANs) [2], normalizing flows [40 5) etc [48
17 g Particularly, enhancing the
disentanglement and independence of latent dimensions has been attracting the attention of the
community 4 43 3 34 45| 23), enabling controllable generation that generates data with desired
properties by interpolating latent variables [44
13 29
25
38
49]. For instance, CSVAE
Corresponding author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

transfers image attributes by correlating latent variables with desired properties [28]. Semi-VAE pairs latent space with properties by minimizing the meeansquure-erve (MSE) between latent variables VAE (PCVAE) synthesizes image objects with desired positions and scales [3 by enforcing the mutual dependence between disentangled latent variables and properties. Conditional Transformer Language (CTRL) model generates text with task-specific style and contents (26). Despite of the rapid growth of research regarding property (6)
and desired properties 31. Property-controllann
(A]
For Sou
Sun Horr
Rie
1see
Solubility lowicity possible
V-
Potential:
Figure 1:
(a) Correlated properties are common for the real-world object, such as the day (growth time), size and color of wild pepper: (b) Generation of molecules that satisfy desired properties can be viewed as multi-objective optimization task
controllable generation in various domains, critical challenges still remain such as: 1) Difficulty in identifying property correlation. Existing models for data property control typically map each property to its exclusive latent variables. Also, all the latent variables are inherently enforced to be independent of each other. Therefore such complete distentanglement among the consideration of properties disallows the model to characterize the correlation among properties and hence can only work for controlling properties that are independent of each other. However, properties in real data objects are usually correlated (Figure I (a)). For example, in human face image, the face width has a correlation with eye size. The color of the wild pepper has a correlation with the growth time (Figure (a)). The correlation of properties has been under-explored, which largely impairs the effectiveness of generative models; 2) Controlling individual property also enforces an implicit partial control of its correlated properties, which is difficult to model. For those correlated properties, controlling one of them will also constrain the others into some subspace such as hyperplane even non-convex set. For example, when generating face images, if we constrain the width of the face is 100 pixels then the size of the eyes will be constrained to a reasonable range; 3) Difficulty in simultaneously controlling multiple properties under various manners. The real-world application usually requires generated object to satisfy multiple constraints of properties simultaneously. One may want maximize property's 's value, fix another property to certain value, and constrain the third property within a range. Therefore, the data generation problem is entangled with and hardened by the multi-objective optimization goal, which has not been well explored. For example, chemists may design a molecule that has specific potential, minimizes its toxicity and meanwhile possesses solubility within a range (Figure [ (b)). We overcome these challenges by proposing a novel deep generative model, VAF, that recovers semantics and the correlation of properties via disentangled latent vectors. The correlation is handled by an explainable mask pooling layer, and properties are precisely retained by the generated data via the mutual dependence between latent vectors and properties. Our generative model preserves multiple properties of interest while handling correlation and conflicts of properties under multi-objective optimization framework. The contributions of this paper are summarized as follows:
A novel deep generative model for multi-objective control of correlated properties. Beyond disentangled representation learning, we aim at corresponding latent variables to target properties for better interpretability and controllability. The model is generic to different types of data such images and graphs, together with disentangled terms to obtain independent latent variables to jointly handle correlated properties.
A correlated invertible mapping is proposed for mapping correlated real properties latent independent variables. An interpretable mask pooling layer has been proposed to explicitly iden- tify how the real-world properties are generated by the corresponding subsets of latent independent variables. The information of these latent variables will be aggregated and enforced to be mutually dependent on the property via an invertible constraint.
A multi-objective optimization framework is formulated for deep data generation problem. Corresponding latent variables in the low-dimensional representation are optimized under nultiple objectives and constraints for property control purposes. Our framework is generic to various multiple objectives such as optimizing a property value, constraining property values into a range, maximizing or minimizing property value while maintaining the correlation among properties.

---

Extensive experiments are conducted on real-world datasets. The proposed model can generate data with multiple desired properties simultaneously in the generation process, demonstrating the effectiveness of the model. Moreover, our model shows superior accuracy of generated properties against target properties compared with comparison models with multiple real-world datasets.
This paper firstly introduces the general framework of the proposed model. Then we will discuss the details of the model, including the derivation of the overall objective, the mask pooling layer, and invertible mapping between latent space and correlated properties. Lastly, we conduct the comprehensive experiment to compare our model with existing methods.
2
Related works
2.1 Disentangled representation learning
Disentangled representation learning aims to encode information of high-dimensional complex data into low-dimensional space that consists of mutually independent variables to separate out independent factors of variation of data distribution in the representation 43
34 21 19 [0].
Because of the success of VAE and GANs as deep generative models [35 36 15 a few techniques have been developed as variations of VAE or GANs to achieve disentanglement of latent variables in the representation space. For instance, 3- 3-VAE modifies the variational evidence lower bound (ELBO) by adding hyperparameter 3 before the KL-divergence term encourage disentangled latent variables [21]. Instead, cycle-consistent VAE was proposed for supervised disentanglement of specified and unspecified factors of variation using pairwise similarity labels [2). On the other hand, InfoGAN maximizes the mutual information between the latent variable and the generated sample under the framework of GANs. Disentangled Representation Baaning-Geeneratvve Adversarial Network (DR-GAN) disentangles the representation with the pose of the face on the image through the pose code provided to the generator and pose estimation by the discriminator (43
2.2 Property controllable deep generative models
Considerable efforts have been spent on developing deep generative models that generate data with
desired properties 13 47 20 26 25 42 37. Techniques for property controllable generation include but are not limited to 1) reinforcement learning (RL) approach to goal-directed data generation that preserves target properties, such Graph Convolutional Policy Network (GCPN) 47 and GraphAF [41]. and 2) mutual dependence between properties and latent variables to control generation process by manipulating values of latent variables, such PCVAE [13] and Conditional Subspace VAE (CSVAE) 28). RL approach nevertheless suffers from the requirement of a large sample size for the training purpose. Moreover, all the above methods are unable precisely capture complex correlation among properties in an explainable way, nor can they generate data that simultancously satisfy multiple correlated targets of either values or ranges. To fill the gap between the existing methods and the need for controllable generation from various domains, we propose a novel controllable deep generative model that handles the correlation of properties via an explainable mask pooling layer and generates data with desired properties under a multi-objective optimization framework.
3 Problem formulation
Suppose we have a dataset D, in which each sample can be represented as x, along with y = {v1, (v.. Fm as properties of æ, which can be either correlated or independent with each other. For instance, if the data is a molccule, propertics can be molccular weight, polarity or solubility. We further assume that (I, y) is generated via some random processes from continuous latent variables in (w, 2), where w controls the properties of interest in y and z controls all other aspects of x.
We aim to learn a generative model that generates (z, y) conditioning on (w, 2). where is disen- tangled with w and variables w are disentangled with each other to control either correlated or independent properties. Once the model is trained, the user can generate data with target values or ranges of properties via editing the corresponding elements in w. For example, we may want to generate a molecule with a specific value of the weight, and solubility within a range while minimizing its toxicity by changing values of that contribute to those properties. This goal leads to the following questions answered by our work: how to automatically identify the correlation
3

---

among properties, how to control individual property while enforcing an implicit partial control of its correlated properties. and how control multiple properties simultaneously?
Proposed approach
Hey
W1
11
Property
W2
felonl. Pa)
encoder
MLP
3
mask
feclWee Yea)
Wise
Hz
Object
Object
Weighs
encoder
decader
Weight
FOr relu
Figure 2: Overal framework of ConVAR. Cor encodes the information of correlated properties into the latent space w and other information of the object into 2 via the property and the object encoder, respectively. Then the correlation among properties is captured by the mask pooling layer, where the information to predict specific property is aggregated into the bridging latent variable w'. The mutual dependence between w' and the corresponding property is enforced by the invertible constraint achieved by ResNet. Lastly, the data can be generated from (w, 2) via the object decoder.
In general, the proposed approach, CorrVAE, identifies the property correlation via a novel mask pooling layer. It precisely retains properties via the constraint of mutual dependence between latent vectors and properties, and simultaneously controls multiple properties under a multi-objective optimization framework. The overall framework of the model is shown in Figure 2 Specifically, the model contains two phases: (1) Learning phase encodes the information of properties via the property encoder and other information of the data via the object encoder. As shown Figure 2 the correlation information of properties is captured by novel mask pooling layer. The mutual dependence between latent variables and properties is enforced via constraint applied the learning objective. The data is generated via the object decoder in Figure 2 (2) Generation phase generates data with desired properties of specific values within target ranges under the multi-objective optimization framework. In this section, we will introduce two phases in detail.
4.1 Learning phase
4.1.1 Overall objective for disentangled learning latent variables
The goal requires us to not only model the dependence between x and (w, for latent representation learning and data generation, but also model the dependence between y and w for controlling the property. We propose achieve this by maximizing the joint log likelihood p(x, y) via its variational lower bound. Given an approximate posterior (2,, w|æ, y), we can use the Jensen's equality obtain the variational lower bound of log p(z, y) as:
log p(2, y) log Egts.wle..)) [p(2, y, w, 2) ///(z wle, v) 2 eg (log p(x, y, w, 2)/0(2, w/ v)].
(1)
The joint likelihood log p(x, y. w, 2) is further decomposed as log p(æ, y/2, w) + log p(z, w) given Two assumptions: (1) x and y are conditionally independent given w since w only captures informa- tion from y; (2)= is independent from and y, equivalent
z/w. This gives us 1 y(w,, 2).
suggesting that log p(x, y/ ji 2) log p( w, 2) 2)- +log p(v) w,, z) log p(# w, 2) + log (tylw) Con- sequently, we write the joint log-liklldoood and maximize its lower bound:
log po. (2. y, ur, 2) = log po 2) log low, 2) + log (ylw)
log po ur, 2) log pww,
Σ (yg
where we define next-level latent variable wi as the set of values w that are independent with each other and contribute to the i-th property to bridge the mapping wo - y while allowing property controlling. Each value w in w' = [w], w/y ..., wm)) relates to each ys in Since properties are

---

independent conditioning on w', the decomposition of the third term holds in Eq. 2 The relationship
between y and w' will be further explained in Section 4.1.2 Given 4s (w, 2|r,3/ y) ( 2|x)
golwla) go (0(=]x), we rewrite the joint probability in Eq. (2) as the form of the Bayesian variational
inference as the first term of the learning objectíve:
L = (w._(e) (+) [log (z w, =)] (w)l))) (u(le)] + (xz(es =|>))lp(m,, =)).
(3)
Meanwhile, since the objective function in Eq. (3) does not contribute to our assumption that is independent from w and y. and values in w are independent with each other, we decompose the
KL-divergence in Eq. (3) and penalize the term:
C2 p1 (xl w)ll (=))((w)) + pa Dxc(a(w))) ((s))))
(4)
where #1 and p2 are co-efficient hyper-parameters to penalize the two terms. Details the proof and derivation regarding the overall objective can be refereed in Appendix A.
4.1.2 Relating the properties and latent variables
To model the dependence between the correlated properties and the associated latent variables p(ylw) in Eq (3) well as to capture the correlation among properties, we propose to directly learn the specific relationship between disentangled latent variables in w and properties y. The correlations among y are also captured. Specifically, we design mask pooling layer achieved by a mask matrix M E 10 where I is the dimension of the latent vector w. M captures the way how w relates to y, where M,,- = 1 denotes that w, relates to the j-th property yj. otherwise there is no relation. In this way, two properties that relate to the same variable in w can be regarded as correlated. The binary elements in M are trained with the Gumbel Softmax function. In implementation, the I,, norm of the mask matrix is also added to the objective to encourage the sparsity of M.
Next, given the learned mask matrix M, we model the mapping from to to y. For properties y, we can calculate the corresponding w' that aggregates the values in w that contribute to each property as w- J C M, each column of which corresponds to the related latent variables in w to be aggregated to predict the corresponding y. For each property y in y. we aggregate all the information from its related latent variable set in w into the next-level latent variable w'J (i.e., the j-th variable of w') via an aggregation function h:
w' = h(w J" C M;
(5)
where J is a vector with all values as one, @ represents the element-wise multiplication and 3 is the parameter of h. Then the property y can be predicted using w' as:
(6) where f is the set of prediction functions with w/ = h(w JJ @ 3) as the input and y are the parameter which will be further explained in the next section. Thus, we have built one-to-one mapping between w/ and y. In addition, the correlation of y: and yj can be recovered M M,, # O.
y y= fw'
4.1.3 Invertible constraint for multiple-property control
As stated in the problem formulation, our proposed model aims to generate a data point that retains the original property value requirement for the given properties. The most straightforward way to do this is to model both the mutual dependence between each y and its relevant latent variable set w/. However, this can incur double errors in this two-way mapping, since there exists a complex correlation among properties in and there are many cases that M.. M., * O. To address it, we propose an invertible function that mathematically ensures the exact recovery of bridging variables w' given a group of desired properties y based on the following deduction.
As in Eq. (). the set of correlated properties variables w" {uri. wy.... w/n) } in a one-to-one mapping fashion. Thus we assume that y can be sampled from a multivariate Gaussian given w' as follows: m(vllw")
[vn, V2...., 3m } are correlated with the set of latent
f(w'; 7)), ;7),)))))) E); = (v1 (2,-.., w/ = [w/,, Buu[,u/,-- -.., w w/m J. R m
s.t., [[]]]>>]]]] = f(w': >Lil w/,, Lip(F(w': bb))
if ||Walla 1,j L,.., (7)
where Lip denotes to the Lipschitz - constant. Namely, to precisely control the properties we learn a set of invertible functions f'(w('7) indicated in Eq model py (y/wn)) is the set of
parameters in Eq 6 The constraint enforces (w]; to be an invertible function to achieve mutual
dependence between yj and w' 22. As result, have the third term of the objective function:
La - Eur (NWyl 2)] + [(w' (x)))) 1112
(8)
5

---

We evaluate the proposed and comparison models on two molecular datasets and two image datasets: 1) The Quaternary Ammonium Compound (QAC) dataset is real dataset that contains 462 quaternary ammonium compounds processed by the Minbiole Research Lab [1] An open-source 4.2 Multi-objective data generation phase
In this section, we introduce how to control the property of the generated data based on the well well-trained model. Based on the problem formulation, aim to generate data that holds the correlated properties = {yr 1/, .... /m via w, where the properties need to meet a series of value requirements. We approach the goal by firstly mapping properties y back to w' via the invertible function learned in Eq. 77. then optimizing w under a multi-objective optimization framework. Finally, w combined with that controls other aspects of data to generate the data via the objective decoder (Figure 2
To obtain the corresponding w' from properties y, conditioning on the fact that (1) w' = h(w JJ" @ M; 3) where h is the aggregation function indicated in Eq (5) and (2) w p(w), we optimize the conditional distribution of P. w') by maximizing the probability that y = 9 as follows:
M argmax P- 1 w'): w' argmax
log log N(y ff ())
W/ argmax ΣΣ (0 f( 10) f(d'
(9)
where dij is the element of the i-th row and the j-th column of Σ in Eq. 7 ÿ is the set of properties of interest. The above operations are based on the fact that y is vector that contains continuous variables. Since E is unknown, we optimize Eq. 9 under the same condition by alternatively solving
the following problem based on the Theorem
w' = aigmax 10,
(0, - +b)))
(10)
Theorem 4.1. Solution of Eq. 10 is also the solution to optimizing Eq. 9
The proof of the theorem 4.1 is trivial and can be referred to Appendix this way, we can obtain w' directly from the invertible function fww'; :7 with ý as the input.
Lastly, given the optimal w', aim realize the requirements that posed the correlated properties of the generated data, such as being specific values, lying in a range or reaching the maximum value. To this end, naturally formalize the process of searching the satisfied w" multi-objective optimization framework. The requirements can be defined as range of properties in y. C111 yi S G,2, while it can also be the exact values when 1, C,2- These properties can be divided into two groups based on the requirement of the target value: (1) Y. represents the set of properties that are required to be a specific value, namely, Yi- = 1 and (2) Y, represents the set
of properties, the value of which should lie in range, namely, G,1 S S C,2- Then the overall optimization objective is formalized as:
U (p c.,lww)}
W argmax
w>p(w) B.EY,
s.t. w = hV=:(w;B)) (())) 4,1 S S c,2, Vy E Y,.
(11)
where Y, and Y, are defined based on different applications. The border of the range can even be set to be infinity or negative infinity. When 4,2 is set to be infinity, is maximized in Eq. (11)
When G1 is set to be negative infinity, ys is minimized in Eq.
The details of overall practical implementation of the aforementioned distributions to model the whole learning and generation process are presented in Appendix C.
Experiments
5.1 Dataset
cheminfornatics and machine learning library were used to generate number of properties or features for each of the compounds, in which molecular weight and the log? value were used as data properties in our experiments; 2) QM9 dataset is an enumeration of 1 134,000 stable organic molecules
'The Minbiole Research Lab: http://kminbiol.c clasit. org/

---

110
1011
(4)
[e)
Figure 3: Generated images of Cor by traversing five latent variables in for dSprites dataset according to the mask matrix (Figure The corresponding properties are illustrated at the top right corner of each image. (a) Traversing on the wi that only controls shape; (b) Traversing on the w2 that only controls size; (c) Traversing on the w4 that controls both x position and x+yy position; (d) Traversing on the W2 that controls both y position and x+y position; (e) Traversing on the Wi that simultaneously controls position, position and x+y position.
with up 9 heavy atoms (39] Molecular weight and the logP values serve as the target properties for the comparison with the QAC dataset; 3) dSprites contains 737,280 total images regarding 2D
shapes procedurally generated from ground truth independent latent factors [33, in which shape, scale, x position and position were employed in our experiments. To construct correlated properties, we additionally formed and tested new property, x+y positions by summing up x position with y position; and 4) Pendulum dataset was originally synthesized to explore causality of the model [46]. Pendulum contains 7,308 images in total with 3 entities (pendulum, light, shadow) and 4 properties (ppendulum angle, light position) - (shadow position, shadow length)).
5.2 Comparison models
We compare the proposed model with three comparison models that can be used for property controllable generation: 1) semi-VAE pairs the latent space with desired properties and minimizes MSE between latent variables and desired properties during the training process (T:: 2) CSVAE correlates the subset of latent variables with properties by minimizing the mutual information (28); 3) PCVAE implements an invertible mapping between each pair of latent variables and desired properties, while it solely relies on the disentanglement assumption of the learned latent space and is short for capturing correlated properties [3]. Besides three comparison models, we consider two other models adapted from Cor for the ablation study: 1) CorrVAE-1: we replace the mask pooling layer of the proposed model with the ground-truth mask that is manually obtained from the data; 2) CorrVAE-2: We replace the MLP that maps to w' with the simple linear regression to evaluate the significance of non-linear correlation among properties. The linear regression model employs the w' as the response variable while the corresponding values in as the independent variables.
5.3 Quantitative evaluation
5.3.1
Evaluation metrics
In this section, we quantitatively evaluate the proposed model and comparison models on both molecular and image datasets. For image data, we evaluate the model by controlling three properties shape, size, position, y position and x+y position.
Molecule generation evaluation metries. We evaluate the performance of molecule generation via three common evaluation metrics which focus on the validity, novelty and uniqueness of the generated molecules as follows: Validity measures the percentage of valid molecules overall gen- erated molecules. Novelty measures the percentage of new molecules in the generated molecules
7

---

that are not in the training set. Uniqueness measures the percentage of the unique molecules overall generated molecules. The results are shown in Appendix, Table ??.
Controllable molecule generation evaluation metrics. To evaluate the performance of con- trollable generation on molecules, we evaluate the proposed model in generating molecules with desired properties. Specifically, we evaluate the MSE between the molecular proper- ties of the generated molecules and the expected molecular properties. The results based on predicting three properties size, x position and x+y position are shown in Table
Image property prediction evaluation metrics. As an
additional benefit of the proposed model, the prediction of image property from the latent space could be utilized as an image property predictor. On the other side, the prediction performance of the predictor could reflect the quality of the model in learning image properties. We evaluate MSE between the ground truth image property value and the predicted image property value. The results based on predicting three properties size, x position and x+y position are shown in Table [] We also quantitatively evaluate the quality of generated images via the FID score, reconstruction error and negative log likelihood shown in Appendix Table ??.
RORE RAEST
2352 COOTEE LOGG
4AY poypttoon1Ja 2.027
couiton B3715
ypeafoo 0.024
Disentanglement evaluation metrics. We evaluate the disentanglement of latent variables from different perspec- tives. First of all, we borrow the metric avg./ [31 to evaluate the overall performance of the proposed model and comparison models. avgMl measures the mutual de- pendence between latent variables and properties, and is calculated by the Frobenius norm regarding the mu- tual information matrix and the ground mask matrix: argM/ = [[, y) - M ||2 where [(w, y) is the pair- wise mutual information matrix between latent variables of w and properties. M is the ground-truth mask matrix indicating of the contribution of latent variables to proper- ties. The results are show in Appendix Table ??. Note that for Cor instead we calculate avg using [w', y).
Figure 4: Generation of dSpirates im- ages under different constraints. Prop- erties extracted via pre-trained models are illustrated the top right corner of each image. (a) shape=l (square), size=0.9, x position=0.8, y position=0.8, x+y position= 1.6; (b) shape=l (square), size=0.9, position=0.6, y positionE [0.3, 0.4] (c) shape=2 (ellipse), size=0.9, x position=-20, y position=0.4; (d) shape=2 (ellipse), size=0.5, posi- oc,=-oc,y position= oo
5.3.2 Overall performance
We use avgMI to evaluate the overall performance of the proposed model and comparison models. As shown in Appendix Table ??, CorrVAE achieves aligned avgMI with PCVAE and semi-VA in both dSprites and Pendulum datasets, the value which is rather small, suggesting that VAE achieves strong mutual dependence between the latent variables and the corresponding properties. By contrast, CSVAE has a larger avgMI, which results from the fact that CSVAE bonds the property with all latent variables rather than a single variable in w. This might affect the efficiency of the model to enforce the mutual dependence between latent variables and properties. We also visualize generated images of CAI in Appendix Figure and they all look similar with those in dSprites. According to Appendix Table ?? and Appendix Table ??, both molecule and image data are generated well by the proposed model since CArVVE achieves 100% validity and novelty on molecular generation and comparable reconstruction error and FID values on image generation.
5.3.3 Property prediction evaluation
We evaluate the learning ability of the proposed model and comparison models by the MSE between predicted properties and the true properties. As shown in Table T CorrVAE achieves the lowest size MSE of 0.0016 compared with comparison models, except CSVAE. Nevertheless, VAE has a smaller MSE (0.0066) of x+y position than CSVAE (0.3563), showing the superior ability of Cor to handle correlated properties. On Pendulum dataset, CorVVAE achieves the lowest MSE of pendulum angle (36.37) among all models except Semi- VAE (9. 9455). This might result from the fact that the correlation of properties in Pendulum dataset much more complex than that in dSprites, which is hard for the mask layer to capture. However, the performance of CorVVA is also satisfying
8

---

Table I: CorrVAE compared state-of-the-ar methods on dSprites and Pendulum datasets according MSE
between predicted correlated properties and true properties.
Malood
Abee
Epusiiin
dSprits
ponelalore
Aight pussiion
Penduium
shndow
shadley lenpik
CSVAE
B00006
030085
0,356
184-057
106 3864
4.3027
VAL
B0001
000000
00033
93655
5,5635
1.3296
0,7315
PONME
00008
0.007
00038
37.3644
12,2373
2,2977
0.7669
10025
303255
11,2973
6,379
Von
100.9
1555.35
475,3411
300.753
72:4597
VonVAE
36.3000
15,3000
regarding the MSE of light position and shadow position, compared with comparison models. Not surprisingly, shown in Table [T, CorrVAE-1 achieves better performance than CorVVE on those correlated properties since it uses the ground-truth mask to control the correlation among properties, which is challenging for Corr VAE to learn. Besides, the performance of Corr and CorrVAE are rather close on independent properties such as size for dSprites and pendulum angle as well as shadow position for Pendulum. This might be due to that size is independent with two other variables in dSprites while pendulum angle and shadow position are independent conditioning on light position and shadow position Pendulum. The independence can be well captured by CorrVAE (Figure 5 and Appendix Figure 4) so that it has comparable performance on those variables with CorrVAE- VAE2 has worse performance in both dSprites and Pendulum datasets, as shown in Table T This is because that Corr VAE-2 models w to w' using simple linear regression, which cannot capture the non-linear correlation among properties that might exist in the dataset.
5.3.4 Controllable generation quality
As shown in Appendix Table ??, for the MSE between generated and expected MolWeight compared with comparison models, CorrVAE achieves the lowest MSE of MolWeight(356701.. .5) and the MSE of logP(24.01) only larger than the Semi-VAB(15.1)) 13) on the QAC dataset; CorrVAE achieves a comparable MSE(lggP: 2.75, MolWeight: 4476.54) on the QM9 dataset. Meanwhile, CorrVAE has a lower MSE between generated and expected logP than Semi- VAE. We also evaluate the quality of generated molecules based on the QAC and QM9 dataset, in which CorrVAE and all comparison models have a satisfying performance (Appendix Table ??).
5.4 Qualitative evaluation
We qualitatively evaluate the ability of the property control of Cor using the dSprites dataset. We firstly interpret the mask pooling layer learned in the training process. Then we visualize the change of properties when manipulating corresponding latent variables in w. Lastly, we visualize the images that preserve target properties achieved based on section 4.2 In addition, we pre-trained four predictors extract properties, including shape, scale, x position, y position and x+y position, from generated images. All these models were pre-trained in the training set of CorrVAE with the MSE less than 0.001. The generated images in Figure Figure 4 are annotated by their corresponding features extracted by these pre-tained models. Details regarding the structure of pre-trained models are explained in Appendix Table ??.
5.4.1 Interpreting mask matrix
We train CorrVAE using shape, scale and three correlated properties x position, y position and x+y position while setting the dimension of w 8. As shown in Figure eventually we obtained an interpretable mask matrix indicating that w1 only controls shape and Wy only controls seale. This aligned with the fact that those two properties are independent with others in the data. We also observed that wy simultaneously controls position, position and x+y position, indicating that they are correlated since x+ position is generated from x position and y position. The correlation among those three variables is also captured by u4 that simultaneously controls x position and x+ position, and W3 that simultaneously controls y position and x+y position. In addition, there is one single variable We that only controls y position and another single variable wg that only controls position.
5.4.2 Property control by manipulating latent variables
The mask matrix learned in the training process (Figure enables the mask pooling layer of Cor- rVAE (Figure 2 to control properties accordingly. Based on the mask matrix shown in Figure 5 as shown in Figure 3 (a), we traverse the value of wh within [-5, 5] and the shape of the pattern changes accordingly from ellipse to square. Moreover, traverse the value of wz that controls

---

size of the object within [- and expectedly, the size of the pattern keep decreasing from 1.3877 to 0.3406, as shown Figure (b). If we traverse on Wa that controls both x position and x+y position within [-5, 5). we find that the x position of the object moves from the left to the right while x+ position changes accordingly (Figure 3 (c)). Similarly, when we traverse Wi that controls both y position and x+y position within 1-5,50 5/. the position moves from the bottom to the top (Figure (d) while x+y position also changes accordingly. We also evaluate the more complex setting by traversing the value of wa within |-5, 5] that simultancously controls x position, y po- sition and x+y position. Not surprisingly, the position of the pattern changes in both horizontal and vertical directions, corresponding to x+y position. At the mean time, position and posi- tion change accordingly, as shown in Figure (e). We also showcase the whole batch (eight) of generated images in Appendix Figure 2 corresponding each constraint of Figure 4 All images for the same constraint look similar, indicating the consistency and the replicability of our model.
5.4.3 Multi-objective generation
Wg
Wg
Wg
W4
W2
M4
Wg
We evaluate the performance of VAI for the multi-objectiv generation on the dSprites dataset. The experiments are performed based on the model that controls five properties, shape, scale, x position, y position and y+y position, and the mask matrix learned from the training process (Figure5)). Four sets of property constraints are considered and the corresponding w" are obtained according to section 4.2) Then the image is generated by the object decoder of Cor from w". Specifically, when the set of target properties is shape=square, size=0.9, position=0.8, y position=0.8 and x+y position=1.6, the location of the pattern leans roughly toward the lower right corner and has shape of square (Figure 4 (a)). Besides, when the constraints of target properties are set as shape=square, size=0.9, x position=0.6 and y positiong [0.3, 0.4), that the generated pattern has a large size and is located towards the right hand side, shown in Figure (b). (b) If minimize the x position and set the shape=ellipse, size=0.9 and position=0.4, as shown in Figure (c), a pattern of an ellipse with large size but at the very top of the image is observed while the y position roughlt aligns with the constraint. Lastly, we decrease the size, minimize x position and maximize position. Based on Figure 4 (d), the generated pattern is located the very lower left corner and has much smaller size compared with other generated images. In conclusion, the results show our model can generate objects with target properties based on the multi-objective optimization framework. All properties including shape are roughly aligned with the constraints, as shown in Figure 4
shape
0
0
D
0
0
size
0
position
y position
0
1
xey position
0
0
1
Figure 5: The mask matrix learned by the training process. Each column corresponds to one latent variable in w. Each row corresponds to prop- erty. In our experiments set- ting, five properties, shape, scale, x position, position and x position, are handled.
can visualize
6 Conclusion
In this paper, we attempt to tackle several challenges in multi-objective data generation by proposing a novel deep generative model. Firstly, we identify three challenges in generating data preserving target correlated properties. Secondly, we propose CorrVAE that includes mask pooling layer to identify and control correlation among properties, and a multi-objective optimization framework to generate data with desired properties. Comprehensive experiments were conducted on real-world datasets and our model shows superior performance than comparison models. Future work will be done on testing other multi-objective optimiation techniques.
Acknowledgments and Disclosure of Funding
This work was supported by the NSF Grant No. 2007716, No. 2007976, No. 1942594, No. 1907805, No. 1841520, No. 1755850, Meta Research Award, NEC Lab, Amazon Research Award, NVIDIA GPU Grant, and Design Knowledge Company (subcontract number: (0827.002. 120.044..
References
[1] A. Alemi, I. Fischer, Dillon, and K. Murphy. Deep variational information bottleneck. In ICLR, 2017. URL http:httt///arat http///awww..rr/////////...... wwtpaabb///..o... 00410
10

---

[2] J. Behrmann, W. Grathwohl, R. T. Chen, D. Duvenaud. and J.-H. Jacobsen. Invertible residual networks. International Conference on Machine Learning, pages 573-582. PMLR, 2019.
[3] R. Q. Chen, X. Li, R. B. Grosse, and D. K. Duvenaud. Isolating sources of disentangle- ment in variational autoencoders. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Sys- lems, volume 31, 2018. URL htttppp://ppceeeennngggeoee ins.uuuurinnnss.uurpp.......... cc/ //pppr//01////llle Iee3df www.ca/6666222233399999722222222 papp Pap pdf
[4] Y. Deng. J. Yang, D. Chen, Wen, and X. Tong. Disentangled and controllable face image generation via 3d imitative-contrastiee learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5154-5163, 2020.
[5] L. Dinh, J. Sohl-Dickstein, and Bengio. Density estimation using real nvp. 2016.
[6] Y. Du, Wang. X. Guo, H. Cao, S. Hu, J. Jiang, Varala, A. Angirekula, and L. Zhao. Graphgt: Machine learning datasets for graph generation and transformation. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2). 2021.
[7] Y. Du, X. Guo. H. Can, Y. Ye, and L. Zhao. Disentangled spatiotemporal graph generative model. AAAI, 2022.
[8] Y. Du, Guo, H. Cao, Y. Ye, and L. Zhao. Disentangled spatiotemporal graph generative models, arXiv preprint arXiv:2203.000 00411,, 2022.
[9] Y. Du, X. Guo, A. Shehu, and I.. Zhao. Interpretable molecular graph generation via monotonic constraints. In Proceedings of the 2022 SIAM International Conference on Data Mining (SDM), pages 73-81. SIAM, 2022.
[10] Y. Du, X. Guo, Y. Wang, A. Shehu, and L. Zhao. Small molecule generation via disentangled representation learning. Bioinformatics (Oxford, England), page btac296, 2022.
[11] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial networks. Communications of the ACM, 63(11):139-144, 2020.
[12] X. Guo and L. Zhao. A systematic survey on deep generative models for graph generation. arXiv preprint arXiv:2007.06666, 2020.
[13] X. Guo, Y. Du, and I.. Zhao. Property controllable variational autoencoder via invertible mutual dependence. In International Conference on Learning Representations, 2020.
[14] X. Guo, S. Tadepalli, L. Zhao, and A. Shehu. Generating tertiary protein structures via an interpretative variational autoencoder. arXiv preprint arXiv:2004.071199 2020.
[15] X. Guo, L. Zhao, Z. Qin, L. Wu, A. Shehu, and Y. Ye. Interpretable deep graph generation with node-edge co-disentanglement. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pages 1697-1707, 2020.
[16) X. Guo Y. Du*, and L. Zhao. Disentangled deep generative model for spatial networks. ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2021.
[17] X. Guo, Wang, and L.. Zhao. Graph neural networks: Graph transformation. Graph Neural Foundations, Frontiers, and Applications, pages 251-275. Springer, 2022.
Networks:
[18] A. Gupta, A. Agarwal, P. Singh, and P. Rai. A deep generative framework for paraphrase generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
[19] V. Hahn and C. K. Mechefske. Self-supervised learning for tool wear monitoring with a disentangled- disentangied-varaationaaaaau variationnal-atloeooooe International Journal of Hydrromechatronics, 4(1):69-98, 2021.
[20] G. Henter, J. Lorenzo- Trueba, X. Wang, and Yamagishi, Deep encoder-decoder models for unsupervised learning of controllable speech synthesis. arXiv preprint arXiv:1807.11470 2018.
11

---

[21] I. Higgins, L. Matthey, A. Pal, C. Burgess, X. Glorot, M. Botvinick, S. Mohamed, and A. Ler- chner. beta-vae: Learning basic visual concepts with a constrained variational framework. 2016.
[22] T. Iqbal and S. Qureshi. The survey: Text generation models in deep learning. Journal of King Saud University- Coopuute and Information Sciences, 2020.
[23] A. H. Jha, S. Anand, M. Singh, and V. Veeravasarapu. Disentangling factors of variation with cycle-consistent variational auto-encoders. In Proceedings of the European Conference on Computer Vision (ECCV), pages 805-820, 2018.
[24] W. Jin, R. Barzilay, and T. Jaakkola. Junction tree variational autoencoder for molecular graph generation. In International conference on machine learning, pages 2323-2332. PMLR, 2018.
[25] W. Jin, R. Barzilay, and T. Jaakkola. Multi-objective molecule generation using interpretable substructures. In International Conference on Machine Learning, pages 9849-4859, PMLR, 2020.
[26] N. S. Keskar, B. McCann, I.. R. Varshney, C. Xiong, and R. Socher. Ctrl: A conditional transformer language model for controllable generation. arXiv preprint 1exivv1909/0 9988085888, 2019.
[27] D. P. Kingma and M. Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 2013.
[28] J. Klys, J. Snell, and R. Zemel. Learning latent subspaces in variational autoencoders. Advances in Neural Information Processing Systems, 31, 2018.
[29] Li, M. Liu, and C. Walder. Editvae: Unsupervised part-aware controllable 3d point cloud shape generation. In AAAI, 2022.
[30] Y. Liao, K. Schwarz, L. Mescheder, and A. Geiger. Towards unsupervised learning of generative models for 3d controllable image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5871-5880, 2020.
[31] Locatello, M. Tschannen, S. Bauer, G. Rätsch, B. Schölkopf, and O. Bachem. Disentangling factors of variation using few labels. In ICLR, 2020.
[32] D. Marcheggiani and L. Perez- data to text generation. In Proceedings of the 11th International Conference on Natural Language Generation, pages 1-9, Tilburg University, The Netherlands, Nov. 2018. Association for Computational Linguistics. URL http:::tt/:/ttt http::/w/w/wwwwwwbbbbb/... aclweb. orgg/nnhology//1B--666
Deep graph convolutional encoders for structured
[33] L. Matthey, I. Higgins, D. Hassabis, and A. Lerchner. dsprites: Disentanglement testing sprites dataset. https://github.comm// http:///ww.occcco/////seeeeeeeee datasss 2017.
[34] D. N. Metaxas, L.. Zhao, and X. Peng. Disentangled representation learning and its application to face analytics. In Deep Learning-Baste Based Face Analytics, pages 45-72. Springer, 2021.
[35] A. Oussidi and A. Elhassouny. Deep generative models: Survey. In 2018 International Conference on Intelligent Systems and Computer Vision (ISCV), pages 1-8. IEEE, 2018.
[36] Z. Pan, W. Yu, X. Yi, A. Khan, F. Yuan, and Y. Zheng. Recent progress on generative adversarial networks (gans): A survey. IEEE Access, 7:36322-36333, 2019.
[37] Panwar and P. Michael. Empirical modelling of hydraulic pumps and motors based upon the latin hypercube sampling method. International Journal of Hydromechatronics, 1(3):272-292. 2222, 2018.
[38] A. Plumerault, H. L. Borgne, and C. Hudelot. Controlling generative models with continuous factors of variations. In ICLR, 2020.
[39] R. Ramakrishnan, P. O. Dral, M. Rupp, and O. A. Von Lilienfeld. Quantum chemistry structures and properties of 134 kilo molecules. Scientific data, 1(1):177,20 2014.
12

---

[40] D. Rezende and S. Mohamed. Variational inference with normalizing flows, In International conference on machine learning, pages 1530-1538. PMLR, 2015.
[41] C. Shi, M. Xu, Z. Zhu, W. Zhang, M. Zhang, and J. Tang. Graphaf: a flow-based autoregressive model for molecular graph generation. In ICLR,
[42] V. K. Singh and T. Nath. Energy generation by small hydro power plant under different operating condition. International Journal of Hydromechhrronics, 4(4))331-34) 2021.
[43] L.. Tran, X. Yin, and X. Liu. Disentangled representation learning gan for pose-invariant face recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1415-1424, 2017.
[44] S. Wang. Y. Du, X. Guo, B. Pan, and L. Zhao. Controllable data generation by deep learning: A review. arXiv preprint arXiv:2207.0955 2022.
[45] Wang, X. Guo, and L. Zhao. Deep generative model for periodic graphs. arXiv preprint arXiv:2201.11932, 2022.
[46] M. Yang. F. Liu, Z. Chen, X. Shen, J. Hao, and J. Wang. Causalvae: Disentangled representation learning via neural structural causal models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 9593-9602, 2021.
[47] You, B. Liu, R. Ying, V. Pande, and Leskovec. Graph convolutional policy network for goal-directed molecular graph generation. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS, 2018, page 6412 6422, Red Hook, NY, USA, 2018.
[48] J. You, R. Ying, X. Ren, W. Hamilton, and J. Leskovec. Graphrnn: Generating realistic graphs with deep auto-regressive models. International conference on machine learning, pages 5708-5717. PMLR, 2018.
[49] Z. Zhang and L. Zhao. Representation learning spatial networks. Advances in Neural Information Processing Systems, 54:2303-2318, 2021.
13


Reviews:
[{"review_id": "smBe3JwDx2b", "reviewer": "Reviewer_GPtJ", "summary": "The paper tackles the problem of controllable data generation when the controlled attributes y are correlated.\nTo do so, it proposes to separate the latent code in a disentangled VAE into two parts, z and w, where w contains the variables that are correlated to y, and z the variables independent of w (and hence to y). The disentangled variables w are combined into correlated variables w' via a learned mask M that indicates for each value of w', which values of w contribute to it. The values of w' are mapped one-to-one to the attributes y using an invertible network.\nThe model is used for controlled generation, to identify correlations between attributes, and for generation using multi-objective constrained optimization.", "questions": "-  The samples shown in Fig3 and Fig4 for dShapes are very bad, even in disentanglement VAE models standard. The shape attribute, that should be encoded in the independent z variables, are not only not conserved with attribute manipulation, but also they do not seem to be shapes that are in the dataset. This drastically limits the usefulness and significance of the method for data generation. Can the authors provide reconstruction errors and FID for generated data, for both seen and unseen combination of attributes?\n- The authors discuss in section 5.4 and show in Fig 3 latent traversals of w. However, it would appear that the w space is not that interesting, as they do not align well with w' and y. For instance, both the \"x position\" and \"x+y position\" attributes are diluted into two variables (figure 5). This is especially surprising since it increases the KL term in Eq3, and the mask sparcity loss. Meanwhile, the arguably more interesting variable w' that would be used for attribute manipulation is barely investigated. Can the authors provide qualitative samples for latent traversal of w', and quantitative figures on how well the attributes are retained?\n- I would also like to see the full mask M in Figure 5. What about \"y position\" for instance?\n- While the authors claim that the ablation CorrVAE-1 that use ground truth masks are achieving better performance than CorrVAE, it is not clear in the result Table 1. This could also indicate that the model is not working as expected and that different variables in the model do not exactly capture the information they are intended to get. Why would CorrVAE-1 would be slightly worse than CorrVAE on some task? How significant is the difference?\n- Important implementation details are in Appendix C. At the very least, the fact that the mask is encouraged to be sparse should be mentioned in the main paper. I would argue that approximations and relaxation of the problem, such as Monte-Carlo, Gumbel-SoftMax and Spectral Normalization should also be mentioned in the main paper when used as they are not exact implementations of the provided formula.\n\nOverall, I believe the paper overclaims what is the proposed model able to do. It indeed preserve the controlled attributes, arguably better than the baselines, but seemingly at the cost of loosing the other attributes, even some as fundamental as shape or orientation.\nI could change my opinion if the authors can provide evidence that this assessment is wrong (by answering the questions for instance), or if they can show that the trade-off is a desirable feature.\n\n\nMinors typos: \n\nl199: thrid\n\nl183 w_i^T ·  w_j  : w_i \n\nl326-327: it seem it should be CorrVAE-2 insteand of CorrVAE-1?\n\nVertical spacing after subsection titles are very unusual.", "limitations": "The paper do not mention its limitations.\nI believe the weaknesses raised above do qualify as limitations in term of data generation and should be adressed in the paper.", "rating": 6, "confidence": 4, "soundness": 1, "presentation": 3, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper address the problem of attributes correlation that is often ignored in controlled generation in a principled way.\n- Experiments have been conducted on real molecular data, in addition to synthetic images.\n\nWeaknesses:\n- The experiments fail to sufficiently back the claims made in the paper. Crucially, the paper is framed as a data generation method but the proposed experimental protocol do not assess the quality of the generated data, only the perservation of a few attributes.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "m3Mgv9w3Xtt", "reviewer": "Reviewer_2J2q", "summary": "Generating data with multiple constraints on its correlated properties is a critical task. The authors address this task by designing a new mask pooling layer to identify and control correlated properties using independent latent variables. These latent variables are bond to properties based on the mutual dependence. Then these latent variables are optimized to generate data with desired properties under a multi-objective optimization framework. The effectiveness of the proposed model has been shown on the molecule and image datasets in the evaluation session.", "questions": "1. Provide more discussion about the results in Table 1. Why CorrVAE has worse performance than CSVAE on two independent variables x position and scale?\n2. Discuss the missing reference [1] in the Related works session.\n", "limitations": "The paper discusses the limitation when pointing out the potential future works in the Conclusion. \nThat the work does not have potential negative social impact.", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The tasks approached by the paper are challenging but practically important. The framework proposed by the authors is novel.\n1.1 The proposed framework includes a mask pooling layer learned to capture and control correlated properties via the independent latent variables. This also to some extent adds interpretability to the model.\n1.2 In addition, the framework employs a set of bridging latent variables w to aggregate information from w to predict properties. The exact recovery of w from properties can be achieved via an invertible constraint of mutual dependence.\n1.3 The authors formally propose a multi-objective optimization framework for simultaneously control correlated properties of generated data. It comes with the generality in terms of accommodating different optimization goals and constraints, which looks reasonable and interesting for controllable data generation.\n\n2. The experiments show the effectiveness of the model. \n2.1.\tThe mask pooling layer works well given the results shown in Figure 5 under the setting that x position and x+y position should be correlated with each other. \n2.2. Figure 3 shows the effect of disentanglement of the latent variable w. Correlated properties x+y position can be well controlled by the corresponding latent variable given by the results of mask pooling layer.\n2.3. As shown in Table 1, CorrVAE can handle correlated properties x+y position better than other comparison models on dSprites dataset. Table 2 shows the effectiveness of CorrVAE on molecule datasets on MolWeight and logP properties.\n\n3 The code of the paper is well packaged and published.\n\nWeakness:\n1. In Table 1, although the proposed method dominating the others, it would be great to provide more analysis about the results why sometimes the degree of superiority is very large while sometimes its performance is close to the others.\n2. The white text on Figure 3 and Figure 4 can be larger to be more readable. The fond size of Table 1 and Table 2 should be aligned.\n3. Missing references: For example [1] below can be discussed in the Related works session.\n\n[1] Xie, Yutong, et al. \"Mars: Markov molecular sampling for multi-objective drug discovery.\" arXiv preprint arXiv:2103.10432 (2021).\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "GU0VImi3IEs", "reviewer": "Reviewer_P6bk", "summary": "The paper tries to address conditional generation where conditioning properties are correlated.\nThey point out that existing works that tackle the controllable property generation do not consider the correlation between multiple properties, as it is difficult to identify property correlation. Therefore, they propose a deep generative model that uses invertible mapping to map correlated properties to latent independent variables. During learning, they learn to encode the property and other information through two different encoders and use a novel mask pooling layer on the latent representation corresponding to the property encoder to learn the correlated properties. \n\n\n", "questions": "\n\nThe proposed model seems really interesting but I had some confusion:\n\n1. corresponding properties in figure3 and 4 are not readable. \n2. It is not very clear to me how  this conditional property is simplified: q(w,z|x,y) = q(w,z|x)\n3. I am not sure why we need the J matrix in w'=wJ^TM , is not the masking matrix M learn if an element of w contribute to w' or not and how big the contribution is?\n4. I did not really understand assumption 2 where they derive x and y are independent given z and w.\n5. In equation (2), from the first line to the next where they assume y_i can be learned from w'_i, where w_is are independent, is this assumption always holds? so each w'_i is independent, each y_i is derived from one w'_i, but y_i are correlated. So I am a bit confused that if each y_i is only derived from one w'_i, and w'_is are independent, how can they produce a set of y_i that are correlated? This leads to my next question the learned latent w itself stated that elements in w are disentangled from each other, so what is the intension behind why we introduce another set w' that are also disentangled. \n", "limitations": "The proposed idea is very interesting, however, for property-controlled generation, each time it needs to solve the optimization problem to find the proper w for the target property, this could be expensive, also the paper did not compare with the approaches that do bayesian optimization in the latent to find a best patent variable that can generate the data with the target properties. ", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "strength: The problem they try to solve is beneficial for the community and the proposed method is interesting\n\nweakness:  the presentation of the paper could be improved, some parts are bit confusing. They did not compare with the model where they explicitly model the dependency between the properties using hirerichical model or with those who do unconditional generative model coupled with Bayesian optimization to do conditional generation. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept