GraphCG: Unsupervised Discovery of Steerable Factors in Graphs
Authors: Shengchao Liu
Chengpeng Wang
Weili Nie
Hanchen Wang
Jiarui Lu
Bolei Zhou
Jian Tang
Venue: Submitted to ICLR 2023 (2023)
Keywords: graph
controllable generation
molecular graph
point clouds

Abstract: Deep generative models have been widely developed for graph data such as molecular graphs and point clouds. Yet, much less investigation has been carried out on understanding the learned latent space of deep graph generative models. Such understandings can open up a unified perspective and provide guidelines for essential tasks like controllable generation. To this end, this work develops a method called GraphCG for unsupervised discovery of steerable factors in latent space of deep graph generative models. We first examine the representation space of the recent deep generative models trained for graph data, and observe that the learned representation space is not perfectly disentangled. Thus, our method is designed for discovering steerable factors of graph data in a model-agnostic and task-agnostic manner. Specifically, GraphCG learns the semantic-rich directions via maximizing the corresponding mutual information, where the edited graph along the same direction will possess certain steerable factors. We conduct experiments on two types of graph data, molecular graphs and point clouds. Both the quantitative and qualitative results show the effectiveness of GraphCG for discovering steerable factors.

Under review as a conference paper at ICLR 2023
GRAPHCG: UNSUPERVISED DISCOVERY OF STEERABLE FACTORS IN GRAPHS
Anonymous authors Paper under double-bling review
ABSTRACT
Deep generative models have been widely developed for graph data such as molec- ular graphs and point clouds. Yet, much less investigation has been carried out on understanding the learned latent space of deep graph generative models. Such understandings can open up a unified perspective and provide guidelines for essen- tial tasks like controllable generation. To this end, this work develops method called GraphCG for unsupervised discovery of steerable factors in latent space deep graph generative models. We first examine the representation space of the recent deep generative models trained for graph data, and observe that the learned representation space is not perfectly disentangled. Thus, method is designed for discovering steerable factors of graph data in a model-agnostic and task-agnostic manner. Specifically, GraphCG learns the semantic-rich directions via maximizing the corresponding mutual information, where the edited graph along the same direc- tion will possess certain steerable factors. We conduct experiments on two types of graph data, molecular graphs and point clouds. Both the quantitative and qualitative results show the effectiveness of GraphCG for discovering steerable factors.
INTRODUCTION
Graph is a general format for many real-world data. For instance, molecules can be treated as graphs [10, 14] where the chemical atoms and bonds correspond to the topological nodes and edges respectively. Processing point clouds as graphs is also a popular strategy [53, 58], where points are viewed as nodes and edges are built among the nearest neighbors. Many existing works on deep generative models (DGMs) focus on modeling the graph data and improving the synthesis quality. However, understanding DGMs on graph and their learned representations has been much less explored, which may hinder the development of important applications like the controllable generation (also referred to the data editing) and the discovery of interpretable data structure.
The graph controllable generation task refers to modifying the steerable factors of graph so as to obtain graphs with desired properties [9, 43]. This is an important task in many applications, but traditional methods (e.g., manual editing) possess certain inherent limitations under certain circumstances. A clas- sic example is molecule editing, which aims at modifying the substructures of molecules [38] and can relate to some key tactics in drug discovery like functional group change [13] and scaffold hopping [2, 233. This is a routine task in pharmaceutical companies, yet, relying on domain experts for manual edit- ing can be subjective or biased [9, 15). Different from previous works, this paper aims to explore the unsupervised graph editing with DGMs. It can
good complementary module to conventional methods and bring many crucial benefits: (1) It enables the efficient graph editing in the sarge-scale set- ting. (2) alleviates the requirements for extensive domain knowledge for factor change labeling. (3) It provides another perspective for editing preference, which reduces biases from the domain experts.
One core property relevant to the general unsupervised data editing using DGMs is the disen- tanglement. While there does not exist a widely-accepted definition of disentanglement, the key intuition [36] is that a disentangled representation should separate the distinct, informative, and steerable factors of variations in the data. Thus, the controllable generation task would become trivial with the disentangled DGMs as the backbone. Such a disentanglement assumption has been widely used in generative modeling on the image data, e.g., 3-VAE [19] learns disentangled representation by forcing the representation to be close to an isotropic unit Gaussian. However, it may introduce extra constraints on the formulations and expressiveness of DGMs (11, 19, 47, 60].

---

Under review as a conference paper at ICLR 2023
Edited Molecule a a--3
Direction
2
Anehor Molecule
L
GraphCG
Decode
a-0
>
Contrast
Edited Molecule
Align
Edited Mololecule
Anchor Molecule
(a) Training phase of GraphCG.
(b) Test phase of GraphCG.
Figure I:
(a) The training phase. Given two latent codes z" and x", we edit the four latent representations along i-th and j-th direction with Step size and / respectively. The goal of GraphCG is to aliga the positive pair (z",.. and ) and contrast them with zj,8 and zÿ,e respectively. (b) The test phase. We will first sample an anchor molecule, and adopt the learned directions in the training phase for editing. With step size E [-3,3] 3). we can generate a sequence of molecules. Specifically, after decoding, there a functional group change shown up: the number of hydroxyl groups decreases along the sequence in the decoded molecules.
For graph data, one crucial question remains: is the latent representation space learned from DGMs on graph data disentangled? In image generation, prior work [36] shows that without inductive bias, the latent space learned by VAEs is not guaranteed to be disentangled. However, the disen- tanglement property of graph DGMs is much less explored. In Sec. 3, we first study the latent space of DGMs on two typical graph data (molecular graphs and point clouds), and empirically illustrate that the learned space is not perfectly disentangled. This observation then raises the second question: Given a pretrained DGM with not perfectly disentangled latent space, is there flexible framework enabling the graph controllable generation in an unsupervised manner? To tackle this, we propose -agns-age and task lask-agnostic framework called GraphCG for unsupervised graph controllable generation. GraphCG has two main phases, as illustrated in Fig. 1. During the training phase (Fig. (a)), GraphCG starts with the assumption that the steerable directions can be learned by maximizing the mutual information (MI) among the semantic directions. We formulate GraphCG with an energy-based model (EBM). which provides large family of solutions. Then during the test phase, with the learned semantic directions, we can carry out the editing task by moving along the direction with certain step sizes. As the example illustrated in Fig. 1(b), the molecular structure (hy- droxyl group) changes consistently along the editing sequence. For evaluation, we visually verify the learned semantic directions on both types of graph data. Further for the molecular graphs, we propose novel evaluation metric called sequence monotonic ratio (SMR) to measure the output sequences.
We summarize our contributions as follows: (1) We conduct an empirical study on the disentanglement property of three pretrained deep generative models (DGMs) on two types of graph data, molecular graphs and point clouds. We find that the latent space of these pretrained graph DGMs is not perfectly disentangled. (2) We propose model-agnostic and task-agnostic method called GraphCG for the unsupervised graph controllable generation. GraphCG aims at learning the semantic directions by maximizing their corresponding mutual information, and its outputs are sequences of graphs. (3) We qualitatively evaluate the proposed methods on two types of graph data, molecular graphs and point clouds. Besides, the quantitative results further show the clear improvement over the baselines.
Related work. Recent works leverage the DGMs for various controllable generation tasks [5, 61], where the inherent assumption is that the learned latent representations encode rich semantics, and thus traversal in the latent space can help steer factors of data [17, 16, 52]. Among them, one research direction [40, 52] is using supervised signals learn the semantic-rich directions, and most works on editing the graph data focus on the supervised setting [27, 57, 64). However, these approaches can not be applied many realistic scenarios where extracting the supervised labels is difficult. Another research line [17, 45, 51] considers discovering the latent semanties in an unsupervised manner, but most unsupervised methods are designed to be either model-specific or task-specific, making them not directly applicable to the graph data. More comprehensive discussion is in Appendix B.
2 BACKGROUND AND PROBLEM FORMULATION
Graph and deep generative models (DGMs). Each graph data (including nodes and edges) is denoted as x: E X, where A' is the data space, and DGMs learn the data distribution, iee, p(a)). Our
2

---

Under review as a conference paper at ICLR 2023
proposed graph editing method (GraphCG) is model-agnostic, we briefty introduce the mainstream DGMs for graph data as below. Variational auto-encoder (VAE) [19, 31] measures a variational lower bound of p(x) by introducing a proposal distribution; flow-based model [8, 46] constructs revertible encoding functions such that the data distribution can be deterministically mapped to a prior
distribution. Note that these mainstream DGMs, either explicitly implicitly, contain encoder (f(-)) and a decoder (g()) parameterized by neural networks as:
f(=),
x' 9())
where z E 2 is the latent representation, is the latent space, and x' the reconstructed output graph. Since in the literature [51, 52], people also call latent representations as latent codes or latent vectors, in what follows, we will use these terms interchangeably.
Semantic direction and step size. In the latent space Z. assume there exist D semantically mean- ingful direction vectors, i.e., d: with E {0, 1....... D 1}. There is also scalar variable, step size a, which controls the degree edit the sampled data with desired steerable factors (as will be introduced below), and we follow the prior work [51] on taking a E 1-3.3) 3]. Each direction corresponds to one or multiple factors, such that by editing the latent vector with direction d; and step size a, the reconstructed graph will be augmented with the desired factors, leading to certain structural changes.
Steerable factors. The steerable factors attributes of DGMs, which usually refer the semantic information of data that we can explicitly discover from the pretrained DGMs. In this work, we focus on the steerable factors of graph data, which are data- and task-specific. Yet, there is one category of factors that is commonly shared among all the graph data: the structure information. Concretely, these steerable factors can be the functional groups or fragments in molecular graphs and shapes or sizes in point clouds. In Appendix C, we provide a detailed description of these steerable factors.
Problem formulation: graph controllable generation. Given a pretrained DGM (i.e., the encoder and decoder are fixed), our goal is to learn the most semantically rich directions (d.) in the latent space 2. Then for each latent code 2, with the i-th semantic direction and step size a, we can get an edited latent vector ži,a a and edited data o', as:
(2) where d, and h(.) are the edit direction and edit function that we want to learn. We expect that ž can inherently possess certain steerable factors, which can then be reflected in the graph structure of ø'.
f(x),
2,a = h(z,d,,000 d),
a' g(
Energy-based model (EBM). EBM is a flexible framework for distribution modeling:
p exp(-E(x))
exp(-F(xx)
A
fo exp((E(z))da
where E(-) is the energy function and A the partition function. In EBM, the bottleneck is the estimation of partition function A: it is commonly intractable due to the high cardinality of X. Various methods have been proposed handle this issue, including but not limited to contrastive divergence [20], noise-contrastive estimation [4, 16), and score matching [24, 56].
3 DISENTANGLEMENT OF LATENT REPRESENTATION
In this section, we quantify the degree of disentanglement of the existing DGMs for graph data. In specific, we adopt six disentanglement measures, and the observed low disentanglement scores show that compared to DGMs for image data, such as StyleGANs [28, 29], the attributes in latent space of graph DGMs are not well disentangled.
The key intuition [36] behind disentanglement is that disentangled representation space should separate the distinct, informative, and steerable factors of variations the data. In other words, each latent dimension of the disentangled representation corresponds to multiple factors. Therefore, the change of the disentangled dimension can lead to the consistent change in the corresponding factors of the data. This good property has become a foundational assumption in many existing controllable generation methods [17, 51, 52].
Is the latent space of graph DGMs disentangled? In image generation, [36] shows that without inductive bias, the representation learned by VAEs is not perfectly disentangled. To verify if this claim is also valid for the mainstream DGMs on graphs, we conduct the following experiment.
3

---

Under review as a conference paper at ICLR 2023
Table The six disentanglement metrics on three pretrained DGMs and two graph types, All measures range from O to I, and higher scores mean more disentangled representation.
Graph Type
Model
Dutuset
BetaVAB t
FactorVAB
MIG t
DCI 1
Modularity
t
Molecular Graph
MoFlow
ZINC250K
0.260
0.175
0.031
0.953
0.620
0.009
HierVAl
CLEMBI.
0.178
0.165
0.022
0.114
0.601
0.026
Point Cloud
PointFlow
Airplane
0.022
0.025
0.029
0.160
0.745
0.022
Steerable factors and experiments on disentanglement measure. There have been series of works exploring the disentanglement of the latent space in DGMs, and here we take six widely-used ones: Beta VAE [19], FactorVAE [30], MIG [6], DCI [1 IJ, Modularity [47], and SAP [32]. Each measure has its own bias, and we put a detailed comparison in Appendix C. Meanwhile, they all share the same high-level idea: given the latent representation from a pretrained DGM, they are proposed to measure how predictive it is to certain steerable factors.
To adapt them to our setting, first we need to extract the steerable factors in graph, which requires the domain knowledge. For instance, in molecular graphs, we can extract some special substructures named fragments or functional groups. These substructures can be treated as steerable factors since they are the key components of the molecules and are closely related to certain molecular properties [50]. We use RDKit [33] to extract 9 most distinguishable fragments as steerable factors for disentanglement measurement. For point clouds, we use PCL tool [48] to extract 75 VFH descriptors [49] as steerable factors, which depicts the geometries and viewpoints accordingly.
Then for measuring the disentanglement, backbonn models. All the metric values range from to I, and the higher the value, the more disentangled the DGM is. According to Table 1, We can observe that the most of disentanglement scores are quite low, except the DCI [11] on MoFlow. Thus, we can draw the conclusion that generally these graph DGMs not perfectly disentangled. More details of this experiment (the steerable factors on two data types and six disentanglement metrics) can be found in Appendix C.
consider six metrics two data types with three
OUR METHOD
The analysis in Sec. 3 naturally raises the next research question: given a not well-disentangled representation space, is there a flexible way to do the graph data editing? The answer is positive. We propose GraphCG, flexible model-agnostic and task-agnostic framework to learn the semantic di- rections in an unsupervised manner. It starts with the assumption that the latent representations edited with the same semantic direction and step size should possess similar information (corresponding to the factors) to certain degree, thus by maximizing the mutual information them, can learn the most semantic-rich directions. Then we formulate this editing task as a density estimation problem with the energy-basee model (EBM). As introduced in Sec. 2, EBM covers broad range of solutions, and we further propose GraphCG-NCE by adopting the noise-contrastive estimation (NCE).
4.1 GRAPH CONTROLLABLE GENERATION WITH MUTUAL INFORMATION
The mutual information (MI) measures the non-linear dependency between variables. To adapt it to our setting, we set the editing condition as containing both the semantic directions and step sizes, and we assume that maximizing the MI between different conditions can maximize the shared information within each condition. The pipeline is as follows.
We first sample two codes in the latent space, z" and z", Then we pick up the i-th semantic direction and one step size a to obtain the edited latent points in 2. The corresponding points are as
th = h===,,==" ,d,
: = h(2", d., a).
(4)
Under our assumption, expect that these two edited points share certain information with respect to the steerable factors. Thus, we want to maximize the MI between 2 and T Since the MI is intractable to compute, we adopt the EBM lower bound [35] as:
Laal Ei... tl.a)
( = Epcey
2 B RY. lo (x(,) + logg(= ("".)))
(5)
The detailed derivation is in Appendix D. Till this step, we have transformed the graph data editing task into the summation of two conditional log-likelihoods estimation problem.
4

---

Under review as a conference paper at ICLR 2023
4.2 GRAPHCG WITH ENERGY- BASED MODEL
Following Eq. (5), maximizing the MI between / ( 2aaa) is equivalent to estimating the summa- tion of two conditional log-likelihoodk We then model them using two conditional EBMs. Because these two views in the mirroring direction, we may well take one for illustration. For example, for the first conditional log-likelihoode we can model it with EBM as:
exp( F( ET,.)) I Texp(- Expp(=B(x,,,) 2., ddz
exp(f
p p(
(6)
Ag
where E() is the energy function, Aij is the intractable partition fiucctio, and f(-) is the negative energy. The energy function can be quite flexible, and for simplicity, we use the dot-product:
| a (1(=", d. ax,, .d.
where h(-) is the editing function introduced in Eq. (2). Similarly for the other conditional log- likelihood term, and the objective becomes:
exp(f(Ef... F(.a) + log
exp(f(E),a
(i Fia)))
LomphCG
_[ log
(8)
Aij
Aji
With Eq. (8), we are able to learn the semantically meaningful direction vectors. We name this unsu- pervised graph controllable generation framework as GraphCG. In specific, GraphCG utilizes EBM for estimation, which yields a wide family of solutions. Next we will introduce an intuitive solution.
4.3 GRAPHCG WITH NOISE CONTRASTIVE ESTIMATION
We solve Eq. (8) using the noise contrastive estimation (NCE) [16]. The high-level idea of NCE is to transform the density estimation problem into a binary classification problem that distinguishes if the data comes from the introduced noise distribution from the true distribution. NCE has been widely explored for solving EBM [55], and we adopt it as GraphCG-NCE by optimizing:
ConghOX-NCu -- (Emita 1 1e log (1 of =", Eyun 187 [log [log D.,
(9)
+ (1 loog (1 off(F), (.,D) Bran 14, (ogg (D.)))
where Pdata the data distribution and yn is the noise distribution (derivations are in Appendix D). Recall that the latent pairs are given, and the noise distribution is on the semantic directions and step sizes. In specific, the step sizes (a * (9) are randomly sampled from [-3, 3), and the latent direction indices (i # j) are randomly sampled from (0, I, .... D-1 J. The objective Eq. (9) is for one latent code pair, and we will take the expectation of it over all the pairs from the dataset. Besides, we would like to consider extra similarity and sparsity constraints as:
Cim = Be.jssim(dd (d;, d,)]:
Exprrity = E=[[ II]
(10)
where sim(-) is the similarity function between two latent directions, and we use the dot product. By minimizing these two regularization terms, we can make the learned semantic directions more diverse and sparse. Putting them together, the final objective function is:
C = Eu, E.. [Cangaco NCE] + c2 Laim + cs Lyparityy
(11)
where C1, C22 C3 are coefficients, and we treat them as three hyperparameters (check Appendix E). The above pipeline is illustrated in Fig. 1, and for the next we will discuss certain key modules.
Latent pairs, positive and negative views. We consider two options in designing the latent pairs. (1) Perturbation (GraphCG-P) is that for each latent variable z Z, we apply 2 perturbations (e.g., adding Gaussian noise) on to get perturbed latent codes as 2 and z" respectively. (2) Random sampling ( (GraphCG-R) is that we encode two randomly sampled data points from the empirical data distribution as z" and x" respectively. Perturbation is one of the widely-used strategies [28] for data augmentation, and random sampling has been widely used in the NCE [55] literature. Then we can define the positive and negative pairs in GraphCG-NCE, where the goal is to align the positives and contrast the negatives. As described in Eq. (9), the positive pairs latent pairs moving with the same semantic direction and step size, while the negative pairs are the edited latent codes with different semantic directions and/or step sizes.
5

---

Under review as a conference paper at ICLR 2023
Semantic direction modeling. We first randomly draw basis vector e,, and then model the semantic direction d, as d: = -LP((e),, where MLP(.) is the multi-layer perceptron network.
Design of editing fuucctio. Given the semantic direction and two views, the next task is to design the editing function (()) in Eq. (2). Since our proposed GraphCG is flexible, and the editing function determines the energy function Eq. (7), we consider both the linear and non-linear editing functions as:
2 = + a d,,
Σ, = + a d + MLP(= + d. @ [a]),
(12)
where @ is the concatenation of two vectors. Noticing that for the non-linear case, we are adding an extra term by mapping from the latent code, semantic direction, and step-size simultaneously. We expect that this could bring in more modeling expressiveness in the editing function. For more details, e.g., the ablation study to check the effect on the design of the views and editing functions, please refer to Appendices F and G, while more potential explorations are left for future work.
4.4 IMPLEMENTATIONS
During training, the goal of GraphCG
Algorithm Training Phase of GraphCG
is to learn semantically meaningful di-
1: Input: Given pretrained generative model, f()) and g(-).
rection vectors together with edit-
2: Output: Learned direction vector d, and function h(-).
ing function in the latent space, as in Algorithm 1. Then we need man-
3: Select data points, z" z",=" x² E Z.
4: for each step size and each direction do
ually annotate the semantic directions
5: Set æ, = h(=", a).
with respect to the corresponding fac- tors, using certain post-training eval- uation metrics. Finally for the test phase, provided with the pretrained
6: Set 2 = h(2", d,,a).
7: Assign positive to pair (#). EY,a).
8:
for step size * a and direction * i do
9: Set x = h(=".. (1= d;. B)-
DGM on graph and a selected seman- 10:
Set 2j,3 = h(=", d).
tic direction (together with a step size),
11:
Assign negative to pair (z 2 (18)
we can sample a molecule and use GraphCG for editing, as described
12:
Assign negative to pair (2), =..a)
end for
14: Do SGD w.r.t. GraphCG in Eq. (11).
in Eq. (2). The detailed algorithm is
15:
end for
illustrated in Algorithm 2. Next, we
highlight several key concepts in GraphCG and briefly discuss the differences to other related methods.
NCE and contrastive representation learning. GraphCG-NCE is apply- ing EBM-NCE, which essentially contrastive learning method, and another dominant contrastive loss is
Algorithm 2 Test Phase of GraphCG
1: Input: Given pre-trained generative model (f(-) and g()) ())),
learned direction vector d.
2: Output: A sequence of edited graphs.
3: Sample molecule with DGM or from a large molecule pool.
the InfoNCE [42]. We summarize
4: Encode the molecule to get latent code
their relations as below. (1) Both con-
5: for step size E [-3,3] do
trastive methods are doing the same
6: Do graph edit in the latent space to get 2i,a = h(2, d, at).
thing: align the positive pairs and con-
7: Decode to the graph space with x' = g(z..))
trast the negative pairs. (2) EBM-
8: end for
NCE [18, 35] has been found to out- perform InfoNCE on the certain graph
9: Output is thus sequence of edited graphs. (Ø').
applications like representation learning. (3) What we want to propose here is a flexible framework. Specifically, EBM provides a more general framework by designing the energy functions, and EBM- NCE is just one effective solution. Other promising directions include the denoising score matching or denoising diffusion model [56], while InfoNCE lacks such nice extensibility attribute.
GraphCG and self-supervised learning (SSL). GraphCG shares certain similarities with the self- supervised learning (SSL.) method, however there are some inherent differences, as summarized below. (1) SSL aims at learning the data representation by operating data augmentation on the data space, such as node addition and edge deletion. GraphCG aims at learning the semantically meaningful directions by editing on the latent space (the representation function is pretrained and fixed). (2) Based on the first point, SSL aims at using different data points as the negative samples. GraphCG, on the other hand, is using different directions and step-sizes as negatives. Namely, SSL is learning data representation in the inter-data level, and GraphCG is learning the semantic directions in the inter-direction level.

---

Under review as a conference paper at ICLR 2023
Output sequence in the discrete space. Recall that during inference time (Algorithm 2), GraphCG takes a DGM and the learned semantic direction to output a sequence of edited graphs. Comparing to the vision domain, where certain models [51, 52] have proven their effectiveness in many tasks, the backbone models in the graph domain have limited discussions. This is challenging because the graph data is in discrete and structured space, and the evaluation on such space non-trivial. Meanwhile, GraphCG essentially provides another way to testify the quality of graph generation models. We would like leave this for future exploration.
5 EXPERIMENTS
In this section, we show both the qualitative and quantitative results of GraphCG on two types of graph data: molecular graphs and point clouds. Due to the page limit, We put the experiment and implementation details in Appendix E. For reproducibility, the code will be public in the near future.
5.1 GRAPH DATA: MOLECULAR GRAPHS
Backbone DGMs. We consider two state-of-the-art DGMs for molecular graph generation. MoFlow [65] is a flow-based generative model on molecules which adopts an invertible mapping between the input molecular graphs and latent prior. HierVAE [27] is a hierarchical VAE model which encodes and decodes molecule atoms and motifs in a hierarchical manner. Besides, the pretrained checkpoints are also provided, on ZINC250K [25] and ChEMBL [37] dataset respectively.
Editing sequences and anchor molecule. As discussed in Sec. 4, the output of the inference in GraphCG is sequence of edited molecules with the i-th semantic direction, ( We first randomly generate a molecule using the backbone DGMs (without the editing operation), and we name such molecule the anchor molecule, æ= Then take 21 step sizes from -3 3, with interval 0.3, to obtain a sequence of 21 molecules following Eq. (2). Note that the edited molecule with step size O under the linear editing function is the same as the anchor molecule, i.e., "²
Change of structure factors and evaluation metries. We are interested in the change of the graph structure (the steerable factors) along the output sequence edited with the i-th semantic direction. To evaluate the structure change, we apply the Tanimoto similarity between each output molecule and the anchor molecule. Besides, for the ease of evaluating the monotonicity, we apply a Tanimoto similarity transformation on the output molecules with positive step sizes by taking the deduction from 2. We call this calibrated Tanimoto similarity (CTS) sequence, marked as (s(Ø') J. An illustration is shown in Fig. 2. Further, propose metric called Sequence Monotonic Ratio (SMR), ØSMR (7, T);, which measures the monotonic ratio of M generated sequences edited with the i-th direction. has two arguments: the diversity threshold y constrains the minimum number of distinct molecules, and the tolerance threshold т controls the non-monotonic tolerance ratio along each sequence.
Evaluating the diversity of semantic directions. SMR can evaluate the monotonic ratio of output sequences generated by one direction. To better illustrate that GraphCG is able to learn multiple directions with diverse semantic information, also consider taking the average of top- SMR to reveal that all the best K directions are semantically meaningful, as in Eq. (15).
Original Tanimote Samilarity
len(set((=( 2 monotonic, ({a(=))?") (13)
Tanimote Similarity
Callbrated Tanimoto Similarity
deMM ((s(#')) (_s(""))""">> (,7))
o.
otherwise
desM (7/, 7);
dhsMa ({s(a))) ,,.7))
(14)
35
top-K(y,T)
(Swa(T,7).))
(15)
K
-1.8
1.5
EElip dinasions
Step Size
Figure 2: This shows the sequence monotonic ratio (SMR) calibrated Tanimoto similarity (CTS). Eqs. (13) and (14) are the SMR on each sequence and each direction respectively, where M is the number of generated sequences for the i-th direction and {=(ø"))," is the CTS of m-th generated sequence with the i-th direction. Eq. (15) is the average of Kop-K SMR on D directions. More details are in Appendix F.
7

---

Under review as a conference paper at ICLR 2023
Table 2: This table lists the sequence monotonic ratio (SMR, %) on calibrated Tanimoto similarity (CTS) with respect to the top-1 1 and top-3 directions. The hest performances are marked in bold.
Tanimoto top-1
Tanimoto top-3
Model
Dataset
diversity 7
3
4
tolerance
0
0.2
0
0.2
0
0.2
0.2
Random
23.0
25.0
12.0
15.0
22.0
24.0
11.0
13.7
Variance
24.0
28.0
12.0
16.0
20.0
25.0
10.0
15.0
SeFa [51]
4.0
4.0
0.0
0.0
3.3
3.3
0.0
MoFlow
ZINC250K
DisCo [45]
7.0
14.0
2.0
8.0
11.7
GraphCG-P
32.0
34.0
16.0
18.0
29.0
31.0
13.7
GraphCG-R
25.0
26.0
11.0
14.0
22.0
24.3
10.3
13.3
Random
14.0
45.0
14.0
43.0
10.0
42.3
41.7
Variance
23.0
59.0
19.0
55.0
18.3
52.7
15.7
50.3
ScFa [5]]
4.0
41.0
4.0
41.0
2.3
36.0
2.3
36.0
HierVAEE
ChEMBL
GraphCG-P
40.0
73.0
32.0
65.0
36.0
64.3
26.3
57.7
GraphCG-R
42.0
67.0
30,0
55.0
38.0
62.3
28.7
53.7
6969
6969
20
w/
CT6 0.28
CTS. 1.00
1.20
0.39
cTS 1.00
1M
CT8:027 027
CTB
CTG 1.00
CTS 1:0
CTS
CTS: 6:45
CTS:
CTS 100
CTS mm
CTR
(a) Steerable factor: number of halogens.
(b) Steerable factor: number of hydroxyls.
Quaar
1.77 vnnil
IPSA
IPSA
IPSA: 51
SPAA: 53
IPA
NRB:
MRt:
Nnn.
NRt:
Ruuor EX
x
2
CT8 0:10
CT8
CT8 1.00
CTB: 1.83
CTS
CTB: 6-10
GT9
CTS 1:00
101
CTS:
(c) Steerable factor: number of amides.
(d) Stecrable factor: chain length.
Figure 3: GraphCG for molecular graph editing. We visualize the output molecules and CTS on four directions with two sequences each, where each sequenee consists of five steps. The center point is the anchor molecule, the other four points correspond to step size -3.- 1.8, and respectively. Figs. 3(a) to 3(c) show how functional groups in the molecules can be viewed as the stecrable factors as they change along the sequence, such as halogen atoms, hydroxyl groups and amides. Fig. 3(d) illustrates the effect on the steerable factor on the length of flexible chains in the molecules. Notably, certain properties change with molecular structures accordingly, like topological polar surface (tPSA) and numher of rotatable bonds (NRB).
Baselines. For baselines, consider four unsupervised editing methods. (1) The first is Random. It randomly samples normalized random vector in the latent space as the semantic direction. (2) The second one is Variance. We analyze the variance on each dimension of the latent space, and select the highest one with one-hot encoding as the semantic direction. (3) The third one is SeFa [51] [51]. It first decomposes the latent space into lower dimensions using PCA, and then takes the most principle components (eigenvectors) as the semantic-rich direction vectors. (4) The last one is DisCo [45].
maps each latent code back the data space, followed with an encoder for contrastive learning. Namely, it requires the backbone DGMs to be end-to-end and is infeasible for HierVAE.
Quantitative results. We take D 10 train GraphCG, and the optimal results on 100 sampled sequences are reported in Table 2. We can observe that GraphCG can show consistently better structure change with both top-1 and top-3 directions. This can empirically prove the effectiveness of proposed GraphCG. More comprehensive results are in Appendix F.
Analysis on steerable factors in molecules: functional group change. For visualization. we sample 8 molecular graph sequences along 4 selected directions in Fig. 3, and the backbone DGM is HierVAE pretrained on ChEMBL. The CTS holds good monotonic trend, and each direction shows certain unique changes in the molecular structures. i.e., the steerable factors in molecules. Some structural changes are reflected in molecular properties. We expand all the details below. In Fig. 3(a) and Fig. 3bb,, the number of halogen atoms and hydroxyl groups (in alcohols and phenols) in the
8

---

Under review as a conference paper at ICLR 2023
(a) Steerable factor: engine.
(b) Steerable factor: engine.
Steerable factor: size.
(d) Steerable factor: leg height.
Figure 4: GraphCG for point clouds editing. We show four editing sequences, where each sequence consists of five point clouds, and the center one is the anchor point clouds, ie.. with step size O. The other four point clouds correspond to step size with -3, -1.8. 1.8, and respectively. Fig. 4(a) and Fig. 4(b) refer the semantic direction, and they are showing how to steer the factor engine: number engines will be decreased and increased with the negative (left) and positive (right) step size respectively. Similarly, Figs. 4(c) and 4(d) illustrate the effect on the steerable factors on the car size and the chair leg height.
molecules decrease from left right, respectively. In Fig. 3(c), the number of amides in the molecules increases along the path. Because amides are polar functional groups, the topological polar surface area (tPSA) of the molecules also increase accordingly, which is key molecular property for the prediction of drug transport properties, e.g., permeability [12]. In Fig. 3(d), the flexible chain length, marked by the number of ethylene (CH&CH2) units, increases from left to right. Since the number of rotatable bonds (NRB) measures the molecular flexibility, it also increases accordingly [57].
5.2 GRAPH DATA: POINT CLOUDS
Backbone DGMs. We consider one of the latest DGMs on point clouds, PointFlow [62]. It is using the normalizing flow model for estimating the 3D point clouds distribution. Then consider PointFlow pretrained on three datasets ShapeNet [3]: Airplane, Car, and Chair. All point clouds obtained by sampling points uniformly from the mesh surface.
Analysis on steerable factors in point clouds: shape change. To train GraphCG, we take D = 10 directions, and sample point cloud sequences along 3 directions for visualization in Fig. More results are in Appendix G. It is observed that GraphCG can steer the shape of the point clouds, e.g., the size of cars and the height of chair legs. We also find interesting that GraphCG can steer more finger- finger-trained factors. like modifying the number jet engines of airplanes in Figs. 4(a) and 4(b).
CONCLUSION AND DISCUSSION
In this work, are interested in unsupervised graph editing. It is well-motivated task for various real-world applications, and we discuss two mainstream data types: molecular graphs and point clouds. We start with exploring the latent space of mainstream deep generative models and propose GraphCG, model-agnostic and aask-agnottic unsupervised method for graph data editing. The key component of GraphCG EBM, and we take the GraphCG-NCE as the solution for now. For the future work, we may well extend it to more advanced solutions like denoising diffusion model [21],
One limitation of GraphCG (as well as the solutions to general unsupervised data editing) [17, 45, 51] is that we may need some post-training human selection (as shown in Algorithms and 2) to select the most promising semantic vectors to steer factors. Another issue is the lack of open-sourced evaluation metrics. This requires both deep understanding of the representation space of deep generative models and domain knowledge of the problem. For instance, activity cliff a challenging task [22] for editing, while current measures fail capture it. To set up constructive evaluation metrics can help augment our understandings from both the domain and technique perspectives. This beyond the scope of our work, yet would be interesting explore a future direction.

---

Under review as a conference paper at ICLR 2023
ETHICS STATEMENT
We authors acknowledge that have read and commit to adhering the ICLR Code of Ethics.
REPRODUCIBILITY STATEMENT
To ensure the reproducibility of the empirical results, we provide the implementation details (hyperpa- rameters, dataset specifications, pretrained checkpoints, etc.) in Sec. 5 and appendices C and G, and the source code will be released in the future. Besides, the complete derivations of equations and clear explanations given in Sec. and appendix D.
Specifically,, we provide the details for reproducing the results:
In Table GraphCG-P with Eq. (26) and GraphCG-R with Eq. (24) reported in Table In Table 6, GraphCG-P with Eq. (25) and GraphCG-R with Eq. (25) are reported in Table 2.
For the visualization in Fig. take the GraphCG-P with Eq. (25), and the backbone generative model is HierVAE pretrained on ChEMBL. Further, we provide an anonymous link here. In these CSV files:
Direction O is the halogen fragment (data 4, 71).
Direction 5 the amide fragment (data 95, 61).
Direction 6 is the chain length (data 57, 14).
Direction 7 the alcohol and phenol fragments (data 10, 8).
For the visualization in Fig. 4, we take the GraphCG-R with Eq. (25) on PointFlow, for all three datasets.
REFERENCES
[1] Michael Arbel, Liang Zhou, and Arthur Gretton. Generalized energy based models. arXiv:2003.05033 2020. 17
[2] Hans-Joachim Böhm, Alexander Flohr, and Martin Stahl. Scaffold hopping. Drug discovery today: Technologies, 1(3):217-22,, 2004.
[3] Angel Chang, Thomas Funkhouser, Leonidas Guibas, Hanrahan, Qixing Huang, et al. Shapenet: An information rich 3d model repository. arXiv:1512.03012. 2015.
Tong Che, Ruixiang Zhang, Jascha Sohl-Dickstein, Hugo Larochelle, Liam Paull, et al. Your gan is secretly an energy-based based model and you should use discriminator driven latent sampling. arXiv:2003. 06060, 2020.
[5] Hongming Chen, Ola Engkvist, Yinhai Wang, Marcus Olivecrona, and Thomas Blaschke, The rise of deep learning in drug discovery. Drug discovery today, 23()):2241-1250 2018. 2
[6] Tian Qi Chen, Xuechen Li. Roger Grosse, and David Duvenaud. Isolating sources of disentanglement in variational autoencoders. In NeurlPS, 2018. 4,
Gabriele Corso, Luca Cavalleri, Dominique Beaini, Pietro Liò, and Petar Veličković, Principal neighbour- hood aggregation graph nets, arXiv:2004.05718, 2020.
[8] Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components estimation. arXiv:1410.8516 2014.
[9] Jürgen Drews. Drug discovery: historical perspective. Science, 287(5460):1964 1964, 2000. 1, 14
[10] David Duvenaud, Dougal Maclaurin, Jorge Hirzel, et al. Convolutional networks on graphs for learning molecular fingerprints. arXiv:1509.09200
Rafael
Timothy
2015. 14
[11] Cian Eastwood and Christopher KI Williams. A framework for the quantitative evaluation of disentangled representations. In ICLR, 2018. 1, 15
[12] Peter Ertl, Bernhard Rohde, and Paul Selzer. Fast calculation of molecular polar surface area a sum of fragment-hased contributions and its application the prediction of drug transport properties. Journal of medicinal chemistry, 43(20):3714-377 2000. 9,
10

---

Under review as a conference paper at ICLR 2023
[13] Peter Ertl, Eva Altmann, and Jeffrey M McKenna. The most common functional groups in bioactive molecules and how their popularity has evolved over time. Journal of medicinal chemistry, 63(15); 8408-841 8418, 2020.
[14] Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In /CMI,, 2017. 1, 14
[15] Laurent Gomez. Decision making in medicinal chemistry: The power of our intuition. ACS Medicinal Chemistry Letters, 9(10)9556 6588, 2018. 1
[16] Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In AISTATS, 2010. 3,5
[17] Erik Härkönen, Aaron Hertzman, Jaakko Lehtinen, and Sylvain Paris. Ganspace: Discovering interpretable gan controls. In NeurlPS, 2020. 3, 14
[18] Kaveh Hassani and Amir Hosein Khasahmadi. Contrastive multi-view representation learning on graphs. In ICML, 2020.
[19] Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, et al. beta-vae: Learning basic visual concepts with constrained variational framework. In ICLR, 2017. 1, 4, 15
[20] Geoffrey E Hinton. Training products of experts by minimizing contrastive divergence. Neural computation, 14(8):1771-188) 2002.
[21] Jonathan Ho. Ajay Jain, and Pieter Abbeel. Demoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840- 1556840-6811, 2020. 9
[22] Hu and Bajorath Jurgen. Extending the activity cliff concept: structural categorization of activity cliffs and systematic identification of different types of cliffss in the chembl database. Journal of chemical information and modeling, (807))8806-1888 2012.
[23] Yc Hu, Dagmar Stumpfc, and Jurgen Bajorath. Recent advances in scaffold hopping: miniperspective. Journal of medicinal chemistry, 60(4):1238 1246-1246, 2017. 1
[24] Aapo Hyvärinen and Peter Dayan. Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research, 6(4), 2005. 3
[25] John Irwin and Brian K Shoichet. Zinc: free database of commercially available compounds for virtual screening, Journal of chemical information and modeling, 45(1):177-182, 2005.
[26] Ali Jahanian, Lucy Chai, and Phillip Isola. On the" steerability" of generative adversarial networks. In ICLR, 2019. 2, 14
[27] Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Hierarchical generation of molecular graphs using structural motifs. In ICML, 2020. 2, 7, 14
[28] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In CVPR, 2019. 3. 14
[29] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, et al. Analyzing and improving the image quality of stylegan. CVPR, 2020.
[30] Hyunjik Kim and Andriy Mnih. Disentangling by factorising, In ICML, 2018. 4, 15
[31] Diederik Kingma and Max Welling. Auto-encoding variational bayes. arXiv: 20133 3
[32] Abhishek Kumar, Prasanna Sattigeri, and Avinash Balakrishnan. Variational inference of disentangled
latent concepts from unlabeled observations. In /CLR, 2018. 4, 15
[33] Greg Landrum et al. RDKit: A software suite for cheminformatics, computational chemistry, and predictive modeling, 2013,
[34] Shengehao Liu, Mehmet Furkan Demirel, and Yingyu Liang. N N-gram graph: Simple unsupervised representation for graphs, with applications to molccules. arXiv: 092006 2018. 14
[35] Shengchao Liu, Hanchen Wang, Weiyang Liu, Joan Lasenby, Hongyu Guo, al. Pre-training molecular graph representation with 3d geometry. In ICLR, 2022, 4, 6, 16, 17
11

---

Under review as a conference paper at ICLR 2023
[36] Francesco Locatello, Stefan Bauer, Mario Lucic, Gumnar Raetsch, Sylvain Gelly, et al. Challenging common assumptions the unsupervised learning disentangled representations. In ICML, 2019, 1,
15
[37] David Mendez, Anna Gaulton, A Patrícia Bento, Jon Chambers, Marleen De Veij, et al. Chembl: towards direct deposition of bioassay data. Nucleic acids research, 47(D1):D980--9900, 2019.
[38] Zlatko Mihalié and Nenad Trinajstic. A graph-theoretical approach to structure-property relationships. Journal of Chemical Education, 6919):00,, 1992.
[39] Andriy Mnih and Yee Whye Teh. A fast and simple algorithm for training neural probabilistic language models. arXiv preprint 12Xiv 2012. 17
[40] Weili Nie, Arash Vahdat, and Anima Anandkumar. Controllable and compositional generation with latent-space energy-based models. In NeurIPS, 2021. 2,
[41] Marcus Olivecrona, Thomas Blaschke, Ola Engkvist, and Hongming Chen. Molecular de-novo design through deep reinforcement learning, Journal of cheminformaties, 9(1):1 14, 2017.
[42] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, arXiv:1807.0555 00774777....... 2018.
[43] Yael Pritch, Eitam Kav-Venaki, and Shmuel Peleg. Shift-may map image editing. In ICCV, 2009. 1
[44] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d
classification and segmentation. In CVPR, 2017. 14
[45] Xuanchi Ren, Tao Yang, Yuwang Wang, and Wenjun Zeng. Do generative models know disentanglement? contrastive learning is all you need, arXiv:2102. 10543, 2021. 2, 8, 8,9,1 14
[46] Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In /CML, 2015. 3
[47] Karl Ridgeway and Michael C Mozer. Learning deep disentangled embeddings with the f-statistic loss. In NewrIPS, 2018. 1, 4, 15
[48] Radu Bogdan Rusu and Steve Cousins. 3D here: Point Cloud Library (PCL... In ICRA, 2011, 4, 15
[49] Radu Bogdan Rusu, Gary Bradski, Romain Thibaux, and John Hsu. Fast 3d recognition and pose using the viewpoint feature histogram. IROS, 2010. 15
[50] Paul G. Seybold, Michael May. and Ujjvala A. Bagal. Molecular structure: Property relationships. Journal of Chemical Education, 64(7):575, 1987. 4
[51] Yujun Shen and Bolei Zhou. Closed-fome form factorization of latent semantics in gans. In CVPR, 2021. 2,
8, 9,
[52] Yujun Shen, Ceyuan Yang, Xiaoou Tang, and Bolci Zhou. Interfacegan: Interpreting the disentangled face representation learned by gans. IEEE TPAMI, 2020. 2, 3, 7,
[53] Weijing Shi and Raj Rajkumar. Point-gnn: Graph neural network for 3d object detection in point cloud. In CVPR, 2020. 14
[54] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. 10X.0:9977005000 2019. 3
[55] Yang Song and Diederik P Kingma. How to train your energy-based models. arXiv:2101.03288. 2021. 5,
17
[56] Yang Song, Jascha bohl-Dicksteei. Diederik P Kingma, Abhishek Kumar. Stefano Ermon, et al. Score- based generative modeling through stochastic differential equations. arXiv:2011.13:: 1.1456.. 2020. 6,6
[57] Daniel F Veber, Stephen R Johnson, Hung- Yuan Cheng, Brian R Smith, Keith W Ward, and Kenneth D Kopple. Molecular properties that influence the oral bioavailability of drug candidates. Journal of medicinal chemistry, 45(12):2615 2623, 2002. 2, 9, 21
[58] Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E Sarma, Michael M Bronstein, et al. Dynamic graph enn for lcarning on point clouds. ACM ToG, 2020.
[59] Zichao Wang, Weili Nie, Zhuoran Qiao, Chaowei Xiao, Richard Baraniuk, and Anima Anandkumar. Retrieval-hased controllable molecule generation. arXiv preprint arXiv:2208. // /26,, 2022. 14
12

---

Under review as a conference paper at ICLR 2023
[60] Zongze Wu, Dani Lischinski, and Eli Shechtman. Stylespace analysis: Disentangled controls for stylegan image generation. In CVPR, 2021.
[61] Weihao Xia, Yulun Zhang. Yujiu Yang, Jing- Hao Xue, Bolei Zhou, al. Gan inversion: A survey. arXiv:2101.05275 2021.
[62] Guandao Yang. Xun Huang, Zekun Hao, Ming- Yu Liu, Serge Belongie, al. Pointflow: 3d point cloud generation with continuous normalizing flows. In ICCV, 2019, 14
[63] Kevin Yang, Kyle Swanson, Wengong Jin, Connor Coley, Philipp Eiden, al. Analyzing learned molecular
representations for property prediction. Journal of chemical information and modeling, 59(8):3370-3388 2019. 14
[64] Jiaxuan You,
Liu, Zhitao Ying. Vijay Pande, and Jure Leskovec. Graph convolutional policy network for goal-directed molecular graph generation. Advances in neural information processing systems, 31, 2018. 2, 14
[65] Chengxi Zang and Fei Wang. Moflow; an invertible flow model for generating molecular graphs. In KDD, 2020. 7, 14
13


Reviews:
[{"review_id": "aUvZIcuoH0", "reviewer": "Reviewer_CQ2j", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper tries to develop a method called GraphCG for the unsupervised discovery of steerable factors in the latent space of deep graph generative models. The authors observe that the learned representation space of some methods is not perfectly disentangled.  Thus, they propose to learn the semantic-rich directions by maximizing the corresponding mutual information for tackling this problem. The experiments on molecular graphs and point clouds show the effectiveness of the proposed GraphCG.", "summary_of_the_review": "In summary, this paper focuses on an important problem and proposes an effective method to tackle this problem. But I still have some concerns above. Overall, I recommend weak reject for this paper.\n", "strength_and_weaknesses": "The strengths of this paper are as follows:\n1. This paper focuses on a fundamental and important research problem which is discovering steerable factors in graphs.\n2. This paper is clearly written and the proposed method is easy to understand to me. Figure 1 clearly presents the training phase of the method, which is also clearly listed in Algorithm 1. \n3. The experimental results especially for Figures 3 and 4 well show the effectiveness of the proposed method.\n\nThe weaknesses of this paper are as follows:\n1. The authors seem to ignore some literature on disentangled graph learning methods. The main difference between disentangled latent factors in disentanglement literature and the steerable factors in the paper is not well discussed. I am not sure whether the disentangled graph learning method is good enough for solving the problem, which raises my concerns about the technical novelty of this paper.\n2. The time complexity as well as the empirical efficiency of the proposed method is not well discussed, since relatively complex techniques are proposed in the model.\n3. Another concern is about the experiment. I think the experiments should consider more baselines (e.g., disentangled graph learning methods and SSL methods as discussed in section 4.4) for more convincing results.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "XfAnAw9xjvS", "reviewer": "Reviewer_wzWp", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": " The paper proposes a method denoted as GraphCG to perform unsupervised discovery of steerable factors in latent space of deep graph generative models. The paper is to answer questions:  (1) is the latent representation space learned from DGMs on graph data disentangled\n(2) Given a pretrained DGM with not perfectly disentangled latent space, is there a flexible framework enabling the graph controllable generation in an unsupervised manner?\n\n", "summary_of_the_review": "The paper asked two questions in the introduction; however, I do not feel fully persuaded by the solution proposed in the paper. The evaluation of the paper can be more improved. ", "strength_and_weaknesses": "The paper is solving an interesting problem.\n\nWeakness: \n1. Idea: \nThe paper combines ideas like graph encoder/decoder, mutual information, contrastive estimation, EBM. I do not see how these ideas help with better understanding of the latent space \n\nCan the authors give some intuition about \"It starts with the assumption that the latent representations edited\nwith the same semantic direction and step size should possess similar information (corresponding\nto the factors) to certain degree, thus by maximizing the mutual information them, we can learn the\nmost semantic-rich directions.\" and \"we set the editing condition as containing both the semantic directions and step\nsizes, and we assume that maximizing the MI between different conditions can maximize the shared\ninformation within each condition. \" -- why doing so we can learn meaningful latent representation? Is figure 4 can show some intuition?\n\n2: Experiments: \nCan the paper include comparison with other disentanglement methods? e.g. in Table 1\n\n3 evaluation: \nHow do we sure the learned latent representation is really meaningful? The authors give Tanimoto measure for molecular editing. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "InUbIbK-BWA", "reviewer": "Reviewer_VdZt", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose GraphCG, a method for finding steerable factors in the latent space of already-learned deep generative model (DGM) for graphs. The method learns an energy-based model (EBM) in the latent space of a DGM with the goal of finding direction vectors $d_i$ that correspond to meaningful semantic modification of the latent vectors. The authors evaluate their method on molecular graphs and point clouds. \n", "summary_of_the_review": "The current manuscript's weaknesses outweigh the merits in my view. The most important drawbacks to me are:\n* The vague and (at least to me) not sufficiently clear model description.\n* Labeling the method as graph-specific while there is (as far as I can see) nothing graph-specific about it.  \n* Insufficient ablation/ hyperparameter studies are provided. The only ablation study is visual in nature and thus not a quantitative comparison of the variants.", "strength_and_weaknesses": "Strengths:\n* Steerable generation is a key challenge in generative modeling.\n* The paper is generally well-written and the presentation is mostly clear.\n* The authors present some interesting experimental results.\n\nWeaknesses:\n* There seems to be nothing graph-specific about the proposed method itself.\n* The authors only evaluate their method on molecular graphs and point clouds, where the latter are not graphs in the narrow sense.\n* Some aspects of the model description are confusing to me (see next section for details).\n* No results about ablation or hyperparameter studies are provided, e.g., comparing the linear and non-linear editing functions. It is not clear which version the authors used in each of their experiments. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "4ZrxZRpg1u", "reviewer": "Reviewer_F7x5", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper aims at finding the steerable factors of a graph. It probes the latent space, trains an edit function and a distance function.\nThe objective function of learning the edit function and the distance function consists of three quantities--- the noise contrastive estimation based objective, sparsity prone regularizer and the diversity encouraging similarity regularizer.\nExperiments have been conducted to support the efficacy of their model.", "summary_of_the_review": "My main concern is clarity (this can impact quality and technical soundness too) of the paper, as I mentioned above. Also, a not-so-minor point is that in ICLR, citations are given in author-year format rather than the citation-number format, whereas the authors follow latter. Note that, changing the current format to citation-number format would change the content significantly.", "strength_and_weaknesses": "> Strengths\n\n1. The paper tackles an important problem for controllable graph generation\n2. Apparently simple approach seems to work well\n\n> Weakness\n\n1. Lack of clarity in many places\n2. Lack of contrast from prior work ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject