Unsupervised Discovery of Steerable Factors When Graph Deep Generative Models Are Entangled
Authors: Shengchao Liu
Chengpeng Wang
Jiarui Lu
Weili Nie
Hanchen Wang
Zhuoxinran Li
Bolei Zhou
Jian Tang
Venue: Accepted by TMLR ()

Abstract: Deep generative models (DGMs) have been widely developed for graph data. However, much less investigation has been carried out on understanding the latent space of such pretrained graph DGMs. These understandings possess the potential to provide constructive guidelines for crucial tasks, such as graph controllable generation. Thus in this work, we are interested in studying this problem and propose GraphCG, a method for the unsupervised discovery of steerable factors in the latent space of pretrained graph DGMs. We first examine the representation space of three pretrained graph DGMs with six disentanglement metrics, and we observe that the pretrained representation space is entangled. Motivated by this observation, GraphCG learns the steerable factors via maximizing the mutual information between semantic-rich directions, where the controlled graph moving along the same direction will share the same steerable factors. We quantitatively verify that GraphCG outperforms four competitive baselines on two graph DGMs pretrained on two molecule datasets. Additionally, we qualitatively illustrate seven steerable factors learned by GraphCG on five pretrained DGMs over five graph datasets, including two for molecules and three for point clouds.

Published in Transactions on Machine Learning Research (01/2024)
Unsupervised Discovery of Steerable Factors When Graph Deep Generative Models Are Entangled
Shengchao Liu
shengchao. htw.:hhao..l..@mmmttrrea...ooo
Quebec AI Institute (Mila) University de Montréal
Chengpeng Wang
cw83deilinoos. edu
Universily of Ilimois Urbana- Champaign
Jiarui Lu
jiarui. bu @umontreal.cc
Quebec Al Institute (Mila) University de Montréal
Weili Nic Nvidia Research
wric @nwidda com
Hanchen Wang University of Cambridge
hw501 @com. ac. uk
Zhuoxinran Li
University of Toronto
Bolei Zhou University of California, Los Angeles
bolei@cs. ucla edu
Jian Tang
fian. Sonnlanggnn wwwggaann..h.c.ccoo on
Quebec Al Institute (Mila)
HEC Montréal
Reviewed on OpenReview: https: // openreview. net/ forum? id= wy0uya00g
Abstract
Deep generative models (DGMs) have been widely developed for graph data. However, much less investigation has been carried out on understanding the latent space such pretrained graph DGMs. These understandings possess the potential to provide constructive guidelines for crucial tasks, such as graph controllable generation. Thus in this work, we are interested in studying this problem and propose GraphCG, method for the unsupervised discovery of steerable factors in the latent space of pretrained graph DGMs. We first examine the representation space of three pretrained graph DGMs with six disentanglement metrics, and we obscrve that the pretrained representation space is entangled. Motivated by this observa- tion, GraphCG learns the steerable factors via maximizing the mutual information between semantic-rich directions, where the controlled graph moving along the same direction will share the same steerable factors. We quantitatively verify that GraphCG outperforms four competitive basclines two graph DGMs pretrained on two molecule datasets. Addition- ally, we qualitatively illustrate seven steerable factors learned by GraphCG on five pretrained DGMs over five graph datascts, including two for molecules and three for point clouds.

---

Published in Transactions on Machine Learning Research (01/2024)
1 Introduction
The graph is general format for many real-world data. For instance, molecules can be treated as graphs (uu- venaud et al., 2015 Gilmer et al. 2017) where the chemical atoms and bonds correspond to the topological nodes and edges respectively. Processing point clouds as graphs is also a popular strategy (Shi & Rajkumar 2020: Wang et al. 2020), where points are viewed as nodes and edges are built among the nearest neighbors. Many existing works on deep generative models (DGMs) focus on modeling the graph data and improving the synthesis quality, However, understanding the pretrained graph DGMs and their learned representa- tions has been much less explored. This may hinder the development of important applications like graph controllable generalion (also referred to as graph ediling) and the discovery of interpretable graph structure.
Concretely, the graph controllable generation task refers to modifying the steerable factors of the graph so as to obtain graphs with desired properties easily (Drews] 2000: Pritch et al., 2009). This is an important task in many applications, but traditional methods (e.g., manual editing) possess inherent limitations under particular circumstances. A typical example is molocule editing: aims at modifying the substructures of molecules (Mihalić& & Trinajstic 1992) and is related to certain key tactics in drug discovery like functional group change (Ertl et al.) 20200] and scaffold hopping (Böhm et al., 2004 Hu et al. 2017). This is a routine task in pharmaceutical companies, yet, relying on domain experts for manual editing can be subjective or
biased (Drews 2000 Gomez 2018). Different from previous works, this paper starts to explore unsupervised
graph editing on pretrained DGMs. It can act as a complementary module to conventional methods and
(2) It alleviates the requirements for extensive domain knowledge for factor change labeling. (3) It provides a constructive perspective for editing preference, which can reduce biases from the domain experts.
bring many crucial benefits: (1) It enables efficient graph editing in a large-scale setting.
Disentanglement for editing. One core property relevant to the general unsupervised data editing using DGMs is disentanglement. While there does not exist a widely-accepted definition of disentanglement, the key intuition Locatello el al., 2019) is that a disentangled representation should separate the distinct, informative, and stcerable factors of variations in the data. Thus, the controllable generation task would become trivial with the disentangled DGMs as the backbone. Such a disentanglement assumption has been widely used in generative modeling on the image data, e.g., 3-VAE (Higgins et al. 2017) learns disentangled representation by forcing the representation to be close to isotropic unit Gaussian. However, it may
introduce extra constraints on the formulations and expressiveness of DGMs HHiginn et al. 2017 Ridgeway
& Mozer 2018
Eastwood & Williams 2018: Wu el al. 2021).
Entanglement on pretrained graph DGMs. Thus for graph data, one crucial question arises: Is the latent representation space from pretrained graph DGMs disentangled or not? In image gencration, a scries of work (Collins et al., 2020 Shen et al., 20200a Härkönen et al., 2020 Tewari et al.] 2020 Wu et al.] (202) has shown the disentanglement properties on pretrained DGMs. However, such property of pretrained graph DGMs is much less explored. In Scction 3 we first study the latent space of three pretrained graph DGMs and empirically illustrate that the learned space is not perfectly disentangled entangled. In what follows, we adopt the term "entangled" for graph DGMs.
Our approach. This observation then raises the second question: Given a pretrained yet entangled DGM for graph data, is there a fiexible framework enabling the graph controllable generation in an unsupercised manner? To tackle this problem, we propose a model-agnostic framework coined GraphCG for unsupervised graph controllable generation. GraphCG has two main phases, as illustrated in Figure [ During the learning phase (Figure [aa]]. GraphCG starts with the assumption that the steerable factors can be learned by maximizing the mutual information (MI) among the semantic directions. We formulate GraphCG using energy-based model (EBM), which offers a large family of solutions. Then during the inference phase, with the learned semantic directions, we can carry out the editing lask by moving along the direction with certain step sizes. As the example illustrated in Figure I(b] the graph structure (hydroxyl group) changes consistently along the learned oditing direction. For evaluation, we qualitatively verify the learned directions of five pretrained graph DGMs. Particularly for the molecular datasets, we propose a novel evaluation metric called sequence monotonic ratio (SMR) to quantitatively measure the structure change over the put sequences.
2

---

Published in Transactions on Machine Learning Research (01/2024)
Edited Molecule
a ==-3
100
Direction
Anchor Molecule
Laa
GraphCG
Decode
a a=0
Contrast
Edited Molecule
Align
Edited Mololecule
a a=3
Anchor Molecule
(a) Learning phase of GraphCG.
(b) Inference phase of GraphCG.
Figure 1: (a) The learning phase. Given two latent codes z" and z", we edit the four latent representations along i-th and j-th direction with step size and respectively. The goal of GraphCG, is to align the positive pair (2%, and ), and contrast them with 5j.9 and Ki,3 respectively. (b) The inference phase. We will first sample an anchor molecule and adopt the learned directions the learning phase for editing. With step size [-3, 3), we can generate a sequence of molecules. Specifically, after decoding, there is a functional group change shown up: the number of hydroxyl groups decreases along the sequence the decoded molecules.
Our contributions. (1) We conduct an empirical study on the disentanglement property of three pretrained graph DGMs using six metrics, and we observe that the latent space of these pretrained graph DGMs is entangled.
(2) We propose a model-agnostic method called GraphCG for the unsupervised graph controllable generation or graph editing. GraphCG aims at learning the steerable factors by maximizing the mutual information among corresponding directions, and its outputs are sequences of edited graphs. (3) We quantitatively verify that GraphCG outperforms four competitive baselines when evaluated on two pretrained graph DGMs over two molecule datasets. (4) We further qualitatively strengthen the effectiveness of GraphCG by illustrating seven semantic factors on five pretrained graph DGMs over five graph datasets, including two for molecular graphs and three for point clouds.
Related work. Recent works leverage the DGMs for various controllable generation tasks (Chen el al. 2018a; Xia al. 2021), where the inherent assumption that the learned latent representations encode rich semantics, and thus traversal in the latent space can help steer factors of data (Jahanian al.) 2019
Shen et al. 2020b Härkönen et al. 2020). Among them, one research direction (Shen et al.) 2020b Nie
et al. 2021 is using supervised signals to learn the semantic-rich dirrectinss, and most works on editing the
graph data focus on the supervised setting (Jin et al. 2020a. Veber et al.) 2002 You et al., 2018). However,
these approaches can not be applicd to many realistic scenarios where extracting the supervised labels is
difficult. Another research line (Härkönen et al.) 2020 Shen & Zhou 2021 Ren et al. 2021) considers
discovering the latent semantics an unsupervised manner, but these unsupervised methods are designed to be either model-specific or task-specific, making them not directly generalizable to the graph data. A more comprehensive discussion is in Appendix
2 Background and Problem Formulation
Graph and deep generative models (DGMs). Each graph data (including nodes and edges) is denoted as x E X, where X' is the data space, and DGMs learn the data distribution, i.e., p(x).. Our proposed graph editing method (GraphCG) is model-agnostic or DGM-agnostie, so we briefly introduce the mainstream DGMs for graph data as below. Variational auto-encoder (VAE) (Kingma & Welling 2013 Higgins et al., 2017) measures a variational lower bound of p(z) by introducing a proposal distribution; flowbbaeed model (Dinh et al.) 2014 Rezende & Mohamed 2015) constructs revertible encoding functions such that
the data distribution can be deterministically mapped to prior distribution. Note that these mainstream
DGMs, either explicitly or implicitly, contain encoder (f(.)) and decoder (g(-)) parameterized by neural networks:
= f(x),
x'=((x),
(1)
where z E Z is the latent representation, Z the latent space, and x' is the reconstructed output graph. Since in the literature (Shen& & Zhou 2021 Shen al. 2020b), people also call latent representations latent
3

---

Published in Transactions on Machine Learning Research (01/2024)
codes or latent vectors, in what follows, we will use these terms interchangeably. Note that the encoding and decoding functions in Equation ① (f.g) can be stochastic depending on the DGMs we are using.
Steerable factors. The steerable factors are key attributes of DGMs, referring to the semantic information of data that we can explicitly discover from the pretrained DGMs. For instance, existing works (Shen & Zhou 2021 Ren al. (021) have shown that using unsupervised methods on facial image DGM can discover factors such as the size of eyes, smiles, noses, etc. In this work, we focus on the steerable factors of graph data, which are data- and task-specific. Yet. there is one category of factor that is commonly shared among all the graph data: the structure factor. Concretely, these steerable factors can be the functional groups in molecular graphs and shapes or sizes in point clouds. The details of these stecrable factors are in Appendix C
Semantic direction and step size. To learn the steerable factors using deep learning lools, we will introduce the semantic directions defined on the latent space of DGM. In such a space Z, we assume there exist D semantically meaningful dircction voctors, d, with i E {0, 1, ...... .,....., - 1 There is also a scalar variable, step size a, which controls the degree to edit. the sampled data with desired steerable factors (as will be introduced below), and we follow the prior work (Shen & Zhou 2021) on taking a E [-3,3]. 3]. Each direction corresponds to one or multiple factors, such that by editing the latent vector along d; with step size a, the reconstructed graph will possess the desired structural modifications. The editing with a sequence of step sizes a E [-3,3] along the same direction d, leads to a sequence of edited graphs.
Problem formulation: graph editing or graph controllable generation. Given a pretrained DGM (ú..,, the encoder and decoder are fixod), our goal is to learn the most semantically rich directions (d.) in the latent space Z. Then for each latent code 2, with the i-th semantic direction and step size a, we can get an edited latent vector ž and edited data x' after decoding Zi,a, as:
z f(x),
2.a = l1zz, a),
n' ='=9(ā,,a)
(2)
where d, and h(-) are the edit direction and edit functions that we want to learn. We expect that Zí, can inherently possess certain steerable factors, which can be reflected in the graph structure of ø'.
Energy-based model (EBM). EBM is a flexible framework for distribution modeling:
p(x)= exp(-E(x))
exp(-E(x)) =xx)==E
A
where B(-) is the energy function and A is the partition function. In EBM, the bottleneck is the estimation of partition function A. It is often intractable due to the high cardinality of X. Various methods have been proposed to handle this issue, including but not limited to contrastive divergence (Hinton) 2002), noise-
contrastive estimation (Gutmann & Hyvärinen 2010: Che et al. 2020 and score matching (Hyvärinen &
Dayan 2005 Song & Ermon 2019: Song et al., 2020
3 Entanglement of Latent Representation for Graph DGMs
In this section, we quantify the degree of disentanglement of the existing DGMs for graph data. Recall that the key intuition (Locatello et al.] (219g) behind disentanglement that a disentangled representation space should separate the distinct, informative, and steerable factors of variations in the data. In other words, each latent dimension of the disentangled representation corresponds to one multiple factors. Therefore, the change of the disentangled dimension can lead to the consistent change in the corresponding factors of the data. This good property has bocome a foundational assumption in many existing controllable generation methods (Shen et al. 2020b Shen & Zhou, 2021 Härkönen et al. 2020
In the computer vision domain, StyleGAN Karras et al. 2019 is one of the most recent works on image gencration, and several works have proven its nice disentanglemen property (Collins et al. 2020 Shen al. 2020at Härkönen et al. 2020 Tewari et al. 2020; Wu et al., 2021]. In image generation, (Locatello et al. 2019h shows that without inductive bias, the representation learned by VAEs is not perfectly disentangled. Then the next question naturally ariscs: Is the latent space of pretrained graph DGMs disentangled or not!
To answer this question, we conduct the following experiment.
In unsupervised editing, the steerable factors an each semantic direction known by tost-training human selection.
4

---

Published in Transactions on Machine Learning Research (01/2024)
Table 1:
The six disentanglement metrics on three pretrained DGMs and two graph types. All measures range from 0 to 1, and higher scores mean more disentangled representation.
Graph Type
DGM
Dataset
BetaVAE f
FactorVAE t
MIG f
DCI 1
Modularity t
SAP t
Molecular Graph
MoFlow HierVAR
ZINC250K
0.260
0.175
0.031
0.953
0.620
0.009
ChEMBBL
0.178
0.165
0.022
0.114
0.606
0.026
Point Cloud
PointFlow
Airplane
0.022
0.025
0.029
0.160
0.745
0.022
There have been a series of works exploring the disentanglement of the latent space in DGMs, and here we take six widely-used ones: BetaVAE (Higgins et al.) 2017). FactorVAE (Kim & Mnih 2018). MIG (Chen eet al. 2018b), DCI (Eastwood & Williams| 2018); Modularity (Ridgeway & Mozer 2018); and SAP (Kumar let al., 2018). Each measure has its own bias, and we put detailed comparison in Appendix C Meanwhile, they all share the same high-level idea: given the latent representation from pretrained DGM, they are proposed to measure how predictive it is certain steerable factors.
To adapt them to our setting, first, we need to extract the steerable factors in graph DGMs, which requires domain knowledge. For instance, in molecular graphs, we can extract some special substructures called fragments or functional groups. These substructures can be treated as steerable factors since they are the key components of the molccules and are closely related to certain molecular properties (Seybold et al. 1987). We use RDKit (Landrum et al. 2013) to extract the 11 most distinguishable fragments as steerable factors for disentanglement measurement. For point clouds, we use PCL tool (Rusu & Cousins 2011 to extract 75 VFH descriptors Rusu et al. 2010 as steerable factors, which depicts the geometries and viewpoints accordingly.
Then to measure the disentanglement on graph DGMs, we consider six metrics on three datasets and two data types with three backbone models. All the metric values range from 0 to 1, and the higher the value, the more disentangled the DGM is. According to Table we can observe that most of the disentanglement scores are quite low, except the DCI (Eastwood Williams 2018) on MoFlow. Thus, we draw the conclusion that, generally, these graph DGMs are entangled. More details of this experiment (the steerable factors on two data types and six disentanglement metrics) can be found in Appendix C
4 Our Method
The analysis in Section naturally raises the next rescarch question: Given an entangled DGM, is there flexible way to do the gruph data editing in an unsupervised manner? The answer is positive. We propose GraphCG, a flexible model-agnostic framework to learn the semantic directions in an unsupervised manner. It starts with the assumption that the latent representations edited with the same semantic direction and step size should possess similar information (with respect to the factors) to a certain degree, thus by maximizing the mutual information among them, we can learn the most semantie-rich directions. Then we formulate this editing task as density estimation problem with the energy-based model (EBM). As introduced in Section [ EBM covers broad range of solutions, and we further propose GraphCG-NCE by adopting the noise- contrastive estimation (NCE) solution.
4.1 GraphCG with Mutual Information
Motivation: learning semantic directions using MI on entangled DGM. Recall that our ultimate goal is to enable graph editing based on semantic vectors. Existing deep generative models are entangled, thus obtaining such semantic vectors is a nontrivial task. To handle this problem, we propose using mutual information (MT) to learn the semantics. MI measures the non-linear dependency between variables. Here we set the editing condition as containing both the semantic directions and step sizes. We assume that maximizing the MT between different conditions can maximize the shared information within each condition, i.e., graphs moving along the same condition share more semantic information. The pipeline is as follows.
We first sample two latent codes in the latent space, x" and z". Such two latent codes will be treated as positive pairs, and their construction will be introduced in Section 4.3] Then we pick up the i-th semantic direction and one step size a to obtain the edited latent codes in the latent space Z as:
Z h(=", ,d,
E h(z", a).
(4)
5

---

Published in Transactions on Machine Learning Research (01/2024)
Under assumption, expect that these two edited latent codes share certain information with respect to the steerable factors. Thus, want to maximize the MI between 2, and " Since the MT is intractable to compute, we adopt the EBM lower bound from Liu et al. 2022)
as:
DaalFr,ar 21,a) E logg(e"), + log lo
(5)
The detailed derivation is in Appendix Till this step, we have transformed the graph data editing task into the estimation of two conditional log- likeliiiisoods.
4.2 GraphCG with Energy-Basee Based Model
Following Equation 5 maximizing the MI between 1 [ xi,a) is equivalent to estimating the summation of two conditional log- licehiocls We then model them using two conditional EBMs. Because these two views are in the mirroring direction, we may as well take one for illustration. For example, for the first conditional log- likelihood, we can model with EBM as:
exp( ( 2),)) p( (1,a)) Teep(- B( "" dey.
(6)
Ai
where E5.) is the energy function, Agj is the intractable partition function, and f(-) is the negative energy. The energy function is flexible and we use the dot- dot-product:
S ) = (h(=", d, a), M(=", ,d, a)),
(7)
where h(-) is the editing function introduced in Equation ( Similarly for the other conditional log- log-likelihood term, and the objective becomes:
CoappCo = B exp(f(E), R),o))
exp( 2) ))
log
(8)
As
Age
With Equation 88] we are able to learn the semantically meaningful direction vectors. We name this unsupervised graph controllable generation framework as GraphCG. In specifie, GraphCG utilizes EBM for estimation, which yields wide family of solutions, as introduced below.
4.3 GraphCG with Noise Contrastive Estimation
We solve Equation (8) using the noise contrastive estimation (NCE) (Gutmann & Hyvärinen 2010 The high-level idea of NCE is to transform the density estimation problem into binary classification problem that distinguishes if the data comes from the introduced noise distribution or from the true distribution. NCE has been widely explored for solving EBM (Song & Kingma 2021), and adopt it as GraphCG-NCE by optimizing the following objective function:
LGrghCCG-NCE = =- (Bpp. 18 [oog (1 o((()), xi,,)) + Pann 1
(9)
+ E log (1 of ))))) + (y [log logg ".,))
where Pdata is the empirical data distribution and Pa is the noise distribution (derivations are in Appendix D Recall that the latent code pairs (x",,") are given in advance, and the noise distribution is on the semantic directions and step sizes. In specific, the step sizes (a * B) are randomly sampled from [-3, 3), and the latent direction indices (i * j) are randomly sampled from {0, 1, ... D-1}. Equation (9) is for one latent
code pair, and we take the expectation of it all the pairs sampled from the dataset. Besides, would like to consider extra similarity and sparsity constraints
Lsim = Ei, (sim(d, dj)], Leparsity = B. d:
(10)
where sim(-) is the similarity function between two latent directions, and we use the dot product. By minimizing these two regularization terms, we can make the learned semantic directions more diverse and sparse. Putting them together, the final objective function is:
c1 By, CGraphoG NCE] c2 Lotm Laparsity;
(11)

---

Published in Transactions on Machine Learning Research (01/2024)
where C1 02, coefficients, and we treat them three hyperparameters (check Appendix E pipeline is illustrated in Figure 1 and for the next we will discuss certain key modules.
The above
Latent code pairs, positive and negative views. We consider two options for obtaining the latent pairs. (1) Perturbation (GraphCG-P) is that for each data point Φ, we obtain its latent code x = f(x). Then we apply two perturbations (e.g., adding Gaussian noise) on z to get two perturbed latent codes as z" and z", respectively. (2) Random sampling (GraphCG-R) is that we encode two randomly sampled data points from the empirical data distribution as 2" and z" respectively. Perturbation is one of the widely-used strategies (Karras et al. 2019 for data augmentation, and random sampling has been widely used in the NCE (Song & Kingma, 2021 literature. Then we can define the positive and negative pairs in GraphCG- NCE, where the goal is to align the positives and contrast the negatives. As described in Equation (9, the positive pairs are latent pairs moving with the same semantic direction and step size, while the negative pairs are the edited latent codes with different semantic directions and/or step sizes.
Semantic direction modeling. We first randomly draw a basis veclor Gi, and then model the semantic direction d, as d; MLP(e;), where MLP(.) is the multi-layer perceptron network.
Design of editing function. Given the semantic direction and two views, the next task is to design the editing function h(-) in Equation
2 Since our proposed GraphCG is flexible, and the editing function determines the energy function Equation we consider both the linear and non-linear editing functions as:
di,
===+ar d. MLP(= d, @ [a]).
(12)
where @ is the concatenation of two vectors. Noticing that for the non-linear case, we are adding an extra term by mapping from the latent code, semantic direction, and step-size simultancously. We expect that this could bring in more modeling expressiveness in the editing function. For more details, e.g., the ablation study to check the effect on the design of the views and editing functions, please refer to Appendices F and G while more potential explorations are left for future work.
4.4 Implementations Details
During training, the goal of GraphCG is to learn semantically meaningful direction vectors together with an editing function in the latent space, as in Algorithm || Then we need to manually annotate the semantic directions concerning the corresponding factors using certain post-training evaluation metrics. Finally, for the inference phase, provided with the pretrained graph DGM and a selected semantic direction (together with a step size) learned by GraphCG, we sample graph -> conduction editing in the latent space -> decoding to generate the edited graph, as described in Equation ( The detailed algorithm is illustrated in Algorithm 2 Next, we highlight several key concepts in GraphCG and briefly discuss the differences from other related concepts.
NCE and contrastive representa- tion learning. GraphCG-NCE is ap- plying EBM-NCE, which is essentially contrastive learning method, and an- other dominant contrastive loss is the In- foNCE (Oord et al. 2018). We sum- marize their relations below. (1) Both contrastive methods are doing the same thing: align the positive pairs and con-
Algorithm 1 Learning Phase GraphCG
1: Input: Given a pretrained generalive model encoder, f(-).
2: Output: Learued direction vector d, and function h(-).
3: Select latent codes z" ==""" E Z from empirical dataset and f(-)-
4: for each step size a and each direction do
5: Set zy. = 11(=", a)
6: Set æy,a h(22" dd,"),,,,))
7: Assign positive to pair (Z".. Z.,a)
8:
for step size a a and direction j # do Set = h(z". 8)
trast the negative pairs. (2) EBM-
9:
NCE Hassani & Khasahmadi 2020 Liu
10:
Set 2).o = h(2",
ct al. 2022) has been found to outper-
11:
Assign negative to pair (#) a).) Assign negative to pair (2), 4..))
form InfoNCE on certain graph appli-
12:
cations like representation learning. (3)
13:
end for
14: Do SGD w.r.t. GraphCG in Equation
What we want to propose here is a flex-
15: end for
ible framework. Specifically, EBM pro-
vides a more general framework by designing the energy functions, and EBM-NCE is just one effective so-
7

---

Published in Transactions on Machine Learning Research (01/2024)
lution. Other promising directions include the denoising score matching denoising diffusion model (Song
et al. 2020). while InfoNCE lacks such nice extensibility attribute.
GraphCG and contrastive self- supervised learning (SSL). GraphCG shares certain similarities with the self- supervised learning (SSL) method, how- ever, there are some inherent differences, summarized below. (1) SSL aims at learning the data represcntation by op- erating data augmentation on the data space, such as node addition and edge deletion. GraphCG aims at learning the semantically meaningful directions by editing on the latent space (the represen-
Algorithm 2 Inference Phase of GraphCG
1: Input: Given pre-trained generative model, J(-) and g()),
learned direction vector d.
2: Output: A sequence of edited graphs.
3: Sample x with DGM or from graph dataset.
4: If the latter, get latent code 2 = f(x).
5: for step size a E [-3,3] do
6: Do graph edit in the latent space to get 2i,0 = hlz, a.).
7: Decode to the graph space with n' = g(ā,,))
8: end for
9: Outpul is thus sequence of edited graphs, (ø').
(2) Based the first point, SSL aims at using different data points as the negative samples. GraphCG, on the other hand, is using different directions and step-sizes negatives, Namely, SSL is learning data representation in the inter-data level, and GraphCG is learning the semantic directions in the inter-direction level.
tation function is pretrained and fixed).
Output sequence the discrete space. Recall that during inference time (Algorithm 2 GraphCG takes a DGM and the learned semantic direction to output a sequence of edited graphs. Compared to the vision domain, where certain models (Shen & Zhou, 2021 Shen al. 2020bb have proven their effectiveness in many tasks, the backbone models in the graph domain have limited discussions. This is challenging because the graph data is in a discrete and structured space, and the evaluation of such space is non-trivial. Meanwhile, GraphCG essentially provides another way to verify the quality of graph DGMs. GraphCG paves the way for this potential direction, and we would like to leave this for future exploration.
5 Experiments
In this section, we show both the qualitative and quantitative results of GraphCG, two types of graph data: molecular graphs and point clouds. Due to the page limit, We put the experiment and implementation details in Appendix
5.1 Graph Data: Molecular Graphs
Background of molecular graphs. molecule can be naturally treated as a graph, where the aloms and bonds are nodes and edges, respectively. The unsupervised graph editing tasks can thus be formulated editing the substructures of molecular graphs. In practice, people are interested in substructures that are critical components of molecules, which are called the 'raagments'. In recent years, graph representation
Original Tanimoto Similarity
len(set((s(a 27
Tanimoto Similarity
Calibrated Tanimoto Similarity
ØSMR ((=(#')) (..)))))))))))))))))))))))))))
monotonic, (1(()())**)
(13)
otherwise
1
Σ øsvn (Is(a"))",,.1)
Φswn
(14)
M
tap.K(7,7)
Σ (MSMR(TTT))
(15)
K
-1.8
kEtop-K directions
Stop Size
Figure 2: This illustrates the sequence monotonic ratio (SMR) calibrated Tanimoto similarity (CTS), Equa-
tions 13 and (14 are the SMR on each sequence and each direction respectively, where M the number of generated sequences for the i-th direction and (s(#'))," is the CTS of the m-th generated sequence with the i-th direction. Equation (15 the average top-K SMR on directions.
8

---

Published in Transactions on Machine Learning Research (01/2024)
learning has been extensively explored on the molecular graph (Duvenaud et al.
2015 Gilmer et al. 2017
Liu et al. 2018: Yang et al. 2019b Corso et al. 2020
Backbone DGMs. We consider two state-of-the-ar DGMs for molecular graph generation. MoFlow Zang & Wang 2020 is a flow-based generative model on molecules that adopts an invertible mapping between the input molecular graphs and a latent prior. HierVAF (Jin et al. (020aa) is a hierarchical VAE model that encodes and decodes molecule atoms and motifs in a hierarchical manner. Besides, the pretrained checkpoints are also provided, on ZINC250K (Iwwin& & Shoichet, 2005) and ChEMBL [Mendez et al. 2019 respectively.
Editing sequences and anchor molecule. As discussed in Section the output of the inference in GraphCG, is a sequence of edited molecules with the i-th semantic direction, {z').- We first randomly generale a molecule using the backbone DGMs (without the editing operation), and we name such molecule as the anchor molecule, Œ². Then we take 21 step sizes from -3 to 3, with interval 0.3, to obtain a sequence of 21 molecules following Equation ( Note that the edited molecule with stop size 0 under the lincar oditing function is the same as the anchor molecule, i.e., x".
Change of structure factors and evaluation metrics. We are interested in the change of the graph structure (the stecrable factors) along the output sequence edited with the i-th semantic direction. To evaluate the structure change, we apply the Tanimoto similarity between each output molecule and the anchor molecule. Besides, for the ease of evaluating the monotonicity, we apply a Tanimoto similarity transformation on the output molecules with positive step sizes by taking the deduction from 2. We call this calibrated Tanimoto similarity (CTS) sequence, marked as {s(=')}, An illustration is shown in Figure 2 Further, we propose a metric called Sequence Monotonic Ratio (SMR), ØSMR (7, which measures the monotonic ratio of M generated sequences edited with the i-th direction. It has two arguments: the diversity threshold y constr the minimum number of distinct molecules, and the tolerance threshold controls the non- non-monoto tolerance ratio along each sequence.
Evaluating the diversity of semantic directions. SMR can evaluate the monotonic ratio of output sequences generated by one direction. To better illustrate that GraphCGG is able to learn multiple directions with diverse semantic information, we also consider taking the average of top-K SMR to reveal that all the best K directions are scmantically meaningful, as in Equation (15).
Baselines For baselines, we consider four unsupervised editing methods. (1) The first is Random. It randomly samples a normalized random vector in the latent space as the semantic direction. (2) The second one is Variance. We analyze the variance on each dimension of the latent space, and select the highest one with one-hot encoding as the semantic direction. (3) The third one is SeFa (Shen & Zhou 2021 It first decomposes the latent space into lower dimensions using PCA, and then takes the most principal components (eigenvectors) as the semantic-rich direction vectors. (4) The last one is DisCo (Ren et al.] [021]. It maps
each latent code back to the data space, followed by an encoder for contrastive learning, it requires the backbone DGMs to be end-to-end and is infeasible for Hie.
Quantitative results. We take D = 10 to train GraphCG, and the optimal results on 100 sampled sequences are reported in Table _ We can observe that GraphCG, can show consistently better structure change with both top-1 and top-3 directions. This can empirically prove the effectiveness of our proposed GraphCG. More comprchensive results are in Appendix
Analysis steerable factors in molecules: functional group change. For visualization, we sample 8 molecular graph sequences along selected directions in Figure 3 and the backbone DGM is HierVAE pretrained on ChEMBL. The CTS holds a good monotonic trend, and cach direction shows certain unique changes in the molecular structures, i.e., the steerable factors in molecules. Some structural changes are reflected in molecular properties. We expand all the details below. Tn Figures 3(a) and 3(b) the number of halogen atoms and hydroxyl groups (in alcohols and phenols) in the molecules decrease from left to right, respectively. In Figure 3(c] the number of amides in the molecules increases along the path. Because amides are polar functional groups, the topological polar surface area (IPSA) of the molecules also increases, which is a key molccular property for the prediction of drug transport properties, e.g., permcability (Ertl et al., 2000). In Figure 3(d) the flexible chain length, marked by the number of ethylene (CH,CH2) units, increases
9

---

Published in Transactions on Machine Learning Research (01/2024)
Table 2: This table lists the sequence monotonic ratio (SMR, %) on calibrated Tanimoto similarity (CTS) for the top-1 and top-3 directions. The best performances are marked in bold.
Tanimoto top-1
Tanimoto top-3
Model
Dataset
diversity 7
4
3
tolerance
0.2
0.2
01.
0
0.2
Random
23.0
25.0
15.0
22.0
24.0
11.0
13.7
Variance
24.0
28.0
16.0
20.0
25.0
10.0
15.0
SeFa Shen & Zhou 2021
4.0
0.0
3.3
3.3
0.0
MoFlow
ZINC250K
DisCo Ron
2021
7.0
14.0
8.0
2.0
7.7
GraphCG-P
32.0
34.0
16.0
18.0
29.0
31.0
13.7
16.3
GraphCG-R R
25.0
26.0
11.0
14.0
22.0
24.3
10.3
13.3
Random
14.0
45.0
14.0
43.0
10.0
42.3
9.3
41.7
Variance
23.0
59.0
19.0
55.0
18.3
52.7
15.7
50.3
ScFa Shen Shen & Zhou 2021
4.0
41.0
41.0
2.3
36.0
2.3
36.0
VAI
ChEMBL
GraphCG-P
40.0
73.0
32.0
65.0
36.0
64.3
26.3
57.7
GraphOG-R
42.0
67.0
30.0
55.0
38.0
62.3
28.7
53.7
9369
CTS: 0.29
CTS. 0.71
CTS: 1.00
CTB, 1:78
CTS:
CTS. 0.59
CTS. 0.58
CTB: 1.00
CTS: 1.68
CTS: 8.83
CTS: 0.37
CTS 6.85
CTS: 1.00
CTS: 1.11
CTS:
cTS 0.49
CTS: 1.00
CT8: 1.00
CTS: 7788
CT8: 1
(a) Steerable factor: number of halogens.
(b) Steerable factor: number of hydroxyls.
CTS: 1010 IPSA:
C.1. IPSA
CTS: 1.00 IPSA:
CTS: 1.88 IPSA
CTB: 1.78
cTS.0.10 NRt 0
CTS. 0.30 NRB
CTB: 1.00 Ni3
CTS- 1.84
CTB: 1.85 NRB
(c) Steerable factor: number of amides.
(d) Steerable factor: chain length.
Figure 3: GraphCG for molecular graph editing. We visualize the output molecules and CTS in four directions with two sequences each, where each sequence cousists of five steps. The five steps correspond five step sizes: -3, -1.8, 0, 1.8, and 3, where marks the anchor molecule (center point of reach sequence). Figure (aa visualizes how the number of halogens (marked in green) decreses from -3 Figure 3bb visualizes how the number of hydroxyls (marked in red) decreases from -3 to 3. Figure 3(c] visualizes how the number of amides (marked in red and blue) increases from -3 Notably, certain properties change with molecular structures accordingly, like topological polar surface area (tPSA) and the number of rotatable bouds (NRB).
Figure 3(d) visualizes
the number of chains (marked green) increases from
from left to right. Since the number of rotatable bonds (NRB) measures the molecular flexibility, it also increases accordingly (Veber et al. 2002
5.2 Graph Data: Point Clouds
Background of point clouds. A point cloud is represented as a set of points, where each point Pt is described by a vector 3D Euclidean coordinates possibly with extra channels (e.g., colors, surface normals, and returned laser intensity). In 2017, Qi et. (Qi et al. (017) designed deep learning framework called PointNet that directly consumes unordered point sets as inputs and can be used for various tasks such classification and segmentation.
10

---

Published in Transactions on Machine Learning Research (01/2024)
(a) Steerable factor: engine.
(b) Steerable factor: engine.
(c) Steerable factor: size.
(d) Steerable factor: leg height.
Figure 4: GraphCG for point clouds editing. We show four editing sequences, where each sequence consists of five point clouds, and the center one is the anchor point clouds, i.e., with step size 0. The other four point clonds correspond step size with -3, -1.8, 1.8, and 3, respectively. Figure [[a] and Figure 4(b) refer the same semantic direction, and they are showing how to steer the factor engine: the number of engines will be decreased and increased
with the negative (left) and positive (right) step size respectively. Similarly, Figures [(c) and ((d) illustrate the effect. of the steerable factors on the car size and the chair leg height.
Backbone DGMs. We consider one of the latest DGMs on point clouds, PointFlow YYngg et al., 2019a It using the normalizing flow model for estimating the 3D point cloud distribution. Then we consider PointFlow pretrained on three datasets in ShapeNet {Chang et al., 2015): Airplane, Car, and Chair. All point clouds are obtained by sampling points uniformly from the mesh surface.
Analysis on steerable factors in point clouds: shape change. To train GraphCG, we take D = 10 directions, and we sample 8 point cloud sequences along 3 directions for visualization in Figure | More results are in Appendix[G] of cars and the height of chair legs. We also find it interesting that GraphCG, can steer more finger-trained factors, like modifying the number of jet engines.
is observed that GraphCG, can steer the shape of the point clouds, e.g., the size
6 Conclusion and Discussion
In this work, interested in unsupervised graph editing. It is well-motivated task for various real- world applications, and we discuss two mainstream data types: molecular graphs and point clouds. We start with exploring the latent space of mainstream deep generative models and propose GraphCG, model- agnostic unsupervised method for graph data editing. The key component of GraphCG, is EBM, and we take the GG-F as the solution for now. For future work, we may as well extend to more advanced solutions like denoising diffusion model (Ho al. 2020).
One limitation of GraphCG, (as well as the solutions to general unsupervised data editing) Härkköen al. 2020 Shen & Zhou 2021 Ren et al. 2021) is that may need post-training human selection (as shown in Algorithm to select the most promising semantic vectors to steer factors. Another issuc is the lack of open-sourced evaluation metrics. This requires both a deep understanding of the representation space of deep generative models and domain knowledge of the problem. For instance, activity cliff is a challenging task HHu & Jurgen 2012) for editing, while current measures fail to capture it. To set up constructive evaluation metrics can help augment our understandings from both the domain and technique perspectives. This is beyond the scope of our work, yet would be interesting to explore future direction.
11

---

Published in Transactions on Machine Learning Research (01/2024)
Code and Data Availability
The codes and data download scripts are available at this GitHub repository
References
Michael Arbel, Liang Zhou, and Arthur Gretton. Generalized energy based models. arXtv:2003 05033, 2020.
G Richard Bickerton, Gaia Paolini, Jérémy Besmard, Sorel Muresan, and Andrew Hopkins. Quantifying the chemical beauty of drugs. Nature chemistry, 4(2):90-98, 2012.
Hans-Joachim Böhm, Alexander Flohr, and Martin Stahl. Scaffold hopping. Drug discovery today: Tech- nologies, 1(3):217 224, 2004.
Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, et al. Shapenet: An information-rich 3d model repository. arXiv:1512.00 03012, 2015.
Tong Che, Ruixiang Zhang, Jascha Sohl--iccstein Hugo Larochelle, Liam Paull, et al. Your gan is secretly an energy-based model and you should use discriminator driven latent sampling. arXiv:2003, 06060, 2020.
Hongming Chen, Ola Engkvist, Yinhai Wang, Marcus Olivecrona, and Thomas Blaschke. The rise of deep
learning in drug discovery. Drug discovery today, 2346)1124 2018a.
Tian Qi Chen, Xuechen Li, Roger B Grosse, and David K Duvenaud. Isolating sources of disentanglement in variational autoencoders. In NewrIPS, 2018b.
Edo Collins, Raja Bala, Bob Price, and Sabine Susstrunk. Editing in style: Uncovering the local semantics of gans. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pp. 17771-5700 2020.
John Comer and Kin Tam. Lipophilicity profiles: theory and measurement. Pharmacokinetic optimization drug research: biological, physicochemical, and computational strategies, pp. 275 275-304, 2001.
Gabriele Corso, Luca Cavalleri, Dominique Beaini, Pietro Liò, and Petar Veličkovió. Principal neighbourhood aggregation for graph nets. arXiv:2004.05118, 2020.
Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components estimation. arXiv:1410.8516, 2014.
Jürgen Drews. Drug discovery: a historical perspective. Science, 287(5460):1960 1964, 2000.
David Duvenaud, Dougal Maclaurin, Jorge
Rafael
Timothy Hirzel,
et al. Convolutional networks on graphs for learning molecular fingerprints. arXi::1509.0000 2015.
Cian Eastwood and Christopher KI Williams, framework for the quantitative evaluation of disentangled representations. In ICLR, 2018.
Peter Ertl and Ansgar Schuffenhauer. Estimation of synthetic accessibility score drug-like molecules basod on molecular complexity and fragment contributions. Journal of cheminforrmtisss, 1(1):1-11 2009.
Peter Ertl, Bernhard Rohde, and Paul Selzer. Fast calculation of molecular polar surface area as sum of fragment-basee based contributions and its application to the prediction of drug transport properties. Journal of medicinal chemistry, 43(20):37143377 2000.
Peter Ertl, Eva. Altmann, and Jeffrey M McKenna. The most common functional groups in bioactive molecules and how their popularity has evolved over time. Journal of medicinal chemistry, 63(15):8408 8418, 2020.
Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In JCML, 2017.
12

---

Published in Transactions on Machine Learning Research (01/2024)
Laurent Gomez. Decision making in medicinal chemistry: The power of our intuition. ACS Medicinal Chemistry Letters, 9(10):956 000-956--58, 2018.
Michael Gutmann and Aapo Hyvärinen. Noise-con estimation: A new estimation principle for un- normalized statistical models. In ATSTATS, 2010.
Kai Han, Yunhe Wang, Jianyuan Guo, Yehui Tang, and Enhua Wu. Vision gnn: An image worth graph nodes. arXiv preprint arXiv:2206.002 00272, 2022.
Erik Härkönen, Aaron Hertzman, Jaakko Lehtinen, and Sylvain Paris. Ganspace: Discovering interpretable gan controls. In NeurIPS, 2020.
Kaveh Hassani and Amir Hosein Khasahmadi. Contrastive multi-view representation learning on graphs. In ICML, 2020.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, et al. beta-vac: Learning basic visual concepts with coistrained variational framework. In ICLR, 2017.
Geoffrey E Hinton. Training products of experts by minimizing contrastive divergence. Newral computation. 14(8):1771-1800) 2002.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Newral Information Processing Systems, 33:6840- 6851, 2020.
Ye Hu and Bajorath Jurgen. Extending the activity cliff concept: structural categorization of activity cliffs and systematic identification of different types of cliffs the chembl database. Journal of chemical information and modeling, 52(7):1806 1811, 2012.
Ye Hu, Dagmar Stumpfe, and Jurgen Bajorath. Recent advances in scaffold hopping: miniperspoctive. Journal of medicinal chemistry, 60(4):1238-13 1246, 2017.
Aapo Hyvärinen and Peter Dayan. Estimation of non-normalized statistical models by score matching, Journal of Machine Learning Research, 6(4), 2005.
John Irwin and Brian Shoichet. Zine: free database of commercially available compounds for virtual screening. Journal of chemical information and modeling, 45(1):177-182, 2005.
Ali Jahanian, Lucy Chai, and Phillip Isola. On the" steerability' generative adversarial networks. ICLR, 2019.
Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Hierarchical generation of molecular graphs using structural motifs. In ICML, 2020a.
Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Multi-objective molecule generation using interpretable substructures. In ICML, 2020b.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In CVPR, 2019.
Hyunjik Kim and Andriy Mnih. Disentangling by factorising, In ICML, 2018.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv:1312.6114 2013.
Abhishek Kumar, Prasanna Sattigeri, and Avinash Balakrishnan. Variational inference disentangled latent concepts from unlabeled observations. In ICLR, 2018.
Greg Landrum et al. RDKit: A software suite for cheminformatics, computational chemistry, and predictive modeling, 2013.
Shengehao Liu, Mehmet Furkan Demirel, and Yingyu Liang. N-gram graph: Simple unsupervised represen- tation for graphs, with applications to molecules. arXio:1806 090.6, 2018.
13

---

Published in Transactions on Machine Learning Research (01/2024)
Shengchao Liu, Hanchen Wang, Weiyang Liu, Joan Lasenby, Hongyu Guo, et al. Pre-training molecular graph representation with 3d geometry. In TCLR, 2022.
Francesco Locatello, Stefan Bauer, Mario Lucie, Gunnar Raetsch, Sylvain Gelly, al. Challenging common assumptions in the unsupervised learning of disentangled representations. In JCML, 2019.
David Mendez, Anna Gaulton, A Patrícia Bento, Jon Chambers, Marleen De Veij, et al. Chembl: towards direct deposition of bioassay data. Nucleic acids rescarch, (TDD)):D30-D940, 2019.
Zlatko Mihalié and Nenad Trinajstic. A graph-theoretical approach to structure-property relationships. Journal of Chemical Education, 69(9):701, 1992.
Andriy Mnih and Yee Whye Teh. fast and simple algorithm for training neural probabilistic language models. arXiv preprint arXiv:1206. 6426, 2012.
Weili Nie, Arash Vahdat, and Anima Anandkumar. Controllable and compositional generation with latent- space energy-based models. In NewrIPS, 2021.
Marcus Olivecrona, Thomas Blaschke, Ola Engkvist, and Hongming Chen. Molecular de-novo design through deep reinforcement learning. Journal of cheminformatics, 9(1):1-14, 2017.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv:1807.03745. 2018
Yael Pritch, Eitam Kav- Venaki, and Shmuel Peleg. Shift-map image editing. In ICCV, 2009.
Charles R. Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d
classification and segmentation. In CVPR, 2017.
Xuanchi Ren, Tao Yang, Yuwang Wang, and Wenjun Zeng. Do generative models know disentanglement? contrastive learning is all you need. arXiv:2102.105. 2021.
Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In ICML, 2015.
Karl Ridgeway and Michael C Mozer. Learning deep disentangled embeddings with the f-statistic loss. In NewrIPS, 2018
Radu Bogdan Rusu and Steve Cousins. 3D is here: Point Cloud Library (PCL). In ICRA, 2011.
Radu Bogdan Rusu, Gary Bradski, Romain Thibaux, and John Hsu. Fast 3d recognition and pose using the viewpoint feature histogram. In TROS, 2010.
Paul G. Seybold, Michael May, and Uijvala A. Bagal. Molecular structure: Property relationships. Journal of Chemical Education, 64(7):575, 1987.
Yujum Shen and Bolei Zhou. Closed-fome form factorization of latent semantics in gans. In CVPR, 2021.
Yujun Shen, Jinjin Gu, Xiaoou Tang, and Bolei Zhou. Interpreting the latent space of gans for semantic face editing. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9243-9252, 2020a.
Yujun Shen, Ceyuan Yang, Xiaoou Tang, and Bolei Zhou. Interfacegan: Interpreting the disentangled face representation learned by gans. IEEE TPAMI, 2020b.
Weijing Shi and Raj Rajkumar. Point-gun: Graph neural network for 3d object detection in a point cloud. In CVPR, 2020.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. arXiv:1907.05600, 2019.
Yang Song and Diederík P Kingma. How to train your energy-based models. arXio:2101.03288. 2021.
14

---

Published in Transactions on Machine Learning Research (01/2024)
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, et al. Score-based generative modeling through stochastic differential equations. arXiv:201.133556 2020.
Jiangming Sun, Nina Jeliazkova, Vladimir Chupakhin, Josc-Felipe Golib-Dzib, Ola Engkvist, al. Excape- db: integrated large scale dataset facilitating big data analysis in chemogenomics. Journal of chemin- formatics, 9(1):1-9, 2017.
Ayush Tewari, Mohamed Elgharib, Gaurav Bharaj, Florian Bernard, Hans-Peter Seidel, Patrick Pérez, Michael Zollhofer, and Christian Theobalt. Stylerig: Rigging stylegan for 3d control over portrait im- ages. In Proceedings of the IEEE/CVF Conference Compuler Vision and Pallern Recognilion, pp- 5142-6151, 2020.
Daniel F Veber, Stephen R Johnson, Hung- Yuan Cheng, Brian R. Smith, Keith W Ward, and Kenneth Kopple. Molecular propertics that influence the oral bioavailability of drug candidates. Journal of medicinal chemistry, 45(12):2615 2623, 2002.
Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E Sarma, Michael M Bronstein, et al. Dynamic graph enn for learning on point clouds. ACM ToG, 2020.
Zichao Wang, Weili Nie, Zhuoran Qiao, Chaowei Xiao, Richard Baraniuk, and Anima Anandkumar. Retrieval-based controllable molecule generation. arXie preprint 111 2022.
Zongze Wu, Dani Lischinski, and Eli Shechtman. Stylespace analysis: Disentangled controls for stylegan image generation. In CVPR, 2021.
Weihao Xia, Yulu Zhang, Yujiu Yang, Jing- Hao Xue, Bolei Zhou, et al. Gan inversion: survey. wrXiv:2101.05215 2021.
Guandao Yang, Xun Huang, Zekun Hao, Ming- Yu Liu, Serge Belongie, et al. Pointflow: 3d point cloud generation with continuous normalizing flows. In ICCV, 2019a.
Kevin Yang, Kyle Swanson, Wengong Jin, Connor Coley, Philipp Eiden, et al. Analyzing learned molccular representations for property prediction. Journal of chemical information and modeling, (9(8))3370-333) 2019b.
Jiaxuan You, Bowen Liu, Zhitao Ying, Vijay Pande, and Jure Leskovee. Graph convolutional policy network for goal-directed molecular graph generation. Advances in neural information processing systems, 31, 2018.
Chengxi Zang and Fei Wang. Moflow: an invertible flow model for generating molecular graphs. In KDD, 2020.
15


Reviews:
[{"review_id": "eTXhw7dHyW", "reviewer": "Reviewer_jtQQ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### _Main comments_\n1. __Related work:__  The presentation of the background and related work needs to be significantly expanded to fully understand the problem formulation and contextualize the contributions of GraphCG. Currently, there is very little discussion of other approaches for graph editing, and the discussion of the datasets and experiments is lacking, thus the interpretation of their results, especially for the molecular examples is difficult. \n\n2. __Ablations of losses in Eq 11:__ The loss in Eq (11) has a number of components. However, it’s unclear how these different components impact the performance of their steering method.\n\n3. __Clarity of problem formulation and method description:__ The authors' description of the problem is lacking in detailed assumptions and clear definitions, especially regarding the 'steerable factors' and their manifestation within graph structures. The formulation appears to be intuitive rather than rigorously defined, which makes it hard to understand precisely what the method entails. By the time we get to Equation (12), it seems to come too late. The reader is also left confused for some time about what a “semantic direction is”, how positive and negative views are generated, and how the directions are learned. The small subsection on “semantic direction modeling” should be integrated with the subsection w Eq (12) and not left alone without context.\n\n4. __Assumptions behind disentanglement experiment:__  The assumption that certain molecular fragments can serve as steerable factors and the subsequent use of RDKit for their extraction lacks justification. It would be useful to have more explanation of why these fragments are chosen and how they indeed reflect the latent representations’ structure. How can you be sure that these initial preprocessing steps are correct, and what if the DGM is disentangled with respect to different sets of factors?\n\n5. __Experimental details:__  The lack of discussion on the experimental setup and datasets within the main text makes it difficult to understand and assess the impact of the results. \n\n6. __Accessibility for non-experts:__ Finally, the paper would benefit from a more thorough walk through of the experiments and results concerning these datasets to enhance its accessibility and ensure that readers across various domains can really appreciate the contributions. The results and experiments on molecular datasets are not easy to understand without domain expertise.\n\n### _Additional questions:_\n1. How stable are the learned factors? Please provide some information and experiments to document the consistency of the factors and the sensitivity to hyperparameters.\n\n2. How does the number of learned semantic directions impact performance and interpretable factors? Are there cases where a model doesn’t learn a good set of representations and your approach can provide insight into these failure modes? \n\n3. “GraphCG aims at learning the steerable factors by maximizing the mutual information among corresponding directions, and its outputs are sequences of edited graphs.” - Make more clear what you mean by “sequences of edited graphs” at this point, consider using figures of point clouds later to illustrate your point\n\n4. How is the method connected to other applications of contrastive learning on graphs?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "### _Summary_\nThis paper introduces GraphCG, a new method for unsupervised graph controllable generation that can be applied to reveal steerable factors in pretrained deep generative models (DGMs) for graph data. The main contributions lie in the identification and utilization of steerable factors within these latent spaces. They do this by maximizing mutual information among semantically-rich directions, enabling the generation of modified graphs that vary along the learned factors. The authors’ method is tested on two datasets (molecules, point clouds) and the authors further provide experiments to quantify disentanglement of the different pretrained graph models. \n\nOverall, the paper tackles an important problem in unsupervised graph controllable generation and editing, and provides a well motivated solution to the problem. However, it requires improvements in clarity, contextualization, and justification to solidify its contributions to the field. Further comments and questions are provided below.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "### _Critical requested changes_\n1. Please provide the requested revisions to the background, method description and related work as requested above to improve clarity of the presentation. \n2. Experiments and insights into the stability of the method in discovering the same steerable factors.\n3. More discussion of related work and other approaches to controllable generation on graphs, and related work in disentangled factors for point clouds.\n4. Discussion of the hyperparameters and the different components of the loss in Eq (11) and how they impact the factors and performance. Visualizations and examples would be helpful (see additional suggestions below).\n5. Additional explanations of the molecular graph problem and datasets, and further insights into how to interpret your results on these datasets.\n\n### _Additional requested changes_ \n1. Additional experiments on more datasets to further highlight the application of the method.\n2. Quantification of the stability of steerable factors as you ablate different parts of the loss in Eq (11)", "broader_impact_concerns": "No broader impact concerns are noted.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "xBxO4jImWH", "reviewer": "Reviewer_cmNf", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is well written and provides sufficient context and is properly placed within the existing literature, to allow the reader to understand the motivation as well as the main intuitions behind the method. \n\nTo me a potential weakness is that the presentation of the paper tends to be somewhat generic. On one hand side, this makes a lot of sense, the framework introduced is generic, and not specific to a particular generative model. On the other hand, grounding things a little bit more can help understand what is going on (e.g. specifying things in terms of a particular generative model, or talk about the exact form of the function h). Some of these details are pushed towards appendix E, but even there I think things are somewhat brief. Overall I do think that someone familiar with the topic could figure out these details and reproduce most results, but someone less familiar might have a much harder time figuring out where to start. There is an algorithm provided in the paper, but I still think it keeps things quite generic.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The work introduces GraphCG a framework to understand the structure of a generative model, and provide means to edit examples by moving in meaningful directions. The core idea is to use a contrastive loss, and rely on ideas around mutual information and energy based models to identify semantically meaningful directions d to move in the latent space and to learn an editing function h (which can be linear) that allows moving in this directions. Specifically the work focuses on performing this on graph data, which is of particular importance for molecule editing.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I think the paper could be accepted in its current form. I'm not sure changes are strictly required. \n\nTo strengthen the paper I think a more detailed appendix E could help (e.g. what does norm stand for? whose norm? which norm?). Maybe some snippets of code could help there. In general how you initialize these directions, ablation or discussion about robustness of hyperparameters choices.", "broader_impact_concerns": "The paper proposes a technique that allows editing graphs in a semantically meaningful way. A target application widely discussed in the paper is molecule editing. As with any technology of this form, it has a dual use. It could be used for drug discovery and provide solutions for existing illnesses, as well as possibly for creating viruses. Or more generally, such techniques can be used on social graphs to understand how to modify them to maximize certain properties. \nHowever neither of these applications are a straight forward application of this work. And in particular, there exists a large body of works in this space, to which the current publication belongs, and this particular paper does not stands out from an ethical implications point of view compared to the others. \n\nTherefore I think there are no ethical implications that would raise specific concerns for this work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "w8BUFOnp3L", "reviewer": "Reviewer_dKVp", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "__Strengths:__\n- Most sections, apart from the method section read well, present the problem at hand relatively  clearly, and succeed in explaining why this is an interesting problem.\n\n__Weaknesses:__\n- The main weakness comes from the method section, which is unclear enough to leave doubts about the validity of the method.\n- Some of the experimental results are not convincing enough.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper has two main contributions:\n- It shows that the latent representation of Graph Deep Generative Models is not disentangled/does not isolate steerable factors.\n- It proposes a model agnostic method to find latent disentangled/steerable directions ex post, after the model has been trained.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "From most important to least important:\n- __Method section (critical):__ The method section needs to be thoroughly clarified. Section 4.1. lacks necessary details to be fully understandable. Typically, the authors mention that they need to sample latent codes from the latent space. This sampling procedure will differ from model to model, but this can be done either in a data dependent fashion or in a data independent fashion. For instance, for a simple non hierarchical gaussian VAE, or a gaussian based flow based model, this sampling can be done either by sampling from the gaussian prior, or by sampling data from the dataset, then encoding those points using the encoding function. When using the first sampling method, samples do not depend on the original data distribution, or on the generative model at play, and the quantities that are mentioned below (i.e. the MI between the considered distribution) could be analytically expressed and optimized. However, this would lead to discovery of steerable factors that are independent of the model and the data distribution, which I don't think is a reasonable thing to look for. By not specifying out of the box what sampling method is considered, the method section is very hard to read and get intuition about. This confusion is made even clearer in Algorithm 1, where f and g are defined as inputs to the algorithm, but never actually used. Besides I am not sure I understand fully why optimizing mutual information to get steerable directions is a meaningful thing to do, since what we seemingly expect from them is for the directions to be, on the one hand, informative, but mostly independent from one another. Maximizing mutual information seems to only maximize informativity, but not independence. To palliate for this issue, the authors seem to introduce an additional orthogonality constraint as an afterthought.\n- __Semantic direction modeling (critical):__ I don't understand why the authors are optimizing the parameters of an MLP taking as input (fixed) basis vectors, and not directly the coordinates of the d_i's. Could you clarify?\n- __Experimental evaluation (critical):__  As I am not very familiar with chemistry, I cannot evaluate the qualitative evaluation on the chemistry dataset. However, on the cloud point datasets, I am a bit doubtful of the independence of the factors discovered: for all of the examples given, the form of the generated object significantly varies along with the supposed steerable factor.\n- __Entanglement/Steerable factors (strengthen):__ A proper explanation of what steerable is, how it is defined, how it is measured and how it differ from disentanglement could be pushed at the beginning of the paper. I still don't understand the distinction between steerable factors and disentangled representation. If there are none, I would suggest framing the paper in term of disentanglement. I only got a clear idea of how one could define and measure steer-ability in the 'Entanglement of latent representation', which I find to be a bit late.\n- __Additional experiments in other modalities (strengthen):__ Given that the method seems quite general and not restricted to the graph modality, I think adding experiments on other modalities, potentially where more comparisons would be available, would add a lot to the paper", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers agree that the paper is worthy of publication in TMLR. Their main concerns regarding clarity were mostly addressed by the authors.

Here are som comments that should be addressed in a minor revision:
- it would be helpful to point out earlier on in Section 4 that the method relies on *positive* pairs z^u, z^v -- this is deferred to section 4.3, but it would be helpful to clarify this earlier.
- as pointed out by reviewer dKVp, it is not directly clear from the MI motivation in Section 4.1 how independence/disentanglement is enforced. In particular, the MI objective in eq.(5) seems to only consider two views from the *same* editing direction, while the repulsion between different direction i and j seems to first appear in the contrastive objective (9). Perhaps this comes from the fact that $i$ itself is randomly chosen, but is always the same in positive pairs? Please clarify this.
- In algorithm 1, steps 3 and 5 seem redundant?
Decision: Accept (poster)