Discrete Graph Auto-Encoder
Authors: Yoann Boget
Magda Gregorova
Alexandros Kalousis
Venue: Accepted by TMLR ()

Abstract: Despite advances in generative methods, accurately modeling the distribution of graphs remains a challenging task primarily because of the absence of predefined or inherent unique graph representation.
Two main strategies have emerged to tackle this issue: 1) restricting the number of possible representations by sorting the nodes, or 2) using permutation-invariant/equivariant functions, specifically Graph Neural Networks (GNNs). 

In this paper, we introduce a new framework named Discrete Graph Auto-Encoder (DGAE), which leverages the strengths of both strategies and mitigate their respective limitations. In essence, we propose a strategy in 2 steps. We first use a permutation-equivariant auto-encoder to convert graphs into sets of discrete latent node representations, each node being represented by a sequence of quantized vectors. In the second step, we sort the sets of discrete latent representations and learn their distribution with a specifically designed auto-regressive model based on the Transformer architecture. 

Through multiple experimental evaluations, we demonstrate the competitive performances of our model in comparison to the existing state-of-the-art across various datasets. Various ablation studies support the interest of our method.

Published in Transactions on Machine Learning Research (03/2024)
Discrete Graph Auco-Encoder
Yoann Boget
yourn. wwannn.eogtt@heege....
Geneva School for Business administration HES-SO
University of Geneva
Magda Gregorova
magda. meg.la.gregorocatttuuuuuu
Center for Artificial Intelligence and Robotics (CAIRO) Technische Hochschule Würzhwrg-Schhucinffrt ( (THWS)
Alexandros Kalousis
Secrandros.klllousisshhheo
Geneva School for Business administration HES-SO
Reviewed on OpenReview: https: // openreveew. form?? id= b280b0ub9
Abstract
Despite advances in generative methods, accurately modeling the distribution of graphs remains challenging primarily because of the absence of predefined or inherent unique graph representation. Two main strategies have emerged to tackle this issue: 1) restrict- ing the number of possible representations by sorting the nodes, or 2) using permutation- functions, specifically Graph Neural Networks (GNNs).
invariant/equiie
In this paper, we introduce a new framework named Discrete Graph Auto-Encodee (DGAE), which leverages both strategies" strengths and mitigates their respective limitations. In essence, we propose strategy in two steps. We first use permtaation-cquivariaat auto- encoder convert graphs into sets of discrete latent node representations, each node being represented by sequence of quantized vectors. In the second step, we sort the sets of discrete latent representations and learn their distribution with specifically designed auto- regressive model based on the Transformer architecture.
Through multiple experimental evaluations, demonstrate the competitive performances of our model in comparison to the existing state-of-the-art across various datasets. Various ablation studies support the interest of our method.
Introduction
Graph generative models are a significant research area with broad potential applications. While most existing models focus on generating molocules for de novo drug design, there has been interest in using these models for tasks such as material design (Lu al, 2020), protein design (Ingraham programming modeling (Brockschmidt et al., 2019), semantic graph modeling in natural language processing (Chen et al., 2018; Klawonn & Heim, 2018), scene graph modeling in robotics (Li et al., 2017).
al., 2019), code
Despite advances in generative methods, accurately modeling the distribution of graphs remains complex task. The main challenge lies the absence of predefined or inherent unique graph reprosentations. Given a graph of nodes, there are nl permutations, each correspouding a distinct represenlation of the same graph.
Two main strategies have emerged to tackle this issuc:
The first accepts multiple representations of the same graph but restricts their quantity by sorting the nodes thanks heuristic such as Breadth-First Search (BFS), as in (You al., 2018; Shi et al., 2020; Luo et al., 2021).

---

Published in Transactions on Machine Learning Research (03/2024)
2. The alternative approach employs models that are inherently invariant to node ordering permuta- lions. This method achieves permutation invariance through the use of Graph Neural Networks, in (Vignac et al., 2023; Jo et al., 2022).
Both methods have their own challenges, which elaborate on in Section 2.
Our main contribution is introducing a new generative model designed to address this issue of multiple graph representations. We use a permutation-cquivaiinn auto-encoder to convert graphs into sets of node embeddings and reconstruct graphs from sets of node embeddings. Coutrary to graphs, any sorting algorithu yields a unique representation for sets. So, we transform the sets into sequences and leverage an auto- regressive model to learn the implicit distribution over sets.
In implementing this framework, introduce new methods and techniques adapt existing ones. We enhance the node embeddings by introducing an innovative feature augmentation strategy. To ease the modeling of the latent distribution, we remove unnecessary information and reduce the size of the latent space by quantizing the node representations. However, for pragmatic considerations, we initially partition the node embeddings and subsequently quantize these partitions. As a result of this sequentialization, the latent representations form sequences spanning two dimensions: the partitions and the nodes. To model this distribution, we introduce a novel Transformer architecture designed to take advantage of its 2-dimensional structure.
The development and subsequent evaluation of our model yield several significant contributions that we summarize as follows:
We introduce novel generative model for graphs, leveraging graph-to-set discrete auto-encoder invariant to permutations.
We develop a new Transformer architecture, the 2D-Transformer, specifically designed to model sequences across two dimensions.
We design new technique for graph feature augmentation, which we call p- path features.
For the first time, we empirically evaluate various feature augmentation strategies via an ablation study.
Through empirical analysis, demonstrate the competitive performances of our model in compar- ison to the existing state-of-the-art across various datasets.
In the following, Section 2 presents the foundational concepts and issues of graph modeling. Section 3 dis- cusses the literature graph generation. In Section 4, introduce Discrete Graph Auto-Encode Encoder. Lastly, Section 5 presents the experimental evaluation of our model, demonstrating its competitive perfor- mance against baseline models for various datasets, including both simple and annotated graphs.
2 Background
In this section, we provide the necessary background about graph modeling. We define some notations and then review the challenges of learning graph representations, which mainly stems from the graph isomorphism problem. Finally, we introduce Graph Neural Networks (GNN) and discuss some of their limitations.
2.1 Notation
We define simple (unannotated and undirected) graph tuple G (V,E), where V is the set of nodes, whose cardinality is n, and E is the set of edges between these nodes. Given two nodes v; and vj in V, we represent the edge connecting them as Eij (pi, vy) E E. For annotated graphs, we introduce two additional functions V and E that map the nodes and the edges to their respective attributes: V(v) = æ, e g and E(G.)) = Ci,j E RRS. In the case of univariate discrete attributes, the categorical attributes are ono-hot encoded, and R and S are the number of node and edge categories, respectively. We denote an annotated graph G (V,8, V. F.).
2

---

Published in Transactions on Machine Learning Research (03/2024)
2.2 Graph isomorphism
A fundamental properly of graphs is their invariance with respecl to the representation ordering of the node. To put it differently, any permutation applied to the representation ordering of the node will yield an identical graph. Therefore, there are n! possible isomorphic reprosentations of the same graph. In the following, we employ the notation x(-) to indicate that we represent the graph under a specific node ordering.
The graph isomorphism problem, which consists of determining whether two graphs are identical or distinct, follows from these possible multiple representations (Köbler et al., 1993). Algorithms that offer unique rep- resentation for each isomorphism are at least expensive computationally as solving the graph isomorphism problem'. This problem has not been proven to be solvable in polynomial time (Helfgott et al., 2017).
As a result, algo that uniquely sort nodes in any graph are, at least, as expensive as algorithms solving the graph isomorphism problem.
Graph isomorphism a significant issue for graph generative models. Without a specific solution, generative models must learn new set of parameters for each permutation. Given the vast number of possible permu- tations for each graph, this brute-force approach is unreasonable. In fact, to the best of our knowledge, no generative model for graphs follows such procedure.
2.3
Generative models and graph isomorphism
Previous generative models, detailed in Section 3, have predominantly pursued two strategies to deal with the issue of ordering: sequentialization and invariance to permutation.
2.3.1 Sequentialization
The first method aims to establish ordering of nodes that yields sequentialization of the graph. Most models adopt algorithms sorting nodes in non-unique manner. Breadth-first search (BFS), used for instance in (You et al., 2018; Shi et al., 2020; Luo et al., 2021), is the most common heuristic to order nodes in graph generative models. We presume that BFS is also a convenient way of traversing the graph and works as an inductive bias for auto-regressive models. The main drawback BFS that the number of potential representations can still be considerable. the worst case (star graph), BFS does not reduce the number of possible representations at all.
2.3.2 Invariance to permutation
The alternative approach circunvents the issue by developing models invariant to permutation by construc- tion. These models rely on permutation- innvrriiiinnnnniannttt..a and wew.unttttioo-quvviiiinn functions. Permutation- invariance is defined as:
f(g)= = f(T(O)) x.
=
(1)
A function f is said permutation--equivaiinn if:
((T(9)) (((())
(2)
Notably, equivariance to permutation implies that input and output have the same cardinality. In practice, Graph Neural Networks (GNN) entails ionuttatioo-iivaianne and permutation-cquivaiinn In the next Section (2.4), we introduce the fundamental principles of GNNs
2.4 Graph neural networks
GNN layers are cetnuutttiion-quvaiiiaan functions. Since the cardinality of the input and the output of such function must match, GNNs are said to be '2
The graph isomorphis problem involves determining whether graphs are isomorphic. Establishing unique represen- tation for cach isomorphism be considered way to solve the problem. the representations match, then the graphs Isomorphic.
Tt. does not mean that the dimension of each node representation, usually vector, does change.
3

---

Published in Transactions on Machine Learning Research (03/2024)
An aggregating function providing graph-level information often follows the GNNs layers. Such aggrega- tion with commutative function, summation or averaging, results in permatatioon-nnariaat function.
2.4.1 Message Passing Neural Networks
Message Passing Neural Networks (MPNNs) (Scarselli et al., 2009) represent prevalent class of Graph Neural Networks (GNNs), even though other classes of GNN have gained in popularity, in particular, GNN, where each node shares information with all the other nodes as, for instance, in (Yun et al., 2019). The core concept underlying MPNNs involves updating the representation of each node by aggregating information from its neighboring nodes, those with which it shares an edge. Consequently, after layers, each node aggregates information from its I-hop neighborhood. We refer to this L-hop neighborhood as the receptive field of node features. So, each layer increases the size of the receptive field.
Message Passing Layer The update of the pth layer of an MPNN involves two steps. The first step computes the message:
H+1 = Sedge (ar, ady, ely) e 1.
(3)
The second step aggregates information to compute a new node representation:
Foode
a
I+1 i.j
(4)
JEN(()
In these equations, the functions f small neural nelworks, the + denotes any commutative function, and (i) is the set of nodes connected to node i. Note that the 'message' e can also be interpreted edge representation and passed to the next layer. In the following, we will favor this interpretation.
1+1
Limitations MPNNs present the advantage of addressing the graph isomorphism problem but exhibit some shortcomings. Specifically, two challenges have been widely acknowledged: oversmoothing and limitation of their expressive power.
Firstly, all node and edge features tend to the same value as the number of layers increases. We call this phenomenon oversmoothing (Li et al., 2018-02; Oono Suzuki, 2020). Because of it, one should limit the depth of MPNNs few layers.
Secondly, MPNNs exhibit limited expressive capacities in the sense that they cannot discern specific pairs of non-isomorphic graphs (Morris et al., 2019). Standard MPNNs are, at best, as expressive the Weisfeiler- Lehman (1-WL) test (Xu et al., 2019; Loukas, 2019). Consequently, they fail to detect some basic substruc- tures (Arvind et al., 2020; Chen et al., 2020). We illustrate such failure in Appendix D.
Two strategies to enhance the expressive capability of standard MPNNs have been proposed. Developing GNNs more powerful than MPNNs the first strategy (Maron al., 2019; Vignac al., 2020). So far, such GNNs are computationally much more expensive and, therefore, ineffective.
The second approach involves the augmentation of the node features with synthetic attributes, including spectral embeddings, cycle counts, and the integration of random noise. The feature augmentation enhances the ability of GNNs to caplure the graph structure. Using spectral embeddings (Laplacian Positional En- coding), cycle count, or random noise is now usual as additional synthetic features for graph generation (Krawczuk et al., 2021; Vignac et al., 2023). We present the details about these features in appendix D. In Section 4.1.2, we propose a new method for feature augmentation called p-path features, and in Section 5.4, experimentally the effect of the various methods
model.
4

---

Published in Transactions on Machine Learning Research (03/2024)
3
Related work
In this Section, we briefly review existing work on graph generative models. As exposed in Section 2, generative models for graphs have followed two main strategies to address the graph representation issue: sequentialization and invariance to permutation.
3.1 Sequential generation
This sequentialization serves dual purposes: it restricts the number of possible graph representations and facilitates auto-regressive graph generation. Notably, all models following this approach are auto-regressive. Graph canonization is computationally expensive, as discussed in Section 2. To the best of our knowledge, GraphGen (Goyal et al., 2020) stands as the only model employing this particular strategy for generic graphs. Most works only soek to reduce the number of graph representations by adopting a Breadth-First Search (BFS) based method.
We can further categorize rugo-segressive models into two classes. The first class, which includes the seminal GraphRNN (You et al., 2018), employs recurrent framework that necessitates maintaining and updating global graph-level hidden state. This approach inherenlly introduces long-range dependencciss, proximate nodes in graph topology can be distant in the sequential representation. The second class of auto-regressive models sidesteps long-range dependencies by recomputing the state for the partially generated graph at each step (Shi et al., 2020; Luo et al., 2021; Liao et al., 2019; Kong et al., 2023). Although this alleviates the long-range dependency issues, it. drastically increases the computational cost.
Furthermore, auto-regressive models, generating nodes and edges one by one, are inherently slow during the generation phase. Generation slowness is especially true for the second class of models, as highlighted by experimental results in Section 5.3.2 and in (Jo et al., 2022). GRANs (Tiao et al., 2019) attempt to address this speed issue by generating multiple nodes and edges simultaneously, albeit with a trade-off in generation performance.
Models previously discussed are generic, meaning they operate independently of any specific domain knowl- odge. However, many models are tailored particular domains, notably the domain of molccule gen- eration, which represents predominant application of graph generative modeling. Such domain-specific models often employ canonical representations. The Canonical SMILES notation, for instance, is commonly used to represent molecules, as in (Gómez-Bombarde et al., 2018; Kusner et al., 2017). Recently, powerful models representing molecules as 3D objects have emerged (Hoogeboom et al., 2022; Xu et al., 2023). While plenty of domain-specific sequential models exist (Liu et al., 2018; Samanta et al., 2020; Jin et al., 2018; 2020; Kuznetsov & Polykovskiy, 2021; Li et al., 2018), in this work, we focus on developing a generic graph model.
3.2 Models invariant to permutations
The second main strategy to address the multiple potential representations of a graph consists of using models invariant to permutations. By defimition, these models, also referred to as 'one-shot', cannot be sequential. These models implement invariance or equivariance through GNNs. They are thus insensitive to the or- dering of the graphs during training. Various methods, such as Generative Adversarial Networks (GANs), Normalizing Flows (NF), and diffussion/scoro-bass models, have been proposod following this strategy, cach with its merits and issues.
In GANs, discriminator (or critic) classifies (or scores) graphs between true (from the dataset) and gener- ated, whilc generator aims to maximize the error of the discriminator. MolGAN (De Cao & Kipf, 2018) proposes to use permutation-invariall discriminator. GG-GAN (Krawczuk et al., 2021) add a permutation- equivariant generator, and GrannGan (Boget et al., 2023) proposes to generate only the nodes and edges attribute from sampled skeletons. However, the adversarial training is known be unstable. MolGAN exhibits mode collapse and low diversity. GG-GAN also suffers from lack of diversity, while GrannGAN does not generate graphs, but only their attributes.
5

---

Published in Transactions on Machine Learning Research (03/2024)
Normalizing Flows are designed to learn invertible mapping between the data distribution and a known continuous distribution, typically the Standard Normal distribution. Both GraphNVP (Madhawa et al., 2019) and Moflow (Zang & Wang, 2020) use a two-step process, first generating an adjacency tensor, followed by an annotation matrix conditioned on this tensor. GNF (Liu et al., 2019) needs only onc stop but focuses on generating simple graphs only.
However, we hypothesize that, even with dequantization, efficiently spanning the continuous latent space using discrete objects remains significant challenge. Indeed, based their empirical evaluations, while these models surpass the performance of GAN-based approaches, they do not model graph distributions effectively as ruto-regrossive models and those relying on diffusion/scorrcbasedd methods.
Diffusion and score-based models gradually introduce Gaussian noise to the data and learn to denoise it for each noise level. In graphs, some models have suggested adding Gaussian noise to the adjacency matrix (Yang al., 2019) and to the annotation matrix (Jo al., 2022). By doing so, these models generate fully connected graphs, Iacilitating direct information sharing among all nodes.
However, as stated in (Vignac et al., 2023), the continuous noise compromises the graph structures, leading them to collapse into fully connected graphs. To address this concern, DiGress introduced a discrete diffusion model for graphs (Vignac et al., 2023), which integrates discrete noise to maintain a coherent graph structure at every noise level. In practice, however, the discretization does not appear to bring significant improvement over GDSS.
Whether continuous or discrete, the Inffussion/sore--eee based approach effectively captures the global structure of the graph and, as of now, has produced the best results with Digress and GDSS (Jo et al., 2022) considered state-of-the-art, However, the extensive number of denoising steps required by these models make the generation process comparatively slow, as shown experimentally and reported in Section 5.
3.3 Graph Auto-Encoders
Closer to work, Kipf and Welling (Kipf & Welling, 2016) proposed permutationn-equivriinn graph auto-encoder, namely the Graph Auto-Encoder (GAE) and the Variational Graph Auto-Encoder (VGAE) relatively long ago. However, they use it as a feature extractor for downstream tasks rather than for generation. In fact, neither of these two models captures the interaction and dependencies between node embeddings. One of our main contributions consists of modeling these interactions and dependencies nec- essary for generation. On a side note, we also remark that they use a simple activated scalar product as decoder, which can neither capture interactions between multiple node embeddings nor decode node and edge attributes. Like most auto-encoders in generative modeling, we use decoder mirroring the encoder.
Lastly, GraphVAE (Simonovsky & Komodakis, 2018) adopts a distinct approach to handle the multiple possible graph representations. A permutation-invarian encoder captures graph-level latent representation, losing the original node ordering in their aggregation. Then, a decoder, built using a standard feed-forward neural network, aims to reconstruct the original graph to yield a graph in any permutation. Nevertheless, during the training phase, the model requires aligning the original representation with its reconstructed counterpart. This comparison necessitates an expensive graph- maahhin procedure with a computational complexity of order O(n*).
Unlike GraphVAR, our auto-encoder is permutation-equuvarree representation in a single vector but a in a set of node embeddings, each capturing its local neighborhood so that we can reconstruct the graph from its sot of node embeddings. Since the scquentialization of sets does not sulfer the same problem as graphs, we can model the set of node embeddings as a sequence leveraging an regresseee model. To sum up, we follow the permutation invariance strategy to train a graph-to-set auto-encoder, and then follow the sequentialization strategy to model the distribution over sets.
end-to-end. It does not to capture the graph
4 Discrete Graph To Set Auto-Encoder
In a nutshell, our approach involves a two-step strategy. The first step graph auto-encoder. We encode graph into set of n node embeddings. For practical reasons detailed hereunder, we partition each node
6

---

Published in Transactions on Machine Learning Research (03/2024)
embedding into C' vectors and quantize each vector independently. Then, decode the quantized latent representation, aiming to recover the original graph. This first step generates discrete latent distribution over sets of node embeddings, yet this latent distribution remains unknown.
In the second step, our objective is to learn this latent distribution. Rather than directly modeling a distri- bution over sets, we transform the sets into sequences. We then leverage model based on the Transformer architecture to learn these sequences auto-regressively.
The rationale behind this two-step strategy stems from using cettantttaioo-quvvvriiiii functions, particularly Message Passing Neural Networks (MPNNs), to project the graph into latent space and decode it. This ap- proach prevents the node ordering issue, making the computation of the reconstruction loss straightforward without needing & matching procedure. However, as discussed in Section 2.4.1, MPNNs presenl inherent limitations, such as being constrained depth and only capturing information from their L-hop neighbor- hood, L being the number of layers in the MPNN. Consequently, MPNNs are unable model interactions between distant nodes. Therefore, we introduce the second step to model the latent distribution to address these long-range dependencies between node embeddings.
In this section, we present and motivate these two steps in detail (Sections 4.1 and 4.2), and we explicitly explain how we perform generation (Section 4.3. The source code of our model is publicly available at https://github.ce http::///gtabb..comyooooggggggg com/yobogge//daa
4.1 Discrete Graph Auto-Encode Encoder
In this initial step, our primary objectives when encoding graphs are twofold. Firstly, we aim to learn node embeddings that facilitate the accurate reconstruction the original graph. Secondly, we strive to create a latent distribution that handy to learn in the second step.
To achieve this, enhance the input graph representation with feature augmentation (Section 4.1.1).
We use Message Passing Neural Network (MPNN) to encode graphs G into sets of node embeddings (=!,..,, . 2h, mapping a space over graphs to space over sets Semender G - 2h (Section 4.1.2). Partitioning and quantizing the node embeddings results in a handy discrete latent space with known support, facilitating the modeling of the latent distribution in the second step. Specifically, we partition the node embeddings in a sequence of C vectors z = (zf, =L)). We then quantize each vector independently (Section 4.1.3 4.1.3), replacing them with their closest neighbor a codebook. Finally, another MPNN decodes the sets of quantized vectors 29, also called codewords, graphs G, formally fdecoder 2 + G. Figure 1 visually summarizes the entire procedure.
n
n
g
Encoder
zh Quantization Za
Decoder
G
Figure Diagram of our auto-encoder. The encoder an MPNN transforming the graph into a set of node embeddings zh 2. The elements of the set Zh are partitioned and quantized, producing a set of codeword sequences Z', 3. The decoder, other MPNN, takes the set 2 and reconstruct the original graph.
4.1.1 Input Graph and Feature Augmentation
As explained in Section 2.4, conventional MPNNs cannot identify certain fundamental graph substructures. Such shortcoming adversely impacts the information captured by the encoder. Tnevitably, the node
7

---

Published in Transactions on Machine Learning Research (03/2024)
beddings miss the information the encoder fails to capture, preventing accurate reconstruction. A common mitigation strategy involves enhancing MPNN capacities by augmenting the input features with synthetic attributes. We employ this approach to generate more informative node embeddings, allowing for better roconstruction performance.
Here, we propose a new augmentation scheme that aggregates information from the p-path neighborhood. Specifically, we first create virtual edges (with vector of zeros as attributes) between nodes that are connected by paths of length 2 to p (the edges between adjacent nodes, i.. connected by a path of length 1, are already represented by vector of edge attributes). We then concatenate synthetic p-dimensional vector to cach vector of edge attribute, whose 8th entry corresponds to the number of paths of length connecting the two endpoints. Similarly, we augment the node features with p-dimensional vector, whose it entry indicates the number of paths of length emanating from the node. In Appendix D, we provide more details on how we compute the path lengths as well as information about the other feature augmentation schemes used for our experiments.
Our method presents two benefits compared to other methods. First, unlike other methods, our p p-path augmentation strategy produces synthetic attributes for nodes and edges, providing richer information about the substructures around each node. Second, the virtual edges aggregate information from larger neigh- borhood. The computation of the synthetic features a unique prepossessing step. In our experiments, we use the p-path method setting p 3.
Our solution related to the recent k- hop message passing presented in (Feng et al., 2022), both using the path length information. Our proposition differs two main points. First, create synthetic features, while k-hop message passing uses this information to compute different message passing functions for each path length. Second, we take into account the number of paths of length p between 2 nodes, while they only consider if there is at least one such path.
the path information to
In Section 5.4, we empirically show that our method increases the reconstruction performance of our auto- encoder the most compared to other methods.
4.1.2 Encoder
We aim eucode graphs as sets of node embeddings, where each embedding captures its local graph structure so that we can effectively recover the original graph from these embeddings. We use an MPNN as encoder since they are specifically designed and efficient for modeling the local neighborhood of each node.
We formally define our encoder L-layer MPNN whose layers follow these two equations:
bn(fedge ([( 2/), el,y))
(5)
x 1+1 bn x
Σ Snodo ([x), a/j, e|,5))
(6)
JEA(()
where bn stands for batch normalization (TolTe Szegedy, 2015) and [, - is the concalenation operator. After the last layer, we discard the edge representation ex, and keep only the node representation so that 2/¹ = of E Rederenee The complete graph latent representation is the set of all the node embedding gh = {=,"..., where the superscript h indicates the set before quantization. The superscript will refer to the latent representations after quantization.
4.1.3 Discrete Latent Distributions
In the context of our auto-cncoder, quantization consists of replacing a dimensional vector by its nearest neighbor in learned codebook H gmxd m being the codebook size. By partitioning, we refer a simple reshaping operation, where vector of dimension d is reshaped in C vectors of dimension d/C. Both
8

---

Published in Transactions on Machine Learning Research (03/2024)
operations follow the same objectives: produce latent distribution that is handy to model (in the second step) and flexible enough to yield good reconstruction.
Specifically, the encoder outputs n node embeddings z E Rdducent So, we partition these node embed- dings in C vectors ( x(,,, where each x is in Rduttee/C Then, we quantize each vector inde- pendently. We refer to the partitioned and quantized representations as the discrete latent representation 2a ,..,, =f,c),,.. (I =)) Figure 2 depicts the quantization procedure. Partitioning and quantization result in discrete latent distributions.
Codebooks He
11,1
h1,0
h h
hm,C
9 Z
1,1
h Z
zh
2q
Figure Diagram of the quantization. We represent each node embedding by C partition vectors =,;-- Then, we quantize each of these vectors by replacing them with their closest neighbor from the corresponding codebook He. The vectors in the codebooks are parameters learned during training.
Discrete Latent Representations The node embeddings z/, as they are output by the encoder, are real-valued vectors Rdiaceme We make two observations regarding these vectors.
First, if the original graphs have only discrete node and edge attributes, the support of this distribution is discrete. It corresponds to all the possible subgraphs within the L-hop neighborhood, where L is the number of encoder layers. However, enumerating the support of this discrete distribution is intractable in almost all practical scenarios, which makes the modeling of this discrete distribution unpractical. In the continuous case, the distribution of the node embeddings remains also unknown and challenging to model.
Second, the encoder cannot model interactions between distant nodes. By consequence, even we could model the distribution of each node embedding individually, the distribution over the sets of node embeddings would remain unknown.
Both observations, i.e., the potential discrete support of the node embedding distributions and the impossible modeling of distant nodes, make standard variational auto-encoder unappropriated for the task. More generally, the distribution of node embeddings 2 as they are output by the encoder, is challenging. On the other hand, discrete latent models have shown impressive results with both discrete and continuous data (van den Oord et al., 2017; Razavi et al., 2019). Therefore, we leverage such a model using quantization to produce discrcte latent distributions with known support.
Quantization The quantization of the node embeddings into discrete latent representations presents many advantages. First, discrete latent representation is natural fit for modeling discrete structures like graphs. In particular, discrete representation prevents approximating step functions or Dirac delta functions with neural networks, which require smooth differentiable functions.
Quantization also enforces the codewords follow a m-dimensional categorical distribution, where m, the codebook size, is a rrper-parameter Knowing that the latent vectors follow an m-dimensional categorical distribution, learn their parameters in the second stage of our model.

---

Published in Transactions on Machine Learning Research (03/2024)
Moreover, quantization acts an information bottleneck. Assuming no prior information, the information contained in each selected codeword is I (m) bits and depends only on the codebook size m. By training the auto-encoder with a reconstruction objective, we encourage the quantization process to retain only the most valuable information for reconstruction. Also, by reducing the codebook size, we shrink the number of categories and, therefore, the number of parameters of the calegorical distribution; making the latent distribution casier to learn. Limiting the codebook size also reduces the risk of codebook collapse (Kaiser et al., 2018) as observed in our experiments (See Section 5.4), favoring better coverage of the space and preventing sampling from regions out of the distribution. However, restraining the information passed to the decoder makes the reconstruction harder. As result, there is trade-off between the reconstrue- tion quality and the dimensionality of the categorical distribution stemming from the codebook size. We empirically show the elfect of this trade-off in our experiments (See Section 5.4). We indeed observe that the reconstruction loss decreases when the codebook size increases. However, larger codebook sizes do not yield better generation performances despite better reconstruction performance. We explain this observation because larger codebooks result in larger categorical distributions that are harder to learn.
Partitioning In practice, we often need a large discrete latent space to model cach node represcntation with enough flexibility. We can obtain this flexibility either with a single codeword with large codebook or with multiple codewords, each coming from a (much) smaller codebook.
Large codebooks imply high dimensional categorical distributions, which are inconvenient in practice. Specif- ically, modeling a high-dimensional categorical dist ributtinn in the second stage of our model would require an output layer whose size corresponds to the dimension of this distribution. Large output layers, particularly those larger than their hidden layers, are ineffective in practice. Our experiments show that our model's generation performance drops for the larger categorical distribution.
So, instcad of representing cach node embodding with single codeword, we use multiple codewords for each node representation. To that end, we partition the node embeddings zf E Idiatent in C' vectors z/ C Rdvaien/C by reshaping them.
In consequence, cach node embedding is represented by C partition vectors. We quantize each of these partition vectors independently, each partition with its own codebook. In fact, in graphs, as in images, codebooks shared across features reflect an invariance assumption: the permutation invariance for graphs and the translation invariance for images. We use different codebooks for the different partitions as we do not assume any invariance across partitions. In other words, there is no reason to share codebooks across partitions because the various partitions do not belong to the same feature map.
After partitioning and quantization, the latent representation of each node is sequence of codewords. We call dictionary the set all possible codeword arrangements. Therefore, for a fixed codebook size m, the dictionary size is M = m.. We empirically assess the effect of partitioning and present the results Section 5.4.
Formalisation Specifically, the partition function fp defines simple reshaping operation transforming a vector of dimension diatent in C' vectors of dimensions diatent/C, that is fp : Rddiaent + RC' xd((ttonn)/)))
We define the quantization function ge: Bddatens/CC + (hr.c...hm.dd e as the mapping of the c-th partition of node i, to its closest neighbor in the corresponding codebooks H. E BB such that:
(e(2 = 2º.0 = -hk,e with h == arg min(l - -hg.elz).
(7)
9
Notably, the quantization procedure acts independently on every element in the set. It is, therefore, permutation-equuiarian..
We underline that a bijection exists between the quantized vectors and their corresponding indices in the code- books. Therefore, the quantized vectors and their indices hold the same information. In the following, we refer to the set of indices corresponding to the set of quantized vectors as Z {(2 zi,c), ... (2%,1: z%,c)) K. ((k,,,,.... ki,c), .-., (kn,1, "h..,, kn, where ki,c E (1,-.., m).
10

---

Published in Transactions on Machine Learning Research (03/2024)
4.1.4 Decoder
After quantization, the decoder has to recover the graph structure and the node and edge attributes for annotated graphs. Formally, it a sel-lo-graph function Sdecader 2 H G. Since we have dropped any explicit representation of the graph (see Section 4.1.2), when we decode it, we assume fully connected graph among the elements of 24. We subsequently feed the fully connected graph into an MPNN similar to the one used as encoder but without feature augmentation. The decoder output serves logit to model the discrete distributions over the edges and the nodes when the graph has node attributes. We present the implementation details in Appendix A.1.
4.1.5 Training
Our training strategy follows principles developed to train models performing quantization of the latent space (van den Oord et al., 2017). The training objectives are simultaneously reconstruction and quantization. We define reconstruction loss for the first objective. We update the codebook vectors center using Exponential Moving Average (EMA) as the second objective. Lastly, add regularization term to prevent the latent space from growing indefinitely. In the following, we present these clements and provide the implementation details in Appendix A.2.
We train the parameters of MPNNs (encoder and decoder) to minimize the recon- struction loss. We define it as the expected negative log- likelioed the graph given the set of codewords z
Reconstruction loss
Lrecon. l2 (po(G(()))
Thanks to the into-encoder's permutation permatation--quiiaaaann, the loss computation is straightforward since the representation order of the output corresponds to the input one.
The main difficulty comes from the quantization operation, which is non-differentiable. Therefore, no gradient can flow back to the encoder. We circumvent this difficulty by following (van den Oord et al., 2017) and passing the gradient with respect to the codeword directly to the encoder's output, bypassing the quantization operation.
Vector Quantization Objective The Vector Quantization (VQ) objective updates the vector in the codebooks. The objective function, acting as an online k-means algorithm, updates the vectors the codebook using FMA as proposed (van den Oord et al., 2017). We adopt the k-mean+ initialization for the codebooks proposed in (tańcucki al., 2020).
Commitment loss Finally, nothing prevents the embeddings from taking arbitrarily large values. To circumvent this issuc, follow again (van den Oord et al., 2017). We incorporate commitment loss function, which pushes the encoder outputs close to their corresponding codewords.
4.2 Modeling The Latent Distribution
The auto-encoder implicitly defines distribution over the latent space Z4, which translates over the space of the corresponding sets indices K defined in Section 4.1.3. For generation, we need to sample from one of these distributions. So, we decide to model the distribution of indices. After sampling, we recover the corresponding codewords from their indices.
Instead of lcarning a distribution over the sets of index partitions (.f,,....,,,,c.......... (kn,1. kn,c)), we turn the sets into sequences by sorting them. After sorting, the sequences of indices K E RRaxC have two dimensions: the partition dimension and the node dimension. Instead of flattening the sequence, we propose a new model basod on the Transformer architecture (Vaswani et al., 2017) that leverages the dimensional structure of the sequences implement parameter sharing for computational efficiency and better generalization. We call it AD--rrnsffrmer.
11

---

Published in Transactions on Machine Learning Research (03/2024)
Our main contribution concerns the attention function. Before presenting it, we first detail the sequen- tialization, the auto-regressive process, the input and output of our model, and recall some basics of the Transformer architecture.
4.2.1 Sequentialization
The existence of multiple permutations carries over from the graphs to the sets in the latent space. The multiple possible representations as well as the limitations of iinitaattio-euuvvriiant and permutation- invariant functions are still an issuc.
However, unlike graphs, sorting sets in a unique representation is easy. Any sorting algorithm can sort sets, yielding unique representation. We can then represent the latent distributions as sequences and learn them auto-regressively.
Additionally, we remark that sorting is specific kind of permutation. Since the decoder is permutation- equivariant, permuting the node embeddings in the latent space only changes the representation of the output graph but not the graph itself. So, we sequentialize the sets and learn the sequential distribution ruto-regressvvely.
In practice, we sort the set of index soquences K. {(k1,1...., kI,c), (kn.1, ..., kn,c)) (the indices corre- sponding to the codewords 2 1(=f.1...., ,).,)...., "f.,),).,,,,............. (2% (2 =),c)),, see Section 4.1.3) by increasing order using the first partition the primary criterion. In case of tie, use the index of the second partition as secondary criterion, etc. We permute the set 2 accordingly. We obtain sequences of indices K E R and, the corresponding sequence of codewords Z E pnxCxdiatas//
Since we perform the sorting operation as a preprocessing step for the second stage of our model, there is no backpropagation through the sorting operation.
The ordering obtained is arbitrary, but it has the advantage of being completely generic. Domain-specifie information can certainly lead to orderings that improve learning performance. We let this question open for further domain-specifie research.
Regarding notation, we remove the superscript g from the latent vector z since we only work with the selected codewords in this second stage. Instead, we use the superscript to indicate the number of Transformer blocks traversed so that z/_c correspond to the representation Zí, after blocks.
4.2.2 Auto-regressive setup two dimensions
Our model uses the codewords vectors as input to model the distribution of indices K. Formally, we have:
12 C
P ((kk,,..., kI.C), (kn.11 kn, .)) = IIII
Rei,C, Zi,co),
(9)
where øcij refers to all the vectors having the first index smaller than i and as second index.
We introduce the description of the inputs and outputs of our model as it should help to understand how our auto-regressive model accesses all and only allowed positions.
For the input, we offset the first dimension (i.e., the node dimension) by one position to predict the partition indices of the next node, and we concatenate the known partitions of the current node to predict the next partition. Since the number of known partitions varies, so does the size of the input vectors. Therefore, we linearly project these vectors so that all the input vectors have the same dimension dmodet- So, we have:
2
W [2] 1,1,,,1,, 1... 1.c,
(10)
and
2 Win 11-1,11 -1-100, Zt,1 Zi,el:
(11)
where the Win E Rddoodd x (diatent + (e/C)dleeet) are matrices of parameters and [.] is the concatenation opera- tor. To obtain the vector in the first position 21 we introduce virtual node (Zo,1, 20,c) made of vectors of zeros.
12

---

Published in Transactions on Machine Learning Research (03/2024)
After L layers, we project the output z/e and apply softmax to model the index distribution of the next partition.
(o(k,.) softmax(fou
(12)
We pad the sequence with an end-of-sequence token so that the sequence can take various sizes. Since we sorted the indices the training set, the indices lower than the last. known index are out of the distribution; for instance, 1 >= k-1,1) = 0. Even though the auto-regressive model should learn generate the
soquence following the order in the training set, case the task of our model by enforcing this constraint with masking.
4.2.3 Transformer
The Transformer is based on atlention. The atlention function takes queries qi, keys k;, and values vi, all are transformations from the corresponding representation z,. The attention output is a weighted sum of the values, where the weights correspond to a compatibility function, such as the scalar product, between queries and keys. The queries correspond to the aggregating position, while keys and values correspond to aggregated positions. The original attention from (Vaswani et al., 2017) includes scaling factor so that the attention output is computed as:
q k V
a, Σ softmax jSi
(13)
In practice, all attention functions are computed simultaneously using matrix multiplication. Masking M ensures that the weighted sum includes allowed positions only. We have: A = softmax OK"
@ MMV. Follow-
va.
ing the original paper (Vaswani al., 2017), we use multi-head attention, residual connections, position-wise fully connected ferd-forward networks, and positional encoding.
2D Attention The main idea behind TD-nrsssformer lies in the computation of the attention function. Instead of computing the attention function for all the partitions, we compute it over the node embeddings whole. In addition to the benefits offered by transformers over other auto-regrossive models, such the computational complexity per layer, the parallelizable computation, and the short path lengths between long-range dependencies (Vaswani et al., 2017), our T--rrasfformer implements parameter sharing for com- putational efficiency and better generalization. So, the keys and values are computed once for the whole node, so that we have: ky = f1(+),1,), and of = (2( One can interpret it as a form of parameter sharing since we use the same keys and values and, therefore, the same set of parameters for all the partitions. Nevertheless, the attention weights depend on the partition through the queries, which are different for each partition, qh = (z),)) In practice, the functions fk, fu, and fg are small neural networks.
Thanks to the queries, the attention score, kj also depends on both dimensions. So, the main change of our model compared to the standard Transformer appears in the attention function, which we compute as:
J. Vdl
ai.c Σ softmax(
(14)
Duc to the unique key and value for the first dimension, we compute the summation along the second dimension by including only the previous representations (j i). As the original Transformer, we scale the attention score by vdg. where dæ the dimensionality of the vector The information about previous partitions in the current node passes through the residual connections as presented hereunder.
Transformer blocks Except for the attention function, our model follows the Transformer architecture. We compute attention every head of every layer and use element-wise forward neural network, layer normalization, well as residual connections:
13

---

Published in Transactions on Machine Learning Research (03/2024)
1+1 Ri
LayerNorm(diee f((,)))
(15)
where
d_. LayerNormm ( + [d),
(16)
and where Of is the concatenation of the H heads. After L blocks, take the outputs ii to model the probability distribution of the next index, as described in Equation 12.
I+1
4.3 Generation
For generation, we iteratively sample from Po(ki,e) and get the corresponding zi.-- Sequence generation stops upon sampling the end-of-sequence token reaching the maximum number of node embeddings mmaz- So, we generate graphs of various sizes, and we need at most Am iterations to generate instance,
Despite the regresseee method, generation with our model is significantly faster than other methods. There are three reasons for this. First, at each iteration, the computational complexity bounded by the vector-matrix product of K", which Oddmodet) where most models require at least O(n"), where w is the matrix mulliplication exponent. Second, the length of the auto-regressive sequence for graphs usually depends on the number of (potential) edges and is proportional to n², Instead, our method is auto-regressive For the current size of the generated graphs, nC is also smaller
sequence depending lincarly than the number of time steps in score-based models. We input the generated sets into the decoder and generate the resulting graphs simultaneously. Finally, the efficiency of our method allows us to keep the size of our model small. As presented in Appendix A.3, the largest hidden dimension of our experimental models never exceeds 256 hidden units, far from the dimension of current large models. In Section 5, we experimentally show that our method is comparatively fast al generation.
5
Evaluation
This section presents the results of our experiments evaluating our model. We first compare the performance of our DGAE with baseline models, both on simple and annotated graphs. We then perform various ablation studies to evaluate and understand our model better.
5.1 Experimental setup
We evaluate our model on both synthetic datasets and molecular datasets. In the following, we present these datasets, the metrics used to assess the quality of generated samples, and the baseline models used for comparison. Appendix A.3 reports the details of the models, the hyperparameters choices and the resulting number of parameters used for the experiments. For fair comparison, we adopt the experimental procedure followed in (Jo et al., 2022) regarding metrics, split between training and test sets, number models and runs, and sample sizes described hereafter.
5.1.1 Simple graphs
For simple graphs, we evaluated the performance of our model on two datasets of small graphs, namely Ego-Small and Community-Small, with 200 and 100 graphs, respectively as well as Enzymes, a dataset of larger graphs. In particular, split the datasets between training and test sets with ratio of 80% 20% as (Jo et al., 2022) and use the exact same split when available.
The Ego-Small dataset is constructed by extracting ego-networks from the large citeseer network, a real-world social network. The dataset comprises 200 graphs having between 4 and 18 nodes.
The Community-Small dataset is purely synthetic. The number of nodes each graph is uniformly sampled from the set (12, 14, 16, 18, 20). Each graph is divided into two communities of equal size. The probability of edge intra-community is set to Pintra 0.7, while the probability of edge between inter-commuities is set
14

---

Published in Transactions on Machine Learning Research (03/2024)
to Pinter 0.03, with the constraint of having at least edge between communities. We use the version of the dataset proposed by (Jo et al., 2022) containing 100 graphs.
Enzymes is a dataset containing 587 protein graphs that represent the protein tertiary structures of enzymes. It extracted from the BRENDA database. The graphs in this dataset are larger than in Ego-small and Community-Small, with a node range between 10 and 125.
We employ the maximum mean discrepancy (MMD) to compare the distributions of graph statistics between generated and test graphs (You et al., 2018). The MMDs computed over the distributions of degrees (deg.), clustering coefficients (clust.), and the number of occurrences of orbits with up to four nodes (orbit) Similar to (Jo et al., 2022), we utilize the Gaussian Earth Mover's Distance (EMD) kernel to compute the MMDs.
The MMDs are computed by comparing the test set to generated samples of the same size as the test set. For Ego-Small and Community-Small, our reported results are the average of fifteen runs, i..,, fifteen generation batches with the lest, set size used for evaluation against the test set: three runs from five models trained independently. For the Enzymes dataset, which contains much larger graphs, we follow (Jo et al., 2022) and average over three runs from single model. Due to space limitation, we report the standard deviation in the appendix B.
5.1.2 Molecular datasets
We evaluated the performance of our model on two widely used datasets for molecule generation: Qm9 (Ranakrishnan et al., 2014) and Zinc250k (Irwin et al., 2012).
The Qm9 dataset (Ramakrishnan et al., 2014) comprises 133,885 small organic molecules with up to nine heavy atoms, consisting of only four atom types, namely C, O. N, F. We followed the convention in previous works and employed the kekulized version of the dataset with three edge types (single, double, and triple).
The Zine250k dataset is a subset of the Zine database (Irwin el al., 2012) that includes 250,000 molecules with up to 38 heavy atoms of nine types. We also used the kekulized representation of this dataset.
We used the test sets provided by (Jo et al., 2022). The metrics are calculated over 10,000 samples from training and Lest sets when needed.
Traditionally, molecule generation was evaluated through three metrics: validity, uniqueness, and novelty. We argue that these metrics are unsuitable for evaluating the generative graph models.
Recent works mostly use valency corrections, bringing the validity rate to 100%. Consequently, this metric has become uninformative. Uniqueness and novelty primarily indicate potential mode collapse and over- fitting, respectively. Since most models, including ours, achieve a high score, these metrics provide little information for evaluation. Therefore, we do not use these metrics for evaluation. Nonetheless, we report them in the Appendix B for completeness.
Instead, we use the Fréchel ChemNet Distance (FCD) (Preuer al., 2018) and the Neighborhood subgraph pairwise distance kernel (NSPDK) MMD (Costa & Grave, 2010) metrics. As highlighted in (Jo al., 2022) 'FCD and NSPDK MMD are salient metrics that assess the ability to learn the distribution of the training molecules, measuring how close the generaled molecules lie to the distribution.. FCD assesses the generated molecules in chemical space, while NSPDK MMD evaluates the distribution of the graph structures.
In addition to FCD and NSPDK metrics, we also include the validity rate without correction as a supplemen- tary evaluation metric. This metric calculates the fraction of valid molecules without any valency correction or edge resampling, and we employ the version that allows for formal charges as in (Jo al., 2022).
We remind that an orbit is the equivalence class of uodes of graph under the action of its automorphisms. Less formally, it set of nodes that are structurally similar. In Figure 13 appendix the nodes with the same color belong the same orbit.
15

---

Published in Transactions on Machine Learning Research (03/2024)
5.2 Baseline models
We present the results of six baseline models in our experiments for simple graphs: three sequential and three seimuntttioo-equivariinn
models. The first model, GraphRNN (You et al., 2018), is an RNN-based auto-regressive model and is the paper that introduced the experiments and metrics we use. The second model, GraphDF (Luo el. al., 2021), is a discrete flow-based model, and GraphARM (Kong et al., 2023) is the recent diffusion-based sequential model. The third and fourth models are one-shot continuous diffusion models, namely EDP-GNN (Yang et al., 2019) and GDSS (Jo et al., 2022). EDP-GNN is initially designed for simple graphs, while GDSS can generate both simple and annotated graphs. Finally, DiGress (Vignac et al., 2023) is díscrete diffusion model.
In addition to these models, we randomly sample the same number of graphs from the training set as in the test set to use as baseline. It gives us the distance between two random samples from the true distribution. Models under this baseline indicate that the generated sample overfits the test set. We report the average MMDs between the test set and 15 random samples drawn from the training set.
For molecular graphs, we also present the results of six baseline models. We utilize GraphAF, GraphDF (Luo et al., 2021) and GraphARM. Additionally, we compare our model against MoFlow (Zang & Wang, 2020), one-shot flow-based model, EDP-GNN (Yang et al., 2019) score-based model adapted by (Jo et al., 2022) to handle annolated graphs, GDSS(Jo et al., 2022), another score-based model and DiGress (Vignac et al., 2023), a discrete diffusion model. We take the results from (Jo et al., 2022) for all baselines except GraphARM and DiGress taken from (Kong et al., 2023).
As discussed in Section 3, there are models specifically designed for graph generation using different types of representation. In particular, some recent models, such as E(3) Equivariant Diffusion Model (EDM) and Geometric Latent Diffusson Models (GEOLDM) leverage diffusion models representing molecules as 3D coordinates point clouds. These models predict the connections between atoms post hoc based on the distance between atoms. Compared to our experiments, these models use additional domain-knowleddg information, particularly the atom positions. So, they must use datasets including this information, such as Qm9, but excluding 250k Moreover, they use specific metrics such as atom and molecule stability. Conversely, they do not report specific metrics for graphs such as NSPDK. Consequently, the only point of comparison is the validity rate on QM9. On this metric, EMD and GEOLDM obtain close and comparable results with the models used as baselines for our experiment with 91.9% and 93.8%, respectively.
5.3 Results
This section reports quantitative evaluation of the experiments. We provide visualization of the generated graphs in Appendix C.
We give the evaluation results on Ego-Small and Community-Sman Small datasets in Table 1. In the case of Ego-Small, which is relatively small and straightforward dataset, we observe that our model matches the scores obtained with random samples from the training set. GDSS and DiGress (Jo et al., 2022; Vignac et al., 2023) also reaches this level (or better!) on all metrics. Therefore, this dataset does not allow to discriminate between the best models.
On the Community-Small dataset, our model presents the best average results compared to baseline models. Table 2 presents the results on the Enzymes datasets. Again, our model outperforms baseline models on average.
5.3.1 Molecular graphs
As presented in Table 3, DiGress outperforms other models over all metrics in the QM9 datasct. However, it performs much worst on the larger molecules of the Zinc250k dataset. Except. DiGress on QM9, our DGAE model exhibits superior performance to all other models regarding FCD and NSPDK metrics, frequently surpassing them by a substantial margin. However, we observe that it is slightly under the best models regarding the validity without correction metric while remaining competitive. These results suggest that, compared to baseline models, our model performs better at capturing global graph features as reported
16

---

Published in Transactions on Machine Learning Research (03/2024)
Table 1: MMDs distances based on degrees (deg.), clustering coefficients (clust.), and the number of occur- rences of orbits with four nodes (orbit) and their averages (avg.) on datasets of small, simple graphs.
SmolSmaal
Community Small
Model
Deg.4
Clust.
Orbit.
Avgl
Deg.
Clust.,
Orbit.
Avg.
Training
0.022
0.043
0.0062
0.024
0.015
0.026
0.0032
0.015
GraphRNN
0.090
0.220
0.003
0.104
0.080
0.120
0.040
0.080
Auto-reg.
GraphDF
0.04
0.13
0.01
0.060
0.06
0.12
0.03
0.070
GraphARM
0.019
0.017
0.010
0.015
0.034
0.082
0.004
0.040
EDP-GNN
0.052
0.093
0.007
0.051
0.053
0.144
0.026
0.074
One-shot
GDSS
0.021
0.024
0.007
0.017
0.045
0.086
0.007
0.046
DiGress
0.015
0.029
0.005
0.016
0.047
0.041
0.026
0.038
Ours
DGAE
0.021
0.041
0.007
0.023
0.032
0.062
0.0046
0.033
Table 2: MMDs distances based on degrees (deg.), clustering coefficients (clust.), and the number of occur- rences of orbits with up to four nodes (orbit) and their averages (avg.) on the Enzymes dataset.
Enzymee
Model
Deg.J Clust. Orbit Avgl
Training set 0.004
0.021
0.001
0.008
GraphRNN
0.017
0.062
0.046
0.042
Auto-reg,
GraphDF
1.503
1.061
0.202
0.922
GraphARM
0.029
0.054
0.015
0.033
EDP-GNN
0.052
0.093
0.007
0.051
One-shot
GDSS
0.026
0.061
0.009
0.032
DiGress
0.004
0.083
0.002
0.030
Ours
DGAE
0.020
0.051
0.003
0.025
17

---

Published in Transactions on Machine Learning Research (03/2024)
by the FCD and the NSPDK metrics larger molecules. It remains competitive at capturing the local node-edge interactions necessary to generate valid molecules, even though some baseline models are better in this respect.
Table 3: MMD distances based Neighborhood subgraph pairwise distance kernel (NSPDK) and Fréchet Net Distance on the Qm9 and Zinc250k datasets.
Qm9
Zinc
Model
NSPDK
FCD Valid. %o NSPDK FCD, Valid. %0
GraphAF
0.021
5.53
74.4
0.044
16.0
68.5
auto-reg.
GraphDF
0.064
10.92
93.88
0.177
33.5
90.6
MoFlow
0.017
4.47
91.4
0.046
20.9
63.1
One-shot EDP-GNN
0.005
2.68
47.5
0.049
16.7
83.0
GDSS
0.003
2.90
95.7
0.019
14.7
97.0
DiGress
0.0005
0.36
99.0
0.082
23.1
91.0
Ours
DGAE
0.0015
0.86
92.0
0.007
4.4
77.9
5.3.2 Generation time
Finally, we use the Zine dataset to assess our model's generation time. We compare against the best auto-regressive model (Luo al., 2021) and the best one-shot models GDSS and DiGress (Jo et al., 2022; Vignac et al., 2023). We compute the clock time to generate 1000 graphs in the rdkit format one GeForce RTX 3070 GPU and 12 CPU cores. We report the result in Table 4.
Our model not only presents very high performances at modeling graph distribution, but it is also several orders of magnitude faster at generation than the baseline models.
Table 4: Generation time in seconds to generate 1000 graphs with models trained on molocular datascts..
Time (s)
Models
Qm9 +
Zinc
GraphDF
3791.67
3859.23
GDSS
28.07
300.44
DiGress
54.01
799.43
DGAE (Ours)
0.33
1.80
5.4 Ablation study
We two types of ablation studies. First, we empirically observe the effect of the various types of data augmentation. We then evaluate the effect of the codebooks and partition size. For all the ablation studies, we use the Zinc250k dataset, which is by far the largest dataset used for the experiments (regarding the number of graphs).
5.4.1 Graph Features Augmentation
Numerous feature augmentation strategies for graphs have been proposed to overcome the inherent con- straints of GNNs (sce Scction 4.1.2 and 2.4). Using such feature augmentations for graph generation has become a prevalent practice. However, we are not aware of any work evaluating these methods in generative modeling. The primary objective of these auxiliary features is to enhance the identification of graph fea- tures. convenient way evaluating a model's capability of capturing graph features and substructures is by assessing its reconstruction accuracy. Thus, we evaluate the methods for synthetic graph feature methods based their reconstruction performance.
18

---

Published in Transactions on Machine Learning Research (03/2024)
Therefore, we have conducted comparative analysis of several feature augmentation techniques, including spectral embedding, cycle counts, random features, and our p-paths method. The impact of each augmen- tation is evaluated independently against a scenario where no augmentation is applied. Further, we assess the impact of omitting one method compared to the combined use of all techniques. In figure 3, we report the reconstruction loss over 20 epochs.
Effect of the feature augmentation
Effect of the feature augmentation
0.012
0.005
no feature augmentation p-paths
no prpatts
0.010
spectral embedding
no spectral embedding randem features E-cyclec
Reconstruction loss
random features
Reconstructton loss
0.000
c cycles
0.003
0.006
0.007
0.004
0,001
0.000
12.5
20.0
0.000
5.0
12.5
20.0
Epochs
Esochs
Figure 3: Left: The effect of each feature augmentation scheme on the reconstruction loss compared to the absence of feature augmentation. Right: The effect of removing each feature augmentation scheme compared to using all of them. The lines represent the average over three runs, and the shaded area is the standard deviation.
When used alone, the reconstruction loss curves suggest that the spectral embedding and our p-paths method dramatically improve the reconstruction loss. Conversely, the cycle count and random feature techniques demonstrate negligible, if any, effects. Upon removal of single method, the performance degradation is most noticeable in the absence of our p-path method. The effect is also evident for the random and spectral features. Conversely, the cycle count contributes marginally, if at all, to the overall effect. We conclude that our feature augmentation method is effective at enhancing MPNN as feature extractor on the node neighborhood, showing competilive performance against the computationally expensive hod using spectral features.
In table 5, we present the impact of the feature augmentation on the pre-processing time and the training times. We measure the pre-processing time, i.e., the time to transform the dataset stored numpy object to a pytorch-geometzic dataset, including the augmented features. We also compute the time for 1000 training iterations, i.c., 1000 paramcter updates. For both, we compare our p path method alone against the dataset without feature augmentation and with the spectral feature. We report the clock time measured on the Zinc250k dataset with one GeForce RTX 3070 GPU and 12 CPU cores.
Table 5: Pre- Pre-processing and training time for some feature augmentation methods in seconds. We compute the training time over 1000 training iterations.
Time (s)
Methods
Pre-proccssing 4. Training $
No augmentation
74.81
38.19
Spectral feature
151.22
38.09
P path feature (ours)
231.18
38.71
We observe that the pre- proccessc time for our p- path scheme is significantly slower than the pre- pre-prrcessiee processing without fcature augmentation and the method based on spectral features. Nevertheless, the method remains short in comparison to the training time of graph generative models, which typically spans hours, if not days. Regarding the training, all methods are very close, our p-path being marginally slower.
19

---

Published in Transactions on Machine Learning Research (03/2024)
5.4.2 Influence of Codebook Sizes and Partitioning
This section evaluates the impact of varying the codebook and partitioning sizes. We first evaluate their effect on dictionary usage. We then study the effects on the autoencoder training and the prior training, given that the size of the codebook and the partitioning determine both the length of the ruto-regressive sequence (nC) and the dimensionality of the categorical distribution" (m (m+1), For these experiments run on the Zinc250k dataset, we compare ten configurations for various codebook size m and number of partition C, representing three sizes of possible codeword sequences M: 256' 162 44, 1024 32 103, and 4096 - 64 16 8
We first analyze the dictionary usage, reporting the normalized perplexity. The perplexity is a measure of the dictionary usage computed as eH(P()))), where H(((z))) is the entropy over the dictionary likelihood. The closer the distribution is to the uniform distribution, the higher the perplexity. The maximun of perplexity corresponds to the dictionary size M. To compare various dictionary sizes, we use the normalized perplexity 1 ( We present the result in Figure 4. We see that the normalized perplexity slightly decreases as the dictionary size increases. We also see that using single large codebook, in this case, a single codebook of size 4096, leads to codebook collapse.
Dictonary Usage
M 256
M= 1000
0.8
Normalized Perplexity
M 4096
0.4
0.0
256
16a
1024
32
103
40961
64?
16
Codebook size m and number of vectors C: mc
Figure 4: effect of the codebook size and the partitioning on the dictionary usage. We report the normalized perplexity averaged over three runs. The black lines indicate the standard deviations.
We then assess the impact of the codebook size m, the number of partitions C, and the dictionary size (i.c., the number of possible codeword sequences M = m)) on the reconstruction loss of the auto-encoder. We analyze the same ten configurations. For comparison, we also include an experiment without quantization. We then selected the best reconstruction loss on the test set (calculated after each epoch) and reported the average across three runs. Figure 5 depicts the outcome. Our findings suggest that the mumber of possible sequences is critical, particularly when small (256 in our experiment). Conversely, the absence quantization yields almost perfect reconstruction (so that its representation in Figure 5 is barely perceptible). However, the benefits of increasing the number of possible sequences become limited once its number surpasses a certain threshold. In this experiment, the advantage of increasing the mumber of possible sequences from 1024 to 4096 is unclear. Likewise, the partitioning effects become significant when the number of possible sequences is low, with smaller partitions proving more beneficial. However, the impact decreases with larger numbers of possible sequences. In our experiments, we do not see any effect when increasing the number of possible sequences to 4096. In Appendix C, we present similar figures reporting node and edge error rates. With a dictionary size larger than 256, we reach close to perfect reconstruction, with both crror rates lying below 0.001 for almost all configurations.
4The 1412 refers to the end-of--equence token
20

---

Published in Transactions on Machine Learning Research (03/2024)
We finally observe the effect on generation. We do not present the result without quantization because our Tr-arossfrrrrr requires quantized vectors for training. Therefore, our model is not able to perform generation without quantization. We report here the NSPDK metric, but the FCD, reported in Appendix C, presents similar results. The benefits of a large number of arrangements and a small number of partitions vanish. The most significant. observation is the poor result. for m 4096 and C C= 1. We assumne that the drop in performance comes from codcbook collapse observed in Figure 4, which results in high probability of sampling indices out of the distribution. All the other settings result in comparable NSPDK values, which are lower than those of the baseline models. We also observe that the configuration with two partitions (C (C=22) and a codebook size of m = 32 presents slightly better generation metrics than the other configurations.
Reconstruction
Generation
0.007
M4 254
34 1000
0.00
M 1000
M- 226
Reconstruction Loss
0.008
No quant:
6.5
0.005
0.64
NSPOK
0.004
0.003
0.22
0.002
_67
home
26'
Codebook size m and number of vectors C: m
Codebook size and number of vectors C: mf
Figure 5: effect of the codebook size and the partitioning on reconstruction (left) and generation (right). We report the best reconstruction loss and the best NSPDK averaged over three runs. The black lines indicate the standard deviations.
6 Conclusion
Our work introduces DGAE, a powerful generative model for graphs in two stages. In the first stage, we leverage auto-encoder, mapping graphs to sets and recovering graphs from sets. We use permutation- equivariant functions to prevent the difficulties caused by the multiple possible representations of graphs. In the second stage, we sort the sets into sequences and learn them ruto-regressively. In its present stage, we do not see any ethical concerns with our model.
Our model achieves excellent performance at learning graph distributions. Our empirical evaluation shows that it is competitive with the best baseline models. Moreover, our model is an order of magnitude faster generation than competitive bascline models.
Implementing our model, we introduce two innovative techniques: new feature augmentation method and new Transformer architecture for sequential data over two dimensions.
However, our model also presents some limitations. First, our model requires two stages training. This makes the hyperparameter tuning complicated, particularly for hyperparameters that influence both training stages, such as the number of partitions or the codebook size m, for instance. Second, our model cannot perform conditional generation yet.. We hypothesize that conditioning the auto-regressive model might be sufficient. Nevertheless, the elaboration and validation of conditional generation remain topics for future investigation.
Lastly, addressing the challenge of scaling models to accommodate larger graphs is prevalent concern in generative modeling, one that our model does not currently tackle. We have evaluated our model on graphs with up to 125 nodes. Adapting our model handle larger graphs would raise challenges related to computation, memory, and overall model performance. This is another line of research open for further exploration.
21

---

Published in Transactions on Machine Learning Research (03/2024)
Distinct from the two predominant approaches in current literature, i.e., the fully sequential and the fully permutation-qquivaiian methods, our generative model for graphs introduces an innovative and competitive framework. We view its present limitations as promising opportunities for future research.
Acknowledgments
We acknowledge the financial support of the Swiss National Science Foundation within the LegoMol project (grant no. 207428). The computations were performed at the University of Geneva on 'Baobab" and 'Yggdrasil" HPC clusters.
References
V. Arvind, Frank Fuhlbrücc,, Johannes Köbler, and Oleg Verbitsky. On weisfeiler-leman invariance: Sub- graph counts and related graph properties. Journal of Computer and System Sciences, 113:42 59, 2020. ISSN 0022-0000. doi: https///lo..rr//111100///.s
2020.04.003. 20..0.......00.................. URL
Yoann Boget, Magda Gregorova, and Alexandros Kalousis. Graph annotation generative adversarial net- works. In Emtiyaz Khan and Mchmet Gonen (eds.), Proceedings of The 14th Asian Conference on Machine Learning, volume 189 of Proceedings of Machine Learning Reseorch, pp. 16-16. PMLR, 12-14 Dec 2023.
URL https::/proceeddngg: .ml.. http://eweeaaaaa.....eeeee/////e press//189/boget222 html.
Marc Brockschmidt, Miltiadis Allamanis, Alexander L. Gaunt, and Oleksandr Polozov. Generative code modeling with graphs In International Conference on Learning Representations, 2019. URL https:
//openreview.net/oorrreeee aett/forrmmidd-BeeKKAAF.
Bo Chen, Le Sun, and Xianpei Han. Actiio: End-to-End Semantic Graph Generation for Semantic Parsing. ACL 2018 56th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Long Pupers), 1:766 777, 2018. doi: N..1853/V1/P18-1000 1071. URL https:
//aclanthology. www.aaooooooooggg/ee1eeeee org/P18-1071
Zhengdao Chen, Lei Chen, Soledad Villar, and Joan Bruna. Can graph neural networks count In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances Neural Information Processing Systems, volume 33, pp. 10383 10395. Curran Associates, Inc., 2020. URL
substructures?
nww.ips.ccccccoeoeoo httprppss..pppppp..eee/////////e fte/////////////////////////// pdf.
75577667715500004666777781111111
Fabrizio Costa and Kurt De Grave. Fast neighborhood subgraph pairwise distance kernel. In Johannes Fürnkranz and Thorsten Joachims (ods.). Proceedings of the 27th International Conference on Machine Learning (ICMI-10), June 21-24, 2010, Haifa, Israel, pp. 255 262. Omnipress, 2010. URL https://
Nicola De Cao and Thomas Kipf. MolGAN: An implicit generative model for small molecular graphs. arXiv:1805. 11973 [es, stat], May 2018. URL http://arxta: http:///aaxiv..oo/a///////ee//// 11973. arXiv: 1805.11973.
Jiarui Feng, Yixin Chen, Fuhai Li, Anindya Sarkar, and Muhan Zhang. How powerful are k-hop message passing graph neural networks. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (ods.), Advances in Neural Information Processing Systems, 2022. URL https://opeeneve//// www.errreiioonne/fouuurr net/forum? id=nN3aVRQszZa
Nikhil Goyal, Harsh Vardhan Jain, and Sayan Ranu. Graphgen: A scalable approach to domain-agnostic labeled graph generation. In Yennun Huang, Irwin King, Tie-Yan Tüu, and Maarten van Steen (eds.). WWW '20: The Web Conference 2020, Taipei, Taiwan, April 20-24, 2020, pp. 1253 1263. ACM IW3C2, 2020. doi: 00055/3366443333022220 URL https://doi..... org/0..0......... 1ttg//0.114555 1145/3366423 3380201.
Rafacl Gómez-Bombarell, Jennifer N. Wei, David Duvenaud, José Miguel Hornández-Lobtto, Benjamín Sánchez- Lengelin,,,,,,, Dennis Sheberla, Jorge and Alán Aspuru-Guzik. Automatic Chemical Design Using a Data-Driven Continuous Representation of
Timothy D. Hirzel, Ryan Adams,
22

---

Published in Transactions on Machine Learning Research (03/2024)
Molecules. ACS Central Science, 4(2):268-2766 February 2018. ISSN 2374-7943. URL https://dbi.org/ 10. 00.102//acccentaci.. 7b00572. Publisher: American Chemical Society.
Harald Andrés Helfgott, Jitendra Bajpai, and Danicle Dona. Graph isomorphisms in quasi-polynomial time, 2017.
Emiel Hoogeboom; Victor Garcia Satorras, Clément Vignac, and Max Welling. Equivariant diffusion for molecule generation in 3D. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (ods.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 8867 8887. PMLR, 17 23 Jul 2022. URL https: htaps:///proceee https://procceeding. meriiinggmlr..... www.:mlt.....e...m..mm.mmmmmlete waasss/////hhoooooo///// html.
John Ingraham, Vikas Garg, Regina Barzilay, and Tommi Jaakkola. Generative models for graph-basod protein design. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32, Curran Associates, Inc., 2019.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 448 456,, Lille, France, 07 09 Jul 2015. PMLR. URL https:/
mwr.mmlrmme...ee wtt.peese////itfeeeeeeee html.
John J. Irwin, Teague Sterling, Michael M. Mysinger, Erin S. Bolstad, and Ryan G. Coleman. Zinc: free tool to discover chemistry for biology. Journal of Chemical Information and Modeling, 52(7):1757- 2012. doi: 10.1021/ci3001277 URL https://doi.org///
PMID: 22587354.
Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Junction tree variational autoencoder for molecular graph generation. In Jennifer Dy and Andreas Krause (eds.). Proceedings of the 35th International Con- ference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 2323-2332. PMLR, 10 15 Jul 2018. URL https://proceeding: mlr. press/v80/jina html.
Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Hierarchical Generation of Molecular Graphs us- ing Structural Motifs. In 37th International Conference Machine Learning, 1CML 2020, volume PartF16814, pp. 4789 4798, 2020. ISBN 9781713821120. URL httpp:///ithuu..omt// hgraph2graph.
Jaehyeong Jo, Seul Lee, and Sung Ju Hwang. Score-based Generative Modeling of Graphs via the Sys- tem of Stochastic Differential Equations. Proceedings of the 39th International Conference on Machine Learning, 162:10362 10383, 2022. URL https: / /gtgtttttbb... http////aaayyyy////.....tt..... 025144.
aba/
Lukasz Kaiser, Samy Bengio, Aurko Roy, Ashish Vaswani, Niki Parmar, Jakob Uszkoreit, and Noam Shazeer. Fast decoding in sequence models using discrete latent variables. In Jennifer Dy and Andreas Krause (eds.); Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 2390-2390 2399. PMLR, 10-15 Jul 2018. URL https://proceddings mlr. press/v80 http///00/aaaeeeeaaaaa html.
Thomas N. Kipf and Max Welling, Variationa graph auto-encoders. In NewrIPS Workshop on Bayesion Deep Learning, 2016. URL http:///aayesiandeeeearnritt http://///aannnon/////////////// http://16//pappr//BLLL2
Matthew Klawomn and Eric Heim. Generating Triples With Adversarial Networks for Scene Graph Con- struction. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1))6992-6999, apr 2018. ISSN 2374-3468. doi: 10.1609//AAM A3A1 12321. URL https://ojs aaai waaaaai.r//aa..aaa..aaaaa wetgg/indxx.ppp httg://ndx.ppp/AAA... httpp//pp/////raiil/llele view/12321.
Johannes Köbler, Uwe Schöning, and Jacobo Torán. Introduction, pp. 1-4. Birkhäuser Boston, Boston, MA, 1993.
23

---

Published in Transactions on Machine Learning Research (03/2024)
Lingkai Kong, Jiaming Cui, Haotian Sun, Yuchen Zhuang, B. Aditya Prakash, and Chao Zhang. Autoregres- sive diffusion model for graph generation. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 17391-17408. PMLR, httl..
23 23-29 Jul 2023. URL https:// httpss///roceedig..oooe .ml wwt.se/ww......r.eeeeeeeemeeeeee hatssssv222/kooggo
Igor Krawezuk, Pedro Abranches, Andreas Loukas, and Volkan Cevher. (GG)-1gan): A geometric graph generative adversarial network, 2021. URL https://openrevii/ http://pwwaaieeeeeeeeeor//aaal/a wwtt:/orrumiddii/xlloo.....
Matt J. Kusner, Brooks Paige, and José Miguel Hernández-Lobate Grammar Variational Autoencoder. In Proceedings of the 94th International Conference on Machine Learning, pp. 1945-1954. PMI.R, July 2017.
URL https:// /ttp:////ccceeeessoeo mlr. htl..prees/vv//kueee http//////uneeeaaaaaaaaaa.aaaaaa html. ISSN: 2640-3498.
Maksim Kuznetsov and Daniil Polykovskiy. MolGrow: A Graph Normalizing Flow for Hierarchical Molecular Generation. Proceedings of the AAAI Conference on Artificial Intelligence, 35(9): 8234, feb 2021.
ISSN 2159-5399. doi: 10.48550/amma arxiv wtt.aaiw.v........... 2106.05856. URL https://arxiv. wwwpa////gg////////////oo http://11....88666..
Qimai Li, Zhichao Han, and Xiao-Ming Wu. Deeper insights into graph convolutional networks for semi- supervised learning. In Proceedings of the Thirty--ecood AAAI Conference Artificial Intelligence (AAAT-18), pp. 3538 - 3545, Palo Alto, CA, 2018-02. Association for the Advancement of Artificial Intelligence. ISBN 978-1-57735-800-8. 32nd AAAI Conference on Artificial Intelligence / 30th Innovative Applications of Artificial Intelligence Conference / 8thh AAAI Symposium on Educational Advances in Artificial Intelligence; Conference Location: New Orleans, LA; Conference Date: February 2-7, 2018.
Y. Li, W. Ouyang, B. Zhou, K. Wang, and X. Wang. Scene graph generation from objects, phrases and region captions. In 2017 IEEE International Conference on Computer Vision (ICCV), pp. 1270 1279, Los Alamitos, CA, USA, oct 2017. IEEE Computer Society. doi: 10.1109/ICCV URL https: //doi. ieeecomputersocietyy org/10. 1109/ICCV. 2017. 142.
Yujia Li, Oriol Vinyals, Chris Dyer, Razvan Pascanu, and Peter Battaglia. Learning deep generative models of graphs. In ICLR 2018, 6th. International Conference on Learning Representations, feb 2018. URL. attps://openreviiee net net//oruumidd-Hyyddeeb
Renjie Liao, Yujia Li, Yang Song, Shenlong Wang, Will Hamilton, David K Duvenaud, Raquel Urtasun, and Richard Zemel. Efficient Graph Generation with Graph Recurrent Attention Networks. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https:// proceedings wwwpiiinnnnneeurr.peeeeeeeee wwwpuuups.ccpppee/////eeee /ttp////////00112222100000000000
waw.tt1t--bsttrat..... html.
Jenny Liu, Aviral Kumar, Jimmy Ba, Jamie Kiros, and Kevin Swersky. Graph normalizing flows. In Advances in Neural Information Processing Systems, volume 32, 2019.
Qi Liu, Miltiadis Allamanis, Mare Brockschmidt, and Alexander L Gaunt. Constrained Graph Variational Autocncoders for Molecule Design. The Thirty-second Conference on Neural Information Processing Sys- lems, 2018.
Andreas Loukas What graph neural networks cannot learn: depth vs width. In International Conference on Learning Representations, September 2019. URL https:///ppenreview.ct http://wwwwweeeeeoorr/ee///a/aaa
Chengqiang Lu, Qi Liu, Qiming Sun, Chang Yu Hsieh, Shengyu Zhang, Liang Shi, and Chee Kong Lee. Deep Learning for Optoelectronic Properties of Organic Semiconductors. Journal of Physical Chemistry C, 124(13):7048 70667)0004-7760 apr 2020. ISSN 19327455. URL https://pubs: acs wwtp.o///////a//////////a/oooae/ http////aa///...0//aaac.... jpcc. 0c00329.
Youzhi Luo, Keqiang Yan, and Shuiwang Ji. GraphDF: A Discrete Flow Model for Molecular Graph Gener- ation. Proceedings of the 38th International Conference Machine Learning, 139:7192-7203. 2021. URL http:///axxivorr/abbae
01189.
Kaushalya Madhawa, Katushiko Ishiguro, Kosuke Nakago, and Motoki Abe. Graphnyp: An invertible flow model for generating molecular graphs, 2019.
24

---

Published in Transactions on Machine Learning Research (03/2024)
Haggai Maron, Heli Ben- Ben-Hamu, Hadar Serviansky, and Yaron Lipman. Provably powerful graph networks. In H. Wallach, H. Larochelle, A. Beygekimer, F. d'Alché-Bue, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran As- newriiipsss
sociates, Inc., 2019. URL 3004:d077cceeeae30055///11111719 pdf.
neurips. http://....rrrppppppppr.r.rii.aa httppo__fiiee/pppr/001/////
Christopher Morris, Martin Ritzert, Matthias Fey, William L. Hamilton, Jan Eric Lenssen, Gaurav Rattan, and Martin Grohe. Weisfeiler and leman go neural: Higher-order graph neural networks. In 33rd AAAT Conference on Artificial Intelligence, AAAI 2019, 31st Innovaivve Applications of Artificial Intelligence Conference, IAAI 2019 and the 9th AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, volume 33, pp. 4602 4609. AAAI Press, jul 2019. ISBN 9781577358091. doi: 10.1609 aaai. v33i01 4602
URL www. aaai.org.
Kenta Oono and Taiji Suzuki. Graph neural networks exponentially lose expressive power for node classifi- cation. In International Conference on Learning Represenlations, 2020. URL https://openreviiet net//
forum?id=S11dD2EFP
Kristina Preuer, Philipp Renz, Thomas Unlerthiner, Sepp Hochreiter, and Günter Klambauer. Fréchet chemnet distance: A metric for generative models for molecules in drug discovery. Journal of Chemical Information and Modeling, 58(9):1736-1741, 2018. doi: 1..021aacs.ciim.B0e
PMID: 30118593.
Raghunathan Ramakrishnan, Pavlo O. Dral, Matthias Rupp, and O. Anatole von Lilienfeld. Quantum chemistry structures and properties of 134 kilo molecules. Scientific Data, 1(1):140022, August 2014. Number: 1 Publisher: Nature
TSSN 2052-4463. URL https: http:://wwwwwawt.eeteeet www.nattuuecomm http.//aattleee///saaaaaaaaa Publishing Group.
Ali Razavi, Aaron van den Oord, and Oriol Vinyals. Generating diverse high-fidelity im- ages with vq-vae-2. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buce E. Fox, and R. Garnett (eds.), Advances in Newral Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https://proceeddnne wtw.oeeeddinnn.uuuu........ (adoffaa77aaltbbeffffccccccc
neurips httphippp.cpppppprreeeeee http://ccccppppp....eep/p/p///// fiDes//pper//219//1ll
Paper. paf.
Bidisha Samanta, Bidisha@iitkgp Ac In, Abir De, Vicenç Gomez, Pratim Kumar Chattaraj. Niloy Ganguly, Manuel GonezzRodr,guez, Jana Gourhari, and Kumar Chattaraj. NEVAE: A Deep Generative Model for Molecular Graphs * Journal of Machine Learning Research, 21:1-33, 33,, 2020.
Franco Scarselli, Gori Marco, Tsoi Ah Chung, Hagenbuchner Markus, and Monfardini Gabriele. The Graph Neural Network Model. JEEE Transactions on Neural Networks, 20, jan 2009. URL https://ieeexploor ieee. wwt..orgg/ooooooo... htg//document4470002
Chence Shi, Minkai Xu, Zhaocheng Zhu, Weinan Zhang, Ming Zhang, and Jian Tang, Graphaf: a flow- based autoregressive model for molecular graph generation. In International Conference Learning Representations, 2020. URL https://openreeven met/forrmmiddS1eet
Martin Simonovsky and Nikos Komodakis. GraphVAF: Towards Generation Small Graphs Using Varia- tional Autoencoders. arXiv: 100.00800000480 [es], February 2018. URL http: http://arxxw.orr//bbbst.. wttg//s////////////////////////e 03480.
arXiv: 1802.03480.
Aaron van den Oord, Oriol Vinyals, and koray kavukcuoglu. Neural discrete representation learning. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Newral Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Transformer: Attention is all you neod. Aduances in Neural Information Processing Systems 30, pp. 5998 6008, 2017. ISSN 10495258.
Clément Vignac, Andreas Loukas, and Pascal Frossard. Building powerful and equivariant graph neural networks with structural message-passing. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and
25

---

Published in Transactions on Machine Learning Research (03/2024)
H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 14143-14155 Cur- ran Associates, Inc., 2020. URL. a32d7 3527707mee01001110000777662222p2
Clement Vignac, Igor Krawczuk, Antoine Siraudin, Bohan Wang. Volkan Cevher, and Pascal Frossard.
Digress: Discrcte denoising diffusion for graph generation. In The Eleventh International Conference
Learning Represenlations, 2023. URL http:://pppeeeeeeeeeeeeeerrrrrrr
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanic Jegelka. How powerful are graph neural networks? In International Conference on Learning Represenlations, 2019. URL https:///ppenreviirt htw.ppparrrrriee..ttttoororriiao net/forummid= ryGs61A5Km.
Minkai Xu, Alexander Powers, Ron O. Dror, Stefano Ermon, and Jure Leskovec. Geometric latent diffusion models for 3D molecule generation. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 38592- 386100 PMLR, 23 29 Jul 2023. URL https://proceeding.. mlr /rees//v02//ae htt0/xu2nn.htt html.
Carl Yang, Peiye Zhuang, Wenhan Shi, Alan Luu, and Pan Li. Conditional Structure Generation through Graph Variational Generative Adversarial Nets. In Advances in Neural Information Processing Systems, volume 32, 2019.
Jiaxuan You, Rex Ying, Xiang Ren, William Hamilton, and Jure Leskovee. GraphRNN: Generating Realistic Graphs with Deep Auto-regressive Models. Proceedings of the 35th International Conference on Machine Learning, pp. 5708 5717. PMLR, July 2018. URL https: http://pooceeee...e.eee/ mlr, gress/v00youl.. html. ISSN: 2640-3498.
Scongjun Yun, Minbyul Jeong, Rachyun Kim, Jacwoo Kang, and Hyunwoo J Kim. Graph trans- former nelworks. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Newral Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https: https://proceedingg neurips. wwtrripssccppppppiee .dff.
htt.ar/ppppcppppppppeeeeeeeeeeee httprppprrrfisssppprrrrr/ea//eee /t1es/ppppr/2019/f/ll/
Chengxi Zang and Fei Wang. MoFlow: An Invertible Flow Model for Generating Molecular Graphs. Pro- ceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 10: 617-626, aug 2020. doi: 0011455/333448664403144 URL attps://di: acm org//oi//0/ 11041/34/339496. 3403104.
Adrian Laúcucki, Jan Chorowski, Guillaume Sanchez, Nanxin Marxer, Ricard andChen, Dolfing Hans J.G.A., Samoer Khurana, Tanel Alumãc, and Antoine Laurent. Robust training of vector quantized bottleneck models. wwXYi:2005.08520 fes, stat/, May 2020. URL http:///aaxvv..///a///////
26


Reviews:
[{"review_id": "zHk4HvtS1y", "reviewer": "Reviewer_uXMb", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall the paper is well written and easy to understand. The description of the problem, state of the art, and solution is clear and choices are well motivated throughout the text.\n\nThe experiments are convincing enough that this method works, although generic graph generation is not my area of expertise enough to comment on the choice of baselines in the simple graphs section. I was surprised not to see any methods from 2023 compared in the paper - given the pace of the field I would usually expect something new to have occurred in the last year that is of relevance.\n\nIn the experiments on molecular graphs, perhaps some additional references could be included. For example:\n- Equivariant Diffusion for Molecule Generation in 3D\n- Geometric Latent Diffusion Models for 3D Molecule Generation\nare both methods that perform well on the QM9 task, one of them performing better than the method in this paper in fact.\n\nThe ablation studies in the end of the appendix are good to see and clearly demonstrate benefits of the different components of the model", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a new method for the generative modeling of graphs.\n\nThis is achieved with an autoencoder model with enforced clustering in the latent space to produce representations of the graph, and an auto regressive model with a new transformer architecture that models the latent distribution as the generative component of the model.\n\nThe method is compared on a number of tasks with reasonably convincing results, alongside ablation studies of the method.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I would like to see a better discussion of related work in the experiments section, comparing to more recent works, and state of the art methods that are of a different type to this work. There are particularly some related diffusion works not discussed.\n\nOne ablation not included that I think should be is not using a codebook at all. It would be very interesting to see if this feature actually helps, as it is not evident to me that it will.\n\nIt is not clear to me that the autoregressive model will generate sorted lists of codebook keys - could the authors comment on this?\n\np 13. seciton 4.4 paragraph 2, states \"there are 4 reasons...\" but only 3 are listed as \"secondly\" is skipped.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "gltwaLvTEd", "reviewer": "Reviewer_2GRG", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Main Strengths\n\n1. The paper combines different but prominent strategies tackling the equivariance issues raised when working with graph structures, with impressive performance.\n\n2. Introduces a novel 2D transformer architecture modeling 2D sequences.\n\n3. It offers a new technique for graph augmentation, named p-path and importantly validates graph augmentation techniques in an ablation study showcasing the importance of such procedures.\n\n4. It offers competitive performance against prominent baselines based on multiple networks while massively outperforming the baselines in the generation time.\n\n## Weaker Elements\n\n1. The paper does not contain any more recent graph diffusion generative models but rather an older model from 2019. They could potentially introduce more recent works such as Digress ([link](https://openreview.net/forum?id=UaAD-Nu86WX)).\n\n2. The defined model produces a two-stage training which, as also mentioned by the authors, can increase the training complexity, as it defines multiple hyperparameters.\n\n3. It does not offer any concrete ideas on how to potentially scale the model to large graphs, a very important aspect.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces the **Discrete Graph Auto-Encoder (DGAE)**, a novel approach that addresses challenges related to the equivariance property in graph generative models. DGAE employs two key strategies to tackle these issues. First, it projects a graph into a set using a discrete and invariant encoder, thereby eliminating the need for costly graph matching procedures.\n\nTo enhance the latent space, DGAE incorporates a clustering procedure, facilitating the learning of informative latent embeddings. This is achieved through a codebook containing various codeword sequences. Notably, the use of sorted sets enables the autoregressive learning of unique representations.\n\nThe decoder in DGAE incorporates a specialized **2D attention mechanism** designed for modeling sequences in two dimensions. Additionally, the paper introduces various feature augmentation techniques and empirically validates these strategies through multiple experiments on diverse datasets. The results are compared against various baselines, providing a comprehensive evaluation of the proposed model.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The paper is well-written, addressing my questions and concerns about the model's training and technical aspects through multiple experiments. I recommend the authors to explore recent graph diffusion generative models for comparison with the current state-of-the-art. Additionally, it would be beneficial for the authors to provide a more detailed discussion and ideas on how to scale their proposed model to larger graphs. Overall, the paper is very good.", "broader_impact_concerns": "No particular need for a Broader Impact Statement,", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "2SOWh7Ss3T", "reviewer": "Reviewer_xt8A", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strength**\n\n1] The way in which you discuss related work in Section 3 is rather nice. \n\n2] In general the mathematical notation is often well-defined and clear. \n\n3] In Section 5.1.2 you provide a good discussion of the shortcomings of different metrics used for graph generation on molecular datasets and propose a seemingly improved evaluation scheme. \n\n**Weaknesses**\n\n1] In the abstract and introduction you heavily focus the discussion on the graph isomorphism problem. I am not convinced that this is the best aspect of your model to place the focus on. The fact that GNNs are permutation equi-/invariant is well-known and the resulting advantages have also been discussed in many papers. I therefore, find your initial discussion of the two possible approaches to modelling graphs (BFS and GNNs) not to be overly insightful. I suggest you shift the focus from graph isomorphism initially.\n\n2] I am not convinced of the necessity of the clustering step in your architecture. You make several comments speaking to its advantages such as, \n\n- \"To ease the modeling of the latent distribution, we remove unnecessary information and reduce the size of the latent space by clustering the node representations.\" and equally, \"The primary objective of clustering is to ease the learning of the latent distribution. We achieve this thanks to two main consequences of clustering: first, by removing superfluous information\". It is unclear to me whether you truly remove \"unnecessary information\" and I don't see evidence in your experiment substantiating this statement.\n\n- \"clustering also ensures better coverage of the latent space\". I don't see how reducing the number of utilised embeddings from a set of embeddings containing one embedding per node to a set of embeddings containing one embedding per centroid ensures better coverage of the latent space.\n\n3] It seems to me that your model should be presented in greater detail and more clearly. (see Point 2 in the Requested Changes for more detail.)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In the submitted manuscript, the authors propose a model for graph generation. Their proposed DGAE model combines a graph autoencoder with a clustering step, a sorting step of the learned embeddings, and a Transformer model that is fit to the sorted embedding sequences. They present a range of experiments on five different datasets in which they demonstrate good performance of their proposed DGAE model.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Generally speaking, it seems to me that many changes are required for the submitted manuscript to be ready for publication. It is my impression that the newly introduced ideas need to be presented more clearly and in particular your model needs to be better described. I now present my proposed changes in detail.\n\n\n1] It would significantly strengthen Section 3.2 if you could discuss Graph Autoencoders [1] and related literature in greater detail. Given that your model is also a graph autoencoder, this would help place your model into context.  \n\n2] It would be better if you could describe your model in greater detail. I feel I may have misunderstood certain elements of your model despite my best efforts to grasp the sequence in which you apply the different model components. This would certainly help my understanding of its different components. \n\n2.1] It is unclear to me how and for what reason the partitioning of the learned embeddings is performed. According to which criterion do you partition the learned representations and why do you see the need for partitioning? It would be great if you could describe this in greater detail in your paper. \n\n2.2] I was unable to understand why you deem the clustering step in your architecture to be necessary. i) Without experimental evidence, I am not convinced that it removes \"superfluous information\". When clustering it seems to me that you may be losing valuable information. It would be great if you could clarify this part in the paper by stating the practical reason that necessitates the use of a clustering operation. ii) In particular, I wonder whether it would be possible to run our architecture without the clustering step. Such an experiment would help motivate your use of clustering. iii) It might furthermore be interesting to explore learnable alternatives to clustering such as a DiffPool layer [1]. \n\n\n2.3] I have several questions regarding your choice of decoder. i) Using an MPNN as a decoder in a graph autoencoder is not a standard choice. It is more common to work with activated inner products [2], i.e., \\sigma(ZZ^T), as a decoder. It would be good if you could motivate your choice of decoder and potentially run an ablation experiment for it. ii) In the decoder it is unclear to me how you get back to having $n$ representations when it follows the clustering step. To me, it seems that the clustering step reduced the number of represented objects the number of centroids. iii) It is furthermore unclear to me why you chose to work with a complete graph instead of the original graph in the decoder. iv) If nodes are represented via their corresponding clustering centroids then they cannot be distinguished in a complete graph. Is this indeed the case?\n\n\n2.4] In your feature augmentation scheme you draw virtual edges between nodes that are connected by paths of length up to $p.$ i) It would be interesting if you could discuss whether these edges are then considered in addition to the original edges in the GNN encoder and whether the virtual edges distinguish themselves from the originally present edges in any way. ii) Since the computational complexity of GNNs scales linearly in the number of edges it seems to me that your proposed feature augmentation scheme might add a significant computational expense to the model. It would be good if you could alleviate this concern by not only recording the infernce but also the training and potential pre-processing time to your experiments in Section 5. iii) While this is certainly not required for publication, it may be a nice addition to your paper to make a theoretical statement on what kind of graphs, that are indistinguishable using traditional MPNNs, can be distinguished as a result of your feature augmentation scheme.\n\n2.5] In Section 4.3 you state that you sort node representations by index. i) It is unclear to me which indexes you use in your sorting. Are you sorting by first entries of the learned embeddings? ii) While you discuss that the clustering step in your architecture is not differentiable and how this issue is resolved in your training, you do not mention the fact that the sorting operation you perform is not differentiable either. It may be good to also discuss how you manage to backpropagate through the sorting operation. \n\n2.6] In Section 4.3.2 you state that you \"linearly project these vectors so that all the input vectors have the same dimensiond model\". It is unclear to me why there were input vectors of unequal dimension at this stage. It might help explain your model more clearly if you could specify the source of these vectors and why there are vectors of unequal dimension.\n\n\n3] In Section 5.3 it would also be good to mention the number of parameters of the different compared models. It seems to me that your model involves many parameters, which may result in an unfair comparison if fewer parameters were fitted for your baseline models.\n\n4] It would be interesting if you could include the recent DiGress model [3] in your experiments to place your performance in the context of recent high-performing models. \n\n5] Minor Changes:\n-In the abstract the formulation \"and there is no known fast algorithm.\" is a bit ambiguous and out of context. On first reading, it's unclear what kind of algorithm you are referring to here. \n\n-It is unclear what kind of graph generator you draw your random graph from here \"we refer to the random graph with G\" and I do not see where these random graphs are used later on. So, I would either specify that you consider the Erdos-Renyi model, or a comparable one, or to remove this statement. Similarly, I do not understand what the definition of the probability of a graph in Section 2.2 adds to your paper. I do not see it being used later on and it could confuse certain readers. \n\n- The statement \"algorithms that uniquely sort any graph\" is imprecise. I think you are referring to algorithms that uniquely sort nodes in a graph. \n\n- It would be nice to add a few references to Section 2.3.1 to more firmly root your statements in the literature. In general, would it be nice to point to specific papers using BFS in graph generation early on in your introduction to ensure that readers become aware that this was indeed an approach that was proposed and successfully used. \n\n- The following description of GNNs could easily be misunderstood \"Since the cardinality of the input and the output of such function must match, GNNs are said to be ’flat’.\" While indeed the number of learned representations corresponds to the number of nodes throughout the architecture (for the majority of GNNs), the dimension of each learned representation often varies at the different layers. \n\n- When you say that \"Our solution is related, but not identical, to the recent work presented in Feng et al. (2022).\" it would be nice to state specific differences between your approaches to make your statement more readily credible to the reader. \n\n- In Equation (10) you have the notation $z_{<i,1}.$ I am unfamiliar with the notation $<i$ in your subscript. Could you please define this?\n\n- The formulation \"we introduce a virtual node (z 0,1 , ..., z 0,C ) made of vectors of zeros.\" is unclear to me. How can a single node be \"made of\" several vectors?\n\n- It would be good if you could more clearly define what \"orbits with four nodes\" are.\n\n- In Section 5.3.1 you state that your model \"remains competitive\" in the generation of valid molecules, while it performs almost 20% worse on the Zinc dataset. It seems to me that your discussion of these results should be weakened to be more realistic. \n\n- In Figure 3 the legend is rather small on a printed copy and the colour scheme makes it challenging to differentiate the different lines. \n\n\n\n[1] Ying, Zhitao, et al. \"Hierarchical graph representation learning with differentiable pooling.\" Advances in neural information processing systems 31, 2018.\n\n[2] Kipf, Thomas N., and Max Welling. \"Variational graph auto-encoders.\" . NeurIPS Workshop on Bayesian Deep Learning, 2016.\n\n[3] Vignac, Clement, et al. \"Digress: Discrete denoising diffusion for graph generation.\" ICLR, 2023.", "broader_impact_concerns": "I do not have any concerns about this work that would necessitate a broader impact statement.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper was reviewed by three reviewers. The reviewers raised concerns about the significance of the empirical results since the proposed model was not compared against any recent method. The reviewers also complained about the absence of an ablation study to investigate whether the clustering/quantization component actually helps. One reviewer also expressed concerns about the clarity of presentation of the approach and requested several changes. Most of the reviewers' concerns were addressed in the revision, and all three reviewers are in favor of accepting this paper. Two reviewers recommended "Leaning Accept", and one reviewer recommended "Accept". I am recommending accept with a minor revision, and I request that the claim discussed above (that the proposed model outperforms state-of-the-art models ) is modified in the final version of the manuscript. Also, while reading the paper, I noticed that the authors use \cite instead of \citep throughout the manuscript and that there are a few typos, e.g., page 1: "and mitigate their" -> "and mitigates their". I expect the authors to fix those presentation issues in the final version.
Decision: Accept (poster)